
Thoughts on cost evaluations
Our world is filled with cost evaluations. Under the conventional assumptions that dominate most governments’ thinking these days, decision-makers are always seeking to justify new spending based on the idea that it will provide some sort of long-term cost saving to government and/or economic benefit to society. In terms of autism research and supports, if you are planning an intervention program and want somebody to fund it, it’s pretty much essential to include a cost evaluation.
However, I have something of a rant on the topic, because when you look at these cost evaluations, there are so many problems spanning across all sorts of different levels. To be clear, plenty has been written about this already – but I often see critiques that focus on only a few issues, whereas I’m trying here to be a bit more integrative, although I am sure there are other issues I’m forgetting or haven’t thought of.
I should admit up front that this is not my area of research, so I apologise for whatever errors or misrepresentations I have probably committed, and for my presumption in commenting on it at all…
Autistic people as a burden
Is autism a burden, or the other way around?
First of all, the cost evaluations often tend to imply that what we’ve historically thought of as the “disorder” and “deficits” of autistic people creates costs for society. For example, because autistic people are often unemployed or underemployed, autism is a cost. Or because autistic people may require more support from caregivers, preventing them from doing other things like working for more money, autism is a cost.
As many autistic and disabled advocates have pointed out, this reflects a rather biased and frankly offensive outlook. You could just as easily argue that neurotypical and abled people are a cost to society! After all, autistic and disabled people are part of society and face all kinds of costs as a result of the way that society has been set up by and for this majority: effects of discrimination, inaccessible and overwhelming environments, exclusion, and so on. It’s not like autistic people asked to be discriminated against.
Oddly, some people seem to draw a line between autistic people with generally-lower support needs and those with generally-higher needs, suggesting that these sorts of neurodiversity-aligned ideas only apply to people with generally-lower needs. This is odd, since if anything, people with higher needs are even more vulnerable to discrimination and exclusion and the impacts and costs they impose.
Who gets the money?
Moreover, there’s another issue that’s very similar to the first, but I think subtly different.
There is often an underlying assumption in these cost evaluations that we are somehow spending money on autistic people. That is, we’re taking other people’s hard-earned tax dollars and spending it on autistic people. Autistic people are recipients of society’s generosity, or even a burden on society.
Framing vulnerable minorities as burdens can be dangerous – consider how the Nazi regime justified mass murder of disabled people by describing them as “useless eaters.” Moreover, it’s empirically problematic. Some autism and disability spending is actually going to autistic and disabled people, and can by spent by them on whatever they want, but this is usually a small amount of money. Disability benefits in Canada are a pittance, and the situation in the United States is even worse. Even though disabled people face additional barriers that mean they probably need more money than others, those prevented from working by their disabilities get much less than an ordinarily-livable wage.1
Instead, lots of “autism spending” goes to usually-non-autistic people and their organisations. It goes to professionals and clinicians, paraprofessionals, support workers, education systems, and so on. Of course, these people often do provide vitally-needed services and supports, and I certainly don’t want to imply that the autistic community opposes autism supports. However, there are certain kinds of autism spending that can sometimes be an unwanted imposition. For example, surveys suggest a lot of interventionists today are trying to normalise neurodivergent behaviours even when they are merely different, not harmful or disruptive. This is fundamentally contrary to what autistic advocates in the neurodiversity movement and autistic community want, and thus, clearly isn’t being driven by them. Rather, a lot of this money may in part reflect the vested interests of an autism industry.
Counting costs and benefits
Scope
Cost evaluations can vary greatly in their scope. For example, does the evaluation only examine spending on the program itself as a cost? Or does it also consider other kinds of costs, such as in terms of individual and family earnings?
This is especially important because many autism interventions these days are parent- and carer-mediated. Ideally, this means that one just quickly teaches parents some useful skills so that when they are interacting with their children they can be more responsive caregivers and support their children’s wellbeing more effectively. However, there’s always the risk of effectively turning parents into interventionists, which could be awfully time-consuming and stressful, and might force them to reduce their involvement in the labour market. And that’s assuming that they are in a secure enough position to be able to do things like reducing work commitments.
If an evaluation looks at things from the perspective of an agency funding it, ignoring these financial and non-financial implications for families or individuals, one might end up thinking that a program that relies on parent effort is tremendously cost-effective. One might think that hiring experts to do anything more than consult and train is wasteful. But that would obviously be a very narrow outlook.
Apples and oranges
In a cost evaluation, we care about money. But how does one include things that are qualitatively different from money? What exactly is gaining an extra IQ point worth in terms of dollars (or pounds, yen, crowns, whatever the case may be)? What’s the monetary value of communication methods that allow you to share your thoughts and needs? How do you translate the quality of the parent-child relationship into money? How much is mental health and wellbeing worth?
I suppose you could, if you really wanted to, use economic utility functions to figure out how much people are willing to pay for these things – but I’m pretty sure that such an endeavour would be deeply wrongheaded. For starters, there are all kinds of issues:
- People have differing goals.
- People might say they’d do one thing when they’d really do another – what is their real preference?
- People may have a differing ability to pay – is the happiness of a billionaire worth more than the happiness of a regular person?
- You can’t actually buy something like mental heath or happiness. You can make decisions that may affect these outcomes, but also affect many other things.
And ultimately, even if you could overcome these issues, why would you even want to put a price on these things, especially in an impersonal cost-benefit evaluation? Intuitively, I think most people have a strong intuitive sense that we don’t want to reduce human lives to cold dollar calculations.
But when one is doing cost evaluations, one might either decide to simply ignore things like human wellbeing, or one might put them into an oversimplified framework that allows one to reduce people to numbers. Evaluations sometimes use metrics like “quality-adjusted life years” or “QALYs,” and “disability-adjusted life years” or “DALYs,” whereby some people’s lives are judged more valuable than others based on their perceived quality. I, for one, am hard-pressed to think of anything more arrogant than to deem oneself capable of comparing people’s lives and numerically rating their respective worth!
Besides the inherent discomfort that I think we should have when we reduce human lives to these sorts of quantitative metrics, QALYs/DALYs are incredibly vulnerable to biases. For one thing, there’s a widespread assumption in this literature that autistic lives are less valuable, and while there’s no denying that autistic people face many challenges, there can also be incredibly positive aspects to being autistic. For another, what exactly would be making autistic lives less valuable? When we calculate QALYs, are we simply basing them on the “severity” of autism characteristics (which, of course, are not acceptable intervention targets in the neurodiversity movement)? Or IQ? Language? Communication generally? Mental health? When we award ourselves this extraordinary power of judging the worth of human lives, on what grounds do we do so?
Perhaps, rather than trying to convert human lives into economic value, or come up with crude metrics allowing us to rate the value of a life, we just need to describe the actual outcomes we might get for a given investment in human terms. Then we can decide whether they are worth it.
Time
The other issue is that we need to understand the outcomes, costs and benefits of a program, but let’s say that it is an early-childhood program: perhaps one of those parent-mediated ones I mentioned earlier. In theory, many of these programs are supposed to take advantage of early neuroplasticity in ways that have cascading effects and will change people’s lives with repercussions for decades to come.
Now, we may not want to wait decades to figure out what the outcome of our intervention is. Indeed, even if we did wait decades, society would probably have changed so much in the meantime that the results would no longer be exactly the same in younger generations anyway. Moreover, it would be unethical to deny people an intervention we think is probably helpful and that they could otherwise obtain, so we can’t always compare the intervention to nothing. Instead, we are often comparing one intervention to another intervention(s), often whatever is normally delivered in the community. If outcomes in each condition are similar, we might not know if that means the interventions are all ineffective or if they are all highly effective.
To be clear, I’m not saying that we shouldn’t do long-term follow-ups of autism interventions. In fact, I think they are desperately needed. We need to know not only if these interventions have long-term benefits, but since some autistic advocates have described experiences of being harmed and even traumatised by some interventions, we also need to examine whether they cause harm. But most of the time, we won’t have a great understanding of what our interventions do in the long term. This means that cost evaluations can either ignore long-term effects, which takes us back to the scope problem and distorts our results, or we can try our best to guess even though we could be spectacularly wrong.
The multiplier effect
In the Great Depression, governments all over the world flailed desperately as they totally failed to stem job losses, panic and collapse. Many people had an intuition that an orthodox, classic-liberal economic response of cutting government spending and letting the currency deflate in value seemed to be hurting more than it was helping, but why? One key reason is the Keynesian multiplier effect. When the government spends money on something – for example, autism services – it doesn’t stay put. Let’s say that the occupational therapist Philippe was struggling to find clients, but when our newly funded program hires him, his income increases. Philippe starts spending more of this money: for example, he eats out at restaurants, supporting the local economy. Let’s say this contributes a bit more towards helping a local restaurant hire a young worker, Mila, helping with youth unemployment and allowing Mila to save some money for her education. Thus, the direct impact of giving Philippe the money is multiplied. As people spend more money, there’s more money sloshing about, so they can spend even more money, etc. Spending can be stimulus.2
Now, this multiplier effect often falls outside the scope of cost evaluations, or is dealt with in only a very vague and superficial way. But it’s as complicated as it is important. Depending on how one spends money, different kinds of programs might have very different multiplier effects. Consider these examples:
ABA in the United States
In the United States, the main autism intervention is Applied Behaviour Analysis (ABA). Americans rely heavily on largely-privatised3 medical insurance to fund ABA coverage, which is mostly provided by private agencies. These are often for-profit, and there’s a lot of private equity moving into the area, which is supposed to result in profits for wealthy investors. On the other hand, much of the boots-on-the ground service is delivered not by board-certified behaviour analysts (BCBAs), but by behaviour technicians under their (sometimes loose) supervision. Behaviour technicians only need about forty hours of training (and the training content may not be great either), so the job is very much the sort of poorly-paid thing that young people do temporarily as a stepping-stone to something better.
Leaving aside how this model might affect the quality of ABA itself,4 this model means that some of the money is going to the private equity investors and other parts to the young technicians who don’t have a great deal of money.5
How might these young behaviour technicians use the money? Because they probably don’t have high incomes, it seems likely they are going to be spending it, putting us firmly on the “demand side” of the economy. Maybe it helps them pay for their education (hopefully encouraging university investment in staffing and class delivery), and pay for more and better-quality food (supporting grocery chain and cafe profits, but also shop clerks, farmers, etc.), and so on. What about the private equity investors? This is the “supply side” of the economy. In our current economy, given the extraordinary expansion of the capital supply that economists like Thomas Piketty have documented, there’s actually quite a lot of money on the supply side already. Indeed, there seems to be a growing imbalance between the demand side and the supply side. Now, maybe some of this private equity is regular people’s retirement savings, which will be saved and then later moved back to the demand side. However, it’s also possible the return on investment will simply be used to generate even more money. While this could theoretically take the form of productive investment, in today’s economy the money might be more likely to end up being used for rent-seeking6 or speculation.7
The NDIS in Australia
The National Disability Insurance Scheme in Australia is similar to the American system in some ways. For example, both systems rely heavily on private service providers, including for-profit entities. However, there are key differences.
For one, although NDIS funding can be used to pay for nonexpert coaches and support workers, when it is used to fund formal intervention, this has tended to be delivered by clinicians such as occupational therapists and speech-language pathologists, who are much better trained and therefore make more money than behaviour technicians. What might these people spend their money on? Perhaps, rather than buying better groceries, they would instead be acting like Philippe and using any extra money they earn to dine in restaurants? Perhaps, being able to meet their everyday expenses, they would save more of their money and thus move it onto the supply side? How will this change the operation of the multiplier effect?
Another difference is that the NDIS was introduced somewhat abruptly and in a situation where numbers of people being identified with some key disabilities – especially autism – were increasing. Thus, the NDIS dramatically increased demand for supports, but it takes time for more professionals to be trained to meet that demand. In situations of scarcity, prices rise. Trained professionals might be able to command a premium for their services, while others scramble to fill gaps. On the other hand, the NDIS is ultimately reliant on government funding, and when a market is dominated by a single purchaser like a government, there’s a lot that purchaser can do to impose limits on how much it will pay. How do all of these institutional and contextual dynamics change the operation of the multiplier effect? There seems to be no reason to believe that the multiplier effect will behave the same way in the American ABA industry as in the Australian NDIS.
Moreover, the Australian government is currently trying to reduce spending on disability supports and remove people – especially autistic people – from the NDIS. If the government succeeds, then instead of there being a shortage of professionals, one could imagine there might instead be a glut, and if so, one might expect them to have to accept work for less pay. This could not only affect spending on the program itself, but also change its workers’ spending habits, perhaps leading them to reign in any nonessential spending – which would have negative repercussions for any local businesses they would otherwise be supporting.
Nova Scotia
The Canadian province of Nova Scotia is admittedly small compared to America or Australia, but it has developed an interesting model for delivering autism supports. The government started paying a specific provider to provide intensive supports delivered by well-trained staff.
On the one hand, in some ways, this approach might allow for better-quality service than one would get with models that rely heavily on the private sector. On the other hand, this has led to chronic waitlist issues, because this provider can’t meet demand. In response, the government began to pay for the private sector to fill gaps. Further, when people are on waitlists, they might end up paying out of pocket for private supports if they can afford it.
I also imagine running a program in this very different way has important economic implications in the indirect realm of multiplier effects, not just in terms of direct outcomes. When the system operated solely through the public and nonprofit/academic sectors, before privatised options were added to fill gaps, it theoretically removed private equity from the picture. Government spending would be channelled exclusively to programs and staffing, and thus be more likely to end up on the demand side than the supply side. On the other hand, one would also have to consider the effects of increased out-of-pocket spending on privatised services from waitlisted families.
All in all, clearly, this is quite a different model from either American ABA delivery or the Australian NDIS, and the multiplier effect would presumably operate differently under it.
Community health workers in the Global South
So far, we’ve considered only models in high-income countries with highly-educated populations. What if you are trying to do things in a very different context, where people have lower incomes and where you can’t assume the availability of an educated workforce?
There are programs like PASS Plus and the WHO Caregiver Skills Training program that are specifically tailored to these Global South contexts. These programs can train community health workers who may have limited education compared to their counterparts in the Global North. For example, they may not have completed secondary education.
What are the economic implications of these programs? I can imagine they might be very different from those in the Global North. For example, those with limited means in Global South contexts might not be able to borrow money except at extremely high interest rates (e.g., 50% or more). If undertaking a brief training and helping to deliver these programs provides extra income to an experienced parent, maybe it removes the necessity to take out such a loan. Conversely, if trained professionals set up private practices based on Global North models, even some families who could technically minimally afford their services might be severely burdened by the expense, and this might redistribute money away from those with less and towards those with more.
What other sorts of multiplier effects might we observe in this very different context? I don’t know enough to say, but I imagine they could be meaningfully different from the Global North context.
Direct payments
Of course, as mentioned above, many costs “of autism” actually end up being expenses that go to others – private equity, professionals, paraprofessionals, community workers, etc. – not the autistic person themselves. So far, I’ve just been talking about various options for paying people to provide supports to autistic people or carers, but what about directly paying autistic adults? As many disabled people are not working, and if we can’t change that situation easily,8 governments generally provide direct welfare payments like American SSI9 to disabled people they consider deserving of them. However, as we saw, these are typically grossly inadequate, falling well below the poverty line. In a recent news story, one disabled Canadian shared his humble ambition: the hope that government would increase disability benefits to a level allowing him to purchase lactose-free milk. Basically, one can assume that any increase in these grossly-inadequate supports will be funnelled directly into increasing consumer demand, probably for quite basic goods, which will then have multiplier effects as it benefits those producing and selling those goods.
Summary
These multiplier effects are clearly very complex and difficult to estimate,10 but on the other hand, if one were to simply ignore them and assume that all costs of a program are money lost rather than stimulus or investment, one would be systematically biasing one’s evaluation. Through multiplier effects, society may benefit from increased spending, especially when delivered within the range of what the available pool of trained or trainable personnel can reasonably support. Eliminating these effects from consideration will systematically discourage spending, discourage services and encourage austerity.
Modelling chaos
However, the very complexity of these multiplier effects speaks to a larger issue: constructing economic models and trying to understand the downstream effects of any change, like the introduction of a new program, is not simple!
Economies are not just complicated, but chaotic. They contain numerous agents making decisions based on complex, nonlinear and often-irrational considerations. How can we forecast how much the labour market will value 5 extra IQ points for one person thirty years from now? We can’t even predict what the stock market will look like tomorrow, in part because as people try to predict the future, they use that information to make decisions, which changes the future.
The economics field, like a few other fields dealing with incredibly-complicated phenomena (for example, computational neuroscience), loves models and equations: beautiful numbers showing elegant relationships and neat systems. But models of complex and chaotic systems such as human mind-brains and human national and global economies are necessarily extraordinarily oversimplified and based on numerous and controversial and contingent assumptions. For example, I tend to think that spending money on the demand side of the economy is more beneficial these days than adding increased speculative and rent-seeking capital to the supply side, but a lot of people apparently disagree with that.
Indeed, in cost evaluations, there’s likely to be all kinds of opportunities to fiddle with subjective assumptions to produce very different effects. For instance, how many people will actually apply for benefits under some new program? Somebody who wants the program to go ahead might have good reason to come up with a lowball estimate, in order to keep projected costs down, while somebody who doesn’t want the program to be funded would presumably do the opposite. Or let’s say a low-quality study of the program shows strong positive effects, while a higher-quality study shows only modest benefit: does one use only the more rigorous estimate, does one treat both as equally valid, does one weight them based on study quality (and how do you quantify that quality)? Any assumption can be an opportunity to tweak a model to obtain a desired result. If the model is remotely realistic – if it approaches anything like the complexity and chaos of the real-world economy – a small change to parameters and assumptions could lead to very different outcomes.
How, in these complex and chaotic systems, in which actors have biases and interests that may cloud their objectivity, can we possibly come up with a model capable of predicting fine details? By all means, let us gather empirical data to test our ideas and theories about causal relationships between variables, but for the foreseeable future, vast gaps will remain in our understanding. We should not use the illusory certainty of models and equations to deceive ourselves.
You might think we can get around this issue by trying to formally quantify uncertainty, such as by drawing confidence intervals around our estimates, but that doesn’t actually work out. Confidence intervals make sense if, say, we are preforming a single empirical measurement and we aren’t sure we got it exactly right. But in these sorts of complex and chaotic systems, we don’t know enough about the complex relationships to even be able to quantify how uncertain we are.
At the end of the day, I think we have to acknowledge radical uncertainty and do our best to muddle through. Instead of trying to make everything neat and simple, we can just enumerate and consider the things we reasonably can infer, while frankly discussing what we can’t. We can quantify the general costs of a program, and do our best to consider if scaling it up might affect wages or expenses in any systematic way based on economies of scale, overextending a limited workforce, etc. We can also do our best to empirically capture other short-term money effects: for example, if a job placement increases an autistic adult’s income, or an early intervention reduces families’ caregiving demands and allows them to earn a bit more money. This can all be combined to provide a sort of short-term net financial cost estimate.
We can then compare this financial outcome to short-term benefits to wellbeing, communication, mental health, developmental skills, and other outcomes that we can observe through research trials, focusing on socially-valid desired outcomes and also considering any potential harmful side-effects. How much are these outcomes worth to us? That’s an inherently subjective question. There is no commensurability between these and the money, nor should we lightly assign crude numeric values to judge how much more worthwhile one life is than another. We need to carefully consider all of the likely short-term effects to decide whether an intervention is worth the price.
Beyond that, we can do our best to follow up with people in the longer-term and get a sense of how interventions affect people after years and decades – this is, after all, the ultimate purpose behind many of our interventions, especially the early-childhood ones. However, without solid data about these long-term downstream impacts, we’ll likely have to rely a lot on guesswork. We’ll have to be comfortable with uncertainty. We cannot afford to ignore the crucial possibility of long-term benefits, but we should be hesitant about putting numbers to outcomes that are so uncertain. This is something we will often have to discuss qualitatively and theoretically.
Multiplier effects are similarly uncertain. We might be able to get some general numbers about how much program spending goes to the supply side or the demand side of the economy, and perhaps even the income levels of the first order recipients, but the ultimate implications of such spending are not something all the experts are likely ever going to be able to quantitatively agree on. Once again, we will have to rely heavily on qualitative and theoretical discussion of the possibilities, but we must not ignore the possibility of these multiplier effects.
When we acknowledge the subjectivity of these qualitative issues, this also limits our ability to easily compare different programs. That’s okay. If aspects of the programs are genuinely qualitatively different, trying to force them into a framework for quantitatively comparing them with each other doesn’t make sense in the first place.
As we consider different program options before us, we also have to think about the bigger picture. What are we trying to do? Increase societal wealth? Increase happiness and wellbeing? For whom – the average person, or the most vulnerable and desperate?
We need empirical data and solid evidence to help our society choose appropriate policies, but we mustn’t let simplistic or dangerous modelling assumptions, or artificially narrowed scopes and boundaries, bias our better judgement. To paraphrase Mark Carney, some questions need to be answered based not on monetary value, but on societal and moral values: what we as humans decide is worthwhile.
- And that’s not even getting into an issue I mentioned in this post, which is that both leftist thinkers (through the “reserve army of labour” idea) and right-wing thinkers (by arguing it is necessary to use monetary policy to keep inflation down, rather than to pursue other goals like reducing unemployment) say that the current economic system requires unemployment – it requires people who want jobs but can’t get them. Leftist thinkers are more likely to think this is problematic and must change, but I think everyone basically agrees that this is how things are. And autistic and disabled people, even if they are capable of working and want to work, are often going to end up in this unemployed “reserve army” because they may be perceived as less desirable employees…
- In a way, this is arguably even more true today than when the idea of the multiplier was invented in the Great Depression, because back then, governments actually had to raise money through borrowing or taxation in order to spend it, so they were, in a way, just transferring money from less productive to more productive uses. Since the 1970s, some key countries – the USA, Canada, Australia, Japan, etc. – have had floating fiat currencies, so they can basically just wave a magic wand and poof money into existence if they want. (Though they have to be careful about managing inflation when doing so.) Mind you, there is another difference between now and the time of the Great Depression. The global economy was interconnected then, but now it is even more so. Not only is there a lot of global trade today, but there’s a lot of market concentration and a lot of specialisation by particular countries in producing particular types of goods and services. Thus, more of the multiplied benefits of spending might move overseas these days, creating something of a collective action problem.
- Although many disabled people are eligible for the United States’ publicly-funded health coverage.
- This sort of thing is one reason why results obtained in efficacy studies in laboratory contexts may not always match the results obtained in real-world settings.
- For now, let’s forget about other expenses like the BCBAs themselves to keep this simple. We can also ignore whatever money the big private profiteering insurance companies use to administer the system, the money to construct any facilities needed for the ABA service delivery, and so on…
- Basically, when rich people make others pay rent to use things they own, thereby in the long run transferring money from those with less to those who already have more. Importantly, these richer people can reap such rewards without having to do anything productive.
- For example, cryptocurrency. Buying things because one expects they will increase in value, which can be a source of instability and problems in the economy. You get these cycles where everyone is buying on the theory that they’ll sell at a profit to somebody else before things go bad. Eventually, the overinflated market inevitably crashes, and people lose a lot of money.
- Recalling that our current economic system, as noted earlier, requires a certain level of unemployment. So while there’s a lot we can and should do to promote autistic and disabled employment, especially direct job placement supports, this has limits even outside of the direct abilities and support needs of individuals.
- SSDI in the case of those acquiring disabilities after they began working and contributing to retirement savings and such.
- Especially if we consider that we aren’t just estimating the effects of spending money on the program, but also considering what will happen if the government doesn’t fund a program. Does the government just spend less money? Does it spend the money on a different program? If the government doesn’t spend as much money, does this mean less money is created, or does it mean that the money stays somewhere else? What would the ripple effects of these different scenarios be?