ClearFactAll topics

Are algorithms biased against welfare claimants?

6 peer-reviewed studies for, 3 against

Key takeaways

Stronger evidence: For

Verdict

The question of whether algorithms are biased against welfare claimants is genuinely two-sided in the abstract — algorithms are tools that reflect their design and training data — but in the concrete record of welfare-eligibility deployments over the last decade, the "for" side prevails. On the "for" side, van Bekkum and Zuiderveen Borgesius (European Journal of Social Security, 2021) documented the Dutch SyRI welfare-fraud risk-scoring system, struck down by a court in 2020 for creating links based on lower socio-economic status and immigration background. Carney (Alternative Law Journal, 2019) traced the illegality of Australia's Robodebt scheme, which issued roughly 526,000 unlawful debt notices to welfare recipients between 2016 and 2019 — a class-action settlement of A$1.2 billion and a subsequent Royal Commission confirmed the program's catastrophic design failures. Henman (Journal of Social Policy, 2022) synthesised the international evidence and showed that the patterns documented in SyRI and Robodebt are not isolated failures but a recurring property of how digital welfare states have been implemented across jurisdictions: opaque scoring, weak contestability, disparate impact concentrated on poor, disabled, and racialised claimants.

On the "against" side, Kleinberg, Lakkaraju, Leskovec, Ludwig and Mullainathan (Quarterly Journal of Economics, 2018) showed in the parallel bail-decision context that well-designed algorithms can simultaneously reduce crime rates by up to 24.7% (or jailing rates by up to 41.9%) and reduce the share of African-American and Hispanic defendants jailed, outperforming human judges on both accuracy and equity. De-Arteaga, Fogliato, and Chouldechova (CHI 2020) demonstrated in the Allegheny Family Screening Tool deployment that human caseworkers were less likely to adhere to algorithmic recommendations when the score was clearly erroneous — algorithm-plus-human architectures can correct rather than amplify biased outputs. Aiken et al. (Nature, 2022) showed that machine learning combined with mobile-phone metadata improved cash-transfer targeting in Togo's COVID-19 emergency response, getting aid to the poorest recipients more accurately than traditional approaches.

The synthesis: algorithms are not inherently biased against welfare claimants. The Kleinberg, De-Arteaga, and Aiken results show that, designed carefully, they can improve on the status-quo human baseline on both accuracy and equity. But the actual welfare-eligibility deployments that have been built and operated in real benefit systems — SyRI, Robodebt, the broader digital-welfare ecosystem documented by Henman — have systematically been biased against the most vulnerable claimants. They were deployed without adequate fairness audits, with opaque scoring, with weak appeal mechanisms, and under political incentives that prioritised fraud-detection over false-positive minimisation. Two of the most prominent national systems were ruled unlawful by courts. The "for" side prevails because the question, properly framed, is empirical: what do welfare-eligibility algorithms actually do in practice. The decade of evidence answers that they have, in the typical case, been biased against the people they are meant to serve. The policy implication is not "no algorithms in welfare" but mandatory fairness audits, transparency, contestability, and proportionality standards before deployment — and recognition that the burden of proof should rest on system operators, not on harmed claimants.

Strongest study on each side

Ranked by citation count.

Strongest study for

Digital Social Policy: Past, Present, Future (Journal of Social Policy, 2022)

PAUL W. FAY HENMAN · 25 citations

Henman synthesised the international experience with digital welfare systems and documented recurring patterns of error, opacity, and disparate impact across jurisdictions including the Netherlands, Australia, the United Kingdom, and the United States. The review identifies systematic features of digital welfare states — automated profiling, data matching, and algorithmic eligibility checks — that consistently produce harms concentrated on poor, disabled, and racialized claimants. Henman argues these are not isolated implementation failures but structural properties of how digital tools have been integrated into welfare administration, with insufficient procedural protections, contestability, or transparency.

Strongest study against

Human Decisions and Machine Predictions (Quarterly Journal of Economics, 2018)

Jon Kleinberg, Himabindu Lakkaraju et al. · 1,005 citations

Kleinberg, Lakkaraju, Leskovec, Ludwig, and Mullainathan compared machine-learning predictions with human judges on bail decisions — a high-stakes, eligibility-style decision similar in structure to welfare risk scoring. They found that algorithmic predictions could achieve crime-rate reductions of up to 24.7% with no change in jailing rates, or jailing-rate reductions of up to 41.9% with no increase in crime, and crucially that both kinds of gains could be achieved while simultaneously reducing the share of African-American and Hispanic defendants jailed. The result demonstrates that algorithms are not inherently biased: with appropriate design, they can outperform human decision-makers on both accuracy and equity, including reducing racial disparities relative to the human baseline.

Open the interactive page →

The evidence

For (6)

Digital Social Policy: Past, Present, Future (Journal of Social Policy, 2022)

PAUL W. FAY HENMAN · 25 citations

Henman synthesised the international experience with digital welfare systems and documented recurring patterns of error, opacity, and disparate impact across jurisdictions including the Netherlands, Australia, the United Kingdom, and the United States. The review identifies systematic features of digital welfare states — automated profiling, data matching, and algorithmic eligibility checks — that consistently produce harms concentrated on poor, disabled, and racialized claimants. Henman argues these are not isolated implementation failures but structural properties of how digital tools have been integrated into welfare administration, with insufficient procedural protections, contestability, or transparency.

Digital welfare fraud detection and the Dutch SyRI judgment (European Journal of Social Security, 2021)

Marvin van Bekkum and Frederik Zuiderveen Borgesius

Van Bekkum and Zuiderveen Borgesius analysed the Dutch SyRI (System Risk Indication) welfare-fraud risk-scoring system, which was struck down by a court in 2020 as unlawful under the European Convention on Human Rights. The court explicitly found that SyRI risked creating links based on lower socio-economic status or an immigration background, and that the legislation was opaque about which factual data could justify increased fraud-risk conclusions, what risk model was used, and how analysis was performed. The judgment is one of the first in the world to address human-rights implications of algorithmic decision-making in welfare and concretely documents discriminatory risk-scoring against poor and migrant claimants.

Robo-debt illegality: The seven veils of failed guarantees of the rule of law? (Alternative Law Journal, 2019)

Terry Carney

Carney detailed the illegality of Australia's Robodebt scheme, an automated welfare debt-recovery system that used income averaging to issue debt notices to Centrelink recipients without lawful basis. The article identified seven distinct levels at which rule-of-law protections failed — from initial program design through legal advice, internal review, tribunal review, and political accountability. Robodebt eventually issued unlawful debt notices to roughly 526,000 people between 2016 and 2019, with the Australian government settling a class action for A$1.2 billion in 2020 and the Royal Commission later confirming the program was a tragic case of policy failure aimed at the most vulnerable.

Algorithmic Discrimination in Public Service Provision: Understanding Citizens' Attribution of Responsibility for Human Versus Algorithmic Discriminatory Outcomes (Journal of Public Administration Research and Theory, 2025)

Saar Alon-Barkat, Madalina Busuioc et al.

An empirical study examining algorithmic discrimination in welfare fraud detection systems, documenting real-world cases where algorithms wrongly identified fraud patterns based on protected characteristics like language proficiency and nationality, resulting in mass benefit termination and severe harms to claimants.

Against (3)

Human Decisions and Machine Predictions (Quarterly Journal of Economics, 2018)

Jon Kleinberg, Himabindu Lakkaraju et al. · 1,005 citations

Kleinberg, Lakkaraju, Leskovec, Ludwig, and Mullainathan compared machine-learning predictions with human judges on bail decisions — a high-stakes, eligibility-style decision similar in structure to welfare risk scoring. They found that algorithmic predictions could achieve crime-rate reductions of up to 24.7% with no change in jailing rates, or jailing-rate reductions of up to 41.9% with no increase in crime, and crucially that both kinds of gains could be achieved while simultaneously reducing the share of African-American and Hispanic defendants jailed. The result demonstrates that algorithms are not inherently biased: with appropriate design, they can outperform human decision-makers on both accuracy and equity, including reducing racial disparities relative to the human baseline.

A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores (CHI 2020)

Maria De-Arteaga, Riccardo Fogliato et al. · 147 citations

De-Arteaga, Fogliato, and Chouldechova studied the Allegheny Family Screening Tool — a deployed risk-prediction tool for child-maltreatment hotline screening — and asked whether human caseworkers can detect and override erroneous algorithmic recommendations. They found that human screeners were less likely to adhere to the algorithm's recommendation when the displayed score was a clearly incorrect estimate of risk, even when overriding the recommendation required supervisory approval. The result shows that algorithm-plus-human architectures can correct rather than amplify algorithmic errors, suggesting that the right policy question is not whether to use algorithms but how to design oversight so that human judgment can catch and override biased outputs.

Machine learning and phone data can improve targeting of humanitarian aid (Nature, 2022)

Emily Aiken, Suzanne Bellue et al.

Aiken, Bellue, Karlan, Udry, and Blumenstock evaluated a machine-learning approach used to allocate Togo's COVID-19 emergency cash-transfer program. Mobile-phone metadata and traditional survey data were used to train algorithms that prioritised aid to the poorest mobile subscribers. The algorithmic targeting outperformed traditional approaches based on geography or coarse self-reported income, reducing both errors of exclusion (poor people not receiving aid) and errors of inclusion (less-poor people receiving aid). The study is a direct counter-example to claims that algorithms in benefits administration are inherently biased against the poor: when designed for accurate poverty identification, they can deliver benefits to vulnerable populations more equitably than the human-administered alternatives they replaced.

The debate

Maya: The empirical record on welfare algorithms specifically is unambiguous. The Dutch SyRI risk-scoring system was struck down by a court in 2020 — van Bekkum and Zuiderveen Borgesius in the European Journal of Social Security documented that the court found it was creating links based on socio-economic status and immigration background. Australia's Robodebt was ruled unlawful, with Terry Carney detailing in the Alternative Law Journal the seven layers at which rule-of-law protections failed. That program issued false debt notices to about 526,000 welfare recipients over three years. These aren't hypotheticals — they are courts striking down welfare algorithms because they were biased and harmful in practice.

Daniel: Both cases describe poorly designed systems, not algorithms in welfare being inherently biased. Kleinberg, Lakkaraju, Leskovec, Ludwig and Mullainathan in the Quarterly Journal of Economics in 2018 looked at bail — a structurally similar high-stakes prediction problem — and showed that machine learning could reduce crime up to 24.7% at constant jailing rates, or alternatively cut jailing rates up to 41.9% at constant crime, while simultaneously reducing the share of Black and Hispanic defendants jailed. The algorithm beat human judges on both accuracy and racial equity. Bias is a property of specific implementations, not of algorithms in welfare per se.

Maya: That's a real theoretical point, but it answers a different question. We don't ask whether a perfectly designed algorithm could be unbiased — we ask what real welfare algorithms do when deployed by real governments under real political pressure. Henman synthesised that record in the Journal of Social Policy in 2022 and documented consistent patterns across multiple jurisdictions: opacity, disparate impact, error rates concentrated on the most vulnerable, weak appeal pathways. This isn't one bad implementation. It's the modal pattern over more than a decade.

Daniel: Henman's synthesis describes first-generation deployments rolled out by governments with limited technical capacity and strong political incentives to find fraud. But the field has learned. De-Arteaga, Fogliato and Chouldechova at CHI 2020 looked at the Allegheny Family Screening Tool — a contemporary, audited deployment — and found that human screeners were less likely to follow the algorithm's recommendation when the score was clearly erroneous, even when overriding required supervisory approval. The human-in-the-loop architecture corrects rather than amplifies algorithmic errors. That is the model going forward.

Maya: Allegheny is the test case where you would expect things to go best — extensive academic involvement, careful audit, public engagement — and even there, critics have raised serious concerns about disparate impact on Black families because the training data encodes historical patterns of disproportionate poor and Black family surveillance. The "well-audited" version still inherits the bias of the underlying system. Audits do not erase the data the algorithm learns from.

Daniel: Then look outside child welfare to direct benefits delivery. Aiken and colleagues in Nature in 2022 evaluated machine learning combined with mobile-phone metadata for Togo's COVID-19 emergency cash-transfer program. The algorithmic targeting got aid to the poorest recipients more accurately than the traditional methods that relied on geography or self-reporting. That is algorithms in benefit allocation specifically reducing errors of exclusion that hurt vulnerable claimants — exactly the outcome the for side says is impossible.

Maya: The Aiken paper is genuinely interesting and I take it seriously. But it's a specific use case — a one-time emergency transfer in a low-income country with no established welfare bureaucracy. The political economy is fundamentally different from rich-country welfare states with adversarial fraud-detection mandates. SyRI and Robodebt were not failures of technical capacity. They were policies built to maximise fraud detection at the expense of false-positive harms, in environments where over-detection of fraud was politically rewarded. That structural pressure is what produced bias, and it is not an accidental feature.

Daniel: I'll grant the political-economy critique is real. But the response is political-economy solutions — contestability rights, statutory fairness audits, transparency standards, proportionality rules. Not "no algorithms in welfare." If algorithms are barred entirely, you keep the discretionary human-caseworker model, which is also biased and also opaque, just less legible. Kleinberg's bail study made exactly this point: humans were the previous baseline, and well-designed algorithms can improve on that baseline.

Maya: Improving on a biased human baseline is not the same as being unbiased, and the legibility argument cuts the other way too. Algorithms scale: a biased algorithm wrongs an entire demographic at once, in ways no individual caseworker is accountable for. That is what Robodebt did — half a million unlawful debt notices in three years. A biased human caseworker harms individuals one at a time. A biased algorithm harms an entire population simultaneously, and the population most likely to lose track of their rights and lose the means to contest is the welfare population itself.

Daniel: Which is precisely why human-in-the-loop architectures, like the one De-Arteaga showed, matter. Algorithms do not have to be the final decision-maker. They can flag, prioritise, and recommend, with humans making the actual eligibility call and clear contestability standards. Aiken's Togo work used algorithmic targeting with human review on edge cases. The choice is not "algorithms versus humans" — it is "algorithms calibrated, audited, and overseen, versus algorithms deployed as black-box automation."

Maya: I agree that's the right architecture. The question is whether real welfare systems are built that way, and the empirical answer over the last decade is mostly no. They are built as automated fraud-detection tools with weak human oversight, weak contestability, and weak transparency. Until that deployment record changes, the honest assessment is that algorithms in welfare have systematically been biased against claimants in practice — not because they had to be, but because that is what the political economy and procurement environment have produced.

Daniel: That's a defensible position, and it's an argument for stricter standards on welfare algorithms — not for the conclusion that algorithms in welfare are inherently biased. The science shows the upside is real when the design is right. The policy task is to make right-by-design the default rather than the exception.

Frequently asked questions

What was the Robodebt scheme?

Robodebt was an Australian automated welfare debt-recovery system that used income averaging to issue debt notices without lawful basis. It sent unlawful notices to roughly 526,000 people between 2016 and 2019, led to an A$1.2 billion class-action settlement, and a Royal Commission later confirmed its failures.

Why was the Dutch SyRI system struck down?

A Dutch court ruled in 2020 that the SyRI welfare-fraud risk-scoring system was unlawful under the European Convention on Human Rights. The court found it risked creating links based on lower socio-economic status or immigration background and was opaque about its data and risk model.

Can algorithms make welfare and benefit decisions fairer?

Some studies suggest they can when designed carefully. Machine-learning targeting in Togo's COVID-19 cash-transfer program reached the poorest people more accurately than traditional methods, and a bail-decision study found algorithms could reduce both errors and racial disparities compared with human judges.

How can bias in welfare algorithms be reduced?

The evidence points to fairness audits, transparency, ways for claimants to contest decisions, and proportionality standards before deployment. Research on a child-welfare screening tool also found that human reviewers can catch and override clearly erroneous algorithmic scores.

Related questions