ClearFactAll topics

Should AI companies be required to obtain explicit informed consent before training models on personal data?

9 peer-reviewed studies for, 3 against

Key takeaways

Stronger evidence: For

Verdict

Requiring explicit consent before training on personal data is justified because modern ML systems can leak information about individuals in ways users cannot reasonably anticipate or detect. Foundational work shows “membership inference” can reveal whether a specific person’s record was in a training set, and “model inversion” can reconstruct sensitive attributes from model outputs—meaning the trained model itself can become a privacy exposure channel. Meanwhile, the idea that companies can avoid consent simply by “anonymizing” data is shaky: empirical and policy-facing research emphasizes that anonymization is often brittle and reidentification risk persists, especially when datasets can be linked. These risks strengthen the argument that people should have real agency over downstream uses of their personal information.\n\nThat said, the best counterargument is practical: demanding explicit consent for all training could be operationally difficult and could reduce data availability, potentially harming model quality and representativeness. Technical approaches such as differential privacy and secure aggregation can materially reduce privacy risk and may enable socially beneficial training with stronger safeguards. A sensible policy compromise can allow narrow, audited exceptions when strong privacy guarantees are in place. Still, as a default rule for personal data, explicit consent is the more defensible baseline—so the stronger side is for.

Strongest study on each side

Ranked by citation count.

Strongest study for

AI, Big Data, and the Future of Consent (AI & Society, 2022)

Adam J. Andreotta, Nin Kirkham et al. · 37 citations

Examines how current big data practices erode informed consent by identifying three critical problems: transparency/explanation issues, data re-purposing beyond original intent, and lack of meaningful alternatives for users. Proposes that personal data use for AI should be regulated through ethical governance frameworks like biomedical research oversight.

Strongest study against

Open the interactive page →

The evidence

For (9)

AI, Big Data, and the Future of Consent (AI & Society, 2022)

Adam J. Andreotta, Nin Kirkham et al. · 37 citations

Examines how current big data practices erode informed consent by identifying three critical problems: transparency/explanation issues, data re-purposing beyond original intent, and lack of meaningful alternatives for users. Proposes that personal data use for AI should be regulated through ethical governance frameworks like biomedical research oversight.

Patient Consent for the Secondary Use of Health Data in Artificial Intelligence (AI) Models: A Scoping Review (International Journal of Medical Informatics, 2025)

Khadijeh Moulaei, Saeed Akhlaghpour et al.

A comprehensive scoping review examining patient consent practices for AI training on health data. Identifies 65 barriers and 101 facilitators to informed consent, emphasizing that inadequate consent processes and privacy breaches are major obstacles to responsible data sharing for AI development.

Can AI be Consentful? (arXiv, 2025)

Through legal and ethical analysis, this paper demonstrates that while individuals can consent to initial AI training data use, they cannot meaningfully consent to the numerous potential outputs their data might enable. The authors identify three fundamental challenges—the scope problem, temporality problem, and autonomy trap—that create a "consent gap" in AI systems, arguing current frameworks inadequately address individual autonomy rights.

Information Provision for Informed Consent Procedures in Psychological Research Under the General Data Protection Regulation: A Practical Guide (Advances in Methods and Practices in Psychological Science, 2023)

Dara Hallinan, Franziska Boehm et al.

Hallinan et al. provide comprehensive guidance establishing that informed consent procedures requiring specific information disclosure are essential ethical and legal standards for psychological research involving personal data collection. This directly supports requiring explicit informed consent before training AI models on personal data.

Using personal data in AI model training under EU law (Digital Law Journal, 2025)

This paper examines the legal permissibility of AI model training under joint application of the AI Act and GDPR, identifying risks of legally defective training due to excessive data collection, and proposing a framework with impact assessments to ensure lawful personal data processing while respecting fundamental rights.

Against (3)

The dilemma of consent for AI in healthcare (Surgery, 2024)

Jeremy A. Balch, Barbara J. Evans et al.

Argues that mandatory informed consent requirements for AI training data can create selection bias and disparate impacts on underrepresented populations, potentially worsening AI performance for marginalized groups who have historically lower consent rates.

The debate

David: James, we need explicit consent before using personal data to train AI systems. Modern machine learning can expose people's private information in ways they never agreed to or could even imagine.

James: But David, that's incredibly impractical. Do you want to contact millions of users every time we improve a recommendation algorithm? We'd grind innovation to a halt while our competitors overseas race ahead.

David: Innovation built on secretly exploiting people's data isn't progress worth having. Research shows these models can leak sensitive information through membership inference attacks - hackers can literally determine if your medical records were in the training set.

James: Look, we already anonymize data before training. Remove the names and social security numbers, problem solved. Plus, these theoretical attacks you're worried about rarely happen in practice.

David: Anonymization is a myth, James. Studies prove you can re-identify people by linking supposedly anonymous datasets. Netflix learned this the hard way when researchers de-anonymized their "privacy-safe" movie ratings using IMDb cross-references.

James: Fine, but explicit consent would devastate data quality. We'd only hear from privileged users who understand these systems, creating biased models that ignore marginalized communities. Is that really more ethical?

David: You're assuming people won't consent when given a real choice. Many would gladly participate if companies were transparent about their intentions instead of hiding behind dense privacy policies. Besides, model inversion attacks can reconstruct sensitive attributes from outputs - the trained model itself becomes a privacy risk.

James: I'll grant you the privacy risks are real, but we have technical solutions. Differential privacy and secure aggregation can protect individual data while preserving model utility. Why throw the baby out with the bathwater?

David: Those techniques help, but they're not bulletproof. Even with differential privacy, determined attackers find workarounds. And who's auditing these protections? We're asking people to trust black-box systems with their most sensitive information.

James: So we create auditing frameworks and regulatory oversight. We can have narrow exceptions for socially beneficial applications like medical research or climate modeling, with strong technical safeguards in place.

David: Now you're talking sense, but that proves my point. If we need special exceptions and oversight, it shows explicit consent should be the default rule. People deserve real agency over their personal information, not corporate promises that "trust us, we'll be careful."

James: I worry we're being idealistic at the expense of practical progress. These models power everything from fraud detection to accessibility tools. Requiring explicit consent could break systems people rely on daily.

David: Then let's build better systems that respect privacy by design. The current model of "collect everything, ask forgiveness later" is unsustainable. When researchers can extract training data verbatim from large language models, we've crossed a line that only explicit consent can address.

James: You make compelling points about the technical risks. Maybe the solution is stronger default protections with clear opt-in mechanisms for users who want to contribute their data.

Frequently asked questions

Can an AI model reveal personal information from its training data?

Yes. Studies on membership inference show attackers can tell whether a specific person's record was used in training, and model inversion research shows sensitive attributes can be reconstructed from model outputs.

Is anonymizing data enough to avoid needing consent for AI training?

The evidence suggests it often is not. Reviews of anonymization find that reidentification risk persists in practice, particularly when datasets can be linked together.

Are there technical alternatives to requiring consent for AI training?

Differential privacy can limit what a model reveals about any single individual, and secure aggregation lets training updates be combined without exposing each user's contribution. Some argue these safeguards could justify narrow, audited exceptions to consent requirements.

What are the downsides of requiring consent for AI training data?

Consent requirements can be operationally difficult and may reduce the amount of data available. Research in healthcare warns they can create selection bias, since some underrepresented groups have historically lower consent rates, which could make AI perform worse for those groups.

Related questions