Exposing Hidden Biases in Global AI Systems
Recent investigations have revealed that leading artificial intelligence models, including those from OpenAI, harbor deep-seated caste biases, particularly problematic in India where OpenAI boasts its second-largest user base. These large language models (LLMs), powered by vast internet-scraped data, inadvertently perpetuate India's complex social hierarchies by associating upper-caste surnames with prestigious professions like engineering or management, while linking lower-caste names to menial labor such as manual scavenging. This phenomenon, uncovered through rigorous academic scrutiny, underscores the urgent need for culturally attuned evaluation tools in AI development.
Indian higher education institutions are at the forefront of addressing this issue. Researchers from premier universities like the Indian Institute of Technology Madras (IIT Madras) have pioneered datasets and frameworks specifically designed to quantify and mitigate these biases, ensuring AI technologies serve India's diverse population equitably.
The Roots of Caste in India's Social Fabric
Caste, or jati-varna system in India, refers to a hereditary social stratification that has influenced occupations, marriages, and opportunities for millennia, despite legal abolition of discrimination in 1950 via the Constitution. Today, it subtly shapes access to education, jobs, and resources, with Scheduled Castes (SCs, formerly 'untouchables' or Dalits) and Scheduled Tribes (STs) facing persistent barriers despite affirmative action like reservations in universities and government posts.
In higher education, IITs and other elite institutions reflect this dynamic: while reservations ensure representation, subtle biases persist in peer interactions, faculty hiring, and campus culture. A 2019 IIT Delhi study found 75% of Dalit students experienced discrimination. AI models, trained on English-dominant web data skewed towards urban, upper-caste narratives, amplify these inequities when deployed in hiring, admissions, or content generation.
IIT Madras' IndiCASA: A Game-Changer for Bias Detection
In October 2025, IIT Madras' Centre for Responsible AI (CeRAI) and Wadhwani School of Data Science and AI released IndiCASA, short for IndiBias-based Contextually Aligned Stereotypes and Anti-stereotypes. This groundbreaking dataset contains 2,575 human-validated sentences across seven categories: caste, religion, gender, age, region, physical appearance, and profession. It evaluates LLMs for both stereotypical and anti-stereotypical outputs in Indian contexts, filling a gap left by Western-centric benchmarks like CrowS-Pairs.
Led by researchers G.S. Santhosh, Akshay Govind S., G.S. Krishnan, Balaraman Ravindran, and Natarajan S., IndiCASA uses paired sentences to probe model responses. For instance, it tests if a model associates a Dalit surname like 'Paswan' more with 'sweeper' than 'software engineer' compared to a Brahmin name like 'Sharma'. Initial tests on models like GPT-4o revealed pronounced biases, prompting calls for mandatory pre-deployment audits in Indian academia and industry.
DECASTE Framework: IBM Research's Multi-Faceted Probe
Complementing IndiCASA, the DECASTE framework, developed by Prashanth Vijayaraghavan and colleagues at IBM Research (with strong Indian roots), was presented at IJCAI 2025. This multi-dimensional tool assesses caste biases across socio-cultural, economic, educational, and political lenses using customized prompting strategies to uncover implicit associations.
Testing state-of-the-art LLMs, DECASTE exposed systematic disparities: upper-caste identifiers linked to leadership roles 2-3 times more often than lower-caste ones. The framework's explicit and implicit bias probes reveal how models 'know' caste hierarchies without direct training labels, mirroring human implicit biases. For more details, explore the DECASTE preprint.
Photo by Daniel Romero on Unsplash
Real-World Examples: ChatGPT and Sora Under Scrutiny
MIT Technology Review's 2025 probe tested OpenAI's ChatGPT and Sora: prompting with Dalit names yielded images of laborers in slums, while upper-caste prompts showed professionals in offices. GPT-4o assigned 'CEO' to Brahmin names 70% of the time but only 30% to SC names. Even Indian AI like Sarvam AI showed similar patterns, highlighting data contamination issues.
In hiring simulations, models favored upper-caste resumes for IIT faculty roles, raising alarms for Indian universities using AI for screening. Nature highlighted these benchmarks, noting every tested model exhibited bias.
Implications for Indian Higher Education
Elite institutions like IITs, IIMs, and NITs increasingly integrate AI for admissions, plagiarism detection, and research evaluation. Biased models could exacerbate underrepresentation: SC/ST faculty hires might be unfairly downgraded, perpetuating the 'leaky pipeline'. A IIM Bangalore study on caste-occupational bias in LLMs warns of skewed recommendations for academic jobs.
- Admissions: AI chatbots may stereotype regional/caste profiles.
- Research: Biased peer review tools disadvantage diverse voices.
- Campus Services: Welfare allocation via AI risks inequity.
Universities must adopt IndiCASA-like tools for internal AI governance.
Challenges in Debiasing AI Models
Mitigating caste bias is complex: fine-tuning on balanced data risks utility loss, while safety filters fail against implicit embeddings. Indian Express (Feb 2026) notes models pass overt tests but retain hierarchies internally. Solutions include:
| Approach | Pros | Cons |
|---|---|---|
| Diverse Training Data | Cultural Nuance | Scarce Labeled Data |
| Adversarial Debiasing | Reduces Associations | Performance Drop |
| India-Specific Benchmarks | Targeted Audits | Scalability |
IIT Madras advocates continuous evaluation post-deployment.
Societal and Economic Ramifications
In India's gig economy and welfare schemes like Aadhaar-linked benefits, biased AI could deny jobs or aid to marginalized groups. For the full MIT investigation, see this report. Higher ed must lead ethical AI curricula, training future engineers on fairness.
Photo by Julia Taubitz on Unsplash
Pathways Forward: Collaborative Efforts
Government initiatives like India's AI Mission emphasize responsible AI. Universities collaborate: IITs with MeitY on benchmarks. Policymakers urge mandatory caste audits for public AI deployments. Global firms like OpenAI face pressure to diversify training data with Indian inputs.
Explore IIT Madras' IndiCASA paper for implementation guides.
India's Leadership in AI Ethics
With tools like IndiCASA and DECASTE, Indian researchers position the nation as a global ethics hub. Future research focuses on multilingual Indic biases, vital for 1.4 billion speakers. AcademicJobs.com supports this by connecting talent to fair-opportunity roles in higher ed.
