AI Auditing Engineer (Temporary) - Penn Center for Media, Technology, and Democracy - Penn Engineering
Job Description Summary
The Penn Center for Media, Technology, and Democracy (Penn MEDIATED) works to advance the scientific understanding of the information ecosystem and leverage that research to strengthen the foundations of democracy.
Penn MEDIATED seeks a part-time AI Auditing Engineer to develop and productionize auditing and monitoring of large language models (LLMs). The AI Auditing Engineer will unify existing research prototypes from several codebases in different academic projects to create an interoperable set of tools that is generally useful for longitudinal monitoring of LLMs. The tooling will be used to automatically and longitudinally monitor rates of refusal or soft censorship, political lean, sourcing information, and other features of LLM responses on political topics.
We seek an AI Auditing Engineer experienced in developing efficient and scalable data pipelines in Python, designing and managing NoSQL databases, working with cloud computing platforms, and in working with LLM APIs. Experience with and knowledge with natural language processing methods for LLM evaluations is also beneficial.
The AI Auditing Engineer will work closely with Danae Metaxa, Raj & Neera Singh Assistant Professor in the Department of Computer and Information Sciences. The position will be jointly supervised by Danae Metaxa and Alex Engler, Executive Director of the Penn Center on Media, Technology, and Democracy. This role can be fully remote or hybrid. This is a temporary role, and may be either part-time or full-time, depending on the applicants, with the expected term during summer 2026 at between 20-40 hours per week.
Job Description
- Develop and deploy an interoperable set of tools that is generally useful for longitudinal monitoring of LLMs.
- Design and manage appropriate databases for LLM monitoring.
- Advance research methods for LLM monitoring of political topics.
Qualifications
- Bachelor's degree in a relevant field of study
- Required proficiency in developing efficient and scalable data pipelines in Python.
- Required proficiency in designing and managing NoSQL databases.
- Required proficiency in working with cloud computing platforms.
- Preferred understanding or prior experience working with LLM APIs.
- Preferred understanding or experience with natural language processing methods for LLM evaluation.
- Preferred understanding or experience with cloud cost engineering and other aspects of DevOps, especially cost efficient use of LLMs.
- Preferred experience with data visualization in interactive dashboards.
- Interest in the political outputs and impacts of large language models (LLMs) is a bonus.
To Apply
Please Submit a Resume and Cover Letter
Job Location
Philadelphia, Pennsylvania
Department / School
School of Engineering and Applied Science
Pay Range
$7.25 - $22.00 Hourly Rate
Find Your Best Opportunity
Tell them AcademicJobs.com sent you!
