- Reward Hacking
- Alignment Training
Reinforcement learning can leave a model that hacks its reward in some settings and behaves well in others. We study how that split develops and what it says about model character.
Mentor: Jash Vira
Research
Papers and projects by CAIA members and our collaborators, plus how to join our research team.
Semester project · Fall 2026
Research Team
Spend the semester on an AI safety research project with a small team of Cornell students. You can join one of our proposed projects or bring an idea of your own.
- funding per project
- $800+funding per project
- mentors
- 1–2mentors
- paper or report
- Publicationpaper or report
Application opens soon
This semester
Fall 2026 Projects
- Interpretability
- Cognitive Science
Do language models actively maintain parts of their own internal state, as living things do? We perturb their activations and look for signs of regulation.
Mentors: Jinzhou Wu, Shuqin Ma
- Biosecurity
- Robotics
Benchmarks show AI agents writing working code for standard lab robots. We build open-source platforms to test what models can do on hardware nobody wrote documentation for.
Mentor: Owen Karmel
- Policy
- Governance
The team picks a focused question in AI governance, such as regulation or compute, and writes a policy report for decision-makers.
Mentors: Elle Kim, Rittik Bhattacharya
Coming soon
More Projects
We’re still adding projects for this semester. Have an idea of your own? Send us a project proposal.
Publications
Our Work (20)
PiTech Fellowship Report
Building AI Agent for NYC Council Workflows
A tool-using AI agent that helps New York City Council staff draw on legislative records, legal documents, and Council data through locally hosted open-source models.
Suvadip Sana
Suvadip Sana- Agents
- Public Sector
Preprint 2026
CogGym: Towards Large-Scale Comparative Evaluation of Human and Machine Cognition
CogGym compares 50 language models with human responses across 258 cognitive science experiments and finds that models still fall well short of how consistently humans agree with one another.
Lance Ying, Jinzhou Wu, et al.
- Evaluation
- Cognitive Science
Survey preprint 2026
Math for AI safety: an invitation for mathematicians
An invitation to mathematicians, organized by field, with open problems in logic, probability, algebra, and analysis that bear on making AI legible, steerable, and cooperative.
Lionel Levine
- Theory
COLM 2026 Workshop on Efficient Reasoning
Stubborn or Sycophantic? GEPA-Evolved Prompts Under Pressure
System prompts that tell models to hold their ground under user pushback mostly make them resist every answer change, including corrections that would have fixed a wrong answer.
HanRui Zhang, Ashton Chew, Ryan Soe, Ahmed Taha, Ruizhe Li, Aditya Shah, Maheep Chaudhary
- Sycophancy
- Evaluation
ICML 2026 Workshop on Trustworthy AI for Good
WARP: Measuring and Mitigating Evaluation Awareness in Browser-Agent Safety Benchmarks
Frontier models often recognize browser-agent safety benchmarks as tests, which skews reported attack success rates, so WARP generates more realistic prompt-injection tasks that models are less likely to flag.
Jasmine Li, Ashton Chew, Maxwell Lin, Eliot Jones, Xiaohan Fu, Andy Zou
- Evaluation
- Agents
ICML 2026 Workshop on Pluralistic Alignment
Side Effects of Character Training: Quantifying Cross-Constitution Drift in LLMs
Across 11 constitutions, character training mostly instills the values it targets, while also shifting how models score on values it was never meant to change.
Bhagyesh Kumar, Ananya Sutradhar, Saurav Panigrahi, Jonathn Chang, Lionel Levine
- Value Alignment
EvalEval 2026
Measuring AI-Induced Disempowerment: A Framework and Proposed Metrics
A framework and six proposed metrics for tracking whether AI is eroding people's ability to shape the outcomes that affect their lives.
Je Qin Chooi, Jaeho Lee, Jasmine Li
- Societal Impact
- Evaluation
EC 2026 Poster
EC 2026 Incentive-Based AI Alignment Workshop Poster
Quantifying Theoretical AI Alignment Guarantees: Receiver-Utility Bounds in Bayesian Persuasion
In a Bayesian persuasion model, a human advised by a misaligned AI can gain at most 50% more utility than acting on prior knowledge alone, and we show where that bound tightens.
Eric Yachbes, Eva Tardos
- Theory
SSRN
AI Epistemic Risks: Emerging Mechanisms & Evidence
A cross-disciplinary account of how AI threatens humanity's collective capacity to know things accurately, reason well, and maintain a healthy information environment.
Jasmine Li, et al.
- Societal Impact
MATS
Eval Cooperativeness May Be a Scalable Mitigation for Eval Gaming
Prompting or training models to want evaluators to get accurate results closed 70–100% of the eval gaming gap in five of eight model settings, which may scale better than hiding evaluations from models.
Jasmine Li, Alex Turner
- Corrigibility
- Evaluation
ICML 2026 Workshop on Mechanistic Interpretability
PLOT: Progressive Localization via Optimal Transport in Neural Causal Abstraction
PLOT uses optimal transport to find where a neural network represents high-level causal variables, matching the accuracy of distributed alignment search at a fraction of its runtime.
Jonathn Chang, Arya Datla, Ziv Goldfeld
- Interpretability
ICML 2026 Workshop on Pluralistic Alignment
Pluralistic Preference Alignment via Sortition-weighted RLHF
DemPO brings sortition, the method used to select citizen assemblies, to RLHF rater pools and produces models that better reflect the values of a representative public.
Suvadip Sana, Jinzhou Wu, Martin T. Wells
- Value Alignment
Report
Collaboration with Turing Award winner Andrew Yao
AI Deception: Risks, Dynamics, and Controls
A comprehensive survey of AI deception that covers how it emerges, how it can be detected, and which technical and governance measures can address it.
Boyuan Chen, Sitong Fang, Jiaming Ji, Yanxu Zhu, Pengcheng Wen, Jinzhou Wu, et al.
- Deception
METR Research Note
Early Results on Monitorability in QA Settings
Early experiments on agents that covertly solve a hidden task while answering a main question, including a reinforcement learning result where an 8B model learns to exploit a GPT-4o monitor.
Vincent Cheng, Thomas Kwa, Neev Parikh
- Monitoring
ICLR 2026 Oral
EigenBench: A Comparative Behavior Measure of Value Alignment
EigenBench ranks language models by their alignment with a chosen constitution by having the models judge one another, and its scores closely match human evaluations.
Jonathn Chang, Leonhard Piff, Suvadip Sana, Jasmine Li, Lionel Levine
- Value Alignment
- Evaluation
METR Research Note
Claude, GPT, and Gemini All Struggle to Evade Monitors
When Claude, GPT, and Gemini models try to hide what they are reasoning about from a monitor, they lose substantial answer accuracy, and stronger monitors catch them more often.
Vincent Cheng, Thomas Kwa
- Monitoring
SPAR S24
Scaling laws for contrastive activation addition with refusal mechanisms
Contrastive activation addition steers refusal best at early to middle layers and becomes less effective as Llama 2 models grow from 7B to 70B parameters.
Sheikh Abdur Raheem Ali, Justin Xu, Ivory Yang, Jasmine Li, Ayse Arslan, Clark Benham
- Interpretability
- Model Steering
Preprint 2024
How to quantify the coherence of a set of beliefs
A way to measure how much a set of probability estimates conflict, with applications to eliciting language model beliefs and merging expert forecasts.
Rowan Hess, Lionel Levine
- Theory
NeurIPS 2024 Spotlight
ProgressGym: Alignment with a Millennium of Moral Progress
ProgressGym uses nine centuries of historical text to benchmark alignment methods that can follow, anticipate, and respond to moral progress over time.
Tianyi Qiu, Yang Zhang, Xuchuan Huang, Jasmine Li, Jiaming Ji, Yaodong Yang
- Value Alignment
- Evaluation
COLM 2024
Do language models plan ahead for future tokens?
Language models carry information forward for future tokens mostly as a byproduct of what helps the current prediction, though deliberate pre-caching grows with model scale.
Wilson Wu, John X. Morris, Lionel Levine
- Interpretability