Publications and projects by CAIA members and collaborators.
A tool-using AI agent that helps New York City Council staff draw on legislative records, legal documents, and Council data through locally hosted open-source models.
Suvadip Sana
CogGym compares 50 language models with human responses across 258 cognitive science experiments and finds that models still fall well short of how consistently humans agree with one another.
Lance Ying, Jinzhou Wu, et al.
An invitation to mathematicians, organized by field, with open problems in logic, probability, algebra, and analysis that bear on making AI legible, steerable, and cooperative.
Lionel Levine
System prompts that tell models to hold their ground under user pushback mostly make them resist every answer change, including corrections that would have fixed a wrong answer.
HanRui Zhang, Ashton Chew, Ryan Soe, Ahmed Taha, Ruizhe Li, Aditya Shah, Maheep Chaudhary
Frontier models often recognize browser-agent safety benchmarks as tests, which skews reported attack success rates, so WARP generates more realistic prompt-injection tasks that models are less likely to flag.
Jasmine Li, Ashton Chew, Maxwell Lin, Eliot Jones, Xiaohan Fu, Andy Zou
Across 11 constitutions, character training mostly instills the values it targets, while also shifting how models score on values it was never meant to change.
Bhagyesh Kumar, Ananya Sutradhar, Saurav Panigrahi, Jonathn Chang, Lionel Levine
A framework and six proposed metrics for tracking whether AI is eroding people's ability to shape the outcomes that affect their lives.
Je Qin Chooi, Jaeho Lee, Jasmine Li
In a Bayesian persuasion model, a human advised by a misaligned AI can gain at most 50% more utility than acting on prior knowledge alone, and we show where that bound tightens.
Eric Yachbes, Eva Tardos
A cross-disciplinary account of how AI threatens humanity's collective capacity to know things accurately, reason well, and maintain a healthy information environment.
Jasmine Li, et al.
Prompting or training models to want evaluators to get accurate results closed 70–100% of the eval gaming gap in five of eight model settings, which may scale better than hiding evaluations from models.
Jasmine Li, Alex Turner
PLOT uses optimal transport to find where a neural network represents high-level causal variables, matching the accuracy of distributed alignment search at a fraction of its runtime.
Jonathn Chang, Arya Datla, Ziv Goldfeld
DemPO brings sortition, the method used to select citizen assemblies, to RLHF rater pools and produces models that better reflect the values of a representative public.
Suvadip Sana, Jinzhou Wu, Martin T. Wells
A comprehensive survey of AI deception that covers how it emerges, how it can be detected, and which technical and governance measures can address it.
Boyuan Chen, Sitong Fang, Jiaming Ji, Yanxu Zhu, Pengcheng Wen, Jinzhou Wu, et al.
Early experiments on agents that covertly solve a hidden task while answering a main question, including a reinforcement learning result where an 8B model learns to exploit a GPT-4o monitor.
Vincent Cheng, Thomas Kwa, Neev Parikh
EigenBench ranks language models by their alignment with a chosen constitution by having the models judge one another, and its scores closely match human evaluations.
Jonathn Chang, Leonhard Piff, Suvadip Sana, Jasmine Li, Lionel Levine
When Claude, GPT, and Gemini models try to hide what they are reasoning about from a monitor, they lose substantial answer accuracy, and stronger monitors catch them more often.
Vincent Cheng, Thomas Kwa
Contrastive activation addition steers refusal best at early to middle layers and becomes less effective as Llama 2 models grow from 7B to 70B parameters.
Sheikh Abdur Raheem Ali, Justin Xu, Ivory Yang, Jasmine Li, Ayse Arslan, Clark Benham
A way to measure how much a set of probability estimates conflict, with applications to eliciting language model beliefs and merging expert forecasts.
Rowan Hess, Lionel Levine
ProgressGym uses nine centuries of historical text to benchmark alignment methods that can follow, anticipate, and respond to moral progress over time.
Tianyi Qiu, Yang Zhang, Xuchuan Huang, Jasmine Li, Jiaming Ji, Yaodong Yang
Language models carry information forward for future tokens mostly as a byproduct of what helps the current prediction, though deliberate pre-caching grows with model scale.
Wilson Wu, John X. Morris, Lionel Levine
