CS 1998: Intro to AI Safety & Alignment

Topics

• Why increasingly capable AI systems do not become safe or aligned by default.
• Evidence from current systems: sycophancy, deception, specification gaming, and agentic failure modes.
• Risk models spanning misuse, accidents, systemic harms, and loss of control.
• How to reason about uncertain, low-probability, high-impact outcomes.
• Capabilities forecasts, scenarios, and the assumptions behind them.

Topics

• The post-training pipeline: supervised fine-tuning, preference data, reward models, and policy optimization.
• RLHF with PPO and direct preference methods such as DPO.
• Constitutional AI, reinforcement learning from AI feedback, and rule-based alignment.
• What present-day alignment methods accomplish in practice.
• Known limitations: reward misspecification, overoptimization, plural values, and dependence on human oversight.

Topics

• Goodhart's law, proxy objectives, and specification gaming.
• Reward hacking and reward tampering in agents and language models.
• Goal misgeneralization versus ordinary capability failures under distribution shift.
• Sycophancy and emergent misalignment after narrow fine-tuning.
• Mitigation strategies and why behavioral training may not remove the underlying failure mode.

Topics

• What interpretability is for and what kinds of evidence it can provide.
• Probes, logit attribution, activation patching, and causal interventions.
• Sparse features, circuit tracing, and representation engineering.
• The Jacobian lens and the J-space hypothesis of a verbalizable global workspace.
• Limits of current methods and the challenge of scalable, safety-relevant auditing.

Topics

• Threat modeling and capability elicitation before benchmark design.
• Evaluations for cyber, biological, persuasion, deception, autonomy, and self-proliferation capabilities.
• Construct validity, contamination, sandbagging, and evaluation awareness.
• Manual red teaming, automated red teaming, and scalable evaluation pipelines.
• Using evaluation evidence to inform deployment safeguards and safety cases.

Topics

• The supervision gap and weak-to-strong generalization.
• Scalable oversight through debate, critique, decomposition, and recursive supervision.
• AI control protocols: trusted monitoring, untrusted monitoring, trusted editing, and defer-to-human strategies.
• Control evaluations against intentionally subversive model behavior.
• Safety-usefulness tradeoffs and evidence for an AI control safety case.

Topics

• Capability, algorithmic, and compute trends and what they can and cannot predict.
• Forecasting timelines and takeoff under deep uncertainty.
• Governance tools: evaluations, safety cases, transparency, incident reporting, and compute governance.
• Domestic regulation and international coordination at the frontier.
• Open research questions and technical, policy, and governance career paths.