Webpage for STAT240 (Robust Statistics)

Instructor: Jacob Steinhardt (jsteinhardt@berkeley)
Lectures: T/Th 3:30-5 (Zoom)
Office Hours: F 2-3 (Zoom)

TA: Serena Wang (serenalwang@berkeley)
Office Hours: Th 2:30-3:30 (Zoom)

Syllabus: link

IMPORTANT: If you plan to take the class, sign up here to be added to the class mailing list. We will use this to communicate instead of bCourses.

Prerequisites

No formal requirements, but this class will be fast-paced and assume mathematical maturity.

Lecture Notes

Updated periodically: link (last update 02/25/2021, through Lecture 14)

Please e-mail typos/corrections to me (jsteinhardt@berkeley with a dot edu at the end).

See here if you prefer notes broken up on a per-lecture basis.

Problem Sets

Problem Set 1 (due February 9th before class [note extension]) tex source
Problem Set 2 (due February 23rd before class) tex source
Problem Set 3 (due March 9th before class) tex source

Schedule

Feedback form for lectures

Lecture 1: Overview and 1D Robust Estimation (video) (tablet)
Lecture 2: Minimum Distance Functionals and Resilience (video) (tablet)
Lecture 3: Concentration Inequalities (video) (tablet)
Lecture 4: Bounding Suprema via Concentration Inequalities (video) (tablet)
Lecture 5: Finite-Sample Analysis via Generalized KS Distance (video) (tablet)
Lecture 6: Finite-Sample Analysis via Expanding the Destination Set (video) (tablet)
Lecture 7: Truncated Moments and Ledoux-Talagrand (video) (tablet)
Lecture 8: Efficient Algorithms: Projecting onto Maximum Eigenvector (video) (tablet)
Lecture 9: Approximation Oracles and Grothendieck's Inequality (video) (tablet)
Lecture 10: Resilience Beyond Mean Estimation (video) (tablet)
Lecture 11: Resilience For Linear Regression (video) (tablet)
Lecture 12: Efficient Algorithms for Robust Linear Regression (video) (tablet)
Lecture 13: Resilience for Wasserstein Distances
Lecture 14: Wasserstein Resilience for Moment Estimation and Linear Regression
Lecture 15: Model Mis-specification in Generalized Linear Models
Lecture 16: Robust Inference via the Bootstrap
Lecture 17: Robust Inference via Partial Specification
Lecture 18: Nonparametric Regression I
Lecture 19: Nonparametric Regression II
Lecture 20: Domain Adaptation under Covariate Shift
Lecture 21: Doubly-Robust Estimators and Semi-Parametric Estimation
Lecture 22: Neural Networks and Pre-training
Lecture 23: Robustness of Neural Networks
Lecture 24: Scaling Laws for Neural Networks
Lecture 25: Partial Specification and Agnostic Clustering
Lecture 26: Agnostic Clustering via Resilience
Lecture 27: Efficient Clustering via SVD + k-means

Supplementary Reading List

These are not necessary to follow the class, but may be interesting further reading.

Jerry Li taught a class related to the first 14 lectures.

Robust Learning: Information Theory and Algorithms (Jacob Steinhardt's thesis)
Concentration of Measure (lecture notes by Terence Tao)
  • Alternate reference: Concentration Inequalities (notes by Boucheron, Lugosi, and Bousquet)
    Generalized Resilience and Robust Statistics (Zhu, Jiao, Steinhardt)
    Principled Approaches to Robust Machine Learning and Beyond (Jerry Li's thesis)
    Probability Bounds (John Duchi; contains exposition on Ledoux-Talagrand)
    Approximating the Cut-Norm via Grothendieck's Inequality (Alon and Naor)
    Better Agnostic Clustering via Relaxed Tensor Norms (Kothari and Steinhardt)
    Ricci curvature of Markov chains on metric spaces (Ollivier; relation between Poincaré inequalities and Markov chain convergence)
    Concentration inequalities: A nonasymptotic theory of independence (Boucheron, Lugosi, and Massart; good general survey that contains discussion of Poincaré inequalities)
    Provable Defenses against Adversarial Examples via the Convex Outer Adversarial Polytope (Eric Wong and Zico Kolter)
    Training Verified Learners with Learned Verifiers (Krishnamurthy Dvijotham et al.)
    Semidefinite relaxations for certifying robustness to adversarial examples (Aditi Raghunathan et al.)