CS 281A / Stat 241A, Spring 2007:

Statistical Learning Theory

People

			Office hours
Professor	Peter Bartlett	bartlett@cs	Tue 3-4, 399 Evans. Wed 3-4, 723 Soda.
TAs	Alexandre Bouchard	bouchard@cs	Tuesday 1-2, Thursday 1-2, 751 Soda.
	Mikhail Traskin	mtraskin@stat	Wednesday 10-12, 387 Evans.

Lectures: Soda 306. Tuesday/Thursday 11-12:30.

Discussion section: Evans 334. Thursday 5-6.

Course description

This course will provide an introduction to probabilistic and computational methods for the statistical modeling of complex, multivariate data. It will concentrate on graphical models, and in particular issues of representation, estimation, and inference in parametric models. [More details: html pdf]

Prerequisites:

The prerequisites are previous coursework in linear algebra, multivariate calculus, and basic probability and statistics. Previous coursework in graph theory, information theory, and optimization theory would be helpful but is not required. Familiarity with Matlab, Splus or a related matrix-oriented programming language will be necessary.

Text book:

The course will follow the unpublished manuscript An Introduction to Probabilistic Graphical Models, by Michael I. Jordan, which is available from Copy Central at 2483 Hearst Avenue. It is reader number 54 (ask for CS C281A) and costs $23.
For an introduction, see the review paper Michael I. Jordan, Graphical models. .

Assignments:

There will be a substantial project (40% of the grade), plus regular homework assignments (60% of the grade; approximately one every two weeks).
It is appropriate to discuss homework assignments with other students, but homeworks must be written up individually. If you discuss the assignment with other students, please list their names on your homework. See the policy on academic dishonesty.

Homework 1, due 11am, Thursday February 8: homework: pdf solution: pdf
Homework 2, due 11am, Tuesday February 27: homework: pdf data: ascii solution: pdf
Homework 3, due 11am, Thursday March 15: homework: pdf (Revised March 8, 2007) hw3-1.data hw3-2.data solution: pdf
Homework 4, due 5pm, Friday, April 6 (see announcement): homework: pdf (Revised March 23, 2007) hw4-1.data hw4-3.data hw4-3.test solution: pdf
Homework 5 (the final homework) is due 11am, Tuesday, April 17: homework: pdf (Revised April 11, 2007) hw5-1.data hw5-2.data hw5-2.true solution: pdf

Lectures:

The chapter numbers under 'Reading' refer to the text An Introduction to Probabilistic Graphical Models, Michael I. Jordan.

	Topic	Reading
Jan 16	Introduction; directed graphical models	1, 2
Jan 18	directed graphical models	2.1
Jan 23	undirected graphical models	2.2, 2.3
Jan 25	elimination algorithm (guest lecturer: Martin Wainwright)	3
Jan 30	graph elimination, sum-product algorithm	3,4
Feb 1	factor graphs, poly trees	4
Feb 6	max-product algorithm, parameter estimation	4,5
Feb 8	parameter estimation	5
Feb 13	linear regression	6
Feb 15	linear classification	7
Feb 20	exponential family	8 (+ notes on exponential family
Feb 22	exponential family	8 (+ notes on exponential family
Feb 27	estimation with complete observations	9 (+ notes on chordal graphs)
Mar 1	estimation with complete observations: IPF	9
Mar 6	estimation with hidden variables: EM	10
Mar 8	estimation with hidden variables: EM	11 (+ notes on EM implementation)
Mar 13	inference in HMMs	12
Mar 15	estimation in HMMs	12
Mar 20	factor analysis	14
Mar 22	state space models	15
Mar 27	Spring Break
Mar 29	Spring Break
Apr 3	inference and estimation in state space models	15 (+ An approach to time series smoothing and forecasting using the EM algorithm. R. Shumway and D. Stoffer. J. Time Series Analysis 3(4):253-264, 1982.) (+ notes on ML Gaussian estimates)
Apr 5	junction tree	17
Apr 10	junction tree, HMMs	17,18
Apr 12	junction tree and HMMs	18
Apr 17	junction tree and Kalman filter; maximum entropy	18,19
Apr 19	iterative scaling	20
Apr 24	iterative scaling	20
Apr 26	approximate inference: variational methods (guest lecturer: Alex Bouchard)	A variational principle for graphical models. M. Wainwright and M. Jordan. 2005. Tutorial on variational approximation methods. T. Jaakkola. 2000. Alex's slides
May 1	approximate inference: sampling methods	21
May 3	CS281A project poster session
May 8	Stat241A project poster session

Announcements:

[Wed, May 2] The poster session on Thu May 3 and Tue May 8 will be in the usual lecture time and place. It will be just like a poster presentation at a conference: If you are presenting a poster, you will need to stick your poster up on a wall somewhere (we'll have materials to do that), stand beside it, and answer any questions that come up as others wander by and look at it. It's common to offer to give a brief overview of the material presented in your poster, so be prepared to do that. Remember it should be brief - each person will be spending only a small fraction (approx two minutes) of the poster session in front of your poster.
[Tue, May 1] There will not be a discussion section on Thu May 3.
[Fri, Apr 20] The project report is due at 5pm on May 10 (note the extension). Please submit the report (in hard copy) to the box outside Soda 723. This deadline is firm; reports submitted after 5pm on May 10 will not be accepted.
[Thu, Apr 12] The project poster presentations will be during the lecture periods on Thursday, May 3 and Tuesday, May 8. If you are enrolled in CS281A, please present your poster in the session on May 3. If you are enrolled in Stat241A, please present it on May 8. If you cannot present your poster on the assigned day, please arrange to swap with someone in the other session and send me an email describing the swap. For joint projects with both courses represented, choose a day and email me letting me know.
[Wed, Apr 11] Homework 5 has been modified: In the first sentence of Question 1, the number of locations should be 2n+1, not n. Homework 5 is due at the lecture (11am) on Tuesday, April 17.
[Wed, Apr 4] The deadline for homework 4 has been extended to 5pm, Friday April 6. Please leave homeworks in the box at the door of Soda 723. Late homeworks will not be accepted.
[Fri, Mar 23] Homework 4 has been posted (and the final question added today). It is due at the lecture (11am) on Thursday, April 5.
[Thu, Mar 8] Homework 3 has been posted (and some clarifications added today). It is due at the lecture (11am) on Thursday, March 15.
[Thu, Mar 8] The project proposal is due on March 20. Some additional information on projects is available: [Projects]
[Thu, Mar 1] Notes on chordal graphs are available: [pdf]
[Tue, Feb 27] Graded homeworks will be available in the discussion sections.
[Thu, Feb 22] Alex's office hours will change from Thursday 12:30-2:30 to Tuesday 1-2 and Thursday 1-2, starting on Tuesday, February 27. Still in 751 Soda.
[Tue, Feb 20] A course wiki has been created. Go to http://bspace.berkeley.edu, log in through calnet, and search for "COMPSCI 281 Sp07." It will include an FAQ, a list of text typos, and a discussion section wish list.
[Thu, Feb 15] Homework 2 has been posted. It is due at the lecture (11am) on Tuesday, February 27.
[Mon, Jan 29] Homework 1 has been posted: pdf. It is due at the lecture (11am) on Thursday February 8.
[Thu, Jan 18] Martin Wainwright will give a guest lecture on Thursday, Jan 25.
[Tue, Jan 16] The discussion section starts on Thursday, Jan 18. It is in Evans 334, 5-6pm.