University of California at Berkeley
Dept. of Electrical Engineering & Computer Science
Dept. of Statistics

EECS 281a / STAT 241a
Statistical Learning Theory --- Graphical Models

Fall Semester 2011

Practical information
Course description
Updates
Supplementary reading
Handouts

Practical information

Lectures: 306 Soda Hall, Monday and Wednesday, 10:30--12.

Recitations (optional): 306 Soda Hall, Thursday 5:30--7pm.

Course reader: An Introduction to Probabilistic Graphical Models, by M. Jordan. Available at Northside Copy Central, Hearst & Euclid, starting 8/31.

Grading: Homework (60%) and Course Projects (40%).

Instructors:

Michael Jordan

Office Hours: Wednesday 12--1pm (427 Evans)

Email: jordan AT eecs DOT berkeley DOT edu

Phone: 642-3806

Office: 427 Evans Hall

Martin Wainwright

Office Hours: Monday 12--1pm (263 Cory)

Email: wainwrig AT eecs DOT berkeley DOT edu

Phone: 643-1978

Office: 263 Cory Hall

Graduate student instructors:

Yangqing Jia

Office Hours: Thursday 4:30--5:30pm (751 Soda Hall)

Email: jiayq AT eecs DOT berkeley DOT edu

Po-Ling Loh

Office Hours: Tuesday 3:30--4:30pm (307 Evans)

Email: ploh AT berkeley DOT edu

Course description: This course is a 3-unit course that provides an introduction to the area of probabilistic models based on graphs. These graphical models provide a very flexible and powerful framework for capturing statistical dependencies in complex, multivariate data. Key issues to be addressed include representation, efficient algorithms, inference, and statistical estimation. These concepts will be illustrated using examples drawn from various application domains, including machine learning, signal processing, communication theory, computational biology, computer vision, etc.

Outline:

Basics on graphical models, Markov properties, recursive decomposability, elimination algorithms
Sum-product algorithm, factor graphs, semi-rings
Markov properties of graphical models
Junction tree algorithm
Chains, trees, factorial models, coupled models, layered models
Kalman filtering and Rauch-Tung-Striebel smoothing
Hidden Markov models (HMM) and forward-backward
Exponential family, sufficiency, conjugacy
Frequentist and Bayesian methods
The EM algorithm
Conditional mixture models, hierarchical mixture models
Factor analysis, principal component analysis (PCA), canonical correlation analysis (CCA), independent component analysis (ICA)
Importance sampling, Gibbs sampling, Metropolis-Hastings
Variational algorithms: mean field, belief propagation, convex relaxations
Dynamical graphical models
Model selection, marginal likelihood, AIC, BIC, and MDL
Applications to signal processing, bioinformatics, communication, computer vision, etc.

Required background: The prerequisites are previous coursework in linear algebra, multivariate calculus, basic probability and statistics (at the level of EE 126). Some degree of mathematical maturity is also required. Coursework or background in graph theory, information theory, optimization theory, and statistical physics is relevant, and could be helpful but is not required. Familiarity with a matrix-oriented programming language (e.g., MATLAB, R, Splus, etc.) will be necessary.

Homework: Although it is acceptable for students to discuss the homework assignments with one another, each student must write up his/her homework on an individual basis. Each student must indicate with whom (if anyone) they discussed the homework problems. Homeworks must be turned in at the beginning of class on the due date. Late homeworks will not be accepted.

Course project: The course project will involve independent work on a topic of the student's own choosing. Course projects will be presented in an informal poster session at the end of semester, and the work will be summarized in a write-up.

Updates and Announcements

(12/07): HW #5 has been graded and was returned at the poster session today. Out of 40 points, the average was 32 with a standard deviation of 7.
(12/05): The poster session will be held at 1-3pm, in the Wozniak's Lounge (4th floor Soda). The final project write-up is also due on the same day *during the poster session*.
(11/29): HW #4 has been graded and will be returned on Wednesday. Solutions are posted below. Out of 35 points, the average was 27 with a standard deviation of 6.
(11/06): Prof. Wainwright will not be holding office hours on Mon, 11/28. Prof. Jordan will hold office hours in his place.
(11/15): This week's recitation will be in 405 Soda at the usual time (5:30-7pm Thursday). Next week, there will be no recitation due to Thanksgiving. The final recitation will be in 306 Soda from 5:30-7pm on 12/1.
(11/10): HW #5 has been posted and is due at the beginning of class on Wed, Nov 30, 2011. Download data files .
(11/06): Prof. Jordan will not be holding office hours on Weds, 11/9. Prof. Wainwright will hold office hours in his place.
(11/02): Clarification re HW 4.2(b): when verifying the correctness of your implementation, you should check that your algorithm returns a probability distribution that is the same as the one given by the specified closed form estimates. (Due to overcompleteness of the representation, your algorithm may not be comparable on a per edge/node basis.)
(10/31): Po-Ling's office hours this week will be Tues, Nov 1, from 1-2pm in 387 Evans.
(10/31): HW#3 has been graded and will be returned today. Solutions are posted below. Out of 40 points, the average was 36 with a standard deviation of 5.
(10/24): HW #4 has been posted and is due at the beginning of class on Monday, Nov 7, 2011. Download IPF data file .
(10/18): Supplementary notes on IPF/exp. families.
(10/17): HW #2 has been graded and will be returned today. Solutions are posted below. Out of 40 points, the average was 27 with a standard deviation of 9. Note that point totals will be reweighted so each homework set is worth the same number of points, and we will continue to grade just a subset of problems on future homeworks.
(10/10): HW #3 has been posted and is due at the beginning of class on Monday, Oct 24. Download LMS data file .
(9/28): HW #2 has been posted and is due at the beginning of class on Monday, Oct 10.
(9/28): The course projects will be due on Weds, Dec 7th, and the poster session will also be held on that day.
(9/26): Prof. Jordan's office hours will be held from 3-4 on Weds this week (9/28) instead of the usual 12-1 time slot.
(9/28): HW #2 has been posted and is due at the beginning of class on Monday, Oct 10.
(9/26): Prof. Jordan's office hours will be held from 3-4 on Weds this week (9/28) instead of the usual 12-1 time slot.
(9/26): HW #1 has been graded and will be returned today. Solutions are posted below. Out of 100 points, the average was 85 with a standard deviation of 10.
(9/20): We have managed to secure 330 Evans for recitations from 5:30--7pm on 9/22, 9/29, and 11/17. Therefore, all recitations will be held at the usual time (Thursday 5:30--7pm) until the end of the semester. Recitations are in 306 Soda except for the three dates listed above.
(9/12): This week's recitation (Thurs Sep 19, 5:30--7pm) will take place in 332 Evans. All remaining recitations will be held in 306 Soda. Recitations on 9/22, 9/29, and 11/17 will be from 4:30--6pm.
(9/8): Yangqing's office hour has changed to 4:30-5:30 due to the schedule of recitation sessions.
(9/8): Download HW #1. It is due on Monday, Sep 19th. The first two problems are review of undergraduate level probability.
(9/6): We have created a course mailing list. Go to lists.berkeley.edu and subscribe to cs281a. Please let Yangqing know if you encounter any problems when subscribing.
(9/6): This week's recitation (Thurs Sep 8, 5:30--7pm) will take place in 306 Soda.
(9/5): Prof. Wainwright will hold office hours (in place of Prof. Jordan's hour) at the same time, Wed 12--1pm (Sep 7th) in 263 Cory.
(9/1): The first recitation will be Friday, Sep 2, from 5:00--6:30pm at Soda 306 (note the date change). Further sessions' dates and place TBA.

Handouts

(12/29): Homework #5 Solutions. Sample R code for PCA and FA.
(11/29): Homework #4 Solutions. Sample Python code for IPF (4.2(a)).
(10/31): Homework #3 Solutions. Sample Python code for LMS problem.
(10/24): Homework #4.
(10/10): Homework #3.
(10/17): Homework #2 Solutions.
(9/28): Homework #2.
(9/26): Homework #1 Solutions.
(9/8): Homework #1.

Supplementary reading

Recitation 1: Fundamentals [pdf]
IPF Lecture (10/17): Supplementary notes on IPF/exp. families.
Exponential families and graphical models: more advanced monograph , by Wainwright and Jordan, Foundations and Trends in Machine Learning

EECS 281a / STAT 241a Statistical Learning Theory --- Graphical Models