Stat 700
Probabilistic graphical models
Instructor: Long Nguyen
Class Hours: Thursday, 1--3pm, Location: 1250 USB
Office Hour: Weds 1:30--2:30pm, 445B West Hall (or by appointment)
Fall 2009
[ Description in pdf]
[ Reading Assignments]
[ Homeworks]
[ Projects]
[ Topics ]
[ Text books ]
[ Prerequisites ]
[ Structure/Eval]
[ Data sets ]
Announcements
- Nov 13: Homework 4 is issued.
- Oct 26: Homework 3 is issued.
- Oct 15: Partial solutions to homework1 is posted in Ctools.
- Homework 2 is out on Oct 6.
- A typo in problem 6(c) (Homework 1): The cost of the computation
of $p(x_i)$ is (at most) proportional to the number of edges in $G$.
- Office hour changed to Weds 10-11am, or by appointment.
- Homework 1 is out on Thurs Sept 17.
- Basics on graphical models, Markov properties, recursive decomposability, elimination algorithms
- Sum-product algorithm, factor graphs, semi-rings
- Frequentist and Bayesian methods
- Bayesian classification, linear models and generalized linear (GLIM) models, on-line methods
- Exponential family, sufficiency, conjugacy
- Legendre-Fenchel duality, information theoretic concepts
- The EM algorithm
- Mixture models, conditional mixture models, hierarchical mixture models
- Hidden Markov models (HMM) and forward-backward algorithms
- Factor analysis, principal component analysis (PCA), canonical correlation analysis (CCA), independent component analysis (ICA)
- Kalman filtering and Rauch-Tung-Striebel smoothing
- Markov properties of graphical models
- Junction tree algorithm
- Chains, trees, factorial models, coupled models, layered models
- Importance sampling, Markov Chain Monte Carlo methods (Gibbs/ Metropolis-Hastings samplers)
- Variational inference algorithms: mean field, belief propagation, convex relaxations
- Model selection, marginal likelihood, AIC, BIC and MDL
- Nonparametric methods: Gaussian processes, Dirichlet processes, infinite mixture models
- Applications to signal processing, bioinformatics, communication, natural language processing, computer vision etc.
The prerequisites are previous coursework in linear algebra, multivariatecalculus, and basic probability and statistics. Previous coursework
in graph theory, information theory, optimization theory and statistical
physics would be helpful but is not required. Familiarity with R,
Matlab, Splus or a related matrix-oriented programming language will
be necessary.
Required text:
-
M. I. Jordan, An introduction to probabilistic graphical models.
This book is not yet published.
Hard copies of selected book chapters will be distributed in a classpack
from the Dollar Bill Copying on Church Street.
Additional reading assignments will be posted in the course website.
Supplementary texts:
- S. Lauritzen, Graphical models, Publisher: Oxford University
Press, 1996.
- J. Whittaker, Graphical Models in Applied Multivariate Statistics,
Wiley, 2009.
- N. Cressie, Statistics for Spatial Data, Wiley, NY, 1993.
- S. Banerjee, B.P. Carlin and A.E. Gelfand, Hierarchical Modeling
and Analysis for Spatial Data, Chapman and Hall/CRC Press, 2004.
This will be continuously updated to reflect the true progress.
-
9/10: Intro to GM's and exact inference: GM with directed
graphs, undirected graphs, factor graphs; Elimination algorithms,
probability propagation algorithms
-
Graphical models. M. I. Jordan. Statistical Science (Special Issue on Bayesian Statistics), 19, 140-155, 2004. Section 3 in this overview article
can be skipped at the first reading.
- Read MJ's Chapter 2.
- 9/17: Continue on exact inference (sum-product, max-product algorithms),
basic statistical concepts (frequentist and Bayesian estimation methods)
- Chapters 3/4, and beginning of Chapter 5
- Hw 1 out, due 9/24
-
9/24: Canonical estimation problems from the viewpoint of graphical
models (density estimation, linear regression and classification;
supervised vs unsupervised learning; generative vs discriminative
models )
- 10/1: Exponential family, convexity and Legendre-Fenchel duality,
sufficiency and some information-theoretic concepts, conjugate priors
- Chapter 8, and Section 3 of the Wainwright-Jordan paper (see the
link below)
-
10/8: Continue on exponential families
-
10/15: Mixture models and estimation(Chap 9/10/11):
Completely observed GMs, mixtures and conditional mixtures, the
EM algorithm
-
10/22: Continue on mixture models and the EM algorithm
-
10/29:
Hidden Markov models (Chap 12), Multivariate Gaussians (Chap 13)
-
11/5:
Factor analysis (Chap 14), Kalman filter and smoothing (Chap 15)
-
11/12: Approximate inference: Sampling methods (important sampling, Markov chain Monte Carlo)
-
11/19:
The junction tree algorithm (Chap 17),
Approximate inference: variational inference
methods for Markov random fields and directed GM's
-
11/26: Thanksgiving break
-
12/3: Variational inference (cont.); Model selection (Chap 26)
-
12/10: Model selection methods:
- Frequentist approaches using penalties on model complexity
- Bayesian approaches via the marginal likelihood
- Nonparametric Bayesian methods using random measures/
stochastic processes as the prior on models
(Link to relevant Data sets )
- Homework 1 (out Sep 17, due Sep 24) [ pdf ]
- Homework 2 (out Oct 6, due Oct 15) [ pdf ]
- Homework 3 (out Oct 26, due Nov 5) [ pdf ]
- Homework 4 (out Nov 13, due Dec 3) [ pdf ]
Suggestions for possible class projects.
The course will meet once a week and will follow a regular lecture format.
There will be bi-weekly homework assignments, due (approximately)
one week after being passed out. There will be a final course project
in forms of either a survey paper, an application research or a
methodological research project. The homeworks count for 60%
of the grade, and the project counts for 40% of the grade.