Lectures

Lectures

Rocco's lecture notes will be posted soon after the class.

Warning: the notes below were generated in real time and have not been edited. They may contain typos or other errors. They may also contain aesthetically jarring color combinations.

Number	Date	Topics	Notes	References
1	Tues Sept 2	Introduction, basics	PDF	Blum survey sec. 3.0
2	Thurs Sept 4	Online mistake-bound learning, elimination algorithm, decision lists	PDF	Blum survey sec. 3.0, 3.1
3	Tues Sept 9	Learning decision lists, Winnow1	PDF	Blum survey sec. 3.2, Littlestone paper sec. 5 (just through Theorem 7)
4	Thurs Sept 11	Winnow2, Perceptron	PDF	Blum survey sec. 3.2, Littlestone paper sec. 5 (just through Theorem 7), handout on Perceptron and kernel methods
5	Tues Sept 16	Perceptron, dual Perceptron, kernel methods	PDF	handout on Perceptron and kernel methods
6	Thurs Sept 18	General bounds on OLMB learning: Halving Algorithm, Randomized H.A., start VC Dimension	PDF	Blum survey sec. 2.0, 2.1, 2.2, Littlestone paper sec. 1-3 (don't worry about the stuff about the SOA)
7	Tues Sept 23	General bounds on OLMB learning: VC dimension, Weighted Majority algorithm	PDF	Blum survey sec. 2.0, 2.1, 2.2, Littlestone paper sec. 1-3 (don't worry about the stuff about the SOA)
8	Thurs Sept 25	Randomized Weighted Majority algorithm, intro to PAC learning, PAC learning intervals	PDF	Kearns and Vazirani chapter 1.1-1.3
9	Tues Sept 29	finish PAC learning intervals, OLMB to PAC conversion, definitional issues	PDF	Kearns and Vazirani chapter 1.1-1.3
10	Thurs Oct 2	Chernoff bounds, learning by finding consistent hypotheses, Occam's Razor	PDF	Kearns and Vazirani chapters 1,2, appendix (Chapter 9), this handout on probability basics, this handout on Chernoff bounds
11	Tues Oct 7	PAC sample-efficient learning sparse disjunctions via Occam and greedy set cover, start proper versus improper learning	PDF	Kearns and Vazirani chapters 1,2
12	Thurs Oct 9	Improper PAC learning of 3-term DNF is computationally easy, proper PAC learning of 3-term DNF is computationally hard	PDF	Kearns and Vazirani chapters 1,2
13	Tues Oct 14	Lower bound on PAC learning sample complexity based on VC dimension; start upper bound	PDF	Kearns and Vazirani chapter 3
14	Thurs Oct 16	Upper bound on PAC learning sample complexity based on VC dimension: Sauer-Shelah-Perles lemm, ``double sample'' argument	PDF	Kearns and Vazirani chapter 3
15	Tues Oct 21	No lecture (midterm exam)
15	Thurs Oct 23	Upper bound on PAC learning sample complexity based on VC dimension: ``double sample'' argument, application to PAC learning LTFs over \R^n	PDF	Kearns and Vazirani chapter 3 (you can peek at Chapter 4.0-4.3.2 as a head start for next time)
16	Tues Oct 28	Confidence boosting; accuracy boosting overview; start simple 3-stage accuracy improving procedure		Kearns and Vazirani Chapter 4.0-4.3.2

Schedule of Topics

Here is an anticipated list of topics. Note that the ordering of some topics may change, and we may spend more or less than one lecture per topic.

• Introduction to machine learning theory. Learning models and learning problems.
• Online mistake bound learning. Algorithms for simple concept classes. Attribute efficient learning with the Winnow algorithm.
• Winnow algorithm continued. Perceptron algorithm. General bounds for online mistake bound learning. The Halving algorithm, the Standard Optimal Algorithm.
• Best Experts and Weighted Majority.
• The Probably Approximately Correct(PAC) Learning model. PAC learning algorithms for simple concept classes.
• More PAC learning algorithms. Conversions from online learning to PAC learning. (KV chapter 1)
• Occam's Razor: learning by finding consistent hypotheses. Applications (KV chapter 2).
• Computational hardness results for finding consistent hypotheses.
• Vapnik-Chervonenkis dimension. Upper and lower bounds on sample complexity (KV chapter 3).
• VC dimension and sample complexity continued.
• VC dimension and sample complexity continued.
• Weak learning, strong learning, and Boosting (KV chapter 4).
• Boosting continued.
• Boosting continued.
• Learning in the presence of noise. Classification noise, malicious noice,
• Statistical Query learning (KV chapter 5).
• Learning with noise continued.
• Learning with noise continued.
• Cryptographic limitations on efficient learning (KV chapter 6).
• Cryptographic limitations on learning continued.
• Cryptographic limitations and reductions in PAC learning (KV chapters 6,7).
• The model of exact learning from membership and equivalence queries. Learning monotone DNF formulas.
• Learning deterministic finite automata.
• Learning deterministic finite automata continued.