Unsupervised Machine Learning

Tentative Schedule

Introduction
Introduction, classic problems in unsupervised learning, overview of: clustering, dimensionality reduction, density estimation, discoversing intrinsic structure and organizing data
[slides]

reading: review your math
Topic 1
Metrics spaces and coverings, clustering in metric spaces, k-center problem, k-means problem, hardness results, approximation guarantees, other variants
[lec1,2, lec3,4, slides]

clustering general:
- hartigan
- gonzalez

k-means reading:
- NP hardness
- approximation for local updates
- smart seeding
Topic 2
More clustering: hierarchical, spectral, axiomatic view, impossibility theorem, clustering graph data and planted partition models
[lec5,6, lec7,8, slides1, slides2]

more clustering reading:
- hierarchical clustering guarantees
- tutorial on spectral clustering
- axiomatic view
- impossibility theorem
- graph clustering in planted partitioning models
Topic 3
Dimensionality reduction, embeddings in metric spaces, Frechet and Bourgain embeddings, linear dimensionality reduction, Principal Components Aanalysis (PCA), Factor Analysis (FA), Independent Component Analysis (ICA), Blind Source Separaction (BSS), randomized maps and Johnson-Lindenstrauss Lemma
[lec9,10, lec11, lec12, slides1, slides2]

reading:
- notes on Bourgain's theorem
- linear dimension reduction survey
- probabilistic proof of JL lemma
Topic 4
Non-linear dimensionality reduction, manifold learning, spectral methods: (LLE, isomap, LE, HE, LTSA, ...), tSNE, other techniques
[lec13,14, lec15,16, lec17, tsne slides, slides ]

reading:
- non-linear dimension reduction survey
- tSNE, guarantee
- random projections for manifolds
- algorithmic construction for Nash's embedding
Topic 5
Density estimation minimax results, assumed structure: Gaussian mixture models, latent dirichelet allocation (LDA), tensor methods to learn latent models
[lec18,19,20]

reading:
- L2 density estimation rates
- learning mixtures of Gaussians
- latent dirichelet allocation (LDA)
- tensor decompositions for latent models
Topic 6
Structure discovery, horseshoe effect, topological data analysis, Fast near neighbor search, locality sensitive hashing
[lec21,22, slides (tensors), lec23 slides (tda)]

reading:
- horseshoe effect
- toplogical data analysis
- near neighbor search (short survey)
- locality sensitive hashing (LSH)
Topic 7
Text and Document Embeddings, Latent Semantic Analysis (LSA), Latend Dirichlet Allocation (LDA), parameter learning using Method of Moments, Word2Vec, Theory behind Analogies and Vector Additivity in Skipgram Models
[slides]

reading:
- Latent Dirichlet Allocation (LDA)
- Word2Vec (Skipgram Model)
- Why analogies occur as vector additivity in Skipgrams
Topic 8
Beyond Euclidean Emeddings, Introduction to Hyperbolic Embeddings (and beyond)
[slides]

reading:
- Hyperbolic Geometry on the Hyperboloid
- Sarkar's Construction for (approx.) isometric embeddings of trees in hyperbolic spaces
- Hyperbolic MDS
- Hyperbolic SVMs
- Hyperbolic (and manifold) Metric Learning

Resources

Papers
A list of relevant papers on Unsupervised Learning can be found here
Books on ML
The Elements of Statistical Learning by Hastie, Tibshirani and Friedman (link)
Pattern Recognition and Machine Learning by Bishop (link)
A Course in Machine Learning by Daume (link)
Deep Learning by Goodfellow, Bengio and Courville (link)
Software
MATLAB: download info, learning the basics.
Python: download info, tutorial
LaTeX: download info, learning the basics, typesetting common math formulas.
Background
Probability: Events, random variables, expectations, joint, conditional and marginal distributions, and independence. (refresher 1, refresher 2, refresher 3, refresher 4)
Statistics: Bayes' Rule, Priors, Posteriors, Maximum Likelihood Principle (MLE), Basic distributions such as Bernoulli, Binomial, Multinomial, Poisson, Gaussian. (refresher, reference sheet)
Linear Algebra: Vector spaces, subspaces, matrix inversion, matrix multiplication, linear independence, rank, determinants, orthonormality, basis, solving systems of linear equations. (refresher 1, refresher 2, refresher 3, refresher 4)

Multivariate Calculus: Take derivatives and integrals of common functions, gradient, Jacobian, Hessian, compute maxima and minima of common functions. (basic calculus identities, multivariable differentiation, extrema refresher, refresher 1, refresher 2)

Algorithm basics: Ability to write and analyze simple algorithms; basic understanding of time and space complexity. (refresher 1, refresher 2)

Mathematical maturity: Ability to communicate technical ideas clearly. (refresher 1, refresher 2)

Programming: Ability to program in a high-level language, and familiarity with basic algorithm and datastructure design and good coding principles.
Misc.
The Matrix Cookbook
Convex Optimization

Policies

Homework submission & formatting
Homeworks will contain a mix of programming and written assignments.

The written segment of the homeworks must be typesetted as a PDF document, with all mathematical formulas properly formatted. It is strongly recommended that you prepare your write-up as a LaTeX document (you can use the following templates: homework.tex, homework.cls and homework.pdf); see the Resources section for more details on LaTeX. An MS Word writeup is also fine; you must make sure that all math formulas are typesetted properly.

The programming segment can be done in any high-level programming language. You are expected to have good working knowledge of the programming language you choose, and will receive very little 'debugging' help from the staff. You may use supporting math/scientific libraries for auxiliary tasks (such as matrix multiplication, matrix inversion, etc.), however you must write code for the core algorithms yourself.

Please include your name and UNI on the first page of the written assignment and at the top level comment of your programming assignment.

The written segment of the homework (including plots and comparative experimental studies) must be submitted via Gradescope, and (if the homeworks specifies) the a tarball of the programming files should be handed to the TA by the specified due dates. No late homeworks will be accepted.
Collaboration
Discussion of the homework problems is encouraged, but you must write the solution individually or in small groups of 2-3 students (as specified in the Homeworks). Detailed discussion of the solution must only be discussed within the group. General discussion on problem clarification and possible approaches can be discussed with others over Piazza. Each group must write up their own solutions independently.
Late submissions
No late homeworks will be accepted.
Academic rules of conduct
Students are expected to adhere to the Academic Honesty policy of the Computer Science Department, this policy can be found in full here.
Violations
Violation of any portion of these policies will result in a penalty to be assessed at the instructor's discretion. This may include receiving a zero grade for the assignment in question and a failing grade for the whole course, even for the first infraction.

Unsupervised Learning

COMS 4774 Fall 2022

Tentative Schedule

Course Info

Resources

Policies