Syllabus
Part I (The Basics):
*Fundamental Information Measures
*Entropy, Relative Entropy, and Mutual Information (for discrete distributions)
*Basic properties, convexity, log-sum inequality
*Definitions for general distributions
*f-divergence
*Key properties: tensorization, data-processing inequality, variational definition
----
* Compression and Gambling
*Definition of source codes
*Non-singular, uniquely-decodable, and instantenously-decodable codes
*Kraft's inequality, Optimal compression rate, achievability
*Connections to gambling on horse races
*Operational meanings of entropy, relative entropy, and mutual information
----
----------------------------------------
Part II (Applications):
Application 1: From Compression to Sequential Inference
*Universal Compression and gambling
*Principle of testing by betting
*Constructing Confidence Sequences
*Constructing sequential nonparametric tests
----
Application 2: Error exponents in hypothesis testing
*Basics of large deviations principle (LDP)
*Introduction to the method of types
*Sanov's Theorem and information projection
*Chernoff Stein Lemma
----
Application 3: Information Theoretic analysis of ML algorithms
*Mutual Information based generalization bounds
*Regret Analysis of Thompson Sampling [IF TIME PERMITS]
***OR***
Review and discussion of omitted topics
----