Probability Theory assisted by Information Theory

 

Navigation

Introduction Subjects Prerequests Time and Place Notes Syllabus Exercises Links

The purpose of the course is to prove the most important theorems in probability theory using methods from information theory. The chosen theorems will be formulated in stronger versions than normally found in the literature and the proofs will be more direct. Finally, this approach will be closer to statistical practice than one normally finds in probability theory.  

back to top

 

Subjects

  • Entropy .
  • Information divergence (Kullback-Leibler discrimination).
  • Law of Large Numbers.
  • Poisson's Law.
  • The Central Limit Theorem.
  • Ergodicity.
  • Markov chains.
  • Conditional Limit Theorem and Sanov property.
  • Convergence of martingales and inverse martingales.

A detailed list of topics which have been discussed in the lectures can be found here.

back to top

 

Prerequests

Knowledge of elementary probability theory and measure theory.

back to top

 

Time and place

Wednesday 8-10 in Aud. 7 at H. C. Ørstedsinstituttet.  back to top

Notes

An updated version of the notes is available here. Any feed-back on the notes is deeply appreciated.

PostScript

PDF

Updated

Part 1

Part 1

13/11

7-8

7-8

12/11

9

9

12/11

11-12

11-12

16/12

14-16

14-16

16/12

Part 4

Part 4

16/12

bibliography

bibliography

3/9

 

back to top

 

Syllabus

3/9: Coding and Kraft’s inequality. Entropy and divergence.

10/9: Inequalities, convexity and continuity.

17/9: Pinsker’s inequality and Poisson's law.

24/9: No lecture.

1/10: Universal source coding and information projection.

8/10: Information projection.

22/10: Strong Law of Large Numbers.

29/10: Increasing and decreasing information.

5/11: Markov chains

12/11: Reversible Markov chains and Ergodicity.

19/11: The Conditional Limit Theorem and Sanov property. Stein's Lemma and Chernoff information.

26/11: Convergence of direct and inverse martingales.

3/12: Hewit-Savage 0-1 Law. Central Limit Theorem.

10/12: The Central Limit Theorem.

17/12: Edgeworth expansion.

back to top

 

A Mathematical Theory of Communication by Claude Shannon 1948.

Entropy on the world wide web.

Information Theory Society contains a lot of links to relevant material.

Lecture notes on information theory made by Flemming Topsøe.

Lecture notes on inequalities of information theory made by Flemming Topsøe.

 

 

Last modified: 12/11 2003 by Peter Harremoës - moes@math.ku.dk