Jacob J. Walker's Blog

Scholarly Thoughts, Research, and Journalism for Informal Peer Review

Calling all Fellow Math Nerds: Let’s Learn Information Theory Together

with 5 comments

I believe I finally have done enough research about statistical forms of regression, to see that the methods I will need to use in my doctoral research will require Bayesian methods and/or knowledge from Information Theory.  So I am starting to dive into these, starting with Information Theory, and I want to see if any of my friends are interested in having a study group to learn this together.  I believe if we do it with sufficient rigor, we can earn upper division math credit through LearningCounts; so this could be of value both personally and professionally.

I would like to meet as a group one time per week, probably after work, but weekends might be possible, depending upon what the group wants.  The location of the meeting will depend upon who is in the study group, and what is most central to all of us.  I am estimating the cost of materials to be around $25, after purchasing the various books we will use (Thank you Samamtha, for finding the free online version of the last textbook).   If you want to get math credit for this course, there are additional fees from LearningCounts, which currently would be about $275 ($125 for the portfolio review, and $149 for the required course on how to create a portfolio).

This is usually a semester long course, and I am estimating it will take us at least that long, especially since I want to start with easy conceptual materials before moving into the formula and algorithm driven materials.  So if we start sometime next week, then I’d like to be finished with the self-study by March or April.  Because this is a rigorous topic, in which we will be ultimately completing the curriculum of an upper division math course, I think those who want to be able to get the math credit from LearningCounts should be willing to dedicate 5 to 10 hours per week to the topic (1 to 2 hours per day), in addition to the time we meet each week.

If you just want to join our study group to gleam what you can, without wanting to earn any credits, you won’t need to do any outside studying, but you won’t get as much out of the course, and the later meetings will probably be more difficult to understand without spending additional time learning between meetings.  But the early meetings should be comprehensible and interesting for anyone.

Here is my current thoughts on materials and curriculum:

  • We start with Khan Academy’s excellent course Journey into Information Theory, which are well made, easy to understand videos.
  • We will start our reading with The Logician and the Engineer, which will help give some foundational understanding of the basic concepts of Information Theory, and learn about two of the primary inventors, in a manner that starts to ease our way into the math, and by taking a biographical stance, will help us connect with the people involved
  • We will then move into reading An Introduction to Information Theory: Symbols, Signals and Noise, which is considered a classic text that is easy to grasp, although mathematically “light”, it has more maths than the previous materials mentioned, and helps to develop an appreciation for mathematics even for those who are not yet mathematically inclined.
  • Last, we will dive into Information Theory, Inference and Learning Algorithms, which is the most expensive book, and most challenging.  But it is the primary text used in the Math 345 course that we are paralleling, so that this self-study can be worth college credit.  This book also will bridge into Bayesian statistics and Machine Learning, which will be the next planned self-study courses.

I’m sure will find other free online videos and maybe other online texts that will be used as well.  My goal is to have each session have one video that we watch, a discussion of our reading assignment that we had for the week, and then tackling one problem as a group.

If you are interested, send me an email at my EffectiveEducation.org address, or post a comment here.  I’ll also be contacting a few of my friends who I know are

Post Revisions:

Written by Jacob Walker

November 17th, 2015 at 11:59 am

Posted in Data Science

5 Responses to 'Calling all Fellow Math Nerds: Let’s Learn Information Theory Together'

Subscribe to comments with RSS or TrackBack to 'Calling all Fellow Math Nerds: Let’s Learn Information Theory Together'.

  1. The most expensive book can be viewed/downloaded (but not printed) as a pdf file from: http://www.inference.phy.cam.ac.uk/itprnn/book.pdf

    Also while Khan Academy’s course video does look nice and easy to understand (at least from the first video,) searching for other legitimate places to find the other ebooks lead me to the following videos: https://www.youtube.com/watch?v=BCiZc0n6COY

    (I just figured that extra resources can be good when learning at home.)

    Anyways, I am not exactly reliable at the moment (nor the fastest reader,) but depending on location/time, I may be interested in “sitting in” on the group.


    18 Nov 15 at 2:12 pm

  2. (P.S. I trust the original website for the main book was: http://www.inference.phy.cam.ac.uk/itila/book.html

    Since it talks on the page about the copyright law, I am willing to believe it is a legitimate source…

    The other books didn’t seem to have an equivalent legitimate source for etext downloaded without pay, but less the combined Kindle prices for the other 2 books costs about $23.)


    18 Nov 15 at 2:23 pm

  3. Yes, that source is legit. I was actually going to post about it also. And don’t worry, the initial parts of the self-study are going to start at a much more basic level, and we will try to build our way to the higher math.

    Jacob Walker

    18 Nov 15 at 2:37 pm

  4. […] I posted recently, there is a self-study group starting about learning Information Theory, that will ultimately […]

  5. […] I’ve been posting about, I’ve been working to learn more about information theory, to bridge my way to learning […]

Leave a Reply

%d bloggers like this: