Department of Electrical Engineering and Computer Science  Department of Mechanical Engineering  6.050J / 2.110J     Information and Entropy     Spring 2004

Unit 7: Processes

Schedule

Lecture Handouts

Students who for any reason did not receive these items can pick up a copy in Room 38-344. Most of this material is also available on the 6.050J/2.110J Web site http://mtlsites.mit.edu/Courses/6.050.

• Unit 7 Resources (this page)
• Problem Set 7
• 6.050J/2.110J Notes
  • Processes
• A summary in concise mathematics (mostly algebra) of many topics in information theory, coding, communications, by Dr. M. Z. Wang, Hong Kong, October 1996:
  http://www.eie.polyu.edu.hk/~enmzwang/adc/l-notes/node4.html. There is a section on Discrete Memoryless Channels.
• Robert G. Gallager, "Claude E. Shannon: A Retrospective on His Life, Work, and Impact," IEEE Transactions on Information Theory, vol. 47, no. 7, pp. 2681-2695; November, 2001.
• Professor Wyatt's Recitation Notes regarding Encoding

Reading Assignment

• Processes

Resources

Technical

• Claude E. Shannon, "A Mathematical Theory of Communication," Bell System Technical Journal, vol. 27, pp. 379-423 (Part I), 623-656 (Part II); July and October, 1948. These seminal papers are available in several forms (see bibliographic notes)
  • PDF version of original papers, with corrections but without Shannon's 1949 modifications
  • Claude E. Shannon and Warren Weaver, "The Mathematical Theory of Communication," University of Illinois Press, Urbana, IL; 1949, with later editions 1963 and 1998 (incorporating a number of modifications and corrections by Shannon)
  • Claude E. Shannon, "A Mathematical Theory of Communication," 50th Anniversary Edition, printed for the 1998 IEEE International Symposium on Information Theory, MIT, Cambridge, MA; August 16-21, 1998 (based on 1949 book, with corrections)
  • Reprinted in D. Slepian, editor, Key Papers in the Development of Information Theory, IEEE Press, New York; 1974
  • Reprinted in N. J. A. Sloane and A. D. Wyner, editors, Claude Elwood Shannon: Collected Papers, IEEE Press, New York; 1993
  Because he treated a general case, Shannon was able to distinguish loss and noise, in the way done in 6.050J/2.110J. Many others treat only channels with similar inputs and outputs in which case often L = N so it may not be obvious why they should be different concepts.

Historical

• Claude E. Shannon (1916 - 2001)
      obituary, Tech Talk, MIT
• Aaron D. Wyner, The Significance of Shannon's Work
• Robert G. Gallager, "Claude E. Shannon: A Retrospective on His Life, Work, and Impact," IEEE Transactions on Information Theory, vol. 47, no. 7, pp. 2681-2695; November, 2001.
• The process model is also known as Markov Processes or Markov Chains after Andrei A. Markov (1856 - 1922).

General Technical Books

There are many excellent texts on communications, most of which assume a familiarity with mathematics beyond introductory calculus. Some have treatments of the discrete memoryless channel, but not with the same emphasis given here. Here are a few:

• Robert G. Gallager, "Information Theory and Reliable Communications," John Wiley & Sons, Inc., New York, NY; 1968. One of the early textbooks, designed for first-year graduate students
• Allan R. Hambley, "An Introduction to Communication Systems," Computer Science Press; 1990
• Thomas M. Cover and Joy A. Thomas, "Elements of Information Theory," John Wiley & Sons, Inc., New York, NY; 1991. Aimed at university seniors and first-year graduate students. One of several excellent books of that era
• Simon Haykin, "Communication Systems," 4th edition, John Wiley and Sons, Inc.; 2001.

Help Wanted

6.050J/2.110J students: be the first to suggest a resource, for example a useful Web site or a good book or article, to add to the list above. If your suggestion is accepted by the 6.050J/2.110J staff, you will get a $5 ice-cream gift certificate. Send your suggestion by e-mail during Spring 2004 to 6.050-staff (at) mit.edu.