Information and Entropy, by Paul Penfield, Jr. ECEDHA 2008 Annual Meeting, Coronado, CA, March 18, 2008 SLIDE 1 (blank) Hello. I'm pleased to be here today. It's great to be back at ECEDHA (which I knew back before it was globalized, when it was still called NEEDHA). SLIDE 2 (title) The theme of the session is "Curricula to Encourage Innovation." I will tell you about a new course, which a colleague and I have been developing at MIT, which is itself innovative, which may have a role in innovative curricula, and which may be useful for a department that wants to add innovative new elements to its mission. SLIDE 3 (Information and Entropy) This course addresses the fact that some of our students are never exposed to what is perhaps the most profound and mysterious achievement in all of science, namely the Second Law of Thermodynamics. This means they miss out of some pretty important things: the concept of irreversibility, and a quantity (entropy) that is not conserved but instead "monotonic", steadily increasing. Their education is incomplete. They are not well rounded scientists, they cannot appreciate other disciplines, and they won't be able to keep up with future trends in EECS. We want to fix this. My colleague, Seth Lloyd, is in the Department of Mechanical Engineering so our course is joint. We talked about it nine years ago at this very same meeting. Today I'll give you a progress report. SLIDE 4 (Underlying Ideas) The essential concept behind the entire course is the fact that entropy (that mysterious quantity from thermodynamics) and information (the quantitative measure used in communications theory, defined by Shannon in 1948) are interchangeable. That is, they are the same concept. It's just like energy and mass, which were considered separate concepts before the famous E = M C squared formula of Einstein. I say here entropy is arcane. Why? As usually presented, the Second Law is part of thermodynamics, which deals with such things as ideal gasses, steam engines, and stuff we EECS people don't care much about. But it is really much broader, with applications in communications and computing. We decided to recast entropy in this course as a form of information, thereby making it accessible to lots more people. Including freshmen. And including those not majoring in science or engineering. And particularly including those who don't care much about thermodynamics. So how can we possible teach this to freshmen? SLIDE 5 (Style) It's easy. We keep it simple. We don't go into the depth that normal courses do. Yet this is not a survey course. It gets to fundamental issues of what information is, how it is defined and measured, and how it affects efficiency of communication systems and computation. And when we look at information in physical systems, we get to the heart of the concept of reversibility, and then to this thing called temperature. Any finally, why entropy never decreases. Another thing that works in our favor is the use of EECS input/output diagrams. Another is sticking to discrete rather than continuous systems; so we don't need calculus. The students discover that information, like energy, is a physical quantity that can be moved from place to place, stored for future use, and converted from one form to another. They see that information has dimensions just like length and time. The commonly used units of information are bits and joules/kelvin. We made a decision (not yet fully implemented) to keep the entire course consistent with quantum mechanics. What does this mean? As you probably realize, all of the models EECS deals with need to be thought through and made consistent with quantum theory. Yes, this even applies to the bit. The classical bit and the qubit are different things. One is an approximation of the other. They have different properties and different fundamental limits. SLIDE 6 (What's in the Course?) So what do we actually teach? The basic idea is to go from what students know (information can be quantified) to what they do not (entropy). So our first applications are to computation and communications. Nothing deep -- we state Shannon's various theorems in a digital context (not continuous) but do not prove them all. We do carefully define entropy in terms of probability distributions and give them Kraft's Inequality and the Gibbs Inequality. We also show lots of neat examples that grab their attention, including LZW compression, Huffman coding, and Hamming codes. We develop and present information-flow diagrams which portray loss and noise in a communications system, and define carefully the mutual information. Then we transition into physical systems using the Principle of Maximum Entropy. We use extremely simply models. One such model is a linear array of magnetic dipoles. Believe it or not, this is really all that is needed to understand diffusion, heat, work, temperature, and the Carnot efficiency. Understand at an introductory level, that is. Not in enough detail to do anything practical. SLIDE 7 (This Course is Not . . .) You may be thinking to yourself, how can they possibly cover computation, information theory, communication systems, digital logic, statistical mechanics, and thermodynamics, all for freshmen? Well of course we don't. But we DO lay a foundation on which any of these can be taught in detail later. Our students will then see the ideas for the second time, and recognize the universal nature of information. By keeping it simple we think this material is accessible to students in programs who are not majoring in science or engineering. SLIDE 8 (Trend: Expanding EECS Scope) Let's look at what we are doing in a historical context. As EE and CS have evolved in the past century, we have to deal with more and more technologies, and know more and more scientific disciplines. This is surely not a surprise to anyone here. Information science may be one of the most important fundamental sciences in this, the information age. Our new course is aligned with these trends. We bring together classical physics with other disciplines more recently recognized as relevant, including information science, a bit of modern biology, and especially quantum physics. What we are doing is right up to date. SLIDE 9 (Trend: What our Graduates Do) But will the students benefit from what they learn? Here is my take on what society expects engineers to do. A century ago we educated people to practice electrical engineering. In the decades since, expectations have risen. Now some of our graduates go into engineering science research, or the practice of computer science or engineering, or other professions such as medicine, law, or management. As the scope of technical material relevant to our graduates has increased, they are increasingly going for master's degrees (because we cannot cram everything into four years) so the M.S. is now the de facto first professional degree for EECS. That is where we stand today. In the future, I believe that we should also prepare some students with the skills needed for societal leadership, and that means B.A. degree programs, which are not vocational, but are designed to prepare people for citizenship. Our new course serves all these areas of activity. SLIDE 10 (Why is This a Good Idea Now?) You may be wondering why we should be doing this now. Three reasons. First, the students are different. They show up knowing very well the size of their hard disks and data rates for information transfer. In other words, they are used to thinking of data quantitatively. Data and information are not quite the same, but it is not hard to go to information content by thinking about redundancy and compression (that is where the units on compression, errors, and probability come in). Second, the technologies they will need are different now. Quantum engineering is at hand. Note that there are two big ideas here -- on one hand, Moore's law is letting us control bits with fewer and fewer atoms. Pretty soon quantum fluctuations will be important in data systems, and analysis cannot be done without dealing with both informational entropy and physical entropy. In other words, memories and logic will be acting as entropy converters. And also, quantum systems can do things, such as factoring numbers, that classical systems cannot. The underlying models are different and therefore so are the fundamental limits. If our graduates are to avoid premature obsolescence they need some quantum models. Third, our universities should all be thinking about what we can do to help society adapt to the torrent of new technology. That means helping educate people who will be tomorrow's leaders of society (not just leaders of technology). Our nation (and others) draw on B.A. programs for national leaders, so anything we can do to improve scientific content in those programs is useful. A good example is the debate about global climate change. Ideally, all citizens should understand, so some degree, the relevant science and technology. More realistically, our national leaders should. Regrettably, that is not the case. This course, together with a similar one about a conserved quantity such as energy, would be a wonderful science component of a B.A. program. SLIDE 11 (Results: Good News and Bad News) So let me report now how things have been going. As you might expect, there is some good news and some bad news. Our students love the course, and that is partly because they self select. This course does not fulfill any degree requirement, so they are there because they want to be, not because they have to be. Our educational materials are on the Web, and you are welcome to use them in any way you wish. However, the notes are not complete yet, and in particular do not yet fulfill our intention of keeping quantum consistency all semester long. After introducing qubits in the first lecture, and the classical bit as an approximation, we proceed to do most of the communication material only classically. In fact, we do not even have a good way to construct codes for symbol sets in qubits. And some of the material on physical systems is still a little rough. And now the bad news. We carefully constructed the course to fit into a niche in freshman schedules, and attracted a good number of students at first. Then other departments noticed that niche, and now there are a lot of courses for students to choose from. The enrollment in recent years has been lower. You cannot sustain a course for only ten students. What the future holds in store is unclear. I have officially retired, so I can do what I want, but my colleague has been doing his teaching in part as an overload. That can't go on forever. We need to do a better job of convincing both of our departments that the course, though not part of either degree program, is important. So far we have not attempted to identify any B.A. program where we could try this out. MIT is not the place to do this. We hope that when the notes are done, that will be the time. SLIDE 12 (Why Should You Care?) However, if you are so inclined, we would be happy to work with you or some of your faculty. If you want your students to know about irreversibility and the Second Law, this is an efficient way to do it. If you want your graduates to avoid obsolescence when quantum engineering arrives, this is one way to do it. And if you want your department to contribute to the education of tomorrow's society leaders, either through a B.A. program of your own or via collaboration within your university, this course could be an excellent part of it. We would be happy to do what we can to help you. SLIDE 13 (Thank you) It has been a pleasure for me to be here with you. Thank you for your attention.