Seth Lloyd and Paul Penfield, Jr., Hot Bits: Seeing the World in Terms of Information and Entropy, NEEDHA Annual Meeting, Destin, FL; March 8, 1999.

Hot Bits:
Seeing the World in Terms of
Information and Entropy

MIT logo  . .

Seth Lloyd

Paul Penfield, Jr.

Massachusetts Institute of Technology
Cambridge, MA 02139-4307

(617) 252-1803(617) 253-2506
slloyd@mit.edupenfield@mit.edu

Hot Bits

Outline
You're going to teach freshmen WHAT????
What is this thing called Energy?
What is this thing called Information?
What is this thing called Entropy?
Can this be taught?
Our plans
What will the students end up with?

You're going to teach freshmen WHAT????

The Second Law of Thermodynamics is one of science's priceless intellectual achievements!
 . . It is grounded in, and informs, fundamental issues:
 . .  . . . . . time reversibility, the nature of measurement, . . .
 . . It is universal.
 . . There is a non-conserved, monotonically increasing, quantity.
But as normally presented, entropy is difficult.
 . . It is usually considered part of thermodynamics.
 . . EE and CS students are not motivated to study it.
 . . The Second Law, as usually stated, is only approximate.
We will cover information, of which entropy is part.
 . . This matches today's student backgrounds and interests.
 . . This leads to universal and exact form of the Second Law.
 . . It serves the information age.
 . . It gives students a useful model to help interpret the world.


Reminder: What is Energy?

It is found in different forms.
 . . Kinetic
 . . Potential
 . . Electric
 . . Magnetic
 . . Chemical
 . . . . .
It is conserved.
 . . This property is so important that when a leak is found, we
 . .    invent a new form just to preserve it.
 . .  . . . . . heat, mass . . .
Energy can be moved in space and time.
You can design energy converters.


What is Information?

It has a quantitative theory.
 . . The basic unit is the bit.
 . . Arrays of bits can represent complex objects.
It is found in different forms.
 . . Information we have
 . . Information we don't have
Information we have is inherently subjective.
Information we don't have is also inherently subjective.
However, sometimes total information is not subjective.
Information is not conserved.
 . . We lose track of things.
 . . Noise is the enemy.
Information can be moved in space and time.
You can design information converters.


What is Entropy?

Entropy is information we don't have.
Entropy is therefore subjective.
Traditionally, entropy is a property of physical systems.
 . . Avogadro's number = 6 x 10^23
 . .  . . With that many atoms there is lots of information you don't have.
 . .  . . So entropy is not very subjective (knowledge is only a few bits).
But things are changing.
 . . Today we can manipulate bits with only a few atoms.
 . . Semiconductor devices are pushing the quantum limits.
Note: Entropy is not the monotonic quantity in the Second Law of Thermodynamics.
 . . Information is.
 . . Entropy is only the information we don't have.
You can design information converters.
 . . . . . quantum computers, Maxwell's Demon, measurements,
 . .    even steam engines . . .


Challenge: Need Unified View of Information

Useful for
 . . . . . Computing, Communications, Physical Systems . . .
Key notion is partitioning the universe.
 . . Space
 . . Observer / observed
 . . Visibility filter
 . . Known / unknown
 . . Abstraction level
 . . Type
 . .  . . . . . electrical, kinetic, chemical . . .
Difficult issues need to be addressed.
 . . Information may be correlated across the partition.
 . . Things may be irreversible.
Analogy to Energy


Can Freshmen Really Learn This Stuff?

We believe so!
 . . There is no need for calculus.
 . . Of course only limited depth is possible at the freshman level.
We will rely on student background and interest in information.
 . . Today all students are computer literate.
 . .  . . This is, after all, the information age.
Information is now one of engineering's foundational sciences:
 . . . . . Physics, Chemistry, Biology, Information . . .
 . .  . . Mathematics is also needed, to provide notation, concepts, and tools for many fields.

Planned New Course

Topics
 . . The digital abstraction and the bit
 . . Representation of objects by fixed-length codes
 . .  . . . . . ASCII, genetic code . . .
 . . Applications to computing
 . .  . . . . . Compression, LZ, GIF (reversible), JPEG (irreversible) . . .
 . . Probability and variable-length codes
 . .  . . Maximum entropy technique for calculating probabilities
 . .  . . . . . Morse code, Huffman codes . . .
 . . Applications to communications
 . .  . . Noise and irreversibility
 . .  . . Shannon theorems
 . . Applications to physical systems
 . .  . . Boltzmann probability distribution
 . .  . . Second Law of Thermodynamics


Planned Course (cont)

Timetable
 . . Informal seminar, Fall 1998
 . . Serious course development, Fall 1999
 . . Pilot offering, Spring 2000
 . . Note writing, serious revisions, Fall 2000
 . . First normal offering, Spring 2001
Dramatis Personae
 . . Seth Lloyd, Department of Mechanical Engineering, MIT
 . .  . . His research interests include quantum computing.
 . .  . . He currently offers a similar senior-level course.
 . . Paul Penfield, Jr., Department of Electrical Engineering and Computer Science, MIT
 . .  . . He has a background in thermodynamics, noise, & computing.
 . .  . . No longer Department Head, he can do fun stuff now.


Planned Course (cont)

Venue
 . . First, MIT
 . . Next, other engineering schools
 . . If successful, general universities
Where does it fit at MIT?
 . . Half-course slot is available in spring of freshman year.
 . . The course cannot be required.
 . . It will compete with other offerings.
 . . It cannot be a prerequisite for anything else.
 . .  . . But it will be consistent with approaches taken in later courses.


Will Our Students Be Better Educated?

They will have a new model with which to interpret the world.
 . . A quantity that is monotonic (as opposed to constant)
They will see information everywhere they look.
Who needs it? Just about everybody!
 . . Some engineers will use the concepts routinely.
 . .  . . Chemical engineers, specialists in communications or information processing
 . . Others will use the concepts mainly for motivation.
 . . Some science majors will need it.
 . . Liberal arts majors need the broadening exposure.
 . .  . . This may be an excellent way to satisfy a science requirement.
Information may become the most important science in this information age.


URL of this page: http://www-mtl.mit.edu/users/penfield/pubs/needha-99.html
Created: Mar 4, 1999  |  Modified: Mar 9, 1999
Related page: Penfield publication list
Site map  |  To Paul Penfield's home page  |  Your comments are welcome.