Simple

## Paul Penfield, Jr.## Professor of Electrical Engineering
Cambridge, MA 02139-4307
(617) 253-2506 |

**Outline****Arcane****Pity****Perspective****Information****Course****Status**

**Surely one of science's most glorious accomplishments**

Also one of the most profound and mysterious

What is this thing called "entropy?"

Does the Second Law really tell us which way clocks run?

Deep related concepts

Complexity

Randomness

Fluctuations

Dissipation

Order

The public's imagination is captured

**Nobody really understands it**

At least that is what we believed as graduate students

It is too complex to be taught

**Because people think it is part of thermodynamics**

Thermodynamics really is hard to understand.

Many concepts -- heat, work, temperature, ...

You have to care about things like ideal gases.

Most engineers don't need to know thermodynamics.

Most don't care.

**Entropy and the Second Law are taught as part of
thermodynamics, so most people miss them.**

All scientists, all engineers, indeed all educated people need

to understand that some operations are reversible and

some are not.

They need to be able to tell them apart.

They need to know there is a way to quantify reversibility.

They need a "monotonic model"

To complement models of conserved quantities

Both are helpful for understanding the world

Both are prototypes for other quantities

It's just that thermodynamics is where they have traditionally

been taught.

**Thermodynamics always involves energy**

Entropy need not

**Outside of thermodynamics, without links to energy**

Entropy is less complex

There are plenty of reversible and irreversible operations

The Second Law, or something much like it, exists

Monotonic models can be taught more easily

**The more general the context, the simpler the concept**

This is the secret of making complexity simple

**And entropy is too important to be left to the physicists**

* George Clémenceau (1841 - 1929)

French Premier, 1906 - 1909, 1917 - 1920

**Start with information**

Entropy is one kind of information.

Entropy is information we do not have

**See reversible and irreversible data transformations**

In computation and communications

**Note that irreversible operations destroy information**

This is the Second Law in this context

**Apply to a physical system with energy**

Use maximum-entropy principle

Voila, thermodynamics!

Temperature is energy per bit of entropy (sort of)

Intensive vs. extensive variables

Second Law in traditional setting

Carnot efficiency

**The basic idea of reversibility is not difficult to understand**

**Why is this possible?**

Today's students are different

Best to start from the known

Data, disks, Internet, packets, bits, ...

Consistent with the coming information age

Go toward the unknown

Thermodynamics, equilibrium, heat engines, refrigerators

Relevant to the current industrial age

Physical view of information . . . like energy, information

can be of many types

can be converted from one form to another

can exist in one place or another

can be sent from here to there

can be stored for later use

There are interesting applications

Biology (genetic code)

Communications

Quantum computing

**A Freshman Course**

12 weeks:

1. Bits

2. Codes

3. Compression

4. Errors

5. Probability

6. Communications

7. Processes

8. Inference

9. Entropy

10. Physical systems

11. Temperature

12. Myths

**This course is NOT**

Introduction to Computing

Introductory Communications

Thermodynamics 101

**1. Bits**

Restoring logic

Digital abstraction

Signals and streams

Boolean algebra

**2. Codes**

Bytes

Fixed-length codes

ASCII

Genetic code

Binary code, gray code

Variable-length codes

Morse code

Telegraph codebooks

**3. Compression**

Helps with low channel capacity

Codebooks

Irreversible -- fidelity requirement

JPEG

MP3

Reversible

Run length encoding

LZW

The LZW patent issue

**4. Errors**

Physical sources of noise

Detection -- parity

Correction

Triple redundancy

Hamming code

**5. Probability**

Racing odds

Random sources

coins, dice, cards

Probabilities are subjective

Information can be quantified

**6. Communications**

Model with source, coder, channel, decoder, receiver

Huffman codes

Symmetric binary channel

Lossless

Noisy

Client-server model

TCP and IP

Strategies for recovery from lost packets

**7. Processes**

Discrete memoryless channel

Noise, loss

M = I_{IN} - L

= I_{OUT} - N

Cascade inequalities in L, N, and M

L_{1} <= L

L_{1} + L_{2} - N_{1} <= L
<= L_{1} + L_{2}

**8. Inference**

Given received signal, what is input

How much information have we learned?

**Discrete Memoryless Channel**

**9. Entropy**

Entropy is information we do not have

Input probabilities consistent with constraints

Minimum assumptions, maximum entropy

Lagrange multipliers

Deer hunters

Fishermen

**10. Physical systems**

Energy per state

Expected value of energy

Boltzmann distribution

Lagrange multipliers are intensive variables

Equilibrium

**11. Temperature**

One of the Lagrange multipliers is temperature

Heat, work

Carnot efficiency

**12. Myths**

Order out of chaos

Miracle needed

Heat death

Evaporation of black holes

Difficulty of extensions to social science

**We are finding out**

Fall 1999, course development

Faculty: Paul Penfield, Seth Lloyd, Sherra Kerns

Students: small set of freshmen serving as guinea pigs

Spring 2000, pilot offering

Limited to 50 freshmen

Of course we have a Web site -- everybody does

http://www-mtl.mit.edu/users/penfield/6.095-s00/

Fall 2000, revisions, note writing

Spring 2001, first full offering

**The devil is in the details**

E.g., which is the best statistical mechanics model to use?

**Entropy is the information we don't have**

Therefore entropy is subjective (some people don't like that)

**Math generally simple -- discrete, not continuous processes**

But Lagrange multipliers are not easy

**Skill in modeling is still important**

No magic here

**At the freshman level you cannot go very deeply**

All we can do is provide simple ideas to be built on later

We want to be consistent with later learning in specific areas

**Stay in touch -- we will let you know how it turns out**

**If we are successful**

The course will be permanent

We will help other universities start similar courses

We will advocate it as a science exposure in the liberal arts

URL of this page:

Created: Nov 19, 1999 | Modified: Dec 29, 1999

Related page: Penfield publication list

Site map | To Paul Penfield's home page | Your comments are welcome.