"Anything you can do, I can do meta."
--Charles Simonyi

I have created a place with an absence of information and offer YOU, dear reader, the ability to insert information.

Try it for yourself. I'll be here to explain what you just did when you get back.

So you probably started by watching a sine wave slowly draw itself across your web browser. Sorry it took so long. Once it draws out one full period it should move quite a bit faster. I could probably use a trick or two to get rid of that initial slowdown, but I'm lazy. That sine wave is handily labeled "Source".

Below it you might notice a graph that says "Information from Source". That graph lines up vertically with Source to show you how many bits of information each value shown in source gives you upon observation. If you haven't inserted any information yet, the graph will be all the way at the bottom, at 0 bits. Below it you can see Entropy, which tells you how many bits of information, on average, you are getting from each value you are currently observing.

If you click the Insert Information button, you will see Source get a little more random. Information from source will go up and fluctuate with the randomness of the Source. Entropy will also increase. You can click again to add more and more information.

So why does inserting information cause Source to become more random? The information being measured is Shannon Information. In the late 40s and early 50s, Claude Shannon introduced Information Theory. One of the problems he worked on was an efficient way to code messages. An optimal encoding takes advantage of the naturally occurring asymmetries in the frequencies and patterns in the messages you want to send. You can give more typical messages smaller codewords and more atypical messages bigger codewords. If you have a probability distribution P over your message space, then the optimal length for any message m is at least -log(P(m)).

The units of information depend on the base of the logarithm. A base 10 log measures information in digits. Base 2 measures information in bits, and base e (the natural logarithm) measures information in a purely abstract unit called nats.

Since Shannon's introduction multiple actual coding strategies are able to get arbitrarily close to that theoretical lower bound, even when that lower bound is not a whole number, by putting multiple messages together and encoding them as one. Perhaps the most clever is called Arithmetic encoding.

In the above application, the message is the next value from Source. Before inserting information, the next value is perfectly predictable: it is whatever comes next in the sine wave. When an outcome is deterministically predictable, it takes 0 bits to describe. Its probability is 1, and log(1) = 0. You get zero bits of information by observing a perfectly predicted outcome.

The entropy H of a distribution X is the expected information per draw.

This quantity was named entropy for it's similarity to thermodynamic entropy.

So as you insert information, the typical value of the Information from Source graph increases steadily and the entropy goes up. It starts at the zero entropy, perfectly predictable distribution where at each time you get the next piece of the sine wave. Adding information makes the next piece distributed according to a Poisson(0.5) distribution, then Poisson(2), Poisson(5), Poisson(10), and Poisson(30). Finally, it goes to a uniform distribution, where every value provides a constant 7.7something bits of information (you can use that number to calculate how many pieces I cut my sine wave into).