Information-Theory on CS Theorems

The No-Cloning Theorem

Fri, 02 Jul 2027 12:00:00 +0100

Symptom

You have spent this whole part learning that information wants to be copied. Entropy bounds compression (T018), Huffman hits the bound (T020), redundancy buys error correction (T021). Every result so far treats “make another copy” as the free operation — the thing you do without thinking, the reason cp has no interesting failure modes.

The Nyquist–Shannon Sampling Theorem

Fri, 25 Jun 2027 12:00:00 +0100

Symptom

Your dashboard shows CPU utilization averaging 40%, comfortably under budget. Users are reporting timeouts. You add more logging and the average stays at 40%.

Eventually someone captures a one-second trace and finds the truth: the service spikes to 100% for eight seconds out of every ninety, and your monitoring samples every sixty seconds. The spikes are real, periodic, and invisible.

Streaming Lower Bounds and Sketching

Fri, 30 Apr 2027 12:00:00 +0100

Symptom

Product wants the daily unique-visitor count. You have a firehose of events.

The obvious implementation is a set. Add each visitor ID, report the size. At a billion distinct IDs, eight bytes each, that is 8 GB before any hash table overhead, and in practice a HashSet will cost you two to three times that. Per day. Per dimension you want to slice by. Multiply by country, by platform, by campaign, and the memory bill is absurd for a number nobody looks at past two significant figures.

Minimum Description Length and Occam's Razor

Fri, 16 Apr 2027 12:00:00 +0100

Symptom

You fit a model. It scores 94% on training data and 71% on held-out data.

So you simplify: fewer parameters, more regularization. Training drops to 88%, held-out rises to 84%. You simplify further and both drop. Somewhere in there was an optimum, and you found it by trial and error, with a validation set and patience.

Kolmogorov Complexity

Fri, 09 Apr 2027 12:00:00 +0100

Symptom

You have two files, each exactly one megabyte.

The first is a megabyte of the byte 0x00. The second is a megabyte from /dev/urandom. Gzip the first and you get a few hundred bytes. Gzip the second and you get slightly more than a megabyte, because the header costs something and there is nothing to exploit.

The Hamming Bound and Error-Correcting Codes

Fri, 05 Mar 2027 12:00:00 +0100

Symptom

You are specifying ECC memory for a fleet. The vendor quotes SECDED: single error correct, double error detect, at 8 check bits per 64 data bits. That is 12.5% overhead, and someone in the room asks the obvious question — why not correct two errors? Or three? What does it cost?

The Noisy-Channel Coding Theorem

Fri, 26 Feb 2027 12:00:00 +0100

Symptom

Your link drops 1% of bits. You need reliable delivery, so you triple every bit and take a majority vote. Now an error needs two of three bits to flip, which happens with probability about $3 \times 10^{-4}$ — better, but you paid 3x in bandwidth and the error rate is still not zero.

Huffman Coding Is Optimal

Fri, 19 Feb 2027 12:00:00 +0100

Symptom

You are compressing a log file. The symbols are wildly skewed: INFO is 90% of the lines, WARN is 9%, ERROR and FATAL split the rest. Fixed-width two-bit codes give you exactly 2 bits per symbol, and entropy (T018) says the floor is about 0.53 bits. There is a factor of four sitting there.

Shannon Entropy and the Source Coding Theorem

Fri, 13 Nov 2026 12:00:00 +0100

Symptom

You gzip a log file and it drops to 8% of its original size. You gzip the already-gzipped file and it gets slightly bigger. Somebody asks why, and the honest answer you have is “because it’s already compressed,” which is a restatement, not a reason.

No Universal Lossless Compressor

Fri, 04 Sep 2026 12:00:00 +0200

Symptom

Someone sends you a pitch deck. The claim is a compression algorithm that reduces any file by a guaranteed ratio, and — this is always the tell — that it can be applied repeatedly. Four gigabytes down to one, then one down to two fifty, and so on until the whole film is a few hundred bytes and a clever decoder.