Distributed-Systems on CS Theorems

Consistent Hashing

Fri, 29 Oct 2027 12:00:00 +0100

Symptom

You shard a cache across ten servers with the obvious rule:

server = servers[hash(key) % len(servers)]

It works. Then you add an eleventh server, and % 10 becomes % 11.

Every key now maps somewhere else. Not one tenth of them — essentially all of them. A key landing on the same server after the change is a coincidence with probability about $1/11$, so roughly 91% of your cache is instantly invalid. Every request misses, every miss hits the database, and the database, sized for a 95% cache hit rate, now receives twenty times its provisioned load and falls over.

The CALM Theorem and CRDTs

Fri, 03 Sep 2027 12:00:00 +0100

Symptom

Every distributed result so far has been a prohibition. Two Generals: no agreement over a lossy channel. FLP: no deterministic asynchronous consensus. Byzantine: not below $3f+1$. CAP: not all three.

The CAP Theorem and PACELC

Fri, 27 Aug 2027 12:00:00 +0100

Symptom

Someone in a design review says “we’re AP, not CP,” and the room nods.

Nobody asks which consistency, which availability, or what happens when there is no partition, which is 99.9% of the time. The phrase has done the work of a decision without containing one.

Paxos, Raft, and the Shape of Consensus

Fri, 20 Aug 2027 12:00:00 +0100

Symptom

Your database has two nodes that both believe they are the primary.

Each has been accepting writes. Each has a coherent log. Neither has crashed. The old primary was declared dead by a health check that timed out during a GC pause, a new one was elected, and then the old one woke up still holding what it thinks is a valid lease. You now have two divergent histories of the same data and no principled way to merge them, because both contain acknowledged writes that customers were told had succeeded.

Quorum Intersection (R + W > N)

Fri, 13 Aug 2027 12:00:00 +0100

Symptom

You have three replicas of a value. A client writes v2 and gets an acknowledgement. Another client reads and gets v1.

Nothing crashed. No network partition healed badly. Both operations completed successfully and reported success. The write went to one replica, the read came from another, and the two never spoke.

The Byzantine Generals Problem

Fri, 18 Jun 2027 12:00:00 +0100

Symptom

A node in your cluster is not down. It is worse than down.

Its disk is returning corrupted blocks that pass the checksum because the checksum is corrupted too. Or a NIC with a firmware bug is duplicating and reordering frames. Or a bad memory module flipped a bit in a config value and the node now believes it owns a shard it does not. Or someone has root on one machine.

FLP Impossibility

Fri, 11 Jun 2027 12:00:00 +0100

Symptom

Your cluster of five nodes elects a leader. It works. It has worked for a year.

Then one afternoon a garbage collection pause on the leader runs for twelve seconds. The followers time out, start an election, and elect a new leader. The old leader wakes up, has no idea it was deposed, and keeps serving writes. For a few hundred milliseconds you have two leaders, and if your fencing is not airtight, two writes that should have been ordered land in an order nobody chose.

Vector Clocks and Causal Consistency

Fri, 04 Jun 2027 12:00:00 +0100

Symptom

You took the lesson from T058 and stopped trusting wall clocks. Every event now carries a Lamport timestamp, and your writes are ordered by it. The vanishing updates stopped.

Happens-Before and Lamport Clocks

Fri, 28 May 2027 12:00:00 +0100

Symptom

Two writes to the same key. Which one wins?

Your first instinct is timestamps: whichever has the later wall-clock time is newer. This works in testing. In production you get a bug report saying an update vanished, and when you dig in, node B’s clock was 40 ms behind node A’s, so B’s later write carried an earlier timestamp and lost. NTP is running. NTP does not make clocks identical, it makes them close, and “close” is not “ordered.”

Balls Into Bins and the Power of Two Choices

Fri, 12 Feb 2027 12:00:00 +0100

Symptom

You have 100 backends and a load balancer hashing request IDs to pick one. Expected load per backend is exactly 1%, and you have checked the hash is good.

The Two Generals Problem

Fri, 30 Oct 2026 12:00:00 +0100

Symptom

You call the payment API. The request goes out. Nothing comes back.

Did the charge happen? You genuinely do not know. Retrying might double-charge the customer. Not retrying might drop the payment. There is no third option available, and no amount of care in your client library creates one.

Gustafson's Law and the Universal Scalability Law

Fri, 25 Sep 2026 12:00:00 +0200

Symptom

Two measurements that both look like they break the previous post.

The first: your Spark job takes four hours on ten nodes. You move it to a hundred nodes and it still takes four hours — but it is now processing ten times the data. T081 said the ceiling was 20×, and nobody hit a ceiling.