Symptom A node in your cluster is not down. It is worse than down. Its disk is returning corrupted blocks that pass the checksum because the checksum is corrupted too. Or a NIC with a firmware bug is duplicating and reordering frames. Or a bad memory module flipped a bit in a config value and the node now believes it owns a shard it does not. Or someone has root on one machine. ...
Symptom Your cluster of five nodes elects a leader. It works. It has worked for a year. Then one afternoon a garbage collection pause on the leader runs for twelve seconds. The followers time out, start an election, and elect a new leader. The old leader wakes up, has no idea it was deposed, and keeps serving writes. For a few hundred milliseconds you have two leaders, and if your fencing is not airtight, two writes that should have been ordered land in an order nobody chose. ...
Symptom You took the lesson from T058 and stopped trusting wall clocks. Every event now carries a Lamport timestamp, and your writes are ordered by it. The vanishing updates stopped. ...
Symptom Two writes to the same key. Which one wins? Your first instinct is timestamps: whichever has the later wall-clock time is newer. This works in testing. In production you get a bug report saying an update vanished, and when you dig in, node B’s clock was 40 ms behind node A’s, so B’s later write carried an earlier timestamp and lost. NTP is running. NTP does not make clocks identical, it makes them close, and “close” is not “ordered.” ...
Symptom You have 100 backends and a load balancer hashing request IDs to pick one. Expected load per backend is exactly 1%, and you have checked the hash is good. ...
Symptom You call the payment API. The request goes out. Nothing comes back. Did the charge happen? You genuinely do not know. Retrying might double-charge the customer. Not retrying might drop the payment. There is no third option available, and no amount of care in your client library creates one. ...
Symptom Two measurements that both look like they break the previous post. The first: your Spark job takes four hours on ten nodes. You move it to a hundred nodes and it still takes four hours — but it is now processing ten times the data. T081 said the ceiling was 20×, and nobody hit a ceiling. ...