[{"content":"Symptom You need to prove a language is not regular. Everyone points you at the pumping lemma, and you spend an afternoon losing to it.\nThe statement is a nest of quantifiers: for every regular language there exists a pumping length $p$ such that for every string $w$ with $|w| \\ge p$ there exists a decomposition $w = xyz$ with $|xy| \\le p$ and $|y| \u0026gt; 0$ such that for all $i \\ge 0$, $xy^i z$ is in the language. To use it you negate all of that and play a game against an adversary who picks $p$ and the decomposition while you pick $w$ and $i$.\nWhen it works, it works. But it fails in two ways that waste real time. First, picking the wrong $w$ leaves you stuck with no indication that the string was the problem. Second, and worse, the pumping lemma is not an iff. There are non-regular languages that pump. So a failed attempt tells you nothing at all, and you cannot use it to prove a language is regular.\nMeanwhile a different question is nagging: your regex library minimized a DFA from 40 states to 12, and you want to know whether 12 is really the floor or whether a smarter algorithm finds 11.\nBoth questions have the same answer, and it is not the pumping lemma.\nStatement For a language $L$ over $\\Sigma$, define strings $x$ and $y$ to be equivalent when no suffix distinguishes them:\n$$x \\equiv_L y \\iff \\forall z \\in \\Sigma^* : (xz \\in L \\iff yz \\in L).$$This is the Nerode right congruence. Two prefixes are equivalent when the future cannot tell them apart.\n$L$ is regular if and only if $\\equiv_L$ has finitely many equivalence classes. Moreover, the number of classes is exactly the number of states in the unique minimal DFA for $L$.\n$L$ is regular iff the index of $\\equiv_L$ is finite, and the minimal DFA has state set $\\Sigma^*/{\\equiv_L}$ with $\\delta([x], a) = [xa]$, which is well-defined precisely because $\\equiv_L$ is a right congruence. This DFA is unique up to isomorphism.\nThree results in one. A characterization (iff, unlike the pumping lemma), an exact state count, and a uniqueness claim — the minimal DFA is not one minimal DFA among several but the minimal DFA, canonical up to renaming.\nArgument Regular implies finite index. Let $M$ be a DFA for $L$ with states $Q$. If $\\delta(q_0, x) = \\delta(q_0, y)$, then $x$ and $y$ drive the machine to the same state, and everything afterwards is identical, so $xz$ and $yz$ are accepted together. Hence $x \\equiv_L y$. So the map from states to classes is onto, and the index is at most $|Q|$, which is finite.\nNotice this already proves the lower bound: any DFA for $L$ has at least as many states as $\\equiv_L$ has classes.\nFinite index implies regular. Build a DFA whose states are the classes. Start state $[\\varepsilon]$; transition $\\delta([x], a) = [xa]$; accept $[x]$ when $x \\in L$.\nBoth need checking. The transition is well-defined: if $x \\equiv_L y$ then $xa \\equiv_L ya$, since any $z$ distinguishing $xa$ from $ya$ means $az$ distinguishes $x$ from $y$. The accept set is well-defined: taking $z = \\varepsilon$ in the definition shows equivalent strings agree on membership. Finite index makes it a finite automaton, and by construction it accepts exactly $L$.\nUniqueness. Any DFA for $L$ with the minimum number of states must have its states in bijection with the classes, and the transitions are then forced by the construction. So minimal DFAs are isomorphic. Contrast minimal NFAs, which are neither unique nor efficiently computable — NFA minimization is PSPACE-complete. Determinism is what buys canonicity.\nUsing it: $L = {a^n b^n}$. Consider $a^i$ and $a^j$ for $i \\ne j$. The suffix $b^i$ distinguishes them: $a^i b^i \\in L$ but $a^j b^i \\notin L$. So every $a^i$ is in its own class, giving infinitely many, so $L$ is not regular.\nThat is the whole proof. No quantifier alternation, no adversary, no choosing a clever $w$. Exhibit an infinite family of pairwise distinguishable prefixes and you are done, and the family is usually the obvious one.\nWhy it subsumes the pumping lemma. The pumping lemma is a consequence of finite index: with $k$ states, any string of length $\\ge k$ revisits a state, and the loop is the pumpable part. But it is a one-way consequence, which is why it cannot certify regularity. Myhill-Nerode is the underlying fact, and it goes both ways. Anything the pumping lemma proves, Myhill-Nerode proves, usually in fewer lines. The reverse fails — ${a^i b^j c^k : i = 1 \\implies j = k}$ pumps but is not regular, and Myhill-Nerode disposes of it directly.\nWhere minimization algorithms come from. Hopcroft\u0026rsquo;s $O(n \\log n)$ algorithm computes $\\equiv_L$ by partition refinement: start by splitting accepting from non-accepting states, then repeatedly split any block whose members transition into different blocks. It terminates at the Nerode classes, and the theorem is why the answer is canonical rather than dependent on the order of refinement.\nA worked count, since the theorem gives an exact number and that is unusual. Take $L$ = binary strings whose value is divisible by 3, read most significant bit first. How many states does the minimal DFA have?\nDo not build an automaton. Just count Nerode classes. Reading a prefix $x$, the only thing the future needs to know is the value of $x$ modulo 3, because appending a bit $b$ maps value $v$ to $2v + b$, and that map depends on $v$ only through $v \\bmod 3$. So there are at most three classes, and they are genuinely distinct: $\\varepsilon$ (value 0), 1 (value 1), and 10 (value 2) are pairwise distinguishable, since appending 0 accepts exactly the first. Exactly three states, proved without drawing anything, and a brute-force count of the classes over all prefixes up to length 14 confirms it.\nNow change the language slightly: binary strings divisible by 3 whose length is even. The future needs the residue and the parity of the length, and all six combinations are reachable and distinguishable, so the minimal DFA has exactly six states. The product structure appears in the count directly.\nContrast $L = {a^n b^n}$ from above, where the classes $[a^0], [a^1], [a^2], \\dots$ are pairwise distinguishable and there are infinitely many. The same counting procedure answers both \u0026ldquo;how many states\u0026rdquo; and \u0026ldquo;is it regular at all,\u0026rdquo; which is what makes this the tool to reach for first. You are always doing the same thing: asking what the minimum is that a prefix must tell you about itself, and then checking whether the answer fits in a bounded amount of memory.\nForbids A DFA for ${a^n b^n}$, palindromes, balanced parentheses, or ${ww}$, all by the same one-line argument.\nA DFA smaller than the Nerode index. This is a hard floor. No optimization, no state encoding, no clever alphabet trick goes below it.\nMultiple structurally different minimal DFAs. They are all the same machine with relabelled states, which is what makes DFA equality decidable: minimize both and check isomorphism.\nFinite automata for anything requiring unbounded memory of the input. If distinguishing prefixes requires remembering an unbounded quantity, the index is infinite. This is the honest formulation of \u0026ldquo;regular languages cannot count,\u0026rdquo; and it is the mechanism behind T038\u0026rsquo;s ladder.\nDoes not forbid It does not forbid small NFAs for languages with huge DFAs, which is the point people miss when they conclude a language is \u0026ldquo;expensive.\u0026rdquo; The language \u0026ldquo;the $k$-th symbol from the end is $a$\u0026rdquo; has exactly $2^k$ Nerode classes — a million-plus states at $k = 20$ — because you must remember the last $k$ symbols. But an NFA does it with $k+1$ states by guessing. The theorem bounds DFAs only, and this gap is why RE2 and Rust\u0026rsquo;s regex simulate NFAs with lazy DFA caching rather than building the DFA up front (T039).\nIt does not make minimization expensive. Hopcroft\u0026rsquo;s algorithm is $O(n \\log n)$ and is what flex, ANTLR\u0026rsquo;s lexer generator, and hardware synthesis tools run routinely on generated automata. The theorem says a canonical answer exists; the algorithm finds it quickly.\nIt does not mean non-regular languages are impractical. Balanced parentheses are not regular and every parser handles them. You move up the hierarchy to a pushdown automaton and get a stack. Non-regular means \u0026ldquo;needs more than finite memory,\u0026rdquo; not \u0026ldquo;hard.\u0026rdquo;\nIt does not apply to the extended regexes in your language\u0026rsquo;s standard library. Backreferences are not regular, so a re pattern using them has no DFA and no Nerode index, and reasoning about it with this theorem is a category error.\nIt does not mean the minimal DFA is the best implementation. Minimal state count is not minimal memory — a 12-state DFA over a 256-symbol alphabet has a 3072-entry transition table, and a larger automaton with a compressible structure may be faster in cache. Real lexers use table compression, and sometimes a non-minimal automaton wins. The theorem answers \u0026ldquo;how few states\u0026rdquo; and that is a different question from \u0026ldquo;how fast.\u0026rdquo;\nBoundary The theorem is specifically about DFAs over finite alphabets, and its relatives mark the edges:\nNFAs: no unique minimum, and minimization is PSPACE-complete. Context-free languages: no analogue. There is no canonical minimal pushdown automaton, and equivalence of CFGs is undecidable — which is exactly why grammar conflicts in a parser generator are reported rather than resolved. Weighted and probabilistic automata: analogues exist via the Hankel matrix, where rank plays the role of index. This is the basis of spectral learning for hidden Markov models. Infinite alphabets: register automata and nominal automata, where the theory partially survives. The idea to keep is the reframing. A DFA state is not a place; it is an equivalence class of histories, the answer to \u0026ldquo;what is the minimum I must remember about the past to behave correctly in the future?\u0026rdquo; That question makes sense far beyond automata — it is what a state machine in your codebase should be storing, and it is a good test to run on one. If two of your states never lead to different behaviour, they are the same state, and Myhill-Nerode says so.\nRead next ","permalink":"https://cs.lozic.me/posts/t040-the-myhill-nerode-theorem/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eYou need to prove a language is not regular. Everyone points you at the pumping\nlemma, and you spend an afternoon losing to it.\u003c/p\u003e\n\u003cp\u003eThe statement is a nest of quantifiers: for every regular language there exists\na pumping length $p$ such that for every string $w$ with $|w| \\ge p$ there exists\na decomposition $w = xyz$ with $|xy| \\le p$ and $|y| \u0026gt; 0$ such that for all $i\n\\ge 0$, $xy^i z$ is in the language. To use it you negate all of that and play a\ngame against an adversary who picks $p$ and the decomposition while you pick $w$\nand $i$.\u003c/p\u003e","title":"The Myhill–Nerode Theorem"},{"content":"Symptom Your service goes down. CPU pinned at 100% on one core, no memory growth, no error logs, requests timing out.\nThe cause is a validation regex — something reasonable-looking like ^(a+)+$, or in real life a URL or email validator with nested quantifiers — matched against a 30-character string that happens not to match. The engine is exploring an exponential number of ways to split the input among the nested groups, and it will get there eventually, some time after the heat death of everything.\nThis has a name, ReDoS, and it has taken down Stack Overflow (2016, a trailing-whitespace regex) and Cloudflare (2019, a WAF rule that took out a large fraction of the internet for half an hour).\nHere is the part that should bother you: matching a regular expression against a string can always be done in linear time. There is no exponential blowup anywhere in the theory. The engine that hung your service is not implementing the theory. It is implementing something more powerful and much worse, and the theorem explains exactly what got lost.\nStatement A language is describable by a regular expression if and only if it is recognized by a finite automaton.\nThe class of languages generated by regular expressions over $\\Sigma$ equals the class recognized by DFAs, which equals the class recognized by NFAs. Both directions are effective: each representation converts to the other algorithmically.\nTwo formalisms that look nothing alike. One is an algebraic notation with concatenation, alternation, and Kleene star. The other is a machine with states and transitions. They have exactly the same expressive power, and the equivalence is constructive in both directions, which is what makes it an engineering result rather than a classification.\nCombined with T038: this is the bottom rung of the Chomsky hierarchy, described three ways. The equivalence is why \u0026ldquo;regular\u0026rdquo; is a robust notion rather than an artifact of one notation.\nArgument Direction one: regex to NFA (Thompson\u0026rsquo;s construction). Structural induction on the expression, where each case builds a fragment with one start state and one accept state.\n$\\emptyset$: two states, no transition. $\\varepsilon$: an $\\varepsilon$-edge. A single character $a$: an $a$-edge. Concatenation $RS$: $\\varepsilon$-edge from $R$\u0026rsquo;s accept to $S$\u0026rsquo;s start. Alternation $R \\mid S$: new start with $\\varepsilon$-edges into both, and both accepts $\\varepsilon$-edge to a new accept. Kleene star $R^*$: new start/accept pair, $\\varepsilon$-edges to enter, to skip, and to loop back. Each operator adds at most two states, so an expression of size $m$ yields an NFA with $O(m)$ states. Linear, which is the fact the whole practical story rests on.\nDirection two: NFA to DFA (subset construction). A DFA state is a set of NFA states — everywhere the NFA could currently be. Start from the $\\varepsilon$-closure of the NFA\u0026rsquo;s start; on input $a$, move to the closure of everything reachable by $a$; accept if the set contains an NFA accept state. Finitely many subsets, so it terminates.\nThe cost is that $n$ NFA states give up to $2^n$ DFA states, and this is sometimes unavoidable: the language \u0026ldquo;the $k$-th symbol from the end is $a$\u0026rdquo; needs $k+1$ NFA states and provably $2^k$ DFA states, because the DFA must remember the last $k$ symbols. At $k = 20$ that is 21 states versus 1,048,576.\nDirection three: DFA to regex. State elimination. Label edges with regexes, then remove states one at a time, replacing paths $p \\to q \\to r$ with a direct edge labelled $R_{pq} R_{qq}^* R_{qr}$ — the star capturing loops at the eliminated state. When only start and accept remain, the edge label is the answer.\nHow this makes matching linear. Simulate the NFA directly, keeping the set of current states. Each input character updates the set once. With $m$ NFA states and length-$n$ input, that is $O(mn)$, and with on-the-fly DFA construction plus caching it is effectively $O(n)$.\nNo backtracking. No exponential blowup. The reason is structural: the NFA explores all alternatives simultaneously as a set, whereas backtracking explores them sequentially and re-explores overlapping work. Same automaton, different traversal, and the entire difference between microseconds and forever.\nSo why does Perl backtrack? Because it is not matching regular expressions. Backreferences — (a+)\\1, meaning \u0026ldquo;some text, then the same text again\u0026rdquo; — are not regular. The language ${ww}$ is not even context-free. Once your syntax includes backreferences you have left the theorem\u0026rsquo;s scope entirely, and the only known general algorithm is backtracking. Lookahead, recursive patterns, and atomic groups similarly exceed it. The performance disaster is the price of features that are not regular, paid on every pattern including the ones that do not use them.\nWatching the blowup happen. Take the pattern that hung the service, ^(a+)+$, against the input aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa! — thirty as and a character guaranteeing failure. A backtracking engine must try every way to partition those thirty as among repetitions of the inner group, and the number of compositions of 30 is $2^{29} \\approx 5.4 \\times 10^8$. At a hundred million steps per second that is about five seconds. Add ten more as and it is an hour and a half; at sixty as it is roughly 180 years, and at seventy it is about 190,000.\nNow run the same pattern through Thompson\u0026rsquo;s construction. The expression has about six operators, so the NFA has roughly a dozen states. Simulating it keeps a set of at most twelve states and updates that set once per input character. Thirty characters, twelve states, and the work is bounded by $12 \\times 31 = 372$ state updates. Microseconds, and the growth is linear no matter how many as you append.\nThe two engines are exploring the same graph. The backtracker walks paths one at a time and revisits shared prefixes exponentially often; the automaton walks the frontier of all paths at once and visits each state at most once per position. The exponential is entirely an artifact of the traversal order, which is why the fix requires no cleverness about the pattern and no timeout tuning — only a different engine.\nThis is also why input length limits are a poor mitigation. The blowup is exponential in length, so the difference between a safe limit and a fatal one is a handful of characters, and it depends on the pattern in a way no reviewer can eyeball.\nForbids Regexes matching balanced parentheses, or any properly nested structure. Requires unbounded counting, and a finite automaton has finitely many states. This is the theorem behind the famous refusal to parse HTML with a regex, and the reason is not stylistic.\nRegexes recognizing $a^n b^n$, palindromes, or ${ww}$. All above the regular rung of T038.\nA finite automaton counting without bound. \u0026ldquo;Even number of $a$s\u0026rdquo; is fine — two states. \u0026ldquo;Equal numbers of $a$s and $b$s\u0026rdquo; is not, at any state count.\nExponential worst-case matching for genuinely regular patterns. If your engine blows up on a regular pattern, the engine is at fault, not the pattern. This is a real and actionable distinction during an incident.\nDoes not forbid It does not mean regex engines are broken, and treating backtracking as a bug misses what those engines are for. PCRE, Perl, Python\u0026rsquo;s re, Java\u0026rsquo;s java.util.regex, and JavaScript\u0026rsquo;s built-in engine all implement a strict superset of regular expressions. Backreferences and lookahead are genuinely useful and genuinely non-regular. The engineering error is not that these exist; it is deploying them on untrusted input without a timeout.\nIt does not mean you have to accept ReDoS, and the fix is to use an engine that made the other choice. RE2 (Google) and Rust\u0026rsquo;s regex crate guarantee linear time by refusing to implement backreferences — the theorem, as a product decision. Go\u0026rsquo;s regexp is RE2-based. Cloudflare rewrote its WAF onto RE2 after the 2019 outage, and Rust\u0026rsquo;s crate is what powers ripgrep, which is why it outruns grep on large trees. If your regex runs on user input, this is the choice to make, and the theorem is what guarantees it is safe to make it.\nIt does not forbid practical extensions that stay regular. Bounded repetition a{2,5}, character classes, anchors, and case-insensitivity are all syntactic sugar over the regular core, expanding to plain automata. Capture groups can be handled in linear time too — RE2 does it with tagged transitions. Only backreferences and unbounded lookahead genuinely escape.\nIt does not mean DFA conversion is always worth it. The $2^n$ blowup is real, which is why RE2 builds DFA states lazily and caches them, falling back to NFA simulation when the cache fills. You get DFA speed on common inputs and a bounded memory footprint on adversarial ones, which is the practical resolution of the exponential.\nIt does not say all regex features are equally costly. A pattern using only regular features runs in linear time even in a backtracking engine if the engine recognizes that. Some do optimize this. Relying on it is unwise, since whether your specific pattern hits the optimized path is an implementation detail that can change on upgrade.\nBoundary The theorem\u0026rsquo;s exact scope, in terms that matter when choosing a tool:\nRegular operations only. Union, concatenation, star, and anything definable from them, including intersection and complement, which are closed for regular languages but can blow up the automaton. Backreferences leave the class. Matching becomes NP-hard in general. Lookahead and lookbehind of bounded width stay regular in principle but are usually implemented by backtracking. Recursive patterns (PCRE\u0026rsquo;s (?R)) reach into context-free territory, at which point you should be using a parser. The practical rule that falls out: if the input is untrusted, use an engine with a linear-time guarantee, and if you need backreferences on untrusted input, you need a different design rather than a longer timeout. And when you find yourself reaching for recursion or nested-structure matching, T038 already told you the answer, which is that you have a parsing problem and regular expressions are the wrong rung of the ladder.\nRead next ","permalink":"https://cs.lozic.me/posts/t039-kleenes-theorem-regexes-are-finite-automata/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eYour service goes down. CPU pinned at 100% on one core, no memory growth, no\nerror logs, requests timing out.\u003c/p\u003e\n\u003cp\u003eThe cause is a validation regex — something reasonable-looking like\n\u003ccode\u003e^(a+)+$\u003c/code\u003e, or in real life a URL or email validator with nested quantifiers —\nmatched against a 30-character string that happens not to match. The engine is\nexploring an exponential number of ways to split the input among the nested\ngroups, and it will get there eventually, some time after the heat death of\neverything.\u003c/p\u003e","title":"Kleene's Theorem: Regexes Are Finite Automata"},{"content":"Symptom You need to test whether two arithmetic circuits compute the same polynomial. The deterministic approach is to expand both into normal form, and the expansion is exponentially large.\nThe randomized approach takes four lines. Pick a random point, evaluate both circuits there, compare. If they differ as polynomials, the Schwartz-Zippel lemma says a random point catches it with probability at least $1 - d/|S|$. Repeat a few times and the error is negligible.\nThis is a pattern you have seen throughout Part II. Randomized quicksort dodges adversarial inputs. Miller-Rabin tests primality in milliseconds. Randomized load balancing beats every deterministic scheme you could tune by hand. Random sampling turns intractable counting into an estimate with a confidence interval.\nSo randomness is clearly powerful. It solves problems you do not know how to solve deterministically.\nExcept almost every complexity theorist believes it adds no power at all, and the reason is genuinely strange: whether you can eliminate randomness depends on whether hard problems exist. Not \u0026ldquo;hard problems make derandomization harder\u0026rdquo; — the opposite. Hardness is the resource that buys you fake randomness, and if computation is easy, randomness becomes irreplaceable.\nStatement $\\mathsf{BPP}$ is the class of languages decidable by a probabilistic polynomial-time machine with error probability at most $1/3$ on every input.\nConjecture: $\\mathsf{P} = \\mathsf{BPP}$. Randomness does not extend what is efficiently computable.\nTheorem (Impagliazzo-Wigderson, 1997): If some language in $\\mathsf{E}$ requires circuits of size $2^{\\Omega(n)}$, then $\\mathsf{P} = \\mathsf{BPP}$.\nIf there exists $L \\in \\mathsf{DTIME}(2^{O(n)})$ and $\\epsilon \u0026gt; 0$ such that $L$ requires circuits of size $2^{\\epsilon n}$ almost everywhere, then $\\mathsf{P} = \\mathsf{BPP}$.\nRead the conditional carefully, because the direction is the surprise. A statement that something is hard implies that randomized algorithms can be made deterministic. Hardness is not the obstacle. Hardness is the input.\nThe reason $1/3$ in the definition is not arbitrary but also not special: amplification makes any constant below $1/2$ equivalent. Run the algorithm $k$ times and take the majority. Chernoff (T008) gives exponential decay in $k$, and the concrete numbers make it vivid: one run with error $1/3$; majority of 11 runs gives error $0.122$; 41 runs gives $0.0135$; 101 runs gives $2.7 \\times 10^{-4}$; 301 runs gives $1.3 \\times 10^{-9}$. To push below $2^{-100}$ takes 1121 runs — a constant factor, which is why the error bound in the definition carries no information.\nArgument Why anyone believes $\\mathsf{P} = \\mathsf{BPP}$. Two pieces of evidence.\nFirst, problems keep getting derandomized. Primality was the flagship randomized-only problem for thirty years until Agrawal, Kayal, and Saxena gave a deterministic polynomial algorithm in 2002. Undirected connectivity was randomized-only until Reingold\u0026rsquo;s 2004 log-space algorithm. The list of problems that genuinely need randomness keeps shrinking, and the biggest survivor, polynomial identity testing, is exactly the one known to be equivalent to circuit lower bounds.\nSecond, and more decisively, the conditional theorem\u0026rsquo;s hypothesis is very weak. Everyone believes $\\mathsf{E}$ contains hard problems — it would be astonishing if every exponential-time language had subexponential circuits. So almost everyone believes the conclusion.\nThe mechanism: hardness to pseudorandomness. This is the construction worth carrying away.\nA pseudorandom generator stretches $\\ell$ truly random bits into $m \\gg \\ell$ bits that no small circuit can distinguish from random. If you have one with $\\ell = O(\\log m)$, you derandomize completely: enumerate all $2^{\\ell} = \\text{poly}(m)$ seeds, run the algorithm on each, take the majority. Polynomial time, no randomness.\nNisan and Wigderson showed how to build the generator from a hard function $f$. Take the seed, extract many subsets of its bits using a combinatorial design in which any two subsets overlap in few positions, and output $f$ applied to each subset. If some circuit distinguished this output from random, you could convert that distinguisher into a small circuit computing $f$ — the design\u0026rsquo;s low overlap is what makes the reconstruction possible — contradicting $f$\u0026rsquo;s hardness.\nImpagliazzo and Wigderson closed the remaining gap. Nisan-Wigderson needed $f$ to be hard on average, which is a strong assumption; IW showed worst-case hardness suffices, via hardness amplification using error-correcting codes. Encoding a worst-case-hard function with a suitable code produces one that is hard on average, because a good average-case algorithm would be a decoder.\n$$\\text{worst-case hard} \\xrightarrow{\\ \\text{codes}\\ } \\text{average-case hard} \\xrightarrow{\\ \\text{NW}\\ } \\text{PRG} \\xrightarrow{\\ \\text{enumerate}\\ } \\mathsf{P} = \\mathsf{BPP}$$The converse, which makes it a genuine equivalence. Kabanets and Impagliazzo proved in 2004 that derandomizing polynomial identity testing implies circuit lower bounds. So you cannot derandomize without proving hardness, and the two problems are the same problem. This is why $\\mathsf{P} = \\mathsf{BPP}$ is believed but unproven: proving it requires circuit lower bounds, and T030 explains why those are hard to come by.\nThe five worlds, because they make the trade concrete. Impagliazzo\u0026rsquo;s 1995 framing is the standard way to hold what depends on what. Each world is a consistent possibility given current knowledge, and which one we live in determines both cryptography and derandomization at once.\nAlgorithmica. $\\mathsf{P} = \\mathsf{NP}$. Optimization is free, and every cryptographic primitive fails. No PRGs, so randomness is not eliminable by this route. Heuristica. $\\mathsf{P} \\ne \\mathsf{NP}$ but NP problems are easy on average. No cryptography, since hardness on random instances is what encryption needs. Pessiland. Hard instances exist but nobody can generate them with known solutions. The worst world: no cryptography and no efficient algorithms. Minicrypt. One-way functions exist, so symmetric cryptography and PRGs work, but public-key does not. Cryptomania. Trapdoor functions exist. Public-key cryptography, secure multiparty computation, everything we currently deploy. We behave as though we live in Cryptomania, and the entire internet is built on that assumption. Note what this post adds: in Minicrypt and Cryptomania, hardness is abundant, PRGs exist, and $\\mathsf{P} = \\mathsf{BPP}$ follows. In Algorithmica and Heuristica, hardness is scarce and derandomization by this construction is unavailable.\nSo the same fact that secures your TLS session is the fact that makes randomized algorithms dispensable. Cryptographers and complexity theorists are betting on identical propositions with opposite hopes, and that is the cleanest statement of why hardness is a resource rather than an obstacle.\nForbids Derandomization without hardness. By Kabanets-Impagliazzo, a proof that $\\mathsf{P} = \\mathsf{BPP}$ yields circuit lower bounds. There is no easy route.\nA world with both easy computation and unavoidable randomness. If everything in $\\mathsf{E}$ has small circuits, no PRG of this kind exists. Hardness and derandomizability rise and fall together.\nRandomness as a source of computational power in the usual sense. Under the believed hypothesis, $\\mathsf{BPP}$ collapses to $\\mathsf{P}$ and every randomized algorithm has a deterministic equivalent within a polynomial factor.\nCryptography in a world without hard problems. The same construction that yields PRGs yields pseudorandom functions, so if $\\mathsf{E}$ has small circuits, cryptography as we know it fails. This is Impagliazzo\u0026rsquo;s \u0026ldquo;Algorithmica\u0026rdquo; world, and it is the same trade viewed from the other side.\nDoes not forbid It does not mean randomized algorithms are pointless, and dropping them because \u0026ldquo;$\\mathsf{P} = \\mathsf{BPP}$ anyway\u0026rdquo; is the misreading that would cost you the most in practice. The derandomization is polynomial-time but the polynomial is dreadful, and no one implements it. Randomized quicksort, Miller-Rabin in OpenSSL and GnuPG, Monte Carlo integration, randomized routing, and hash-based load balancing are all faster, simpler, and more robust than their deterministic counterparts. $\\mathsf{P} = \\mathsf{BPP}$ is a statement about asymptotic classes and not about what to write.\nIt does not apply to the interactive or space-bounded settings. Randomness is provably powerful in interactive proofs — that is IP = PSPACE (T036) — and $\\mathsf{RL}$ versus $\\mathsf{L}$ remains open. The collapse is specific to polynomial time.\nIt does not mean true randomness is unnecessary for cryptography. Very much the opposite: the same theory says cryptographic PRGs require hardness assumptions, and a PRG seeded predictably is broken. The Debian OpenSSL bug of 2008 reduced the entropy pool to the process ID, generating 32,767 possible keys total. The Sony PS3 ECDSA break came from a reused nonce. Both are the theorem\u0026rsquo;s warning in production, and both cost far more than any asymptotic question.\nIt does not forbid quantum speedups. $\\mathsf{BQP}$ is a different class, and Shor\u0026rsquo;s algorithm is not evidence about $\\mathsf{BPP}$ at all. Quantum advantage, if it exists, comes from interference rather than randomness.\nIt does not say the hypothesis is proven. No superpolynomial circuit lower bound is known for any explicit function in $\\mathsf{NP}$ — the record is about $5n$ gates. The strongest evidence for $\\mathsf{P} = \\mathsf{BPP}$ rests on an assumption nobody can currently establish, and honesty about that is part of understanding the result.\nBoundary The class landscape around $\\mathsf{BPP}$ is worth holding precisely:\n$\\mathsf{RP}$ and $\\mathsf{coRP}$ — one-sided error. Miller-Rabin is $\\mathsf{coRP}$: a composite verdict is certain, a prime verdict is probable. $\\mathsf{ZPP} = \\mathsf{RP} \\cap \\mathsf{coRP}$ — always correct, expected polynomial time. Las Vegas rather than Monte Carlo. $\\mathsf{BPP}$ — two-sided error, and the class in question. $\\mathsf{PP}$ — error just under $1/2$, and much larger; $\\mathsf{PP}$ contains $\\mathsf{NP}$, so the gap in the error bound matters enormously. Known unconditionally: $\\mathsf{BPP} \\subseteq \\mathsf{P}/\\text{poly}$ (Adleman), and $\\mathsf{BPP} \\subseteq \\Sigma_2 \\cap \\Pi_2$ (Sipser-Gács-Lautemann). So $\\mathsf{BPP}$ is not wildly large. What is not known is whether $\\mathsf{BPP} \\subseteq \\mathsf{NP}$, which is a good calibration point for how little is settled here.\nThe lasting idea is the exchange rate. Randomness and hardness are convertible: a hard function is a randomness generator, and a distinguisher is an algorithm. Once you see that, cryptography and derandomization stop being separate subjects and become the same theorem read in two directions.\nRead next ","permalink":"https://cs.lozic.me/posts/t035-bpp-pseudorandomness-and-derandomization/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eYou need to test whether two arithmetic circuits compute the same polynomial.\nThe deterministic approach is to expand both into normal form, and the expansion\nis exponentially large.\u003c/p\u003e\n\u003cp\u003eThe randomized approach takes four lines. Pick a random point, evaluate both\ncircuits there, compare. If they differ as polynomials, the\nSchwartz-Zippel lemma says a random point catches it with probability at least\n$1 - d/|S|$. Repeat a few times and the error is negligible.\u003c/p\u003e","title":"BPP, Pseudorandomness, and Derandomization"},{"content":"Symptom You have a string algorithm. Edit distance between two sequences, the classic dynamic program, $O(n^2)$ time. It has been in production for years.\nNow the inputs are genome-scale. At $n = 10^5$ characters, $n^2 = 10^{10}$ operations, about 10 seconds at a billion ops per second. At $n = 10^6$ it is $10^{12}$ operations, about 1000 seconds — seventeen minutes for one pair of strings. You need to do a million pairs.\nSo you go looking for a subquadratic algorithm. You find heuristics, bit-packing tricks worth a constant factor of 64, banded variants that assume the answer is small. What you do not find is an $O(n^{1.9})$ algorithm, and after a week you start wondering whether you are missing something obvious.\nYou are not. And the classical theory is useless here, because edit distance is in P. Polynomial time, solved, efficient by the standard everyone taught you. P vs NP has nothing to say about the gap between $n^2$ and $n^{1.9}$, and that gap is where your actual problem lives.\nThat is the hole fine-grained complexity fills. P vs NP asks whether a problem is tractable. Fine-grained complexity asks whether your particular exponent is optimal, and it answers with conditional lower bounds that are just as actionable.\nStatement Three hypotheses, in increasing strength.\nETH. 3-SAT cannot be solved in $2^{o(n)}$ time on formulas with $n$ variables.\nSETH. For every $\\epsilon \u0026gt; 0$ there is a $k$ such that $k$-SAT cannot be solved in $O((2-\\epsilon)^n)$ time.\nFine-grained reductions transfer these lower bounds to problems in P.\nETH: there exists $\\delta \u0026gt; 0$ with no $O(2^{\\delta n})$ algorithm for 3-SAT. SETH: $\\lim_{k \\to \\infty} s_k = 1$ where $s_k = \\inf{\\delta : k\\text{-SAT} \\in O(2^{\\delta n})}$. SETH implies ETH implies $\\mathsf{P} \\ne \\mathsf{NP}$.\nSETH says brute force over $2^n$ assignments is essentially optimal — you can shave the base below 2 for any fixed $k$, but not by a constant that works for all $k$.\nThese are conjectures, and that is the honest framing. But they are load- bearing conjectures with decades of failed attacks, and a refutation would be a larger result than most people are trying to prove. Building on them is the same bet as building on the hardness of factoring.\nArgument Why ETH is plausible. Fifty years of SAT solving has produced enormous practical progress and no asymptotic progress. The best 3-SAT algorithms run in about $O(1.308^n)$ — Hertli\u0026rsquo;s refinement of Schöning\u0026rsquo;s random-walk algorithm. That is a real improvement over $2^n$: at $n = 50$, $1.308^{50} \\approx 6.8 \\times 10^5$ versus $2^{50} \\approx 1.1 \\times 10^{15}$, nine orders of magnitude. But it is still exponential, and the base has crept down glacially while the form has never changed.\nThe key move: fine-grained reductions. A classical reduction preserves polynomial-time solvability and is allowed to blow up the instance polynomially — which destroys exponent information and is why Karp reductions (T028) cannot answer this question. A fine-grained reduction preserves the exponent: it maps an instance of size $n$ to instances of size $O(n^{1+o(1)})$ such that an $O(n^{2-\\epsilon})$ algorithm for the target yields an $O((2-\\delta)^n)$ algorithm for SAT.\nThe hub problem: Orthogonal Vectors. Given two sets of $n$ vectors in ${0, 1}^d$, is there a pair with zero dot product? The obvious algorithm is $O(n^2 d)$.\nSETH implies OV needs $n^{2-o(1)}$ time. Split the $n$ variables of a CNF formula in half. Enumerate all $2^{n/2}$ assignments to each half, and for each one build a $d$-bit vector recording which clauses it leaves unsatisfied. A pair of half-assignments is orthogonal exactly when together they satisfy every clause. So SAT becomes OV on $N = 2^{n/2}$ vectors, and an $O(N^{2-\\epsilon})$ OV algorithm gives $O(2^{(1-\\epsilon/2)n})$ for SAT, contradicting SETH.\nAnd OV reduces into everything. Backurs and Indyk showed in 2015 that a truly subquadratic edit distance algorithm gives a subquadratic OV algorithm and therefore refutes SETH. The same holds for longest common subsequence, dynamic time warping, Fréchet distance, and local alignment. Your $O(n^2)$ edit distance dynamic program is optimal unless SETH is false, and that is a complete answer to the week you spent searching.\nThe other two roots. APSP: all-pairs shortest paths is conjectured to need $n^{3-o(1)}$, and negative-weight triangle, graph radius, and replacement paths are all equivalent to it — a subcubic algorithm for any one gives subcubic for all. 3SUM: given $n$ integers, are there three summing to zero? Conjectured $n^{2-o(1)}$, and it implies quadratic lower bounds across computational geometry, including collinearity testing and polygon containment.\nThree conjectures, one reduction web, and most everyday polynomial problems hang off one of them.\nA worked reduction, so the transfer is concrete. Take the SETH-to-OV argument and put numbers on it. A CNF formula with $n = 60$ variables and $d = 200$ clauses splits into two halves of 30 variables each. Enumerating one half gives $2^{30} \\approx 1.07 \\times 10^9$ assignments, and each becomes a 200-bit vector marking which clauses that half fails to satisfy.\nA pair of vectors is orthogonal exactly when no clause is left unsatisfied by both halves — that is, when the combined assignment satisfies the formula. So the OV instance has $N = 2^{30}$ vectors, and $N^2 = 2^{60}$, which is precisely the brute-force SAT cost. Nothing has been gained yet.\nThe leverage is in what a faster OV algorithm would mean. An $O(N^{1.9})$ algorithm would solve this SAT instance in $2^{57}$ steps, and more importantly in $2^{0.95n}$ steps for every $n$. That is a base of $2^{0.95} \\approx 1.93$ rather than 2, uniformly across all clause widths, which is exactly what SETH says is impossible. The reduction converts a modest polynomial improvement on a quadratic problem into a modest exponential improvement on SAT, and the reason the exponents line up is the halving: squaring $2^{n/2}$ recovers $2^n$, so an exponent below 2 downstream is an exponential base below 2 upstream.\nThis is why fine-grained reductions must be near-linear in the instance size. A reduction that blew the instance up to $N^{1.5}$ would dilute the exponent saving to nothing, which is exactly what classical Karp reductions do and why they cannot see this structure.\nForbids Subquadratic edit distance, LCS, dynamic time warping, and Fréchet distance, unless SETH fails.\nSubcubic all-pairs shortest paths, unless the APSP conjecture fails. Floyd-Warshall\u0026rsquo;s exponent is not laziness. Williams\u0026rsquo; $n^3/2^{\\Theta(\\sqrt{\\log n})}$ shaves a subpolynomial factor and stops there.\nSubquadratic 3SUM and its geometric descendants, unless 3SUM fails.\n$2^{o(n)}$ algorithms for 3-SAT, vertex cover, Hamiltonian path, and graph colouring, unless ETH fails. ETH also rules out $n^{o(k)}$ for $k$-clique, which is the lower-bound half of parameterized complexity (T033) — W[1]-hardness becomes a concrete time bound rather than just a class membership.\nA subexponential-time approximation scheme for many problems, via ETH-based lower bounds on PTAS running times, which is a sharper statement than the inapproximability of T031.\nDoes not forbid It does not forbid fast edit distance in practice, and treating the lower bound as a reason not to optimize is the misreading that costs the most real performance. Myers\u0026rsquo; bit-vector algorithm processes 64 cells per word operation, a genuine 64x speedup that leaves the exponent untouched. Landau- Vishkin runs in $O(nk)$ when the edit distance is at most $k$, which is fast whenever the strings are similar — the usual case in bioinformatics. Minimap2 and BLAST align genomes daily by seeding and filtering so the quadratic kernel only ever runs on short candidate regions. Asymptotic optimality says nothing about constants, parameters, or input structure.\nIt does not forbid approximation. Edit distance is approximable within a constant factor in near-linear time — Andoni and Nosatzki reached $O(n^{1+ \\epsilon})$ in 2020. The lower bound is for exact computation, and relaxing exactness escapes it completely. This is the single most useful escape and it is routinely forgotten.\nIt does not mean SETH is true. It is a conjecture, and there is real doubt. SETH has already been refuted in restricted settings, and Ryan Williams — who proved several of the strongest SETH-based results — has said publicly he suspects it is false. The results are all stated conditionally for exactly this reason, and a refutation would be a triumph rather than an embarrassment.\nIt does not apply to quantum algorithms directly. Grover gives $O(2^{n/2})$ for SAT, which violates SETH\u0026rsquo;s classical form; the quantum analogue QSETH is formulated separately and is much less studied.\nIt does not forbid better algorithms for structured inputs. These are worst-case bounds. Real strings have low edit distance, real graphs are sparse and often planar, and real geometric inputs are not adversarial. Parameterized algorithms (T033) exploit this systematically, and the fine-grained bounds are what tell you which parameter to reach for.\nBoundary The conjectures sit in a hierarchy, and knowing which one your lower bound rests on tells you how much to trust it:\n$\\mathsf{P} \\ne \\mathsf{NP}$ — weakest, most believed. ETH — implies the above; rules out subexponential exact algorithms. SETH — implies ETH; the least believed of the three, and the one carrying the most quadratic lower bounds. APSP and 3SUM conjectures — independent of the SAT-based ones, each anchoring its own equivalence class. What the field genuinely delivers is a shift in what \u0026ldquo;I cannot find a better algorithm\u0026rdquo; means. Before fine-grained complexity, that was a statement about your week. After it, you can often prove it is a statement about the problem, conditional on a hypothesis you were implicitly relying on anyway. And when the answer comes back \u0026ldquo;your exponent is optimal,\u0026rdquo; the productive next moves are named and finite: approximate it, parameterize it, exploit structure, or buy constants. That is a far better place to end a week than where this post started.\nRead next ","permalink":"https://cs.lozic.me/posts/t037-the-exponential-time-hypothesis-and-fine-grained-complexity/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eYou have a string algorithm. Edit distance between two sequences, the classic\ndynamic program, $O(n^2)$ time. It has been in production for years.\u003c/p\u003e\n\u003cp\u003eNow the inputs are genome-scale. At $n = 10^5$ characters, $n^2 = 10^{10}$\noperations, about \u003cstrong\u003e10 seconds\u003c/strong\u003e at a billion ops per second. At $n = 10^6$ it is\n$10^{12}$ operations, about \u003cstrong\u003e1000 seconds\u003c/strong\u003e — seventeen minutes for one pair of\nstrings. You need to do a million pairs.\u003c/p\u003e","title":"The Exponential Time Hypothesis and Fine-Grained Complexity"},{"content":"Symptom You have just read T030 and you are demoralized. Relativization kills the techniques that treat machines as black boxes. Natural proofs kill the combinatorial ones, assuming pseudorandom functions exist. Between them they appear to rule out everything anyone knows how to do, and the honest summary of fifty years is that nobody can prove any interesting problem is hard.\nExcept that summary is wrong, and believing it is the reason people leave this field.\nHere is a problem: given $n$ input bits, output their XOR. It is trivial — one line of code, and a chain of $n-1$ XOR gates computes it with $n$ gates.\nNow add one constraint. Your circuit has AND and OR gates of unbounded fan-in, so a single gate can read all $n$ inputs at once, plus free NOT gates. The only restriction is constant depth: a fixed number of layers, independent of $n$.\nYou cannot do it. Not with a clever encoding, not with a million gates. Any constant-depth circuit computing parity needs exponentially many gates, and this is not a conjecture. It is proved, unconditionally, with no assumptions about anything.\nStatement $\\mathsf{AC}^0$ is the class of languages decided by circuit families of constant depth and polynomial size, over unbounded fan-in AND, OR, and NOT.\nParity is not in $\\mathsf{AC}^0$. Any depth-$d$ circuit computing the parity of $n$ bits requires at least $2^{\\Omega(n^{1/(d-1)})}$ gates.\n$\\mathrm{PARITY} \\notin \\mathsf{AC}^0$. Håstad\u0026rsquo;s bound is tight: depth-$d$ circuits of size $2^{O(n^{1/(d-1)})}$ do compute parity.\nTwo things deserve emphasis.\nIt is unconditional. No assumption that $\\mathsf{P} \\ne \\mathsf{NP}$, no cryptographic hypothesis, no oracle. This is a theorem in the same sense that the pigeonhole principle is a theorem.\nIt is tight. The bound is matched by a construction, so this is the exact answer rather than the best current lower bound.\nPut numbers on it. At $n = 1000$ and depth 3, the exponent $n^{1/2}$ is about 31.6, so you need roughly $2^{31.6} \\approx 3.3 \\times 10^9$ gates to XOR a thousand bits in three layers. At depth 5 the exponent drops to about 5.6 and the bound becomes vacuous, which is exactly what \u0026ldquo;constant depth costs exponentially\u0026rdquo; means: buying one more layer is worth an enormous amount, and no fixed number of layers is ever enough.\nArgument The proof is Håstad\u0026rsquo;s switching lemma, and the strategy is to attack the circuit rather than the function.\nThe setup. Assume a depth-$d$, size-$S$ circuit computes parity. Normalize it so the bottom layer consists of small-width AND gates (a DNF at the bottom).\nThe move: random restriction. Pick each input independently. With probability $p$ leave it free; otherwise fix it to 0 or 1 at random. This kills most inputs and simplifies the circuit drastically.\nThe lemma. If a function is computed by a width-$w$ DNF, then after a random restriction keeping each variable free with probability $p$, the restricted function is computed by a width-$k$ CNF — except with probability at most $(5pw)^k$.\nThat is the whole engine, and the point is the swap. An OR of ANDs becomes an AND of ORs. Once the bottom two layers are both ANDs, they merge into one, and the circuit\u0026rsquo;s depth drops by one at the cost of a probability penalty.\nThe induction. Apply the lemma repeatedly with $p \\approx 1/(10w)$ at each stage. Each application costs one layer. After $d-2$ applications the circuit has collapsed to depth 2 — a single small-width DNF or CNF — and a union bound over all $S$ gates keeps the total failure probability below 1, so some restriction achieves every collapse simultaneously.\nThe contradiction. Parity has a property nothing else has: every restriction of parity is parity or its negation, on the surviving variables. Fixing some inputs never simplifies it, because every remaining variable still flips the answer. So if enough variables survive — and with $p$ chosen as above, about $n^{1/(d-1)}$ do — the collapsed depth-2 circuit must compute parity on them.\nBut a depth-2 circuit computing parity on $m$ variables needs width $m$: any term of width less than $m$ leaves a variable free, and that variable flips the output, so the term cannot be decisive. A width-$m$ DNF needs $2^{m-1}$ terms.\nSince we assumed the circuit was small, contradiction, and the surviving count $m \\approx n^{1/(d-1)}$ is exactly where the bound\u0026rsquo;s shape comes from. $\\blacksquare$\nWhy this evades the barriers of T030. Relativization does not apply because circuit lower bounds are not statements about oracle machines — there is no machine to relativize. Natural proofs does not apply because the switching lemma argument is not large: the property \u0026ldquo;is not simplified by any restriction\u0026rdquo; is satisfied by essentially no functions, whereas a natural proof requires a property that most functions have. The proof exploits a highly specific structural feature of parity, and that specificity is what lets it through.\nThis is the reason the post exists. The barriers rule out entire families of technique, not the possibility of progress, and $\\mathsf{AC}^0$ is the standing proof that unconditional lower bounds against real computational models are obtainable.\nA second reading of the same fact: parity has no low-degree approximation. Razborov and Smolensky\u0026rsquo;s route to the same conclusion is worth carrying because it generalizes further. Represent each gate by a polynomial over $\\mathbb{F}_2$ that agrees with it on most inputs. An unbounded fan-in OR of $n$ inputs is approximable by a polynomial of degree $O(\\log n)$ by randomly subsampling the inputs, so a depth-$d$ circuit of size $S$ is approximable by a polynomial of degree $(\\log S)^d$, agreeing with the circuit on most inputs.\nNow compute parity\u0026rsquo;s degree. Over $\\mathbb{F}_2$, parity is the degree-1 polynomial $x_1 + \\cdots + x_n$, so that field is useless. Move to $\\mathbb{F}_3$, encode bits as $\\pm 1$, and parity becomes the product $\\prod x_i$, which has degree $n$ and provably admits no approximation of degree below $\\sqrt{n}$ on more than a small fraction of inputs. Setting $(\\log S)^d \u0026lt; \\sqrt{n}$ gives $S \u0026gt; 2^{n^{1/(2d)}}$ — the same shape of bound from a completely different technique.\nTwo independent proofs matter here for the same reason two independent measurements matter. It is not a fluke of one clever argument, and the polynomial method is what extends to $\\mathsf{AC}^0[p]$ for prime $p$, which is why the wall at composite modulus is precisely where $\\mathbb{Z}/6$ stops being a field.\nForbids Constant-depth polynomial-size circuits for parity, and hence for anything parity reduces to: majority, sorting, integer multiplication, and connectivity are all outside $\\mathsf{AC}^0$ by consequence.\nA polynomial-size constant-depth CRCW PRAM algorithm for parity. The $\\mathsf{AC}^0$ class corresponds exactly to constant-time PRAM computation with polynomially many processors, so this is a genuine parallel-computing lower bound: XOR cannot be done in $O(1)$ parallel time with polynomial hardware.\nCertain circuit designs, physically. A parity tree over 64 bits has logarithmic depth for a reason, and no layout trick flattens it. Every carry chain and every ECC syndrome generator in silicon pays this depth, which is why carry-lookahead adders buy depth with area and cannot buy it down to constant.\nHope that $\\mathsf{AC}^0$ captures useful computation. It contains addition and comparison, and stops almost immediately after.\nDoes not forbid It does not forbid fast parity in hardware, which is the misreading that makes people think the theorem is wrong. Your CPU has a parity flag and computes it in one cycle. POPCNT counts set bits in a 64-bit word in a single instruction. There is no contradiction: $n$ is fixed at 64, and \u0026ldquo;constant depth, polynomial size\u0026rdquo; is an asymptotic statement about a family of circuits. The theorem says depth must grow with $n$, and 64-bit hardware uses a fixed logarithmic-depth tree of six layers. Fixed-width hardware never feels an asymptotic bound.\nIt does not forbid parity in slightly larger classes, and the boundary is startlingly sharp. $\\mathsf{AC}^0[\\oplus]$ — the same circuits plus MOD-2 gates — contains parity by definition. $\\mathsf{NC}^1$, allowing logarithmic depth with bounded fan-in, contains it easily. Razborov and Smolensky extended the result to $\\mathsf{AC}^0[p]$ for prime $p$, and then progress stopped: nobody can prove a lower bound against $\\mathsf{ACC}^0$ with composite modulus, say MOD-6 gates. Williams\u0026rsquo; 2011 result separating $\\mathsf{NEXP}$ from $\\mathsf{ACC}^0$ was celebrated precisely because that wall had stood for twenty-five years.\nIt does not mean lower bounds are easy now. The distance from $\\mathsf{AC}^0$ to $\\mathsf{P}$ is enormous, and the current frontier for general circuits is embarrassing: the best known lower bound for an explicit function in $\\mathsf{NP}$ is around $5n$ gates. Five times $n$. Proving anything superlinear for general circuits is open.\nIt does not make parity hard in any practical sense. It is $O(n)$ time, $O(\\log n)$ depth, and about as cheap as computation gets. The theorem is about one specific restricted model, and \u0026ldquo;not in $\\mathsf{AC}^0$\u0026rdquo; is a statement about parallel depth rather than about difficulty.\nIt does not rely on any unproven assumption, unlike almost everything else in Part IV. If you take one structural fact from this post, it is that this result would survive a proof that $\\mathsf{P} = \\mathsf{NP}$ tomorrow.\nBoundary The result is exactly about constant depth with unbounded fan-in, and every neighbouring model behaves differently:\nAdd MOD-$p$ gates ($\\mathsf{AC}^0[p]$, prime $p$): Razborov-Smolensky still gives lower bounds, by approximating circuits with low-degree polynomials over $\\mathbb{F}_p$. Composite modulus ($\\mathsf{ACC}^0$): open, and the polynomial method breaks because $\\mathbb{Z}/6$ is not a field. This is the current frontier. Logarithmic depth, bounded fan-in ($\\mathsf{NC}^1$): parity is in, and no separation from $\\mathsf{P}$ is known. Threshold gates ($\\mathsf{TC}^0$): parity is in, and this matters practically because $\\mathsf{TC}^0$ is essentially constant-depth neural networks. The reason a small transformer struggles to learn parity while learning far more complex-looking functions easily is this hierarchy showing through, and it is a live thread in the length-generalization literature. What the theorem ultimately provides is a proof of concept. The barriers say certain techniques cannot work; $\\mathsf{AC}^0$ says lower bounds themselves are attainable when you find a structural handle. The field is not stuck for lack of possibility. It is stuck for lack of handles.\nRead next ","permalink":"https://cs.lozic.me/posts/t034-parity-is-not-in-ac0/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eYou have just read T030 and you are demoralized. Relativization kills the\ntechniques that treat machines as black boxes. Natural proofs kill the\ncombinatorial ones, assuming pseudorandom functions exist. Between them they\nappear to rule out everything anyone knows how to do, and the honest summary of\nfifty years is that nobody can prove any interesting problem is hard.\u003c/p\u003e","title":"Parity Is Not in AC^0"},{"content":"Symptom You have spent this whole part learning that information wants to be copied. Entropy bounds compression (T018), Huffman hits the bound (T020), redundancy buys error correction (T021). Every result so far treats \u0026ldquo;make another copy\u0026rdquo; as the free operation — the thing you do without thinking, the reason cp has no interesting failure modes.\nThen you write your first quantum program and the compiler rejects a line that looks completely ordinary. You wanted to save a qubit\u0026rsquo;s state before a measurement so you could check your work. There is no instruction for it. Not a slow one, not an approximate one. The operation does not exist.\nThe instinct is that this is an engineering gap — quantum hardware is young, someone will build a qubit copier eventually. It is not a gap. And the reason matters far beyond quantum computing, because if you could copy an unknown quantum state, you could signal faster than light, and the theorem is what stands between quantum mechanics and a contradiction with relativity.\nStatement There is no unitary operation that copies an arbitrary unknown quantum state.\nThere is no unitary $U$ such that for all states $\\lvert\\psi\\rangle$,\n$$U \\big( \\lvert\\psi\\rangle \\otimes \\lvert 0 \\rangle \\big) = \\lvert\\psi\\rangle \\otimes \\lvert\\psi\\rangle .$$ The quantifier is the whole theorem. Arbitrary and unknown. You can copy a state you already know the description of — just prepare a second one the same way. You can copy states drawn from a known orthogonal set, which is exactly what a classical bit is. What you cannot do is build one device that takes in any state, including ones you have no description of, and emits two.\nAnd the proof is not deep. It is linearity. Quantum evolution is linear, copying is not, and that is the entire conflict.\nArgument Proof one: linearity. Suppose $U$ clones two particular states,\n$$U(\\lvert\\psi\\rangle \\lvert 0\\rangle) = \\lvert\\psi\\rangle\\lvert\\psi\\rangle, \\qquad U(\\lvert\\phi\\rangle \\lvert 0\\rangle) = \\lvert\\phi\\rangle\\lvert\\phi\\rangle .$$Now feed it the superposition $\\lvert\\chi\\rangle = \\alpha\\lvert\\psi\\rangle + \\beta\\lvert\\phi\\rangle$. Because $U$ is linear, it must distribute:\n$$U(\\lvert\\chi\\rangle\\lvert 0\\rangle) = \\alpha \\lvert\\psi\\rangle\\lvert\\psi\\rangle + \\beta \\lvert\\phi\\rangle\\lvert\\phi\\rangle .$$But cloning $\\lvert\\chi\\rangle$ demands\n$$\\lvert\\chi\\rangle\\lvert\\chi\\rangle = \\alpha^2 \\lvert\\psi\\rangle\\lvert\\psi\\rangle\n\\alpha\\beta \\lvert\\psi\\rangle\\lvert\\phi\\rangle \\beta\\alpha \\lvert\\phi\\rangle\\lvert\\psi\\rangle \\beta^2 \\lvert\\phi\\rangle\\lvert\\phi\\rangle .$$ These agree only when the cross terms vanish and $\\alpha^2 = \\alpha$, $\\beta^2 = \\beta$ — that is, only when one of $\\alpha,\\beta$ is 1 and the other 0. So $U$ clones $\\lvert\\psi\\rangle$ and $\\lvert\\phi\\rangle$ but no genuine superposition of them. $\\blacksquare$\nCopying is a quadratic operation on amplitudes; unitary evolution is linear. There is nothing more to it than that mismatch.\nProof two: inner products. Unitaries preserve inner products. If $U$ cloned both states, then\n$$\\langle \\psi \\vert \\phi \\rangle = \\langle \\psi \\vert \\phi \\rangle \\cdot \\langle \\psi \\vert \\phi \\rangle,$$so $\\langle\\psi\\vert\\phi\\rangle \\in {0, 1}$: the states are either orthogonal or identical. A cloner can therefore only handle a set of mutually orthogonal states — which is precisely a classical alphabet. Classical information is the part of quantum information that happens to be clonable, and that reframing is the one worth keeping.\nWhy this saves relativity. Entanglement creates correlations that appear instantaneous. Alice and Bob share a Bell pair; Alice measures in one of two bases and Bob\u0026rsquo;s qubit collapses correspondingly, immediately, at any distance. No signal is sent, because Bob\u0026rsquo;s outcomes are uniformly random whichever basis Alice chose, and his marginal distribution is identical either way. He learns nothing.\nUnless he could clone. Given a million copies of his qubit, Bob could do tomography, determine the state, infer Alice\u0026rsquo;s basis choice, and read one bit — faster than light, in flat contradiction with causality. Nick Herbert proposed exactly this in 1981, and Wootters, Zurek, and Dieks published no-cloning in 1982 in direct response. The theorem is not a curiosity about qubits; it is the reason quantum mechanics and special relativity coexist.\nWhere the information actually is. It is worth being precise about what a qubit contains, because the usual \u0026ldquo;infinite information in one qubit\u0026rdquo; line makes no-cloning sound like a tragedy. A pure qubit state is a point on the Bloch sphere, specified by two real angles, so it does take infinitely many bits to write down. But Holevo\u0026rsquo;s theorem says you can extract at most one classical bit from measuring one qubit. The description is continuous and the accessible content is a single bit.\nSo no-cloning is not withholding a treasure. It is the flip side of the same fact: the reason you cannot copy the state is the reason you cannot read it, and both come from the same place, which is that measurement returns one outcome from a basis you had to choose in advance. A cloner would let you defeat that by measuring each copy in a different basis. The theorem and the measurement postulate are two views of one restriction, and a universe with one but not the other would be inconsistent.\nForbids Copying an unknown qubit, exactly, ever, by any means.\nPerfect quantum error correction by naive redundancy. The classical repetition code — store it three times, take a majority — is not available, and this shaped the entire field. Quantum error correction had to be invented differently: Shor\u0026rsquo;s nine-qubit code spreads one logical qubit\u0026rsquo;s information across entanglement among nine physical qubits without ever copying it, and syndrome measurement extracts error information without measuring the data.\nBacking up a quantum state, saving it before a risky operation, or checkpointing a quantum computation. Every debugging technique that relies on inspecting intermediate state is unavailable. This is a genuine reason quantum programming is hard, not an incidental tooling gap.\nSignalling faster than light using entanglement. The theorem closes the loophole directly, and every subsequent proposal for FTL signalling with entanglement fails at the same step.\nUndetectably eavesdropping on a quantum channel. An eavesdropper cannot copy a qubit in transit and forward the original untouched. She must measure, and measuring in the wrong basis disturbs the state. That disturbance is the detection mechanism, and it is the entire security argument for QKD.\nDoes not forbid It does not forbid quantum key distribution being attacked, and \u0026ldquo;no-cloning makes QKD unbreakable\u0026rdquo; is the misreading this post exists to kill. The theorem protects the channel, not the endpoints. Every practical QKD break has been an implementation attack: Vadim Makarov\u0026rsquo;s group blinded commercial ID Quantique and MagiQ detectors with bright laser light, turning single-photon detectors into classical ones and reading the entire key undetected. Photon-number splitting exploits attenuated lasers that occasionally emit two identical photons, letting Eve keep one — no cloning required, because the source did the duplication. Trojan-horse attacks probe Alice\u0026rsquo;s modulator from outside. The physics is sound and the devices are not, and the security proof\u0026rsquo;s assumptions about the hardware are where the entire attack surface lives.\nIt does not forbid approximate cloning. The Bužek-Hillery universal cloner produces two copies with fidelity $5/6 \\approx 0.833$ for arbitrary qubit inputs, and that is provably optimal. Perfect cloning is impossible; imperfect cloning is a well-characterized resource with a known bound, and quantum cryptanalysis takes it seriously.\nIt does not forbid teleportation, which is the confusion the name invites. Quantum teleportation moves a state from Alice to Bob using a shared Bell pair plus two classical bits. It does not violate no-cloning because the original is destroyed by Alice\u0026rsquo;s measurement — it is a move, not a copy, and at no instant do two copies exist. It also does not exceed lightspeed, because the two classical bits travel by ordinary channel and Bob\u0026rsquo;s state is useless until they arrive.\nIt does not forbid copying classical information encoded in quantum states. If your bits are encoded in orthogonal basis states and you know the basis, you can measure and re-prepare freely. Every quantum computer\u0026rsquo;s classical readout does this. No-cloning bites only on genuinely unknown superpositions.\nIt does not forbid quantum error correction, despite forbidding the obvious approach. Shor, Steane, and surface codes work by encoding logical information in entangled subspaces and measuring syndromes — operators that reveal which error occurred without revealing the encoded state. The threshold theorem says arbitrary-length quantum computation is possible with imperfect components. The field\u0026rsquo;s founding achievement was routing around this theorem, and it took a decade.\nBoundary The precise scope is narrower than the slogan:\nUnitary evolution only. Measurement is not unitary, and does destroy states rather than copying them, so it is no loophole. Arbitrary unknown states only. Known states are freely reproducible. Orthogonal sets are freely copyable. The classical world is the copyable subset. Exact copying only. Fidelity up to $5/6$ is achievable, and the exact optimum is known for every input ensemble. Pure states, in the original statement. The mixed-state generalization is the no-broadcasting theorem: a set of mixed states can be broadcast if and only if they commute. Same structure, same conclusion. There is a companion worth knowing: no-deleting. Given two identical copies of an unknown state, you cannot delete one and leave the other intact. Quantum information can be neither created nor destroyed by local unitary means, only moved. Taken together the two theorems say quantum information is conserved in a way classical information is not, and that conservation is what makes it a genuinely different resource rather than classical bits with extra steps.\nRead next ","permalink":"https://cs.lozic.me/posts/t110-the-no-cloning-theorem/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eYou have spent this whole part learning that information wants to be copied.\nEntropy bounds compression (T018), Huffman hits the bound (T020), redundancy\nbuys error correction (T021). Every result so far treats \u0026ldquo;make another copy\u0026rdquo; as\nthe free operation — the thing you do without thinking, the reason \u003ccode\u003ecp\u003c/code\u003e has no\ninteresting failure modes.\u003c/p\u003e","title":"The No-Cloning Theorem"},{"content":"Symptom Your dashboard shows CPU utilization averaging 40%, comfortably under budget. Users are reporting timeouts. You add more logging and the average stays at 40%.\nEventually someone captures a one-second trace and finds the truth: the service spikes to 100% for eight seconds out of every ninety, and your monitoring samples every sixty seconds. The spikes are real, periodic, and invisible.\nWorse, when you plot the sampled series over an hour, you see a clean slow oscillation with a three-minute period that does not exist in the system at all. It is an artifact. Your team spends two days looking for a three-minute cycle in the workload.\nHere is that number, because it is not intuitive. A 90-second-period signal sampled every 60 seconds appears at frequency $|1/90 - 1/60| = 1/180$ Hz, a 180-second period. The graph shows a real, smooth, persistent three-minute wave that corresponds to nothing.\nSame phenomenon, other guises: wagon wheels turning backwards on film, moiré patterns on a photographed screen, jagged edges before antialiasing, a 15 kHz whine appearing in a downsampled audio file. These look like unrelated rendering bugs. They are one theorem.\nStatement A signal is bandlimited to $W$ hertz if its Fourier transform vanishes above $W$ — it contains no frequency component faster than $W$.\nA signal bandlimited to $W$ hertz is completely determined by its samples taken at any rate $f_s \u0026gt; 2W$, and can be reconstructed exactly. Below that rate, information is destroyed and no reconstruction can recover it.\nIf $x(t)$ has $X(f) = 0$ for $|f| \\ge W$, then\n$$x(t) = \\sum_{n=-\\infty}^{\\infty} x\\!\\left(\\tfrac{n}{f_s}\\right) \\operatorname{sinc}\\!\\left(f_s t - n\\right), \\qquad f_s \u003e 2W.$$ Two claims, and both are surprising in opposite directions.\nThe positive claim: countably many samples reconstruct a continuous function exactly. Not approximately. Every value between the samples is determined.\nThe negative claim: below $2W$, the loss is irreversible. High frequencies do not vanish, they fold down and masquerade as low ones. The corrupted component is indistinguishable from a genuine low-frequency signal, so no post-processing separates them.\nThe rate $2W$ is the Nyquist rate. Half the sampling rate, $f_s/2$, is the Nyquist frequency — the highest frequency you can represent. These get swapped constantly, so: CD audio samples at 44.1 kHz, giving a Nyquist frequency of 22.05 kHz, which covers human hearing to 20 kHz with a 2050 Hz guard band for the filter to roll off in.\nArgument Why sampling causes folding. Multiplying $x(t)$ by an impulse train of period $1/f_s$ is, in the frequency domain, convolution with an impulse train of spacing $f_s$. Convolving a spectrum with impulses at multiples of $f_s$ produces copies of the spectrum centred at every multiple of $f_s$:\n$$X_s(f) = f_s \\sum_{k=-\\infty}^{\\infty} X(f - k f_s).$$That is the entire mechanism. Sampling in time replicates the spectrum in frequency.\nNow the geometry. Each copy occupies a band of width $2W$ around its centre. If $f_s \u0026gt; 2W$, consecutive copies do not touch, and an ideal low-pass filter keeps the original and discards the rest, recovering $X(f)$ exactly. Take the inverse transform of that rectangular filter and you get the sinc function in the formula above.\nIf $f_s \u0026lt; 2W$, the copies overlap. In the overlap region the sampled spectrum is a sum of contributions from the original band and its neighbour, and addition is not invertible. Given the sum, no filter recovers the summands. That is aliasing, and its irreversibility is arithmetic rather than technological.\nThe folding formula. A component at $f_0 \u0026gt; f_s/2$ appears at $|f_0 - k f_s|$ for whichever integer $k$ lands it in $[0, f_s/2]$. Sampling at 44.1 kHz, a 30 kHz component appears at $|30000 - 44100| = 14.1$ kHz — squarely audible, and now permanently part of the signal. A 25 kHz component lands at 19.1 kHz. An ultrasonic component you never intended to record becomes a tone you cannot remove.\nThis is why every ADC has an anti-aliasing filter in front of it, in analog hardware, before the sampler. It must be analog and it must come first, because after sampling the damage is arithmetic. A digital filter applied afterwards removes the 14.1 kHz alias and the genuine 14.1 kHz content together, since by then they are the same numbers.\nCounting, which is Shannon\u0026rsquo;s framing and the one worth keeping. A signal bandlimited to $W$ over $T$ seconds has about $2WT$ degrees of freedom. The theorem says a sample rate of $2W$ supplies exactly that many numbers per second: enough, and not more than enough. Sampling faster is redundancy; sampling slower loses dimensions. It is a dimension count, and the reason it connects to T018 is that both are statements about how many bits a signal actually contains as opposed to how many you chose to write down.\nForbids Recovering detail above half your sampling rate. Not with a better algorithm, not with a neural network, not ever, because the information is not present in the samples.\nRemoving aliasing after the fact. Once folded, an aliased component is arithmetically identical to genuine content at that frequency.\nDetecting events shorter than twice your monitoring interval. A 60-second scrape cannot see a 30-second outage. Not \u0026ldquo;might miss\u0026rdquo; — cannot see. This governs Prometheus scrape intervals, CloudWatch periods, and every APM sampling config, and it is why p99 latency computed from 1-minute aggregates is not the p99 your users experienced.\nPerfect real-time reconstruction, since the sinc function has infinite support in both directions. Exact reconstruction requires all samples, including future ones. Every practical reconstructor is a finite approximation, and its error is the gap between the theorem and your hardware.\nDoes not forbid It does not forbid sampling below $2W$ when the signal is sparse, and this is the misreading with the biggest modern consequences. Compressed sensing reconstructs signals from far fewer samples than Nyquist demands, provided the signal is sparse in some basis. This is not a violation: Nyquist assumes only bandlimiting, while compressed sensing assumes much more, and stronger assumptions buy stronger conclusions. MRI scanners ship this — modern scans are several times faster than Nyquist-rate acquisition would allow, and the reconstruction is a convex optimization rather than a sinc sum. The Event Horizon Telescope black hole image came from radically sub-Nyquist sampling plus sparsity priors.\nIt does not forbid undersampling on purpose. Bandpass sampling (or undersampling) deliberately aliases a high-frequency band down to baseband, using the fact that if you know which copy you are looking at, folding is invertible. Software-defined radios like the RTL-SDR and HackRF use this routinely to capture GHz signals with MHz-rate converters. The requirement is $f_s \u0026gt; 2B$ for bandwidth $B$, not $2f_{\\max}$.\nIt does not mean higher sample rates sound better. 192 kHz audio captures frequencies to 96 kHz, which no human hears and most speakers cannot produce. The genuine arguments for high-rate audio are about filter design headroom and processing during production, not about the captured content. Nyquist says 44.1 kHz already covers human hearing exactly, and treating the theorem as approximate is where a lot of audiophile marketing lives.\nIt does not apply to non-bandlimited signals, which is every real signal, strictly speaking. A signal of finite duration cannot be perfectly bandlimited — the two are Fourier-incompatible. So all real sampling has some aliasing, kept below the noise floor by filtering rather than eliminated. The theorem describes an idealization you approach, not a state you reach.\nIt does not require uniform sampling. Non-uniform and randomized sampling schemes achieve the same reconstruction at the same average rate, and randomized sampling has the useful property of turning aliases into broadband noise rather than coherent phantom signals. This is a real fix for the monitoring case: jitter your scrape interval and the phantom three-minute wave becomes noise you can see through instead of a pattern you chase.\nBoundary The theorem\u0026rsquo;s assumptions are where all the engineering lives:\nPerfect bandlimiting — impossible; approximated by analog anti-aliasing filters with finite roll-off, which is what the guard band is for. Infinite samples — impossible; approximated by windowed sinc interpolation, and the window\u0026rsquo;s shape is the entire subject of filter design. Exact sample values — impossible; quantization adds noise, and the $\\approx 6.02b + 1.76$ dB signal-to-noise ratio for $b$ bits is the other half of digital audio\u0026rsquo;s design. Uniform timing — approximated; clock jitter converts to noise proportional to signal frequency, which is why high-frequency ADCs care about clock quality far more than low-frequency ones do. For monitoring specifically, the practical rule falls straight out: to detect an event of duration $d$, sample at least every $d/2$. To characterize its shape, much faster. And if you cannot sample that fast, do not interpolate the gaps — switch to event-based instrumentation, where the system reports occurrences rather than being asked. Counters and histograms are aliasing-immune because they are not samples of a continuous signal at all, and that is the real reason they are preferred over gauges for anything bursty.\nRead next ","permalink":"https://cs.lozic.me/posts/t106-the-nyquist-shannon-sampling-theorem/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eYour dashboard shows CPU utilization averaging 40%, comfortably under budget.\nUsers are reporting timeouts. You add more logging and the average stays at 40%.\u003c/p\u003e\n\u003cp\u003eEventually someone captures a one-second trace and finds the truth: the service\nspikes to 100% for eight seconds out of every ninety, and your monitoring\nsamples every sixty seconds. The spikes are real, periodic, and invisible.\u003c/p\u003e","title":"The Nyquist–Shannon Sampling Theorem"},{"content":"Symptom A node in your cluster is not down. It is worse than down.\nIts disk is returning corrupted blocks that pass the checksum because the checksum is corrupted too. Or a NIC with a firmware bug is duplicating and reordering frames. Or a bad memory module flipped a bit in a config value and the node now believes it owns a shard it does not. Or someone has root on one machine.\nEvery failure model you have used so far assumes nodes stop cleanly. FLP (T062) assumes crash faults. Raft assumes crash faults. Your health checks assume a sick node goes quiet. This node is not quiet. It is actively saying wrong things, confidently, to different peers, possibly saying different wrong things to different peers.\nThe operational signature is distinctive: the cluster disagrees about what happened, every node\u0026rsquo;s logs are internally consistent, and no two nodes\u0026rsquo; logs agree. There is no crash to point at. And a majority vote does not save you, because the faulty node votes too.\nStatement A commanding general must send an order — attack or retreat — to $n-1$ lieutenants. Some generals, possibly including the commander, are traitors who may say anything to anyone, including different things to different people.\nThe protocol must guarantee:\nIC1. All loyal lieutenants obey the same order. IC2. If the commander is loyal, every loyal lieutenant obeys the commander\u0026rsquo;s order. With $f$ traitors, a solution using oral messages exists if and only if there are at least $3f+1$ generals.\nByzantine agreement among $n$ processes tolerating $f$ arbitrary faults is solvable with unauthenticated messages if and only if $n \\ge 3f+1$. With unforgeable signatures, the bound relaxes to $n \\ge f+1$ for agreement, and $n \\ge 2f+1$ when combined with a liveness requirement in the usual partially synchronous setting.\nBoth directions matter. There is a protocol at $3f+1$, and there is no protocol at $3f$. The bound is tight, so this is not a limit of current technique.\nConcretely: tolerating one liar takes four nodes, two takes seven, three takes ten. Not three, six, and nine. The extra node in each case is the entire content of the theorem.\nArgument The impossibility, for $n=3$, $f=1$. This case carries the whole idea.\nThree generals: commander $C$ and lieutenants $L_1$, $L_2$. Exactly one may be a traitor. Consider two scenarios.\nScenario A. $C$ is loyal and sends \u0026ldquo;attack\u0026rdquo; to both. $L_2$ is a traitor. $L_1$ receives \u0026ldquo;attack\u0026rdquo; from $C$, and then receives from $L_2$ the report \u0026ldquo;the commander told me retreat.\u0026rdquo; $L_1$ sees a disagreement and knows one of $C$ or $L_2$ is lying, but not which.\nScenario B. $C$ is the traitor, sending \u0026ldquo;attack\u0026rdquo; to $L_1$ and \u0026ldquo;retreat\u0026rdquo; to $L_2$. $L_2$ is loyal and truthfully reports \u0026ldquo;the commander told me retreat.\u0026rdquo;\nNow look at what $L_1$ actually received. In both scenarios: \u0026ldquo;attack\u0026rdquo; from the commander, and \u0026ldquo;$L_2$ says the commander told me retreat.\u0026rdquo; The two scenarios are indistinguishable to $L_1$. Every bit it has is identical.\nSo $L_1$ must act the same way in both. In Scenario A, IC2 forces $L_1$ to attack, since the commander is loyal. So $L_1$ attacks in Scenario B too.\nApply the mirror argument to $L_2$ in the scenario where $C$ is loyal and sends \u0026ldquo;retreat,\u0026rdquo; with $L_1$ as the traitor. By symmetry, $L_2$ retreats in Scenario B.\nBut Scenario B has both lieutenants loyal, and they have done different things. IC1 is violated, and no protocol avoids it, because the violation came from indistinguishability rather than from any choice the protocol made. $\\blacksquare$\nWhy the general bound is $3f+1$. The scaling argument is a simulation. If a protocol existed for $n \\le 3f$, partition the generals into three groups of at most $f$ each and have three \u0026ldquo;super-generals\u0026rdquo; each simulate one group. A single traitorous super-general corresponds to at most $f$ real traitors, which the hypothetical protocol tolerates — and you have solved the three-general problem with one traitor, which we just proved impossible.\nWhere the number comes from, intuitively. You have $n$ nodes and $f$ are lying. To reach a decision, you must be able to gather a quorum whose honest members alone are a majority of the honest nodes, while also allowing that $f$ nodes may be unreachable. You wait for $n-f$ responses, since $f$ may never answer. Of those $n-f$, up to $f$ may be lies, leaving $n-2f$ honest. For the honest votes to outnumber the lies you need $n - 2f \u0026gt; f$, so $n \u0026gt; 3f$. Each $f$ is subtracted for a distinct reason — the ones you cannot wait for, and the ones who lie to you — and the third comes from needing a strict majority among the remainder.\nThe signature escape. The impossibility rests on $L_1$ being unable to verify $L_2$\u0026rsquo;s claim about what $C$ said. Give every general an unforgeable signature, and $L_2$ must produce $C$\u0026rsquo;s signed order. A traitorous $L_2$ cannot fabricate one, and a traitorous $C$ that signs two contradictory orders produces a transferable proof of its own treachery. The bound collapses to $n \\ge f+1$ for agreement alone. This is why blockchains, which have signatures by construction, work with quorums that would otherwise be impossible, and why PBFT-family protocols still use $3f+1$: they want liveness under partial synchrony, not just agreement.\nForbids Byzantine fault tolerance in a three-node cluster. A three-node etcd or ZooKeeper survives one crash. It does not survive one node lying, and the distinction is invisible in normal operation and total when it matters.\nMajority voting as a Byzantine defence. With $f$ liars among $2f+1$ nodes, a \u0026ldquo;majority\u0026rdquo; of $f+1$ can consist of $f$ liars plus one honest node. The vote returns a wrong answer with full confidence.\nDetecting a Byzantine node from its behaviour alone in a general system. A node that lies consistently and plausibly is indistinguishable from a node that is correct in a different scenario. That is exactly the indistinguishability the proof used.\nUnauthenticated agreement at $n \\le 3f$, unconditionally. No cleverness, no extra rounds, no better cryptographic hash.\nDoes not forbid It does not mean you need BFT, and this is the misreading that wastes the most engineering effort. Byzantine fault tolerance costs $O(n^2)$ messages per decision — a 100-node PBFT cluster exchanges 10,000 messages to agree on one value — and it is the right tool only when you genuinely have mutually distrusting parties. Google, Amazon, and Meta run planetary infrastructure on crash-fault-tolerant consensus. Inside a trusted datacenter the realistic Byzantine faults are corruption and bugs, and those are handled far more cheaply by checksums, ECC memory, and end-to-end verification than by a consensus protocol. Bitcoin, Ethereum, Tendermint, and Diem need BFT because their participants are strangers, and that is the actual criterion.\nIt does not forbid tolerating more than $n/3$ failures when they are not Byzantine. A protocol can tolerate $f$ Byzantine faults and additional crash faults simultaneously under hybrid fault models, because crashes are a strictly easier failure to survive.\nIt does not apply to synchronous systems with signatures. With both, Dolev and Strong showed agreement is possible with any $n \u0026gt; f$, in $f+1$ rounds. The $3f+1$ bound is specifically the price of asynchrony without authentication, and stating it without those qualifiers is how it gets misapplied.\nIt does not mean cheap defences are worthless, which is the practical point most teams miss. Most real \u0026ldquo;Byzantine\u0026rdquo; faults in a datacenter are not adversarial. ZFS and Btrfs checksums catch silent disk corruption. ECC memory catches bit flips. TLS catches on-path tampering. Amazon\u0026rsquo;s S3 does end-to-end checksums across every hop. Each of these eliminates a class of arbitrary fault at a fraction of BFT\u0026rsquo;s cost, and stacking them is why crash-fault-tolerant systems work in practice despite hardware being imperfect.\nIt does not say the faulty node must be malicious. \u0026ldquo;Byzantine\u0026rdquo; means arbitrary, not hostile. A memory-corrupted node with no attacker anywhere near it is Byzantine in the technical sense, and this is the far more common case. Treating the model as being about attackers causes people to dismiss it in environments where it genuinely applies.\nBoundary The $3f+1$ bound holds for asynchronous or partially synchronous systems without authentication. Each assumption you relax moves the bound:\nAdd signatures. $n \\ge f+1$ for agreement with synchrony; $n \\ge 2f+1$ in the partially synchronous protocols people actually deploy. Signatures are cheap now, which is why every modern BFT protocol assumes them. Add synchrony. Dolev-Strong reaches agreement in $f+1$ rounds with any $n \u0026gt; f$, given signatures and a known message-delay bound. Add randomization. Randomized BFT protocols circumvent FLP\u0026rsquo;s termination problem the same way crash-fault protocols do, without changing the $3f+1$ resilience bound. Weaken the requirement. Bitcoin does not achieve agreement in this sense at all. It achieves eventual probabilistic agreement under an honest-majority hashpower assumption, which is a different theorem with a different bound, and the confirmation-depth convention exists precisely because the guarantee is probabilistic. What remains true throughout: the price of arbitrary faults is at least one extra node per fault beyond what crashes cost, and that extra node is buying you the ability to tell a liar from a silence. Whether you should pay it is a question about your trust boundary, not your uptime target.\nRead next ","permalink":"https://cs.lozic.me/posts/t063-the-byzantine-generals-problem/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eA node in your cluster is not down. It is worse than down.\u003c/p\u003e\n\u003cp\u003eIts disk is returning corrupted blocks that pass the checksum because the\nchecksum is corrupted too. Or a NIC with a firmware bug is duplicating and\nreordering frames. Or a bad memory module flipped a bit in a config value and\nthe node now believes it owns a shard it does not. Or someone has root on one\nmachine.\u003c/p\u003e","title":"The Byzantine Generals Problem"},{"content":"Symptom Your cluster of five nodes elects a leader. It works. It has worked for a year.\nThen one afternoon a garbage collection pause on the leader runs for twelve seconds. The followers time out, start an election, and elect a new leader. The old leader wakes up, has no idea it was deposed, and keeps serving writes. For a few hundred milliseconds you have two leaders, and if your fencing is not airtight, two writes that should have been ordered land in an order nobody chose.\nYou fix it by raising the timeout. Now failover takes thirty seconds instead of five, and an actual crash costs you thirty seconds of downtime. So you lower it. Now spurious elections come back.\nYou are tuning a dial with a bad outcome at each end, and there is no setting that makes both problems go away. That is not a gap in your understanding of the configuration. It is FLP.\nThe question the timeout is trying to answer is: is that node dead, or slow? And in an asynchronous system, that question has no answer. A message that has not arrived yet and a message that will never arrive look identical from where you are standing, and no amount of waiting distinguishes them, because waiting longer only ever rules out \u0026ldquo;arrived by now.\u0026rdquo;\nStatement The setting is the asynchronous model. Messages are eventually delivered, but with no bound on how long that takes. Processes have no synchronized clocks and no timeouts, because a timeout is a bound and there are none. Exactly one process may fail, and only by crashing — it stops, silently, and never lies.\nConsensus requires three properties: agreement (no two correct processes decide differently), validity (the decided value was proposed by someone), and termination (every correct process eventually decides).\nIn an asynchronous system with even one possible crash failure, no deterministic protocol solves consensus.\nThere is no deterministic algorithm that satisfies agreement, validity, and termination in the asynchronous message-passing model tolerating a single crash fault.\nNote how weak the failure model is. One crash. No Byzantine behaviour, no message loss, no network partition, no corruption, no lying. The channel is reliable. This is much weaker than Two Generals (T061), which needed message loss, and the impossibility still bites.\nAnd note which property fails. Agreement and validity are achievable — a protocol that never decides anything satisfies both trivially. It is termination that cannot be guaranteed. FLP does not say your consensus protocol will produce a wrong answer. It says there is an execution in which it produces no answer, forever.\nArgument The proof is a strategy for an adversary who controls only message scheduling, not content, and it has two moves.\nConfigurations and valence. A configuration is the full global state: every process\u0026rsquo;s local state plus every message in flight. Call a configuration 0-valent if every reachable decision from it is 0, 1-valent if every reachable decision is 1, and bivalent if both outcomes remain reachable. Bivalent means undecided in the strongest sense: the outcome is not yet determined by the state.\nMove one: some initial configuration is bivalent.\nLine up the $2^n$ initial configurations so that adjacent ones differ in exactly one process\u0026rsquo;s input. The all-zeros configuration must decide 0 by validity, and the all-ones must decide 1. So somewhere along the line there are two adjacent configurations $C$ and $C\u0026rsquo;$, differing only at process $p$, with different decisions.\nNow run the execution in which $p$ crashes immediately, at the very start. The remaining processes cannot see $p$\u0026rsquo;s input, so $C$ and $C\u0026rsquo;$ are indistinguishable to all of them, and they must reach the same decision. But $C$ decides 0 and $C\u0026rsquo;$ decides 1. Contradiction — unless at least one of them was bivalent all along.\nThis is where the \u0026ldquo;one crash fault\u0026rdquo; is spent. It is not used to disrupt the protocol; it is used to make two inputs indistinguishable.\nMove two: from any bivalent configuration, the adversary can reach another bivalent configuration.\nLet $C$ be bivalent and let $m$ be any message in flight. The adversary wants to show it can delay $m$, let the protocol take some steps, and still be bivalent when $m$ is finally delivered.\nLet $\\mathcal{C}$ be the set of configurations reachable from $C$ without delivering $m$, and let $\\mathcal{D} = { e(m)(E) : E \\in \\mathcal{C} }$ be what you get by delivering $m$ last. Suppose for contradiction that every configuration in $\\mathcal{D}$ is univalent. Since $C$ is bivalent, $\\mathcal{D}$ contains both a 0-valent and a 1-valent configuration, so somewhere in $\\mathcal{C}$ there are neighbours $E_0$ and $E_1$, one step apart, whose $m$-deliveries have opposite valence.\nSay $E_1 = e\u0026rsquo;(E_0)$ where $e\u0026rsquo;$ is the delivery of some message $m\u0026rsquo;$ to process $q$. Two cases:\nIf $m\u0026rsquo;$ goes to a different process than $m$, the two steps commute — they touch disjoint local states — so delivering $m$ then $m\u0026rsquo;$ gives the same configuration as $m\u0026rsquo;$ then $m$. But those are claimed to be 0-valent and 1-valent. Contradiction.\nIf $m\u0026rsquo;$ goes to the same process $q$ as $m$, consider the execution where $q$ crashes right after $E_0$ and takes no further steps. The remaining processes must still decide, by termination, reaching some configuration $A$. But $A$ is reachable both from $e(m)(E_0)$ and from $e(m)(E_1)$ by having $q$ crash, and those have opposite valence. Contradiction again.\nSo $\\mathcal{D}$ contains a bivalent configuration. $\\blacksquare$\nPutting it together. Start bivalent by move one. By move two, the adversary can always delay the oldest undelivered message just long enough to land in another bivalent configuration, then deliver it. Every message is eventually delivered, so the execution is legal in the asynchronous model. And the system is bivalent at every step, so it never decides.\nThe adversary never drops a message, never corrupts one, and crashes at most one process. It only ever chooses the order. That is the whole attack.\nWhat the proof is really about. Every contradiction above came from indistinguishability: two different global situations that some process cannot tell apart, forcing it to act identically in both. It is the same engine as Two Generals. The difference is that FLP needs only reordering where Two Generals needed loss, which makes FLP the stronger statement about a weaker world.\nForbids A consensus protocol with a guaranteed termination bound in an asynchronous network. Not a slow one, not an unlikely one — no bound exists.\nA perfect failure detector. If you could reliably distinguish crashed from slow, consensus would be solvable, so FLP says you cannot build one. Every health check, heartbeat, and liveness probe you have ever configured is an unreliable failure detector making a guess with a false-positive rate.\nAtomic broadcast, leader election, and distributed locking with guaranteed termination, since each is equivalent to consensus. If your service discovery promises a bounded time to converge on a new leader in an asynchronous network, it is promising something no algorithm has.\nDeterministic termination for blockchain consensus in an asynchronous network. Tendermint and HotStuff are explicit about this: they guarantee safety always and liveness only under partial synchrony.\nDoes not forbid It does not forbid Paxos, Raft, or ZooKeeper, and \u0026ldquo;FLP proves consensus is impossible so Raft must be broken\u0026rdquo; is the single most common misreading of this result. These protocols are always safe — they never violate agreement, in any execution, including the pathological FLP one. What they give up is guaranteed liveness. Raft in an unlucky execution keeps holding split elections and never commits. That is not a bug in Raft; it is Raft being correct about a world where progress cannot be guaranteed. etcd, Consul, and ZooKeeper all ship this tradeoff and run global infrastructure on it.\nIt does not forbid termination with probability 1. FLP rules out deterministic protocols. Ben-Or\u0026rsquo;s randomized consensus (1983) terminates with probability 1, and randomized leader election in Raft — that jittered election timeout — is the same trick in production. The probability of never terminating is zero, but there is no bound on when, so the theorem is untouched. This is the most practically important escape and the most frequently overlooked.\nIt does not apply to the partially synchronous model, which is where every real system lives. Dwork, Lynch, and Stockmeyer showed in 1988 that if the network is eventually synchronous — bounds exist, you just do not know them or when they start holding — consensus becomes solvable. Real networks behave this way: mostly well-behaved, occasionally awful, eventually well-behaved again. Raft and Paxos are designed exactly for this model, and their liveness argument is \u0026ldquo;once the network settles, we make progress.\u0026rdquo;\nIt does not mean timeouts are wrong. A timeout is not a failure detector; it is a suspicion generator, and protocols built on it treat suspicion as a hint rather than a fact. Raft\u0026rsquo;s key design property is that a wrong suspicion costs you an election, not correctness. Understanding that timeouts affect performance and never safety is the difference between tuning a cluster and being afraid of it.\nIt does not apply to a synchronous system. With a known bound on message delay and clock drift, a missing message is a crash, and consensus is solvable with a simple round-based protocol. Real-time systems, avionics buses, and lockstep replication live here — at the cost of enforcing the bound physically.\nIt says nothing about Google Spanner being impossible. Spanner uses TrueTime, atomic clocks and GPS giving a bounded uncertainty interval, and then simply waits out the interval. That is buying partial synchrony with hardware. The commit-wait latency is the theorem\u0026rsquo;s price, paid in milliseconds.\nBoundary The exits are exactly four, and every production consensus system uses one or more:\nRandomization. Termination with probability 1. Ben-Or, and Raft\u0026rsquo;s randomized election timeouts. Partial synchrony. Assume bounds eventually hold. Paxos, Raft, PBFT, Tendermint. This is the mainstream answer. Failure detectors. Chandra and Toueg showed $\\Diamond W$ — eventually weak, meaning eventually some correct process is never suspected — is the weakest detector sufficient for consensus. This is the same assumption as partial synchrony in different clothing, and it tells you the minimum you must assume. Buy synchrony. TrueTime, atomic clocks, dedicated networks. What survives all four: in any given moment, you cannot know whether a silent node is dead or slow, and any action you take is a bet. The right engineering response is not to make the bet more accurate but to make being wrong survivable — fencing tokens, leases with generation numbers, idempotent operations. That is the same conclusion T061 reached from a different direction, and the agreement between them is not a coincidence.\nRead next ","permalink":"https://cs.lozic.me/posts/t062-flp-impossibility/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eYour cluster of five nodes elects a leader. It works. It has worked for a year.\u003c/p\u003e\n\u003cp\u003eThen one afternoon a garbage collection pause on the leader runs for twelve\nseconds. The followers time out, start an election, and elect a new leader. The\nold leader wakes up, has no idea it was deposed, and keeps serving writes. For a\nfew hundred milliseconds you have two leaders, and if your fencing is not\nairtight, two writes that should have been ordered land in an order nobody\nchose.\u003c/p\u003e","title":"FLP Impossibility"},{"content":"Symptom You took the lesson from T058 and stopped trusting wall clocks. Every event now carries a Lamport timestamp, and your writes are ordered by it. The vanishing updates stopped.\nThen a different bug arrives. A user edits their profile from a phone and a laptop within the same minute. The two edits never saw each other, so the correct behaviour is to detect a conflict and either merge or ask. Instead one silently overwrote the other, because the laptop\u0026rsquo;s write happened to carry Lamport timestamp 47 and the phone\u0026rsquo;s carried 44, and 47 \u0026gt; 44, so the system concluded the laptop\u0026rsquo;s write came after and knew about the phone\u0026rsquo;s.\nIt did not. The two are concurrent. Nothing about them is ordered. Your clock manufactured an order that the underlying causality does not support, and then your last-write-wins rule acted on that fiction and threw away real data.\nHere is the sharper version of the problem. Given two Lamport timestamps $L(a) \u0026lt; L(b)$, what can you conclude?\nAlmost nothing. Either $a \\to b$, or $a$ and $b$ are concurrent. The one thing you can conclude is the contrapositive: if $L(a) \\ge L(b)$ then $a$ definitely did not happen before $b$. That is a real fact and it is much weaker than what your code assumed.\nThe symptom, in one line: Lamport clocks can tell you that an order exists, but not that one doesn\u0026rsquo;t. And detecting concurrency, not detecting order, is what conflict resolution actually needs.\nStatement A vector clock fixes exactly this, and the fix is complete.\nEach of $n$ processes keeps a vector $V$ of $n$ counters. Process $i$ increments $V[i]$ on each local event, attaches $V$ to every message, and on receipt sets $V[k] \\leftarrow \\max(V[k], V_{\\text{msg}}[k])$ for every $k$, then increments its own entry.\nDefine $V(a) \\le V(b)$ to mean $V(a)[k] \\le V(b)[k]$ for all $k$, and $V(a) \u0026lt; V(b)$ to mean $V(a) \\le V(b)$ with at least one strict inequality.\n$a \\to b$ if and only if $V(a) \u0026lt; V(b)$. Two events are concurrent exactly when neither vector dominates the other.\nVector clocks characterize the happens-before relation: the map from events to vectors under the product order is an order-isomorphism onto its image. Lamport clocks are only order-preserving, giving the one-way implication $a \\to b \\implies L(a) \u0026lt; L(b)$.\nThe word doing the work is iff. Lamport gives you an implication; vector clocks give you an equivalence. That upgrade is the entire content of the result, and it is what makes concurrency detectable.\nAnd there is a price, which is also a theorem: any timestamping scheme that characterizes happens-before in a system of $n$ processes needs $n$ components. Charron-Bost proved this in 1991. The $O(n)$ metadata is not an artifact of Fidge\u0026rsquo;s particular construction, it is the information-theoretic cost of the guarantee. You cannot be clever and get it in $O(\\log n)$.\nArgument Forward direction: $a \\to b \\implies V(a) \u0026lt; V(b)$.\nBy induction over the happens-before relation\u0026rsquo;s three generating cases. If $a$ and $b$ are consecutive events in the same process, $b$\u0026rsquo;s vector is $a$\u0026rsquo;s with one entry incremented, so $V(a) \u0026lt; V(b)$. If $a$ is a send and $b$ is the matching receive, the merge takes a componentwise max with $V(a)$, so every entry is at least as large, and the local increment makes one strictly larger. Transitivity of the componentwise order closes the induction.\nReverse direction: $V(a) \u0026lt; V(b) \\implies a \\to b$. This is the direction Lamport clocks fail, and the proof is short.\nLet $a$ be an event at process $i$. By construction $V(a)[i]$ is the count of events at process $i$ up to and including $a$, and this entry is only ever incremented by process $i$ itself. Every other process learns about it solely through the max-merge on message receipt.\nNow suppose $V(a) \\le V(b)$, so in particular $V(b)[i] \\ge V(a)[i]$. If $b$ is also at process $i$, then $b$ is at least as late in the local sequence, so $a \\to b$ or $a = b$. If $b$ is at some other process $j$, then process $j$ can only have $V[i]$ that large by having received a chain of messages that ultimately originates at process $i$ at or after event $a$. That chain is a happens-before path from $a$ to $b$. $\\blacksquare$\nThe mechanism, stated plainly: entry $i$ of a vector is a claim about what process $i$ has done, and the only route by which that claim can travel to another process is a message. So a large entry is evidence of a causal path, and the vector is a record of the frontier of everything you have transitively heard about.\nConcurrency, then, is the failure of both comparisons. If $V(a) \\not\\le V(b)$ and $V(b) \\not\\le V(a)$, each event knows something the other does not, so no causal path runs in either direction. That is exactly the conflict your last-write-wins rule was papering over.\nA trace, so the vectors are concrete. Three processes, $A$, $B$, $C$.\n$A$ writes. $V_A = (1,0,0)$. $A$ sends to $B$. $B$ receives and writes: $V_B = (1,1,0)$. Independently, $C$ writes, having heard from nobody: $V_C = (0,0,1)$. Compare $B$\u0026rsquo;s write and $C$\u0026rsquo;s write. $(1,1,0)$ versus $(0,0,1)$: the first is larger in components 1 and 2, the second is larger in component 3. Neither dominates, so they are concurrent, correctly. Compare $A$\u0026rsquo;s write with $B$\u0026rsquo;s: $(1,0,0) \\le (1,1,0)$ and they differ, so $A \\to B$, correctly. Lamport clocks would have given these events the scalars 1, 2, and 1, and the comparison between $B$ at 2 and $C$ at 1 would have reported a spurious order.\nWhat causal consistency then means. A store is causally consistent if, whenever $a \\to b$, every process observes $a$ before $b$. Concurrent writes may be observed in different orders by different processes, and that is permitted. This is the strongest consistency model achievable without giving up availability during a partition, which is why it sits exactly where it does in the CAP landscape (T066).\nForbids A single scalar that detects concurrency. By Charron-Bost\u0026rsquo;s lower bound, no timestamp of dimension less than $n$ characterizes happens-before among $n$ processes. Sequence numbers, hybrid logical clocks, and Lamport clocks all fail this, necessarily and not through poor design.\nLast-write-wins as a correct conflict resolution strategy. LWW does not resolve conflicts, it hides them by picking one. When two writes are concurrent, there is no fact of the matter about which is later, so any rule that names a winner is discarding a real update. This is fine when the data is a cache entry and catastrophic when it is a shopping cart.\nConstant-size causal metadata in a system with unboundedly many writers. If every client is its own actor in the vector, the vector grows without bound. This is a genuine operational problem and the mitigations (below) are all lossy in some direction.\nDetecting causality between events in systems that never exchanged messages. Vector clocks track potential causality through the message graph only. If process $A$ influences process $B$ through a side channel — a shared database, a human reading a dashboard and typing a command, a file on disk — the clocks see independence where a real causal link exists. Lamport is explicit about this: happens-before is about the message-passing structure, and anything outside it is invisible.\nDoes not forbid It does not forbid smaller causal metadata in practice, and the systems that do it are worth naming. Riak\u0026rsquo;s dotted version vectors bound the vector by the number of server replicas, not clients, by having the coordinating server own the entry. A three-replica cluster carries three entries regardless of how many million clients write. Cassandra skips vectors entirely and uses LWW with client timestamps, accepting silent loss as a documented tradeoff. CockroachDB and YugabyteDB use hybrid logical clocks, which are scalars plus bounded physical-clock error, and get most of the ordering benefit for $O(1)$ metadata while explicitly not characterizing concurrency. Each of these is a real engineering answer to the $O(n)$ cost, and each pays somewhere.\nIt does not forbid pruning. Entries for actors that have been inactive longer than a known bound can be dropped, at the cost of occasionally reporting concurrent events as ordered. Riak prunes on both age and vector length. The guarantee degrades gracefully rather than breaking, which is the right shape for a production system.\nIt does not mean causal consistency is enough for everything. It is not. Causal consistency permits two concurrent \u0026ldquo;withdraw\u0026rdquo; operations to both succeed and drive the balance negative, because they are genuinely concurrent and causal consistency has nothing to say about them. If you need a global invariant across concurrent operations, you need consensus (T064), and the CALM theorem (T067) tells you exactly which invariants have this property.\nIt does not require vector clocks specifically, which is the misreading that sends teams down a long implementation road they did not need. If your data type is a CRDT — a G-counter, an OR-set, a LWW-register with a proper lattice — concurrent updates merge deterministically and you never need to detect the conflict because you never need to resolve it. Automerge and Yjs power collaborative editors this way, and Redis CRDTs and Azure Cosmos DB ship it commercially. Detecting concurrency matters only when resolution requires a decision. Design the type so it doesn\u0026rsquo;t, and the whole apparatus becomes unnecessary.\nIt does not mean version vectors and vector clocks are the same thing, and conflating them causes real bugs. A vector clock timestamps events and its entries are per-process. A version vector timestamps replicas of an object and its entries are per-replica. They have identical mechanics and different domains, and code that indexes one by the other\u0026rsquo;s actor set will produce comparisons that are meaningless rather than merely wrong.\nBoundary The $O(n)$ cost is real and it is where all the engineering happens. Concretely, in a 1000-node cluster with 8-byte counters, a full vector is 8000 bytes of metadata per version. For an object whose payload is a 200-byte JSON blob, the causality tracking is forty times the size of the data. This is why nobody runs unbounded vector clocks at scale.\nThe escape routes, ordered by how much they give up:\nBound the actor set. Server-side dotted version vectors: entries scale with replicas (typically 3 to 5), not clients. Nearly free, and the standard answer. Prune stale entries. Bounded size, occasional false ordering. Use a scalar plus bounded clock error. Hybrid logical clocks. $O(1)$, and you lose concurrency detection but keep causal ordering under a real-time bound. Make conflicts impossible. CRDTs. You stop needing the detection at all, at the cost of constraining the data type to something with a merge. What remains, after all of them: causality tracked through message passing only. Any influence that leaves your system and re-enters it — a user, a cron job, an external API — is a causal edge your clocks do not have and cannot infer. And the direction the whole subject is heading is the last bullet: the cheapest concurrency detector is a data type that does not care.\nRead next ","permalink":"https://cs.lozic.me/posts/t059-vector-clocks-and-causal-consistency/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eYou took the lesson from T058 and stopped trusting wall clocks. Every event now\ncarries a Lamport timestamp, and your writes are ordered by it. The vanishing\nupdates stopped.\u003c/p\u003e","title":"Vector Clocks and Causal Consistency"},{"content":"Symptom Two writes to the same key. Which one wins?\nYour first instinct is timestamps: whichever has the later wall-clock time is newer. This works in testing. In production you get a bug report saying an update vanished, and when you dig in, node B\u0026rsquo;s clock was 40 ms behind node A\u0026rsquo;s, so B\u0026rsquo;s later write carried an earlier timestamp and lost. NTP is running. NTP does not make clocks identical, it makes them close, and \u0026ldquo;close\u0026rdquo; is not \u0026ldquo;ordered.\u0026rdquo;\nIt gets worse. Clocks go backwards during NTP corrections, so your monotonically increasing sequence is not. Leap seconds have taken down production systems. Virtual machine migration can jump a clock by seconds.\nThere is no global \u0026ldquo;now\u0026rdquo; you can consult. This is not a limitation of your infrastructure budget. It is a property of distributed systems that no amount of better hardware removes, and once you accept it the question becomes: what ordering can you have?\nStatement Define the happens-before relation $\\to$ on events, as the smallest relation satisfying:\nIf $a$ and $b$ occur on the same process and $a$ precedes $b$, then $a \\to b$. If $a$ is the send of a message and $b$ is its receive, then $a \\to b$. Transitivity: if $a \\to b$ and $b \\to c$ then $a \\to c$. If neither $a \\to b$ nor $b \\to a$, the events are concurrent, written $a \\parallel b$. Happens-before is a partial order, not a total one, and the partiality is the physics, not a defect.\nLamport clocks. Give each process a counter $C$. Increment before each local event. Attach $C$ to every message sent. On receive, set $C \\leftarrow \\max(C, C_{\\text{msg}}) + 1$.\nThe theorem:\n$$a \\to b \\implies C(a) \u003c C(b)$$And the crucial converse failure: $C(a) \u0026lt; C(b)$ does not imply $a \\to b$. Lamport clocks are consistent with causality but do not characterize it. Small timestamps tell you nothing; only the implication in one direction holds.\nArgument Why wall clocks cannot work, in principle. Determining that $a$ happened before $b$ on different machines requires comparing clocks, and comparison requires communication, and communication takes unbounded and variable time. You can bound the error — that is what NTP does, to milliseconds on a LAN — but you cannot eliminate it. For events closer together than the uncertainty, wall-clock comparison is a coin flip. The uncertainty interval is where the ordering lives, and it never shrinks to zero.\nProof that $a \\to b \\implies C(a) \u0026lt; C(b)$. By induction on the derivation.\nSame process: the counter increments before every local event, and never decreases, so a later local event has a strictly larger value.\nSend/receive: if $a$ is the send with $C(a) = t$, the message carries $t$, and the receiver sets $C(b) = \\max(C_{\\text{recv}}, t) + 1 \\ge t + 1 \u0026gt; C(a)$.\nTransitivity: $\u0026lt;$ is transitive, so chaining works.\nEvery happens-before edge is one of the two base cases, so any causal chain gives a strictly increasing sequence of counter values.\nWhy the converse fails, concretely. Two processes that never communicate both increment freely. Process A reaches $C = 5$; process B independently reaches $C = 3$. Nothing connects them, so $3 \u0026lt; 5$ means nothing — those events are concurrent. The order is real but its meaning is one-directional, and treating a Lamport timestamp as a real ordering is the classic misuse.\nWhy the partial order is the right object. Concurrency is not ignorance. It is a positive fact: if $a \\parallel b$, then no observer could have seen $a$ influence $b$ or vice versa, so the system is genuinely free to order them either way. Any total order you impose is a choice, and it must be one you can defend.\nTotal order by tiebreaking. Order by $(C(e), \\text{process id})$. This is a total order consistent with happens-before, which is exactly what state machine replication needs: every replica applies the same operations in the same order, so every replica reaches the same state. That is the foundation the whole consensus literature is built on, and Lamport\u0026rsquo;s paper derives it in the same few pages.\nThe connection to relativity, which is not decoration. Lamport\u0026rsquo;s insight came from special relativity: in spacetime, events outside each other\u0026rsquo;s light cones have no observer-independent order, and different observers disagree about which came first. Distributed systems have the same structure, with message latency playing the role of the speed of light. The impossibility of global time is the same impossibility in both cases, which is why no engineering advance removes it.\nA concrete trace. Three processes, all counters starting at 0. A does a local event ($C_A = 1$) and sends to B ($C_A = 2$, message carries 2). B is at $C_B = 4$ from its own local work, receives, and sets $C_B = \\max(4, 2) + 1 = 5$. Meanwhile C has done five local events and sits at $C_C = 5$ having spoken to nobody.\nNow B\u0026rsquo;s receive and C\u0026rsquo;s fifth event both carry the timestamp 5, and they are genuinely concurrent. Worse, A\u0026rsquo;s send at 2 is causally before B\u0026rsquo;s receive at 5, while C\u0026rsquo;s second event, also at 2, is causally unrelated to anything. The same number means \u0026ldquo;caused by\u0026rdquo; in one place and nothing at all in another, and no inspection of the numbers distinguishes the cases. That is the converse failure, in five lines, and it is worth writing out once because the abstract statement does not produce the same discomfort.\nForbids Ordering distributed events by wall-clock timestamp. Clock skew makes this wrong, silently, and last-write-wins conflict resolution on wall clocks loses data. Cassandra\u0026rsquo;s LWW behaviour is the well-documented example.\nA globally consistent \u0026ldquo;now.\u0026rdquo; No protocol gives every node the same instant. The best available is a bounded interval.\nInferring causality from timestamp comparison. $C(a) \u0026lt; C(b)$ is compatible with concurrency. This is the single most common misuse of Lamport clocks in practice.\nDetecting concurrency with Lamport clocks. You cannot tell $a \\parallel b$ from the counters, because concurrent events still get comparable numbers. That requires vector clocks, and it is exactly why they exist.\nDoes not forbid It does not make NTP or PTP useless. They are essential for log correlation, certificate validity, metrics alignment and human debugging. Millisecond-accurate wall clocks are extremely useful; they are just not an ordering primitive. Using them for one thing and Lamport clocks for the other is the correct architecture.\nIt does not stop bounded-uncertainty clocks from providing real guarantees. Google Spanner\u0026rsquo;s TrueTime is the counterexample everyone should know: with GPS and atomic clocks it reports an interval $[t_{\\text{earliest}}, t_{\\text{latest}}]$ with bounded width, typically single-digit milliseconds, and Spanner simply waits out the uncertainty before committing. That buys externally consistent distributed transactions across continents. The theorem is not violated — Spanner pays for ordering with latency, in the open, rather than pretending clocks agree. AWS Time Sync now offers microsecond-level bounds, so this approach is no longer Google-only.\nIt does not mean hybrid logical clocks are unsound. HLC combines physical time with a logical counter, giving timestamps close to wall clock and consistent with happens-before. CockroachDB, MongoDB and YugabyteDB all use it. You get human-meaningful timestamps without breaking causality, which is usually what you actually wanted.\nIt does not mean last-write-wins is always wrong. For genuinely idempotent or commutative updates — a cache entry, a heartbeat, a presence flag — LWW is correct and cheap. The failure mode is applying it to non-commutative updates on structured data, which is where CRDTs earn their keep.\nIt does not require message passing to be visible. Causality can flow through channels your system does not see — a user reads a value on one screen and types it into another. Lamport clocks track only the causality you instrument, and external causality is invisible to them. This is a real source of anomalies in production and not a theoretical footnote.\nBoundary Vector clocks characterize causality exactly. One counter per process gives $a \\to b \\iff V(a) \u0026lt; V(b)$ componentwise, so concurrency becomes detectable. The cost is $O(n)$ metadata per message, which is what T059 is about. Version vectors and dotted version vectors are the production form, used by Dynamo, Riak and Voldemort, with pruning strategies to bound growth. Consistent snapshots. The Chandy–Lamport algorithm records a globally consistent cut without stopping the system, using exactly this causal structure. Flink\u0026rsquo;s checkpointing is a direct descendant. Where this feeds forward. Total order broadcast is equivalent to consensus, which FLP (T062) proves impossible in an asynchronous system with one crash failure. Lamport clocks give you a consistent order cheaply; agreeing on the order is the expensive problem. Causal ordering is cheaper than total ordering. Delivering messages in an order consistent with happens-before requires only local bookkeeping, and it is what causal consistency in Dynamo-style stores provides. Agreeing on a single total order requires consensus and a majority round-trip, which is orders of magnitude more expensive. Knowing which of the two your application actually needs is usually worth more than any optimization downstream of the choice. The practical rule. Use physical clocks for humans and logical clocks for machines. Any code that compares timestamps from two machines to decide correctness is a bug waiting for the right amount of skew. Read next ","permalink":"https://cs.lozic.me/posts/t058-happens-before-and-lamport-clocks/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eTwo writes to the same key. Which one wins?\u003c/p\u003e\n\u003cp\u003eYour first instinct is timestamps: whichever has the later wall-clock time is\nnewer. This works in testing. In production you get a bug report saying an\nupdate vanished, and when you dig in, node B\u0026rsquo;s clock was 40 ms behind node A\u0026rsquo;s,\nso B\u0026rsquo;s \u003cem\u003elater\u003c/em\u003e write carried an \u003cem\u003eearlier\u003c/em\u003e timestamp and lost. NTP is running.\nNTP does not make clocks identical, it makes them close, and \u0026ldquo;close\u0026rdquo; is not\n\u0026ldquo;ordered.\u0026rdquo;\u003c/p\u003e","title":"Happens-Before and Lamport Clocks"},{"content":"Symptom Your problem is NP-hard, and yet it keeps being easy.\nYou are computing a minimum vertex cover to select monitoring points in a network. Twelve thousand nodes. NP-hard, per T028, so you brace for the worst — and the answer comes back in under a second, every time. The cover is small, around thirty nodes, because the network is sparse and the interesting nodes are few.\nThen your colleague\u0026rsquo;s instance, on a graph a tenth the size, runs for hours.\n\u0026ldquo;NP-hard\u0026rdquo; is not distinguishing between these two cases, and something clearly is. The classification you were taught puts them in the same box. Something about the shape of the instance, not its size, decides which world you are in, and the theory that names that something is the most practically useful refinement of NP-hardness available.\nStatement A parameterized problem takes an instance of size $n$ and a parameter $k$. It is fixed-parameter tractable (FPT) if it is solvable in time\n$$f(k) \\cdot n^{O(1)}$$for some computable $f$. The exponential blowup is confined to $k$, and the dependence on $n$ is polynomial with a degree that does not grow with $k$.\nCompare $n^{O(k)}$, which is the naive brute-force shape (class XP). The distinction is everything. Vertex cover with $k = 20$ on a million-node graph: brute force over all $\\binom{n}{k}$ subsets is about $10^{102}$ operations, while the FPT algorithm at $1.2738^k \\cdot n$ is about 127 million — a second\u0026rsquo;s work. Same problem, same instance, opposite outcomes.\nVertex cover is FPT, in $O(1.2738^k + kn)$.\nClique is not, apparently. Clique parameterized by size is W[1]-complete, and W[1] $\\ne$ FPT is the field\u0026rsquo;s working hypothesis, standing in the same role as P $\\ne$ NP. It follows from the Exponential Time Hypothesis, so the two beliefs are linked.\nSo NP-hardness splits. Some NP-hard problems collapse to easy when a parameter is small, and others do not, and which is which is not visible from the NP-hardness proof.\nArgument Vertex cover is FPT, by bounded search tree. Pick any edge $(u,v)$. Any cover contains $u$ or $v$ — no third option. Branch: recurse having taken $u$ with budget $k-1$, and recurse having taken $v$ with budget $k-1$. Depth $k$, branching factor 2, so at most $2^k$ leaves, each doing $O(n)$ work. That gives $O(2^k n)$ immediately, and refinements on high-degree vertices reduce the base to 1.2738.\nThe key is that the branching is on a bounded local choice, and the budget strictly decreases. The instance size is irrelevant to the recursion depth. That is the FPT pattern in one sentence.\nKernelization, which is the technique you will actually deploy. Preprocess in polynomial time until what remains has size bounded by a function of $k$ alone. For vertex cover, two rules (Buss):\nAny vertex of degree greater than $k$ must be in the cover. Otherwise all its neighbours are, exceeding the budget. Take it, decrement $k$. Delete isolated vertices; they are irrelevant. After exhaustive application, every remaining vertex has degree at most $k$, and $k$ vertices of degree $\\le k$ cover at most $k^2$ edges. If more edges remain, answer no. The instance is now $O(k^2)$ regardless of whether it started with a million vertices. Solve the kernel by brute force if you like.\nKernelization is a theorem about preprocessing — a provable guarantee about how much a polynomial-time cleanup pass can shrink an instance. That is unusual and valuable: it turns \u0026ldquo;run some reduction rules first\u0026rdquo; from folklore into something with a bound attached. And a problem is FPT if and only if it is kernelizable, which is a genuinely surprising equivalence.\nColour coding, for finding small patterns. To find a path of length $k$, randomly colour vertices with $k$ colours. If a $k$-path exists, it is colourful — all distinct colours — with probability $k!/k^k \\ge e^{-k}$. Colourful paths are findable by dynamic programming over colour subsets in $O(2^k \\cdot \\text{poly})$. Repeat $e^k$ times for high probability. Total $O((2e)^k \\cdot \\text{poly})$: FPT. Randomization converts a global search into a local one, and the technique derandomizes with perfect hash families.\nW[1]-hardness, and why clique resists. The W-hierarchy is defined by weighted circuit satisfiability at increasing depths, and it is layered:\n$$\\text{FPT} \\subseteq \\text{W}[1] \\subseteq \\text{W}[2] \\subseteq \\cdots \\subseteq \\text{XP}$$Clique and independent set are W[1]-complete; dominating set is W[2]-complete. Hardness is shown by FPT reductions, which must run in FPT time and bound the new parameter by a function of the old — ordinary Karp reductions are useless here, since they routinely blow the parameter up. Rebuilding the reduction catalogue under this stricter notion is exactly what the field did, mirroring what T031 did for approximation.\nUnder ETH (no $2^{o(n)}$ algorithm for 3-SAT), clique requires $n^{\\Omega(k)}$, so the brute-force exponent is essentially right. W[1]-hardness is a real barrier, not an absence of ideas.\nChoosing the parameter is the actual skill. Solution size is the obvious choice and often the wrong one. Treewidth is frequently better: most NP-hard graph problems are FPT in treewidth by Courcelle\u0026rsquo;s theorem, and real networks — control-flow graphs, road networks, dependency graphs — often have small treewidth. Other useful parameters are the number of distinct values, the maximum degree, the distance from a tractable class, and the solution\u0026rsquo;s structural depth. The same problem is FPT under one parameter and W[1]-hard under another, so \u0026ldquo;is it FPT\u0026rdquo; is not a well-posed question until you say parameterized by what.\nForbids FPT algorithms for W[1]-hard problems, unless FPT = W[1]. Clique, independent set and dominating set parameterized by solution size. The $n^{O(k)}$ brute force is essentially optimal.\nPolynomial kernels for problems without them. Some FPT problems provably have no polynomial kernel unless the polynomial hierarchy collapses — $k$-path is the standard example. So there are problems that are FPT and yet cannot be preprocessed down to polynomial size, which is a genuinely non-obvious separation.\nReading tractability off NP-hardness. Vertex cover and clique are both NP-complete and are complements of each other on the same graph, yet one is FPT and the other is W[1]-hard. The classical theory cannot see this at all.\nKarp reductions transferring parameterized hardness. They do not preserve parameters. Every parameterized hardness result needs its own FPT reduction.\nDoes not forbid It does not mean W[1]-hard problems are unsolvable in practice. Cliquer and modern maximum-clique solvers handle graphs with thousands of vertices routinely, and clique enumeration is standard in bioinformatics on real protein-interaction networks. W[1]-hardness is worst-case over all instances and all parameter values.\nIt does not mean FPT means fast. Courcelle\u0026rsquo;s theorem gives FPT algorithms in treewidth for a huge class of problems, with $f(k)$ a tower of exponentials — completely unimplementable. The Robertson–Seymour graph-minor machinery gives FPT algorithms with constants that have been described, accurately, as galactic. FPT is a classification, and $f(k)$ is the thing you must actually look at.\nIt does not mean parameterization is only theoretical. SAT solvers exploit backdoor sets, which is a parameterized idea. Kernelization rules are what every good ILP presolver does — Gurobi\u0026rsquo;s presolve is kernelization without the theorems. Treewidth-based inference is how probabilistic graphical models are solved, junction trees being exactly the treewidth algorithm.\nIt does not require you to compute the parameter first. Many FPT algorithms work by iterative deepening on $k$, and treewidth has good FPT approximations. You do not need to know the parameter\u0026rsquo;s value in advance to benefit.\nIt does not conflict with approximation. They compose: parameterized approximation gives $(1+\\epsilon)$ answers in $f(k, \\epsilon) \\cdot n^{O(1)}$, and some problems hard under either relaxation alone yield to both together. Recent parameterized-inapproximability results (T031) map where even that fails.\nBoundary Treewidth is the parameter that pays off most often. Courcelle\u0026rsquo;s theorem makes every property expressible in monadic second-order logic FPT in treewidth, which covers most graph problems you will meet. The catch is $f(k)$. Above-guarantee parameterization. MAX-SAT is trivially half-satisfiable, so parameterize by how far above $m/2$ you want to go. Parameterizing by the interesting part of the answer rather than its magnitude is often what makes a problem tractable. Kernel lower bounds. The composition framework proves no polynomial kernel exists for specific problems unless coNP $\\subseteq$ NP/poly. Preprocessing has its own barrier theory, which is a nice parallel to T030. ETH and SETH give finer bounds. Beyond W-hardness, ETH gives statements like \u0026ldquo;no $2^{o(k)}$ algorithm for vertex cover\u0026rdquo; and SETH gives conditional lower bounds on polynomial-time problems too — edit distance in $O(n^{2 - \\epsilon})$ is ruled out under SETH. That is fine-grained complexity, and it applies to problems already in P. The practical reading. When you hit an NP-hard problem, ask what is small about your instances before asking for a heuristic. Solution size, treewidth, degree, number of distinct values, distance from a tractable class. If something is small and the problem is FPT in it, you get an exact answer quickly, which is strictly better than the approximation you were about to settle for. Read next ","permalink":"https://cs.lozic.me/posts/t033-parameterized-complexity-fpt-and-w1/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eYour problem is NP-hard, and yet it keeps being easy.\u003c/p\u003e\n\u003cp\u003eYou are computing a minimum vertex cover to select monitoring points in a\nnetwork. Twelve thousand nodes. NP-hard, per T028, so you brace for the worst —\nand the answer comes back in under a second, every time. The cover is small,\naround thirty nodes, because the network is sparse and the interesting nodes are\nfew.\u003c/p\u003e","title":"Parameterized Complexity (FPT and W[1])"},{"content":"Symptom You have accepted that the problem is NP-hard. Now what?\nThe literature offers a wall of results with numbers attached: 2-approximation, $\\ln n$-approximation, PTAS, FPTAS, 0.878. Nobody explains what these numbers buy you, how they are proved, or how they connect to the heuristic you already wrote. Meanwhile your greedy heuristic is running in production and you have no idea whether it is within 5% or a factor of 50 of optimal, because you cannot compute the optimum to compare against — that was the whole problem.\nThat last point is the one that stings. Measuring approximation quality seems to require solving the problem you cannot solve.\nIt does not, and the technique for getting around it is the actual content of this subject.\nStatement An algorithm is an $\\alpha$-approximation for a minimization problem if it runs in polynomial time and always returns a solution of cost at most $\\alpha \\cdot \\text{OPT}$. For maximization, at least $\\alpha \\cdot \\text{OPT}$ with $\\alpha \\le 1$. The guarantee holds on every input, which is what separates this from a heuristic.\nThe landscape, in increasing order of good news:\nNo constant factor. Set cover: $\\Theta(\\ln n)$, and by T031 that is optimal. General TSP: no constant-factor approximation at all unless P = NP. Constant factor (APX). Vertex cover at 2, metric TSP at 3/2, max-cut at 0.878. These do not admit a PTAS unless P = NP. PTAS. For any $\\epsilon \u0026gt; 0$, a $(1+\\epsilon)$-approximation in time polynomial in $n$ — but possibly $n^{1/\\epsilon}$, which is polynomial and useless. Euclidean TSP has one. FPTAS. Time polynomial in both $n$ and $1/\\epsilon$. Knapsack has one. This is the best possible category, and strongly NP-hard problems cannot have one. The key structural fact: exact-solution equivalence does not survive. All of Karp\u0026rsquo;s 21 are equally hard exactly, and they scatter across every one of these classes. \u0026ldquo;NP-hard\u0026rdquo; tells you almost nothing about how well you can approximate.\nArgument Vertex cover, 2-approximation, and the bounding trick. Find any maximal matching $M$ — repeatedly grab an uncovered edge and delete both endpoints — and output all $2|M|$ endpoints.\nFeasible: if some edge were uncovered, both endpoints would be free and the matching was not maximal.\nWithin 2: the edges of $M$ are disjoint, so any cover must contain at least one endpoint of each, giving $\\text{OPT} \\ge |M|$. We output $2|M| \\le 2,\\text{OPT}$.\nThat is the whole technique, and it answers the symptom. We never computed OPT. We found a quantity — the matching size — that is provably a lower bound on OPT and provably close to our output. Every approximation guarantee is built this way: bound OPT from the side you can compute. Once you internalize this, you can go and bound your own production heuristic.\nGreedy set cover, $\\ln n$. Repeatedly take the set covering the most uncovered elements. If $k$ sets suffice, some set covers at least a $1/k$ fraction of what remains, so each greedy step leaves at most a $(1 - 1/k)$ fraction. After $k \\ln n$ steps, at most $n(1-1/k)^{k\\ln n} \\le n e^{-\\ln n} = 1$ element remains. So greedy uses at most $k \\ln n$ sets, a ratio of $\\ln n$ — about 13.8 at a million elements. By T031 no polynomial algorithm does better, so greedy is not merely good, it is final.\nKnapsack\u0026rsquo;s FPTAS, by rounding. The exact dynamic program runs in $O(n V)$ where $V$ is total value — pseudo-polynomial, exponential in bit length. Scale every value down by $\\epsilon v_{\\max}/n$ and round. Each item loses at most $\\epsilon v_{\\max}/n$, so $n$ items lose at most $\\epsilon v_{\\max} \\le \\epsilon ,\\text{OPT}$. The scaled DP runs in $O(n^3/\\epsilon)$. Throwing away low-order bits of the input is the whole trick, and it is polynomial in $1/\\epsilon$, so $\\epsilon = 0.01$ is genuinely runnable.\nLP relaxation and rounding, the general-purpose method. Write the problem as an integer program, drop the integrality constraint, and solve the LP in polynomial time. The LP optimum bounds the integer optimum — again, a computable bound on OPT. Then round the fractional solution to an integer one. For vertex cover, rounding every $x_v \\ge 1/2$ up gives 2-approximation again, and the LP has a half-integral optimum which makes the analysis clean.\nThe integrality gap is the method\u0026rsquo;s ceiling. The worst-case ratio between the integer and LP optima bounds what any LP-rounding argument can achieve. Set cover\u0026rsquo;s LP gap is $\\Theta(\\log n)$, so no amount of clever rounding beats greedy. Knowing the gap tells you when to stop trying, which is the same service T031 provides one level up.\nRandomized rounding, and derandomization. Treat fractional $x_i$ as a probability, include $i$ with that probability, and use linearity of expectation (T007) plus a concentration bound (T008). MAX-SAT\u0026rsquo;s 3/4-approximation combines randomized rounding with a naive random assignment and takes the better of the two. Goemans–Williamson\u0026rsquo;s 0.878 for max-cut solves a semidefinite relaxation and rounds by a random hyperplane, which is where the odd constant comes from — it is $\\min_\\theta \\frac{2\\theta}{\\pi(1-\\cos\\theta)}$, not a design choice.\nPrimal-dual and local search round out the toolkit: build a feasible dual solution alongside the primal to certify the bound, or make local improvements until stuck and argue the local optimum is globally close. Both are in Williamson and Shmoys, and both give combinatorial algorithms with no LP solver in the loop.\nTightness, and why 2 is really 2. The vertex cover analysis is not loose. A complete bipartite graph $K_{n,n}$ has a maximal matching of size $n$, so the algorithm returns all $2n$ vertices, while the optimum is one side, $n$ vertices. The ratio is exactly 2 on an infinite family, so no sharper analysis of this algorithm exists. Approximation ratios come in pairs — an upper bound from the analysis and a matching lower bound from an adversarial family — and a result without both halves has not been finished.\nThe same discipline explains why greedy set cover is stated as $\\ln n$ rather than something better: there is a known family of instances on which greedy really does use $\\ln n$ times the optimal number of sets.\nForbids A PTAS for APX-hard problems, unless P = NP. Vertex cover, max-cut and metric TSP have hard constant floors, so pushing a 2-approximation to 1.01 is not an engineering project.\nAn FPTAS for strongly NP-hard problems. TSP, graph colouring, bin packing in the general sense. The FPTAS route depends on a pseudo-polynomial algorithm, and strong NP-hardness rules that out.\nConstant-factor approximation of general TSP. Without the triangle inequality, a constant-factor approximation would solve Hamiltonian circuit. Metric structure is doing essential work, not simplifying the analysis.\nGuarantees from unanalyzed heuristics. A heuristic without a bounding argument has no worst case you can state. That is often fine, and it is not the same thing.\nDoes not forbid It does not mean approximation ratios predict practical quality. LKH — a local-search TSP heuristic with no constant-factor guarantee at all — routinely lands within 1% of optimal on instances where Christofides\u0026rsquo;s guaranteed 3/2 is far worse in practice. The guaranteed algorithm often loses to the unguaranteed one on real inputs, and this is the most important thing to know before choosing based on the published ratio.\nIt does not mean the worst case is your case. First-fit-decreasing bin packing has a worst-case ratio of 11/9, and averages around 2% above optimal on realistic item distributions. Ratios are adversarial.\nIt does not mean exact solvers are out of the question. Concorde solves TSP instances with tens of thousands of cities to proven optimality; Gurobi and CPLEX solve enormous integer programs exactly with branch-and-cut. If you need the optimum and can wait, exact is frequently available, and reaching for an approximation reflexively is a mistake.\nIt does not mean a PTAS is usable. A $(1+\\epsilon)$ algorithm running in $n^{O(1/\\epsilon^2)}$ is polynomial and unrunnable at $\\epsilon = 0.1$. The early Euclidean TSP PTASes had exactly this character. \u0026ldquo;Has a PTAS\u0026rdquo; is a classification fact, not a recommendation.\nIt does not mean approximation and heuristics are rivals. The standard production pattern is a guaranteed algorithm for a starting solution plus local search to improve it, keeping the bound while capturing practical gains. You do not have to choose.\nBoundary Bicriteria approximation. Relax feasibility as well as objective: a $(2,3)$-approximation might give twice the cost with three times the resource cap. Frequently the right trade in scheduling and clustering when a hard constraint is soft in reality. Resource augmentation. Compare your algorithm on $m$ machines against the optimum on fewer, which is how competitive-analysis pessimism gets tamed and the standard framing for online scheduling. Online approximation is a different beast. Competitive ratios bound performance without knowledge of the future. Graham\u0026rsquo;s list scheduling is $(2 - 1/m)$-competitive — 1.875 on eight machines — with no future knowledge at all, which is a strong result for an online algorithm. Where the ratio comes from is a design decision. Combinatorial arguments give simple fast algorithms with weaker bounds; LP and SDP rounding give stronger bounds at the cost of solving a relaxation per instance. In production the combinatorial version usually wins on operability. Parameterized approximation combines this post with T033: allow $f(k)\\cdot n^{O(1)}$ time and a $(1+\\epsilon)$ factor, and some problems hard under either relaxation alone become tractable under both. Read next ","permalink":"https://cs.lozic.me/posts/t032-approximation-algorithms-and-ratios/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eYou have accepted that the problem is NP-hard. Now what?\u003c/p\u003e\n\u003cp\u003eThe literature offers a wall of results with numbers attached: 2-approximation,\n$\\ln n$-approximation, PTAS, FPTAS, 0.878. Nobody explains what these numbers\nbuy you, how they are proved, or how they connect to the heuristic you already\nwrote. Meanwhile your greedy heuristic is running in production and you have no\nidea whether it is within 5% or a factor of 50 of optimal, because \u003cstrong\u003eyou cannot\ncompute the optimum to compare against\u003c/strong\u003e — that was the whole problem.\u003c/p\u003e","title":"Approximation Algorithms and Ratios"},{"content":"Symptom You gave up on exact (T028) and went looking for an approximation.\nVertex cover: you find a 2-approximation in four lines. Knapsack: an FPTAS, any accuracy you want. Encouraged, you go looking for the same for max-clique, and you find nothing. Not \u0026ldquo;nothing good\u0026rdquo; — the best known ratio is around $n/(\\log n)^2$, which on a 10,000-vertex graph means the algorithm might return a clique nearly 60 times smaller than the true maximum. That is not an approximation, it is a rumour.\nWhy does approximability vary so wildly across problems that are all equally hard to solve exactly? Karp\u0026rsquo;s problems are interreducible; exact solution is one problem in twenty-one disguises. Approximation shatters that equivalence completely, and the reason it does is one of the deepest results in the field.\nStatement The PCP theorem (Arora–Safra, Arora–Lund–Motwani–Sudan–Szegedy, 1992).\n$$\\text{NP} = \\text{PCP}(\\log n, 1)$$Every NP language has a proof format such that a verifier using $O(\\log n)$ random bits and reading only a constant number of bits of the proof accepts correct proofs always, and rejects claimed proofs of false statements with probability at least 1/2. The constant can be taken to be 3.\nRead that again, because it should sound impossible. You can check a mathematical proof by looking at three bits of it, chosen at random, and catch any flaw half the time. Not three lines. Three bits.\nThe equivalent gap formulation, which is what gets used. It is NP-hard to distinguish 3-SAT instances that are fully satisfiable from those where at most a $(1-\\epsilon)$ fraction of clauses can be satisfied, for some fixed $\\epsilon \u0026gt; 0$.\nThe consequence. If approximating a problem within some ratio would let you distinguish those two cases, then approximating within that ratio is NP-hard. Hardness of approximation follows from hardness of decision — the PCP theorem is the bridge, and before it there was essentially no way to prove any inapproximability result at all.\nSharp consequences, all tight or nearly so:\nMAX-3SAT is NP-hard to approximate better than 7/8 (Håstad). Random assignment achieves exactly 7/8, so the trivial algorithm is optimal. Max-clique is NP-hard within $n^{1-\\epsilon}$ for any $\\epsilon \u0026gt; 0$. Set cover is NP-hard within $(1-o(1))\\ln n$ (Feige, Dinur–Steurer). Greedy matches it. Vertex cover is NP-hard within $1.36$, and within $2-\\epsilon$ under the Unique Games Conjecture. Argument The full proof is long and this post will not give it. The load-bearing ideas are gettable, and they are what you need.\nWhy \u0026ldquo;3 bits\u0026rdquo; is not a trick. The verifier does not read the proof as a transcript. The proof is written in an error-correcting encoding (T022) of the satisfying assignment. In a good code, any two valid codewords differ in a constant fraction of positions, so a corrupted or fabricated proof differs from every valid one almost everywhere. Random spot-checks then catch it with constant probability. Local checkability comes from global redundancy — exactly the property T022 studies, deployed for verification rather than transmission.\nThe easy direction. PCP$(\\log n, 1) \\subseteq$ NP: a verifier using $O(\\log n)$ random bits has polynomially many random strings, so enumerate them all, simulate, accept if all accept. Straightforward.\nThe hard direction, in outline. Given a 3-SAT instance, encode the assignment in a highly redundant form — the original proof used the Walsh–Hadamard code, where the proof is the value of every linear function of the assignment, an exponentially long object. Linearity testing (the BLR test: check $f(x) + f(y) = f(x+y)$ on random $x, y$) verifies the encoding is close to linear using three queries. Then self-correction recovers any desired value reliably from a slightly corrupted table. Composition — recursively applying the same machinery to the verifier\u0026rsquo;s own computation — brings the proof length back down to polynomial while keeping the query count constant. Proof composition is the technical heart, and it is genuinely hard.\nDinur\u0026rsquo;s gap amplification (2007) is the version to know. Instead of building the verifier directly, start from a standard NP-hard constraint system with a tiny satisfiability gap and amplify it. Each round: take the constraint graph, raise it to a power by replacing constraints with walks of length $t$ (which doubles the gap but blows up the alphabet), then compose with a small fixed PCP to shrink the alphabet back. Repeat $O(\\log n)$ times. The gap doubles each round and the size grows only by a constant factor, so a constant gap arrives after logarithmically many rounds at polynomial total size.\nThis reframes the theorem entirely: PCP is not a statement about proofs, it is a statement about amplifying a small gap into a large one, and the amplifier is expander graph powering. That is a comprehensible idea, and it is why Dinur\u0026rsquo;s proof is the one taught.\nHow you actually use it. To prove your problem hard to approximate, build a gap-preserving reduction from gap-3SAT: satisfiable instances must map to instances with high optimum, and $(1-\\epsilon)$-satisfiable ones to instances with low optimum. Then an approximation algorithm beating the ratio between them would decide gap-3SAT. This is a mechanical technique once you have the gap, and the entire inapproximability literature is instances of it.\nDeriving an inapproximability result, end to end. Suppose you had a $(1 - \\epsilon/2)$-approximation for MAX-3SAT with $\\epsilon$ the PCP gap constant. Feed it a gap instance. If the instance is satisfiable, the algorithm returns an assignment satisfying at least $(1-\\epsilon/2)m$ clauses. If at most $(1-\\epsilon)m$ clauses are satisfiable, the algorithm cannot return more than that, since it cannot exceed the optimum. Those two ranges do not overlap, so counting satisfied clauses in the returned assignment decides gap-3SAT, which is NP-hard. Therefore the approximation algorithm cannot exist unless P = NP.\nThat is the entire argument pattern, and every hardness-of-approximation result in the literature is this shape with a gap-preserving reduction bolted on the front. Once you see it once, the wall of ratios in the previous post becomes a set of derived facts rather than a table to memorize.\nForbids A PTAS for MAX-3SAT, max-clique, or set cover, unless P = NP. Not merely undiscovered. Ruled out at specific, known ratios.\nBeating 7/8 for MAX-3SAT. The stupidest possible algorithm — assign each variable at random, satisfying each clause with probability 7/8 — is optimal, and no cleverness can improve it. This is the most striking single corollary.\nBeating $\\ln n$ for set cover. Greedy is optimal, and the thirty years spent looking for better were provably wasted.\nNon-trivial clique approximation. Within $n^{1-\\epsilon}$ is hard, so essentially nothing better than returning a single vertex is guaranteed.\nDoes not forbid It does not stop good approximations for other problems, and the landscape is wildly uneven. Knapsack has an FPTAS. Euclidean TSP has a PTAS (Arora, Mitchell). Metric TSP had 3/2 from Christofides for forty years and was improved to $3/2 - 10^{-36}$ in 2020. Max-cut has the 0.878 Goemans–Williamson semidefinite bound, which is exactly optimal under the Unique Games Conjecture. Exact-solution equivalence does not survive approximation, and that is the main lesson.\nIt does not make heuristics useless. Set cover being $\\ln n$-hard in the worst case does not stop greedy set cover from being excellent on real instances, and LKH solves TSP instances to within fractions of a percent routinely. Worst-case inapproximability is compatible with near-optimal performance on structured inputs.\nIt does not apply to average-case or structured instances. These are worst-case results. Planar and bounded-genus versions of many hard problems admit PTASes via Baker\u0026rsquo;s technique, and geometric instances are frequently much easier.\nIt does not mean the Unique Games Conjecture results are settled. UGC-based tight bounds — vertex cover at 2, max-cut at 0.878 — are conditional on a conjecture that is open and that some researchers doubt. Subexponential-time algorithms for unique games exist, which is unusual for a hardness assumption and worth knowing before citing UGC results as established.\nIt does not mean PCPs are only a proof technique. Probabilistically checkable proofs became practical: SNARKs and STARKs, deployed in zk-rollups on Ethereum and in verifiable computation generally, are engineered descendants of this machinery. A verifier really does check a computation by sampling a few positions of an encoded proof. The theorem became infrastructure, which almost no complexity result does.\nBoundary The approximability classes. APX (constant-factor), PTAS, FPTAS, and log-APX form a hierarchy that is strict unless P = NP. Locating your problem in it tells you what to aim for and when to stop. Gap-preserving reductions are the working tool. Ordinary Karp reductions do not preserve approximation ratios. L-reductions and gap-preserving reductions do, and building the catalogue of inapproximability results was redoing Karp\u0026rsquo;s work with a stronger notion of reduction. The Unique Games Conjecture as the missing piece. Many natural problems have a gap between the best algorithm and the best unconditional hardness. UGC closes almost all of them at once, which is either evidence for it or a reason for suspicion depending on temperament. Håstad\u0026rsquo;s optimal inapproximability results are the sharpest artifacts here: for several problems, the trivial randomized algorithm is provably the best possible. There is something bracing about a theorem whose content is that cleverness cannot help. Parameterized inapproximability. Recent work extends this to FPT approximation, showing some problems resist even the parameter-based escape of T033. The escapes are being closed off one at a time, and knowing which remain open is what tells you where to spend effort. Read next ","permalink":"https://cs.lozic.me/posts/t031-the-pcp-theorem-and-inapproximability/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eYou gave up on exact (T028) and went looking for an approximation.\u003c/p\u003e\n\u003cp\u003eVertex cover: you find a 2-approximation in four lines. Knapsack: an FPTAS, any\naccuracy you want. Encouraged, you go looking for the same for max-clique, and\nyou find nothing. Not \u0026ldquo;nothing good\u0026rdquo; — the best known ratio is around\n$n/(\\log n)^2$, which on a 10,000-vertex graph means the algorithm might return\na clique nearly 60 times smaller than the true maximum. That is not an\napproximation, it is a rumour.\u003c/p\u003e","title":"The PCP Theorem and Inapproximability"},{"content":"Symptom Product wants the daily unique-visitor count. You have a firehose of events.\nThe obvious implementation is a set. Add each visitor ID, report the size. At a billion distinct IDs, eight bytes each, that is 8 GB before any hash table overhead, and in practice a HashSet will cost you two to three times that. Per day. Per dimension you want to slice by. Multiply by country, by platform, by campaign, and the memory bill is absurd for a number nobody looks at past two significant figures.\nSo you try to be clever, and you fail, repeatedly, in a way that feels like it should be solvable. Then somebody tells you it is not solvable, and hands you a 12 KiB structure that answers the question to within 1%.\nBoth halves of that are theorems, and the pair of them is the most useful trade in applied algorithms.\nStatement The lower bound. Computing the number of distinct elements $F_0$ exactly in one pass requires $\\Omega(n)$ bits of space, where $n$ is the universe size. The same holds for exact frequency queries and for exact median. Randomization and approximation separately do not help; you need both.\nThe upper bound. With a $(1 \\pm \\epsilon)$ approximation and failure probability $\\delta$, distinct counting is solvable in $O(\\epsilon^{-2}\\log\\log n + \\log n)$ bits. HyperLogLog achieves a standard error of $1.04/\\sqrt{m}$ using $m$ registers of about 6 bits each.\nThe trade, in numbers. At $m = 2^{14} = 16384$ registers, HyperLogLog uses 12 KiB and has 0.81% standard error, at any cardinality up to billions. Against 8 GB minimum for the exact set, that is a factor of roughly 700,000 in space for one significant figure of loss.\nCount-Min sketch does the analogous thing for frequencies: a $d \\times w$ table of counters gives $\\hat{f}_x \\le f_x + \\epsilon N$ with probability $1 - \\delta$ using $w = \\lceil e/\\epsilon \\rceil$ and $d = \\lceil \\ln(1/\\delta) \\rceil$. At $\\epsilon = \\delta = 0.001$ that is $2719 \\times 7 = 19{,}033$ counters — under 80 KB to track every frequency in a stream of any length to within 0.1% of its total.\nArgument The lower bound, by reduction from communication complexity. This is where the information-theoretic argument of T004 reappears in a new setting.\nAlice holds $x \\in {0,1}^n$, Bob holds $y \\in {0,1}^n$, and they must decide whether $x = y$ (EQUALITY) or whether they share a 1 (DISJOINTNESS). The known result: deterministic EQUALITY requires $n$ bits of communication, and randomized DISJOINTNESS requires $\\Omega(n)$.\nNow suppose a streaming algorithm computes $F_0$ exactly in $s$ bits. Alice feeds her set as a stream, sends the algorithm\u0026rsquo;s $s$-bit state to Bob, and Bob continues with his own elements. From the final $F_0$ Bob learns $|A \\cup B|$, and knowing $|A|$ and $|B|$ gives $|A \\cap B|$, which decides DISJOINTNESS. So $s = \\Omega(n)$.\nThe state of the algorithm is a message, and anything the message cannot carry, the algorithm cannot compute. Once you see streaming as one-way communication, lower bounds follow from a well-developed theory rather than from ad-hoc arguments.\nWhy approximation escapes. The reduction needs an exact answer to recover $|A \\cap B|$. A $(1 \\pm \\epsilon)$ estimate does not distinguish intersection size 0 from 1 when the sets are large, so the reduction breaks and the bound does not apply. Both approximation and randomization are load-bearing.\nHyperLogLog, from first principles. Hash each element to a uniform bit string. For a truly random hash, the probability that a value begins with $k$ zeros is $2^{-k}$. So among $N$ distinct values, the maximum leading-zero count is about $\\log_2 N$. Track that maximum, report $2^{\\max}$.\nDuplicates are free: rehashing the same element gives the same bit string and cannot raise the maximum. That is why the structure counts distinct elements rather than elements, and it needs nothing but the hash\u0026rsquo;s determinism.\nThe single-estimator variance is terrible — each additional zero doubles the estimate. The fix is stochastic averaging: use the first $b$ bits to pick one of $m = 2^b$ registers, keep a per-register maximum, and combine with the harmonic mean, which suppresses the influence of the occasional huge register. The analysis gives standard error $1.04/\\sqrt{m}$. Each register holds a count up to about 64, so 6 bits suffices, and 16384 registers is 12 KiB.\nRegisters hold $\\log\\log n$ bits, which is where the $O(\\epsilon^{-2}\\log\\log n)$ comes from. This is provably near-optimal.\nCount-Min, and why the error is one-sided. Keep $d$ rows of $w$ counters, each row with its own hash from a pairwise-independent family (T104). On update $x$, increment cell $(i, h_i(x))$ in every row. To query, report the minimum across rows.\nEvery counter is at least $f_x$, since $x$\u0026rsquo;s own increments are always there — the estimate can only overshoot, never undershoot. Overshoot in row $i$ is the total from other items colliding with $x$ there. By pairwise independence and linearity of expectation (T007), the expected collision mass is $N/w$. Markov\u0026rsquo;s inequality gives $\\Pr[\\text{overshoot} \u0026gt; eN/w] \\le 1/e$, and $d$ independent rows all failing has probability $e^{-d}$. Set $w = e/\\epsilon$ and $d = \\ln(1/\\delta)$ and you are done.\nBoth structures are mergeable: a union of HLLs is a per-register maximum, a union of Count-Mins is a cellwise sum. Distributed aggregation becomes associative and commutative, which is why every analytics system in production uses them — Redis PFCOUNT, Presto, BigQuery\u0026rsquo;s APPROX_COUNT_DISTINCT, Druid and ClickHouse.\nWhat the constants mean at a chosen size. The $\\epsilon^{-2}$ dependence is worth pricing before you pick registers. At $m = 2^{12} = 4096$, HyperLogLog costs 3 KiB with 1.62% standard error; at $m = 2^{14}$ it is 12 KiB and 0.81%; at $m = 2^{16}$ it is 48 KiB and 0.41%. Four times the memory for half the error, every time, which is the theorem\u0026rsquo;s exponent showing up directly in a capacity-planning spreadsheet. Redis chose $m = 2^{14}$ for exactly this reason: 12 KiB per counter and under 1% error is the point where both numbers stop mattering to most applications.\nForbids Exact distinct counting in sublinear space. Not an implementation limitation. Any exact one-pass algorithm needs space proportional to the universe.\nExact heavy hitters in small space. Same reduction. You get approximate counts with bounded error or you get linear space.\nExact quantiles in one pass in sublinear space. Medians need $\\Omega(n)$ exactly, which is why every latency dashboard reports approximate percentiles whether it admits it or not.\nArbitrarily small error for free. The $\\epsilon^{-2}$ dependence is tight: halving the error quadruples the space. Going from 1% to 0.1% costs 100×, and that is a theorem rather than a tuning problem.\nDoes not forbid It does not stop exact counting when the set is small or bounded. If you have a million users, a bitmap is 125 KB and exact. Roaring bitmaps do exact distinct counts on compressed integer sets at enormous scale and are the right choice when IDs are dense. The lower bound is about the universe, and if your universe is small it does not bite.\nIt does not mean sketches are always the right answer. They cannot be decremented reliably (Count-Min undercounts badly with deletions; use Count-Sketch or a counting Bloom filter), you cannot enumerate members, and you cannot compute exact set differences. Debugging \u0026ldquo;which user is missing\u0026rdquo; is impossible with an HLL, and that is a real operational cost people discover late.\nIt does not mean the error is uniform. HyperLogLog\u0026rsquo;s raw estimator is biased at small cardinalities, which is why the original paper adds linear counting below $2.5m$ and Google\u0026rsquo;s HLL++ replaces that with an empirical bias-correction table and a sparse representation. Naive implementations are visibly wrong under a few thousand elements, and this is the single most common HLL bug.\nIt does not require perfect hash functions. The analysis assumes idealized uniform hashing, but Count-Min needs only pairwise independence, and HLL works fine with a good 64-bit non-cryptographic hash such as xxHash or MurmurHash. The theory is tighter than practice needs, which is fortunate.\nIt does not mean multi-pass algorithms are stuck. The lower bound is for one pass. With two passes or with sorted input, exact distinct counting is easy — sort and scan. Batch systems with data on disk are not in the streaming model at all, so applying streaming bounds to a Spark job is a category error.\nBoundary The AMS sketch and higher moments. $F_2$ — the sum of squared frequencies, used for self-join size estimation — is estimable in $O(\\epsilon^{-2}\\log n)$ space by Alon, Matias and Szegedy, using 4-wise independent $\\pm1$ hashes. $F_k$ for $k \u0026gt; 2$ genuinely needs polynomial space, which is a sharp threshold. Sliding windows. Exponential histograms give $F_0$ over the last $W$ elements at $O(\\epsilon^{-1}\\log^2 W)$ space. Recency costs a factor, and it is the usual production requirement. Quantile sketches. t-digest and KLL give approximate percentiles with strong accuracy at the tails, which is exactly what latency monitoring needs and what a naive equal-width histogram fails at. Turnstile versus cash-register. Insert-only streams are much easier than streams with deletions. Knowing which model you are in determines which sketches are even applicable, and it is the first question to ask. The bigger pattern. These are all the same trade the Bloom filter makes: accept a bounded, one-sided, quantified error and pay a fraction of the space. What makes it engineering rather than gambling is that the error is bounded and computable in advance, so you can decide whether 0.81% is acceptable before shipping. Read next ","permalink":"https://cs.lozic.me/posts/t105-streaming-lower-bounds-and-sketching/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eProduct wants the daily unique-visitor count. You have a firehose of events.\u003c/p\u003e\n\u003cp\u003eThe obvious implementation is a set. Add each visitor ID, report the size. At a\nbillion distinct IDs, eight bytes each, that is 8 GB before any hash table\noverhead, and in practice a \u003ccode\u003eHashSet\u003c/code\u003e will cost you two to three times that.\nPer day. Per dimension you want to slice by. Multiply by country, by platform,\nby campaign, and the memory bill is absurd for a number nobody looks at past two\nsignificant figures.\u003c/p\u003e","title":"Streaming Lower Bounds and Sketching"},{"content":"Symptom Your service went down under a hash collision attack.\nSomebody noticed your web framework put POST parameters into a hash table, found thousands of distinct keys colliding under its hash function, and posted a form with 20,000 of them. Every insert walked a chain. Quadratic behaviour, one CPU pinned per request, service dead. This actually happened, across PHP, Python, Ruby, Java and .NET in 2011, and again against Rust\u0026rsquo;s default HashMap before it switched to SipHash.\nYour hash function was fine. It distributed real-world keys beautifully. That was never the property you needed.\nWhat does a hash function actually guarantee? \u0026ldquo;Looks random\u0026rdquo; is not a guarantee, because an adversary — or an unlucky workload — gets to pick the keys after seeing your function. The fix is not a better function. It is a different kind of promise.\nStatement A family $\\mathcal{H}$ of functions from universe $U$ to ${0,\\dots,m-1}$ is universal if for every pair of distinct keys $x \\ne y$,\n$$\\Pr_{h \\in \\mathcal{H}}[h(x) = h(y)] \\le \\frac{1}{m}$$where the probability is over the random choice of $h$ from the family, not over the keys. That inversion is the entire idea.\n$\\mathcal{H}$ is pairwise independent (or strongly universal) if for distinct $x, y$ and any values $a, b$,\n$$\\Pr_{h}[h(x) = a \\wedge h(y) = b] = \\frac{1}{m^2}$$which is stronger: not only are collisions rare, any two keys land independently.\nCarter–Wegman construction. Pick a prime $p \u0026gt; |U|$ and choose $a \\in {1,\\dots,p-1}$, $b \\in {0,\\dots,p-1}$ uniformly:\n$$h_{a,b}(x) = ((ax + b) \\bmod p) \\bmod m$$This family is universal, and pairwise independent when $m = p$.\nConsequence. With $n$ keys in $m$ buckets, the expected chain length for any key is at most $1 + n/m$. For every input, including adversarial ones, because the randomness is yours and the adversary cannot see your choice of $h$.\nArgument Why the guarantee has to be over $h$. Fix any single hash function $h: U \\to [m]$. Since $|U| \\gg m$, some bucket receives at least $|U|/m$ keys by pigeonhole. An adversary enumerates them and sends those. No fixed function escapes; the only defence is that the adversary does not know which function you picked.\nUniversality of $h_{a,b}$. Work in $\\mathbb{Z}_p$, a field since $p$ is prime. Fix $x \\ne y$ and let\n$$r = (ax + b) \\bmod p, \\qquad s = (ay + b) \\bmod p$$First, $r \\ne s$: their difference is $a(x-y) \\bmod p$, and in a field a product of nonzero elements is nonzero, since $a \\ne 0$ and $x \\ne y$. So there are no collisions before the final mod $m$.\nSecond, the map $(a,b) \\mapsto (r,s)$ is a bijection from the $p(p-1)$ valid pairs onto the $p(p-1)$ pairs with $r \\ne s$. Given any such $(r,s)$, solve: $a = (r-s)(x-y)^{-1}$ and $b = r - ax$, both uniquely determined in the field. So $(r,s)$ is uniform over distinct pairs.\nNow collisions can only come from the final $\\bmod, m$. Given $r$, how many $s \\ne r$ have $s \\equiv r \\pmod m$? At most $\\lceil p/m \\rceil - 1 \\le (p-1)/m$. Dividing by the $p-1$ available values of $s$:\n$$\\Pr[h(x) = h(y)] \\le \\frac{(p-1)/m}{p-1} = \\frac{1}{m}$$Universal. Two facts did the work: a field has no zero divisors, and an affine map with nonzero slope is a bijection.\nFrom the pair bound to chain length. This is where linearity of expectation (T007) makes the analysis trivial. Fix a key $x$ and a set $S$ of $n$ keys. Let $C_y = 1$ if $y \\ne x$ collides with $x$. Then\n$$\\mathbb{E}[\\text{chain length at } x] = 1 + \\sum_{y \\in S, y \\ne x} \\Pr[h(y) = h(x)] \\le 1 + \\frac{n}{m}$$No independence between different $C_y$ is needed, because expectation is linear regardless. Universality alone, a statement about pairs, gives the bound you actually want, a statement about a whole table. With $m \\ge n$ the expected chain is under 2 and hash table operations are $O(1)$ expected — for every input.\nRandomize the algorithm, not the assumption. The classical analysis assumes keys are random. Carter and Wegman assume nothing about keys and make the algorithm random. This is the same move as randomized quicksort: it converts \u0026ldquo;fast on typical inputs\u0026rdquo; into \u0026ldquo;fast on every input, with high probability over my own coins,\u0026rdquo; which is a guarantee an adversary cannot attack.\nWhat stronger independence buys. Pairwise independence is what Chebyshev-based variance arguments need, so 2-universal families are exactly right for sketching. Count-Min needs pairwise; higher moment estimators need 4-wise. The hierarchy is not academic: each level costs more and unlocks a specific concentration inequality, and picking the level is a real design decision in T105\u0026rsquo;s sketches.\nA worked collision count. Take $p = 2{,}147{,}483{,}647$ (a Mersenne prime, $2^{31}-1$, which makes the modulus cheap) and $m = 2^{20}$ buckets holding a million keys. Universality says any specific pair collides with probability at most $2^{-20}$. The expected number of colliding pairs among $n = 10^6$ keys is at most $\\binom{10^6}{2} \\cdot 2^{-20} \\approx 4.77 \\times 10^{5}$, so about half a million colliding pairs — which sounds alarming until you notice it is spread over a million buckets, giving the expected chain length just under 2 that the theorem promised. The pair count is large and the per-key cost is constant, and confusing those two quantities is how people talk themselves out of a correct data structure.\nForbids Any fixed hash function with worst-case guarantees. By pigeonhole. If your function is public and deterministic, adversarial inputs exist and can be found.\nAssuming your keys are random. They are usernames, URLs, sequential IDs and attacker-chosen strings. The uniformity assumption is not a technicality that usually holds; it is false in production.\nIgnoring the seed. A universal family used with a constant seed is a fixed function, and provides nothing. Seeding per process from a real source is the whole mechanism.\nBelieving collisions can be eliminated. With $|U| \u0026gt; m$ they are forced. The guarantee is on their rate, not their existence.\nDoes not forbid It does not mean non-universal hashing is wrong in practice. MurmurHash3, xxHash and FNV are not universal families and are used everywhere, correctly, because they are far faster and most workloads are not adversarial. Google\u0026rsquo;s Abseil and Rust\u0026rsquo;s hashbrown use fast non-cryptographic hashes with randomized seeds — the seed is doing the security work, and the hash is doing the speed work. That split is the actual engineering answer.\nIt does not require cryptographic strength. SipHash-1-3, the default in Rust and Python since 3.4, is a keyed PRF rather than a Carter–Wegman family, and it is chosen because it resists key recovery at a few cycles per byte. Universality gives collision-probability bounds; a keyed PRF gives a stronger practical property at comparable cost. Both solve the 2011 attack; neither is the unique answer.\nIt does not make expected-case bounds worst-case. The $1 + n/m$ chain is an expectation. Some chains will be longer — the maximum is $\\Theta(\\log n / \\log\\log n)$ under full randomness (T009). Cuckoo hashing and perfect hashing exist precisely because expectation is sometimes not enough.\nIt does not mean you need pairwise independence for hash tables. Plain universality suffices for chaining, and linear probing needs 5-independence for its classical bound, though tabulation hashing gets there cheaply. Reaching for the strongest family available is a common and costly mistake.\nIt does not stop hash flooding on its own if the seed leaks. Timing side-channels can reveal seeds. This is why Python randomizes per process and why long-lived servers with observable timing still need rate limiting. The theorem protects against key-guessing, not against key-learning.\nBoundary Perfect hashing (FKS). For a static key set, two levels of universal hashing give worst-case $O(1)$ lookup in $O(n)$ space, with the second-level table for a bucket of size $b$ sized $b^2$ so it is collision-free with probability at least 1/2. Static sets escape the expectation entirely. Cuckoo hashing. Two functions, two candidate slots, worst-case $O(1)$ lookup with amortized expected $O(1)$ insert. The trade is rehashing on failure, which universality bounds. Tabulation hashing. XOR precomputed random tables indexed by each byte. Only 3-independent, yet it behaves like full randomness for linear probing, min-wise hashing and Chernoff-type bounds, and it is faster than multiply-shift. The best speed-to-guarantee ratio available. Multiply-shift. $h(x) = (ax \\bmod 2^{w}) \\gg (w - \\ell)$ with odd $a$ is 2-universal up to a factor of 2 and needs one multiply and one shift, no division and no prime. This is what production code should use when it wants a provable family. The seed has to come from somewhere. A universal family is only as good as the randomness selecting from it. Python seeds from os.urandom at interpreter start, Rust\u0026rsquo;s RandomState draws from the OS per HashMap instance, and a hardcoded seed in a config file silently reverts you to a fixed function with none of the guarantees. This is the commonest way the protection is lost in practice, and it is invisible in testing. Where this feeds forward. Bloom filters (T073), Count-Min and HyperLogLog (T105) all assume a hash family with a stated independence level. Their error bounds are theorems about that assumption, so knowing which level you have is the difference between a proved bound and a hope. Read next ","permalink":"https://cs.lozic.me/posts/t104-universal-hashing/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eYour service went down under a hash collision attack.\u003c/p\u003e\n\u003cp\u003eSomebody noticed your web framework put POST parameters into a hash table, found\nthousands of distinct keys colliding under its hash function, and posted a form\nwith 20,000 of them. Every insert walked a chain. Quadratic behaviour, one CPU\npinned per request, service dead. This actually happened, across PHP, Python,\nRuby, Java and .NET in 2011, and again against Rust\u0026rsquo;s default \u003ccode\u003eHashMap\u003c/code\u003e before\nit switched to SipHash.\u003c/p\u003e","title":"Universal Hashing"},{"content":"Symptom You fit a model. It scores 94% on training data and 71% on held-out data.\nSo you simplify: fewer parameters, more regularization. Training drops to 88%, held-out rises to 84%. You simplify further and both drop. Somewhere in there was an optimum, and you found it by trial and error, with a validation set and patience.\nEveryone tells you to prefer the simpler model. Occam\u0026rsquo;s razor, they say, as if citing a medieval friar settles a question about polynomial degree. But nobody tells you what \u0026ldquo;simpler\u0026rdquo; means quantitatively, and without that the advice is unfalsifiable. Simpler in parameters? In description? In computation? A degree-9 polynomial with small coefficients and a degree-3 polynomial with huge ones — which is simpler, and why should the answer have anything to do with predicting new data?\nMDL answers this. Simplicity is compression, the units are bits, and the tradeoff stops being a matter of taste.\nStatement Minimum description length principle. The best model for a dataset is the one minimizing the total description length of model plus data given model:\n$$\\min_{M} \\; \\left[ L(M) + L(D \\mid M) \\right]$$$L(M)$ is the bits to describe the model; $L(D \\mid M)$ is the bits to describe the data using that model as a codebook. Learning is compression, and the best hypothesis is the one that makes the shortest total message.\nThe two terms pull against each other and that is the point. A complex model fits tightly and costs little to encode residuals — small $L(D \\mid M)$, large $L(M)$. The empty model costs nothing to state and forces you to transmit the data raw. Neither extreme wins, and overfitting is exactly the case where the model costs more bits than it saves.\nTwo connections make this more than a slogan. Via Kolmogorov complexity (T023), $L(M) + L(D \\mid M)$ approximates $K(D)$, so MDL is a computable stand-in for \u0026ldquo;the shortest program generating this data.\u0026rdquo; Via Shannon (T018), the optimal code length for data under a probabilistic model is $-\\log_2 P(D \\mid M)$ bits, so minimizing description length is minimizing\n$$-\\log_2 P(D \\mid M) - \\log_2 P(M)$$which is exactly maximum a posteriori estimation. MDL and Bayesian model selection are the same computation in different units: a prior is a code, and a code is a prior.\nArgument Why the code-length view is forced. Shannon\u0026rsquo;s source coding theorem says that for a distribution $P$, the optimal codeword for outcome $x$ has length $-\\log_2 P(x)$ bits, achievable to within one bit (T018, T020). Probabilities and code lengths are interchangeable. Any probabilistic model is a compression scheme and any compression scheme is a probabilistic model, so the question \u0026ldquo;which model\u0026rdquo; and the question \u0026ldquo;which encoding\u0026rdquo; are one question.\nThe two-part code, worked. Suppose you have 1000 points and are choosing a polynomial degree.\nModel cost. A degree-$d$ polynomial has $d+1$ coefficients. At $b$ bits each, $L(M) = (d+1)b$. There is a classical result that the right precision is about $\\frac{1}{2}\\log_2 n$ bits per parameter — finer precision costs more than it saves, since the estimation error already swamps it. At $n = 1000$ that is about 5 bits per parameter. Data cost. With Gaussian residuals of standard deviation $\\sigma$, encoding $n$ residuals costs about $\\frac{n}{2}\\log_2(2\\pi e \\sigma^2)$ bits. Halving $\\sigma$ saves exactly $n$ bits — one bit per point. So each added parameter costs $\\frac{1}{2}\\log_2 n$ bits and is worth it only if it reduces residual entropy by more. That is the whole tradeoff in one sentence, and it is quantitative.\nNow the numbers. Adding a parameter at $n = 1000$ costs about 5 bits. To pay for itself it must shrink $\\sigma$ by a factor of $2^{5/1000} = 1.0035$ — a 0.35% reduction in residual scale. That is a small bar, which is correct: with a thousand points you can afford parameters. At $n = 20$, the cost is $\\frac{1}{2}\\log_2 20 = 2.16$ bits, and the required shrinkage is $2^{2.16/20} = 1.078$, nearly 8%. The same parameter must earn twenty times as much on small data. MDL derives the folk rule that small datasets demand simple models, rather than asserting it.\nWhere AIC and BIC come from. Take the two-part code with $\\frac{1}{2}\\log_2 n$ bits per parameter, convert from bits to nats, and multiply by two:\n$$-2\\ln P(D \\mid \\hat\\theta) + k \\ln n$$That is BIC exactly. The Bayesian Information Criterion is the two-part MDL code in different clothing, which is why BIC\u0026rsquo;s $\\ln n$ penalty grows with sample size while AIC\u0026rsquo;s constant $2k$ does not. AIC estimates prediction error, BIC and MDL estimate description length, and they disagree because they are answering different questions — AIC does not assume the true model is in your candidate set, BIC does. Knowing which you want stops the endless AIC-versus-BIC argument.\nThe crude code\u0026rsquo;s weakness, and the fix. $L(M)$ depends on how you choose to encode models, and different encodings give different winners. That arbitrariness is real, and it is why naive MDL is unsatisfying. Modern refined MDL removes the choice using the normalized maximum likelihood distribution: encode the data with the code that minimizes worst-case regret against the best model in the class. The resulting complexity term — the parametric complexity — depends on the model class\u0026rsquo;s geometry rather than on your arbitrary parameterization. That is the version worth learning, and Grünwald\u0026rsquo;s tutorial is where.\nWhy compression predicts. Compression is only possible by finding regularity; regularity is what persists in new data. A model compressing your data has found structure, and structure generalizes while noise does not. This is the intuition behind Solomonoff induction, and MDL is its computable approximation.\nForbids Model selection by training fit alone. Training likelihood always improves with parameters, so it cannot select. Something must price complexity, and MDL says the price is in bits.\nFree lunch from complexity. Every parameter costs. A model must earn its bits through improved data compression or it loses, and this is not a heuristic but an accounting identity.\nThe idea that \u0026ldquo;simplicity\u0026rdquo; is subjective here. Once you fix a description language, simplicity is a number. Language choice still matters, but it shifts everything by a constant (T023), not arbitrarily.\nComparing models across different data encodings. Both terms must be in the same units on the same data, or the comparison is meaningless. This is the commonest way MDL is misapplied.\nDoes not forbid It does not say big models cannot generalize, and modern deep learning is the loud counterexample. GPT-scale networks have hundreds of billions of parameters and generalize well past the point classical complexity penalties predict disaster. The resolution is that parameter count is a terrible proxy for description length: trained networks are massively compressible — pruning and 8-bit or 4-bit quantization routinely shrink them by an order of magnitude with little loss — and implicit regularization from SGD keeps the effective description short. Double descent, where test error falls again past the interpolation threshold, is the empirical shape of this. MDL is not refuted; the naive counting proxy is.\nIt does not make cross-validation obsolete. CV is the workhorse and usually the right default, since it needs no coding scheme and directly estimates what you care about. MDL is most valuable where CV is awkward: tiny datasets, time series where splitting breaks dependence, and online settings with no held-out set.\nIt does not force the simplest model always. MDL selects the model with the best total, and on large data that is often a large model. \u0026ldquo;Prefer simplicity\u0026rdquo; is a consequence of the accounting under scarce data, not an axiom.\nIt does not mean regularization is just MDL in disguise, though the correspondence is exact where it holds. L2 regularization is a Gaussian prior and hence a code; L1 is Laplace and induces sparsity because zeros are cheap to encode. Dropout and batch normalization have no such clean description-length reading, and pretending otherwise is retrofitting.\nIt does not require your model class to contain the truth. Refined MDL is explicitly a worst-case-regret criterion relative to the class you brought, which makes it usable when you know your model is wrong — which is always.\nBoundary The coding scheme is a modelling choice. Crude two-part MDL is only as good as your encoding of models. Refined MDL and NML remove the arbitrariness at real computational cost, since the parametric complexity integral is often intractable. Bayes is the same thing in probability units. $-\\log P(M)$ is $L(M)$. Choosing a prior is choosing a code, and people who find priors philosophically troubling but code lengths obvious should notice they have committed to the same object. Solomonoff induction is the ideal limit. Weight hypotheses by $2^{-K}$ and you dominate every computable predictor, uncomputably. MDL is what you do instead, and the gap is exactly T023\u0026rsquo;s uncomputability. Compression as a benchmark. The Hutter Prize pays for compressing a Wikipedia snapshot, on the thesis that better compression is better understanding. Large language models are now evaluated by bits-per-byte, which is the same measurement — this connection is not metaphorical. Where it feeds forward. PAC learning (T096) prices hypothesis-class complexity by VC dimension instead of description length. Both bound generalization by a complexity term; MDL\u0026rsquo;s is in bits and VC\u0026rsquo;s is combinatorial, and the two agree qualitatively about what makes a class safe. Read next ","permalink":"https://cs.lozic.me/posts/t025-minimum-description-length-and-occams-razor/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eYou fit a model. It scores 94% on training data and 71% on held-out data.\u003c/p\u003e\n\u003cp\u003eSo you simplify: fewer parameters, more regularization. Training drops to 88%,\nheld-out rises to 84%. You simplify further and both drop. Somewhere in there\nwas an optimum, and you found it by trial and error, with a validation set and\npatience.\u003c/p\u003e","title":"Minimum Description Length and Occam's Razor"},{"content":"Symptom You have two files, each exactly one megabyte.\nThe first is a megabyte of the byte 0x00. The second is a megabyte from /dev/urandom. Gzip the first and you get a few hundred bytes. Gzip the second and you get slightly more than a megabyte, because the header costs something and there is nothing to exploit.\nShannon (T018) does not quite explain this. Entropy is a property of a distribution, and you do not have a distribution — you have two specific files. Asking \u0026ldquo;what is the entropy of this file\u0026rdquo; is a category error, and yet the intuition that one file is simple and the other is complicated is obviously correct and obviously about the files themselves.\nYou need a notion of information content that applies to a single object. That is what Kolmogorov complexity is, and having it changes how you think about randomness, compression, and simplicity.\nStatement The Kolmogorov complexity $K(x)$ of a string $x$ is the length of the shortest program that outputs $x$ and halts, on some fixed universal machine $U$ (T013):\n$$K(x) = \\min\\{|p| : U(p) = x\\}$$Three facts make it worth the trouble.\nInvariance. For universal machines $U$ and $V$, there is a constant $c$ depending on the pair but not on $x$ with $|K_U(x) - K_V(x)| \\le c$. Choice of language shifts every complexity by at most an additive constant, so $K$ is well-defined up to that constant.\nIncompressibility. For every $n$ and $k$, at least $2^n - 2^{n-k} + 1$ strings of length $n$ have $K(x) \\ge n - k$. Most strings are incompressible: at most one in $2^{k}$ can be compressed by $k$ bits, by counting.\nUncomputability. $K$ is not computable. No program takes $x$ and returns $K(x)$.\nDefine a string as random when $K(x) \\ge |x|$: it has no description shorter than itself. This is the definition of randomness for individual objects — not \u0026ldquo;produced by a random process,\u0026rdquo; which is a claim about history, but \u0026ldquo;has no structure to exploit,\u0026rdquo; which is a claim about the object.\nArgument Invariance. $U$ can simulate $V$: there is a fixed interpreter program of some length $c$ that reads a $V$-program and runs it. So any $V$-description of $x$ becomes a $U$-description of length at most $|p| + c$. Symmetrically the other way. The constant is the size of an interpreter, and that is all.\nThis is why $K$ is meaningful. Rewriting your compressor in a different language does not change what is compressible, only the fixed overhead.\nIncompressibility, by counting. Programs shorter than $n-k$ number at most $2^{n-k} - 1$. Each outputs at most one string. Strings of length $n$ number $2^n$. So at most $2^{n-k}-1$ of them have $K(x) \u0026lt; n-k$, leaving more than $2^n - 2^{n-k}$ with $K(x) \\ge n-k$. At $k = 10$, fewer than one in a thousand strings compresses by ten bits.\nThis is exactly the pigeonhole fact behind \u0026ldquo;no compressor compresses everything.\u0026rdquo; Any lossless compressor that shrinks some inputs must expand others. The counting argument is not a limitation of current compressors; it is arithmetic.\nUncomputability, via Berry\u0026rsquo;s paradox. Suppose $K$ were computable. Write a program: for a fixed integer $m$, enumerate strings in order and output the first $x$ with $K(x) \u0026gt; m$. Such an $x$ exists by counting. Call this program $P_m$; its length is $\\log m + c$ for some constant $c$ covering the enumeration and the call to $K$.\nBut $P_m$ outputs a string of complexity greater than $m$ using only $\\log m + c$ bits. For $m$ large enough that $\\log m + c \u0026lt; m$ — which holds for essentially every $m$ past a small threshold — we have a contradiction: $x$ is produced by a description shorter than $m$, so $K(x) \\le \\log m + c \u0026lt; m$.\nHence $K$ is not computable. This is \u0026ldquo;the smallest number not describable in fewer than twenty words,\u0026rdquo; made rigorous. It also reduces to the halting problem (T010): if you could compute $K$ you could solve halting, and conversely.\nA subtlety worth knowing: $K$ is upper semi-computable. You can compute better and better upper bounds — run all programs in parallel by dovetailing and record the shortest that has produced $x$ so far. The estimate decreases monotonically toward $K(x)$ and never tells you when it has arrived. Every real compressor is an upper bound on $K$ that halts.\nRelation to Shannon entropy. They agree on average. For a computable source with distribution $p$, the expected Kolmogorov complexity satisfies $H(p) \\le \\mathbb{E}[K(x)] \\le H(p) + c$ for a constant depending on the source description. Shannon measures the average over the distribution; Kolmogorov measures the individual object; and the two coincide in expectation. That is the bridge between T018 and this post, and it is the reason both notions deserve the name \u0026ldquo;information.\u0026rdquo;\nThe incompressibility method. Because most strings are incompressible, you can prove things about typical objects by assuming randomness and deriving a contradiction. Want a lower bound on the average time of a sorting algorithm? Argue that a fast run on an incompressible input yields a short description of that input. This turns the counting argument into a proof technique, and it routinely gives one-paragraph proofs of results that take pages otherwise.\nThe incompressibility method, in one example. Claim: any algorithm sorting $n$ distinct items by comparisons needs $\\Omega(n \\log n)$ comparisons on average. Take a permutation $\\pi$ with $K(\\pi) \\ge \\log_2(n!) - 10$, which exists because at most one permutation in $2^{10}$ is that compressible. The sequence of comparison outcomes during the sort is a bit string that, together with the algorithm\u0026rsquo;s fixed description, reconstructs $\\pi$ — replay the comparisons and you recover the input order. So the number of comparisons is at least $K(\\pi) - c \\ge \\log_2(n!) - c\u0026rsquo; = \\Omega(n \\log n)$ by Stirling.\nThe counting argument of T003 and this argument are the same argument, but the incompressibility version needs no decision-tree formalism and generalizes to average case for free, which is the sort of leverage the method gives.\nForbids A universal compressor that compresses every input. By counting, not by engineering. Any claimed universal compressor either loses data or expands some inputs, and no amount of cleverness changes this.\nComputing the true information content of your data. You cannot know the compression floor for a specific file; you can only exhibit compressors that get close from above.\nA finite test for randomness. No program decides whether a string is Kolmogorov-random. Statistical tests check specific regularities and nothing more, so \u0026ldquo;passed the test suite\u0026rdquo; is not \u0026ldquo;random.\u0026rdquo;\nRecursive compression. Compressing already-compressed data cannot keep gaining. If output were still compressible, the composition would be a shorter program, so good compression output looks random by construction.\nDoes not forbid It does not make compression useless — real data is deeply non-random. Text, images, logs and source code sit in a vanishingly small structured corner of string space. zstd, Brotli and LZMA exploit exactly that. The theorem says most strings are incompressible, and real files are not typical strings; that gap is the entire compression industry.\nIt does not stop practical randomness testing. Dieharder, TestU01 and the NIST suite are used seriously to evaluate PRNGs, and they catch real defects. They cannot certify randomness, but they falsify it, and falsification is what you need to reject a bad generator.\nIt does not mean normalized compression distance cannot work. NCD — using gzip output sizes to approximate the information distance between two objects — does plagiarism detection, phylogeny reconstruction from genomes, and language clustering, and it does them well. The uncomputable quantity is approximated from above by a real compressor, and the approximation is good enough to be useful. A 2023 paper even got competitive text classification from gzip plus k-nearest-neighbours, beating some neural baselines on low-resource tasks.\nIt does not say cryptographic randomness is unattainable. /dev/urandom and ChaCha20-based CSPRNGs produce output that is computationally indistinguishable from random, which is not Kolmogorov randomness — the output has a short description, namely the seed — but it is the right notion for security. The two definitions of \u0026ldquo;random\u0026rdquo; answer different questions, and conflating them causes real confusion.\nIt does not make the constant irrelevant in practice. The invariance constant is asymptotic. For short strings, language choice dominates completely, which is why \u0026ldquo;the Kolmogorov complexity of this 20-byte string\u0026rdquo; is not a meaningful quantity, and why practical applications always work with large objects.\nBoundary Prefix complexity $K$ versus plain complexity $C$. Requiring the program set to be prefix-free makes Kraft\u0026rsquo;s inequality apply and the theory much cleaner, at the cost of $O(\\log n)$ additive terms. It is what makes algorithmic probability and Chaitin\u0026rsquo;s Omega well-defined. Conditional complexity $K(x \\mid y)$ is the shortest program producing $x$ given $y$ for free. Information distance and NCD are built on it, and it is the algorithmic analogue of conditional entropy. Time-bounded complexity $K^t$. Restrict to programs running within $t$ steps and you get a computable quantity. That is what real compressors approximate, and the theory of $K^t$ connects directly to pseudorandomness and to the barriers in T030. Levin\u0026rsquo;s universal search. Dovetail over all programs, giving program $p$ a fraction $2^{-|p|}$ of the time. This solves any inverse problem within a constant factor of optimal, and the constant is astronomically bad. It is the cleanest example of a theoretically optimal, practically useless algorithm. Solomonoff induction. Weight every hypothesis by $2^{-K}$ and you get a universal predictor that dominates every computable predictor — uncomputable, and the formal ancestor of Occam\u0026rsquo;s razor, which is the next post\u0026rsquo;s subject. Read next ","permalink":"https://cs.lozic.me/posts/t023-kolmogorov-complexity/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eYou have two files, each exactly one megabyte.\u003c/p\u003e\n\u003cp\u003eThe first is a megabyte of the byte \u003ccode\u003e0x00\u003c/code\u003e. The second is a megabyte from\n\u003ccode\u003e/dev/urandom\u003c/code\u003e. Gzip the first and you get a few hundred bytes. Gzip the second\nand you get slightly more than a megabyte, because the header costs something and\nthere is nothing to exploit.\u003c/p\u003e","title":"Kolmogorov Complexity"},{"content":"Symptom You have read T026 and T027 and you are wondering the obvious thing.\nFifty years. Thousands of researchers. A million-dollar prize. And P vs NP has not moved. That is strange — most famous problems yield partial results, near misses, special cases that suggest the shape of the answer. Here there is almost nothing. We cannot even prove that NP requires more than linear-size circuits, which is a laughably weak statement compared to what everyone believes.\nIs the problem just very hard, or is something structurally wrong with the way we attack it?\nIt is the second, and we know this precisely, because there are theorems about it. Two of them, and they kill different things. This is the rare case where a field has mapped its own failure and can tell you exactly which tools cannot possibly work.\nStatement The relativization barrier (Baker–Gill–Solovay, 1975). There exist oracles $A$ and $B$ with $\\text{P}^A = \\text{NP}^A$ and $\\text{P}^B \\ne \\text{NP}^B$. Consequently, any proof technique that goes through unchanged when both classes are given the same oracle cannot resolve P vs NP — it would have to prove both outcomes.\nThe natural proofs barrier (Razborov–Rudich, 1994). Call a circuit lower bound argument natural if it proceeds by exhibiting a property of Boolean functions that is (i) constructive — checkable in time polynomial in the truth table — and (ii) large — held by a substantial fraction of random functions. If strong pseudorandom generators exist, no natural property can separate P from NP.\nThe sting is the hypothesis. Pseudorandom generators of the required strength follow from standard cryptographic assumptions, so the natural-proofs barrier says: if cryptography works, the obvious way to prove circuit lower bounds does not. Believing hardness is what blocks proving hardness.\nAlgebrization (Aaronson–Wigderson, 2008) closes the remaining gap: the arithmetization techniques that beat relativization, which gave us IP = PSPACE, still fail against a slightly extended notion of oracle access.\nThree barriers, and the known techniques fall into their union.\nArgument Relativization, via oracles. An oracle is a black box answering membership queries in one step. $\\text{P}^A$ is what polynomial machines decide with free access to $A$. Most classical arguments — diagonalization, simulation — do not care what the machine does internally, so they hold verbatim when both classes get the same oracle. Baker, Gill and Solovay showed this is fatal.\nAn oracle collapsing them. Let $A$ be any PSPACE-complete language. With $A$ in hand, a polynomial machine can decide anything in PSPACE, and PSPACE is closed under the nondeterministic guessing NP does, so $\\text{P}^A = \\text{NP}^A = \\text{PSPACE}$. One oracle, both classes flattened.\nAn oracle separating them. Build $B$ by diagonalization. Let $L_B = {1^n : B \\text{ contains some string of length } n}$. Clearly $L_B \\in \\text{NP}^B$: guess the string, query it. Now construct $B$ in stages against an enumeration of polynomial-time oracle machines. At stage $i$, pick a length $n$ larger than anything decided so far, run machine $M_i$ on $1^n$, and watch: it makes at most $n^{k}$ queries, but there are $2^n$ strings of length $n$, so for large $n$ it must leave one unqueried. If $M_i$ accepts, put no length-$n$ string in $B$; if it rejects, add an unqueried one. Either way $M_i$ is wrong, and it cannot tell, because its answers do not change. So $L_B \\notin \\text{P}^B$.\nThe technique is a counting argument: a polynomial machine cannot look at exponentially many places, so we hide the answer where it did not look. This is exactly T005\u0026rsquo;s adversary argument, applied to oracles.\nWhy this indicts our tools. Cook–Levin\u0026rsquo;s tableau is a simulation and relativizes. The hierarchy theorems are diagonalization and relativize. Everything in the classical toolkit relativizes, so none of it can settle P vs NP.\nNatural proofs, and why the plausible route is blocked. Circuit lower bounds do not relativize, so after 1975 that was the road. It worked at first: Razborov proved exponential lower bounds for monotone circuits, Håstad\u0026rsquo;s switching lemma gave parity\u0026rsquo;s lower bound for AC⁰ (T034). Then it stopped, and progress against general circuits ceased entirely.\nRazborov and Rudich asked what those successful proofs had in common. All worked by exhibiting a property $P$ of Boolean functions: hard functions have $P$, and functions computed by small circuits do not. Both features are natural in practice. The property must be checkable or you cannot argue with it, and it must be common among random functions, because random functions are hard and any property capturing hardness will catch most of them.\nNow the punch. A pseudorandom function family is one no efficient test distinguishes from truly random. Suppose you have a constructive, large property $P$ separating hard functions from easy ones. Then $P$ is an efficient distinguisher: pseudorandom functions are computed by small circuits, so they lack $P$; truly random functions have it with substantial probability. Compute $P$ on the truth table and you have broken the generator.\nSo a natural proof of strong circuit lower bounds breaks cryptography. Since we believe cryptography holds — and its security rests on the same hardness assumptions the lower bound would establish — we believe no natural proof exists. The two beliefs are the same belief, pointed in opposite directions.\nSavitch\u0026rsquo;s theorem, which shows the barriers are not universal. $\\text{NSPACE}(f(n)) \\subseteq \\text{DSPACE}(f(n)^2)$: nondeterminism buys at most a squaring in space, so NPSPACE = PSPACE. The proof is a recursive reachability test — to check configuration $c_1$ reaches $c_2$ in $2^k$ steps, guess the midpoint and recurse on both halves, reusing the same space, giving depth $O(f(n)^2)$. This is exactly the space analogue of the P vs NP question, and it is settled. Space is reusable and time is not, and that difference is why one question fell in 1970 while the other did not. It is also the sharpest available answer to \u0026ldquo;maybe the question is simply unanswerable.\u0026rdquo;\nForbids Any relativizing proof of P ≠ NP or P = NP. Diagonalization and simulation alone, in every arrangement, are ruled out. Most amateur proofs die here.\nNatural circuit lower bounds against P/poly, if strong PRGs exist. The approach that produced every circuit lower bound we have cannot be pushed to the general case.\nArithmetization alone, by algebrization. The technique that beat relativization is itself bounded, and its bound is now known.\nThe expectation of a short elementary proof. Any resolution must be non-relativizing, non-naturalizing, and non-algebrizing simultaneously. Nothing in the standard toolkit qualifies, so a proof will require a genuinely new idea.\nDoes not forbid They do not say P vs NP is unprovable or independent of ZFC. This is the common misreading, and it is wrong. The barriers constrain techniques, not truth. There is no evidence of independence, and unlike the continuum hypothesis, no forcing construction or model-theoretic obstruction is known.\nThey do not stop lower bounds from being proved. Parity is not in AC⁰ (T034), unconditionally. Razborov\u0026rsquo;s monotone bound is unconditional. Ryan Williams proved NEXP $\\not\\subseteq$ ACC⁰ in 2011 by combining circuit analysis with a faster-than-brute-force satisfiability algorithm — explicitly non-natural, and the clearest existence proof that barrier-evading arguments exist. There is also a well-known unconditional time-space tradeoff showing SAT cannot be solved in $n^{1.8}$ time and logarithmic space simultaneously.\nThey do not make the barriers themselves useless. Razborov–Rudich is a specification for a working proof: your property must be non-constructive or rare. Williams\u0026rsquo;s programme deliberately targets that gap, and \u0026ldquo;which barrier does this evade\u0026rdquo; is now a standard question asked of any new technique.\nThey do not undermine complexity theory\u0026rsquo;s practical results. NP-hardness reductions (T028) are unaffected — those are proofs within the theory, not separations of classes. Everything you use complexity theory for at work is untouched.\nThey do not imply the community is stuck for lack of effort or ideas. Interactive proofs, PCP (T031), and the whole hardness-versus-randomness line came out of these decades. The field moved a great deal; it moved sideways relative to the flagship question.\nBoundary The barriers are about the general case. Restricted circuit classes — monotone, constant-depth, bounded fan-in, ACC⁰ — remain open ground, and that is where lower-bound progress actually happens. Williams\u0026rsquo;s route: algorithms as lower bounds. A faster-than-brute-force algorithm for circuit satisfiability implies a circuit lower bound. Turning algorithm design into lower-bound proof is the most promising known barrier-evading direction. Non-uniformity is a real gap. P/poly permits a different circuit per input length, including circuits computing undecidable languages. NP $\\not\\subseteq$ P/poly is stronger than P ≠ NP, and non-uniform lower bounds are where the circuit programme lives. Proof complexity as the other flank. Lower bounds on proof length in specific systems — resolution, cutting planes, Frege — are a concrete sub-goal, since NP ≠ coNP would follow from super-polynomial lower bounds for every propositional proof system. That programme has genuine unconditional results. Savitch is the standing counterexample to fatalism. The space version of the question is solved, by an argument two paragraphs long, exploiting a resource property time does not have. Knowing why space was easy is the best guide to what a time proof would need. Read next ","permalink":"https://cs.lozic.me/posts/t030-the-barriers-relativization-and-natural-proofs/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eYou have read T026 and T027 and you are wondering the obvious thing.\u003c/p\u003e\n\u003cp\u003eFifty years. Thousands of researchers. A million-dollar prize. And P vs NP has\nnot moved. That is strange — most famous problems yield partial results, near\nmisses, special cases that suggest the shape of the answer. Here there is\nalmost nothing. We cannot even prove that NP requires more than linear-size\ncircuits, which is a laughably weak statement compared to what everyone\nbelieves.\u003c/p\u003e","title":"The Barriers: Relativization and Natural Proofs"},{"content":"Symptom Three weeks into a project, you are still trying to write an exact algorithm.\nThe problem is yours and it looks specific: assign delivery vans to routes such that every stop is covered, no van exceeds its capacity, and the total distance is minimized. Nothing in the literature matches it exactly. So you keep going — better data structures, smarter pruning, a cleverer greedy pass with a repair step — and each version works on your test set and falls over on production data.\nNobody told you the search was hopeless, because from the inside it never looks hopeless. Each failure suggests a specific fix. What you needed on day two was the ability to look at the problem and say: that is set cover with a knapsack constraint, that is NP-hard, exact-and-fast is not available, and the real question is which of exactness, speed, or generality I am giving up.\nKarp\u0026rsquo;s paper is where the profession acquired that ability.\nStatement Twenty-one natural combinatorial problems are NP-complete. Karp gave polynomial-time reductions from SAT to each, arranging them in a tree so that each reduction starts from a problem proved complete earlier on the page.\nThe list is the founding catalogue: satisfiability, 0-1 integer programming, clique, set packing, vertex cover, set covering, feedback node set, feedback arc set, directed and undirected Hamiltonian circuit, 3-SAT, chromatic number, clique cover, exact cover, hitting set, Steiner tree, 3-dimensional matching, knapsack, job sequencing, partition, and max cut.\nThe content is not any single one of them. It is that these twenty-one — drawn from logic, graph theory, scheduling, number theory, and operations research, studied for decades by communities that did not talk to each other — are all the same problem wearing different clothes. Solve any one in polynomial time and you solve all of them, plus everything else in NP.\nThe catalogue has since grown past several thousand problems. Garey and Johnson\u0026rsquo;s 1979 book listed roughly three hundred and is still the standard reference.\nArgument Cook–Levin (T027) supplies the first complete problem. Everything after that is transitivity: if $A \\le_p B$ and $B$ is NP-hard, then $B$\u0026rsquo;s hardness transfers to anything $B$ reduces to. So the work is entirely in exhibiting reductions, and the technique is worth seeing in miniature.\n3-SAT $\\le_p$ Independent Set. Given a 3-CNF formula with $m$ clauses, build a graph: one vertex per literal occurrence, so $3m$ vertices. Connect the three vertices within each clause into a triangle, and connect every vertex to every vertex holding its negation. Ask for an independent set of size $m$.\nA triangle admits at most one vertex from each clause, so an independent set of size $m$ picks exactly one literal per clause. The negation edges guarantee it never picks both $x$ and $\\neg x$. So an independent set of size $m$ is exactly a choice of one true literal per clause with no contradictions — a satisfying assignment. Conversely, any satisfying assignment gives such a set. The graph is built in linear time.\nThat is the whole shape: encode the choices as vertices, the constraints as edges, and let the target problem\u0026rsquo;s objective do the counting.\nIndependent Set $\\le_p$ Vertex Cover. $S$ is independent exactly when its complement is a vertex cover, because an edge with both endpoints outside the cover would be an edge inside $S$. So ask for a cover of size $n - k$. The reduction is a subtraction — the same problem, read from the other side.\nVertex Cover $\\le_p$ Set Cover. Elements are edges, and each vertex contributes the set of edges it touches. Covering all elements is covering all edges. A change of vocabulary, nothing more.\n3-SAT $\\le_p$ Hamiltonian Circuit is where the gadgets get real: each variable becomes a long two-way \u0026ldquo;highway\u0026rdquo; traversable left-to-right or right-to-left (true or false), each clause becomes a node that can be visited from any of its three literals\u0026rsquo; highways, and a Hamiltonian circuit exists exactly when every clause node can be picked up by some true literal. Nobody guesses this in an afternoon, which is precisely why the catalogue is valuable: somebody already did it.\nWhy a tree rather than twenty-one proofs from SAT. Reducing knapsack from SAT directly would be brutal; reducing it from partition-flavoured problems is easy. The structure of Karp\u0026rsquo;s paper is the argument — a well-chosen intermediate makes the next reduction short, and building the catalogue in dependency order is what keeps each step checkable by eye.\nThe practical skill this buys. Once you have seen fifteen of these, you stop proving and start recognizing. \u0026ldquo;Choose one from each group, subject to pairwise conflicts\u0026rdquo; is 3-SAT or graph colouring. \u0026ldquo;Cover everything using fewest sets\u0026rdquo; is set cover. \u0026ldquo;Pack items under a budget\u0026rdquo; is knapsack. Your van-routing problem contains an exact cover and therefore will not have an exact fast algorithm, and you know this in ten minutes rather than three weeks.\nForbids A polynomial exact algorithm for any of the twenty-one, unless P = NP. Including the ones that look easy. Partition — split a multiset into two equal halves — has a two-line statement and is NP-complete.\nThe assumption that a small change makes a problem tractable. 2-SAT is in P but 3-SAT is complete; 2-colouring is in P but 3-colouring is complete; shortest path is in P but longest path is complete. The tractability boundary sits between neighbours, so \u0026ldquo;mine is only slightly different\u0026rdquo; is not evidence.\nAny claimed general fast solver for one of these. Such a claim is a claim of P = NP, and gets evaluated at that price.\nThe hope that unfamiliarity means unexplored. Your specific problem is almost certainly a known one with a gadget wrapped around it. Searching the catalogue is faster than searching for an algorithm.\nDoes not forbid It does not stop production systems from solving these daily, at scale. Concorde has solved TSP instances with tens of thousands of cities to proven optimality, including an 85,900-city instance in 2006. CPLEX and Gurobi solve integer programs — Karp\u0026rsquo;s second problem — with millions of variables across logistics and airline scheduling. NP-hardness is a worst-case statement about an infinite family, and your instances are drawn from a structured distribution that branch-and-cut exploits.\nIt does not mean approximation is hopeless. Vertex cover has a trivial 2-approximation (take both endpoints of a maximal matching). Knapsack has an FPTAS: any accuracy you like, in time polynomial in $1/\\epsilon$. Set cover has a $\\ln n$-approximation from the greedy algorithm. That is T032\u0026rsquo;s subject, and the approximability landscape is wildly uneven across a list of problems that are all equivalent for exact solution.\nIt does not mean every instance is hard, or even that hard instances are common. Graph colouring is NP-complete, but register allocation in LLVM and GCC colours interference graphs constantly with a linear-scan or Chaitin-style heuristic, and real interference graphs are chordal or nearly so. The theorem describes the worst adversarial input, and your compiler is not facing an adversary.\nIt does not mean the parameters do not matter. Vertex cover is NP-complete but solvable in $O(1.28^k + kn)$ time for cover size $k$, which is fast whenever $k$ is small even if $n$ is enormous. \u0026ldquo;NP-hard\u0026rdquo; flattens a distinction that parameterized complexity (T033) restores, and it is the single most useful refinement of the theorem for practitioners.\nIt does not say your problem is hard just because it resembles one on the list. Hardness transfers along reductions in one direction only. Matching is in P even though it looks like set packing; 2-SAT is in P even though it looks like 3-SAT; min-cut is in P even though max-cut is complete. You have to actually check which side you are on, and the pairs that straddle the line are the ones worth memorizing.\nBoundary What the catalogue is really for. Not for proving things. For recognizing them. The value is a trained eye, which is why reading the reductions beats reading the list. The min-cut/max-cut asymmetry. Minimizing a cut is polynomial; maximizing it is complete. Same object, opposite direction, opposite complexity. This is the single best cure for the intuition that structurally similar problems have similar difficulty. Weak versus strong NP-hardness. Knapsack has a pseudo-polynomial $O(nW)$ dynamic program, which is fine when weights are small integers and exponential in the input\u0026rsquo;s bit length. TSP is strongly NP-hard and has no such escape. The distinction tells you whether \u0026ldquo;small numbers\u0026rdquo; is a usable restriction. Special structures that collapse hardness. Many of the twenty-one become polynomial on trees, on bounded-treewidth graphs, on planar graphs (often via a PTAS), or on interval graphs. Checking whether your instances have structure is usually more productive than checking whether your heuristic can be tuned. Optimization versus decision. Karp\u0026rsquo;s problems are decision problems, but self-reducibility means a decision oracle gives you the optimum by binary search plus fixing choices one at a time. That equivalence is why nobody worries about the distinction in practice. Read next ","permalink":"https://cs.lozic.me/posts/t028-karps-21-problems/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eThree weeks into a project, you are still trying to write an exact algorithm.\u003c/p\u003e\n\u003cp\u003eThe problem is yours and it looks specific: assign delivery vans to routes such\nthat every stop is covered, no van exceeds its capacity, and the total distance\nis minimized. Nothing in the literature matches it exactly. So you keep going —\nbetter data structures, smarter pruning, a cleverer greedy pass with a repair\nstep — and each version works on your test set and falls over on production\ndata.\u003c/p\u003e","title":"Karp's 21 Problems"},{"content":"Symptom You know reductions (T017): to prove your problem hard, reduce a known-hard problem to it. Fine. But that begs the obvious question, and it is the question a sharp colleague asks the first time you use the technique.\nWhere does the first hard problem come from?\nEvery hardness proof in the literature is a chain of reductions, and chains need a first link. If every NP-hardness result is \u0026ldquo;reduce from something already known NP-hard,\u0026rdquo; the entire edifice is circular unless somebody, once, proved a problem NP-hard from the definition. Somebody did, twice independently — Cook in Toronto in 1971 and Levin in Moscow, working in isolation behind the Iron Curtain — and the argument is the technical heart of the field.\nThe practical version of the symptom is quieter: you are wondering why SAT solvers are the universal tool. Why does everyone encode scheduling, verification, planning, and dependency resolution into Boolean satisfiability rather than building a specialized search? The answer is this theorem. SAT is not one hard problem among many. It is the problem that is computation, written down.\nStatement SAT is NP-complete. That is, SAT is in NP, and every language in NP reduces to SAT in polynomial time.\n3-SAT is NP-complete too, where every clause has exactly three literals, which is the version everyone actually reduces from.\nUnpacking \u0026ldquo;every language in NP reduces to SAT\u0026rdquo;: for any problem with efficiently checkable certificates — any problem at all, including ones nobody has thought of — there is a polynomial-time procedure converting its instances into Boolean formulas that are satisfiable exactly when the original instance is a yes. Boolean logic is expressive enough to encode arbitrary bounded computation, and that is the real content.\nThe consequence for practice: a polynomial SAT algorithm gives a polynomial algorithm for everything in NP. This is why SAT is the reduction target of choice in both directions — to prove hardness you reduce from it, and to solve your problem you reduce to it and call a solver.\nArgument SAT is in NP. The certificate is the satisfying assignment; checking it is one pass over the formula. Trivial, and worth stating only because completeness requires both halves.\nEvery NP problem reduces to SAT: the tableau construction. Let $L \\in$ NP, decided by a nondeterministic machine $M$ in time $n^k$. Given input $w$, we build a formula satisfiable exactly when $M$ has an accepting computation.\nPicture the computation as a table — Cook\u0026rsquo;s tableau — with $n^k$ rows (one per time step) and $n^k$ columns (one per tape cell, since in $n^k$ steps the head cannot move further). Each cell holds a tape symbol, or a state marker if the head is there. An accepting computation is precisely a correctly filled table.\nIntroduce a Boolean variable $x_{i,j,s}$ meaning \u0026ldquo;cell $(i,j)$ contains symbol $s$.\u0026rdquo; There are polynomially many. Now write four groups of clauses:\nWell-formed. Each cell holds exactly one symbol: at least one ($\\bigvee_s x_{i,j,s}$) and at most one ($\\neg x_{i,j,s} \\vee \\neg x_{i,j,t}$ for $s \\ne t$). Start. Row 1 encodes the initial configuration: start state, input $w$, blanks after. Accept. Some cell somewhere contains the accepting state. Move. This is the only interesting one. Every consecutive pair of rows must be related by a legal transition of $M$. The local-window trick, which is the whole idea. Checking that row $i+1$ legally follows row $i$ sounds global, but it is not: a Turing machine changes only the cell under the head and its immediate neighbourhood. So it suffices to check every $2 \\times 3$ window — two rows, three adjacent columns — independently. A window is legal if its contents are consistent with some transition of $M$, and legality is a finite condition over a finite alphabet, so it is a constant-size Boolean formula. If every window is legal, the whole transition is legal; if any is not, it is not.\nThat local check is the reason the reduction is polynomial. Computation is a local process — each step touches a bounded neighbourhood — and locality is exactly what a conjunction of small clauses expresses. The formula has $O(n^{2k})$ variables and clauses, and is constructible in polynomial time by a straightforward program that never runs $M$ at all.\nSatisfying assignments correspond exactly to accepting computations. So $w \\in L \\iff \\phi_w \\in \\text{SAT}$.\nFrom SAT to 3-SAT. Convert each clause to CNF, then split long clauses using fresh variables: $(a \\vee b \\vee c \\vee d)$ becomes $(a \\vee b \\vee z) \\wedge (\\neg z \\vee c \\vee d)$, which is satisfiable exactly when the original is. Repeat. Linear blowup, and now every clause has three literals. The Tseytin transformation does the same job for arbitrary circuits in linear size, and it is what every real encoder uses.\nLevin\u0026rsquo;s version. Levin proved essentially the same result independently and published a short abstract in 1973, framing it in terms of search problems rather than decision problems — closer to what people actually want. The Soviet and Western literatures met later, and the joint name is the field\u0026rsquo;s acknowledgement of a genuinely independent discovery.\nWhy this made the catalogue possible. With one NP-complete problem in hand, transitivity does the rest. Karp\u0026rsquo;s 1972 paper (T028) reduced SAT to 21 problems in one stroke, and the count is now in the thousands. Everything downstream is this theorem plus composition.\nForbids A polynomial SAT algorithm that does not collapse NP into P. SAT\u0026rsquo;s completeness means solving it solves everything in NP. Any claimed polynomial-time SAT algorithm is a claim of P = NP.\nProving your problem is not NP-hard by showing it is unlike SAT. Superficial dissimilarity is not evidence; the tableau construction shows that utterly unrelated-looking problems encode the same thing.\nEfficiently deciding satisfiability of general circuits, unless P = NP. Circuit-SAT is NP-complete, which is why symbolic execution and equivalence checking have exponential worst cases, and why your verification tool sometimes hangs.\nA general polynomial procedure for finding proofs of bounded length. Cook\u0026rsquo;s original motivation: proof search for bounded-length proofs is NP-complete, so automated theorem proving is hard in the worst case by this theorem specifically.\nDoes not forbid It does not make SAT hard in practice, and the gap is enormous. MiniSat, Kissat, and CaDiCaL routinely solve industrial instances with millions of variables. CDCL — conflict-driven clause learning, with unit propagation, VSIDS activity heuristics, watched literals, and restarts — exploits structure that worst-case theory cannot see. SAT is the backbone of hardware equivalence checking at Intel, of apt and Dart\u0026rsquo;s pub dependency resolution, and of symbolic execution engines. The theorem tells you SAT is universal, and universality is precisely why investing in one solver pays off across every domain.\nIt does not mean all SAT instances are equally hard. Random 3-SAT has a sharp phase transition at a clause-to-variable ratio around 4.27: below it instances are almost all satisfiable and easy, above it almost all unsatisfiable and easy to refute, and only near the threshold are they hard. Industrial instances are almost never near it, which is a large part of why solvers work.\nIt does not make every restricted SAT hard. 2-SAT is in P via implication graphs and strongly connected components (linear time), Horn-SAT is in P by unit propagation and is the basis of Datalog and Prolog inference, and XOR-SAT is Gaussian elimination. Schaefer\u0026rsquo;s dichotomy theorem is the sharp statement: every Boolean constraint satisfaction problem is either in P or NP-complete, with exactly six tractable cases and nothing in between. That is an unusually complete answer to \u0026ldquo;which restrictions help.\u0026rdquo;\nIt does not mean encoding to SAT is always the right move. SMT solvers keep theories — arithmetic, arrays, bit-vectors, uninterpreted functions — at the native level instead of bit-blasting, and Z3 usually beats a hand-rolled SAT encoding badly on such problems. Integer programming solvers beat SAT on optimization. The reduction proves possibility; the engineering question of which encoding to use is separate and often goes the other way.\nIt does not say anything about approximation or counting. Cook–Levin is about decision. Counting satisfying assignments (#SAT) is #P-complete and strictly harder than deciding, which is Toda\u0026rsquo;s theorem territory, and it matters because probabilistic inference is a counting problem rather than a decision one.\nBoundary What made SAT the right universal problem. The tableau is a table of local constraints, and CNF is the natural language of local constraints. Any formalism expressive enough to state \u0026ldquo;these adjacent cells are consistent\u0026rdquo; would work, which is why tiling problems and circuit value problems are also complete. The Tseytin transformation is the practical version. Naive CNF conversion is exponential; Tseytin introduces a variable per gate and produces linear-size equisatisfiable CNF. Everyone who encodes to SAT uses it, and knowing it exists is what makes reduction-to-SAT a practical technique rather than a theoretical one. The Karp–Levin distinction. Cook used Turing reductions, Karp used many-one, and the modern definition follows Karp because many-one reductions give a finer theory. The naming convention \u0026ldquo;Karp reduction\u0026rdquo; comes from here. Where the tableau reappears. The same construction proves the time-hierarchy-flavoured completeness results for other classes: bounded-space computation gives PSPACE-completeness of QBF, and alternating quantifiers give the polynomial hierarchy. The technique generalizes cleanly, which is another sign SAT was not an arbitrary choice. The barriers apply here too (T030). The tableau argument is a simulation, so it relativizes. That is fine for proving completeness and useless for separating classes, which is exactly the dividing line the barrier results draw. Read next ","permalink":"https://cs.lozic.me/posts/t027-the-cook-levin-theorem/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eYou know reductions (T017): to prove your problem hard, reduce a known-hard\nproblem to it. Fine. But that begs the obvious question, and it is the question\na sharp colleague asks the first time you use the technique.\u003c/p\u003e","title":"The Cook–Levin Theorem"},{"content":"Symptom You are handed a scheduling problem. Two hundred tasks, precedence constraints, shared resources, and a deadline. Somebody wants an optimal schedule.\nChecking a proposed schedule takes seconds: walk the list, verify each constraint, add up the makespan. Finding one is different. Your search runs for a day and finds nothing. You try simulated annealing, then a genetic algorithm, then branch and bound. Each gives good schedules and none gives the best one, and you cannot tell whether the problem is hard or you are not clever enough.\nThat gap — between the ease of checking and the difficulty of finding — is the most important open question in computer science, and it is not a curiosity about theoretical machines. It is the shape of your week. The practical question underneath is: when should you stop looking for an exact algorithm? The answer depends on a conjecture nobody has proved, and the professional skill is knowing how to act while it stays unproved.\nStatement P is the class of decision problems solvable by a deterministic Turing machine in time polynomial in the input size.\nNP is the class of decision problems whose yes instances have a polynomial-size certificate verifiable in polynomial time. Equivalently, those solvable in polynomial time by a nondeterministic machine.\nThe question: does P = NP?\n$\\text{P} \\subseteq \\text{NP}$ is immediate — if you can solve it you can verify it by ignoring the certificate and re-solving. The question is whether the containment is strict. Nobody knows, and the Clay Institute will pay a million dollars for either answer.\nNote the framing that makes it feel less abstract: P is \u0026ldquo;solvable,\u0026rdquo; NP is \u0026ldquo;recognizable-when-shown.\u0026rdquo; Sudoku, factoring, protein folding, circuit design, theorem proving, and your scheduling problem are all NP: a solution is easy to check. The conjecture $\\text{P} \\ne \\text{NP}$ says that for some of them, finding is genuinely harder than checking — that there is no shortcut past search.\nAn NP-complete problem is one in NP to which every NP problem reduces in polynomial time (T017). These are the hardest problems in NP, and there are thousands. A polynomial algorithm for any single one collapses the whole class.\nArgument There is no proof to present, so what follows is the case for the conjecture and the reasons it has resisted.\nWhy almost everyone believes $\\text{P} \\ne \\text{NP}$. Not proof, but the evidence is substantial. Fifty years of intense effort on thousands of NP-complete problems has produced no polynomial algorithm for any of them, and these are problems with enormous commercial value. The asymmetry between verifying and searching feels fundamental: appreciating a proof is not the same skill as finding one, and if P = NP then it is, in a precise sense. Aaronson\u0026rsquo;s formulation is that P = NP would mean \u0026ldquo;anyone who could appreciate a symphony could write one,\u0026rdquo; and the world does not appear to work that way.\nA 2019 poll of complexity theorists found 88% believing P ≠ NP. That is not evidence about mathematics but it is evidence about where informed effort is being spent.\nWhy it is hard: the barriers (T030). Three proof techniques have been ruled out, which is unusual and is the real reason progress stalled.\nRelativization (Baker–Gill–Solovay, 1975). There are oracles $A$ and $B$ with $\\text{P}^A = \\text{NP}^A$ and $\\text{P}^B \\ne \\text{NP}^B$. Any proof that works by simulation — as the halting proof and the time hierarchy theorem do — would relativize, so it would have to hold for both oracles. It cannot. Every technique inherited from computability theory is dead.\nNatural proofs (Razborov–Rudich, 1994). Circuit lower bound techniques that are \u0026ldquo;constructive\u0026rdquo; and \u0026ldquo;large\u0026rdquo; — applying to most functions and checkable efficiently — would break pseudorandom generators. If strong one-way functions exist, which is roughly what we believe, then no natural proof separates P from NP. Most circuit-complexity methods are natural.\nAlgebrization (Aaronson–Wigderson, 2008). The algebraic techniques that resolved IP = PSPACE relativize in an extended algebraic sense too, and are therefore also insufficient.\nSo the field knows three large families of techniques that cannot work, and does not know what the fourth family looks like. This is a much more specific situation than \u0026ldquo;the problem is hard.\u0026rdquo;\nWhat a proof of P = NP would change. If constructive with a small exponent, essentially everything. Public-key cryptography dies, since breaking RSA and finding discrete logs are in NP. Optimal scheduling, routing, circuit layout, protein structure prediction, and drug design become tractable. Mathematics changes character: finding a proof of length $n$ becomes a polynomial search, so theorem proving is automated up to the length bound. This is why the result would be transformative rather than merely important, and also why it is implausible.\nIf non-constructive, or constructive with an exponent of 100, nothing changes practically and the world learns something profound.\nWhat a proof of P ≠ NP would change. Directly, less than people expect. We would have proved what we already assume. But the techniques needed would almost certainly be worth more than the theorem — the same way Wiles\u0026rsquo;s proof of Fermat mattered mostly for the modularity machinery — and cryptography would gain a necessary condition it currently lacks. Note that P ≠ NP does not by itself give secure cryptography: crypto needs average-case hardness, and P ≠ NP is a worst-case statement.\nForbids Strictly, an open problem forbids nothing. What the surrounding theory forbids is worth stating, because it is what you actually use:\nA polynomial algorithm for one NP-complete problem without collapsing all of them. By reduction closure, one is all. So a claimed polynomial SAT algorithm is a claim of P = NP, and should meet that bar of scrutiny.\nProving P ≠ NP by any relativizing, natural, or algebrizing argument. If your proof strategy simulates machines, or produces an efficiently checkable property of hard functions, it is already known to fail. This has saved enormous amounts of wasted effort and is the practical value of the barrier results.\nEscaping NP-hardness by restricting to a subclass the reduction still reaches. If 3-SAT reduces to your problem using only instances your restriction permits, you have not escaped.\nDoes not forbid It does not say NP-hard problems are unsolvable in practice, and treating it that way is the expensive mistake. SAT solvers dispatch industrial instances with millions of variables daily; Z3 and CVC5 are load-bearing in verification, symbolic execution, and dependency resolution. CPLEX and Gurobi solve integer programs with hundreds of thousands of variables. Real instances have structure — low treewidth, backbone variables, community structure — that worst-case theory does not see. \u0026ldquo;It\u0026rsquo;s NP-complete\u0026rdquo; is a reason to reach for a solver, never a reason to stop.\nIt does not preclude good approximations. Many NP-hard optimization problems have polynomial approximation schemes. Metric TSP has a 1.5-approximation (Christofides) and, since 2020, slightly better; knapsack has an FPTAS; vertex cover has a trivial 2-approximation. Whether a good approximation exists is a separate question answered by the PCP theorem (T031), and the answers vary by problem: max-cut is approximable to 0.878, and max-clique is not approximable to within $n^{1-\\varepsilon}$.\nIt does not mean NP-complete problems are the hardest problems. PSPACE, EXPTIME, and the undecidable sit above. Generalized chess is EXPTIME-complete; halting is not in NP at all. NP-complete is the bottom of the hard problems, not the top, and treating \u0026ldquo;NP-hard\u0026rdquo; as a synonym for \u0026ldquo;impossible\u0026rdquo; mislabels the difficulty.\nIt does not settle whether NP-intermediate problems exist. Ladner\u0026rsquo;s theorem says that if P ≠ NP then there are problems in NP that are neither in P nor NP-complete. Factoring and graph isomorphism are the natural candidates. Babai\u0026rsquo;s quasipolynomial algorithm for graph isomorphism moved it much closer to P without landing there, and factoring being easy on a quantum computer while apparently not NP-complete is exactly the kind of structure this space allows.\nIt does not mean quantum computers solve NP-complete problems. BQP is not believed to contain NP. Grover gives a quadratic speedup on unstructured search — $2^{n/2}$ instead of $2^n$, meaningful but not a collapse — and Shor\u0026rsquo;s algorithm attacks factoring, which is likely NP-intermediate. The popular claim that quantum computers \u0026ldquo;try all possibilities at once\u0026rdquo; is wrong and this is where it does most damage.\nBoundary The parameterized view (T033). Rather than asking whether a problem is polynomial in $n$, ask whether it is polynomial in $n$ for fixed parameter $k$. Vertex Cover is $O(2^k n)$: hard in general, easy when the cover is small. This often matches reality better than the P/NP dichotomy. Average case vs. worst case. NP-hardness is a worst-case claim. Cryptography needs problems hard on average with a known distribution, which is a strictly stronger requirement, and it is why lattice problems with worst-case-to-average-case reductions are the post-quantum favourites. The exponential-time hypothesis. Stronger than P ≠ NP: that 3-SAT needs $2^{\\Omega(n)}$. ETH and SETH give conditional lower bounds on problems already in P, such as edit distance having no truly subquadratic algorithm. This fine-grained programme is where the field\u0026rsquo;s practical energy has moved. What counts as a proof attempt. Gerhard Woeginger maintained a list of over a hundred claimed proofs, all wrong. The barriers explain why: a correct proof must use a technique nobody has, so any argument that feels familiar is almost certainly one of the three ruled-out families. Cook–Levin is the next post. All of this rests on there being a first NP-complete problem to reduce from, and T027 is where SAT gets that role. Read next ","permalink":"https://cs.lozic.me/posts/t026-p-vs-np/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eYou are handed a scheduling problem. Two hundred tasks, precedence constraints,\nshared resources, and a deadline. Somebody wants an optimal schedule.\u003c/p\u003e\n\u003cp\u003eChecking a proposed schedule takes seconds: walk the list, verify each\nconstraint, add up the makespan. Finding one is different. Your search runs for\na day and finds nothing. You try simulated annealing, then a genetic algorithm,\nthen branch and bound. Each gives good schedules and none gives the best one, and\nyou cannot tell whether the problem is hard or you are not clever enough.\u003c/p\u003e","title":"P vs NP"},{"content":"Symptom You are specifying ECC memory for a fleet. The vendor quotes SECDED: single error correct, double error detect, at 8 check bits per 64 data bits. That is 12.5% overhead, and someone in the room asks the obvious question — why not correct two errors? Or three? What does it cost?\nThe vendor says more bits. How many more? Nobody in the room knows, and the options in the catalogue look arbitrary: (72,64) SECDED, (255,223) Reed–Solomon, various LDPC configurations for flash. There is no visible principle connecting redundancy to correction capability, so the choice gets made on price.\nThere is a principle, and it is geometric. Every code is a set of points in a discrete space, correction is a sphere around each point, and the spheres must not overlap. Once you see it that way the whole catalogue becomes a sphere-packing problem, the overhead becomes computable, and the question \u0026ldquo;why not correct two errors\u0026rdquo; gets a number rather than a shrug.\nStatement Hamming distance $d(x,y)$ is the number of positions where two equal-length strings differ. The minimum distance $d$ of a code is the smallest distance between any two codewords. Then:\nA code with minimum distance $d$ detects up to $d - 1$ errors. It corrects up to $t = \\lfloor (d-1)/2 \\rfloor$ errors. Hamming bound (sphere-packing bound). Any binary code of length $n$ correcting $t$ errors has at most\n$$M \\le \\frac{2^n}{\\sum_{i=0}^{t} \\binom{n}{i}}$$codewords. Equivalently, with $k$ information bits, $2^k \\sum_{i=0}^t \\binom{n}{i} \\le 2^n$.\nSingleton bound. $d \\le n - k + 1$, always. Codes meeting it with equality are MDS codes, and Reed–Solomon is the important example.\nA code meeting the Hamming bound with equality is perfect: the correction spheres tile the space exactly, with no point left over. Perfect codes are almost nonexistent, and the list is short enough to state — which is itself a striking fact about the geometry.\nArgument Why distance determines correction. Picture codewords as points in the Hamming cube ${0,1}^n$. Transmission moves a point by the number of flipped bits. If you receive $y$ and decode to the nearest codeword, you are correct whenever no other codeword is at least as close. Draw a ball of radius $t$ around each codeword: if the balls are disjoint, any received word within $t$ of a codeword has exactly one nearest neighbour and decoding is unambiguous. Balls of radius $t$ are disjoint precisely when $d \\ge 2t+1$. That is the whole correction story, and it is geometry rather than algebra.\nDetection is weaker and cheaper: to notice that an error happened you only need the corrupted word to miss every other codeword, which needs $d \\ge e+1$ for $e$ errors. Correction costs roughly twice the distance of detection, which is why SECDED gets one correction and two detections out of $d = 4$.\nWhy the bound is a volume argument. A ball of radius $t$ in the Hamming cube contains $\\sum_{i=0}^{t}\\binom{n}{i}$ points — choose which $i$ positions flipped. If there are $M$ codewords and their balls are disjoint, the total volume cannot exceed the whole space:\n$$M \\cdot \\sum_{i=0}^{t}\\binom{n}{i} \\le 2^n.$$That is the Hamming bound, and it is pigeonhole (T001) with a volume attached, exactly as the birthday bound (T074) was pigeonhole with a probability attached.\nWorking the ECC memory question. For SECDED over 64 data bits: correcting one error needs $2^k(1 + n) \\le 2^n$ with $k = 64$. Try $n = 71$: $1 + 71 = 72$, and $2^{71}/2^{64} = 128 \\ge 72$. So 7 check bits suffice for single-error correction — the (71,64) Hamming code — and the eighth bit buys the double-error detection, giving $d = 4$. That is where the vendor\u0026rsquo;s 8 comes from, and it is forced, not chosen.\nNow correct two errors. We need $2^{64}\\left(1 + n + \\binom{n}{2}\\right) \\le 2^n$. The smallest $n$ that passes is 76: the ball holds $1 + 76 + 2850 = 2927$ points and $2^{76-64} = 4096$, so 12 check bits clear the bound. The bound is necessary, not sufficient — real BCH codes for double correction over 64 bits use around 14 to 16 check bits — but the shape of the answer is clear: the first error correction costs 7 bits and the second costs roughly another 7. Correction capability is roughly linear in redundancy, not free and not exponential.\nWhy perfect codes are rare. For the balls to tile the space exactly is a strong arithmetic coincidence. The complete list of binary perfect codes: Hamming codes ($t=1$), the binary Golay code $(23, 12, 7)$ correcting 3 errors, the repetition codes of odd length, and the trivial ones. That is it — proved by Tietäväinen and van Lint in the 1970s. The Golay code\u0026rsquo;s existence is a genuine accident of arithmetic ($\\sum_{i=0}^{3}\\binom{23}{i} = 2048 = 2^{11}$ exactly), and it flew on Voyager 1 and 2 to send back the Jupiter and Saturn images.\nThe Singleton bound and Reed–Solomon. Delete $d-1$ symbols from every codeword; the results must still be distinct, since codewords differ in at least $d$ places. So $M \\le q^{n-d+1}$, giving $d \\le n-k+1$. Reed–Solomon meets this with equality by treating data as coefficients of a polynomial of degree $k-1$ and transmitting $n$ evaluations: any $k$ evaluations determine the polynomial, so any $n-k$ erasures are survivable. That is the same fact as \u0026ldquo;two points determine a line,\u0026rdquo; and it is why RAID-6, QR codes, CDs, and erasure-coded object storage all use Reed–Solomon.\nForbids Correcting $t$ errors with less redundancy than the bound allows. If someone proposes a (72,64) code correcting two errors, check: the radius-2 ball holds $1 + 72 + \\binom{72}{2} = 2629$ points and you have only $2^{72-64} = 256$ to spend. Impossible, no cleverness available. This is a five-second refutation of a whole class of proposals.\nBeating the Singleton bound. No code has $d \u0026gt; n-k+1$. Any claim of an erasure code surviving more failures than parity symbols is false, and this makes storage-vendor arithmetic checkable: an $(n,k)$ erasure code tolerates at most $n-k$ lost shards.\nCorrecting more errors than you have distance for, however good the decoder is. Beyond $d/2$ the nearest codeword may not be the transmitted one. The information is genuinely gone.\nA code that corrects everything cheaply. Overhead grows with correction capability, and the growth rate is what the bound quantifies.\nDoes not forbid It does not say the bound is achievable, and usually it is not. The Hamming bound is an upper limit on how good a code can be, while the Gilbert–Varshamov bound gives what is guaranteed to exist. Real codes live between them, and for most parameters the gap is never closed. Being told the bound permits 14 check bits does not mean a 14-bit code exists.\nIt does not apply to burst errors, which is what actually happens in hardware. Hamming distance counts scattered flips; real failures are a scratch on a disc, a row failure in DRAM, or a fading interval in wireless. Reed–Solomon operating on symbols rather than bits handles bursts natively — an eight-bit symbol counts as one error whether one bit flipped or all eight — and interleaving spreads bursts across codewords so a scattered-error code sees scattered errors. CD players interleave across a span of several thousand bits, which is why a 2 mm scratch is survivable.\nIt does not describe soft-decision decoding. The bound assumes hard decisions: each received bit is 0 or 1. Real demodulators produce likelihoods, and using them buys about 2 dB — a large amount — which is why LDPC and turbo decoders work with log-likelihood ratios and routinely correct patterns that a hard-decision decoder of the same code could not. The bound is not violated; the channel model is different.\nIt does not conflict with Shannon (T021), though they look contradictory. Shannon says rate $C$ is achievable with vanishing error; Hamming says correcting $t$ errors costs redundancy. The reconciliation is that Shannon allows a small error probability and considers typical error patterns, while Hamming demands correction of every pattern of $t$ errors. Guaranteeing the worst case is much more expensive than handling the average one, and the gap between the two bounds is precisely the price of worst-case guarantees.\nIt does not mean list decoding is impossible past $d/2$. Sudan and Guruswami–Sudan decode Reed–Solomon well beyond half the minimum distance by returning a short list of candidates rather than one answer. If you have any other way to disambiguate — a checksum, a plausibility check — this extends usable correction substantially, and it is used in practice.\nBoundary The three bounds together. Hamming (sphere packing) is an upper bound, Singleton is an upper bound on distance, Gilbert–Varshamov is a lower bound on what exists. Good code design is navigating between them, and asymptotically the best known binary codes still sit at GV for most rates. Linear codes make it tractable. If the code is a linear subspace, minimum distance equals minimum weight of a nonzero codeword, encoding is a matrix multiply, and syndrome decoding is a table lookup. Nearly every deployed code is linear for this reason alone. Where the geometry generalizes. Sphere packing in the Hamming cube is the discrete cousin of sphere packing in $\\mathbb{R}^n$, and the connection is not decorative: the Leech lattice in 24 dimensions is built from the Golay code, and Viazovska\u0026rsquo;s 2016 proof of optimal packing in dimension 8 (and, with coauthors, 24) settled a question that had been open since Kepler\u0026rsquo;s cousin problem. Cryptography reuses the hardness. Decoding a random linear code is NP-hard, and the McEliece cryptosystem is built on it. It is one of the few pre-quantum systems believed to survive quantum attack, which is why a 1978 proposal is a NIST post-quantum candidate. Modern practice. LDPC and polar codes approach Shannon capacity and are analyzed probabilistically rather than by minimum distance; their minimum distances are often unimpressive. That is a hint about which theory to use: Hamming\u0026rsquo;s for guarantees, Shannon\u0026rsquo;s for throughput. Read next ","permalink":"https://cs.lozic.me/posts/t022-the-hamming-bound/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eYou are specifying ECC memory for a fleet. The vendor quotes SECDED: single\nerror correct, double error detect, at 8 check bits per 64 data bits. That is\n12.5% overhead, and someone in the room asks the obvious question — why not\ncorrect two errors? Or three? What does it cost?\u003c/p\u003e","title":"The Hamming Bound and Error-Correcting Codes"},{"content":"Symptom Your link drops 1% of bits. You need reliable delivery, so you triple every bit and take a majority vote. Now an error needs two of three bits to flip, which happens with probability about $3 \\times 10^{-4}$ — better, but you paid 3x in bandwidth and the error rate is still not zero.\nYou want it lower, so you send five copies. Then seven. The error rate falls and the throughput falls with it, and the obvious extrapolation is depressing: arbitrarily reliable communication costs arbitrarily much bandwidth, and the efficiency of a reliable channel tends to zero. That extrapolation is the intuition essentially everyone has, it is what repetition coding demonstrates, and it dominated engineering practice before 1948.\nIt is false, and its falseness is probably the most surprising single result in this series. There is a rate — a strictly positive one, computable from the noise level — below which you can drive the error probability as close to zero as you like without reducing the rate at all. Reliability is not bought with bandwidth. It is bought with block length, which is to say with latency and computation, and those are much cheaper things to spend.\nStatement Define the channel capacity as the maximum mutual information between input and output over all input distributions:\n$$C = \\max_{p(x)} I(X; Y).$$Then for any rate $R \u0026lt; C$ and any $\\varepsilon \u0026gt; 0$, there exists a block length $n$ and a code of rate $R$ with error probability less than $\\varepsilon$. Conversely, for any $R \u0026gt; C$, the error probability is bounded away from zero — no code at any block length achieves reliable communication above capacity.\nTwo halves, and both matter. The forward part says reliability below capacity is free in the currency you expected to pay. The converse says capacity is a hard wall: the failure above it is not gradual degradation but a genuine impossibility, and this is the part that makes $C$ a law rather than a benchmark.\nFor the binary symmetric channel with crossover probability $p$:\n$$C = 1 - H(p).$$At $p = 0.01$, $C = 1 - 0.0808 = 0.919$ bits per channel use. So the 1%-error link can carry 0.919 bits of reliable payload per bit sent — a 9% overhead, not a 200% one. Repetition coding at 3x uses 33% of the channel to get $10^{-4}$; capacity says 92% of it gets you $10^{-15}$.\nArgument Why capacity is mutual information. $I(X;Y) = H(X) - H(X \\mid Y)$ is the uncertainty about the input, minus what remains after seeing the output. That is exactly \u0026ldquo;how much of what I sent got through.\u0026rdquo; Maximizing over input distributions asks how much can get through if I choose my signalling well. For the BSC, symmetry makes uniform input optimal, and $H(X \\mid Y) = H(p)$ because knowing the output leaves precisely the uncertainty of whether this bit was flipped. Hence $C = 1 - H(p)$, and the interpretation is direct: the noise consumes $H(p)$ bits of every bit you send, and the rest is yours.\nThe forward proof: random coding. This is Shannon\u0026rsquo;s move and it is worth following, because it is the reason the theorem arrived thirty years before anyone could use it.\nPick $2^{nR}$ codewords of length $n$ uniformly at random, and give the codebook to both parties. To decode, find the unique codeword that is jointly typical with the received sequence — roughly, the one whose empirical correlation with the output matches what the channel statistics predict.\nNow bound the error. Two things go wrong: the transmitted codeword is not typical with the output (probability $\\to 0$ by the law of large numbers, since $n$ independent uses concentrate — this is T008), or some other codeword is jointly typical by accident. The number of sequences jointly typical with a given output is about $2^{n H(X|Y)}$ out of $2^{nH(X)}$ total, so a random codeword collides with probability about $2^{-nI(X;Y)}$. Union bound over the $2^{nR}$ competitors:\n$$P(\\text{error}) \\lesssim 2^{nR} \\cdot 2^{-nI(X;Y)} = 2^{-n(I - R)}.$$If $R \u0026lt; I$, this goes to zero exponentially in $n$. That is the theorem. The rate stays fixed and the errors vanish, paid for by block length.\nTwo features of this argument are worth naming. First, it proves that the average code over the random ensemble is good, so a good code must exist — without producing one. Second, expurgation (throwing away the worst half of the codewords) upgrades average error to maximal error at negligible rate cost. This is the probabilistic method (T007\u0026rsquo;s existence-proof pattern) doing the heaviest lifting anyone has ever asked of it.\nThe converse. Fano\u0026rsquo;s inequality bounds the residual uncertainty given the output: $H(W \\mid \\hat{W}) \\le 1 + P_e \\cdot nR$. Chain it with the data processing inequality — information cannot increase along the Markov chain $W \\to X^n \\to Y^n \\to \\hat{W}$ — and you get $nR \\le nC + 1 + P_e nR$, so\n$$P_e \\ge 1 - \\frac{C}{R} - \\frac{1}{nR}.$$Above capacity, $P_e$ is bounded below by a positive constant no matter how large $n$ is. Bigger blocks do not help. The wall is real.\nThe fifty-year gap. Shannon proved a good code exists in 1948 with no way to build one; random codes need exponential-time decoding. Hamming and Reed–Solomon codes came close on some channels; convolutional codes with Viterbi decoding got closer. Turbo codes (Berrou, 1993) came within 0.5 dB of capacity and stunned the field. LDPC codes — invented by Gallager in 1962, forgotten as computationally infeasible, rediscovered in the 1990s — do it too and are now in Wi-Fi 6, 5G, and DVB-S2. Polar codes (Arıkan, 2008) are the first provably capacity-achieving construction with practical decoding, and are in 5G control channels. The theorem set the target and the target held for half a century.\nForbids Reliable communication above capacity, at any block length or complexity. If your channel\u0026rsquo;s capacity is 100 Mbps you will not get 110 Mbps reliably. Not with a better code, not with more compute. Claims to the contrary are claims to have broken the converse.\nReliability requiring rate to vanish. The repetition-coding intuition is wrong. Any scheme whose overhead grows without bound as the target error rate falls is leaving capacity on the table, and is beatable by a fixed-rate code with a longer block.\nBeating capacity by retransmission. Feedback does not increase the capacity of a memoryless channel. ARQ and TCP retransmission improve practical throughput on bursty channels and change nothing about the limit.\nDoes not forbid It does not say the code is easy to find, and for fifty years it was not. Shannon\u0026rsquo;s proof is non-constructive, which is why the achievable-rate curve and the practical-rate curve were far apart for decades. This is the honest reading of \u0026ldquo;reliability is free\u0026rdquo;: free in bandwidth, expensive in ingenuity.\nIt does not make latency free, and this is the trade that actually bites. Error probability falls like $2^{-n(C-R)}$, so approaching capacity needs large $n$, and $n$ is buffered symbols before you can decode. This is why deep-space links (Voyager used a concatenated Reed–Solomon/convolutional code) happily use enormous blocks and why real-time voice does not. 5G\u0026rsquo;s control channels use short polar codes and its data channels use long LDPC codes for exactly this reason.\nIt does not apply to adversarial errors. The capacity is computed for a probabilistic channel. An adversary who chooses which bits to flip is a different model, and the achievable rates are lower — this is the domain of Hamming\u0026rsquo;s combinatorial bound (T022) rather than Shannon\u0026rsquo;s probabilistic one, and the gap between the two is why the two theories coexist rather than one subsuming the other.\nIt does not require knowing the channel. Rateless codes (fountain codes, LT and Raptor) generate limitless encoded symbols and let the receiver stop when it has enough. RaptorQ is standardized and used in 3GPP multicast; the receiver does not need to know the erasure rate in advance, which is a genuine practical escape from having to estimate $p$.\nIt does not say error-free. The theorem gives arbitrarily small error, never zero. Every real link has a residual error rate, which is why storage systems layer checksums above error-correcting codes, and why \u0026ldquo;10^-15 uncorrectable bit error rate\u0026rdquo; appears on drive datasheets rather than \u0026ldquo;no errors.\u0026rdquo;\nIt does not mean separate source and channel coding is always right. Shannon\u0026rsquo;s separation theorem says compressing then coding loses nothing asymptotically — but at finite block length, and especially over multiple-user channels, joint source-channel coding wins. This is why video codecs and their transport are co-designed rather than layered.\nBoundary Finite block length. Polyanskiy, Poor and Verdú\u0026rsquo;s normal approximation gives the achievable rate at block length $n$ as roughly $C - \\sqrt{V/n},Q^{-1}(\\varepsilon)$, where $V$ is channel dispersion. This is the theorem made engineering-usable, and it is what tells you whether a 1000-symbol block is enough. The Gaussian channel. $C = \\frac{1}{2}\\log_2(1 + \\mathrm{SNR})$ per use, which becomes the Shannon–Hartley formula $C = B\\log_2(1 + S/N)$. Every claim about how fast a wireless link could go traces here, and it is why increasing bandwidth beats increasing power: capacity is linear in $B$ and logarithmic in SNR. Channels with memory and feedback. Real channels are bursty. Interleaving spreads a burst across blocks so it looks memoryless to the decoder, which is a general and cheap trick. Feedback does not raise capacity but does reduce the block length needed, which is a latency win. The erasure channel is the friendly case. Capacity is $1 - p$, and the codes are simple: any $k$ of $n$ symbols suffice with an MDS code. This is exactly RAID and erasure-coded object storage, and it is why distributed storage math is tractable while wireless is not. The gap to T022. Shannon says what is achievable on average against random noise. Hamming says what is possible in the worst case against any $t$ errors. Both are real bounds and neither implies the other, which is the subject of the next post. Read next ","permalink":"https://cs.lozic.me/posts/t021-the-noisy-channel-coding-theorem/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eYour link drops 1% of bits. You need reliable delivery, so you triple every bit\nand take a majority vote. Now an error needs two of three bits to flip, which\nhappens with probability about $3 \\times 10^{-4}$ — better, but you paid 3x in\nbandwidth and the error rate is still not zero.\u003c/p\u003e","title":"The Noisy-Channel Coding Theorem"},{"content":"Symptom You are compressing a log file. The symbols are wildly skewed: INFO is 90% of the lines, WARN is 9%, ERROR and FATAL split the rest. Fixed-width two-bit codes give you exactly 2 bits per symbol, and entropy (T018) says the floor is about 0.53 bits. There is a factor of four sitting there.\nSo you assign short codes to common symbols. INFO gets 0, WARN gets 1, ERROR gets 01, FATAL gets 10. Then you try to decode 0101 and find it is INFO WARN INFO WARN, or ERROR ERROR, or INFO WARN ERROR\u0026hellip; The stream is ambiguous and there is no way to fix it by adding a rule, because the ambiguity is structural: 0 is a prefix of 01.\nNow you have two questions and they are different. First, which sets of codeword lengths are even possible for an unambiguous code? Second, given that constraint, which assignment is best? The first is Kraft\u0026rsquo;s inequality and it is the cleaner of the two. The second is Huffman\u0026rsquo;s algorithm, and the striking part is not that it works but that a greedy algorithm — the kind that usually gives you an approximation — turns out to be exactly optimal.\nStatement Kraft\u0026rsquo;s inequality. A prefix code over an alphabet of size $D$ with codeword lengths $\\ell_1, \\dots, \\ell_n$ exists if and only if\n$$\\sum_{i=1}^{n} D^{-\\ell_i} \\le 1.$$Huffman\u0026rsquo;s theorem. For a source with symbol probabilities $p_1, \\dots, p_n$, the Huffman code minimizes expected codeword length $\\sum p_i \\ell_i$ over all prefix codes. Combined with T018:\n$$H(X) \\le L_{\\text{Huffman}} \u003c H(X) + 1.$$Three things worth separating here. Kraft is a statement about which length vectors are achievable, and it says nothing about which is good. Huffman picks the best one. And the $+1$ gap is the price of insisting that every codeword be a whole number of bits — the only thing standing between Huffman and the entropy floor, and the reason arithmetic coding exists.\nArgument Kraft, forward direction. Think of the code as a tree: each codeword is a leaf at depth $\\ell_i$, and prefix-freeness means no codeword is an ancestor of another. In a complete binary tree of depth $L = \\max \\ell_i$, a leaf at depth $\\ell_i$ rules out $2^{L - \\ell_i}$ of the $2^L$ possible depth-$L$ nodes, and prefix-freeness means these sets are disjoint. So $\\sum 2^{L-\\ell_i} \\le 2^L$, which is Kraft.\nThe probabilistic reading is nicer: walk down the tree flipping a fair coin. You reach the leaf for codeword $i$ with probability $2^{-\\ell_i}$, and these are disjoint events, so they sum to at most 1. Kraft is just \u0026ldquo;probabilities of disjoint events add up.\u0026rdquo;\nKraft, converse. Given lengths satisfying the inequality, sort them increasing and assign codewords greedily as binary fractions: $c_i$ is the first $\\ell_i$ bits of $\\sum_{j\u0026lt;i} 2^{-\\ell_j}$. The inequality guarantees you never run out of room, and sortedness guarantees prefix-freeness. So the inequality is not merely necessary; it is a complete characterization of which length vectors you may have.\nWhy entropy is a lower bound. Minimizing $\\sum p_i \\ell_i$ subject to Kraft, with the integrality dropped, is a Lagrange problem whose solution is $\\ell_i = -\\log_2 p_i$, giving $\\sum p_i \\ell_i = H(X)$ exactly. So the entropy floor is the Kraft constraint with the integers relaxed, and Huffman\u0026rsquo;s job is to round.\nHuffman\u0026rsquo;s algorithm. Take the two least probable symbols, merge them into a node with the sum of their probabilities, and repeat until one node remains. Read off the tree. That is the whole thing.\nWhy greedy is exactly optimal. The proof is two exchange arguments, and both are worth the space because greedy algorithms are usually not optimal and the reason this one is comes down to two structural facts.\nFact one: an optimal code has the two rarest symbols as siblings at maximum depth. Suppose in an optimal tree a symbol $x$ with lower probability sits higher than a symbol $y$ with higher probability. Swapping them changes the cost by $(p_x - p_y)(\\ell_y - \\ell_x)$, which is $\\le 0$ — the swap does not hurt. So some optimal code has the rarest symbols deepest. Further, an optimal tree has no unmatched leaf (you could promote it and save a bit), so the two deepest leaves are siblings, and we may take them to be the two rarest.\nFact two: merging is cost-preserving. Let $T$ be a code for the merged alphabet, where $x$ and $y$ are replaced by a single symbol $z$ with $p_z = p_x + p_y$. Expanding $z$ back into a pair of leaves costs exactly $p_x + p_y$ extra bits, regardless of where $z$ sits. So\n$$\\mathrm{cost}(T_{\\text{expanded}}) = \\mathrm{cost}(T) + p_x + p_y,$$a constant offset. Minimizing one minimizes the other. Induct on alphabet size: Huffman\u0026rsquo;s merge step reduces the problem to a strictly smaller one whose optimal solution lifts to an optimal solution here.\nThat constant offset is the crux. Greedy algorithms fail when an early choice constrains later ones in a way that depends on the choice; here the merge changes the cost by an amount that does not depend on what the rest of the tree does, so the subproblem is genuinely independent.\nThe numbers. For the log file: $p = (0.9, 0.09, 0.005, 0.005)$. Entropy is $0.526$ bits. Huffman gives lengths $(1, 2, 3, 3)$ for a cost of $1.11$ bits per symbol — better than 2, and nowhere near 0.526, because no codeword can be shorter than one bit however likely it is. That is the $+1$ gap at its worst, and skewed distributions are exactly where it bites hardest.\nForbids A prefix code with lengths violating Kraft. If someone proposes a code with lengths $(1,2,2,3)$ over a binary alphabet, $0.5+0.25+0.25+0.125 = 1.125 \u0026gt; 1$ and no such code exists. This is a one-line check on any proposed format.\nAny prefix code beating Huffman on the same alphabet and probabilities. Huffman is not a heuristic. There is nothing left to find, and \u0026ldquo;we improved on Huffman\u0026rdquo; always means the model or the alphabet changed.\nBeating entropy. $L \\ge H(X)$ for any uniquely decodable code, not just prefix codes — the McMillan converse shows uniquely decodable codes satisfy Kraft too, so dropping the prefix property buys nothing. This is worth knowing because it closes an obvious escape route.\nDoes not forbid It does not stop arithmetic coding from beating Huffman, and in practice everything does. Arithmetic and range coding encode the whole message as a single number in $[0,1)$, so symbols cost fractional bits and the total lands within 2 bits of entropy for the entire message rather than 1 bit per symbol. On the log-file distribution above that is 0.53 versus 1.11 — a 2x difference, not a rounding error. This is why modern compressors use rANS (Zstandard, LZFSE) or context-mixing arithmetic coders, and Huffman survives mainly where decode speed dominates. DEFLATE uses Huffman; Zstandard uses both, picking per block.\nIt does not fix the probability model, which is where the real gains are. Huffman is optimal given $p$. Order-0 Huffman on English text gets about 4.2 bits per character; a good context model gets under 2. Every serious compressor spends its effort on modelling, and the entropy coder is the cheap part. LZ77 plus Huffman beats Huffman alone because the LZ stage changes the alphabet from characters to matches, not because the coder improved.\nIt does not require probabilities to be known in advance. Adaptive Huffman (FGK, Vitter) updates the tree as it encodes, and two-pass Huffman ships the table. Both are used; JPEG and MP3 ship standard tables, and DEFLATE offers fixed, dynamic, and stored block types so the encoder can choose per block.\nIt does not say the Huffman tree is unique. Ties produce different trees with identical cost. Canonical Huffman codes exploit this: fix a tie-break rule so the decoder can rebuild the tree from the lengths alone, which is why JPEG and DEFLATE transmit only a list of code lengths and not the tree.\nIt does not apply when codeword costs are unequal or constrained. If symbols cost different amounts to transmit, the right answer is the Varn code; if you need a maximum codeword length — as JPEG does, capping at 16 bits — the length-limited problem is solved by the package-merge algorithm, not by Huffman, and it is genuinely a different algorithm giving a slightly worse code.\nBoundary The $+1$ is tight and it is about integers. A two-symbol source with $p = (0.99, 0.01)$ has entropy $0.08$ bits and Huffman must spend 1 bit. The loss is entirely the rounding, which is why blocking symbols together — coding pairs or triples — reduces the per-symbol penalty to $1/k$ and why arithmetic coding, which blocks the entire message, essentially removes it. Shannon–Fano is the near miss. Fano\u0026rsquo;s top-down splitting was the state of the art when Huffman took the class, and it is not optimal. The direction of construction is the whole difference: bottom-up commits to the decisions that are forced, top-down commits to the ones that are not. Kraft as a general tool. The inequality is really a statement about any assignment of disjoint probability mass, and it reappears in Kolmogorov complexity (T023) as the reason $\\sum 2^{-K(x)} \\le 1$, which is what makes the universal prior a probability distribution at all. Optimal is not the same as good enough. Huffman is optimal over prefix codes for a memoryless source. Change any of those — memory, non-prefix, fractional bits — and there is room. Most practical progress comes from noticing which assumption is the wrong one, not from beating the theorem. Where the greedy proof pattern generalizes. The two exchange arguments here are the matroid-greedy pattern: local optimality plus a subproblem that is independent of the choice made. Kruskal\u0026rsquo;s algorithm has the same shape, and recognizing it is how you tell in advance whether greedy will work. Read next ","permalink":"https://cs.lozic.me/posts/t020-huffman-coding-is-optimal/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eYou are compressing a log file. The symbols are wildly skewed: \u003ccode\u003eINFO\u003c/code\u003e is 90% of\nthe lines, \u003ccode\u003eWARN\u003c/code\u003e is 9%, \u003ccode\u003eERROR\u003c/code\u003e and \u003ccode\u003eFATAL\u003c/code\u003e split the rest. Fixed-width\ntwo-bit codes give you exactly 2 bits per symbol, and entropy (T018) says the\nfloor is about 0.53 bits. There is a factor of four sitting there.\u003c/p\u003e","title":"Huffman Coding Is Optimal"},{"content":"Symptom You have 100 backends and a load balancer hashing request IDs to pick one. Expected load per backend is exactly 1%, and you have checked the hash is good.\nThen the dashboard shows one backend at three times the mean. You check for a hot key: none. You check the hash: uniform. You add more backends and the imbalance persists. Somebody suggests the hash function is bad after all, and a week goes into replacing it, and the imbalance does not change.\nThe hash function was never the problem. Uniform random assignment produces imbalance as a matter of course, and the size of that imbalance is a theorem. Linearity of expectation (T007) tells you the mean load per bin and says nothing about the maximum, which is the number that determines whether a backend falls over, what your p99 looks like, and how much headroom you must provision.\nThe good news is the second half of this post, and it is one of the highest return-on-complexity results in systems engineering: a change that costs one extra probe per request turns an exponential-ish gap into a doubly logarithmic one.\nStatement Single choice. Throw $n$ balls independently and uniformly at random into $n$ bins. With high probability, the maximum load is\n$$\\Theta\\!\\left(\\frac{\\log n}{\\log \\log n}\\right).$$Two choices (Azar, Broder, Karlin, Upfal). Throw $n$ balls into $n$ bins, but for each ball sample $d \\ge 2$ bins uniformly and place it in the least loaded of the $d$. With high probability, the maximum load is\n$$\\frac{\\ln \\ln n}{\\ln d} + \\Theta(1).$$Read those two together, because the contrast is the entire point:\nGoing from one choice to two takes you from $\\log n/\\log\\log n$ to $\\log\\log n$ — an exponential improvement. Going from two choices to three takes you from $\\log_2 \\ln n$ to $\\log_3 \\ln n$ — a constant factor. The first extra probe buys almost everything; the rest buy almost nothing. That asymmetry is why \u0026ldquo;the power of two choices\u0026rdquo; is the name.\nConcretely, at $n = 10^6$: one choice gives a max load around 10; two choices gives $\\ln\\ln(10^6)/\\ln 2 \\approx 3.8$, so about 4. Take $n$ up to $10^9$ and the one-choice figure keeps climbing while the two-choice figure moves to 4.4. Doubly logarithmic growth is, for practical purposes, constant.\nArgument Why one choice is unbalanced. The load of a particular bin is $\\mathrm{Binomial}(n, 1/n)$, mean 1. The probability it has at least $k$ balls is at most $\\binom{n}{k}(1/n)^k \\le 1/k!$. Setting this to $1/n$ and using Stirling, $k! \\approx (k/e)^k$, gives $k \\log(k/e) \\approx \\log n$, so $k = \\Theta(\\log n/\\log\\log n)$. Now union bound (T008\u0026rsquo;s standard partner) over all $n$ bins: with probability $1 - o(1)$ no bin exceeds this, and a matching argument shows some bin reaches it.\nSo the max is not $O(1)$, and no amount of hash quality changes that. The imbalance is a property of randomness, not of your hash.\nWhy two choices is doubly logarithmic — the layered induction. This is the argument worth carrying around. Let $\\beta_k$ be the fraction of bins with load at least $k$. A ball lands in a bin of load $\\ge k$ only if both its sampled bins already have load $\\ge k$, which happens with probability $\\beta_k^2$. Roughly, then,\n$$\\beta_{k+1} \\lesssim \\beta_k^2.$$Start from $\\beta_2 \\le 1/2$ or so and iterate: the exponent squares each time, so $\\beta_{k} \\approx 2^{-2^{k}}$. The height at which $\\beta_k$ drops below $1/n$ — meaning no bin at all is that tall — is $k \\approx \\log_2\\log_2 n$. The squaring is the whole mechanism, and $d$ choices give $\\beta_{k+1} \\approx \\beta_k^d$, which changes the base of the outer logarithm and nothing else. That is exactly why the third choice is nearly worthless.\nWhy this is more than a curiosity. The single-choice result is what makes hash tables have $\\Theta(\\log n/\\log\\log n)$ worst-case chains despite $O(1)$ expected chains — the fact quoted in T007 as the reason expectation does not determine your p99. The two-choice result then repairs it: cuckoo hashing gives each key two candidate positions, evicting and relocating on collision, and achieves worst-case $O(1)$ lookup with high space efficiency. Same theorem, used as a data-structure design.\nWhere it lands in real systems. NGINX, HAProxy, and Envoy all implement \u0026ldquo;power of two random choices\u0026rdquo; load balancing (Envoy calls it LEAST_REQUEST with a default choice count of 2); it is the default in several service meshes. The reason it beats exact least-loaded in distributed settings is the topic of the next section.\nForbids Perfect balance from uniform random assignment. Expect $\\Theta(\\log n/\\log\\log n)$ imbalance and provision for it. A backend at 3x the mean with 100 backends is unremarkable, and chasing it as a bug wastes weeks.\nFixing single-choice imbalance by improving the hash. A perfectly uniform hash is the model. If the hash is uniform and you still see imbalance, the theorem says you will, and only changing the assignment algorithm helps.\nMeaningful gains past a few choices. $d = 2$ to $d = 3$ is a constant factor in the logarithm; going to $d = 10$ costs 10 probes for essentially nothing. If someone proposes probing more backends, the theorem prices it.\nProvisioning from the mean. With $m$ tasks over $n$ machines you must size for the max, not the mean. Sizing from the mean is how capacity plans fail while being arithmetically correct.\nDoes not forbid It does not say randomization is required, and this is worth saying because deterministic schemes are often better. Round-robin achieves perfect balance for identical requests with no probing at all. Consistent hashing keeps a key on the same backend, which is what makes caches work — and bounded-load consistent hashing (Mirrokni, Thorup, Zadimoghaddam, used by Vimeo) adds a load cap to consistent hashing and gets both properties. Randomization is the answer when you have no coordination, not always.\nIt does not say two choices beats exact least-loaded. Exact least-loaded is better in the model. The reason nobody uses it in a distributed load balancer is the herd effect: with many independent balancers, all of them see the same \u0026ldquo;least loaded\u0026rdquo; backend from stale state and stampede it, which is worse than either. Two random choices is robust precisely because it is not coordinated, and this is a case where a theoretically weaker algorithm wins on a property the theorem does not model.\nIt does not hold when the balls are not exchangeable. Requests have wildly different costs; a single expensive query outweighs a thousand cheap ones. The theorem counts balls. If your load is heavy-tailed in cost, balancing counts balances the wrong thing, and least-outstanding-request or a cost-weighted scheme is what you want. This is the same limitation as T008\u0026rsquo;s on heavy tails.\nIt does not survive stale information gracefully. The analysis assumes the loads you sample are current. With delayed load reports, two choices degrades, and Mitzenmacher\u0026rsquo;s later work on \u0026ldquo;the power of two choices with stale information\u0026rdquo; shows the improvement can disappear entirely. Sampling at the moment of the decision matters.\nIt does not apply when $m \\gg n$. With $m$ balls into $n$ bins for large $m$, single choice gives $m/n + \\Theta(\\sqrt{(m\\log n)/n})$ — the deviation becomes small relative to the mean, so imbalance stops mattering. Two choices gives $m/n + \\ln\\ln n$, where the gap above the mean is independent of $m$. That is the sharper statement for a long-running system, and it is the one to quote for a load balancer that runs for months.\nBoundary The dynamic setting. Real systems have arrivals and departures, not a fixed throw. Mitzenmacher\u0026rsquo;s fluid-limit analysis shows the queue-length distribution decays doubly exponentially with two choices and only exponentially with one, which is the same contrast in the steady state. Memory helps too. Remembering the least loaded bin from the previous round (\u0026ldquo;one choice plus memory\u0026rdquo;) achieves similar gains to two choices, which suggests the real resource is information rather than probes. The $(1+\\beta)$-choice variant. Using two choices only a $\\beta$ fraction of the time still gets you most of the benefit, which matters when probing is expensive. Cuckoo hashing and beyond. Two hash functions with relocation give worst-case $O(1)$ lookups; the load threshold where insertion starts failing (0.5 for two functions, about 0.91 for three) is itself a sharp phase transition, and a nice example of a threshold phenomenon in a data structure. Connection to consistent hashing. T072\u0026rsquo;s rebalancing behaviour depends on load distribution, and this is the result underneath it: how much a node can exceed its share is what determines whether removing a node cascades. Why virtual nodes exist. Consistent hashing with one point per node gives a node\u0026rsquo;s share of the ring as a spacing between uniform points, whose variance is large — the same single-choice imbalance, in a different coordinate. Giving each node a hundred or more virtual points averages a hundred independent spacings and shrinks the deviation by $\\sqrt{100}$. Dynamo, Cassandra, and Riak all do this, and the number of virtual nodes is a variance knob rather than an implementation detail. The lower bound is what makes the result interesting. $\\Omega(\\log\\log n)$ holds for any $d$-choice scheme, so two choices is not merely good, it is within a constant of optimal for this class. Knowing that stops the search for a cleverer probing rule. Read next ","permalink":"https://cs.lozic.me/posts/t009-balls-into-bins/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eYou have 100 backends and a load balancer hashing request IDs to pick one.\nExpected load per backend is exactly 1%, and you have checked the hash is good.\u003c/p\u003e","title":"Balls Into Bins and the Power of Two Choices"},{"content":"Symptom You have a new problem. Your build system needs to decide whether two configuration files can ever produce conflicting outputs. Or your linter needs to decide whether a regex with backreferences can match a string of a given length. Or your scheduler needs to decide whether a set of periodic tasks is feasible.\nYou spend two weeks on an algorithm. It handles the examples, then someone finds a case where it loops. You add a depth limit. Someone finds a case where the depth limit gives the wrong answer. You start to suspect the problem is impossible, but suspecting is not knowing, and you cannot justify abandoning the project on a hunch.\nMeanwhile the halting problem (T010) is undecidable, and you know that, and the knowledge is useless because your problem is not the halting problem. It is about YAML files. There is no visible connection.\nThe connection is a technique, and it is the single most transferable skill in theoretical computer science: make your problem solve the hard one. Once you can do that, one undecidable problem becomes thousands, and the same mechanism turns one NP-hard problem into Karp\u0026rsquo;s list and then into the tens of thousands of hard problems catalogued since.\nStatement A many-one reduction from $A$ to $B$ is a computable function $f$ such that for all $x$:\n$$x \\in A \\iff f(x) \\in B.$$Write $A \\le_m B$. Then:\nIf $B$ is decidable, so is $A$. (Compute $f(x)$, ask about $B$.) Contrapositive, and this is the one you use: if $A$ is undecidable, so is $B$. The direction is the thing people get backwards, so it is worth stating flatly: to prove your problem $B$ is hard, you reduce a known-hard problem $A$ to it. You are showing that solving $B$ would let you solve $A$, which you know is impossible. Reducing your problem to a known-hard one proves nothing at all about your problem\u0026rsquo;s hardness — that only shows yours is no harder.\nFor complexity, the same definition with $f$ required to be polynomial-time gives $\\le_p$, and the same contrapositive: if $A$ is NP-hard and $A \\le_p B$, then $B$ is NP-hard. Cook–Levin (T027) provides the first NP-complete problem, and everything after is reduction.\nTuring reductions are more permissive: $A \\le_T B$ if a machine with an oracle for $B$ decides $A$, with unlimited queries and free use of negation. Many-one is the stricter and more informative notion, which is why hardness proofs prefer it.\nArgument The transitivity that makes it work. If $A \\le_m B$ via $f$ and $B \\le_m C$ via $g$, then $A \\le_m C$ via $g \\circ f$, since computable functions compose. That is why hardness spreads: you never have to go back to the halting problem. You reduce from whatever known-hard problem is structurally closest to yours, and the chain back to the source is implicit. Karp\u0026rsquo;s 21 problems (T028) is precisely a tree of such chains rooted at SAT.\nA worked undecidability reduction. Claim: $E_{TM} = {\\langle M\\rangle : L(M) = \\emptyset}$, \u0026ldquo;does this program accept nothing,\u0026rdquo; is undecidable. Reduce from $A_{TM}$, the acceptance problem. Given $\\langle M, w\\rangle$, construct a new machine:\nM\u0026#39;(x): if x != w: reject else: run M on w and return its answer $M\u0026rsquo;$ accepts nothing unless $M$ accepts $w$, in which case it accepts exactly ${w}$. So $L(M\u0026rsquo;) = \\emptyset \\iff M$ does not accept $w$. Building the source of $M\u0026rsquo;$ from $\\langle M, w \\rangle$ is textual substitution, hence computable. A decider for $E_{TM}$ would therefore decide $A_{TM}$, which is impossible.\nThe shape here is universal: build a machine that ignores its input, performs the hard computation, and reports the answer through the property you are interested in. Almost every undecidability proof about program properties is this, and Rice\u0026rsquo;s theorem (T011) is the observation that it always works for semantic properties.\nA worked NP-hardness reduction. 3-SAT $\\le_p$ Independent Set. From a formula with $m$ clauses, build a graph with three vertices per clause (one per literal), a triangle within each clause, and an edge between any two vertices holding complementary literals. Claim: the formula is satisfiable iff the graph has an independent set of size $m$. An independent set of size $m$ takes exactly one vertex per triangle (a chosen true literal per clause) and never takes both $x$ and $\\neg x$ (no edge violated), which is exactly a satisfying assignment. The construction is linear in the formula.\nNotice what makes this work: the graph problem has no combinatorial connection to Boolean logic. The triangles simulate \u0026ldquo;pick one,\u0026rdquo; and the complement edges simulate \u0026ldquo;be consistent.\u0026rdquo; Designing a reduction is designing gadgets that simulate the source problem\u0026rsquo;s constraints in the target\u0026rsquo;s vocabulary. That is the craft.\nPost\u0026rsquo;s correspondence problem as the great intermediate target. PCP asks: given dominoes with a top and bottom string, is there a sequence whose concatenated tops equal its concatenated bottoms? No machines, no programs, just strings. It is undecidable, proved by encoding a Turing machine\u0026rsquo;s computation history so that the only way to match is to spell out an accepting run — the bottom row is always one configuration ahead of the top, so matching forces correct simulation. PCP then becomes the source for proving ambiguity of context-free grammars undecidable, and equivalence of context-free grammars undecidable, which are results you actually meet building parsers (T038).\nWhy hardness \u0026ldquo;spreads.\u0026rdquo; Each new hard problem is a new source, and problems close to it become easy to reduce from. This is why the NP-complete catalogue grew from Karp\u0026rsquo;s 21 to Garey and Johnson\u0026rsquo;s several hundred within a decade: it is not that people got smarter, it is that transitivity compounds.\nForbids Solving any problem that a known-hard problem reduces to. If halting reduces to your problem, no algorithm decides it, and continuing to look is wasted work. This is the practical payoff: two weeks of failed attempts becomes one page of proof, and the project changes direction rather than dying slowly.\nA polynomial algorithm for any NP-hard problem, unless P = NP. One would collapse the class, so a claimed polynomial algorithm for your scheduling problem is a claimed proof of P = NP and should be treated with the corresponding scepticism.\nEscaping by restricting slightly. If the reduction only uses instances in your restricted subclass, the restriction does not help. This is why \u0026ldquo;our graphs are sparse\u0026rdquo; often fails to rescue anything: 3-SAT reduces to problems on graphs of maximum degree 3.\nDoes not forbid It does not say your actual instances are hard, and this is the misuse that ends useful projects. NP-hardness is a worst-case statement about an infinite family. SAT solvers routinely dispatch industrial instances with millions of variables, because real instances have structure — backbones, small community structure, low treewidth. Z3, CVC5, and MiniSat are load-bearing in verification, symbolic execution, and package management despite SAT being the original NP-complete problem. \u0026ldquo;It\u0026rsquo;s NP-hard\u0026rdquo; is a reason to use a solver, not a reason to give up.\nUndecidability does not stop useful tools either. Termination is undecidable, and Terminator, AProVE, and Rust\u0026rsquo;s MIRAI prove termination of real programs every day. Type inference for System F is undecidable, and Haskell infers types fine. The standard escape is to answer \u0026ldquo;yes,\u0026rdquo; \u0026ldquo;no,\u0026rdquo; or \u0026ldquo;don\u0026rsquo;t know\u0026rdquo; — soundness without completeness — and every static analyzer in production does this. Undecidability forbids the total decider, never the useful partial one.\nReductions do not preserve approximability. A polynomial reduction can destroy the structure of near-optimal solutions, so NP-hardness of the exact problem says nothing about whether a good approximation exists. Knapsack is NP-hard and has an FPTAS; max-cut is NP-hard and a coin flip gets you 0.5. You need approximation-preserving reductions (L-reductions) and the PCP theorem (T031) to rule approximations out, and that is a genuinely different and later result.\nTuring reductions do not preserve everything many-one reductions do. $A$ and its complement are Turing-equivalent but generally not many-one equivalent, which is exactly why co-NP is not obviously NP and why the distinction is not pedantry.\nReductions do not always run the direction you want. Failure to find a reduction is not evidence of easiness. Graph isomorphism has resisted both a polynomial algorithm and an NP-hardness proof for fifty years, and now has a quasipolynomial one (Babai).\nBoundary Choosing the source problem. The skill is picking a hard problem structurally near yours: 3-SAT for constraint problems, Vertex Cover or Clique for graphs, 3-Partition for scheduling and packing (it is strongly NP-hard, which pseudo-polynomial algorithms cannot escape), Hamiltonian Cycle for ordering. Reductions define classes, not just spread hardness. \u0026ldquo;NP-complete\u0026rdquo; means in NP and NP-hard under $\\le_p$; the reduction notion is part of the definition, and using the wrong one (Turing instead of many-one) breaks the theory of completeness. Weaker reductions for finer structure. Log-space reductions are needed to study P-completeness, since polynomial reductions are too coarse when the class itself is P. The general rule: the reduction must be weaker than the class you are classifying. Fine-grained complexity. Reductions from the Strong Exponential Time Hypothesis show that edit distance has no truly subquadratic algorithm, which is a conditional lower bound on a problem already in P. Same technique, applied inside the tractable world. Reductions also transfer algorithms. The positive direction is real and underused: encode your problem as SAT, ILP, or SMT, and inherit decades of solver engineering. Reduction is how you get an answer, not only how you prove you cannot. Read next ","permalink":"https://cs.lozic.me/posts/t017-reductions/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eYou have a new problem. Your build system needs to decide whether two\nconfiguration files can ever produce conflicting outputs. Or your linter needs to\ndecide whether a regex with backreferences can match a string of a given length.\nOr your scheduler needs to decide whether a set of periodic tasks is feasible.\u003c/p\u003e","title":"Reductions"},{"content":"Symptom Production is wedged. Not slow — wedged. CPU is at zero, no errors are being logged, and two threads are sitting in a lock wait with no timeout. The stack traces show thread A holding the account lock and waiting for the ledger lock, thread B holding the ledger lock and waiting for the account lock. Somebody restarts the service and it goes away.\nIt comes back in three weeks. Then a fix ships that \u0026ldquo;adds a timeout so it can\u0026rsquo;t hang,\u0026rdquo; and now instead of hanging the system throws lock-acquisition failures under load and the transactions retry into the same collision. Then someone adds jitter to the retry, and the incidents become rarer and never stop.\nEvery step of that is a real thing teams do, and every step treats deadlock as a probabilistic phenomenon to be made rarer. It is not. Deadlock has an exact characterization — four conditions, all of which must hold — and if you break one of them structurally the bug becomes impossible rather than unlikely. That is the difference between a mitigation and a fix, and it is available here in a way it usually is not.\nStatement Deadlock occurs if and only if all four of the following hold simultaneously:\nMutual exclusion. At least one resource is held in a non-shareable mode. Hold and wait. A process holding at least one resource is waiting to acquire additional resources held by others. No preemption. A resource cannot be forcibly taken from the process holding it; it is released only voluntarily. Circular wait. There is a cycle $P_1 \\to P_2 \\to \\dots \\to P_n \\to P_1$ in which each process waits for a resource held by the next. The four are individually necessary, and together sufficient. That biconditional is the whole engineering value: it means the space of fixes is exactly four options wide, and it is complete. There is no fifth kind of deadlock prevention, and any strategy you can think of is one of these four in disguise.\nA subtlety worth stating: condition 4 implies condition 2 for the general resource model, so the conditions are not independent. But keeping them separate is what makes them useful, because they correspond to four different places you can intervene in a design.\nArgument Necessity. Each condition\u0026rsquo;s absence directly prevents deadlock:\nWithout mutual exclusion, resources are shareable and nobody waits. Without hold-and-wait, a blocked process holds nothing, so it blocks nobody. With preemption, any wait can be resolved by taking the resource. Without a cycle, the wait-for graph is acyclic, so it has a sink — a process waiting for nothing — which can proceed, complete, release, and unblock its predecessor. Induct: the whole graph drains. That last one is the real content, and it is worth noticing that it is the same argument as topological sorting: a finite directed acyclic graph always has a vertex with no outgoing edges, and progress at that vertex propagates backwards.\nSufficiency. Given all four, consider the cycle. Each $P_i$ holds a resource $P_{i-1}$ wants (mutual exclusion means it must wait), holds it while waiting (hold-and-wait), and cannot be forced to release (no preemption). No process in the cycle can advance and nothing outside can help them. The state is permanent.\nThe four fixes, and what they cost.\nBreak mutual exclusion. Use shareable resources. Read-write locks let readers share; lock-free structures using compare-and-swap remove the lock entirely; immutable data has no exclusion problem at all. This is why persistent data structures and MVCC databases sidestep whole categories of this bug — a reader in PostgreSQL never blocks a writer because it reads an older version. Cost: not all resources can be shared, and lock-free code is genuinely hard to write.\nBreak hold-and-wait. Acquire everything up front, atomically, or nothing. Java\u0026rsquo;s tryLock on all resources with rollback does this, as does two-phase locking\u0026rsquo;s growing phase when combined with a total request. Cost: you must know your full resource set in advance, and holding everything from the start destroys concurrency.\nBreak no-preemption. Add timeouts and rollback. Database transaction managers do exactly this: detect the cycle, pick a victim, abort it, and let the rest proceed. PostgreSQL runs a deadlock detector after deadlock_timeout (1s by default) and kills one transaction with error 40P01. Cost: you need the work to be rollbackable, which is why this is natural in databases and awkward in general code holding OS mutexes.\nBreak circular wait. Impose a global ordering on resources and require every process to acquire in increasing order. A cycle would require some process to acquire out of order, so cycles are impossible. This is the one to reach for in application code, because it is static, cheap, and checkable. Linux\u0026rsquo;s lockdep validates lock ordering at runtime and reports violations before they deadlock; the kernel\u0026rsquo;s documented mmap/page-lock ordering is exactly this discipline written down. Cost: you must maintain the order globally, which is a real organizational burden, and the order must be a genuine total order over resource classes, not instances — which is why locking two accounts by ID is the standard trick.\nWhy the timeout fix fails. Adding a timeout does break no-preemption, so it does technically prevent deadlock. What it produces instead is livelock: the threads release, retry, and collide again. The system is not deadlocked and is also not progressing. Deadlock is a safety property violation and livelock is a liveness one, and trading one for the other is not obviously a win.\nForbids Deadlock with an acyclic wait-for graph. If you observe a hang and the wait-for graph has no cycle, you are not looking at deadlock. You are looking at livelock, starvation, an unbounded wait on external I/O, or a lost wakeup. This is a genuinely useful diagnostic: dump the threads, build the graph, check for a cycle, and if there is none, stop looking for a lock-ordering bug.\nDeadlock among lock-free code paths. No mutual exclusion, no deadlock. (ABA problems and livelock remain available.)\nDeadlock under a consistently applied global lock order. Not less likely. Impossible. If it happens anyway, some path violated the order, and that is now a findable bug rather than a mysterious one.\nA fifth prevention strategy. Every real technique is one of the four. Lock hierarchies are #4, tryLock-with-backoff is #3, arena allocation up front is #2, RCU is #1. This is what a characterization theorem buys you: the design space is enumerated.\nDoes not forbid It does not cover starvation or livelock, which are the failures that survive your fix. A thread that repeatedly loses a race for a lock it could acquire is starving, not deadlocked, and no Coffman condition is violated. Priority inversion — where a low-priority thread holds a lock a high-priority thread needs — sank the Mars Pathfinder mission in 1997 and is not deadlock either; priority inheritance is the fix, and it is orthogonal to all four conditions.\nIt does not apply cleanly to distributed systems, where the graph is not observable. Building the wait-for graph requires a consistent global snapshot, which the Two Generals problem (T061) makes expensive and FLP (T062) makes impossible to do reliably with failures. Distributed deadlock detection is a real subject (Chandy–Misra–Haas edge chasing) and it is much harder than the single-node case, which is why distributed systems overwhelmingly use lease timeouts instead: break no-preemption, accept the livelock risk.\nIt does not mean detection is always the wrong choice. Prevention costs concurrency. Databases deliberately allow deadlock and detect it, because transactions can be aborted cheaply and forbidding hold-and-wait would serialize the workload. Ostrich algorithm — ignoring the problem — is also defensible: it is what Linux does for most user-space deadlock, on the grounds that the cost of prevention exceeds the cost of the occasional restart.\nIt does not require locks specifically. The resources can be threads in a bounded pool, database connections, file handles, or network buffers. Thread pool exhaustion where tasks submit subtasks to the same pool and wait for them is textbook hold-and-wait plus circular wait, and it is one of the most common production hangs in JVM services. The fix is the same: separate pools imposes an ordering.\nBoundary Prevention vs. avoidance vs. detection. Prevention breaks a condition structurally. Avoidance (the Banker\u0026rsquo;s algorithm) allows requests only into states from which completion is guaranteed, and requires knowing maximum resource claims in advance, which is why nothing real uses it. Detection lets deadlock happen and recovers. Deadlock is a safety property. \u0026ldquo;Something bad happens\u0026rdquo; — the system enters a state with no successor. Starvation and livelock are liveness violations: \u0026ldquo;something good never happens.\u0026rdquo; The distinction determines which tools apply, since model checkers find safety violations far more easily. Static detection. Lock-order violations are checkable: Linux\u0026rsquo;s lockdep at runtime, Rust\u0026rsquo;s type system by making shared mutable state require a lock, and Java\u0026rsquo;s -XX:+PrintConcurrentLocks plus jstack for post-mortem. RAII and Rust\u0026rsquo;s MutexGuard eliminate the forgot to unlock class entirely, which is a different bug but the same discipline. Where the model is too simple. The conditions assume resources are discrete and requests are for whole resources. Deadlock over quantities of a divisible resource — memory, connection counts — behaves differently and is closer to a scheduling problem than a graph one. Read next ","permalink":"https://cs.lozic.me/posts/t051-the-coffman-conditions/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eProduction is wedged. Not slow — wedged. CPU is at zero, no errors are being\nlogged, and two threads are sitting in a lock wait with no timeout. The stack\ntraces show thread A holding the account lock and waiting for the ledger lock,\nthread B holding the ledger lock and waiting for the account lock. Somebody\nrestarts the service and it goes away.\u003c/p\u003e","title":"The Coffman Conditions"},{"content":"Symptom The proof assistant is failing you. You are trying to verify a piece of concurrent code in Coq or Lean, and the tool will not accept your termination argument. A colleague says, without much conviction, \u0026ldquo;well, Gödel — you can\u0026rsquo;t prove everything anyway.\u0026rdquo;\nThat sentence is almost always wrong in context, and it is the most commonly misapplied result in mathematics. It has been used to argue that human minds exceed computers, that formal verification is pointless, that AI is impossible, that no software can ever be proved correct, and that mathematics is arbitrary. None of these follow.\nBut the theorem does have real engineering content, and it is much more specific and much sharper than the folklore. The second theorem in particular is a hard constraint on the verification stack — it says something concrete about what your proof assistant can and cannot say about itself, and it is the reason Coq\u0026rsquo;s kernel cannot prove its own soundness. That is worth understanding precisely, because the precise version is useful and the vague version is only ever used to excuse giving up.\nStatement First incompleteness theorem. Any consistent formal system $F$ that is effectively axiomatizable and strong enough to express elementary arithmetic contains a sentence $G_F$ that is true but neither provable nor refutable in $F$.\nSecond incompleteness theorem. For such an $F$, the sentence $\\mathrm{Con}(F)$ expressing $F$\u0026rsquo;s own consistency is not provable in $F$, unless $F$ is inconsistent.\nThe three hypotheses are all load-bearing, and every misreading drops one:\nEffectively axiomatizable. The axioms must be recognizable by an algorithm. Drop this and \u0026ldquo;the set of all true arithmetic statements\u0026rdquo; is a complete theory — useless, because you cannot check whether something is an axiom. Strong enough for arithmetic. Specifically, capable of representing primitive recursive functions. Weaker systems escape entirely. Consistent. An inconsistent system proves everything, including its own consistency, which is precisely why the second theorem\u0026rsquo;s converse is worthless as reassurance. The second theorem is the one with teeth. The first says there is a weird sentence you cannot settle; you can shrug and add it as an axiom. The second says the one thing you most want to prove about your system — that it does not prove false things — is exactly the thing it cannot prove.\nArgument Step one: arithmetization. Assign a number to every symbol, then to every formula (as a sequence of symbols), then to every proof (as a sequence of formulas), using prime factorization or any injective encoding. This is Gödel numbering, and it is the same idea as T013\u0026rsquo;s machine encoding: syntax is data, and data is numbers. It is 1931, so this is the first appearance of the idea that would become the stored-program computer.\nStep two: provability becomes arithmetic. Because the axioms are algorithmically recognizable and proof-checking is mechanical, the relation \u0026ldquo;$p$ encodes a proof of the formula encoded by $f$\u0026rdquo; is a primitive recursive relation on numbers, hence expressible inside the system as a formula $\\mathrm{Prf}(p, f)$. The system can now talk about its own proofs, in the only language it has: arithmetic. This step is the hard technical labour of the paper and it is what everyone skips.\nStep three: the diagonal lemma. For any formula $\\psi(x)$ with one free variable, there is a sentence $\\sigma$ with $F \\vdash \\sigma \\leftrightarrow \\psi(\\ulcorner\\sigma\\urcorner)$. The construction is the same self-application as Kleene\u0026rsquo;s recursion theorem (T014): build a formula that, applied to its own number, produces itself. Apply this to $\\psi(x) = \\neg\\exists p, \\mathrm{Prf}(p, x)$ and you get $G$, a sentence equivalent to \u0026ldquo;there is no proof of me.\u0026rdquo;\nStep four. If $F \\vdash G$, then there is a proof of $G$, so $\\neg G$ is provable too, and $F$ is inconsistent. If $F \\vdash \\neg G$, then $F$ proves there is a proof of $G$; with consistency (Rosser\u0026rsquo;s refinement removes the need for the stronger $\\omega$-consistency Gödel originally assumed) this also collapses. So neither is provable. And since $G$ is unprovable, what $G$ asserts is the case — $G$ is true in the standard model, and unprovable in $F$.\nStep five: the second theorem, which is the good one. The argument of step four is itself elementary and arithmetizable. Formalize \u0026ldquo;if $F$ is consistent then $G$ is unprovable\u0026rdquo; inside $F$: you get $F \\vdash \\mathrm{Con}(F) \\to G$. If $F$ could prove $\\mathrm{Con}(F)$, it could prove $G$, which step four forbids. Hence $F \\nvdash \\mathrm{Con}(F)$. The elegance is that the second theorem needs no new construction, only the observation that the first proof can be carried out inside the system.\nThe relationship to the halting problem. T010 is the same theorem in computational dress, and each gives a proof of the other. If provability were decidable you could decide halting by asking for a proof that a machine halts; conversely, the set of provable sentences is recursively enumerable while the set of true arithmetic sentences is not, so they cannot coincide. Gödel came first by five years, which is why this post is placed as the prehistory of Part II rather than a corollary of it. Turing\u0026rsquo;s contribution was to make the diagonalization mechanical, and that is what made it engineering.\nForbids A single formal system settling all mathematical questions. Hilbert\u0026rsquo;s programme, as originally stated, is dead. There is no algorithmic axiom set from which every arithmetic truth follows.\nA system proving its own consistency. If your proof checker verifies a proof of its own soundness, you have found a bug or an inconsistency, not a reassurance. This is a real design constraint: Coq\u0026rsquo;s soundness is proved in a metatheory strictly stronger than Coq (typically ZFC plus inaccessible cardinals), and that regress does not terminate.\nA complete, sound, effective decision procedure for arithmetic. No tool will ever decide arbitrary statements about integers. Every SMT solver\u0026rsquo;s integer arithmetic is either incomplete or restricted to a decidable fragment, and this is why.\nEscaping by adding $G$ as an axiom. The new system $F + G$ is still effectively axiomatizable and still arithmetic, so it has its own $G\u0026rsquo;$. The process never terminates, and transfinite iteration (Turing\u0026rsquo;s ordinal logics) just relocates the problem into naming ordinals.\nDoes not forbid It does not say your program cannot be verified, and this is the misreading that costs money. seL4 is a fully verified microkernel with a machine-checked proof of functional correctness. CompCert is a verified C compiler. The CakeML compiler is verified down to machine code. Incompleteness concerns sentences constructed by diagonalization; it says nothing about whether your concrete specification is provable, and empirically the answer is usually yes. The obstacle in practice is always effort, never Gödel.\nIt does not say humans out-perform machines. The Lucas–Penrose argument claims we \u0026ldquo;see\u0026rdquo; $G$ is true while machines cannot. But we see it only given consistency of $F$ — and any formal system can also prove $\\mathrm{Con}(F) \\to G$. The human is doing exactly the inference the machine can do, and the claim that humans know their own consistency is unsupported and probably false.\nIt does not apply to systems too weak for arithmetic, and lots of useful systems are. Presburger arithmetic (addition, no multiplication) is complete and decidable, and is what array-bounds and loop-dependence analyses actually use. The theory of real closed fields is decidable (Tarski), which is why geometric solvers work. Propositional logic is decidable. Weakness is a feature, and this is the same trade as T012\u0026rsquo;s total languages and T038\u0026rsquo;s parser levels.\nIt does not make consistency unknowable. Gentzen proved the consistency of Peano arithmetic in 1936 using transfinite induction up to $\\varepsilon_0$. That is a real proof; it just uses a principle PA does not have. The theorem forbids self-certification, not certification.\nIt does not make undecidable statements exotic. Independent statements have turned out to be mathematically natural: the Paris–Harrington theorem, Goodstein\u0026rsquo;s theorem, and Kruskal\u0026rsquo;s tree theorem are all independent of PA and all things someone would want to prove. So \u0026ldquo;unprovable statements are contrived self-referential curiosities\u0026rdquo; is a comforting claim that stopped being true in 1977.\nBoundary Löb\u0026rsquo;s theorem. If $F \\vdash (\\mathrm{Prov}(\\ulcorner A \\urcorner) \\to A)$ then $F \\vdash A$. Believing your own proofs only about $A$ already commits you to $A$. The second theorem is the case $A = \\bot$, and Löb is the sharper statement. Tarski\u0026rsquo;s undefinability of truth. Truth is not arithmetically definable at all, a stronger and cleaner result than incompleteness, and it explains why the liar sentence does not simply live inside the system. Completeness vs. incompleteness. Gödel also proved the completeness theorem for first-order logic: every logically valid sentence is provable. There is no conflict. Completeness is about logical consequence from arbitrary axioms; incompleteness is about a specific structure, the standard integers, which no first-order axiom set pins down. The verification stack in practice. Since no system certifies itself, real systems shrink what must be trusted: a small kernel (LCF-style), independently checkable proof objects, and multiple independent checkers. The response to the regress is to make the base of it small enough to audit by eye. Reverse mathematics. Rather than asking what is unprovable, ask which axioms a theorem needs. Most of classical analysis lives in weak subsystems, which is a much more informative map of the terrain than \u0026ldquo;some things are unprovable.\u0026rdquo; Read next ","permalink":"https://cs.lozic.me/posts/t015-godels-incompleteness-theorems/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eThe proof assistant is failing you. You are trying to verify a piece of\nconcurrent code in Coq or Lean, and the tool will not accept your termination\nargument. A colleague says, without much conviction, \u0026ldquo;well, Gödel — you can\u0026rsquo;t\nprove everything anyway.\u0026rdquo;\u003c/p\u003e","title":"Godel's Incompleteness Theorems"},{"content":"Symptom Write a program that prints its own source code. No file I/O, no reading __file__, no cheating.\nThe first attempt fails instantly. To print the source you must contain the source, and then the containing text is also part of the source, so you must contain that too. The regress is obviously infinite, and most people conclude after ten minutes that the task is impossible.\nIt is not. Quines exist in every language, and the standard reaction on seeing one is that it is a clever trick particular to the language\u0026rsquo;s string escaping. That reaction is wrong in an interesting way: quines are not a trick, they are guaranteed. Any Turing-complete language has them, necessarily, and the same guarantee produces things far less cute than quines — self-replicating malware, compilers that reinsert their own backdoors, and the fixed points that make several major undecidability proofs work.\nThe engineering question underneath is: when someone tells you a system cannot be built because its specification is self-referential, are they right? Usually they are not, and this theorem is why.\nStatement (Recursion theorem, second form.) For every total computable function $f$ there exists a program index $e$ such that\n$$\\varphi_e = \\varphi_{f(e)}.$$In words: for any computable transformation of programs, there is a program whose behaviour is unchanged by the transformation. Every computable program transformation has a semantic fixed point.\n(Recursion theorem, first form — the one to remember.) For any computable function $g(x, y)$, there exists a program $e$ such that for all $y$:\n$$\\varphi_e(y) = g(e, y).$$That is the version with engineering content: any program may be written as though it has access to its own source code. You can freely use your own index in your own definition, and the theorem promises such a program exists. Quines are the special case $g(e, y) = e$: a program that ignores its input and prints its own text.\nNothing here is optional or language-specific. It holds in any acceptable programming system, which is any system with a universal machine (T013) and s-m-n.\nArgument The proof is short and feels like sleight of hand until you see what it is doing, so it is worth going slowly.\nThe tool: s-m-n. From a program $p$ of two arguments and a value $a$, we can compute a program $s(p, a)$ of one argument that behaves like $p$ with the first fixed to $a$. This is partial application, and it is computable because you can literally construct the source: take $p$\u0026rsquo;s text and prepend an assignment. Nothing deep, but it is what makes self-reference constructible rather than merely wished for.\nThe construction. Given $f$, define a helper $h$ which on input $x$ computes $s(x, x)$ — the program that runs $x$ with itself as first argument. Now define a two-argument program $v(x, y) = \\varphi_{f(h(x))}(y)$: apply $h$ to $x$, apply $f$ to that, and run the result on $y$. Let $n$ be an index for $v$, and set $e = h(n) = s(n,n)$.\nThen $\\varphi_e = \\varphi_{s(n,n)} = \\varphi_n(n, \\cdot) = v(n, \\cdot) = \\varphi_{f(h(n))} = \\varphi_{f(e)}$. Done.\nWhat actually happened. The move is doubling: $h(x) = s(x,x)$ feeds a program its own description. This is the same self-application that produces the Y combinator in the $\\lambda$-calculus, $Y = \\lambda f.(\\lambda x.f(xx))(\\lambda x.f(xx))$, where the duplicated $xx$ plays exactly the role $s(x,x)$ plays here. It is also the same move as the diagonal argument (T002) and the halting proof (T010) — with one crucial difference in how the result is used. Diagonalization applies a transformation and derives a contradiction, concluding that something does not exist. The recursion theorem applies a transformation and finds a fixed point, concluding that something does. Same machinery, opposite sign.\nBuilding a quine concretely. The pattern in every language is: write a function of a string that prints the string twice, once as data and once as code, then apply it to its own text. In Python:\ns = \u0026#39;s = %r\\nprint(s %% s)\u0026#39; print(s % s) The %r is doing the \u0026ldquo;as data\u0026rdquo; job and the % is doing the \u0026ldquo;as code\u0026rdquo; job. That is $s(x,x)$ with string formatting standing in for partial application. Every quine in every language is this, and knowing so removes the mystery.\nTrusting trust. Thompson\u0026rsquo;s attack has three stages. Stage one: modify the compiler to insert a backdoor when compiling login. Detectable by reading the compiler source. Stage two: also modify it to insert both modifications when compiling a compiler. Now compile the clean compiler source with the dirty binary, and ship the resulting binary with clean source. The source is clean; the binary reproduces the attack forever. Stage two is exactly the fixed point: a program that, transformed, produces itself. The theorem says such a program must exist, and Thompson built it. Diverse double-compiling is the known defence, and it works by breaking the self-reference with an independent compiler.\nForbids No compiler can detect all self-replicating code. Self-reference is available to every program in the language, and detecting a semantic property of programs is undecidable anyway (T011). Antivirus signature matching catches known replicators, not the class.\nNo language design prevents quines while staying Turing-complete. You cannot remove self-reference by removing reflection or eval. The theorem needs only s-m-n and universality. C has no reflection and has quines.\n\u0026ldquo;That specification is self-referential, so it\u0026rsquo;s impossible\u0026rdquo; is not a valid argument. The theorem says self-referential specifications are generally satisfiable. Reject them for other reasons if you like, but not that one.\nThe trusting-trust attack cannot be ruled out by source review. Reviewing every line of the compiler source is compatible with the binary being backdoored, which is precisely Thompson\u0026rsquo;s point and the reason reproducible builds and bootstrappable toolchains are taken seriously.\nDoes not forbid It does not require reflection, eval, or introspection. This is the practical misreading. Removing eval from your language does not remove self-reference — the program gets its own text by construction, not by lookup. Languages without any reflective facility have quines, and eBPF\u0026rsquo;s verifier does not stop a filter from encoding its own description.\nIt does not make the fixed point useful, or unique. There are infinitely many fixed points for any $f$, and most are enormous and computationally useless. The theorem is an existence result. Constructing a small or fast fixed point is a separate problem, and Kolmogorov complexity (T023) has things to say about how small it can be.\nIt does not mean self-reference is paradoxical. The naive regress fails; the construction succeeds. The difference is that the regress tries to contain the text and the construction computes it. Recursive functions, this in OO languages, and self-hosting compilers are all mundane consequences. GCC compiles GCC and rustc compiles rustc; that is the theorem being routine.\nIt does not defeat all defences against trusting trust. Diverse double-compiling (Wheeler) detects the attack: compile the suspect source with a different compiler, use the result to recompile the source, and compare. The fixed point is specific to a compiler; an independent one breaks it. Reproducible builds and bootstrappable-builds projects like live-bootstrap — which starts from a 357-byte hex monitor and builds up to GCC — are real, working responses.\nIt does not require the transformation to preserve anything. $f$ can be any total computable function: an optimizer, an obfuscator, a minifier. Every one has programs it cannot semantically change. That is a mildly alarming statement about optimizers, and a true one.\nIt does not make self-reproduction a security problem by itself. The theorem is neutral about what the fixed point does. make rebuilding itself, a Rust compiler compiling its own source, a Kubernetes operator that manages its own deployment, and a Git repository containing the tooling that builds it are all self-reference doing useful work. The bootstrapping problem — how do you get the first compiler — is the recursion theorem\u0026rsquo;s practical face, and the answer (write a small one in something else, then grow it) is a construction, not a paradox.\nBoundary Rogers\u0026rsquo; fixed-point theorem. The general form: for any total computable $f$ there is $n$ with $\\varphi_n = \\varphi_{f(n)}$, in any acceptable numbering. The theorem is a property of programming systems as such, not of Turing machines. The relationship to Rice\u0026rsquo;s theorem. T011 falls out in one line: if a non-trivial semantic property were decidable, build $f$ mapping each program to one with the opposite property, and its fixed point contradicts. Same machinery, and this is the shortest known proof of Rice. Gödel\u0026rsquo;s first incompleteness theorem is the same construction in logic (T015). The diagonal lemma builds a sentence asserting its own unprovability, and the construction is line-for-line the recursion theorem with provability in place of computation. The Y combinator. In the untyped $\\lambda$-calculus, $Y$ is the recursion theorem made into a term, and it is how recursion exists in a language with no named functions. It is also why the simply-typed $\\lambda$-calculus is not Turing-complete: types reject $\\lambda x.xx$, which kills self-application and thus general recursion. That is the precise mechanism by which total languages (T012) stay total. Self-replication beyond software. Von Neumann\u0026rsquo;s universal constructor predates the biological discovery of DNA\u0026rsquo;s role and has the same structure: a description used both as instructions and as copied data. Life and quines are the same theorem. Read next ","permalink":"https://cs.lozic.me/posts/t014-kleenes-recursion-theorem/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eWrite a program that prints its own source code. No file I/O, no reading\n\u003ccode\u003e__file__\u003c/code\u003e, no cheating.\u003c/p\u003e\n\u003cp\u003eThe first attempt fails instantly. To print the source you must contain the\nsource, and then the containing text is also part of the source, so you must\ncontain that too. The regress is obviously infinite, and most people conclude\nafter ten minutes that the task is impossible.\u003c/p\u003e","title":"Kleene's Recursion Theorem"},{"content":"Symptom The most upvoted answer in Stack Overflow history is a refusal to answer. Someone asked how to match nested HTML tags with a regular expression, and the reply is a page of escalating horror about the centre not holding. It is funny, and it is also a theorem, which is not obvious from reading it.\nThe practical version of the symptom is quieter and more expensive. Someone writes a regex to strip HTML tags, and it works on the test cases. Then a comment contains a \u0026lt;, or an attribute value contains \u0026gt;, or tags nest three deep, and the regex silently produces wrong output rather than failing. The fix is another regex, with a lookahead. Six months later the sanitizer has fourteen alternations, a security advisory, and nobody willing to touch it.\nThe same shape recurs: parsing JSON with string splitting, validating balanced parentheses in a config file with pattern matching, extracting nested function calls from source with a scanner. In each case the tool cannot express the property, so the code accumulates special cases forever and asymptotically approaches a parser that nobody designed.\nStatement Grammars, and the languages they generate, fall into four strictly nested classes, each corresponding exactly to a class of machine:\nType Grammar Machine Production form 3 Regular Finite automaton $A \\to aB$ or $A \\to a$ 2 Context-free Pushdown automaton $A \\to \\gamma$ 1 Context-sensitive Linear-bounded automaton $\\alpha A \\beta \\to \\alpha\\gamma\\beta$ 0 Unrestricted Turing machine $\\alpha \\to \\beta$ The containment is strict at every level:\n$$\\text{Regular} \\subsetneq \\text{Context-free} \\subsetneq \\text{Context-sensitive} \\subsetneq \\text{Recursively enumerable}$$The engineering content is in the strictness. Each inclusion being proper means there is a concrete language you cannot recognize with the weaker machine, no matter how clever you are. $a^n b^n$ separates regular from context-free. $a^n b^n c^n$ separates context-free from context-sensitive. The halting language separates decidable from recursively enumerable.\nAnd the correspondence between grammar shape and machine is not an analogy. It is an exact equivalence, proved by mutual simulation, which is why the shape of your grammar rules tells you what memory your parser will need.\nArgument Why the machines line up with the grammar forms. The key quantity is memory discipline. A finite automaton has a fixed number of states — a bounded amount of memory — and regular productions $A \\to aB$ are exactly rules that carry no information forward beyond a single symbol of state. A pushdown automaton adds a stack, and context-free productions $A \\to \\gamma$ are exactly what a stack can track, because expanding a nonterminal into a string of symbols to be processed later is pushing. A linear-bounded automaton has a tape proportional to the input, matching non-contracting productions, which can never make the working string shorter than the input. Remove the restriction and you have unbounded tape and a Turing machine (T013).\nSo the hierarchy is really a hierarchy of memory: none, a stack, a linear amount, unbounded. Everything else follows.\nWhy regex cannot match nested tags. The pumping lemma for regular languages says: for any regular language there is a length $p$ such that every string of length at least $p$ splits as $xyz$ with $|xy| \\le p$, $|y| \\ge 1$, and $xy^iz$ in the language for all $i \\ge 0$. The reason is pigeonhole (T001): a DFA with $p$ states reading a string of length $\\ge p$ must revisit a state, and the loop between the two visits can be repeated any number of times without the machine noticing.\nApply it to ${a^nb^n}$. Given $p$, take $a^pb^p$. Since $|xy| \\le p$, $y$ is all $a$s, so pumping to $xy^2z$ adds $a$s without $b$s, giving a string not in the language. Contradiction. There is no regular language of balanced things, and nested HTML tags are exactly that. No regex — in the formal sense — will ever match them, however long.\nMyhill–Nerode (T040) is the sharper tool: $a^nb^n$ has infinitely many distinguishable prefixes, since $a^i$ and $a^j$ with $i \\ne j$ are distinguished by $b^i$. Infinitely many equivalence classes, no finite automaton, no argument about pumping needed.\nWhy context-free is where programming languages sit. Nested constructs — parentheses, blocks, expressions — are exactly what a stack handles, and the correspondence is literal: the recursive descent parser\u0026rsquo;s call stack is the pushdown automaton\u0026rsquo;s stack. A grammar that is LL(k) or LR(k) admits a deterministic linear-time parser, which is why real compilers use those subclasses rather than the full class.\nWhy languages are not purely context-free either. \u0026ldquo;Every variable must be declared before use\u0026rdquo; needs unbounded correlation between distant parts of the string, which is the $a^nb^nc^n$ shape. This is why every compiler has a semantic analysis phase after parsing: the type checker is doing the context-sensitive part, and no amount of grammar work will absorb it. C is worse still — x * y; is a declaration or a multiplication depending on whether x is a typedef, so the grammar is genuinely ambiguous without symbol-table feedback, and that is the notorious \u0026ldquo;lexer hack.\u0026rdquo;\nForbids Matching balanced or nested structures with a regular expression. HTML, XML, JSON, S-expressions, nested comments, and balanced parentheses are all context-free and not regular. This is not a limitation of a library, and no feature request fixes it.\nA finite automaton counting without a bound. Anything requiring \u0026ldquo;the same number of X as Y\u0026rdquo; with unbounded counts is out. This is why you cannot validate that a config has matching begin/end markers with a lexer alone.\nA pushdown automaton enforcing cross-references. Declare-before-use, type agreement, and matching function arity to call sites are context-sensitive. Attempting them in the grammar produces exponential rule blowup and still fails.\nA single unified \u0026ldquo;just parse it\u0026rdquo; phase. The hierarchy is why compilers have stages. Each stage handles the level its machine can express, and the structure of the toolchain is a consequence of the theorem, not a convention.\nDoes not forbid Modern \u0026ldquo;regex\u0026rdquo; engines are not regular, and this trips people constantly. PCRE, Perl, Python\u0026rsquo;s re, and Java\u0026rsquo;s regex all have backreferences and recursion. Backreferences alone push you past context-free — (a+)b\\1 is not even context-free — and PCRE\u0026rsquo;s recursive patterns (?R) genuinely can match balanced parentheses. So \u0026ldquo;you can\u0026rsquo;t match HTML with a regex\u0026rdquo; is true of formal regular expressions and false of PCRE, which can do it badly. The reason not to is now practical: backtracking engines on these patterns hit exponential blowup, which is the ReDoS vulnerability class. RE2 and Go\u0026rsquo;s regexp deliberately omit backreferences to stay regular and guarantee linear time, which is the same trade as T012\u0026rsquo;s total languages.\nBounded nesting is regular. \u0026ldquo;Match tags nested at most three deep\u0026rdquo; is a finite condition and a perfectly good regex, if an ugly one. Many real problems have a bound, and if yours does, the theorem does not apply to you. The failure mode is assuming a bound that the input does not respect.\nContext-free parsers handle plenty of context-sensitive-looking things via side channels. Symbol tables, attribute grammars, and two-pass compilation are all ways to do the context-sensitive work outside the grammar. The theorem says the grammar cannot express it, not that your program cannot check it.\nThe hierarchy is not the only classification, and modern parsing lives between the levels. Parsing expression grammars (PEGs) are incomparable with context-free: they handle some non-context-free languages, and their ordered choice makes them unambiguous by construction, but they cannot express some context-free languages. Tree-adjoining grammars and other mildly context-sensitive formalisms sit strictly between types 2 and 1, and are what computational linguistics actually uses.\nPractical parsers are not restricted to LL/LR. GLR (used by tree-sitter and Bison\u0026rsquo;s %glr-parser) handles all context-free grammars including ambiguous ones, in cubic worst case and near-linear practice; Earley parsing does the same. \u0026ldquo;Not LR(1)\u0026rdquo; is a statement about a parser-generator algorithm, not about the hierarchy.\nBoundary Closure properties are the practical tool. Regular languages are closed under union, intersection, and complement; context-free languages are closed under union but not intersection or complement. That is why you can compose lexer rules freely and cannot intersect two grammars, and it explains a lot of parser-generator behaviour. Decidability degrades as you climb. Equivalence of two DFAs is decidable in polynomial time; equivalence of two context-free grammars is undecidable, as is ambiguity. Every step up the ladder buys expressiveness and pays in analyzability, which is the same trade as T012. Complexity, not just computability. Regular is $O(n)$ with $O(1)$ memory. Context-free is $O(n^3)$ in general (CYK, Earley), $O(n)$ for LL/LR. Context-sensitive recognition is PSPACE-complete. The ladder is a cost curve. The security reading. LANGSEC argues that most injection vulnerabilities come from recognizing input with a machine weaker than the language actually requires — ad-hoc parsers for context-free formats. \u0026ldquo;Full recognition before processing\u0026rdquo; is the discipline the hierarchy implies. Two formalisms, one power, repeatedly. Kleene\u0026rsquo;s theorem (T039) shows regular expressions and finite automata are the same class; Codd\u0026rsquo;s theorem (T069) does the same for relational algebra and calculus. This coincidence is common enough to be a genre. Read next ","permalink":"https://cs.lozic.me/posts/t038-the-chomsky-hierarchy/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eThe most upvoted answer in Stack Overflow history is a refusal to answer. Someone\nasked how to match nested HTML tags with a regular expression, and the reply is\na page of escalating horror about the centre not holding. It is funny, and it is\nalso a theorem, which is not obvious from reading it.\u003c/p\u003e","title":"The Chomsky Hierarchy"},{"content":"Symptom You write a Python program. It runs. What ran it?\nCPython — itself a program, written in C, compiled to x86 instructions, executed by a CPU whose control unit is arguably interpreting microcode, possibly inside a virtual machine, on a kernel that scheduled it, all perhaps within a container image. At no point in that stack does anyone find it strange that a program\u0026rsquo;s job is running other programs. We build emulators, JITs, WebAssembly runtimes, Docker, QEMU, and browsers that download and execute arbitrary code from strangers, and treat every layer as ordinary engineering.\nIt is worth stopping to notice how strange this is. A machine that does one fixed thing — a calculator, a washing machine controller — is the intuitive picture of a machine. The universal machine is a device whose fixed behaviour is to have no fixed behaviour, and that is not a design pattern anyone invented. It is a theorem, proved in 1936, before a single one of these machines existed.\nAnd the same theorem has a second face that is much less comfortable: if a machine can run any program you hand it, it can run any program an attacker hands it too. Every sandbox escape, every macro virus, every eval injection is this theorem being used by the other side.\nStatement There exists a Turing machine $U$ such that for every Turing machine $M$ and every input $w$:\n$$U(\\langle M \\rangle, w) = M(w)$$where $\\langle M \\rangle$ is a finite string encoding $M$. $U$ halts exactly when $M$ halts on $w$, and produces the same output.\nOne fixed machine, with a fixed finite state set and a fixed transition table, simulates every machine there is — including ones with far more states than it has, and including itself.\nThree consequences fall out immediately:\nPrograms are data. A machine is a finite string, so it can be an input, stored in memory, generated, transformed, and transmitted like any other data. A single piece of hardware suffices. You do not need one device per problem. This is the stored-program computer. The set of machines is countable. Each is a finite string over a finite alphabet, so they can be enumerated $M_1, M_2, M_3, \\dots$ — which is the premise the diagonal argument (T002) needs to prove things uncomputable. Argument The construction is unglamorous, which is the point: universality is cheap.\nEncoding. Fix a scheme that writes a machine\u0026rsquo;s transition table as a string. States as $q_1, q_2, \\dots$ in unary, symbols likewise, and each transition $\\delta(q_i, a_j) = (q_k, a_l, D)$ as a delimited tuple. Concatenate. Any injective, decodable scheme works, and the theorem does not depend on which.\nSimulation. $U$ uses its tape as three regions: the description $\\langle M \\rangle$, the contents of $M$\u0026rsquo;s simulated tape, and $M$\u0026rsquo;s current state plus head position. The loop is:\nRead the simulated state and the symbol under the simulated head. Scan $\\langle M \\rangle$ for the matching transition tuple. Rewrite the simulated tape cell, move the simulated head marker, update the recorded state. If the state is accepting or rejecting, halt accordingly. Otherwise repeat. Every step is finite bookkeeping over a finite table, so a fixed finite control does it. The overhead is polynomial — each simulated step costs a scan proportional to the description plus the tape used — which is why simulation is practically viable and not merely possible.\nThe size of $U$. People have built very small universal machines: Rogozhin found a family including a 4-state 6-symbol machine, and Wolfram\u0026rsquo;s 2-state 3-symbol machine was proved universal in 2007, though under a nonstandard initial-condition convention that is still debated. Rule 110 is universal. The threshold is low enough to trip over.\nWhy this is the pivot of Part II. Universality is what makes the negative results possible. To diagonalize you must be able to feed a machine its own description, and to feed a machine a description you need descriptions to exist and be executable. $U$ provides both. The halting problem (T010), Rice\u0026rsquo;s theorem (T011), and the recursion theorem (T014) are all consequences of the same fact: the ability to run arbitrary code is inseparable from the inability to analyze arbitrary code. You cannot keep the first and reject the second.\nFrom theorem to hardware. Von Neumann\u0026rsquo;s EDVAC report proposes exactly this: a single memory holding both instructions and data, with a control unit fetching and decoding. Before it, machines like ENIAC were programmed by rewiring — a new problem meant a physical reconfiguration taking days. The stored-program design makes the program a value. That is why compilers, linkers, JITs, and self-modifying code exist, and also why buffer overflows can execute: if the machine does not distinguish instructions from data, neither will its bugs. W^X policies, NX bits, and Harvard-architecture microcontrollers are all retrofitted attempts to put back a distinction the theorem removed.\nForbids It forbids a general-purpose machine that cannot run hostile code. If your platform executes arbitrary user programs, it executes malicious ones. There is no configuration of a universal machine that runs all good programs and no bad ones, because \u0026ldquo;bad\u0026rdquo; is a semantic property (T011). Every practical defense is a restriction of universality: capabilities, seccomp filters, verified bytecode, memory limits, or dropping Turing-completeness entirely.\nIt forbids \u0026ldquo;we\u0026rsquo;ll just check the program before running it.\u0026rdquo; The universal machine\u0026rsquo;s own existence hands the halting problem its input, so the general check is impossible.\nIt forbids needing special-purpose hardware for computability reasons. If a function is computable at all, your laptop computes it. Reasons to build an ASIC are always about speed, energy, or cost, never about capability. Anyone claiming their hardware computes something a CPU cannot is claiming to falsify T012.\nDoes not forbid It does not make simulation free, and the constant matters enormously. Interpretation typically costs 10–100x; QEMU\u0026rsquo;s dynamic translation and Rosetta 2\u0026rsquo;s ahead-of-time translation both exist to claw that back, and Rosetta gets to roughly 80% of native by translating rather than interpreting. A JIT can occasionally beat static compilation using runtime type feedback, which is not a contradiction: the theorem never said the simulated program runs slower, only that simulation is possible.\nIt does not prevent building useful restricted machines. eBPF runs user-supplied code in the Linux kernel safely precisely because its verifier rejects unbounded loops, making it non-universal. Bitcoin Script has no loops. WebAssembly is universal but runs in a sandbox with no ambient authority, which is a different and effective defense: restrict the interface, not the computation. SQL\u0026rsquo;s core is not universal, which is why query planners work.\nIt does not mean all universal systems are interchangeable in practice. The lambda calculus and a modern CPU compute the same functions with wildly different constants and ergonomics. Universality is a statement about the boundary of the possible, and everything inside that boundary is engineering.\nIt does not require self-interpretation to be paradoxical. $U$ can simulate $U$ simulating $U$, and this is fine — each level is just data to the level below, with slowdown compounding. Metacircular interpreters, from the Lisp eval in McCarthy\u0026rsquo;s 1960 paper to PyPy\u0026rsquo;s RPython toolchain, are practical engineering. PyPy in particular is a working demonstration that a program that takes an interpreter as input can produce a fast implementation of it.\nIt does not make every layer of simulation worth having. Each level of interpretation multiplies the constant, which is why production stacks collapse them: a JIT removes the interpreter, a container is not a virtual machine, and hardware virtualization extensions (VT-x, EPT) exist to let the CPU execute guest instructions directly rather than have a hypervisor simulate them. The theorem licenses the tower; engineering flattens it wherever it can.\nIt does not say the encoding matters. Any decodable scheme gives the same theorem, and Gödel numbering, ASCII source, and a serialized AST are all fine. This is why \u0026ldquo;programs are data\u0026rdquo; holds for text files, bytecode, and syntax objects alike, and why Lisp macros are not a special case of anything.\nBoundary The s-m-n theorem. The formal statement of partial application: from a program of two arguments you can compute a program of one, with the first fixed. Together with universality it characterizes what an acceptable programming system is, and it is what specialization, currying, and partial evaluation formalize. Futamura projections. Specializing an interpreter to a fixed program is compilation; specializing the specializer is a compiler generator. Three levels, all consequences of s-m-n plus universality, and all implemented in real partial-evaluation systems. Universality without full computation. Rule 110, the Game of Life, and Magic: The Gathering are universal. That the threshold is this low is why accidental Turing-completeness keeps showing up in font rendering, packet filters, and build systems. The security reading. \u0026ldquo;Programs are data\u0026rdquo; is the theorem; \u0026ldquo;data is programs\u0026rdquo; is the vulnerability class. SQL injection, XSS, deserialization attacks, and format-string bugs are all confusion at the boundary. The language-theoretic security position is that this is a parsing problem, and it connects directly to the Chomsky hierarchy (T038). Read next ","permalink":"https://cs.lozic.me/posts/t013-the-universal-machine/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eYou write a Python program. It runs. What ran it?\u003c/p\u003e\n\u003cp\u003eCPython — itself a program, written in C, compiled to x86 instructions, executed\nby a CPU whose control unit is arguably interpreting microcode, possibly inside\na virtual machine, on a kernel that scheduled it, all perhaps within a container\nimage. At no point in that stack does anyone find it strange that a program\u0026rsquo;s\njob is running other programs. We build emulators, JITs, WebAssembly runtimes,\nDocker, QEMU, and browsers that download and execute arbitrary code from\nstrangers, and treat every layer as ordinary engineering.\u003c/p\u003e","title":"The Universal Machine"},{"content":"Symptom Somebody says \u0026ldquo;CSS is Turing-complete\u0026rdquo; and the room splits. Half the people treat it as a joke about a styling language, the other half as a serious claim with consequences. Both reactions are common and only one is right.\nThe same phrase turns up in places that matter. Your YAML configuration language grew conditionals and templating, and now someone claims it is Turing-complete and therefore you can no longer statically check deployment configs. Your database migration DSL acquired loops. A security researcher demonstrates computation inside x86\u0026rsquo;s MMU fault handler, or inside printf format strings, or in Magic: The Gathering. Every one of these is presented as though it establishes something.\nThe trouble is that \u0026ldquo;Turing-complete\u0026rdquo; sounds like a statement about speed, or expressiveness, or practicality, and it is none of those. It is a statement about a boundary, and understanding why the boundary exists at all — why there is a single line rather than a spectrum of computational power — requires the one entry in this series that is not a theorem.\nStatement Every function that is effectively calculable — computable by any mechanical procedure a human or machine could carry out with unlimited time and paper — is computable by a Turing machine.\nThat is the thesis. Note carefully what it is not: it is not a theorem, and cannot be, because \u0026ldquo;effectively calculable\u0026rdquo; is an informal notion. The thesis identifies an intuitive concept with a formal one, and that identification is not the kind of thing a proof can establish. It is an empirical and conceptual claim, and after ninety years it is as well-supported as anything in science.\nThe formal half of the story is a theorem, and a striking one: all of the proposed formalizations of computation are equivalent. Turing machines, Church\u0026rsquo;s $\\lambda$-calculus, Gödel and Herbrand\u0026rsquo;s general recursive functions, Post\u0026rsquo;s canonical systems, register machines, cellular automata, tag systems, and every real programming language compute exactly the same set of functions. Not similar sets. The same set.\nThis gives the practical corollary that people mean when they use the phrase: a system is Turing-complete if it can simulate a Turing machine, and any two Turing-complete systems can simulate each other. Turing-completeness is not a degree. You have it or you do not, and if you have it you have all of it.\nArgument The thesis rests on three legs, and they are worth separating because people usually only know the first.\nConfluence. In 1936, three people independently formalized computation from completely different starting points. Church built the $\\lambda$-calculus from function abstraction and application. Gödel and Herbrand defined general recursive functions from equations over the naturals. Turing built an idealized clerk with a tape. These have no surface resemblance whatsoever — one is symbolic rewriting, one is number theory, one is a machine — and they were proved equivalent almost immediately. Since then, every new model has landed in the same place: quantum computers change the efficiency landscape and compute exactly the same functions; DNA computing, membrane computing, and the various unconventional models all fall in. Ninety years of people trying to escape, nobody escaping.\nTuring\u0026rsquo;s own argument, which is the actually convincing one. Turing did not argue from examples. In section 9 of his 1936 paper he analyzes what a human computer — the job title, a person doing calculation with pencil and paper — physically can do. The observations: a person can only distinguish finitely many symbols, because arbitrarily similar symbols become indistinguishable; can only be in finitely many states of mind, for the same reason; can only attend to a bounded portion of the workspace at once; and can only move attention a bounded distance in one step. Every one of those is a claim about finite physical beings, not about mathematics. Given all four, the behaviour is exactly a Turing machine. This is a derivation from physical constraints, which is why the thesis feels like more than a coincidence.\nRobustness under modification. Adding power to the Turing machine does not add power. Multiple tapes, two dimensions, nondeterminism, randomness, unbounded parallelism: each is simulable, usually with polynomial overhead (nondeterminism costs exponential time but not computability). Models designed to be tiny are equally powerful — Rule 110, the two-counter Minsky machine, the SKI combinators, Conway\u0026rsquo;s Game of Life. The class is stable in both directions, which is what you expect of something natural rather than arbitrary.\nWhy the line is sharp. The reason Turing-completeness is binary is that once a system can do a small handful of things — store unbounded state, branch on it, and loop — it can interpret a universal machine (T013), and thereby simulate any computation at all. There is no way to have half of that. The accidental Turing-completeness of printf and the x86 MMU is a consequence: the threshold is so low that stumbling over it is easy.\nForbids No physical device computes more than a Turing machine. This is the thesis\u0026rsquo;s real content. Every impossibility result that follows in Part II — the halting problem, Rice\u0026rsquo;s theorem, undecidability of first-order validity — inherits its scope from here. Without the thesis, \u0026ldquo;no Turing machine can decide halting\u0026rdquo; is a narrow statement about one formalism; with it, it is a statement about everything you could ever build.\nNo programming language is more expressive than another, in the sense of which functions it can compute. Assembly and Haskell compute the same set. Arguments about language power are arguments about ergonomics, safety, and concision — real things, but not this.\nNo amount of DSL restraint helps once you cross the line. If your config language is Turing-complete, then \u0026ldquo;does this config terminate?\u0026rdquo; and \u0026ldquo;does it ever access this resource?\u0026rdquo; are undecidable (T011). You do not get to be slightly Turing-complete and keep your static analysis.\nDoes not forbid It does not say all models are equally efficient, and this is the most consequential confusion. Quantum computers compute the same functions and Shor\u0026rsquo;s algorithm still factors in polynomial time. A single-tape Turing machine needs $\\Theta(n^2)$ steps to do what a two-tape machine does in $O(n)$. The Church–Turing thesis is about computability; the extended Church–Turing thesis, which claims all reasonable models are polynomially equivalent, is a separate claim and quantum computing is widely believed to falsify it.\nIt does not say Turing-completeness is desirable. Most well-designed configuration and query languages deliberately stop short. SQL\u0026rsquo;s core relational algebra is not Turing-complete, which is exactly why a query planner can exist. eBPF verifies termination by rejecting unbounded loops. Coq and Agda are total by design, so a well-typed program provably terminates. Dhall, Starlark, and CUE all draw the line on purpose. Non-Turing-completeness is a feature you pay for and get analyzability in return, and Bitcoin Script\u0026rsquo;s deliberate lack of loops is a security property.\nIt does not say a Turing-complete system is practical. CSS with HTML is Turing-complete via Rule 110, and requires a user to click for each step. Magic: The Gathering is Turing-complete. These are real proofs of a mathematical property with zero engineering consequence, and treating \u0026ldquo;Turing-complete\u0026rdquo; as \u0026ldquo;powerful\u0026rdquo; is how the phrase gets misused. What actually matters for a config language is whether the analysis you want is decidable, which is a narrower question than completeness.\nIt does not rule out hypercomputation as mathematics. Oracle machines, infinite-time Turing machines, and Zeno machines are perfectly coherent mathematical objects and are studied seriously; the thesis claims only that none of them can be built. Relativized computation is a central tool in complexity theory (T030), and it does not threaten the thesis.\nIt does not depend on machines being deterministic or discrete-time. Gandy extended the analysis to parallel discrete machines, and the analysis survives. What it does need is discreteness: analog models with infinite-precision reals can compute non-computable functions, which is a statement about the physical plausibility of infinite precision rather than about computation.\nBoundary Physical Church–Turing. The strongest form claims that no physical process computes a non-computable function. This is a claim about physics, not mathematics, and is genuinely open — it is entangled with whether physical quantities are ultimately discrete and whether spacetime permits the relativistic supertasks (Malament–Hogarth spacetimes) that would allow infinitely many steps in finite proper time. What would falsify it. A physically realizable device that decides halting. Nobody has any idea how to build one, and Gandy\u0026rsquo;s principles say which one you would need to break: unbounded information density in a bounded region. Bekenstein\u0026rsquo;s bound from physics says you cannot. Total languages are the useful frontier. The interesting design space is not \u0026ldquo;complete or not\u0026rdquo; but how much you can compute while staying decidable. Primitive recursive functions, System F, and Coq\u0026rsquo;s terminating fragment cover effectively every program anyone writes, and give you termination for free. The thesis is the load-bearing assumption of Part II. T010 and T011 prove things about Turing machines. That those results apply to your Python program, your CI pipeline, and any machine anyone builds is precisely what the thesis buys, and it is worth being conscious that it is an assumption. Read next ","permalink":"https://cs.lozic.me/posts/t012-the-church-turing-thesis/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eSomebody says \u0026ldquo;CSS is Turing-complete\u0026rdquo; and the room splits. Half the people\ntreat it as a joke about a styling language, the other half as a serious claim\nwith consequences. Both reactions are common and only one is right.\u003c/p\u003e","title":"The Church–Turing Thesis"},{"content":"Symptom Your service handles four billion requests a month and you want the error rate. Computing it exactly means a job over four billion log lines. Someone suggests sampling ten thousand of them.\nThe objection arrives immediately: ten thousand out of four billion is 0.00025% of the data. How could that possibly tell you anything? The intuition that a sample must be some meaningful fraction of the population is close to universal and completely wrong, and the wrongness is the point of this post.\nThe same shape appears everywhere. A load balancer hashes requests to 100 backends: the expected load is 1%, but how far from 1% will the worst backend be? A randomized algorithm succeeds with probability 2/3, and you want failure probability under $10^{-9}$ — how many repetitions? A cache with a 90% expected hit rate: how often is a one-minute window under 80%?\nLinearity of expectation (T007) gives you the mean of all of these for free and then goes silent. The mean is not the question. The question is whether you will actually see the mean, and how badly a bad day can deviate. That gap is exactly what concentration bounds fill, and the answer is much stronger than most people\u0026rsquo;s intuition allows.\nStatement Hoeffding\u0026rsquo;s inequality. Let $X_1, \\dots, X_n$ be independent random variables with $X_i \\in [a_i, b_i]$, and let $S = \\sum X_i$. Then for any $t \u0026gt; 0$:\n$$P\\big(|S - E[S]| \\ge t\\big) \\le 2\\exp\\left(\\frac{-2t^2}{\\sum_i (b_i - a_i)^2}\\right).$$For the common case of $n$ variables in $[0,1]$ and the sample mean $\\bar{X} = S/n$:\n$$P\\big(|\\bar{X} - \\mu| \\ge \\varepsilon\\big) \\le 2e^{-2n\\varepsilon^2}.$$Chernoff bound (multiplicative form). For independent $X_i \\in {0,1}$ with $S = \\sum X_i$ and $\\mu = E[S]$, for $0 \u0026lt; \\delta \u0026lt; 1$:\n$$P\\big(S \\ge (1+\\delta)\\mu\\big) \\le e^{-\\delta^2\\mu/3}, \\qquad P\\big(S \\le (1-\\delta)\\mu\\big) \\le e^{-\\delta^2\\mu/2}.$$The two headline facts, and they are the ones to actually remember:\nThe deviation probability decays exponentially, not polynomially. Chebyshev gives you $1/t^2$; these give you $e^{-t^2}$. That difference is the reason sampling works at all. The sample-size requirement $n \\ge \\frac{1}{2\\varepsilon^2}\\ln\\frac{2}{\\delta}$ does not mention the population size. Not as a term, not as a factor. Ten thousand samples say the same thing about four billion requests as about four million. Argument The proof is one idea used three times, and it is worth seeing because the idea generalizes far past this theorem.\nStep one: Markov\u0026rsquo;s inequality. For a non-negative $X$, $P(X \\ge a) \\le E[X]/a$. Proof: $E[X] \\ge E[X \\cdot \\mathbb{1}(X \\ge a)] \\ge a \\cdot P(X \\ge a)$. This is very weak — it uses only the mean — but it is free.\nStep two: apply Markov to the exponential. Instead of bounding $S$, bound $e^{\\lambda S}$ for a parameter $\\lambda \u0026gt; 0$. Since $e^{\\lambda x}$ is increasing, $S \\ge t$ exactly when $e^{\\lambda S} \\ge e^{\\lambda t}$, so\n$$P(S \\ge t) \\le \\frac{E[e^{\\lambda S}]}{e^{\\lambda t}}.$$This is the whole trick. Exponentiating turns a linear deviation into a multiplicative one and, crucially, makes the sum in the exponent into a product of independent terms: $E[e^{\\lambda \\sum X_i}] = \\prod E[e^{\\lambda X_i}]$. That factorization is where independence is spent — the only place, and it is non-negotiable, which is the sharp contrast with T007.\nStep three: bound each factor and optimize $\\lambda$. Each $E[e^{\\lambda X_i}]$ is the moment generating function; for a bounded variable Hoeffding\u0026rsquo;s lemma gives $E[e^{\\lambda(X_i - E X_i)}] \\le e^{\\lambda^2(b_i-a_i)^2/8}$ by convexity. The product is $e^{\\lambda^2 \\sum (b_i-a_i)^2/8}$, so\n$$P(S - E[S] \\ge t) \\le \\exp\\left(\\frac{\\lambda^2\\sum(b_i-a_i)^2}{8} - \\lambda t\\right).$$Now $\\lambda$ was free the whole time, so minimize over it. The exponent is a parabola in $\\lambda$, minimized at $\\lambda = 4t/\\sum(b_i - a_i)^2$, and substituting gives exactly $-2t^2/\\sum(b_i-a_i)^2$. Doubling for the two tails gives the theorem.\nWhy the population size vanishes. Look at the sample-mean form: the only $n$ is the sample count, in the exponent. There is nowhere for a population size to enter, because the samples are drawn i.i.d. and each one is an independent draw from the same distribution whether that distribution describes four million or four billion requests. This is not an approximation that gets worse for large populations. For sampling without replacement from a finite population, Hoeffding\u0026rsquo;s paper shows the bound still holds, and in fact finiteness only helps.\nThe numbers. With $\\varepsilon = 0.01$ and $\\delta = 0.01$, $n \\ge \\frac{\\ln 200}{2 \\times 0.0001} = 26{,}492$. So about 26,000 sampled log lines pin the error rate to within one percentage point, with 99% confidence, for any population. Want a tenth of a point? $\\varepsilon$ is squared, so it costs 100 times as many: 2.6 million. That quadratic term is the real constraint on sampling, and it is why precision, not scale, is what gets expensive.\nBoosting. A randomized algorithm correct with probability $2/3$, run $k$ times with a majority vote, fails only if the number of correct runs drops to $k/2$ — a deviation of $\\delta = 1/4$ below the mean $2k/3$. The lower-tail Chernoff bound gives $e^{-\\delta^2\\mu/2} = e^{-k/48}$, so about 1000 repetitions push a 1/3 failure rate below $10^{-9}$. This is why BPP is a robust class: the exact constant in the definition does not matter, because boosting moves it anywhere you like.\nForbids A load balancer whose worst backend is far from average, at scale. With $n$ requests over $m$ backends and $\\mu = n/m$ each, the probability any particular backend exceeds $1.1\\mu$ is $e^{-\\mu/300}$; a union bound over $m$ backends makes the worst case tight once $\\mu$ is a few thousand. Imbalance at scale is evidence of a broken hash or a hot key, not of bad luck. If you observe a 3x hot backend with millions of requests, do not tune the balancer; find the key.\nA sampling estimate being wildly wrong \u0026ldquo;because the sample was small.\u0026rdquo; Once you have computed the required $n$ and drawn i.i.d., the failure probability is what it is. Estimates that come back wrong indicate a sampling bias — the samples were not independent or not from the distribution you think — not insufficient volume. Almost every real sampling failure is this bug.\nMonte Carlo error decaying faster than $1/\\sqrt{n}$. The $\\varepsilon^2$ in the exponent is a hard floor on general sampling: to halve the error you quadruple the work.\nDoes not forbid It does not apply without independence, and this is the difference from T007. Linearity of expectation is unconditional; concentration is not. Sum $n$ perfectly correlated variables and the sum is $n$ times one variable, with no concentration at all. This matters in practice: request latencies within a service are correlated through shared resources, so sampling p99 latency from a burst of consecutive requests will lie to you in a way that sampling error rates from spread-out requests will not. Correlated failures are the reason availability math over \u0026ldquo;independent\u0026rdquo; replicas in one rack is fiction.\nIt does not need full independence, though. Martingale versions (Azuma–Hoeffding) require only that each step be bounded given the past, and McDiarmid\u0026rsquo;s bounded-differences inequality only requires that changing one input moves the output a little. Those cover randomized algorithms whose steps obviously depend on earlier steps, and they are how you analyze most graph algorithms. Reaching for \u0026ldquo;independence fails, so concentration fails\u0026rdquo; is premature.\nIt does not require bounded variables in every form. Hoeffding needs boundedness, but Bernstein\u0026rsquo;s inequality trades it for a variance condition, and sub-Gaussian and sub-exponential tail conditions cover Gaussian and Poisson data. Heavy-tailed distributions — file sizes, request costs, wealth — genuinely do not concentrate, and using Hoeffding on Pareto-distributed data is a real mistake that produces confident nonsense.\nIt does not make an estimate from a biased sample better. Concentration controls variance, never bias. Sampling only the requests that completed tells you the error rate among successful requests, with beautiful tight bounds and no relevance to your question. This bug is invisible to the mathematics.\nBoundary Chebyshev when independence is absent. $P(|X-\\mu| \\ge t) \\le \\sigma^2/t^2$ needs only pairwise independence to compute the variance of a sum, and often that is all you have. Polynomially weak, but it applies. The union bound is the standard partner. Bounding the maximum of many quantities means bounding each and summing failure probabilities. It costs a $\\log$ factor, needs no independence between events, and is how balls-into-bins (T009) and Johnson–Lindenstrauss (T100) both work. Where it does not concentrate. The maximum load in balls-into-bins is $\\Theta(\\log n/\\log\\log n)$, not $O(1)$, even though every bin has expectation Concentration per-bin plus a union bound gives you exactly that gap, and it is real: the mean is 1 and the max is not. Sub-Gaussian is the right abstraction. The condition that makes the Chernoff argument work is a bound on the moment generating function. Bounded, Gaussian, and sub-Gaussian variables all satisfy it; that is the actual boundary of the technique, and everything above is a special case. This is the machinery under PAC learning. Generalization bounds are Hoeffding applied to the training error, with a union bound over hypotheses — which is why VC dimension shows up as the size of the thing you union over. Read next ","permalink":"https://cs.lozic.me/posts/t008-concentration-bounds/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eYour service handles four billion requests a month and you want the error rate.\nComputing it exactly means a job over four billion log lines. Someone suggests\nsampling ten thousand of them.\u003c/p\u003e","title":"Concentration Bounds (Chernoff/Hoeffding)"},{"content":"Symptom You have a dynamic array. push writes one element and bumps a counter, which is clearly $O(1)$ — except when the array is full, in which case it allocates a new buffer of twice the size, copies every element across, and frees the old one. That is $O(n)$.\nSo what do you tell the person reviewing your design doc? The worst-case cost of push is $O(n)$, and that is a true statement. It is also close to useless: it suggests that $n$ pushes cost $O(n^2)$, which is off by a factor of $n$ from what actually happens. Everyone knows the resize is \u0026ldquo;rare enough not to matter,\u0026rdquo; and everyone says so with a hand-wave.\nThe hand-wave is the problem. \u0026ldquo;Rare enough\u0026rdquo; is not an argument, and the cases where the intuition is wrong look identical to the cases where it is right. If you grow the array by a constant rather than doubling it, resizes are still rare — every 1024 pushes, say — and the total is still $\\Theta(n^2)$. Rarity alone does not save you. Something else does, and you need to be able to say what.\nThere is a second, worse symptom: you reach for the wrong tool. Reviewers who know the worst case is bad will ask for the average case, and someone will compute an average over a probability distribution on inputs that nobody has justified. Dynamic arrays have no randomness in them. The resize is not unlikely; it is entirely determined by the sequence of calls. Averaging is the wrong frame, and it will produce a number that means nothing.\nStatement For a sequence of operations, define a potential function $\\Phi$ mapping each state of the data structure to a real number, with $\\Phi(D_0) = 0$ and $\\Phi(D_i) \\ge 0$ for all $i$. Define the amortized cost of operation $i$ as\n$$\\hat{c}_i = c_i + \\Phi(D_i) - \\Phi(D_{i-1})$$where $c_i$ is the actual cost. Then for any sequence of $n$ operations,\n$$\\sum_{i=1}^{n} c_i = \\sum_{i=1}^{n} \\hat{c}_i - \\Phi(D_n) + \\Phi(D_0) \\le \\sum_{i=1}^{n} \\hat{c}_i.$$The total actual cost is bounded by the total amortized cost. If you can show each $\\hat{c}_i \\le k$, you have shown $n$ operations cost at most $kn$, regardless of which operations they were or what order they came in.\nNote what this is not. It is not a probabilistic claim: there is no distribution, no expectation, no \u0026ldquo;typical\u0026rdquo; input. The bound holds for the adversary\u0026rsquo;s worst sequence. It is also not a claim about any individual operation, which may still cost $\\Theta(n)$. It is a claim about totals, and totals are usually what you actually care about.\nArgument The proof of the theorem itself is one line: the sum telescopes. Every $\\Phi(D_i)$ appears once positively and once negatively except the endpoints, so $\\sum \\hat{c}_i = \\sum c_i + \\Phi(D_n) - \\Phi(D_0)$. Since $\\Phi(D_0) = 0$ and $\\Phi(D_n) \\ge 0$, dropping the potential terms only weakens the bound in the safe direction. That is the entire theorem.\nAll the content is in choosing $\\Phi$. The way to think about it: $\\Phi$ is prepaid work. Cheap operations are charged a little extra, which accumulates in the potential; expensive operations spend it down. The two conditions — starts at zero, never goes negative — are exactly the statement that you cannot spend money you never earned.\nThe dynamic array. Let $n$ be the number of elements and $s$ the allocated size. Take $\\Phi = 2n - s$. Just after a resize, $n = s/2$, so $\\Phi = 0$. Each push increments $n$, raising $\\Phi$ by 2, and costs 1, so $\\hat{c} = 1 + 2 = 3$. When a push triggers a resize at $n = s$, the actual cost is $n + 1$ (copy $n$ elements, write one). Before: $\\Phi = 2n - n = n$. After: $n\u0026rsquo; = n+1$, $s\u0026rsquo; = 2n$, so $\\Phi\u0026rsquo; = 2(n+1) - 2n = 2$. The amortized cost is $(n+1) + 2 - n = 3$. Constant, both cases. So $n$ pushes cost at most $3n$.\nNow watch what happens with a constant growth increment $k$. Between resizes you get $k$ pushes to prepay a copy of size $n$, so each push would have to bank $n/k$ — which grows with $n$, so no constant $\\Phi$ can work. The potential method does not just prove doubling works; it shows precisely why the alternative does not, and it locates the failure in the growth rule rather than in the frequency of resizes. Any geometric growth factor $\\alpha \u0026gt; 1$ works, with amortized cost $1 + \\alpha/(\\alpha - 1)$: 3 for $\\alpha = 2$, and 4 for $\\alpha = 1.5$, which is what several real allocators use so that freed blocks can be reused for a later growth.\nThe binary counter. Incrementing an $n$-bit counter flips a variable number of bits: 0111 → 1000 flips four. Let $\\Phi$ be the number of 1 bits. An increment that flips $t$ trailing ones to zero and one zero to one costs $t + 1$ and changes the potential by $1 - t$. Amortized: $(t+1) + (1-t) = 2$. So $n$ increments flip at most $2n$ bits, no matter the starting value. The proof does not care that the expensive increments are rare; it cares that each 1 bit was paid for when it was created.\nWhere it gets deep. The same machinery, with a cleverer $\\Phi$, is what proves splay trees are $O(\\log n)$ amortized without storing any balance information, and Fibonacci heaps\u0026rsquo; $O(1)$ amortized decrease-key — which is what makes Dijkstra $O(E + V \\log V)$. In both cases no individual operation is fast, and the structure is only efficient as a whole. That is not a weakness of the analysis; it is a design strategy the analysis makes available. You are permitted to build a data structure that is sometimes slow, provided you can name the potential.\nForbids The theorem is a proof technique, so what it forbids is a class of wrong conclusions rather than a class of algorithms:\nIt forbids concluding $O(n^2)$ from a worst case of $O(n)$ per operation. Multiplying the worst case by the number of operations is valid but frequently loose by a factor of $n$. If the operations interact through shared state, worst cases may not be simultaneously achievable, and the potential function is how you prove they are not.\nIt forbids a \u0026ldquo;sometimes slow\u0026rdquo; structure hiding an actual $\\Theta(n)$ average. The bound cuts both ways: if no valid $\\Phi$ exists, no amortized bound exists, and the adversary really can make every operation expensive. Failure to find a potential is evidence, and the constant-growth array is the canonical case.\nIt forbids treating amortization as a probabilistic claim. There is nothing to be unlucky about. An adversary choosing the worst possible sequence gets the same bound.\nDoes not forbid It does not promise any individual operation is fast, and this is the misuse that reaches production. A dynamic array push is amortized $O(1)$ and can still take 200 ms when it copies a gigabyte. If you have a latency SLO, amortized bounds are the wrong instrument — your p99.9 sees exactly the operations the amortization averaged away. This is why real-time systems use std::vector::reserve, why the Go runtime and the JVM both moved to incremental and concurrent collectors rather than faster stop-the-world ones, and why Redis\u0026rsquo;s incremental rehashing moves a few buckets per command instead of rebuilding the table at once. Each of those is the same trade: give up a better amortized bound to get a better worst case.\nIt does not require the data structure to be randomized or the input to be benign. Hash tables with random probing need probability. Dynamic arrays, binary counters, and union-find do not — union-find\u0026rsquo;s $O(\\alpha(n))$ bound is amortized and fully deterministic.\nIt does not stop being valid under adversarial input. This is worth separating from the previous point because it is the most common confusion. People assume that anything with \u0026ldquo;average\u0026rdquo; flavor breaks against an attacker. Amortized bounds do not: an attacker who controls the entire call sequence still cannot exceed $3n$ pushes\u0026rsquo; worth of work. Hash-flooding attacks work against expected-time hashing, not against amortized bounds.\nIt does not require operations to be uniform. Splay trees amortize a mixed workload of searches, insertions and deletions under a single potential, and the deep results in this area (Sleator and Tarjan\u0026rsquo;s access lemma) bound non-uniform access patterns better than uniform ones.\nBoundary Amortized vs. average vs. expected. Three different claims. Amortized is worst-case over sequences, with no probability. Average-case assumes an input distribution. Expected-time assumes internal randomness. Quicksort with a random pivot is expected $O(n\\log n)$; a dynamic array is amortized $O(1)$; they are not the same kind of statement and mixing them up produces guarantees you do not have. The accounting method. An equivalent formulation: assign each operation a charge, let cheap ones bank credit on specific objects, and require the balance to stay non-negative. It is often easier to invent than a potential function, and translating between them is mechanical. Persistence breaks it. Amortized bounds assume the structure is used linearly. If a functional data structure lets you re-run the same expensive operation from an old version repeatedly, the prepaid work is spent many times. Okasaki\u0026rsquo;s lazy evaluation with memoization restores the bounds, which is the whole subject of Purely Functional Data Structures. The same tool proves lower bounds. The weight function in an adversary argument (T005) is a potential function used in reverse: bound how much one comparison can change it, and the start and end values force a minimum operation count. Upper and lower bounds, one technique. Amortization and the online setting. Bounding a sequence when you cannot see the future is the competitive-analysis question, and potential functions are the standard tool there too, which is why this post sits in Part X beside the performance results rather than in Part I. Read next ","permalink":"https://cs.lozic.me/posts/t006-amortized-analysis-and-the-potential-method/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eYou have a dynamic array. \u003ccode\u003epush\u003c/code\u003e writes one element and bumps a counter, which\nis clearly $O(1)$ — except when the array is full, in which case it allocates a\nnew buffer of twice the size, copies every element across, and frees the old\none. That is $O(n)$.\u003c/p\u003e","title":"Amortized Analysis and the Potential Method"},{"content":"Symptom You need the largest and second-largest element of an array. The obvious way is two passes: $n-1$ comparisons for the max, then $n-2$ for the max of the rest. That is $2n - 3$.\nSomebody points out you can do better, and shows you a tournament: pair up the elements, play off the winners, and the second-best must have lost directly to the best — so you only need to re-examine the $\\lceil \\log_2 n\\rceil - 1$ elements the champion beat. That is $n + \\lceil \\log_2 n \\rceil - 2$.\nNow the real question, and the one that is actually hard: is that optimal, or is there a cleverer scheme still? You cannot answer it by trying algorithms. You need an argument that rules out all of them at once, including the ones nobody has invented.\nThe decision-tree bound (T003) will not do it here: it gives $\\log_2$ of the number of outcomes, which for \u0026ldquo;find the top two\u0026rdquo; is far too weak. You need a different tool.\nStatement The technique, stated as a game:\nAn adversary answers the algorithm\u0026rsquo;s queries. It does not fix the input in advance. Instead it answers adaptively, keeping its answers consistent with at least one valid input, and choosing each answer to maximize the work still remaining. If the adversary can always keep two different answers alive after $k$ queries, no algorithm can be correct in $k$ queries.\nThe concrete result this post proves:\nFinding both the maximum and second-maximum of $n$ elements requires at least $n + \\lceil \\log_2 n \\rceil - 2$ comparisons in the worst case.\nSince the tournament achieves this, the bound is tight, and the question is closed. Nobody will ever do better, and that is knowable in a page.\nArgument Two separate accounting arguments, one for each term.\nPart 1: at least $n - 1$ comparisons. Every element except the maximum must lose at least one comparison — otherwise, for any element that never lost, the adversary is free to declare it the maximum, and the algorithm\u0026rsquo;s answer is wrong on some consistent input. Each comparison produces exactly one loser. So at least $n-1$ comparisons are needed just to identify the max.\nPart 2: at least $\\lceil \\log_2 n \\rceil - 1$ more. The second-largest element is exactly the largest of those that lost directly to the maximum. So the algorithm must, in effect, determine the max\u0026rsquo;s set of direct victims and find the best among them. If the eventual champion beat $k$ elements directly, finding the best of those $k$ costs at least $k - 1$ further comparisons by the argument above.\nSo the adversary\u0026rsquo;s job is to force $k \\ge \\lceil \\log_2 n \\rceil$: make the champion win many fights.\nThe adversary strategy — weights. Give every element a weight, initially $w_i = 1$. Interpret $w_i$ as \u0026ldquo;the number of elements this one could still be the best of.\u0026rdquo; When the algorithm compares $x$ and $y$, the adversary answers:\nIf $w_x \u0026gt; w_y$: declare $x$ wins. Set $w_x \\leftarrow w_x + w_y$, $w_y \\leftarrow 0$. If $w_x = w_y$: declare $x$ wins (break ties arbitrarily). Same update. The key property: a winner\u0026rsquo;s weight at most doubles per comparison. When $w_x \u0026gt; w_y$, the new weight $w_x + w_y \u0026lt; 2w_x$. When $w_x = w_y$, it is exactly $2w_x$. So after the champion has won $k$ fights, its weight is at most $2^k$.\nAt the end, the champion must have absorbed everything — its weight must be $n$, since every other element has been shown to be worse than something, and all that \u0026ldquo;worse-ness\u0026rdquo; ultimately routes to the max. So\n$$2^k \\ge n \\quad\\Longrightarrow\\quad k \\ge \\lceil \\log_2 n\\rceil$$The champion won at least $\\lceil \\log_2 n \\rceil$ comparisons. Finding the best of its direct victims costs $\\ge \\lceil \\log_2 n \\rceil - 1$ more.\nTotal: $(n-1) + (\\lceil \\log_2 n\\rceil - 1) = n + \\lceil \\log_2 n\\rceil - 2$. $\\blacksquare$\nWhy the adversary is allowed to do this. The move that feels like cheating is that the adversary has not decided the input yet. It is not cheating, and the reason is worth internalizing: at every point the adversary\u0026rsquo;s answers are consistent with at least one real input. If the algorithm stops early and announces an answer, the adversary exhibits a consistent input on which that answer is wrong. Since the algorithm must be correct on all inputs, it must be correct on that one. The adversary is a bookkeeping device for the sentence \u0026ldquo;there is always a bad input remaining,\u0026rdquo; and it lets you construct that input lazily rather than guessing it up front.\nA second example, in two lines, because the technique generalizes. Searching an unsorted array of $n$ elements needs $n$ probes in the worst case: whenever the algorithm probes a cell, the adversary answers \u0026ldquo;not your target.\u0026rdquo; After $n-1$ probes, one cell is unexamined, and the adversary is free to put the target there or not. The algorithm cannot know. Same shape: keep two worlds alive.\nForbids Finding max and second-max in fewer than $n + \\lceil\\log_2 n\\rceil - 2$ comparisons. For $n = 1024$: at least $1032$. No algorithm, ever.\nFinding the maximum in fewer than $n-1$ comparisons, by Part 1 alone.\nUnsorted search in fewer than $n$ probes.\nA \u0026ldquo;smarter\u0026rdquo; tournament. Since the tournament meets the bound, the entire space of possible improvements is empty. This is the practical payoff of a lower bound: it tells you when to stop optimizing, which is information you cannot get from benchmarking.\nDoes not forbid It does not forbid doing better on real data, and this is where lower bounds get misapplied in engineering discussions. \u0026ldquo;Search is $\\Omega(n)$, so don\u0026rsquo;t bother optimizing lookups\u0026rdquo; ignores that the bound is worst-case over unstructured input. Index the data and you are no longer in the model: a hash index is $O(1)$, a B-tree is $O(\\log n)$, a Bloom filter answers most negatives in constant time. The adversary\u0026rsquo;s power comes entirely from the algorithm having no prior information, and every database ever built is an argument for buying some in advance.\nIt does not forbid randomized algorithms from beating the deterministic bound. A deterministic adversary knows exactly what the algorithm will do next, which is the source of its power. Against a randomized algorithm it cannot, because the next query is not determined. This is a real gap and not a technicality: randomized algorithms genuinely beat deterministic lower bounds for game-tree evaluation, and the correct tool for randomized lower bounds is Yao\u0026rsquo;s minimax principle, which fixes a hard input distribution rather than an adaptive adversary.\nIt does not mean the constant is achievable for related problems. The bound is tight here and famously is not for median selection: the best known algorithm needs about $2.95n$ comparisons, the best known lower bound is about $(2 + \\epsilon)n$, and the gap has stood for decades. Adversary arguments give you real numbers; they do not promise to close.\nIt does not apply once comparisons are not the only operation. Same exemption as T003, and the same reason. If you can read the bits of the keys, the model is gone. The adversary\u0026rsquo;s weights presuppose that the only information ever extracted is \u0026ldquo;which of these two is bigger.\u0026rdquo;\nIt does not say the algorithm is bad if it uses more comparisons. The tournament is optimal in comparisons and has worse cache behaviour than the naive two-pass scan, which is sequential and predictable. On real hardware, for moderate $n$, the \u0026ldquo;suboptimal\u0026rdquo; algorithm frequently wins. A comparison-count lower bound is a statement about a cost model, and the cost model is not your machine.\nBoundary Yao\u0026rsquo;s minimax principle. The right tool when the algorithm may randomize: the expected cost of the best randomized algorithm on its worst input equals the cost of the best deterministic algorithm on the worst input distribution. This converts a randomized lower bound into a deterministic one, at the price of having to invent a hard distribution. Information-theoretic bounds. The decision-tree argument (T003) is the other main technique, and the two have different reach. Counting leaves is easy and gives $\\log_2(\\text{outcomes})$; for max-and-second-max that yields roughly $\\log_2(n(n-1)) \\approx 2\\log_2 n$, hopelessly weak. Adversary arguments track state rather than counting outcomes, which is why they get the linear term the counting argument misses. Reach for the adversary when the answer space is small but the work is large. Potential functions. The weight scheme above is a potential-function argument: define a quantity, bound how much one operation can change it, bound its start and end values, divide. That is the same machinery as amortized analysis (T006), used to prove a lower bound instead of an upper one. Where adversaries fail. They are only as strong as the invariant you can maintain, and inventing the right potential is genuinely hard — which is exactly why the median constant is still open. There is no procedure for finding the weight function; the technique tells you what to look for, not how to find it. Adversaries in complexity theory. The same idea, scaled up: relativization results (Baker–Gill–Solovay) build an oracle adversary that answers queries to keep two worlds alive, one where $P = NP$ and one where it does not. That is this argument, at the level of entire complexity classes, and it is why diagonalization alone cannot settle the question. Read next ","permalink":"https://cs.lozic.me/posts/t005-adversary-arguments/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eYou need the largest and second-largest element of an array. The obvious way is\ntwo passes: $n-1$ comparisons for the max, then $n-2$ for the max of the rest.\nThat is $2n - 3$.\u003c/p\u003e","title":"Adversary Arguments"},{"content":"Symptom You are analyzing a hash table. You want the expected number of buckets that end up empty. The bucket occupancies are all tangled together — one key landing in bucket 3 makes every other bucket slightly less likely to be chosen — and the dependencies look like they will make the sum intractable.\nOr: you want the expected number of comparisons in quicksort, where whether element $i$ is compared to element $j$ depends intricately on which pivots were chosen before.\nOr, the interview version: you shuffle $n$ letters into $n$ addressed envelopes at random. How many land in the right envelope? The events are dependent in an obvious way — if the first $n-1$ are correct, the last one must be too.\nIn every case the honest-looking approach is a nightmare of conditional probabilities. In every case the actual answer takes one line, and the reason is a theorem so mild-sounding that people underestimate it for years.\nStatement $E[X + Y] = E[X] + E[Y]$, for any random variables $X$ and $Y$ on the same probability space.\nFor random variables $X_1, \\dots, X_n$ and constants $c_i$,\n$$E\\left[\\sum_{i=1}^n c_i X_i\\right] = \\sum_{i=1}^n c_i\\, E[X_i]$$with no assumption of independence.\nThat last line is the whole post. The variables may be dependent in any way whatsoever — perfectly correlated, adversarially entangled, defined in terms of each other — and the identity still holds exactly. Nothing else in probability is this forgiving. $E[XY] = E[X]E[Y]$ needs independence. Variances add only when uncorrelated. Expectations just add.\nArgument Work from the definition. For a finite sample space $\\Omega$ with outcomes $\\omega$:\n$$E[X + Y] = \\sum_{\\omega \\in \\Omega} P(\\omega)\\,\\bigl(X(\\omega) + Y(\\omega)\\bigr)$$Split the sum, which is legal because it is a finite sum of real numbers:\n$$= \\sum_{\\omega} P(\\omega)X(\\omega) + \\sum_{\\omega} P(\\omega)Y(\\omega) = E[X] + E[Y]$$$\\blacksquare$\nThat is the entire proof, and its triviality is the point worth dwelling on. Independence never appears because the argument never factors a joint probability — it sums over outcomes, where $X$ and $Y$ are just two numbers attached to the same $\\omega$, and addition does not care where they came from. Every theorem that does require independence requires it because it multiplies.\nThe technique: indicator variables. The proof is trivial; the use is not, and the standard move is worth stating explicitly because it converts almost any counting question into a one-liner.\nTo count things, define an indicator $X_i = 1$ if event $i$ occurs and $0$ otherwise. Then $E[X_i] = P(\\text{event } i)$, directly from the definition. Let $X = \\sum X_i$ be the total count. By linearity,\n$$E[X] = \\sum_i P(\\text{event } i)$$You have replaced a hard question about a total with $n$ easy questions about individual events. That is the trick, in full, and it is the single most useful thing in randomized algorithm analysis.\nEnvelopes. Let $X_i = 1$ if letter $i$ lands in its own envelope. Then $P(X_i = 1) = 1/n$, because letter $i$ is equally likely to land anywhere. So\n$$E[X] = \\sum_{i=1}^n \\frac{1}{n} = 1$$Exactly one letter, on average, for every $n$. The dependencies that made the problem look hard were never consulted. Computing the full distribution here requires the derangement numbers and inclusion–exclusion; computing the mean takes one line.\nEmpty buckets. Throw $n$ keys into $n$ buckets. Let $X_j = 1$ if bucket $j$ is empty. Each key misses bucket $j$ with probability $1 - 1/n$, so $P(X_j = 1) = (1-1/n)^n \\approx e^{-1}$. Hence\n$$E[\\#\\text{empty}] = n\\left(1 - \\tfrac1n\\right)^n \\approx n/e \\approx 0.368\\,n$$About 37% of buckets sit empty in a full table, which is the number behind load factor tuning everywhere.\nQuicksort. Let $X_{ij} = 1$ if elements of rank $i$ and $j$ are ever compared. They are compared exactly when the first pivot chosen from the rank range $[i, j]$ is $i$ or $j$ itself — if any middle element is picked first, $i$ and $j$ are split apart and never meet. That range has $j - i + 1$ elements, so $P(X_{ij} = 1) = 2/(j-i+1)$. Then\n$$E[\\text{comparisons}] = \\sum_{i","permalink":"https://cs.lozic.me/posts/t007-linearity-of-expectation/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eYou are analyzing a hash table. You want the expected number of buckets that end\nup empty. The bucket occupancies are all tangled together — one key landing in\nbucket 3 makes every other bucket slightly less likely to be chosen — and the\ndependencies look like they will make the sum intractable.\u003c/p\u003e","title":"Linearity of Expectation"},{"content":"Symptom Every general-purpose sort in every standard library is $O(n \\log n)$. Timsort, introsort, pdqsort, merge sort, heapsort. Decades of work by extremely motivated people, an enormous amount of money riding on it, and they all land on the same exponent.\nThat is not a coincidence, and it is not a failure of imagination. It is a proof.\nMeanwhile someone on your team points out that radix sort is $O(n)$ and asks why the standard library doesn\u0026rsquo;t just use that. The answer is interesting and it is not \u0026ldquo;radix sort is impractical.\u0026rdquo;\nStatement Any sorting algorithm that gains information about the input only by comparing pairs of elements requires $\\Omega(n \\log n)$ comparisons in the worst case.\nEvery deterministic comparison sort makes at least $\\lceil \\log_2 (n!) \\rceil = n\\log_2 n - n\\log_2 e + O(\\log n)$ comparisons on some input.\nThe qualifier comparison is the entire content of the theorem, and nearly every misunderstanding of it is a failure to notice that word. This bound constrains a model of computation, not the problem of sorting.\nArgument Model any comparison sort as a decision tree. Each internal node is a comparison \u0026ldquo;is $a_i \u0026lt; a_j$?\u0026rdquo;, each of the two edges is an outcome, and each leaf is a final permutation the algorithm outputs. Running the algorithm is walking one root-to-leaf path, so the number of comparisons in the worst case is the tree\u0026rsquo;s height.\nStep 1: the tree needs at least $n!$ leaves. There are $n!$ possible orderings of $n$ distinct elements. If two different orderings led to the same leaf, the algorithm would emit the same permutation for both — and that permutation sorts at most one of them. So the algorithm would be wrong on the other. Distinct inputs need distinct leaves. This is pigeonhole (T001) with permutations as the items and leaves as the boxes.\nStep 2: a binary tree of height $h$ has at most $2^h$ leaves. Induction, or just observe that each comparison at best halves the candidate set.\nPutting them together: $2^h \\ge n!$, so\n$$h \\ge \\log_2(n!)$$Step 3: $\\log_2 (n!) = \\Theta(n\\log n)$. This is the step that costs the post its L1, and it is done with a trick rather than Stirling\u0026rsquo;s approximation, which would be a heavier tool than needed. Keep only the largest half of the factors:\n$$n! = n(n-1)\\cdots\\left(\\frac{n}{2}\\right)\\cdots(2)(1) \\;\\ge\\; \\underbrace{\\frac{n}{2} \\cdot \\frac{n}{2} \\cdots \\frac{n}{2}}_{n/2 \\text{ terms}} = \\left(\\frac{n}{2}\\right)^{n/2}$$because each of the top $n/2$ factors is at least $n/2$, and the remaining factors are at least 1. Take logs:\n$$\\log_2(n!) \\;\\ge\\; \\frac{n}{2}\\log_2\\frac{n}{2} = \\frac{n}{2}\\log_2 n - \\frac{n}{2} = \\Omega(n\\log n)$$And the matching upper bound is immediate, since $n! \\le n^n$ gives $\\log_2(n!) \\le n\\log_2 n$. So the bound is $\\Theta(n \\log n)$, and merge sort achieves it. $\\blacksquare$\nWhat the bound really counts. $\\log_2(n!)$ is the number of bits needed to name one permutation out of $n!$. Each comparison returns one bit. So the theorem is a statement about information: sorting requires learning $\\log_2(n!)$ bits, and a yes/no question yields at most one bit, so you need at least that many questions. Framed this way it is the same argument as the source coding bound (T018) with comparisons in place of code symbols, and it explains why the bound is so robust — you are not out-arguing an algorithm, you are out-arguing arithmetic.\nFor $n = 100$: $\\log_2(100!) \\approx 525$ comparisons, minimum, ever. Merge sort uses about 573. There is very little room left, and this is why sorting research moved to constant factors, cache behaviour, and adaptivity decades ago.\nWhere the constant actually sits. Applying Stirling\u0026rsquo;s approximation to the same quantity sharpens the bound to $n\\log_2 n - 1.44n$, and the $-1.44n$ term is what the good implementations are fighting over. Merge sort spends about $n\\log_2 n - n$, quicksort about $1.39n\\log_2 n$ expected (T007 derives that constant), and the information floor sits below both. That is the entire remaining budget: a constant factor on the leading term and a linear correction. When a new sort is announced, this is the number to check, because the exponent has been settled since 1959.\nForbids A general-purpose comparison sort in $O(n)$. No arrangement of comparisons gets there, regardless of cleverness, data structures, or engineering effort.\nAn $O(n \\log \\log n)$ comparison sort. The bound leaves no room between $n\\log n$ and linear either.\nSorting by \u0026ldquo;just being smarter about pivots.\u0026rdquo; Quicksort\u0026rsquo;s worst case is fixable to $O(n\\log n)$ by median-of-medians or introsort\u0026rsquo;s heapsort fallback, but no pivot strategy breaks the floor.\nA comparison-based priority queue with both $O(1)$ insert and $O(1)$ extract-min. You could sort $n$ items in $O(n)$ with one, which the bound forbids. This is a useful reduction to keep in your pocket: it kills a whole category of data structure proposals without inspecting them. It also explains why Fibonacci heaps give you $O(1)$ amortized insert and decrease-key but pay $O(\\log n)$ on extract-min. The costs can be moved around; the total cannot go below the sorting bound, because the sequence of extract-mins is a sort.\nAn $O(n)$ algorithm for building a binary search tree from unsorted input. An in-order traversal of the finished tree is a sorted list, produced in $O(n)$, so a linear construction would sort in linear time. Same for building a sorted array, a skip list, or any structure a linear scan can read in order. This \u0026ldquo;output is a sort\u0026rdquo; reduction is the fastest way to sanity-check a claimed data-structure bound, and it disposes of most of them.\nDoes not forbid It does not forbid sorting in $O(n)$, and treating \u0026ldquo;sorting is $n \\log n$\u0026rdquo; as a law of nature is the misreading that matters. Radix sort sorts $n$ integers of $w$ bits in $O(wn/\\log n)$ time, which is linear for fixed-width keys, and it is not a loophole or a cheat — it is outside the model, because it never compares two elements. It looks at digits. Counting sort is $O(n + k)$ for keys in a small range. Both are real, both ship in production (LSD radix sort is what fast integer-sorting libraries actually use), and neither contradicts a word of the theorem. The right sentence is \u0026ldquo;comparison sorting is $\\Omega(n \\log n)$,\u0026rdquo; and dropping the first word turns a precise result into folklore that costs people real performance. If your keys are 32-bit integers, you are leaving a large constant on the table by reaching for std::sort out of habit.\nIt does not forbid beating $n \\log n$ on nearly-sorted input. The bound is a worst case over all inputs. Timsort runs in $O(n)$ on already-sorted data and detects existing runs, which is why it is Python\u0026rsquo;s and Java\u0026rsquo;s default. Adaptive sorts are measured against the number of inversions, not against $n!$, and there is no conflict: the theorem promises that some input costs $n\\log n$, not that yours does.\nIt does not apply to randomized or average-case claims — but it survives both. Worth stating precisely because people expect randomization to help here and it does not. The average number of comparisons over uniformly random inputs is also $\\Omega(n\\log n)$, and the expected count for any randomized algorithm is too, since a randomized algorithm is a distribution over decision trees and the leaf-counting argument applies to each. Quicksort\u0026rsquo;s $O(n\\log n)$ expected time is not evading the bound, it is meeting it.\nIt does not say $n\\log n$ is achievable in $n\\log n$ time. The bound counts comparisons, and a comparison is not always $O(1)$. Sorting long strings, or records with expensive comparators, has different arithmetic — which is why suffix-array construction and string sorting are their own fields.\nIt does not mean the bound is exactly achievable. $\\lceil \\log_2(n!)\\rceil$ is the information floor, and for some $n$ no algorithm attains it: the minimum number of comparisons to sort 13 elements is 34, while $\\lceil \\log_2(13!)\\rceil = 33$. Lower bounds are constraints, not predictions, and the gap between \u0026ldquo;no one can do better than this\u0026rdquo; and \u0026ldquo;someone can do this\u0026rdquo; is real even when it is small.\nBoundary Everything that beats the bound does so by leaving the model, and it is worth seeing that they are all the same move:\nLook at the keys, not at pairs. Radix and counting sort read the representation. Cost: keys must have a usable digit structure and bounded width. This is by far the most-used escape and the most-overlooked. Exploit existing order. Timsort, smoothsort, and the adaptive family run faster when the input is partially sorted, measured in inversions rather than $n$. Know the distribution. Bucket sort is $O(n)$ expected for uniformly distributed keys. Learned index structures push this further by fitting a model of the key distribution. Change the machine. Parallel sorts change the depth but not the total work; sorting networks like AKS achieve $O(\\log n)$ depth with $O(n\\log n)$ comparators, so the work bound holds. On a word-RAM, integer sorting is possible in $O(n\\log\\log n)$ (Han and Thorup get $O(n\\sqrt{\\log\\log n})$ expected), which again is not comparison-based. Sort less. Selection and partial sorting have genuinely lower bounds: finding the median is $\\Theta(n)$, and getting the top $k$ costs $O(n + k\\log k)$, not $O(n\\log n)$. If you do not need a total order, do not buy one — this is the boundary with the best practical return, and nth_element exists for exactly this reason. Read next ","permalink":"https://cs.lozic.me/posts/t003-comparison-sort-lower-bound/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eEvery general-purpose sort in every standard library is $O(n \\log n)$.\nTimsort, introsort, pdqsort, merge sort, heapsort. Decades of work by extremely\nmotivated people, an enormous amount of money riding on it, and they all land on\nthe same exponent.\u003c/p\u003e","title":"The Omega(n log n) Comparison-Sort Bound"},{"content":"Symptom You gzip a log file and it drops to 8% of its original size. You gzip the already-gzipped file and it gets slightly bigger. Somebody asks why, and the honest answer you have is \u0026ldquo;because it\u0026rsquo;s already compressed,\u0026rdquo; which is a restatement, not a reason.\nOr: you are choosing a compression level, and you want to know when to stop paying for CPU. Level 9 buys you 2% over level 6. Would level 22 buy another 2%? Is there a number that says where the floor is?\nOr the one from a design review: someone proposes storing a million UUIDs and compressing them. Somebody else says it will not help. Both are guessing.\nT004 told you no compressor shrinks everything. It did not tell you how far a compressor can go on the data you actually have. There is an exact number, it is computable from the data, and it was defined in 1948.\nStatement For a source emitting symbols from an alphabet with probabilities $p_i$, the entropy is\n$$H = -\\sum_i p_i \\log_2 p_i \\quad\\text{bits per symbol}$$ Source coding theorem. For a source of entropy $H$, and any $\\varepsilon \u0026gt; 0$, there is a lossless code with average length $\u0026lt; H + \\varepsilon$ bits per symbol. And no lossless code achieves average length $\u0026lt; H$.\nBoth halves matter and they are usually quoted separately. The second half is the impossibility: $H$ is a floor nothing gets under. The first is the achievability: the floor is reachable, not merely a bound someone proved. It is rare for a lower bound to be tight, and this one is.\nRead $-\\log_2 p_i$ as the \u0026ldquo;surprise\u0026rdquo; of symbol $i$: a symbol with probability $1/8$ carries 3 bits. Entropy is then just the average surprise, and the theorem says average surprise is exactly the cost of writing the data down.\nArgument The floor. Suppose a prefix-free code assigns length $\\ell_i$ to symbol $i$. Kraft\u0026rsquo;s inequality says any prefix-free code satisfies $\\sum_i 2^{-\\ell_i} \\le 1$ — which is a counting fact, not an assumption: reserving a codeword of length $\\ell$ removes a $2^{-\\ell}$ fraction of the space of possible continuations, and you cannot spend more than all of it.\nNow compare the average length $L = \\sum_i p_i \\ell_i$ against $H$:\n$$L - H = \\sum_i p_i \\ell_i + \\sum_i p_i \\log_2 p_i = \\sum_i p_i \\log_2 \\frac{p_i}{2^{-\\ell_i}}$$That last quantity is a relative entropy (a Kullback–Leibler divergence) between the distribution $p$ and the sub-distribution $q_i = 2^{-\\ell_i}$, and it is never negative. The one-line reason is Jensen\u0026rsquo;s inequality applied to the concave $\\log$:\n$$-\\sum_i p_i \\log_2 \\frac{q_i}{p_i} \\;\\ge\\; -\\log_2 \\sum_i p_i \\frac{q_i}{p_i} = -\\log_2 \\sum_i q_i \\;\\ge\\; -\\log_2 1 = 0$$So $L \\ge H$, with equality exactly when $\\ell_i = -\\log_2 p_i$ for every $i$. $\\blacksquare$\nThe ceiling. Set $\\ell_i = \\lceil -\\log_2 p_i \\rceil$. These satisfy Kraft, so such a code exists (Shannon–Fano), and since each length overshoots by less than 1,\n$$L \u003c \\sum_i p_i (-\\log_2 p_i + 1) = H + 1$$That gives $H + 1$, not $H + \\varepsilon$. To close the gap, encode blocks of $n$ symbols at a time. The overhead of at most 1 bit is now spread over $n$ symbols, so the per-symbol cost is below $H + 1/n$, and $n$ can be as large as you like. That blocking step is the whole content of \u0026ldquo;$\\varepsilon$\u0026rdquo;, and it is also why real compressors work on streams rather than symbol by symbol.\nA worked example, since the formula is opaque until you use it. English text, treated as 26 letters plus space with their actual frequencies, has $H \\approx 4.1$ bits per character — against the $\\log_2 27 \\approx 4.75$ bits a uniform code would need. But letters are not independent: q is followed by u, th is common. Taking that structure into account, Shannon estimated English at about 1.0 to 1.5 bits per character. A naive ASCII file spends 8. That factor of six is exactly the room gzip is exploiting, and it is why text compresses well and why nothing compresses text below about 1 bit per character.\nTwo sanity checks on the formula:\nA fair coin. $p = (1/2, 1/2)$, so $H = -2 \\times \\tfrac12 \\log_2 \\tfrac12 = 1$ bit. One bit per flip, which is right, and unimprovable. A biased coin at $p = 0.99$. $H = -0.99\\log_2 0.99 - 0.01\\log_2 0.01 \\approx 0.08$ bits. A sequence of 1000 such flips needs about 80 bits, not 1000. This is the compression the intuition misses. Forbids Compressing uniform random data. $n$ uniform random bits have $H = n$. No code stores them in fewer than $n$ bits on average. This is why the compressed file will not compress again: a good compressor\u0026rsquo;s output is nearly uniform by construction, so it has already spent its slack.\nCompressing a million UUIDs. Each carries 122 random bits, so a million of them hold $1.22 \\times 10^8$ bits $\\approx 15$ MB, and no encoding does better. (The textual form with its dashes and hex is 36 bytes, so there is a 2.4$\\times$ win available from parsing them into bytes — but that is removing encoding overhead, not compressing entropy.)\nBeating $H$ by being clever about the algorithm. The bound does not care what your algorithm is. Arithmetic coding, ANS, Huffman, a neural net: they differ in how close to $H$ they get and how fast, never in whether they can go under.\nA meaningful \u0026ldquo;compression ratio\u0026rdquo; claim without naming the source. Ratios are properties of a source-and-compressor pair. \u0026ldquo;Our algorithm achieves 10:1\u0026rdquo; is only a claim about the test data\u0026rsquo;s entropy until proven otherwise.\nDoes not forbid It does not say your file cannot be compressed below its current entropy estimate, and this misreading gets used to shut down real work. Entropy is defined relative to a model of the source. Measure a file\u0026rsquo;s byte histogram, get 7.9 bits per byte, and conclude it is incompressible — and then a context-mixing compressor finds structure the histogram could not see and halves it. This happens routinely; it is most of what the Hutter Prize is about. The order-0 entropy is a floor for order-0 models only. Every improvement in compression since 1948 has come from better models, not from beating Shannon, and the theorem specifically leaves that door open.\nIt does not forbid lossy compression from doing far better. JPEG, MP3, and H.264 achieve ratios that would be impossible losslessly, because they are answering a different question — how few bits to reproduce something a human cannot distinguish. That is rate–distortion theory, also Shannon\u0026rsquo;s, with its own and much lower floor. When someone says \u0026ldquo;you can\u0026rsquo;t compress video that much,\u0026rdquo; they are quoting the wrong theorem.\nIt does not mean an individual string has an entropy. This is a genuine technical confusion worth naming. $H$ is a property of a distribution, not of a string. The string 0000000000 has no entropy; the source that emitted it does. The per-string analogue is Kolmogorov complexity — the length of the shortest program producing it — which is uncomputable, and which is the subject of its own post. Asking \u0026ldquo;what is the entropy of this file\u0026rdquo; is a type error that everyone commits, including this post\u0026rsquo;s opening section, and it is usually harmless as long as you know it is shorthand for \u0026ldquo;of the model I fitted to it.\u0026rdquo;\nIt does not mean compressed data is random. Compressed output is statistically close to uniform, which is exactly why it resists further compression, and it is completely predictable to anyone holding the decompressor. Encryption is not compression, and using gzip output as a source of entropy for a key is a real mistake people have made.\nIt does not forbid a specific file from shrinking a lot. The floor is on average length over the source distribution. A particular low-probability-looking file may compress magnificently. Averages bind the long run, not any single case, which is the same distinction T001 draws between existence and likelihood.\nBoundary Better models. The most productive direction, and where essentially all practical progress lives. Order-0 entropy assumes independent symbols; conditioning on context lowers $H$, sometimes dramatically. LZ77 exploits repetition, PPM and context mixing predict from history, and modern neural compressors are simply very good models paired with an arithmetic coder. Each is a tighter estimate of the true source entropy, not a violation of it. The gap between $H$ and $H+1$. Huffman coding is optimal among symbol-wise codes but can waste up to 1 bit per symbol, which is catastrophic when $H$ is small — a symbol of probability 0.99 still costs a whole bit. Arithmetic coding and ANS remove this by not requiring integer code lengths, which is why they replaced Huffman where the alphabet is skewed. Universal codes. Lempel–Ziv achieves the entropy rate without knowing the source distribution in advance, asymptotically. That is a remarkable result and the reason gzip needs no training pass over your data. Kolmogorov complexity. Drop the probabilistic source and ask for the shortest program that outputs a specific string. This is the strongest form of the question, it agrees with entropy on average, and it is uncomputable — which is a considerably worse boundary than the one Shannon drew, and gets its own post. Channels, not just sources. The other half of Shannon\u0026rsquo;s 1948 paper turns the question around: given a noisy channel, how much can you reliably send? The channel coding theorem says up to the channel capacity, with arbitrarily low error, which is at least as surprising as this result and is what makes error-correcting codes possible. Read next ","permalink":"https://cs.lozic.me/posts/t018-shannon-entropy-and-source-coding/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eYou gzip a log file and it drops to 8% of its original size. You gzip the\nalready-gzipped file and it gets slightly \u003cem\u003ebigger\u003c/em\u003e. Somebody asks why, and the\nhonest answer you have is \u0026ldquo;because it\u0026rsquo;s already compressed,\u0026rdquo; which is a\nrestatement, not a reason.\u003c/p\u003e","title":"Shannon Entropy and the Source Coding Theorem"},{"content":"Symptom You need a short ID for uploads. Eight hex characters feels generous, so you take the first 32 bits of a hash and move on.\nAt about 80,000 uploads, two files collide, and one of them silently overwrites the other, and the bug report says the customer\u0026rsquo;s invoice contains someone else\u0026rsquo;s line items.\nOr: you generate a nonce per session with 64 bits of randomness. Nonces must never repeat. You reason that $2^{64}$ is astronomically large, and it is, and you will still see a repeat after about five billion sessions, which a busy service reaches.\nOr the git version, which is the famous one: SHA-1 is 160 bits, and yet everyone agreed to migrate, and the reason was not that anyone had enumerated $2^{160}$ hashes.\nPigeonhole (T001) told you collisions exist. It said nothing about when. The answer is much sooner than anyone\u0026rsquo;s intuition, and the shape of the answer is a square root.\nStatement With $N$ equally likely values and $m$ items drawn uniformly at random, a collision becomes more likely than not once $m \\approx 1.177\\sqrt{N}$.\nThe name comes from the classroom version: in a room of 23 people, two share a birthday with probability above one half. $N = 365$, and $1.177\\sqrt{365} \\approx 22.5$.\nThe engineering form is the one to memorize:\nFor $m \\ll N$, the probability of at least one collision is approximately\n$$p(m, N) \\;\\approx\\; 1 - e^{-m^2 / 2N} \\;\\approx\\; \\frac{m^2}{2N}$$the last step holding when $m^2 \\ll N$. Collisions become likely around $m \\approx \\sqrt{N}$, not $m \\approx N$.\nThe headline: an $n$-bit hash gives you $n/2$ bits of collision resistance. A 128-bit hash collides after about $2^{64}$ items. Half your bits are gone before you start, and this is why every collision-resistance number you have ever read is half the digest length.\nArgument Compute the probability that all $m$ items are distinct, which is easier, and subtract.\nDraw items one at a time. The first lands anywhere. The second must avoid 1 of $N$ values, so it succeeds with probability $(N-1)/N$. The third must avoid 2, and so on. The $k$-th item must avoid $k-1$ occupied values:\n$$P(\\text{all distinct}) = \\prod_{k=1}^{m-1}\\left(1 - \\frac{k}{N}\\right)$$Now use $1 - x \\le e^{-x}$, which is the standard move and is tight for small $x$:\n$$P(\\text{all distinct}) \\;\\le\\; \\prod_{k=1}^{m-1} e^{-k/N} = \\exp\\left(-\\frac{1}{N}\\sum_{k=1}^{m-1} k\\right) = \\exp\\left(-\\frac{m(m-1)}{2N}\\right)$$So\n$$P(\\text{collision}) \\;\\ge\\; 1 - e^{-m(m-1)/2N} \\;\\approx\\; 1 - e^{-m^2/2N}$$Set that to $1/2$ and solve: $m^2/2N = \\ln 2$, giving $m = \\sqrt{2\\ln 2},\\sqrt{N} \\approx 1.177\\sqrt{N}$. $\\blacksquare$\nWhy the square root, in one sentence. Collisions are about pairs, not items. With $m$ items there are $\\binom{m}{2} \\approx m^2/2$ pairs, each colliding with probability $1/N$, so the expected number of collisions is $m^2/2N$ — and this is just linearity of expectation (T007), which needs no independence between the pairs even though the pairs plainly overlap. Set the expected count to 1 and you get $m \\approx \\sqrt{2N}$ directly. Intuition fails here because people count items and the mathematics counts pairs, and pairs grow quadratically.\nA worked number, because the abstraction hides the shock. Take a 32-bit hash, $N = 2^{32} \\approx 4.3 \\times 10^9$. Naive intuition says you are safe for billions of items. The bound says $1.177\\sqrt{2^{32}} \\approx 77{,}000$. Seventy-seven thousand. That is a medium-sized S3 bucket, and it is the failure in the opening section. At the $2^{16} = 65{,}536$ mark you are already at roughly a 40% chance.\nDigest $N$ 50% collision at 32-bit $2^{32}$ $\\approx 2^{16}$ = 77 thousand 64-bit $2^{64}$ $\\approx 2^{32}$ = 5 billion 128-bit (MD5, UUIDv4) $2^{128}$ $\\approx 2^{64}$ 160-bit (SHA-1) $2^{160}$ $\\approx 2^{80}$ 256-bit (SHA-256) $2^{256}$ $\\approx 2^{128}$ Forbids Short hashes for content addressing. Any scheme where a collision means data loss needs $2 \\times$ the bits you would naively pick. Git\u0026rsquo;s 160-bit SHA-1 gives 80 bits of collision resistance, which was comfortable in 2005 and is why the migration to SHA-256 happened.\nReusing a 64-bit nonce across a long-lived key. AES-GCM\u0026rsquo;s 96-bit nonce allows about $2^{48}$ messages before random selection becomes unsafe, which is exactly why the specification tells you to use a counter rather than random values when you can.\n\u0026ldquo;We\u0026rsquo;ll truncate the hash to save space, it\u0026rsquo;s still random.\u0026rdquo; Truncating to $n$ bits caps collision resistance at $2^{n/2}$ regardless of how strong the underlying function is. The truncation, not the algorithm, sets the bound.\nTrusting a 128-bit digest against a motivated adversary. $2^{64}$ operations is not out of reach for a well-funded attacker, which is why MD5 and SHA-1 are dead for signatures — and note that SHA-1 actually fell to a cryptanalytic attack cheaper than $2^{80}$, so the birthday bound is the ceiling on security, never a floor.\nAssuming a \u0026ldquo;random enough\u0026rdquo; 53-bit float ID is safe. JavaScript\u0026rsquo;s Number.MAX_SAFE_INTEGER is $2^{53}$, so IDs generated as random doubles collide after about $2^{26.5} \\approx 95$ million — reachable by a single large table. This is a live failure mode wherever an ID crosses a JSON boundary, because the 64-bit value the backend generated is not the value the frontend received.\nDoes not forbid It does not mean UUIDv4 is unsafe, and this is where the theorem gets misused to justify real engineering waste. \u0026ldquo;The birthday bound says collisions happen at $\\sqrt{N}$, so 122 random bits is only 61 bits of safety, so we need a central ID service\u0026rdquo; is an argument that has cost teams a database dependency they did not need. Run the number: 61 bits is $2^{61} \\approx 2.3 \\times 10^{18}$ UUIDs before a 50% chance. Generate a billion a second and you wait about 73 years. The bound is real, the halving is real, and the result is still comfortably beyond any application. The correct response to a factor of two in the exponent is to check the arithmetic, not to add a coordination point.\nIt does not apply to a counter. This is the most useful exemption in the post and the most often missed. The bound is about random selection. A monotonic counter, a database sequence, a Snowflake ID with a machine number and a timestamp, or an AES-GCM nonce driven by a counter never collides at all, by construction. If you can afford the coordination — and within one process you always can — you get $N$ values from $n$ bits instead of $\\sqrt{N}$, which is a free doubling of your effective width. Reach for randomness only when you cannot coordinate.\nIt does not mean a collision is a security break. Collision resistance and preimage resistance are different properties with different bounds. Finding some pair that collides takes $2^{n/2}$; finding a message that hashes to a specific given value takes $2^n$, with no birthday speedup, because you are no longer free to choose both sides. So MD5 is thoroughly broken for certificates (where an attacker crafts both documents) and yet an MD5 preimage attack still does not exist. When someone says a hash is broken, ask which property.\nIt does not apply when the values are not uniform. The bound assumes uniform selection. Real hash functions on real data are close enough to uniform for the estimate to hold, but a bad hash — one with structure, like using the low bits of an auto-increment ID, or a hash the adversary can steer — collides very much sooner. Uniformity is the assumption that makes the square root a best case, not a worst case. Algorithmic complexity attacks work exactly by breaking it.\nIt does not say a collision has occurred, only when to expect one. This inverts pigeonhole\u0026rsquo;s failure mode and is worth keeping straight: T001 says a collision exists somewhere in the input space and is silent about probability; the birthday bound says nothing about existence and gives you the probability. Neither one tells you your specific dataset has collided. That takes checking.\nBoundary Counters and coordination. Covered above and worth repeating as the first boundary because it eliminates the problem rather than pricing it. No randomness, no birthday bound. Widen the space. Every bit you add doubles $N$ and multiplies the collision threshold by $\\sqrt{2}$. Bits are cheap; the standard advice to use 256 bits when you want 128 bits of security is this arithmetic and nothing more. Detect instead of avoid. Content-addressed stores can verify on write: if the digest is present, compare the bytes. Git does this. The collision becomes a detectable event rather than a silent corruption, which changes the failure from data loss to an error message. The other direction — deliberately colliding. Bloom filters and HyperLogLog want collisions and budget for them; the birthday arithmetic is what sizes the filter. Rainbow tables and Pollard\u0026rsquo;s rho use the square root as an attack, finding collisions in $O(\\sqrt{N})$ time and constant memory via cycle detection, which is why the memory-free version of the bound is the one cryptographers quote. Where the square root is too pessimistic. Bernstein\u0026rsquo;s point in the reading below: a $2^{64}$ attack requires not just $2^{64}$ operations but the memory and communication to correlate them, and a parallel attacker\u0026rsquo;s real cost curve is worse than the operation count suggests. The bound is exact about probability and only an approximation of cost. Read next ","permalink":"https://cs.lozic.me/posts/t074-the-birthday-bound/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eYou need a short ID for uploads. Eight hex characters feels generous, so you\ntake the first 32 bits of a hash and move on.\u003c/p\u003e\n\u003cp\u003eAt about 80,000 uploads, two files collide, and one of them silently overwrites\nthe other, and the bug report says the customer\u0026rsquo;s invoice contains someone\nelse\u0026rsquo;s line items.\u003c/p\u003e","title":"The Birthday Bound"},{"content":"Symptom You call the payment API. The request goes out. Nothing comes back.\nDid the charge happen? You genuinely do not know. Retrying might double-charge the customer. Not retrying might drop the payment. There is no third option available, and no amount of care in your client library creates one.\nOr the version in a design doc: the queue promises exactly-once delivery, and you are trying to work out whether to believe it.\nOr the version in an incident: the two services disagree about whether an order was placed, and the reconciliation job has been running for three hours.\nThese are all one problem, and it is provably unsolvable, and knowing that changes what you build instead.\nStatement Two generals must attack a city simultaneously. Attacking alone means defeat. They communicate only by messengers who cross enemy territory and may be captured. Messages that arrive are correct; messages may simply never arrive.\nNo protocol, using finitely many messages, lets both generals reach certainty that the other will attack.\nOver an asynchronous channel with unbounded message loss, common knowledge of a fact is unattainable in a finite number of messages, and therefore no deterministic protocol achieves guaranteed coordinated action between two parties.\nNote what is not assumed. Nobody is lying. Nobody crashes. There is no Byzantine behaviour, no clock skew, no corruption. The single adversity is that a message may be lost, and that alone is fatal. This is what makes the result so much stronger than its cartoon setting suggests: it is the weakest possible failure model, and it already suffices.\nArgument Suppose a protocol exists that solves the problem. Among all its executions, consider one that succeeds — both generals attack — using the fewest messages. Call that number $n$, and let this shortest successful execution be $E$.\nNow consider the last message in $E$, message $n$.\nThat message might have been lost. The channel offers no guarantee, so there is an execution identical to $E$ in every respect except that message $n$ never arrives.\nConsider the sender of message $n$. From their point of view, these two executions are indistinguishable: they sent the message, and in neither case did they receive anything afterward, since message $n$ was the last one. They cannot condition their behaviour on the difference. So they attack in both, or neither.\nSince $E$ succeeds, they attack in both. And the receiver, in the execution where the message was lost, has exactly the information they had before message $n$ was sent — so they behave as they would have without it.\nSo in the lossy execution, both generals still act correctly. But that execution used only $n-1$ delivered messages, and had a successful outcome. So a protocol succeeding with $n-1$ messages exists, contradicting the minimality of $n$.\nTherefore no minimal $n$ exists, and therefore no finite protocol exists. $\\blacksquare$\nThe argument is a descent, and the engine is the indistinguishability step: the sender of the final message cannot know it arrived, so the final message can never be load-bearing. Which means no message can be, since removing it just promotes its predecessor to the same position.\nThe knowledge framing, which is the one that sticks. Write $K_A(p)$ for \u0026ldquo;A knows $p$\u0026rdquo;. What coordinated action requires is common knowledge: A knows, B knows, A knows B knows, B knows A knows A knows, forever. Each message buys exactly one more level of that infinite tower:\n$$K_A(p),\\; K_B K_A(p),\\; K_A K_B K_A(p),\\; \\dots$$An acknowledgment raises the tower by one. An acknowledgment of the acknowledgment raises it by one more. Common knowledge is the limit, and no finite number of messages reaches a limit. Halpern and Moses proved this precisely in 1990: common knowledge is unattainable in a system with unreliable communication, and coordinated simultaneous action requires common knowledge. That is the general theorem; the two generals is its smallest instance.\nForbids Exactly-once message delivery. A receiver cannot distinguish \u0026ldquo;my acknowledgment was lost\u0026rdquo; from \u0026ldquo;the sender never sent,\u0026rdquo; so the sender must choose between resending (at-least-once, risking duplicates) and not resending (at-most-once, risking loss). There is no third protocol. Every queue that advertises exactly-once is doing something else and calling it that; usually at-least-once delivery plus deduplication, which works and is not the same claim.\nA distributed transaction with no uncertainty window. Two-phase commit does not solve this. It relocates the uncertainty into the participants after they vote yes, where they must block until the coordinator returns. That blocking window is not a bug in 2PC, it is this theorem showing through (T068, T064).\nGuaranteed simultaneous action between two parties over a lossy channel. Including: cutting over two services at the same instant, atomically flipping a feature flag in two regions, and confirming both a charge and a shipment together.\nA TCP connection close that both sides agree on. The TCP four-way handshake ends with a TIME_WAIT of twice the maximum segment lifetime, and that timer is not an optimization. It is the protocol admitting it cannot get certainty and substituting a probabilistic wait. The final ACK is unacknowledged, necessarily, because acknowledging it would need an acknowledgment.\nDoes not forbid It does not forbid exactly-once semantics, and conflating this with exactly-once delivery is the misreading that generates the most pointless argument on the internet. Kafka\u0026rsquo;s exactly-once processing is real, works, and does not violate this theorem. It delivers at-least-once and deduplicates using idempotent producers with sequence numbers plus transactional offsets, so the observable effect happens once even though the message may arrive several times. Stripe\u0026rsquo;s idempotency keys do the same. The theorem forbids certainty about delivery; it says nothing about making duplicate delivery harmless, and that is where the entire practical solution lives.\nIt does not forbid consensus in real systems, which is the other big one. \u0026ldquo;Two generals proves distributed consensus is impossible, so Paxos must be snake oil\u0026rdquo; is a thing said with confidence by people who have read one blog post. Paxos and Raft change the problem in two specific ways: they require only a majority, not all participants, and they drop the requirement of simultaneous action in favour of eventual agreement on an ordered log. Both escapes are legitimate and neither contradicts anything here. What they still cannot do is guarantee progress during an arbitrary partition, which is FLP (T062) and CAP (T066), a different and weaker limitation.\nIt does not mean retries are useless. They cannot deliver certainty, but each retry raises the probability of delivery. If a message arrives with probability $0.99$, five independent attempts fail with probability $10^{-10}$. That is not certainty and the theorem is untouched, but it is a better failure rate than most hardware you are running on, and \u0026ldquo;provably impossible\u0026rdquo; is not a reason to skip the retry loop.\nIt does not apply to a reliable channel. If message delivery is guaranteed — a single machine\u0026rsquo;s memory, a synchronous bus, a channel with a known bound on loss — two generals is not a constraint at all. This matters more than it sounds: a great deal of accidental distributed-systems complexity comes from treating an in-process call as if it had the network\u0026rsquo;s failure model, or worse, the reverse.\nIt does not apply once you drop simultaneity. Almost every practical fix is this move. Two-phase commit with a timeout, sagas with compensating transactions, eventual consistency, and CRDTs all abandon \u0026ldquo;both act at the same instant\u0026rdquo; in exchange for \u0026ldquo;both converge eventually.\u0026rdquo; The theorem is specifically about coordinated simultaneous action, and simultaneity turns out to be the requirement almost nobody actually needs once they say out loud why they wanted it.\nBoundary The impossibility is exact, so every workaround is a change of question. There are four, and essentially all production systems use one:\nIdempotency. Make repeated delivery harmless, then use at-least-once and retry freely. You have not solved coordination; you have made the failure to coordinate not matter. This is the single most valuable move in applied distributed systems, and if you take one thing from this post, it is that the right response to \u0026ldquo;did the payment go through?\u0026rdquo; is to make asking twice safe rather than to try harder to know. Probabilistic certainty. Accept a residual failure probability, retry until it is below your error budget. TCP\u0026rsquo;s TIME_WAIT, exponential backoff, and every timeout you have ever tuned are this. Note that a timeout is not a measurement, it is a guess with a confidence level. Drop simultaneity. Sagas, eventual consistency, CRDTs. Converge instead of agreeing at an instant. Drop unanimity. Require a majority rather than everyone, and get consensus in the asynchronous-with-crashes model — Paxos and Raft (T064). This is the deepest of the four and the one that took thirty years to make practical. What remains genuinely impossible after all four: knowing, right now, with certainty, whether the other side received your message. No system on earth has this, including the one you are working on, and designs that quietly assume it fail in ways that take a long time to diagnose because the assumption is never written down.\nRead next ","permalink":"https://cs.lozic.me/posts/t061-the-two-generals-problem/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eYou call the payment API. The request goes out. Nothing comes back.\u003c/p\u003e\n\u003cp\u003eDid the charge happen? You genuinely do not know. Retrying might double-charge\nthe customer. Not retrying might drop the payment. There is no third option\navailable, and no amount of care in your client library creates one.\u003c/p\u003e","title":"The Two Generals Problem"},{"content":"Symptom The service has four layers. Nobody designed four layers. There were four teams.\nOr the version that arrives in a postmortem: the outage happened at the boundary between two services, in the retry logic, because each side assumed the other owned the deduplication. Both teams were right about their own component and neither had ever been in a room with the other.\nOr the one that costs the most and gets noticed the least: the API is ugly in exactly one place, and the ugliness maps precisely to the quarter when the frontend group reported to a different VP.\nYou have been reading these as management failures. They are, but they are also the most reliably predictive regularity in software engineering, and it was written down in 1968.\nStatement Organizations which design systems are constrained to produce designs which are copies of the communication structures of these organizations.\nConway\u0026rsquo;s own phrasing, and the word constrained is doing more work than the popular paraphrase (\u0026ldquo;systems mirror org charts\u0026rdquo;) preserves.\nThere is a homomorphism from the organization\u0026rsquo;s communication graph to the system\u0026rsquo;s dependency graph. If two components must interface, their authors must communicate; where authors do not communicate, no clean interface appears.\nNote the direction. The claim is not that the org chart determines the architecture — it is that the communication structure does, and the org chart is merely its most legible approximation. Two teams on different continents with a shared on-call rotation and a daily sync communicate more than two teams on the same floor under different VPs, and the architecture will show it.\nThis is [empirical], and the tag is the honest one. There is no proof.\nArgument There is no proof, but there is a mechanism, and the mechanism is why this regularity is stable rather than anecdotal.\nStep one: a design is a communication artifact. To specify an interface between components $A$ and $B$, someone must decide what crosses it — types, errors, ordering, ownership of retries, who validates what. That decision is information that must exist in both authors\u0026rsquo; heads. There is no other way for the interface to be coherent.\nStep two: communication is expensive, and superlinearly so. Coordinating $n$ people has on the order of $\\binom{n}{2} = n(n-1)/2$ potential channels. Organizations respond by not paying: they draw boundaries, and inside a boundary communication is cheap and constant, and across one it is scheduled, lossy, and rationed.\nStep three: interfaces form where communication is cheap and calcify where it is not. Within a team, an interface can be renegotiated in an afternoon, so components get refactored until the boundary sits where it belongs. Across a boundary, changing an interface costs a meeting, a migration plan, and possibly a quarter — so the first interface anyone wrote is the one you keep, whether or not it was right.\nStep four: so the architecture converges on the communication graph. Not because anyone chose it, but because every other architecture requires coordination the organization is not paying for. The pressure is continuous and nobody has to agree with it for it to win.\nThe mechanism also predicts the exceptions, which is the test of a mechanism rather than a slogan. Where an organization does pay the coordination cost — a strong architecture review, one author across two teams, a mandated shared interface — Conway\u0026rsquo;s law is locally suspended. It just gets expensive, and it reverts the moment the expenditure stops, which is why architectures decay after a reorg.\nThe evidence. MacCormack, Rusnak, and Baldwin (2012) compared matched pairs of systems built by tightly-coupled commercial organizations against loosely-coupled open-source ones, and found the open-source products significantly more modular — the same product category, different communication structures, different architectures. Microsoft\u0026rsquo;s 2008 study of Windows Vista found organizational metrics (how many engineers touched a binary, how far apart in the reporting hierarchy) predicted post-release defects better than code metrics like churn, complexity, or coverage. That is the strongest form of the claim available without a proof: the org chart out-predicts the code.\nForbids Since this is empirical, \u0026ldquo;forbids\u0026rdquo; means \u0026ldquo;reliably fails in practice,\u0026rdquo; not \u0026ldquo;is impossible.\u0026rdquo; That distinction is the honest one and it matters here.\nAn architecture that ignores the org chart, surviving contact with delivery. You can draw it. You cannot staff it. The design will drift toward the communication structure over the following year, component by component, and each individual drift will look like a reasonable local decision.\nMicroservices from a monolithic organization. If every change still requires sign-off from one central team, splitting the deployable units yields a distributed monolith: all the coordination costs of the monolith, plus network partitions. The communication structure did not change, so the architecture did not change. Only the topology of the failure modes did, and for the worse.\nClean interfaces between teams that do not talk. The interface will exist, because something must compile. It will be a leaky pass-through of whichever side wrote it first, and the semantics — who retries, who deduplicates, what an error means — will be undocumented and discovered in an incident.\nReorganizing without expecting architectural change. The reorg is an architectural change. It has been applied to the system already; the code just has not caught up yet.\nDoes not forbid It does not say the org chart determines the architecture, and taking it that way is what produces the cynical version. Communication structure is the variable, and it is only correlated with the formal hierarchy. Two teams with a shared on-call, a common design doc, and a weekly review are one communication node regardless of who they report to. Conway is not a claim that you are helpless; it is a claim about which lever works.\nIt does not forbid a good architecture from a badly-shaped organization. It prices one. Linux has a famously monolithic kernel built by thousands of loosely-connected contributors, which is close to the opposite of what a naive reading predicts, and it works because the maintainer hierarchy and the subsystem boundaries were deliberately aligned — an inverse maneuver run for thirty years. The mechanism was paid, not evaded.\nIt does not mean microservices are wrong, which is the misapplication doing the most damage right now. \u0026ldquo;Conway\u0026rsquo;s law says you need one team per service\u0026rdquo; has been used to justify splitting a twelve-person company into nine services, and the result is nine deployment pipelines maintained by people who all sit at the same table. The law says the architecture will mirror the communication structure. For a small, high-bandwidth team the mirrored architecture is a monolith, and building one is the Conway-compliant answer, not a failure of nerve. The law is symmetric and people only ever quote the half that recommends splitting.\nIt does not license the fatalist reading. \u0026ldquo;Conway\u0026rsquo;s law means our architecture is determined by the org chart, so there is no point arguing about design\u0026rdquo; is the most common way this gets deployed in a meeting, and it inverts the paper\u0026rsquo;s purpose. Conway\u0026rsquo;s own closing point is that because the homomorphism holds, the organization is a design tool. The law says where the lever is; it does not say the lever is nailed down.\nIt does not predict the quality of the resulting design. It predicts the shape. A well-structured organization produces a well-structured system that may still be slow, wrong, or solving the wrong problem. Conway constrains modularity, and nothing else.\nBoundary The inverse Conway maneuver. Since the homomorphism runs from communication structure to architecture, change the former to get the latter: decide the architecture, then organize teams to match it, and let the pressure work for you. This is the entire operating premise of Team Topologies and of Amazon\u0026rsquo;s two-pizza teams. It is the most valuable practical use of the law and the one most often stated as if it were the law itself. The caveat is real though: you are betting on an architecture before you have built it, and the organization is much harder to reorganize than the code once you are wrong. Where it weakens. Very small teams (everyone talks to everyone, so the communication graph is complete and constrains nothing), and systems where one architect writes all the interfaces. Both are the same escape: the coordination cost has gone to zero, so the mechanism has nothing to push against. Both stop applying at the size where they stopped being true, which is usually around a dozen people and is noticed about a year late. Where it strengthens. Long-lived systems, multiple locations, high turnover, and anywhere interfaces are expensive to change — public APIs, on-disk formats, protocol versions. Hyrum\u0026rsquo;s law (T091) is the reason those become permanent, and Conway\u0026rsquo;s is the reason they landed where they did. The two compose badly and predictably: Conway puts the seam in the wrong place, Hyrum welds it shut. The measurable version. Socio-technical congruence (Cataldo et al.) makes this quantitative: measure the coordination the code requires against the coordination that actually happens, and the gap predicts defects and delay. That turns Conway from an aphorism into an instrument you can point at a repository, which is the honest answer to \u0026ldquo;empirical, so what do I do with it.\u0026rdquo; Read next ","permalink":"https://cs.lozic.me/posts/t089-conways-law/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eThe service has four layers. Nobody designed four layers. There were four teams.\u003c/p\u003e\n\u003cp\u003eOr the version that arrives in a postmortem: the outage happened at the boundary\nbetween two services, in the retry logic, because each side assumed the other\nowned the deduplication. Both teams were right about their own component and\nneither had ever been in a room with the other.\u003c/p\u003e","title":"Conway's Law"},{"content":"Symptom The security team asks for a scanner with no false positives and no false negatives. Every piece of malware caught, nothing legitimate quarantined.\nThe platform team asks whether the analyzer can flag every function that performs I/O, so the pure ones can be cached automatically.\nSomebody in review asks why the compiler cannot just eliminate all the dead code, given that it clearly eliminates some.\nYou have already met the answer for one property: termination is undecidable (T010). What is not obvious, and what makes this the result that actually governs your working life, is that termination was not special. Every one of these questions is undecidable, and so is nearly every question of that form you will ever be asked, and there is a single theorem that says so without requiring you to construct a proof each time.\nStatement Every non-trivial semantic property of programs is undecidable.\nTwo words are load-bearing, and almost every misapplication of this theorem is a failure to check one of them.\nSemantic means the property is about the function the program computes, not about its text. Formally, if $P$ and $Q$ compute the same function, then $P$ has the property exactly when $Q$ does. \u0026ldquo;Halts on every input\u0026rdquo; is semantic. \u0026ldquo;Contains the substring eval\u0026rdquo; is not, and neither is \u0026ldquo;is under 100 lines\u0026rdquo; or \u0026ldquo;has more than three nested loops.\u0026rdquo;\nNon-trivial means some program has the property and some program does not. \u0026ldquo;Is a program\u0026rdquo; is trivial (all of them). \u0026ldquo;Computes a function no program computes\u0026rdquo; is trivial (none of them). Everything else is non-trivial, which is to say: everything interesting.\nLet $\\mathcal{P}$ be a set of partial computable functions with $\\mathcal{P} \\ne \\emptyset$ and $\\mathcal{P} \\ne$ all of them. Then\n$$\\{\\, \\langle M \\rangle : \\text{the function computed by } M \\text{ is in } \\mathcal{P} \\,\\}$$is undecidable.\nRead the quantifier carefully, because it is unusually strong. This is not \u0026ldquo;some properties are undecidable.\u0026rdquo; It is: pick any semantic property at all, at random, sight unseen. If it is not trivial, it is undecidable. There are no interesting exceptions, and you do not have to search for a proof.\nArgument We reduce from halting. Assume a decider $R$ for the property $\\mathcal{P}$, and build from it a decider for HALT, which T010 forbids.\nLet $\\varnothing$ denote the everywhere-undefined function — the one computed by a program that loops on every input. Without loss of generality, assume $\\varnothing \\notin \\mathcal{P}$. (If it is in $\\mathcal{P}$, run the whole argument on the complement property, which is also semantic and non-trivial, and is decidable exactly when $\\mathcal{P}$ is.)\nSince $\\mathcal{P}$ is non-empty, pick some $f \\in \\mathcal{P}$, computed by a program $F$.\nNow, given an arbitrary program $P$ and input $x$ — the halting question we want to answer — construct this program:\ndef M_Px(y): run P on x # ignore the result; may never return return F(y) # only reached if the line above returned Building M_Px from P, x, and F is pure text manipulation. No execution. It always succeeds, and it terminates, because writing a program is not running it. This step is the one worth slowing down on: we are not asking whether M_Px halts, only writing it down.\nWhat function does M_Px compute?\nIf $P$ halts on $x$, the first line returns and M_Px(y) = F(y) for every $y$. So M_Px computes $f$, which is in $\\mathcal{P}$. If $P$ does not halt on $x$, the first line never returns, so M_Px(y) is undefined for every $y$. So M_Px computes $\\varnothing$, which is not in $\\mathcal{P}$. So M_Px has property $\\mathcal{P}$ if and only if $P$ halts on $x$.\nNow run the decider we assumed exists:\ndef Halts(P, x): return R(build_M(P, x)) That is a total, always-correct decider for the halting problem. T010 says there is none. So $R$ does not exist. $\\blacksquare$\nNotice what the proof never did: it never looked at what $\\mathcal{P}$ is. The argument works for \u0026ldquo;computes a total function,\u0026rdquo; \u0026ldquo;always returns 0,\u0026rdquo; \u0026ldquo;is equivalent to this reference implementation,\u0026rdquo; \u0026ldquo;never writes to the network.\u0026rdquo; The property is a black box, which is precisely why the theorem covers all of them at once. Undecidability here is not a fact about any particular property; it is a fact about the fact that programs can be composed.\nCohen\u0026rsquo;s corollary: perfect virus detection is impossible. Fred Cohen ran this argument in 1987 with $\\mathcal{P}$ = \u0026ldquo;propagates itself,\u0026rdquo; and got the result the antivirus industry has lived under ever since. The neat version is a direct diagonal, and it is worth seeing because it needs no reduction at all. Suppose is_virus(P) is a perfect detector. Write:\ndef V(): if not is_virus(V): # ask the detector about myself infect() # ... and if it says I\u0026#39;m clean, infect If is_virus(V) says clean, V infects, so it was a virus and the detector was wrong. If it says infected, V does nothing at all, so it was clean and the detector was wrong. Same structure as D in the halting proof, and it explains the industry\u0026rsquo;s shape: signatures (unsound, miss new malware), heuristics (false positives), sandboxing (bounded, so evadable by waiting). Those are not three competing products. They are the three concessions the theorem leaves available.\nForbids A perfect optimizer. \u0026ldquo;Is this code dead?\u0026rdquo; is semantic and non-trivial, so no compiler eliminates exactly the dead code. Every optimizer is conservative: it removes what it can prove is dead and leaves the rest.\nA perfect equivalence checker. \u0026ldquo;Does $M$ compute the same function as this reference?\u0026rdquo; is the property \u0026ldquo;computes $f$\u0026rdquo;, non-trivial and semantic. So no tool decides whether your rewrite preserved behaviour. This is why refactoring is tested rather than proved, and why superoptimizers work over tiny fragments.\nA sound and complete effect checker. \u0026ldquo;Is this function pure?\u0026rdquo;, \u0026ldquo;does it ever touch the filesystem?\u0026rdquo;, \u0026ldquo;can it throw?\u0026rdquo; — all semantic. Any language that tracks effects does it by making programmers declare them in types, converting a semantic question into a syntactic one, which is the only move available.\nPerfect malware detection, per Cohen, and by the same argument perfect detection of any behavioural class: cryptominers, exfiltration, ransomware.\nA perfect security auditor. \u0026ldquo;Does this program ever leak the secret key?\u0026rdquo; is semantic. Whatever your audit does, it is not this.\nDoes not forbid It does not forbid type systems, and the \u0026ldquo;Rice\u0026rsquo;s theorem says static analysis is impossible\u0026rdquo; line is the single most damaging misuse of a theorem in this series. It shows up in real code review to reject real tooling. But look at what the theorem requires: a decider is sound, complete, and total. Give up completeness and everything opens up. Rust\u0026rsquo;s borrow checker rejects some memory-safe programs and is the reason a systems language shipped without use after free. Java\u0026rsquo;s type checker rejects some programs that would never throw a ClassCastException. mypy, clang-tidy, Infer, and CodeQL all run at scale and find real defects daily. Every one of them answers a three-valued question — yes / no / I reject this rather than guess — while the theorem forbids only the two-valued one. A tool that says \u0026ldquo;I cannot prove this safe, so I will not accept it\u0026rdquo; is not defeated by Rice. It is the intended response to Rice.\nIt does not forbid syntactic properties, and this exemption is larger than it sounds. \u0026ldquo;Does the source contain eval?\u0026rdquo;, \u0026ldquo;is every malloc matched by a free on the same syntactic path?\u0026rdquo;, \u0026ldquo;is this identifier ever assigned twice?\u0026rdquo; — all decidable, all trivially, because they are about the text. A great deal of practical linting is deliberately syntactic for exactly this reason, and the engineering skill is picking a syntactic proxy whose failure modes you can live with. Rice does not apply. It is a theorem about semantics and it says nothing about grep.\nIt does not forbid deciding the property on a restricted language. Rice quantifies over all Turing-complete programs. Terminating languages (Dhall, Starlark, total fragments of Agda), finite-state models (TLA+ with bounded constants, SPIN), straight-line code, and loop-free circuits all escape completely. SQL without recursive CTEs has decidable equivalence. This is the single most-used escape hatch in practical language design, and it is a design decision, not a compromise.\nIt does not forbid deciding the property on your program. The theorem forbids a uniform procedure over all programs. seL4 is a verified operating system kernel. CompCert is a verified C compiler. Both are finished, both make absolute semantic guarantees, and neither contradicts anything here, because a proof about one program is not an algorithm for all programs. When someone says verification is impossible because of Rice, they have swapped a quantifier.\nIt does not apply to intensional properties. \u0026ldquo;Does this program run in under one second?\u0026rdquo;, \u0026ldquo;how many instructions does it execute?\u0026rdquo;, \u0026ldquo;how much memory does it allocate?\u0026rdquo; — these distinguish programs that compute the same function, so they are not semantic in Rice\u0026rsquo;s sense, and the theorem is silent. (Many are still undecidable, by other arguments. But not by this one, and getting the reason right matters when you need to know whether a restriction will help.)\nIt does not say the undecidable cases are common in practice. Rice guarantees that some program defeats your analyzer. It says nothing about the density of such programs in real code, and empirically the density is very low — which is why an escape analysis that works on 95% of real Java methods is possible and shipping, notwithstanding that a program exists which defeats it. The theorem describes the worst case, and the worst case is not the workload.\nBoundary Everything that works lives in a well-mapped set of concessions, and it is worth naming them as a set, because a tool is essentially always making one:\nDrop completeness → sound static analysis. Answer {definitely yes, don\u0026rsquo;t know}, never wrong when it commits. Type systems, abstract interpretation (T047), the borrow checker. The dominant choice where correctness matters. Drop soundness → bug finding. Answer {probably yes, probably no}, wrong sometimes, useful anyway. Most linters, signature-based antivirus, heuristic scanners. The dominant choice where coverage matters more than guarantees. Drop totality → semi-decision. Answer {yes, still running}. Model checkers, symbolic execution, SMT-backed verifiers. Fine when you can afford to wait and to give up. Drop Turing-completeness → decidability returns outright. Total languages, finite-state protocol models, non-recursive query languages. The strongest move available, and the one most often overlooked, because it is made at design time rather than at analysis time. Drop \u0026ldquo;arbitrary program\u0026rdquo; → per-program proof. seL4, CompCert. Expensive, absolute, and the expense is human rather than algorithmic. Abstract interpretation (T047) is the principled version of the first concession: it makes \u0026ldquo;approximate, but only in the safe direction\u0026rdquo; into a mathematical object with a soundness proof, so that \u0026ldquo;I gave up completeness\u0026rdquo; can be stated exactly rather than hoped for. That is the post this one sets up.\nRead next ","permalink":"https://cs.lozic.me/posts/t011-rices-theorem/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eThe security team asks for a scanner with no false positives and no false\nnegatives. Every piece of malware caught, nothing legitimate quarantined.\u003c/p\u003e\n\u003cp\u003eThe platform team asks whether the analyzer can flag every function that\nperforms I/O, so the pure ones can be cached automatically.\u003c/p\u003e","title":"Rice's Theorem"},{"content":"Symptom Your CI has a test that hangs. Not fails — hangs. Somebody suggests the obvious fix: before running a test, check whether it terminates, and skip it if not.\nOr: you are writing a plugin system, and you would like to reject plugins that contain infinite loops before you load them, because an infinite loop in a plugin takes down the host.\nOr the version that gets funded: a static analyzer that flags every unbounded loop in a codebase, with no false positives and no false negatives, so the security team can finally close the \u0026ldquo;unbounded resource consumption\u0026rdquo; ticket class permanently.\nEvery one of these is a request for the same program, and that program does not exist. Not \u0026ldquo;is hard to write.\u0026rdquo; Does not exist, in the way that a largest prime does not exist.\nStatement There is no program $H$ that takes an arbitrary program $P$ and input $x$ and always correctly answers whether $P$ halts on $x$.\nThree words carry the weight. Arbitrary: $H$ must work for every $P$, not for the ones you have. Always: $H$ must itself terminate with an answer, so \u0026ldquo;simulate it and see\u0026rdquo; is not a solution — it never returns on the inputs you most wanted to know about. Correctly: no wrong answers, ever.\nThe language\n$$\\mathrm{HALT} = \\{\\, \\langle P, x\\rangle : P \\text{ halts on input } x \\,\\}$$is undecidable. It is recognizable — a machine can say yes when the answer is yes — but no machine says both yes and no in finite time on all inputs.\nThe gap between recognizable and decidable is the whole content of the theorem, and it is exactly the gap between a debugger and an oracle. You can always confirm halting by waiting. What you cannot do is confirm non-halting.\nArgument Suppose, for contradiction, that $H$ exists: a program where H(P, x) returns true if $P$ halts on $x$ and false otherwise, always terminating.\nThen write this, which is nine lines and does nothing clever:\ndef D(P): if H(P, P): # does P halt when fed its own source? while True: # ... then loop forever pass else: return # ... otherwise halt immediately D is a perfectly ordinary program: it calls a function we assumed exists, and branches. If $H$ is a program, so is D.\nNow ask the only question available. What does D(D) do?\nCase 1: D(D) halts. Then H(D, D) returned true, by the correctness of $H$. But look at the code: when H(P, P) is true, D enters while True and never halts. So D(D) does not halt. Contradiction.\nCase 2: D(D) does not halt. Then H(D, D) returned false. But when H(P, P) is false, D returns immediately. So D(D) halts. Contradiction.\nBoth cases are impossible, and one of them must hold, since a program either halts or does not. The only assumption we made was that $H$ exists. So $H$ does not exist. $\\blacksquare$\nWhere the diagonal went. This is T002 with the rows relabelled. Imagine the infinite table whose rows are programs $P_1, P_2, \\dots$ (countably many, since programs are finite strings), whose columns are inputs in the same enumeration, and whose cell $(i, j)$ records whether $P_i$ halts on $P_j$. D is built to walk the diagonal and disagree with it: D behaves, on input $P_n$, in the opposite way to how $P_n$ behaves on $P_n$. So D differs from every row of the table in at least one cell, and therefore is not any $P_n$. But D is a program, and every program is a row. The only escape is that the table cannot be filled in — that $H$, the thing that computes the cells, is not a program.\nCantor concluded \u0026ldquo;the list is incomplete.\u0026rdquo; Turing arranged matters so the list must be complete — programs really are all enumerable — and therefore the contradiction lands somewhere else, on the halting oracle. Same flip, harder conclusion, and this is why it is worth reading the two proofs together.\nThe self-application is not the trick. Feeding D its own source looks like the sleight of hand, and it is the step people distrust. It should not be distrusted: passing a program its own text is something you do routinely, every time you run a compiler on its own source, and a quine is a fifty-character demonstration that self-reference needs no special power. The recursion theorem (T014) makes this exact: any program can be written to have access to its own source, with no loss of generality. The diagonal is not exploiting a loophole in what programs may do. It is using an ordinary capability at an inconvenient moment.\nForbids The universal termination checker. No tool, in any language, at any budget, now or later, decides termination for arbitrary input programs. This is not a statement about current technology.\nPerfect infinite-loop detection in a plugin sandbox. Which is why every real sandbox uses a timeout or a fuel counter instead — an answer to a different, decidable question (\u0026ldquo;does it halt within $n$ steps?\u0026rdquo;).\nA sound, complete, terminating analysis for any property that encodes halting. And an enormous number of properties do, by reduction: dead code elimination, \u0026ldquo;is this variable ever used\u0026rdquo;, \u0026ldquo;can this pointer be null here\u0026rdquo;, \u0026ldquo;does this regex ever match\u0026rdquo;. Show that solving your property would let you solve halting, and your property inherits the impossibility. That machinery, reductions, gets its own post (T017); the generalization that every nontrivial semantic property falls this way is Rice\u0026rsquo;s theorem (T011), and it is the next one.\nThe mathematician\u0026rsquo;s version. Hilbert\u0026rsquo;s Entscheidungsproblem — a procedure to decide the truth of any first-order statement — dies here too, since a machine\u0026rsquo;s halting can be encoded as such a statement. Turing\u0026rsquo;s paper is titled after that problem, not after halting. Halting was the lemma.\nDoes not forbid It does not forbid termination checkers, and this is the misreading with the highest cost. \u0026ldquo;Halting is undecidable\u0026rdquo; gets used, in real code review, to reject termination analysis as a category error. Meanwhile Coq, Agda, Lean, and Idris all ship termination checkers that work, are sound, and always terminate; Rust\u0026rsquo;s borrow checker and the Linux kernel\u0026rsquo;s eBPF verifier both require proof of termination before accepting code, and the eBPF verifier is in production on essentially every large fleet in the world. The resolution: these tools are allowed to say no to programs that in fact halt. A checker that answers {halts, don\u0026rsquo;t know} is trivially possible, useful, and totally different from what the theorem forbids. The theorem kills the checker with two exact answers, not the checker with a conservative one.\nIt does not forbid deciding halting for restricted languages. Every loop-free program halts, decidably. So does every program in a total language, every primitive recursive function, every terminating rewrite system, every SQL query without recursive CTEs, and every Dhall or Starlark configuration. This is the main reason configuration languages are not Turing-complete on purpose. If your inputs are not arbitrary programs, the theorem simply does not apply to you, and \u0026ldquo;but halting is undecidable\u0026rdquo; is not a valid objection to a language design that has already dodged it.\nIt does not forbid deciding halting for the program in front of you. The theorem quantifies over all programs; it is silent about any particular one. while (true) {} provably does not halt and the proof takes one line. The Collatz iteration is unknown, but \u0026ldquo;unknown\u0026rdquo; is a fact about mathematics in 2026, not a consequence of Turing\u0026rsquo;s theorem. Undecidability of the general problem is compatible with every specific instance you will ever meet being settled.\nIt does not mean \u0026ldquo;you cannot know if your program has bugs.\u0026rdquo; This is the pub version and it is false in a way that discourages useful work. Bounded model checking, exhaustive testing over a finite state space, TLA+ on a fixed configuration, and a type checker rejecting a null dereference all establish real facts about real programs. Undecidability constrains what a single algorithm can do uniformly over all inputs. It places no bound at all on what you can learn about one system.\nIt does not imply human minds exceed machines. The Lucas–Penrose argument says: a machine cannot decide halting, but I can see that while(true){} loops, therefore mind is not machine. The step is invalid. Humans do not decide halting either — nobody has settled Collatz — and the theorem is not about who is clever enough. It says no uniform total procedure exists, and that applies to any procedure a human could follow just as it applies to silicon.\nBoundary The result is sharp, so the interesting question is what sits immediately outside it.\nBounded halting is decidable, and this is what everyone actually ships. \u0026ldquo;Does $P$ halt within $10^9$ steps?\u0026rdquo; is answered by running it for $10^9$ steps. Every timeout, watchdog, query governor, and eBPF instruction limit is this decision procedure. The cost is that you have replaced a question about the program with a question about the program and a budget, and the busy beaver function\u0026rsquo;s growth means no budget is generous enough to be equivalent to the real question. One-sided answers. Give up on always terminating, keep soundness, and you get semi-decision: a recognizer that says yes eventually when the answer is yes and runs forever otherwise. That is what a fuzzer and a symbolic executor are. Give up on completeness instead and you get the abstract interpreters (T047): always terminate, always sound, sometimes say \u0026ldquo;maybe.\u0026rdquo; Ranking functions. The practical technique behind every working termination checker: exhibit a value that strictly decreases on every iteration and is bounded below, and termination follows. Finding one is undecidable in general; finding one for a for i in 0..n loop is trivial, and most real loops are closer to the second case. Terminator and its descendants built a research programme on exactly this gap between \u0026ldquo;impossible in general\u0026rdquo; and \u0026ldquo;usually easy.\u0026rdquo; Change the model and nothing improves. Add randomness, add nondeterminism, add infinite parallelism, add a quantum computer: halting stays undecidable, because the diagonal argument never inspected the machinery. The one thing that does help is assuming a machine strictly more powerful than a Turing machine — an oracle — and then the same proof runs one level up and leaves that machine\u0026rsquo;s halting problem undecidable. The hierarchy is infinite and nothing at any level can see its own level. The boundary of this result is not a wall you can go around; it is the same wall at every altitude. Read next ","permalink":"https://cs.lozic.me/posts/t010-the-halting-problem/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eYour CI has a test that hangs. Not fails — hangs. Somebody suggests the obvious\nfix: before running a test, check whether it terminates, and skip it if not.\u003c/p\u003e","title":"The Halting Problem"},{"content":"Symptom Somebody proposes a tool that will find all the bugs. Not most bugs. All of them. The pitch has the shape: we enumerate the failure modes, we write a checker for each, we keep adding checkers, and the set of undetected bugs shrinks toward zero.\nOr the version you have probably said yourself, over a beer, about a language: eventually the type system will be strong enough to reject every program that goes wrong.\nOr the simplest one, which is a question rather than a claim: there are infinitely many possible programs, and infinitely many possible problems, so surely there is enough supply to meet demand?\nThere is not. There are strictly more problems than there are programs, and the gap is not a small one, and this is provable in about fifteen lines using nothing but a table and the ability to add one.\nStatement Cantor\u0026rsquo;s theorem has a general form, but the version that does the work here is concrete:\nThe set of infinite binary sequences is uncountable. No matter how you list such sequences $s_1, s_2, s_3, \\dots$, some infinite binary sequence is missing from your list.\n\u0026ldquo;Countable\u0026rdquo; means you can put the elements in a list indexed by $1, 2, 3, \\dots$ — every element gets a finite position number, with no element left out. The integers are countable. The rationals are countable, which already surprises people. The infinite binary sequences are not.\nThe general statement, which costs nothing extra once you have the specific one:\nFor any set $S$, there is no surjection from $S$ onto its power set $\\mathcal{P}(S)$. So $|S| \u0026lt; |\\mathcal{P}(S)|$, always, including when $S$ is infinite.\nArgument Suppose someone hands you a list of infinite binary sequences and claims it contains every one of them. Write the list as a table, one sequence per row:\n$$ \\begin{array}{c|cccccc} \u0026 1 \u0026 2 \u0026 3 \u0026 4 \u0026 5 \u0026 \\cdots \\\\ \\hline s_1 \u0026 \\mathbf{0} \u0026 1 \u0026 1 \u0026 0 \u0026 1 \u0026 \\cdots \\\\ s_2 \u0026 1 \u0026 \\mathbf{1} \u0026 0 \u0026 0 \u0026 1 \u0026 \\cdots \\\\ s_3 \u0026 0 \u0026 0 \u0026 \\mathbf{0} \u0026 1 \u0026 1 \u0026 \\cdots \\\\ s_4 \u0026 1 \u0026 0 \u0026 1 \u0026 \\mathbf{1} \u0026 0 \u0026 \\cdots \\\\ s_5 \u0026 0 \u0026 1 \u0026 1 \u0026 0 \u0026 \\mathbf{0} \u0026 \\cdots \\\\ \\vdots \u0026 \u0026 \u0026 \u0026 \u0026 \u0026 \\ddots \\end{array} $$Now build a new sequence $d$ by walking down the diagonal — the bolded entries, $s_1$\u0026rsquo;s first bit, $s_2$\u0026rsquo;s second bit, $s_3$\u0026rsquo;s third — and flipping each one. In the table above the diagonal reads $0,1,0,1,0,\\dots$, so\n$$d = 1,0,1,0,1,\\dots$$Ask the only question that matters: is $d$ somewhere in the list?\nIt is not $s_1$, because $d$ differs from $s_1$ in the first bit — we made it so. It is not $s_2$, because $d$ differs from $s_2$ in the second bit. In general $d$ is not $s_n$ for any $n$, because $d$ and $s_n$ disagree at position $n$ by construction. Two sequences that disagree anywhere are not the same sequence.\nSo $d$ is an infinite binary sequence not in the list. The list was not complete. And this holds for every list, since we never assumed anything about which sequences were in which rows. Therefore no complete list exists.\nThat is the whole proof. Note the shape, because it recurs: we did not find a sequence that lists have trouble with. We built, from any given list, a specific counterexample to that list\u0026rsquo;s completeness — a machine that eats an alleged solution and outputs its refutation.\nWhy this counts programs. A program is a finite string over a finite alphabet. Finite strings over a finite alphabet are countable: list all strings of length 1, then all of length 2, and so on, each block finite, every string eventually reached. So the programs are countable, in a list, with $n$-th entries.\nNow consider decision problems on the naturals — functions $f : \\mathbb{N} \\to {0,1}$. Each such function is an infinite binary sequence. There are uncountably many of them. Countably many programs, uncountably many problems.\nThe counting is not close. Deleting the computable problems from the set of all problems removes so little that the remainder is still exactly as large as it started. Almost every function $\\mathbb{N} \\to {0,1}$ is not computed by anything, and this was known before anybody had built a computer.\nForbids A complete enumeration of the reals. Not \u0026ldquo;we haven\u0026rsquo;t found one\u0026rdquo; — there is none, and this is where the argument was born.\nA program that computes every function. Trivially, from the counting: the programs run out and the functions do not.\nA sound and complete finite checker for every semantic property. Any analysis tool is a program, so tools are countable; the properties they might need to decide are not. This is a counting argument, so it forbids only in aggregate — it says the tools cannot cover everything, without naming a single problem they miss. Naming one takes the diagonal argument applied to machines rather than to sequences, which is the halting problem, and which is the next post.\nAny claim of the form \u0026ldquo;our set is the same size as its power set.\u0026rdquo; Cantor\u0026rsquo;s general theorem kills that for every $S$, finite or infinite. There is no largest infinity; $\\mathcal{P}$ always escapes.\nDoes not forbid It does not mean \u0026ldquo;we can\u0026rsquo;t reach the missing sequence.\u0026rdquo; The diagonal $d$ is not mysterious or unreachable. Given the list, $d$ is completely specified — you can compute any bit of it you like, provided you can compute bits of the listed sequences. The proof is a construction, not an appeal to the unknown. This matters because the mystical reading (\u0026ldquo;some truths are forever beyond us\u0026rdquo;) is both the popular one and the wrong one.\nIt does not say that most problems you will face are unsolvable, and this is the misreading that does real damage. The inference runs: almost all functions are uncomputable, therefore expecting to solve an arbitrary hard problem is naive. It gets deployed against real projects — the SMT-solver-based verifier, the automatic scheduler, the program-synthesis tool — as if the counting argument had ranged over the problems that occur in practice. It has not. The uncountably many functions are, essentially all of them, infinite tables of bits with no finite description. A problem that arrives in a ticket has a finite description by construction, which places it in the countable sliver before anyone starts work. Cantor says the supply of describable problems cannot cover the space of all functions. It says nothing about whether your problem is in the covered part, and the base rate there is far better than the theorem\u0026rsquo;s arithmetic suggests. Z3 solves NP-hard instances daily.\nIt does not forbid deciding a property for the programs you actually have. Uncountability is a statement about an infinite space. Your codebase is finite. \u0026ldquo;Does this specific 200-line function terminate?\u0026rdquo; is very often answerable, and a termination checker that succeeds on 90% of real loops and says \u0026ldquo;don\u0026rsquo;t know\u0026rdquo; on the rest is not defeated by Cantor. Rejecting a tool because a theorem forbids its perfect version is how you end up with no tool.\nIt does not say the uncomputable functions are weird. This inverts the truth and it is worth stating plainly. The computable functions are the vanishingly rare special case — a measure-zero sliver. Every function you have ever written down lives in that sliver, which is why the sliver feels like the whole world. \u0026ldquo;Almost all functions are uncomputable\u0026rdquo; is not a statement about pathology; it is a statement about how extraordinarily special the things we can name are.\nIt does not require the axiom of choice, or any set theory beyond a table. People who half-remember this argument sometimes suspect a trick in the foundations. There is no trick. The proof uses one list, one diagonal, and one negation.\nBoundary The result is exact — $|S| \u0026lt; |\\mathcal{P}(S)|$ with no slack — so the interesting territory is not weakening it but redirecting it.\nDiagonalization against machines rather than sequences. Rows become programs, and the diagonal entry becomes \u0026ldquo;what does program $n$ do on input $n$.\u0026rdquo; Flipping that gives a program that disagrees with every program on at least one input, which is the halting problem (T010) and, with a change of vocabulary, Gödel\u0026rsquo;s first incompleteness theorem (T015). Same table, same flip, different labels on the rows. That is the reason this post exists this early: the technique is worth more than the theorem. Diagonalization inside complexity. Run the diagonal with a clock on it and you separate complexity classes: the time hierarchy theorem says more time strictly buys more computable problems. The same trick, resource-bounded. Where diagonalization stops working. It is not a universal solvent, and knowing where it fails is the mark of understanding it. Relativization (Baker–Gill–Solovay) shows that any argument that survives adding an oracle cannot settle P vs NP, and plain diagonalization survives oracles. That is why fifty years of diagonalizing has not resolved the field\u0026rsquo;s central question, and why the techniques that have made progress look nothing like this one. Constructive readings. In constructive mathematics the diagonal argument survives essentially intact, which is unusual for a proof by contradiction — because it is not really one. It is a construction with a contradiction stapled to the end for presentation. Strip the staple and it still builds $d$. Read next ","permalink":"https://cs.lozic.me/posts/t002-cantors-diagonal-argument/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eSomebody proposes a tool that will find all the bugs. Not most bugs. All of\nthem. The pitch has the shape: we enumerate the failure modes, we write a\nchecker for each, we keep adding checkers, and the set of undetected bugs\nshrinks toward zero.\u003c/p\u003e","title":"Cantor's Diagonal Argument"},{"content":"Symptom Two measurements that both look like they break the previous post.\nThe first: your Spark job takes four hours on ten nodes. You move it to a hundred nodes and it still takes four hours — but it is now processing ten times the data. T081 said the ceiling was 20×, and nobody hit a ceiling.\nThe second, less pleasant: your service does 12,000 requests per second on 40 nodes. You provision 80, expecting somewhere north of 12,000 and prepared to settle for 15,000. You measure 10,400. The graph does not flatten — it turns over and goes down, and the only thing you changed was the amount of hardware.\nAmdahl\u0026rsquo;s law explains neither, because Amdahl\u0026rsquo;s law is missing a term in one direction and an assumption in the other.\nStatement What T081 established, in two sentences: fix the workload, split a fraction of it across $N$ workers, and speedup is $1/((1-p) + p/N)$, bounded forever by $1/(1-p)$. Everything below keeps that algebra and changes what is held fixed.\nGustafson\u0026rsquo;s law. Hold time fixed instead of work. If $\\alpha$ is the serial fraction of the scaled workload — the workload you actually run on $N$ nodes — then the scaled speedup is\n$$S(N) = N - \\alpha(N - 1)$$ Linear in $N$, with a slope slightly under 1. No ceiling.\nThe Universal Scalability Law (Gunther). Relative throughput on $N$ workers:\n$$C(N) = \\frac{N}{1 + \\alpha(N-1) + \\beta N(N-1)}$$where $\\alpha$ is contention (work that serialises) and $\\beta$ is coherency (the cost of workers keeping each other consistent).\nWith $\\beta \u0026gt; 0$ this function rises, peaks, and falls. The peak sits at\n$$N^{*} = \\sqrt{\\frac{1-\\alpha}{\\beta}}$$and beyond it, adding capacity removes throughput.\nArgument Gustafson is Amdahl asked backwards. Amdahl fixes the problem and asks how much faster $N$ workers finish it. Gustafson fixes the wall clock and asks how much more work $N$ workers get through. Normalise the parallel run to 1 unit of time, of which $\\alpha$ is serial and $1-\\alpha$ is parallel. To do that same work on one processor takes $\\alpha + (1-\\alpha)N$, since the parallel part now runs sequentially. Speedup is the ratio:\n$$S(N) = \\alpha + (1-\\alpha)N = N - \\alpha(N-1)$$That is the entire derivation, and it is worth noticing that neither law is more correct than the other. They differ in one assumption — whether problem size is pinned — and the assumption is a fact about your workload, not a modelling choice. A user waiting on a request has a pinned problem: Amdahl. A nightly job that will happily chew a bigger shard: Gustafson.\nThe USL adds the term Amdahl is missing. Start by rewriting Amdahl as a throughput curve. With serial fraction $\\alpha$,\n$$S(N) = \\frac{1}{\\alpha + (1-\\alpha)/N} = \\frac{N}{1 + \\alpha(N-1)}$$which is the USL with $\\beta = 0$. So Amdahl is a special case, and the question is what it left out.\nIt left out the workers talking to each other. Amdahl\u0026rsquo;s penalty term $\\alpha(N-1)$ grows linearly: each additional worker waits on the same serial section. But cache-coherence traffic, distributed lock handoff, gossip, and cross-replica consistency are not one worker waiting on a queue — they are pairs of workers exchanging state, and there are $N(N-1)$ ordered pairs. Hence the term $\\beta N(N-1)$, quadratic in $N$, in the denominator.\nThis is where the compound kind matters. The $\\alpha$ term is algebra: it follows from the definition of a serial section, exactly as in T081. The $\\beta$ term is not. It is a modelling choice — the functional form $N(N-1)$ is chosen because pairwise exchange is quadratic, and the coefficient $\\beta$ is fitted to measurements by regression. That is why this post is tagged empirical first and theorem second. Nobody proved your system\u0026rsquo;s coherency cost is quadratic. Somebody observed that the curve fits, remarkably often, across systems that have nothing else in common.\nThe peak. Differentiate $C(N)$, set to zero, and the roots give\n$$N^{*} = \\sqrt{\\frac{1-\\alpha}{\\beta}}$$Work an example. A service measures $\\alpha = 0.03$ and $\\beta = 0.0005$ — a 3% serial fraction and a coherency cost most people would call negligible. Then $N^{*} = \\sqrt{0.97/0.0005} = \\sqrt{1940} \\approx 44$.\n$N$ $C(N)$ 10 7.6× 20 11.4× 30 13.0× 44 13.6× 88 11.8× Forty-four nodes is the maximum this architecture can deliver, ever. Doubling to 88 does not stall — it loses 13% of the throughput, while doubling the bill. That is the second symptom at the top of this post, and the number that predicts it was available from a load test on 5, 10, 20 and 30 nodes.\nForbids Unbounded scale-out. With any $\\beta \u0026gt; 0$ there is a hard maximum throughput at a computable $N$, and every node past it makes things worse. \u0026ldquo;Throw more nodes at it\u0026rdquo; is not a strategy that degrades gracefully; it reverses.\nReading a ceiling off Amdahl for a distributed system. Amdahl is the optimistic bound. It cannot express retrograde scaling because it has no term that can produce one, so a capacity plan built on it will be wrong in the expensive direction.\nExtrapolating from two load points. Two points fit a line. Fitting $\\alpha$ and $\\beta$ needs enough points on both sides of the knee — six is the usual working minimum, and points beyond the peak are the most informative ones you can collect.\nCiting Gustafson for a latency budget. If a user is waiting, the problem size is pinned by definition and the scaled-speedup framing does not apply.\nDoes not forbid Gustafson does not repeal Amdahl. The commonest misuse, and it usually appears in a document justifying a cluster. Both laws are correct; they answer different questions. Any time the deliverable is \u0026ldquo;this specific request, faster\u0026rdquo;, the workload is fixed and Amdahl\u0026rsquo;s ceiling is the operative one. Gustafson applies to batch and throughput work, where a bigger machine means a bigger problem — which is precisely why HPC calls the two cases strong and weak scaling and insists that a benchmark say which it measured.\nThe USL does not prove that scaling stops around fifty nodes. $\\alpha$ and $\\beta$ are measured, not universal. Genuinely shared-nothing workloads — thumbnailing images off a queue, Monte Carlo runs, per-tenant sharded services — measure $\\beta$ statistically indistinguishable from zero and scale nearly linearly as far as anyone has bothered to test. The law describes the shape of the curve; your architecture decides the constants, and driving $\\beta$ toward zero is a design outcome rather than a fact of nature.\nA good fit is not a diagnosis. Fitting $\\beta = 0.0005$ tells you coherency cost exists. It does not tell you whether it is cache-line ping-ponging, a chatty coordinator, a distributed lock, or a connection pool. Teams that treat the fitted parameter as the finding, rather than as the prompt to go profiling, spend months optimising the wrong layer. The model is a curve fit over a system it knows nothing about.\nRetrograde scaling is not always coherency. The USL folds every superlinear-cost mechanism into one term. In practice the turnover is often queueing at a shared resource, garbage-collection pressure, or a thread pool thrashing — mechanisms with their own theory (T083, T084) and their own fixes. The curve is right about the shape and silent about the cause.\nBoundary Shard until $\\beta$ vanishes. The direct engineering response: partition so that workers do not need to know about each other. Shared-nothing architecture is the USL\u0026rsquo;s $\\beta$ term written as a design constraint. Fit the curve before buying the hardware. Six load points, a two-parameter regression, and $N^{*}$ falls out. This is the rare case where a model genuinely predicts the failure in advance. Weak scaling as an honest benchmark. If a vendor\u0026rsquo;s scaling graph does not say whether the problem grew, the graph does not say anything. The queueing side. Both laws are about parallel work. For a system serving arrivals, Little\u0026rsquo;s law (T083) and the $1/(1-\\rho)$ utilisation blowup (T084) bind sooner and are the correct tools. The degenerate cases are worth holding onto. $\\beta = 0$ gives Amdahl; $\\alpha = \\beta = 0$ gives perfect linear speedup. One formula contains the whole family, which is what the \u0026ldquo;universal\u0026rdquo; in the name is claiming. Read next ","permalink":"https://cs.lozic.me/posts/t082-gustafsons-law-and-the-usl/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eTwo measurements that both look like they break the previous post.\u003c/p\u003e\n\u003cp\u003eThe first: your Spark job takes four hours on ten nodes. You move it to a\nhundred nodes and it still takes four hours — but it is now processing ten times\nthe data. \u003ca href=\"https://cs.lozic.me/posts/t081-amdahls-law/\"\u003eT081\u003c/a\u003e said the ceiling was\n20×, and nobody hit a ceiling.\u003c/p\u003e","title":"Gustafson's Law and the Universal Scalability Law"},{"content":"Symptom The profile was unambiguous: 94% of wall-clock time in one loop, and the loop\u0026rsquo;s iterations are independent. You parallelise it across sixteen cores, the flame graph flattens exactly as predicted, and the end-to-end time goes from 40 seconds to 17.\nNot 2.5 seconds. Seventeen. And the machine with 64 cores gives you 14.\nNobody made a mistake. The 6% you did not touch is now the entire story, and it will stay the entire story no matter what hardware you buy. The arithmetic that says so is one line long, and running it before the sprint rather than after is most of what this post is for.\nStatement Let $p$ be the fraction of the work that can be parallelised, so $1-p$ is the serial fraction — the part that must happen in order, on one worker, no matter how many are available. With $N$ workers:\n$$S(N) = \\frac{1}{(1-p) + \\dfrac{p}{N}} \\qquad\\text{and}\\qquad S(\\infty) = \\frac{1}{1-p}$$ The second equation is the one with teeth. The speedup ceiling depends only on the serial fraction, and $N$ has disappeared from it entirely. Five percent serial caps you at 20×. One percent caps you at 100×. There is no quantity of hardware that moves either number.\nArgument There is no proof technique here to admire; it is division, and that is exactly why the law is trustworthy. It follows from the definition of \u0026ldquo;serial\u0026rdquo; and nothing else — no assumptions about the hardware, the scheduler, the memory system, or the workload.\nTotal time on one worker is the serial part plus the parallel part:\n$$T(1) = T_s + T_p$$Give the parallel part $N$ workers and it takes $T_p/N$; the serial part is unchanged, because that is what serial means:\n$$T(N) = T_s + \\frac{T_p}{N}$$Normalise $T(1) = 1$, so $T_s = 1-p$ and $T_p = p$. Speedup is the ratio:\n$$S(N) = \\frac{T(1)}{T(N)} = \\frac{1}{(1-p) + p/N}$$Let $N \\to \\infty$ and the second term vanishes, leaving $1/(1-p)$.\nThe corollary worth memorising. Ask when you reach half of the theoretical ceiling. Set $S(N) = S(\\infty)/2$:\n$$(1-p) + \\frac{p}{N} = 2(1-p) \\quad\\Longrightarrow\\quad \\frac{p}{N} = 1-p \\quad\\Longrightarrow\\quad N = \\frac{p}{1-p}$$At $p = 0.95$ that is $N = 19$. Nineteen cores gets you halfway to the 20× limit — to 10×. Getting from 10× to 15× costs another 38 cores. Getting to 19× is several hundred. The ceiling is not approached; it is asymptoted at, expensively.\nMeasuring $p$ instead of guessing it. The law is only useful if you know the serial fraction, and the profiler\u0026rsquo;s answer is not it. Rearranging $S(N)$ for the serial fraction gives the Karp–Flatt metric: from a single measured speedup $S$ on $N$ workers,\n$$e = \\frac{1/S - 1/N}{1 - 1/N}$$where $e$ is the experimentally determined serial fraction — everything that failed to parallelise, including the coordination costs the profiler never attributed to your loop.\nRun the opening example through it. The profiler said 6% serial, which predicts $S(16) = 1/(0.06 + 0.94/16) = 8.4$ and a 4.8-second run. The measurement was 40 → 17, so $S = 2.35$, and\n$$e = \\frac{1/2.35 - 1/16}{1 - 1/16} = \\frac{0.4255 - 0.0625}{0.9375} = 0.39$$The effective serial fraction is 39%, not 6%. The gap between those two numbers is the finding: thirty-three points of the workload are being spent on something that is not the loop and is not in the flame graph — false sharing, allocator contention, a mutex around a shared accumulator, thread startup. The ceiling is not 16×; it is 2.6×, and the 64-core machine\u0026rsquo;s 16 seconds was a fair approximation of it. Karp–Flatt run at two different $N$ values is more informative still: if $e$ stays flat as $N$ grows, the serial part is genuinely serial; if $e$ climbs, you are paying coordination overhead that will get worse.\nEfficiency makes it starker. At $p = 0.95$ and $N = 64$, $S = 15.4$, so each core delivers 24% of a core. Three quarters of a machine you are paying for, idle by arithmetic.\n$p$ ceiling $S(\\infty)$ $N$ for half the ceiling 0.50 2× 1 0.90 10× 9 0.95 20× 19 0.99 100× 99 0.999 1000× 999 Forbids Linear scaling of a fixed workload. Any nonzero serial fraction bounds speedup by a constant. This kills \u0026ldquo;we will scale it later by adding cores\u0026rdquo; as a plan, unless someone has measured $p$.\nBuying your way out of a serial section. If a request holds a global lock for 2 ms, no amount of parallelism below that lock reduces the 2 ms, and the p99 will not move. The only available action is removing the lock.\nOrder-of-magnitude wins from small $p$. To get 100× from parallelism, at least 99% of the work must parallelise. Most code that has never been measured is nowhere near that, and the measurement usually takes an afternoon.\nInterpreting a scaling test as a scaling projection. Two data points cannot distinguish $p = 0.95$ from $p = 0.99$, and those differ by 5× at the ceiling.\nDoes not forbid It does not say parallelism is not worth it. \u0026ldquo;Amdahl proves there is no point past eight cores\u0026rdquo; is a real thing people say, and it is not a statement about anything. The ceiling is $1/(1-p)$, and $p$ is a property of your program that you can go and measure this afternoon. For $p = 0.99$ the law permits 100×. It forbids nothing until you know the number.\nIt does not apply when the problem grows. This is the largest misreading and it has its own post: Amdahl assumes the workload is fixed. Rendering farms, model training, and nightly ETL do not have a fixed workload — given a bigger machine, they run a bigger problem in the same time, and the serial fraction of that scaled workload is smaller. Gustafson\u0026rsquo;s law (T082) is the same algebra asked the other way round, and it gives a speedup linear in $N$. Which law applies is decided by one question: is the problem size pinned?\nIt does not govern throughput scaling of independent work. A stateless HTTP service behind a load balancer is not one job being split; it is many jobs being run. Doubling the replicas doubles the throughput, and Amdahl has nothing to say until a shared resource — the database, a cache, a coordination service — becomes the serial part. Applying the law to horizontal scaling of independent requests confuses a latency bound with a throughput one.\nIt does not say $p$ is fixed. The most consequential misreading, because it turns a diagnosis into a fatalism. The serial fraction is a property of the implementation, not of the problem. Per-CPU counters in the Linux kernel, RCU replacing reader-writer locks, lock striping in ConcurrentHashMap, and sharded counters in every high-write database are all the same engineering move: someone looked at the serial part and discovered most of it was not essential. Amdahl tells you what $p$ costs you. It says nothing about how small $p$ can be made.\nSuperlinear speedup is not a refutation. Measured speedups above $N$ are real and are usually cache: a working set that does not fit in one core\u0026rsquo;s L2 fits in the aggregate L2 of eight. The law models time, not memory hierarchy, so this is outside its scope rather than against it.\nBoundary Scale the problem instead of the machine. Gustafson (T082), and the HPC distinction between strong scaling (fixed problem, Amdahl) and weak scaling (problem grows with $N$, Gustafson). Amdahl is the optimistic bound. It has no term for coordination, so it can only ever be too generous. The Universal Scalability Law adds a negative term and predicts that past some $N$ the throughput falls — also T082, and the more useful curve for a distributed system. Attack $p$, not $N$. Almost always the higher-return work, and the law is what tells you how much a given reduction is worth before you attempt it. Latency versus throughput. Amdahl is a statement about one job\u0026rsquo;s elapsed time. For a system serving many jobs, Little\u0026rsquo;s law (T083) and the utilisation curve (T084) are the relevant results, and they will bite long before the parallel ceiling does. Amortise the serial part. If the serial section is a fixed setup cost, its fraction shrinks as the parallel work grows — which is, in one sentence, the entire content of the next post. Read next ","permalink":"https://cs.lozic.me/posts/t081-amdahls-law/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eThe profile was unambiguous: 94% of wall-clock time in one loop, and the loop\u0026rsquo;s\niterations are independent. You parallelise it across sixteen cores, the flame\ngraph flattens exactly as predicted, and the end-to-end time goes from 40\nseconds to 17.\u003c/p\u003e","title":"Amdahl's Law"},{"content":"Symptom You fix a typo in an error message. Four days later a customer escalates: their alerting pipeline greps for the old string and has gone quiet.\nOr you make a function faster, and a test suite that had been green for two years starts failing intermittently — not because the function is wrong, but because it used to be slow enough to hide a race.\nOr you tighten a parser to reject a malformed field that the spec never allowed in the first place, and discover that a third of production traffic sends it.\nIn every case the same objection comes back from the team: that was never part of the contract. Correct, and irrelevant. That is the whole content of the two results in this post — one describing what users do, one describing the design habit that hands them more to do it with.\nStatement Hyrum\u0026rsquo;s law. With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviours of your system will be depended on by somebody.\nHyrum Wright, from a decade of trying to change things inside Google\u0026rsquo;s monorepo. It is an empirical claim, not a theorem, and it is explicitly false for small $N$ — that is what \u0026ldquo;sufficient number of users\u0026rdquo; is doing in the sentence.\nPostel\u0026rsquo;s robustness principle. Be conservative in what you do, be liberal in what you accept from others.\nJon Postel, RFC 761, 1980, in the TCP specification. It is a design principle: justified by argument, never proved, and adopted nearly universally by everything built on the internet for the following twenty years.\nThey run together because Postel is the cause and Hyrum is the bill. A lenient implementation accepts inputs its specification does not describe. Accepting them makes them observable behaviour. Hyrum\u0026rsquo;s law then says that behaviour becomes load-bearing for someone. And a behaviour that is load-bearing for someone can no longer be tightened — including back to what the specification actually said. Neither result reads properly alone: Hyrum without Postel sounds like a complaint about users, and Postel without Hyrum sounds like good manners.\nArgument Hyrum\u0026rsquo;s law is empirical, so honesty requires saying what the argument is and is not. There is no proof. There is a mechanism, and it has two moving parts.\nObservability is much wider than documentation. The set of things a caller can detect about your system includes every documented guarantee, and also: iteration order, error message text, which of two equal-priority results comes back first, wall-clock timing, allocation counts, log format, the exact capitalisation of a header, whether a field is absent or null, and whether an operation happens to be atomic today. The contract is a small subset of the observable surface, and nothing enforces the difference.\nUsers are selected, not sampled. If a given user depends on a given observable with some small probability $p$, the chance that no one among $N$ users depends on it is $(1-p)^N$, which goes to zero. With $N$ in the millions, an observable that one user in a hundred thousand would build on is depended on by dozens of people. This is the only quantitative content in the law, and it is back-of-envelope, not a derivation. Its value is that it identifies the variable that matters: $N$. A library with twelve users has slack that a library with twelve million does not, and the same engineering caution is prudent in one case and theatre in the other.\nThe Postel half has a real argument, and it was a good one in 1980. Networks were heterogeneous, implementations were buggy, and a strict receiver would have been an unreachable island. Leniency bought deployability at a moment when deployability was the binding constraint.\nWhat that argument omits is the second-order effect. A lenient receiver does not merely tolerate a malformed sender; it removes the sender\u0026rsquo;s feedback signal. The bug is never found, the behaviour spreads, and the union of what implementations accept becomes the de facto specification — one nobody wrote, nobody can enumerate, and nobody can change. The classic sequence is HTML: browsers competed on tolerance for broken markup for a decade, and HTML5 eventually had to be written by reverse-engineering what they had converged on, then specifying the error handling exactly, byte for byte. The spec did not describe a language; it documented an accident.\nForbids \u0026ldquo;Internal\u0026rdquo; as a technical guarantee. A comment, a leading underscore, or a line in the docs constrains nobody. Only the compiler, the type system, or the runtime constrains people. If a behaviour is reachable, it is part of your API.\nFixing a bug in a widely-used interface at zero cost. With enough users, the buggy behaviour is someone\u0026rsquo;s workaround. This is not hypothetical: Windows has shipped compatibility shims for individual applications for thirty years, including one that preserved a use-after-free that SimCity depended on, because \u0026ldquo;the new Windows broke SimCity\u0026rdquo; is a Windows problem no matter who wrote the bug.\nKeeping an implementation detail an implementation detail. The pressure runs in both directions, and the second direction is the one people miss. CPython 3.6 changed its dictionary layout for memory reasons, and insertion-ordered iteration fell out as a side effect that the release notes explicitly warned against relying on. By 3.7 it was a guarantee in the language specification, binding every implementation of Python forever, because within one release cycle enough code depended on it that the alternative was breaking the ecosystem. Nobody decided to add that feature. It was observed into existence.\nInteroperability through leniency. Postel-liberal implementations do not converge on the specification. They converge on whatever the dominant implementation accepts, which is why \u0026ldquo;works in Chrome\u0026rdquo; became a synonym for \u0026ldquo;correct\u0026rdquo; and why TLS 1.3 had to be redesigned mid-standardisation to look like a TLS 1.2 session on the wire — middleboxes had been liberal about what they accepted for so long that a genuinely new handshake was undeployable.\nDoes not forbid It does not forbid ever changing anything. The most expensive misreading, and the most common: teams that freeze an interface permanently and cite Hyrum. The law is about cost, not impossibility. Python 3, Rust\u0026rsquo;s edition mechanism, and Go\u0026rsquo;s math/rand/v2 are all deliberate breaks that shipped, because each one paid the price knowingly and provided a migration path. Hyrum\u0026rsquo;s law tells you there will be a bill; it does not tell you the bill exceeds the benefit. Deciding that requires the number of affected users, which is measurable.\nIt does not apply equally at every scale. \u0026ldquo;With a sufficient number of users\u0026rdquo; is a real precondition. A five-person internal service adopting Google\u0026rsquo;s deprecation ceremony — a two-year notice period, a compatibility shim, a migration tool — has misapplied a law about $N$ in the millions to an $N$ of three, and has spent a quarter to avoid a Slack message.\nPostel is not simply wrong. RFC 9413 is careful about this, and so should anyone quoting it be. HTTP\u0026rsquo;s tolerance of header case and surrounding whitespace, and browsers\u0026rsquo; willingness to render imperfect markup, made the early web deployable by people who were not protocol engineers, and that was worth more than the strictness would have been. The modern case against Postel is narrower than the slogan: for protocols intended to evolve, leniency ossifies the extension points, and the failure mode is that you cannot ship version two.\nStrictness does not get you out of it. Go randomises map iteration order specifically so that no program can depend on it — an excellent move, and it closes exactly one hole. Programs still depend on timing, on allocation counts, on the text of error values, on goroutine scheduling. Java specified String.hashCode in the language specification because by the time anyone noticed, changing it was already impossible; specifying it was surrender written as a guarantee. Strictness narrows the observable surface. Nothing closes it.\nBoundary The useful moves all follow from the mechanism rather than from the slogans.\nMake it unobservable, not undocumented. Randomising is the strongest tool available: Go\u0026rsquo;s map ordering, address-space layout randomisation, and TLS GREASE (RFC 8701), which sends deliberately unassigned values so that implementations must tolerate the unknown values a future extension will use. GREASE is the direct engineering answer to ossification — exercise the extension points continuously so they cannot rust shut. Reduce $N$. Unstable feature gates, nightly-only APIs, and genuinely private interfaces work because they hold the user count below the threshold where the law bites. Version instead of mutating. Rust editions and protobuf\u0026rsquo;s field-number discipline both let new behaviour coexist with old rather than replacing it. Turn the surprise into telemetry. Chrome\u0026rsquo;s use counters and deprecation reports measure who depends on what before the change ships, which converts Hyrum\u0026rsquo;s law from an ambush into a number. Be strict on receipt, and say so early. The modern protocol position, and it is a reversal of Postel rather than a refinement of it: reject what the spec does not allow, while the population of implementations is still small enough to fix. Read next ","permalink":"https://cs.lozic.me/posts/t091-hyrums-law-and-postels-principle/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eYou fix a typo in an error message. Four days later a customer escalates:\ntheir alerting pipeline greps for the old string and has gone quiet.\u003c/p\u003e\n\u003cp\u003eOr you make a function faster, and a test suite that had been green for two years\nstarts failing intermittently — not because the function is wrong, but because\nit used to be slow enough to hide a race.\u003c/p\u003e","title":"Hyrum's Law and Postel's Principle"},{"content":"Symptom Someone sends you a pitch deck. The claim is a compression algorithm that reduces any file by a guaranteed ratio, and — this is always the tell — that it can be applied repeatedly. Four gigabytes down to one, then one down to two fifty, and so on until the whole film is a few hundred bytes and a clever decoder.\nOr the same thing at a much smaller scale, in your own terminal:\n$ ls -l archive.tar.gz -rw-r--r-- 1 you staff 4823901 archive.tar.gz $ gzip -c archive.tar.gz | wc -c 4823946 Forty-five bytes bigger. Not a bug, not a bad flag, not a pathological input. You have run into a theorem, and it is the same one that kills the pitch deck.\nStatement Call an encoder $C$ lossless if it is injective: distinct inputs get distinct outputs, which is exactly what you need for a decoder to exist.\nIf a lossless compressor makes any input shorter, it must make some other input longer.\nThere is no way to buy the first without paying the second. Not with better modelling, not with more compute, not with a smarter transform.\nThe quantitative version is the one that ends arguments:\nFix a length $n$. Of the $2^n$ inputs of length $n$, fewer than $2^{1-k}$ of them — that is, fewer than one in $2^{k-1}$ — can be mapped by a lossless compressor to an output of length $n - k$ or less.\nFor $k = 11$: fewer than one input in a thousand can be shortened by even eleven bits. For $k = 21$: fewer than one in a million can lose twenty-one bits. Those numbers are not about a particular algorithm. They bound every algorithm that will ever be written.\nArgument This is T001 with the boxes chosen well. Pigeonhole says $n+1$ items in $n$ boxes forces two into one box; here the items are inputs, the boxes are outputs, and two items sharing a box means two inputs sharing an encoding, which means no decoder.\nStep one: something of length $n$ fails to shrink. There are $2^n$ strings of length exactly $n$. There are\n$$2^0 + 2^1 + \\cdots + 2^{n-1} = 2^n - 1$$strings of length less than $n$. If every length-$n$ input compressed to something shorter, we would be mapping $2^n$ items into $2^n - 1$ boxes. Pigeonhole: two collide, injectivity fails, the decoder cannot exist. So at least one string of each length survives uncompressed.\nStep two: something actually grows. Step one leaves an escape hatch — maybe a compressor shrinks some inputs and leaves the rest exactly as they are, never growing anything. Close it.\nSuppose $C$ is injective and $|C(x)| \\le |x|$ for every $x$. Let $A_n$ be the set of all strings of length at most $n$, so $|A_n| = 2^{n+1} - 1$. Since $C$ never lengthens, $C$ maps $A_n$ into $A_n$; since $C$ is injective and $A_n$ is finite, $C$ is a bijection on $A_n$. This holds for every $n$.\nNow suppose $C$ shrinks something: $|C(x_0)| \u0026lt; |x_0| = k$ for some $x_0$. Then $C(x_0) \\in A_{k-1}$. But $C$ is already a bijection on $A_{k-1}$, so every string in $A_{k-1}$ is the image of some string in $A_{k-1}$ — and $x_0 \\notin A_{k-1}$. Two inputs, one output. Injectivity fails.\nSo a lossless compressor that never grows anything cannot shrink anything either: it is a permutation of each length class, which is to say it is not a compressor. Shrinking and growing come as a pair.\nStep three: the counting bound. Outputs of length $n-k$ or less number $2^{n-k+1} - 1$. Inputs of length $n$ number $2^n$. Injectivity means at most $2^{n-k+1} - 1$ of them can land there, so the fraction is under $2^{n-k+1}/2^n = 2^{1-k}$. That is the boxed claim, and it took one division.\nNothing in any of this inspects the algorithm. That is the point, and it is why the argument is worth ten minutes: you can refute a compression claim without reading a line of its source.\nForbids Recursive compression. If a compressor shrank every input, you could feed its output back in and iterate down to a single bit, and $2^{4,000,000,000}$ distinct files would all decode from one of two starting points. In 1992, WEB Technologies announced DataFiles/16, claiming 16:1 on any input and repeatability down to about 1024 bytes. The product never shipped. It could not have.\nAny claim containing the words \u0026ldquo;including random data\u0026rdquo;. Random data is precisely the input class the counting bound is about. A compressor that shrinks a uniformly random $n$-bit string with probability better than $2^{1-k}$ for a $k$-bit saving does not exist, and the claim can be rejected before the demo.\nMeaningful gains on already-compressed bytes. The forty-five extra bytes at the top of this post are not gzip failing. A well-built compressor detects that it is losing and falls back to storing the data verbatim, and the overhead is the frame around that decision.\nUniversally beating the current format. \u0026ldquo;Our codec is smaller than PNG on every image\u0026rdquo; is a claim about all inputs, and all inputs includes noise. What is achievable, and worth a great deal of money, is being smaller on the images people actually have.\nDoes not forbid It does not forbid compression from working. This is the misreading that makes people dismiss the theorem as a curiosity, and it is worth being precise about why they are wrong to. zstd on a directory of JSON logs routinely returns 10:1. That is not a violation and not luck. The theorem says the wins are paid for by losses on other inputs — and those other inputs are overwhelmingly high-entropy strings that no one has ever wanted to store. Real data lives in a vanishingly small corner of the input space, and every practical compressor is a bet that your file is in that corner. The bet is close to free because the losing side of it is a set you will never draw from.\nIt does not forbid a bounded worst case. A common overcorrection: \u0026ldquo;so any compressor might blow my file up.\u0026rdquo; No. DEFLATE has a stored-block type, so gzip on incompressible input costs about five bytes per 64 KB block plus a small header; zstd and xz do the same thing. The theorem guarantees some input grows. It says nothing about how much, and every serious format caps the damage at a fraction of a percent by refusing to compress when compression is losing. The growth is real, mandatory, and negligible.\nIt does not forbid lossy codecs from doing much better. JPEG, Opus and AV1 are not counterexamples and not cheating. They are not injective — many distinct inputs decode to the same output, which is the entire deal — so they are simply outside the theorem\u0026rsquo;s scope. Anyone citing MP3 as a refutation has misread which word in \u0026ldquo;lossless compressor\u0026rdquo; is load-bearing.\nIt does not say general-purpose compressors are near optimal. This one costs teams real money. \u0026ldquo;Information theory says you can\u0026rsquo;t do much better than gzip\u0026rdquo; is false, and the theorem being invoked says the opposite: since all the gains come from matching the input distribution, a compressor that knows your distribution can beat a general one by a lot. FLAC beats xz on audio. Parquet with dictionary and run-length encoding beats gzipped CSV by an order of magnitude on the same table. Brotli ships a built-in dictionary of common web strings because it knows what it will be asked to compress. Choosing the input distribution is the whole game, and the theorem is what tells you that.\nIt does not cover a corpus. Deduplicating storage — Git packfiles, restic, ZFS — often reports ratios that sound impossible. They are not compressing one string; they are noticing that the same block arrived a thousand times. The theorem is about a single injective map on single inputs, and says nothing about exploiting repetition across inputs.\nBoundary The bound is tight and there is no clever encoding that evades it. What lives just outside:\nDrop injectivity. Lossy compression leaves the theorem\u0026rsquo;s hypotheses entirely, which is why it can offer guaranteed ratios and why it is what you use for anything perceptual. Fix a distribution. Once inputs are drawn from a known source rather than adversarially, the question changes from \u0026ldquo;which inputs shrink\u0026rdquo; to \u0026ldquo;what is the expected length\u0026rdquo;, and the answer is exact: Shannon entropy, in T018. This theorem says a universal win is impossible; entropy says exactly how large the non-universal win is. Move bits out of band. Shared dictionaries — Brotli\u0026rsquo;s, zstd --patch-from, HTTP/2\u0026rsquo;s HPACK table — genuinely shrink the transmitted bytes. The cost has not vanished; it has been paid once, in advance, by both sides. The accounting still balances. Ask about a specific string instead of all of them. The shortest program that outputs a given string is its Kolmogorov complexity (T023). It gives the exact incompressibility of this file rather than a bound over all files, and the price is that it is uncomputable. Amortize across a corpus. Delta encoding and content-addressed deduplication change the unit of work from one string to a collection, which is a different problem with different limits. Read next ","permalink":"https://cs.lozic.me/posts/t004-no-universal-lossless-compressor/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eSomeone sends you a pitch deck. The claim is a compression algorithm that\nreduces \u003cem\u003eany\u003c/em\u003e file by a guaranteed ratio, and — this is always the tell — that\nit can be applied repeatedly. Four gigabytes down to one, then one down to two\nfifty, and so on until the whole film is a few hundred bytes and a clever\ndecoder.\u003c/p\u003e","title":"No Universal Lossless Compressor"},{"content":"Symptom You are asked to build a cache key. You have a 64-bit hash, and someone on the team says: \u0026ldquo;collisions are basically impossible, there are eighteen quintillion values.\u0026rdquo;\nOr: your service assigns short IDs to uploads, six characters of base-36, and you are wondering when you need to worry.\nOr, the one that shows up in a bug report rather than a design doc: a compression routine has been running happily for two years, and then a customer file comes back from the round trip one byte longer than it went in, and nobody can find the bug. There is no bug.\nAll three are the same question, and the answer is not a matter of engineering judgment. It is a two-line proof from 1834.\nStatement If you put $n+1$ items into $n$ boxes, some box contains at least two items.\nThat is the whole thing. The generalized form is barely longer, and is the one you will actually use:\nIf $m$ items are placed into $n$ boxes, some box contains at least $\\lceil m/n \\rceil$ items.\nNote what the statement does not mention: which items, which boxes, or any procedure for finding the collision. It is an existence claim, and it is the cheapest one in mathematics.\nArgument Suppose not. Suppose every box holds at most one item. There are $n$ boxes, so there are at most $n \\cdot 1 = n$ items in total. But we placed $n+1$ items. And $n+1 \u0026gt; n$. Contradiction.\nFor the generalized form, run the same argument with a different bound: if every box held at most $\\lceil m/n \\rceil - 1$ items, the total would be at most\n$$n \\left( \\left\\lceil \\frac{m}{n} \\right\\rceil - 1 \\right) \u003c n \\cdot \\frac{m}{n} = m$$which is fewer than the $m$ items we started with. Contradiction again.\nThe strict inequality in the middle is the only step that needs care, and it holds because $\\lceil x \\rceil - 1 \u0026lt; x$ for every real $x$.\nThat is a complete proof. It is worth pausing on how little it used: no structure on the items, no assumption about how they were placed, no randomness, no limit. Counting two ways and noticing the two counts disagree. Most of the impossibility results in this series are this argument wearing a heavier coat.\nForbids A lossless compressor that shrinks every input. Consider all $2^n$ inputs of length exactly $n$ bits. Suppose your compressor maps each to a shorter string. There are only $2^0 + 2^1 + \\dots + 2^{n-1} = 2^n - 1$ strings shorter than $n$ bits. So $2^n$ inputs must fit into $2^n - 1$ output slots, and two inputs collide, and a compressor that maps two inputs to the same output is not lossless. Every \u0026ldquo;compresses any data by 30%\u0026rdquo; claim dies here, permanently, with no need to inspect the algorithm.\nA hash function with no collisions, once the input space is bigger than the output space. A 64-bit hash of arbitrary-length strings has infinitely many inputs and $2^{64}$ outputs. Collisions do not merely happen; they are guaranteed to exist in unlimited supply.\nTwo people in a room of 366 with distinct birthdays. Boxes are days, items are people. (How quickly collisions arrive once you allow randomness is a very different and much more surprising question, and it gets its own post.)\nDoes not forbid Here is where the principle gets misused, and the misuse is common enough to have a shape.\nIt does not say collisions are likely. This is the one that matters commercially. \u0026ldquo;Pigeonhole proves 64-bit hashes collide, therefore we need 256 bits for our cache keys\u0026rdquo; is an argument that has cost real teams real memory. The principle is an existence result over the whole input space: it says a colliding pair exists somewhere among all possible inputs. It says nothing about whether your particular ten million inputs will contain one. That question needs a probability, and the answer is roughly $m^2/2N$ for $m$ items and $N$ slots — the birthday bound, which is a genuinely different theorem and which gives much scarier numbers than most people expect. Pigeonhole is not the reason to size your hash. It is just the reason you cannot claim zero.\nIt does not forbid compression that works. The zip file on your disk is not a counterexample, and it is not cheating. General-purpose compressors shrink structured inputs and pay for it by lengthening some others, almost all of which are high-entropy strings nobody ever wants to store. The theorem forbids a universal win; it permits an enormously profitable trade, and the entire field of data compression lives in that gap.\nIt does not forbid perfect hashing. This trips people up. If you have a fixed, known set of $k$ keys and you build a table of $n \\ge k$ slots, a collision-free hash function exists and there are standard constructions that find one. No contradiction: pigeonhole only bites when the items outnumber the boxes. Perfect hashing simply refuses to be in that regime, which is exactly why compilers and CDN routers use it for fixed keyword sets.\nIt does not forbid UUIDs from being fine. Version-4 UUIDs have 122 random bits. Pigeonhole guarantees two UUIDs collide somewhere in the space of all possible UUIDs, which is a true and completely useless statement about a space you will never enumerate.\nThe general shape: pigeonhole tells you a thing exists, and people read it as telling them a thing is probable, or as telling them a design is unsound. It does neither. It draws a hard line and says nothing whatsoever about how close to the line you are standing.\nBoundary The principle is tight in the strongest possible sense: with $n$ items and $n$ boxes there may well be no collision, so $n+1$ is exactly the threshold. There is no slack to exploit.\nWhat lives just outside it:\nProbabilistic pigeonhole. Ask not \u0026ldquo;does a collision exist\u0026rdquo; but \u0026ldquo;how many items before one is likely\u0026rdquo;, and you get the birthday bound and a $\\sqrt{N}$ answer instead of an $N$ one. This is the single most practically important refinement, and it is the reason 64-bit IDs are riskier than intuition suggests. Counting with weights. Replace \u0026ldquo;at least two items in a box\u0026rdquo; with averaging: if the mean is 5, some box holds at least 5. That is linearity of expectation, and it is pigeonhole with the integrality relaxed. Changing what a box is. Most clever uses of this principle are not clever about the proof, which never changes. They are clever about choosing the boxes. The next post takes the compression argument above and pushes it to its full strength — no lossless compressor can shrink anything without growing something else — and the decision-tree bound for comparison sorting (T003) is the same principle applied to boxes you would not have thought of. Read next ","permalink":"https://cs.lozic.me/posts/t001-pigeonhole-principle/","summary":"\u003ch2 id=\"symptom\"\u003eSymptom\u003c/h2\u003e\n\u003cp\u003eYou are asked to build a cache key. You have a 64-bit hash, and someone on the\nteam says: \u0026ldquo;collisions are basically impossible, there are eighteen quintillion\nvalues.\u0026rdquo;\u003c/p\u003e\n\u003cp\u003eOr: your service assigns short IDs to uploads, six characters of base-36, and\nyou are wondering when you need to worry.\u003c/p\u003e","title":"The Pigeonhole Principle"},{"content":"","permalink":"https://cs.lozic.me/index-of-results/","summary":"Every result in the series, in book order.","title":"Index of results"}]