Three Ways to Break It
Description
An idea you only ever defend is a belief, not a tool. This episode states the strongest case against criterion sortition — three attacks, hardest first. Attack one, meta-capture: someone has to decide what's on the menu and someone has to run the random draw, and whoever controls those is the new gatekeeper, so the capture has been relocated, not removed. This is real and does not vanish; Gibbard already told us manipulation can't be eliminated. The honest answer is containment, not denial — keep that control thin, public, audited, and held by a separate body that can't rewrite the menu for its own benefit, shrinking the captureable surface from a whole class to one visible procedure. Attack two, measurability — the one that decides everything: the menu only works if these capacities can actually be measured. For some, surprisingly, yes: Tetlock's forecasting research found good judgment is real, measurable, stable from year to year (a correlation around 0.65), and trainable in a way that improves real accuracy, not just the score. But that is the easy case. Wisdom is measurable too, except it swings with the situation, so it needs many readings rather than one. And some capacities — integrity above all — may not be reliably measurable at all; if a capacity can't be measured honestly, it can't be on the menu, so the menu may be shorter than hoped. Attack three, correlated gaming: the moving target works only if the items are different enough that no single preparation covers them all; if one regime — a school, a coach, an AI — could lift many at once, the target stops moving in any way that matters. Whether that's possible is measurable, per menu, and there's a twist: if the regime works by making people genuinely more capable across the board, that's the system succeeding, not failing. Put together, the three give an honest scorecard, and every load-bearing claim now points at the same thing: not more argument, but evidence.
- Gibbard's theorem — Wikipediahttps://en.wikipedia.org/wiki/Gibbard%27s_theorem
- The Good Judgment Project — Wikipediahttps://en.wikipedia.org/wiki/The_Good_Judgment_Project
- The New Science of Practical Wisdom (Grossmann et al.) — PMChttps://pmc.ncbi.nlm.nih.gov/articles/PMC7138215/
- Improving Information from Manipulable Data — Frankel & Kartik, JEEAhttps://academic.oup.com/jeea/article-abstract/20/1/79/6255435
Script
Cold open
An idea you only ever defend is a belief, not a tool. So here is the case against the thing we just built — the three strongest ways it fails. Two of them might be survivable. One of them is the whole ballgame, and it has not been settled.
Frame
We have a mechanism: a fixed menu of merits, a random draw of which ones count, a threshold, a lottery. If it's real, it survives its best objections. If it only survives its weak ones, it's an essay, not a proposal. Three attacks — hardest first.
Who decides what's on the menu and runs the draw — haven't you just moved the capture?
Attack one. Someone has to decide what goes on the menu, and someone has to run the random draw. Whoever controls those two things is the new gatekeeper — which means we've moved the capture, not removed it. This is real, and it does not go away. Gibbard already told us manipulation can't be eliminated, only contained. So the honest answer isn't denial — it's containment. Keep that control thin, public, audited, and in the hands of a separate body that can't rewrite the menu to suit itself. The surface you can capture shrinks from a whole class down to one visible procedure. Smaller. Not gone.
Can these capacities — judgment, integrity — actually be measured at all?
Attack two — and this is the one that decides everything. The menu only works if these capacities can actually be measured. Can they? For some, surprisingly, yes. Philip Tetlock's forecasting tournaments found that good judgment is real, that it's measurable, that it holds steady from one year to the next — and, the crucial part, that training which raises the score raises real accuracy with it, not just the score. But that's the easy case. Wisdom can be measured too — except it swings with the situation, so you need many readings, not one. And some things — integrity, above all — may not be measurable honestly at all. If a capacity can't be measured, it can't be on the menu. The menu may be shorter than we'd like.
What if one regime could game many menu items at once?
Attack three. The moving target only works because the items are different enough that no single preparation covers them all. So what if one regime — a school, a coach, an AI — could lift many of them at once? Then the target stops moving in any way that matters. Whether that's possible isn't a matter of opinion; it's measurable, menu by menu. And notice the twist. If that one regime works by making people genuinely more capable across the board — that isn't an attack. That's the system doing its job. The only thing to fear is a regime that fakes them all at once.
Turn
Put the three together and you get an honest scorecard. The mechanism does not eliminate capture — Gibbard settled that — it shrinks it and contains it. It depends, completely, on whether the merits can be measured, and that runs from 'yes, clearly' for forecasting all the way to 'probably not' for integrity. And its central defense holds only if the menu can't be gamed as a single block, which nobody has tested. This is not a finished answer. It's a specific bet, with named ways to lose. And every load-bearing claim has now arrived at the same place. Not more argument. Evidence.
Closer
That is the uncomfortable shape of an idea actually worth taking seriously: it tells you exactly what would prove it wrong. So there is only one honest thing left to do with it. Stop defending it on paper, and go find out. Which turns out to be a single, concrete experiment — cheap, runnable now, and built to kill the idea early if it deserves to die.