Grading analysts on process hands them an alibi that slows learning

When people are judged on whether they reasoned by the book rather than whether they got it right, following the book becomes the safe move and being wrong stops teaching anything. A wrong answer reached through approved process is not a failure, it is an excused absence.

Sherman Kent supplied the canonical version of the alibi. His September 1962 estimate judged the Soviets would not put offensive missiles in Cuba, because doing so ran against their own interests. Weeks later the missiles were photographed. Kent conceded the board "came down on the wrong side," then argued the analysis was sound and Khrushchev had simply made a dramatically wrong decision no estimating process could divine. Wrong for the right reasons. The defense only exists if estimates are graded on reasoning, and notice it never has to specify what evidence would have counted as the process failing.

Decades later, forecasting tournaments tested the two grading regimes directly. Forecasters held accountable for process conformed to best-practice guidelines and could always claim they did all that current practice allowed. Forecasters held accountable for accuracy alone experimented, noticed faster when their mental models stopped working, and ended up more accurate. The shield was the mechanism: protected from outcomes, process-graded forecasters had no reason to look past the rules, the same way designing a method to counter a bias is not evidence it counters the bias.

The regime you grade under decides whether being wrong is information or just bad luck.

Source claim: Evaluating analysts on the quality of their reasoning rather than the accuracy of their results makes conformity the safe strategy and strips failure of its power to teach.