A company interviews a senior engineer and walks them through the usual tests, the kind of coding and design problems that have been standard for a decade. The candidate does fine on most of them and stumbles on one, and the company passes. Six months later a different candidate, who aced every test, has shipped nothing of value and is quietly managed out.
The tests are still testing for what an IC used to do when the heavy work of the role was done by the human. A senior engineer wrote the code and carried the whole chain of reasoning themselves, and the tests were built to check whether a candidate could do each part of that chain. The chain has changed, because a model now does a big part of the code writing and a reasonable amount of the reasoning. The work an IC does now is mostly around the model: reading what it wrote and catching the places where its output looks right but isn’t.
Judgment about output is the skill that matters most in that work. A model produces confident code that passes every test and is wrong in the fourth line. A model produces a design that reads cleanly and misses the constraint a senior engineer would have caught in ten seconds. The IC who catches both is worth a lot more than the one who only catches the first, and the old tests can’t tell them apart.
Judgment is harder to test for in a short interview than syntax was. You can give a candidate a small function and see whether they can write it. You can’t easily give them a plausible output and see whether they spot the thing wrong with it, because the candidate who saw something like it last week will spot it, and the candidate who didn’t won’t, and neither result tells you much about whether they’d spot the next one.
The hiring process has to do the work of testing for the new skill directly, because the old proxies have broken. One way to do this is to give a candidate a real piece of model output with a real flaw in it, and watch how they work through it. The things to watch are what they ask and how long it takes them to see the flaw. The interview stops testing whether they can write the first version of the work and starts testing whether they can review it.
Most companies haven’t made this change yet, which is part of why so many hires made in the last two years have underperformed. The interview tested for the old shape of the role, and the job is the new shape. The problem doesn’t look like a hiring problem because it shows up months later as a performance problem, and the performance problem looks like a bad hire rather than a stale screen.
Hiring processes have to be rebuilt every few years rather than tuned at the edges. The shape of IC work has changed more in the last two years than in the previous ten, and the screen that was catching the right candidates in 2022 is catching the wrong ones now. The companies that notice early hire the people who can work around a model, and the companies that don’t keep paying for screens that measure the old bar.