The uncomfortable question behind every hiring decision right now is what a person adds when the tools are this good. We have argued before that judgment is the part AI does not replace; hiring is where that argument stops being theoretical. If a model can produce a working implementation of most things in minutes, what exactly are you paying someone for?
We have had to answer that concretely, because we run internship challenges and we have to decide who moves forward. Figuring out how to hire developers in the AI era changed what we look for more than it changed the process itself.
Why finding the answer is no longer the skill
The old proxy for competence was resourcefulness: can this person locate a solution and adapt it. That was a genuinely good filter for a long time, because locating a solution was hard. Knowing which search to run, which answer was outdated, which library was abandoned — that was real skill.
That filter is now broken, and not slightly. Anyone can retrieve a competent-looking answer to almost any well-specified problem. If your interview tests whether a candidate can produce working code for a defined task, you are testing something that no longer distinguishes anybody, and the candidates who score best may simply be the ones fastest at prompting.
The failure mode this produces is specific and we have seen it: someone who ships a working solution quickly and cannot tell you why it works, which parts of it are load-bearing, or what would break if the requirements changed slightly. The code passes. The person cannot maintain it.
What we test instead
We changed the shape of the challenge rather than banning the tools. Telling candidates not to use AI is both unenforceable and dishonest, since it is not how the job works. Our challenges assume they will use everything available.
The brief is deliberately underspecified. We leave real decisions undecided — an edge case that is not mentioned, a requirement that could be read two ways. What we are watching for is whether the candidate notices the ambiguity at all. The strongest ones come back with questions before writing anything. The weakest pick an interpretation silently and never mention it, which tells us they did not see a choice being made.
We ask them to defend a decision they did not have to make. After submission we pick something in their solution and ask why it is that way and what the alternative would have cost. It does not matter whether they wrote that line or generated it. What matters is whether they can reconstruct the reasoning. A person who understands what they submitted can always do this. A person who accepted output cannot, and the gap shows up within two questions.
We change the requirement mid-conversation. A small, realistic change — the kind a client sends on a Friday. Watching someone reason about what their own design makes easy or hard is the closest thing we have found to watching them work.
None of this is about catching anyone. It is about testing the layer above the code, because that layer is what we are actually hiring.
Why this holds up better over time
The practical argument for testing judgment over output is that judgment transfers and specific knowledge does not. Frameworks change. The tools we use today will look primitive in three years. Someone hired for knowing a current stack has to be re-hired, effectively, every time the stack turns over.
Someone who reasons well about trade-offs stays useful across all of it, because what they are good at is not the technology. It is noticing when a problem is underspecified, recognizing when the obvious answer is the wrong one, and being able to explain a decision well enough that someone else can disagree with it productively.
There is also a compounding effect on a small team. When four people can each be trusted to spot a bad requirement before building against it, the whole team moves faster than a larger one where every decision routes through a single person. That only works if judgment was the hiring criterion.
What it does to the culture
The side effect we did not plan for is what this selects out. People who want a fully specified ticket and no ambiguity do not enjoy our challenge, and they self-select away early. That is a real loss in some contexts — plenty of good work is well-specified work — but for a studio this size, where anyone might be the only person looking at a problem for a week, someone who needs the problem pre-solved is going to be blocked constantly.
What is left is a team where the default response to an unclear brief is to ask about it instead of guessing. That is not a hiring outcome so much as a cultural one, and it started with deciding that the thing worth testing was never whether someone could find an answer.