Hiring for production AI — what to actually screen for

The AI hiring market is broken

Everyone with a Coursera certificate is calling themselves an AI engineer. Everyone with three blog posts about RAG is a senior. Pay bands have inflated 30-50% in 18 months. And most of the candidates can't ship anything past a Jupyter notebook to production.

Here's what to actually screen for if you're hiring AI engineers in 2026.

Resume signals that don't matter

• Number of courses completed. Anyone can finish a course. The market is flooded.
• Big-co AI team affiliation. Some of the worst AI engineers we've interviewed came from FAANG AI orgs — they shipped one feature into a stack of 50 and called themselves senior.
• GitHub stars on a notebook repo. Notebooks aren't production. Stars are vanity.
• A Hugging Face Space they made. Same as above. Toy.
• Number of LLM-related blog posts. Sometimes correlates with thinking. Often correlates with marketing.

Resume signals that do matter

• Production traffic numbers. "I shipped a feature handling X requests/day with Y latency target" — they have specifics.
• Eval suite mentions. Anyone who's shipped production AI has fought an eval suite into existence. They'll bring it up unprompted.
• Cost-aware design. They'll describe how they optimized for token cost as well as quality.
• A failure they own. "I shipped this and it broke, here's what we did" — the most reliable senior signal.

Interview questions that actually work

1. "Tell me about an eval suite you built. How did it evolve over the first 90 days?"

Strong candidates describe specific cases that got added, specific regressions caught, and specific things they decided not to score. Weak candidates describe "we wrote tests."

2. "Walk me through the cost trade-off on a recent prompt change."

Senior AI engineers think about token cost the way senior backend engineers think about query cost. They have a number per conversation, an opinion about where it's going, and a plan for when it crosses a threshold.

3. "Show me a refusal pattern you wrote."

When the agent shouldn't answer. Strong candidates have a library of these — they're production scars. Weak candidates think refusal is something the model decides on its own.

4. "Describe an agent action you decided not to ship to auto-execute."

The strong signal: a real example of a decision deferred to a human, with the reasoning. Senior engineers earn auto-execute trust — they don't start with it.

5. "What's your default first-call architecture for a customer-support chatbot?"

A passable answer covers RAG with hybrid search + reranking, citation-required prompting, refusal patterns, eval suite seeded at 20 cases, escalation queue. A great answer mentions the gotchas (chunk boundaries, citation hallucination, multi-turn state).

The take-home that actually predicts performance

Whiteboard exercises tell you very little about production AI work, and most take-homes for these roles are either a toy RAG build or a Kaggle-flavoured notebook. Neither one separates candidates, because the strong and weak both finish them.

The version that discriminates: give them a broken system and ask them to diagnose it.

Hand over a small RAG service that answers badly, with a corpus, twenty example failures, and the prompt. Ask for a written diagnosis and a proposed fix order — not a rewrite. Budget it at two hours and mean it.

What this surfaces that a build task cannot:

Do they separate retrieval failure from generation failure? The strong candidate checks whether the right document was retrieved before touching the prompt. The weak one starts rewriting the system prompt immediately, because that is the part that looks like AI work.

Do they ask about the eval before proposing the fix? Senior candidates want to know how anyone would tell whether the change helped. Junior ones propose confidently and unfalsifiably.

Do they prioritise? Twenty failures rarely have one cause. A ranked list with reasoning is the signal; a list of every possible improvement is the opposite of one.

Do they say what they would not do? "I would not fine-tune here, because the failures are retrieval-side" is the single strongest sentence a candidate can write on this exercise.

It also respects their time, which matters in a market this competitive. Two hours of diagnosis tells you more than a weekend of building, and good candidates notice which companies understand that.

What we screen out

• Candidates who can't reproduce their own results without a notebook.
• Anyone who's never owned a model in production for 6+ months.
• Engineers who haven't argued with a PM about a refusal pattern.
• Anyone who describes LLM development as "configuring prompts."

Interviewing when you're not an AI engineer yourself

Most people hiring for these roles cannot personally evaluate the technical depth of the answers, and pretending otherwise produces bad hires in both directions. You do not need to be able to build the system to interview well for it.

Ask for the decision, then ask for the alternative. "Why that approach?" followed by "what would have made you choose differently?" A candidate who has genuinely made the decision can describe the fork. One who has read about it cannot, and the second question is where that shows.

Listen for numbers, unprompted. Latency, cost per conversation, deflection rate, eval score. People who have run something in production reach for numbers naturally; people who have not speak entirely in qualities.

Listen for the failure being owned rather than narrated. "The retrieval was bad" is narration. "I chunked on token count and split the size tables, so the model got half the answer" is ownership, and it is checkable even by a non-specialist because it is specific.

Ask what they would need from you. Strong candidates ask about data access, who owns the corpus, and whether there is an eval suite, because they know those determine whether the job is possible. Candidates who ask nothing about the operating conditions have not thought about doing the work, only about getting the offer.

Bring a practitioner for one round, not all of them. A single 45-minute technical conversation with someone who has shipped production AI — a contractor, an advisor, a friendly peer — catches almost everything a non-specialist panel misses. It is the cheapest correction available.

The seniority signal that compounds

Senior AI engineers think in production from sentence one. They mention cost, latency, observability, eval suites, refusal patterns, and audit trails without being asked. They've shipped, broken things, fixed them, and have scars.

If your candidate doesn't talk like that, they're not senior, regardless of their title.

Growing the people you already have

The market rates below are the reason most companies cannot hire their way to a team, and the response that consistently works is to convert existing engineers rather than compete for scarce ones.

A strong backend engineer is most of the way there. Production AI work is overwhelmingly ordinary engineering — APIs, queues, retries, observability, cost control — with a probabilistic component bolted into the middle. The genuinely new skills are narrow: evaluation, retrieval design, prompt and context engineering, and knowing where the model's failure modes are. That is weeks of deliberate learning for someone who already ships, not years.

The conversion path that works. Pair them with someone experienced on one real production system, not a course. Give them the eval suite as their first owned artefact, because it forces them to define what "correct" means before they can improve anything. Then let them own one narrow agent end to end, including the on-call for it.

What does not work. Sending a team on a course and expecting production capability, or hiring one senior AI engineer into a team of ten who cannot review their work — that person becomes a single point of failure and usually leaves within a year, because nobody can challenge or support them.

The honest limit. Converting engineers takes a quarter or two before they are independently productive, which does not help if you need something shipped this month. That is the gap a delivery partner fills, and the sensible version is a partner who builds while your people learn on the same codebase, rather than one who builds and leaves.

Compensation reality

US market for senior AI engineers is $250-400K base + equity at well-funded startups. Bay Area FAANG-equivalents go higher. For most companies, that's tough to compete with.

Two options that work: pay above-market for a small in-house team (2-3 people, $300K+ each), or partner with a delivery agency where senior engineering hours are $25-150/hour depending on geography. Either is fine. Hybrid is often the best answer.

When not to hire at all

The most useful thing this post can say is that a chunk of the companies running this search should not be running it yet.

If you have no production use case, you have a research hire and a research budget. Hiring a senior AI engineer to "figure out where AI fits" is expensive discovery. The same money spent on a scoped pilot with a defined success metric answers the question faster and leaves you with a working thing either way.

If your data is not accessible, an AI hire will spend six months on data engineering. Which is fine if you meant to hire a data engineer, and demoralising for everyone if you did not. Check what a new engineer could actually reach in week one before you write the job description.

If nobody internal can own the system after launch, do not build it. This is the same point as the eval-ownership question, one level up. A system with no owner degrades regardless of who built it, and hiring the builder does not create the owner.

If the honest volume is one project a year, hire nobody. A full-time senior AI engineer needs a pipeline of work to stay engaged and to stay sharp. One project a year is a contract or a partnership; making it a permanent role produces an under-used expensive person who leaves for somewhere with more interesting problems.

None of that is an argument against building an in-house team. It is an argument for building one when there is enough work to keep it, and for being honest about which of the four situations above you are actually in.

The first ninety days of an AI hire

Hiring well and then onboarding badly is a common and expensive combination, and AI roles fail here more than most because the work depends on access that takes weeks to arrange.

Week one should not be a project. It should be access: the data, the corpus, the model provider account, the observability stack, and whatever internal system the eventual agent will read from. Sorting this out before they arrive is the single highest-return preparation available, and it is routinely left until their first Monday.

Their first owned artefact should be the eval suite, not a feature. It forces them to learn what the system is supposed to do, exposes them to the real failure history, and produces something the team needs regardless. It is also the fastest way for you to assess whether the hire was right, because a good eval suite is legible to people who cannot review model code.

Give them one narrow thing to own end to end by week six — including the on-call for it. Ownership of a small live system teaches more in a month than a quarter of feature work inside someone else's design.

Expect the first real opinion around week ten. A good hire will tell you something uncomfortable about the existing setup by then. If nobody has heard a strong opinion from them after three months, either the access is still blocking them or the hire is not what you thought.

Trying to hire 8 cheap "AI engineers" in 6 months is the worst answer. You'll get 8 people who can't ship.