Sierra Engineer Explains Redesigned AI-Native Interview Process
Vijay Iyengar, who helped create Sierra's interview process, responds to points in a post by @jdpruettt on hiring experiments when candidates have capable agents. He says Sierra grades product judgment and system understanding via a two-hour prototype, scope decisions, production thinking, short-answer questions and a debugging interview with a semantic code search skill.
Original post · 2 min read
I helped create Sierra's interview process (sierra.ai/blog/the-ai-native-interview), so wanted to reply to some of the points here, which I largely agree with!
Models ship end-to-end demos trivially. What’s the point of testing for this?
We want to evaluate product judgment and system understanding. It’s hard to do this in English, better when you have a real product to look at. The prototype a candidate builds over 2 hours is more a rich visual aid for discussion than the artifact we’re grading. We also find a lot of signal in what scope candidates decide to keep or cut within the time box. The best candidates focus on what makes the product great instead of boilerplate that is trivial to add later. Finally, we spend a good portion of time on how they would take this system to production. We look for whether they really understand what makes the problem nuanced in a real-world setting vs the traditional FAANG style system design where you name-drop consistent hashing and Redis pub/sub to pass.
Short-answer questions
This is something we don’t do in general, outside of certain specialized roles, but I agree it’s valuable. You get a lot of signal about a candidate’s depth in a particular area by whether they can grok and answer questions quickly and concisely. Curious how well this works for generalists vs specialists.
Unassisted code review and tweaking
I really like this idea. We introduced a debugging interview where candidates are given an unfamiliar codebase and have to find/fix a bug in it. We do allow for AI assistance through a skill that doesn’t reveal the problems directly, but acts like a “semantic code search”. We find this to be a happy medium and reasonably representative of the real-world, where you’re just not going to interact with code without an AI anymore.
Doing things that don’t standardize
I very much agree with JD here. The main benefit of Leetcode interviews is that they’re easy to standardize. But imo, you shouldn’t be hiring engineers as quickly anymore, which means it’s worth sacrificing standardization for higher-signal. Of course you want to avoid bias, but I think it’s worth pushing on “what would we do if we didn’t have to standardize?” and questioning whether you really need to double or triple headcount. The three problems with graduated time constraints is a great approach.
Models ship end-to-end demos trivially. What’s the point of testing for this?
We want to evaluate product judgment and system understanding. It’s hard to do this in English, better when you have a real product to look at. The prototype a candidate builds over 2 hours is more a rich visual aid for discussion than the artifact we’re grading. We also find a lot of signal in what scope candidates decide to keep or cut within the time box. The best candidates focus on what makes the product great instead of boilerplate that is trivial to add later. Finally, we spend a good portion of time on how they would take this system to production. We look for whether they really understand what makes the problem nuanced in a real-world setting vs the traditional FAANG style system design where you name-drop consistent hashing and Redis pub/sub to pass.
Short-answer questions
This is something we don’t do in general, outside of certain specialized roles, but I agree it’s valuable. You get a lot of signal about a candidate’s depth in a particular area by whether they can grok and answer questions quickly and concisely. Curious how well this works for generalists vs specialists.
Unassisted code review and tweaking
I really like this idea. We introduced a debugging interview where candidates are given an unfamiliar codebase and have to find/fix a bug in it. We do allow for AI assistance through a skill that doesn’t reveal the problems directly, but acts like a “semantic code search”. We find this to be a happy medium and reasonably representative of the real-world, where you’re just not going to interact with code without an AI anymore.
Doing things that don’t standardize
I very much agree with JD here. The main benefit of Leetcode interviews is that they’re easy to standardize. But imo, you shouldn’t be hiring engineers as quickly anymore, which means it’s worth sacrificing standardization for higher-signal. Of course you want to avoid bias, but I think it’s worth pushing on “what would we do if we didn’t have to standardize?” and questioning whether you really need to double or triple headcount. The three problems with graduated time constraints is a great approach.
JD Pruett @jdpruetttsierra.aiThe AI-native interviewWe’ve redesigned our engineering interview process from the ground up.Results from 7 hiring experiments in drawing out talent when everyone has a capable agent.
♥ 161 · ⟲ 2 · 👁 24.2KView on X ↗
