We were hiring for senior devs java/spring boot so I coded a tiny "Todo app" rest API, basically a single crud controller, and filled the code with many mistakes. Some obvious, some less obvious. Missing authentication, no tests, logging with System.out etc.. then we ask the candidate to review the code (we provide a laptop with a few IDEs but no AI tools). When they point out errors, I get them to explain why they are errors by playing the junior engineer to see how they explain technical issues to someone that might not have the same understanding I am not looking for them to find all issues but if they can't find at least some it's a red flag. It takes about 20-25 minutes and gives really good insights. The other 30 minutes we asked some questions about past projects and some technical questions

So far it provided good signals

The same way we did before. Very very simple code submission (most people using AI use it even though we call out we're going to ask them later to modify later without AI tools writing code for them) then pairing interview where we ask some basic "are you actually at the level you say you are" question, then ask them to extend their program submission with:

- engineers on the call as pairing assistants - google, ai tools, whatever for libraries, syntax, etc. we tell the candidate directly that it's impossible for us to gsther signal on how they think about problems if they ask Claude to just whip them up a solution - the expected output - their own unit test suite

Every candidate that has submitted an ai submission thus far has failed because they have literally no idea where to go. I've interviewed dozens at this point. I'm not saying "they're unfamiliar with the structure", I'm saying "they cannot actually break down the problem even verbally".

It doesn't matter their pedigree or past experience on their resume, if they used AI to generate they don't seem to be able to resurrect the skills that actually matter for the thing, engineering and product work.

Note because I know folks hate code submissions. It's not hard. We give a CSV with 3 columns, 10 lines. Do some basic mapping and some structuring, some basic data modeling. We only expect about 1 actual class or struct. Then unit tests and it should run in the terminal. Max submission length with verbosity has been a java program at something like a hundred lines total if that, most folks complete the submission in an hour or two. Extension is that we modify one of the rules and extend the CSV by 5 lines.

I seem to still be getting good signal from this, since the engineers that I've hired off of this have been fantastic with or without AI tooling immediately in their hands during the day.

To me AI hasn't had any impact on interviewing, only on filtering the CVs as I immediately flag obviously LLM-generated ones that don't have any substance, while those that are LLM-assisted with tasteful judgement are fine.

On the interview side, I have gone through a phase which required do an architectural design task on a whiteboard. I realized how stressful and not very helpful at evaluating it might be when I myself had to sit on the interviewee's side several times. It's difficult to get into this play-design mode.

Currently, I just make natural conversations, ask some technical questions on the level where I estimate the candidate is, try to understand the level of their knowledge and whether I would like to work with them in one team as a person. Maybe it wouldn't work for highly specialized roles, but for a general backend work I have been happy with the results.

Leetcode doesn’t tell you anything apart from someone memorised set of questions and their solution.

For your question, I assume you want people who can solve problems, can explain their thought process and reasons of decisions. I did interview devs recently, and my questions were related to their experience in their CV. Something like, tell me about project X in Y company, what did you do, what did you use, why, what did you learn from it.

My personal opinion is that LLMs are no difference from developers who over engineer and over complicate things. You can get them to do good work with the right steering and right understanding of the whole system within the context of the company.

Measure them on merit, their body of work and how well they will fit into the team culture.

As for the questions you’re asking, instead of asking old-school coding interview questions ask them stuff that is actually relevant to agentic programming.

Ask them to build a harness, have them explain what skills are, have them build a self-made implementation of Claude Code as take-home assignment.

The times have changed, you need to update interviewing questions to meet them. Start with agentic questions and ramp them up into more complex scenarios. This is how you gauge your candidates, see how they think and solve their way out of it.

Try to determine how creative, adaptable, motivated the candidate is. Figure out what there actual skill-sets are that they can bring to you, not whether they know some CS pedantry