In short
- One point each: an expectation is one thing a good answer says or does, specific to the question, written before the interview.
- Must-haves are strict: only what you would reject a candidate without, at this level. Everything else is a nice-to-have.
- Three to six per block: fewer and one slip decides it; more and the answer can't reach them in the time.
- Scored in code: in testee each expectation gets a status with quotes; must-haves weigh twice, and the score is for review, not a hiring decision.
- About the job: never protected traits, proxies such as years or fluency, or "culture fit".
What an expectation is
An expectation is one point a good answer covers. Writing them before the interview, and deciding must-haves vs nice-to-haves, is what turns a list of questions into an interview rubric: every candidate is judged on the same points, set in advance.
- Per block. In testee each block (a question, a task or a role-play) has its own expectations, and at least one of them is a must-have.
- Two levels. A must-have is core: you would reject a candidate without it. A nice-to-have is extra credit.
- Hidden from candidates. Expectations never reach the candidate's browser, and the AI interviewer won't reveal them when asked.
- Judged with evidence. After each block the AI analysis gives every expectation a status and quotes the candidate's exact words behind it.
- Quizzes are computed. A quiz's "Answered correctly" must-have comes from the options picked, not from the AI; the other expectations judge the candidate's explanation.
How statuses and the coverage score work
| Status | What it means | Counts as |
|---|---|---|
| Covered | Clearly and correctly addressed without being pointed to it | Full |
| Partial | Mentioned, but shallow or incomplete | Half |
| Covered after a prompt | Addressed only after the interviewer gave a hint for it | Half |
| Incorrect | Addressed, with a factual mistake | Nothing |
| Not covered | Not addressed | Nothing |
The coverage score, from 0 to 100, is computed in code from those statuses, never by the AI, and a must-have weighs twice as much as a nice-to-have. Take a block with three must-haves and two nice-to-haves. The candidate covers two must-haves, gives a partial answer on the third, covers one nice-to-have and misses the other: 2 + 2 + 1 + 1 + 0 = 6 out of 8 points, a score of 75.
The report also says whether every must-have was met (covered, or covered after a prompt). In this example it wasn't: the partial one doesn't count. The interview's overall score combines the blocks, each weighted by its expectations, so a block with more of them counts more. A reviewer can change any status by hand, and the score is labeled "for review, not a hiring decision" (instructions for use).


Scoring each point and adding the points up by a fixed rule is deliberate. Across studies, combining ratings by a formula predicted job performance better than experts combining them in their heads: a mean validity of .44 against .28 (Kuncel et al. 2013). How to read the result is in how to read an AI interview report.
Write an interview rubric you can see in an answer
A good expectation describes something the candidate says or does, specific to the question, so two reviewers reading the same answer reach the same status. Scales that describe in advance what a good answer contains go back to Smith & Kendall (1963); with them, interviewers agree well on ratings (Taylor & Small 2002).
- One point each. "Checks the logs and rolls back the release" is two checks; split it.
- An action or a statement. Start with a verb: names, explains, asks, verifies, acknowledges.
- Specific to the question. "Stores times in UTC and converts them only for display", not "knows date handling".
- Short. About 20 words at most.
- Examples where several answers count. "Offers a workaround (a setting, an export, a manual step)": one done well is enough.
| Vague | Why it fails | Observable |
|---|---|---|
| Good communication | No two reviewers agree on it | Sums up the customer's problem in one sentence before answering |
| Strong technical skills | Not tied to the question | Names where in the system the time zone is lost |
| Customer-focused | A trait, not an action | Asks how the customer uses the feature before suggesting an alternative |
| Understands sales | Can't be seen in one answer | Reviews what was agreed at the demo before following up |
| Good product sense | Too broad | Uses usage data to find where users drop off |
| Handles pressure well | A personality judgment | Says what they told the customer when a promised date slipped |
How many expectations per block
Three to six for most blocks. With fewer, one slip decides the whole block; with more, each point gets a few seconds and the answer can't reach them all in the time.
| Block | Typical count |
|---|---|
| Conversation, 4–5 min | 3–5 |
| Quiz, 1–2 min | 2: answered correctly, plus what the explanation should cover |
| Role-play or written response, 8–10 min | 4–6 |
| Coding, review or whiteboard, 12–15 min | 5–8; a review gets one per planted problem |
Since the overall score weighs each block by its expectations, a block with eight counts for more than one with three. Give the most expectations to the skills that matter most.
Must-haves vs nice-to-haves: be strict
A must-have is a point you would reject a candidate for missing, at this level. Everything else is a nice-to-have. US disability law draws a similar line between a job's essential functions and the rest (29 CFR 1630.2), and it is a good habit for every role.
Four questions for every must-have
- Rejection: would you really turn down an otherwise strong candidate who missed it?
- Level: is it expected at this seniority? A junior needn't name monitoring tools; a senior should.
- Trainable: could they learn it in the first month? Then it's a nice-to-have.
- Answerable: does the question invite it? A point the question never asks for will be missed by good candidates.
If most of a block's expectations are must-haves, check each one again. Nice-to-haves let strong candidates show more without penalizing the rest.
Keep expectations about the job
An expectation must be about what the job needs, never about who the candidate is. Watch for proxies too: words that sound job-related but stand in for age, origin, disability or personality.
| Avoid | The problem | Write instead |
|---|---|---|
| Culture fit | Undefined, so it rewards people like the interviewer | The behavior you value: "Gives an example of changing their plan after a colleague's feedback" |
| Confident, energetic | A personality judgment | "States a recommendation and the reason for it" |
| 10+ years of experience | A proxy for age, not skill | "Explains why they chose one approach over another in a feature they owned" |
| Native English speaker | A proxy for origin | If writing in English is the job: "Writes a reply a customer can follow without jargon" |
| Speaks fluently, no hesitation | Penalizes accent, stammering and nerves | Judge the content of the answer, not its delivery |
In testee the analysis judges coverage only: it doesn't assess personality, accent, fluency, confidence or "culture fit", and never analyzes emotion, voice or faces. Generation refuses questions about protected characteristics, and company-written text is checked for them at publish. Candidates who can't or prefer not to speak can type every answer, so write expectations about what is said, not how. More in fair AI interviews.
Hiring criteria examples for four roles
One conversation block per role, about five minutes each, with expectations written to the rules above. Use them as models, not as a ready interview.
Software engineer: a time zone bug
"Users in other countries say a page in your app shows the wrong time for their bookings. How would you track it down and fix it?"
| Expectation | Level |
|---|---|
| Checks where the time is stored and where it is converted (database, server, browser) | Must-have |
| Stores or sends times in UTC with an explicit offset, converting only for display | Must-have |
| Tests the fix in at least two time zones, including a daylight-saving change | Must-have |
| Uses the platform's time zone support or a date library instead of manual offsets | Nice-to-have |
| Checks other screens and reports for the same bug | Nice-to-have |
Customer support: a feature that disappeared
"A customer writes that a feature they use every day disappeared after yesterday's update. You don't know whether it was removed on purpose. How do you reply?"
| Expectation | Level |
|---|---|
| Confirms whether the change was intended (release notes, the product team) before answering | Must-have |
| Tells the customer honestly what they know and when they will hear more | Must-have |
| Offers a workaround or alternative where one exists | Must-have |
| Passes the customer's use case to the product team | Nice-to-have |
| Checks whether other customers reported the same thing | Nice-to-have |
Sales: a prospect who went silent
"After a demo that went well, a prospect hasn't answered for three weeks. What do you do?"
| Expectation | Level |
|---|---|
| Reviews what was agreed at the demo and who decides before reaching out | Must-have |
| Follows up with something useful to the prospect, not only "checking in" | Must-have |
| Tries another contact in the account if their main contact stays silent | Must-have |
| Sets a point at which they close the opportunity and move on | Nice-to-have |
| Asks the prospect directly whether their priorities have changed | Nice-to-have |
Product manager: a feature nobody uses
"A feature your team launched a month ago is barely used. What do you do?"
| Expectation | Level |
|---|---|
| Looks at usage data to find where users drop off | Must-have |
| Talks to users, both those who tried the feature and those who didn't | Must-have |
| Decides between improving, repositioning or removing it, with a reason | Must-have |
| Names the success metric that should have been set before launch | Nice-to-have |
| Shares what they learned with the team and stakeholders | Nice-to-have |
Checklist before you publish
For every block
- Three to six expectations, more for long tasks, each one point.
- Observable: each starts with a verb and names something said or done.
- Specific: each could only belong to this question.
- At least one must-have, and each must-have passes the rejection, level and trainable tests.
- The question invites every expectation; nothing is a hidden trick.
- Nothing about who the candidate is: no protected traits, proxies, personality or delivery.
- Weight matches importance: the key skills have the most expectations.
Summary
- An expectation is one observable point a good answer covers, written before the interview and hidden from candidates.
- Must-haves are only what you would reject a candidate without; the rest are nice-to-haves.
- In testee each expectation gets a status with quotes, and the score, computed in code with must-haves counted twice, is for review, not a hiring decision.
- Write about the job and the answer's content, never about the person or how they speak.
Questions and answers
What is the difference between must-have and nice-to-have requirements?
A must-have is something you would reject a candidate without, at this level. A nice-to-have adds to a candidate's case but its absence doesn't rule them out. Keep must-haves few and strict.
What are examples of must-haves vs nice-to-haves in a job?
For a support role, "tells the customer honestly what they know and when they will hear more" is a must-have and "passes the use case to the product team" a nice-to-have. For an engineer fixing a time zone bug, storing times in UTC is a must-have; checking other screens for the same bug is a nice-to-have.
How do you build an interview rubric for hiring?
Pick the skills the job needs, write one question per skill, then list three to six observable points a good answer covers, each marked must-have or nice-to-have. Decide them before the first interview and judge every candidate against the same list.
What does a hiring rubric look like?
A list per question: each line is one point a good answer covers, its level, and, after the interview, a status with the evidence for it. The examples above for four roles show real ones.
Do candidates see the expectations?
Not in testee: they never reach the candidate's browser and the interviewer won't reveal them. Candidates see each question or task and the interview's parts and length.
Sources
- Kuncel, Klieger, Connelly & Ones (2013), Mechanical versus clinical data combination in selection and admissions decisions
- Smith & Kendall (1963), Retranslation of expectations
- Taylor & Small (2002), Asking applicants what they would do versus what they did
- ADA regulations, 29 CFR 1630.2 (essential functions)
- testee, Instructions for use (how the coverage score is computed)