Your hiring AI changed last week.Do you know how?
We test AI hiring tools with synthetic candidates, and keep testing every time they change.
Three things we found in our own tests.
The hidden instruction
Dana Okafor
Applying for operations analyst
synthetic candidate
Hidden from the person reading it. Not from the model.
Score change
+14points
Same résumé, plus one line you can’t see.
In our tests, a hidden instruction raised a widely used model’s scores about 14 points.
The missing skill
The job requires SQL. None of these résumés mention it.
91–96%still advanced.
Same tool, different week
A model, prompt or host changes behind the tool. Nobody announces it. The scores move anyway.
What changed. Without a re-test, nobody sees it.
What we test
- Fairness
- Do equally qualified candidates get equal scores when only their name or background changes?
- Criteria
- Does the tool actually hold candidates to the skills the job requires?
- Consistency
- Does the same résumé get the same score every time?
- Manipulation
- Can hidden text or keyword stuffing move a score?
- Drift
- Did behavior change after a model, prompt or host update?
How it works
- 1
We send synthetic candidates.
Fictional résumés, each built to test one thing. No real people’s data, ever.
- 2
You return scores.
As a batch file or through your API, whichever is easier for your team.
- 3
We report, re-test and alert you.
A written report, then a fresh run every time the tool changes, and an alert when its behavior does.
Then back to step 1, every time the tool changes.
Privacy
No real candidate data.
Nothing leaves your side unless you want it to.
Results are yours, under NDA.
Design partner pilot
Free for 3 months.
- For the first five vendors.
- A weekly check-in with us.
- An anonymized case study at the end.
Find out what changed.
Or write to hello@probityhq.dev