Identify every point that changes candidate visibility
List every step between an application arriving and a recruiter reading it. Include search filters, duplicate handling, eligibility rules, ranking, summary generation and the default sort order. Then ask which of those only organise information and which can prevent an application from being considered.
A screen labelled recommendations may still shape the shortlist if reviewers only read the first few results. Test what reviewers actually do on that screen, because the feature name tells you little. Keep decisions about suitability with authorised people who can examine the underlying evidence and challenge the system.
Establish criteria before configuring the ATS
Agree the essential role requirements with the hiring manager and write down what would count as evidence. Check that the filter or prompt implements those requirements accurately. Then separate a mandatory licence from a preference for a particular employer, institution or career path.
A suggested match or summary inside an ATS is one kind of AI recruitment software in Australia, and the kinds differ most in where the AI sits and where a person signs off. To see where the AI sits in your own system, use a hypothetical applicant who has equivalent experience described in unfamiliar terms. The test should reveal whether the system makes the evidence visible or excludes the application based on a narrow keyword. A reviewer should resolve the equivalence question rather than relying on the software’s label.
Inspect the evidence behind summaries and scores
Ask vendors to point to the original passage that supports each material statement in a summary. Record how missing or contradictory information appears. A model should not fill a gap with a plausible employment date, or infer a personal characteristic to make the profile look complete.
If a vendor proposes a score, ask what it measures, how it was evaluated and whether that evidence fits your role and applicant population. Do not treat a percentage as a probability of job success unless that interpretation is substantiated. Reject a score that cannot be explained well enough for the decision it would inform.
Test permissions and corrections
Check that an ordinary recruiter can access only the records and notes they’re authorised to see. Verify that an AI feature follows the same boundary. Include exports, search results and generated summaries in the test, because information can surface outside the original record screen.
Correct a deliberately wrong field in an authorised test record, then inspect the summaries, filters and drafts downstream. Decide what happens to an earlier assessment that relied on the wrong information. A correction button helps only if the team understands what it changes across the workflow.
Review candidate communications
Explain the actual process in candidate-facing material and give candidates a route for questions or corrections. Ask legal advisers to assess the applicable privacy and employment obligations. Avoid claiming that a human click automatically resolves every concern about an automated contribution.
For a hypothetical rejection draft, require the recruiter to verify the factual basis and approve the message before it goes out. The wording should not invent a detailed assessment that never happened. Keep a record of the actual reason used and the person who made the decision.
Accept a bounded use, not a universal promise
At the pilot review, report the roles tested, the difficult cases, the errors and the reviewer interventions. Distinguish better organisation of records from evidence of improved selection quality. Do not describe an ATS as inherently fair because it uses AI or because a supplier has a general ethics statement.
Use the Australian Human Rights Commission checklist and Fair Work Ombudsman material as starting points for review. Record the approved use, its limits and the events that call for reassessment. Expand only when the organisation has evidence for the additional workflow.
Sources and reading
Primary sources checked 9 October 2026. The practical evaluation steps in this article are editorial recommendations, not measured customer outcomes.
