Prepare test questions

Important This content is part of the Sage HR Assistant Early Adopter release. If we haven't contacted you to be part of this program, refer to future release information.

Understand how to prepare test questions for Sage HR Assistant.

  • Consider if you want a test per population group, per policy or another subset of tests. Prepare the questions accordingly for entry into the testing template later

  • A good-quality test case dataset has three quality attributes:

    • Volume: A sufficient number of test cases to ensure comprehensive coverage of different scenarios and edge cases

    • Diversity: A wide range of inputs, contexts, and variations to test the AI agent's adaptability across real-world uses cases

    • Quality: well-defined, accurate, and relevant test cases you align with the AI agent's objectives

  • You can use the Sage People Insights Generator to provide a list of suggested questions to start. Alternatively, you can use other tools like Copilot or Claude to do the same thing

  • You can access the Insights Generator from the App selector. This provides a list of suggested questions for each group of content you upload. You can use these as the basis for collating your bank of test questions

Important considerations

  • How many questions do you want to ask per policy? For example, longer policies can justify more, and more detailed, questions

  • You can run a different test per policy or mix policies within the same test

  • What are your acceptable threshold scores before you go live?

  • Remember testing will use some of your credits

  • Currently, you can run up to 10 test jobs at once in a 10-hour time frame. You can have up to 1000 test cases per test

  • Try to predict the types of questions users will ask. Try to cover every angle and possibility. Remember this is what you're designing your tests to cover

  • Expect to see errors or issues as you start testing. Test for positive, expected results, then for negative, unexpected, or invalid results. You need to be sure the agent responds correctly, even to "wrong" requests. We call this positive and negative testing, and it makes for a more robust test suite

  • Phrase your questions differently from your document headings. Include the team member name in the question you're asking

  • You can go to the Agentforce Studio for specific detailed analysis. Select Agents then HR Assistant. Open in the Builder, and use the Conversation Preview application

How many questions to ask

  • Ask as many questions as you need to make sure you've covered each policy to a satisfactory level

  • Focus on quality and coverage over volume

  • Aim to reveal patters rather than test every possible scenario

  • We recommend a minimum of 20-35 core policy questions and 10-15 conditional edge-case questions such as eligibility and exceptions

Types of questions to ask

Core policy questions

  • Clear, factual questions the system answers directly from policy documents

    • "What's the annual leave entitlement for Employee Tester?"

    • What's the probation period for Director Tester?"

Policy and employee context questions

  • Policy questions that require grounding in employee data

    • "Does the parental leave policy apply to Manager tester?"

    • "Is Employee Tester eligible for sick pay during probation?"

Boundary and negative questions

  • Questions that test safe handling and guardrails

  • "Can I override approved leave for Manager Tester?"

  • Is salary information available Director Tester?"