Quality > Creating AI Scorecard Instructions

Creating AI Scorecard Instructions

AI scoring and meta data instructions guide how the AI interprets interactions and produces consistent results across all evaluations. The accuracy of your AI evaluations depends on the clarity of these instructions.

Each AI scorecard in FiveLumens includes scoring criteria (used for Yes/No or Selective questions) and meta data fields (used to extract additional insights such as reasons, dates, or identifiers). Both require clear, structured instructions that tell the AI exactly what to look for and how to format its output.

Our Customer Success team can work directly with you to help design and refine these instructions, ensuring they align with your business goals and evaluation standards. However, this guide provides a simple framework should you wish to begin creating them on your own.

AI Scoring Instructions

For scoring criteria, the AI uses your written instructions to determine whether the correct response is Yes, No, or N/A. Each criterion includes structured fields in the prompt generator:

  • Use Case – Required. Describes what is being evaluated and how the AI should interpret it.
  • Yes Exceptions – Optional. Lists any cases that still count as “Yes,” even if they don’t perfectly match the rule.
  • No Exceptions – Optional. Lists any cases that count as “No,” even if they seem partially correct.
  • N/A Exceptions – Optional. Lists cases where the question doesn’t apply.

When writing AI scoring instructions, describe the intent of the question, the behaviors to look for, and what qualifies as each answer option. Be specific and avoid assumptions—the AI will only do what you instruct it to do.

Example: (Yes/No/N/A)

Question: Did the client call to cancel their account?
Use Case: Determine if the client called to cancel their account. Look for different ways the client could express a desire to cancel. 
Answer “Yes” if the client mentions at any point in the call that they want to cancel their account. 
Answer “No” if the client is not requesting a cancellation.
Answer “N/A” if cancellation is never mentioned.

Yes Exceptions: None
No Exceptions: If the client is calling to check the status of a previous cancellation, answer as “No.”
N/A Exceptions: None

AI Meta Data Instructions

Meta Data fields differ from scoring criteria. Instead of evaluating Yes/No/N/A responses, these instructions tell the AI to extract or summarize specific information from the transcript.

When creating Meta Data instructions, focus on:

  1. What you want the AI to find or list (e.g., reasons, dates, product types, issue categories).
  2. How you want the output formatted (e.g., list separated by commas, one phrase, or short paragraph).

Be explicit in describing the output so the AI produces consistent and usable data.

Example: (Meta Data Extraction)

Meta Data Field: Cancellation Reasons
Use Case: Identify all reasons the customer gave for canceling their account. Review the transcript and list each distinct reason mentioned.
Output Format: Return a comma-separated list of cancellation reasons. Example: "Too expensive, Found a better deal, No longer needed."

Best Practices

  • Always write clear and direct instructions—avoid vague terms like “good tone” or “handled well.”
  • Include examples of what qualifies for each answer.
  • For Meta Data, specify both the information type and the exact output format.
  • Review AI results regularly to fine-tune your instructions for accuracy and consistency.

Creating strong AI scoring and meta data instructions is essential to achieving accurate and consistent evaluation results. Well-written instructions ensure that the AI understands your intent, applies your standards correctly, and produces data you can trust.

Start simple—focus on clarity, structure, and precision—and refine over time as you review AI outputs. And remember, our Customer Success team is always available to help you design, test, and optimize your AI prompts to ensure your scorecards deliver the best possible insights.