How to evaluate your generative AI application using Microsoft Foundry. Whether you're assessing single-turn or multi-turn conversations, Microsoft Foundry provides tools for evaluating model performance and safety.
For more details instruction see the Microsoft Foundry Documentation
Here are the steps to get started:
Prerequisites
- A test dataset in either CSV or JSON format.
- A deployed generative AI model (such as Phi-3, GPT 3.5, GPT 4, or Davinci models).
- A runtime with a compute instance to run the evaluation.
Microsoft Foundry allows you to evaluate both single-turn and complex, multi-turn conversations. For Retrieval Augmented Generation (RAG) scenarios, where the model is grounded in specific data, you can assess performance using built-in evaluation metrics. Additionally, you can evaluate general single-turn question answering scenarios (non-RAG).
From the Microsoft Foundry UI, navigate to either the Evaluate page or the Prompt Flow page. Follow the evaluation creation wizard to set up an evaluation run. Provide an optional name for your evaluation. Select the scenario that aligns with your application's objectives. Choose one or more evaluation metrics to assess the model's output.
For greater flexibility, you can establish a custom evaluation flow. Customize the evaluation process based on your specific requirements.
After running the evaluation, log, view, and analyze detailed evaluation metrics in Microsoft Foundry. Gain insights into your application's capabilities and limitations.
Note Microsoft Foundry is currently in public preview, so use it for experimentation and development purposes. For production workloads, consider other options. Explore the official AI Foundry documentation for more details and step-by-step instructions.