The custom agents you create and configure in Microsoft Copilot Studio can be tested from the Power CAT Copilot Studio Kit. To do this, you need to create a Agent Configuration record that will contain the required information to connect to these agents and run tests against them.
From November 2024 release onwards, multiple configuration types are supported:
- Test Automation - Allows bulk testing of custom agents
- Conversation KPIs - Parses conversation transcripts of selected custom agents and generates Conversation KPI records
- File Synchronization - Allows selectively synchronizing content from SharePoint locations to custom agent knowledge base as files
Multiple configuration types are supported in single agent configuration record. Only relevant configuration sections are visible in the agent configuration view.
- Open the Power CAT Copilot Studio Kit application (as seen in installation instructions).
- Navigate to Agents.
- Create a New Agent Configuration record.
- Fill in the following information.
| Column Name | Required | Description |
|---|---|---|
| Name | Yes | Name of the agent configuration. The name doesn't have to be the same as the agent you target. |
| Configuration Type(s) | Yes | Select one or more configuration types as described above. |
Note: These fields are mandatory for all configuration types.
- Select "Test Automation" from Configuration Type(s).
- In addition to Base Configuration, fill in the following information.
| Column Name | Required | Description |
|---|---|---|
| Region | Yes | Region where the agent is deployed. This is required to target the appropriate Direct Line endpoint. |
| Token Endpoint | Depends | If Channel Security isn't used or enforced, use the token endpoint that's available in the Email app channel. |
| Channel Security | Enable this option if web and Direct Line channel security is enabled. That way, a token can be obtained in exchange for a secret. Note: Channel Security needs to be enabled if user authentication is enabled. |
|
| Secret Location | Depends | When Channel Security is checked, choose where you prefer to store the Direct Line channel secret. Dataverse stores the secret in a secured column, while Key Vault requires to use an environment variable of type secret, and storing the secret in a Azure Key Vault. |
| Secret | Depends | When Dataverse is selected as the Secret Location, this column stores the Direct Line channel secret. |
| Environment Variable | Depends | When Key Vault is selected as the Secret Location, this column stores the schema name for the environment variable of type secret that links to the Azure Key Vault secret. See Configure secrets in Azure Key Vault |
| User Authentication | Select Entra ID v2 if end user authentication is required for this agent. See Enable authentication for instructions how to enable user authentication support. Note: Channel Security needs to be enabled if user authentication is enabled. | |
| Client ID | Depends | Enter the application (client) ID of the application created to enable the end user authentication for custom agent (https://learn.microsoft.com/microsoft-copilot-studio/configuration-authentication-azure-ad) |
| Tenant ID | Depends | Enter the tenant ID of the application created to enable end user authentication for custom agent |
| Scope | Depends | Enter the custom scope that you created earlier when enabling authentication support.. Use the full scope URI format: "api://1234-4567/scope.name" |
| Secret Location (user auth) | Depends | When User Authentication is checked, choose where you prefer to store the client secret. Dataverse stores the secret in a secured column, while Key Vault requires to use an environment variable of type secret, and storing the secret in a Azure Key Vault. |
| Client Secret | Depends | Enter the client secret that you created earlier when enabling authentication support.. "KitAuthApp secret" |
| Enrich With Azure Application Insights | Enable this to enrich test results for Generative Answers tests with Azure Application Insights telemetry data. | |
| App Insights Client ID | Depends | Enter the application (client) ID of the application that has been granted the permissions to read data from Application Insights resource |
| App Insights Application ID | Depends | Enter the AppId of your Application Insights resource. Please see here for additional details. |
| App Insights Secret Location | Depends | Choose where you prefer to store the App Insights application secret. Dataverse stores the secret in a secured column, while Key Vault requires to use an environment variable of type secret, and storing the secret in a Azure Key Vault. |
| App Insights Tenant ID | Depends | Enter the tenant ID of the Application Insights resource and the application |
| App Insights Secret | Depends | When Dataverse is selected as the Secret Location, this column stores the App Insights application secret. |
| App Insights Environment Variable | Depends | When Key Vault is selected as the Secret Location, this column stores the schema name for the environment variable of type secret that links to the App Insights application secret. |
| Enrich With Conversation Transcripts | Enable this option to enrich test results with data from Conversation Transcripts stored in Dataverse. | |
| Dataverse URL | Depends | When Enrich With Conversation Transcripts is enabled, URL of Dataverse environment (e.g., https://org123.crm.dynamics.com) The URL can be obtained from Microsoft Copilot Studio settings (⚙️), in session details > instance url. |
| Copy Full Transcript | No | When Enrich With Conversation Transcripts is enabled, copies the full Conversation Transcript JSON as an attachment to the test result record for further analysis. |
| Analyze Generated Answers | When this option is enabled, AI-generated tests are analyzed with a large language model (LLM) to compare the response with a sample answer or validation instructions. | |
| Generative AI Provider | Depends | When Analyze Generated Answers is enabled, LLM model that is used to do the analysis. Only AI Builder is supported at the moment |
- Save
- Select "Conversation KPIs" from Configuration Type(s).
- In addition to Base Configuration, fill in the following information.
| Column Name | Required | Description |
|---|---|---|
| Dataverse URL | Yes | Dataverse URL where the conversation transcripts (and the agent) are located. Example: "https://organizationname.crm.dynamics.com" |
| Agent ID | Yes | Botid of the custom agent you are targeting. You can find this in Copilot Studio from Settings->Session details->Copilot Id. |
| Copy Full Transcript | Yes | Optionally full transcript of the conversation is copied over and associated with the KPI record. This is required if you wish to inspect the conversation in the transcript visualizer (Transcript-tab on the Conversation KPI record.) |
| Tracked Variables | No | You can specify up to five custom variables you wish to track from the conversations (e.g. custom NPS score). The format is JSON array, example: "[ "Activity.Channel", "Activity.Type" ]" |
| Agent Components | No | This is a readonly field which is populated with the components of the agent you have specified, e.g. topics, knowledge sources, etc. |
- Save
- Select "File Synchronization" from Configuration Type(s).
- In addition to Base Configuration, enter the following information:
| Column Name | Required | Description |
|---|---|---|
| Agent ID (botid) | Yes | Botid of the custom agent you are targeting. You can find this in Copilot Studio from Settings->Session details->Copilot Id. |
| Dataverse URL | Yes | Dataverse URL where the custom agent is located (and content will be synchronized to). Example: "https://organizationname.crm.dynamics.com" |
- Add at least one File Indexer Configuration and fill in the following information.
| Column Name | Required | Description |
|---|---|---|
| Name | Yes | Name of the File Indexer configuration. Example: "HR documents" |
| Site Address | Yes | Address of the SharePoint site you are synchronizing the content from. Example: "https://organizationname.sharepoint.com/sites/sitename" |
| Library Name | Yes | The library in the SharePoint site you are synchronizing the content from. Example: "Documents" |
| Agent Configuration | Yes | The agent configuration this record belongs to. This is automatically populated if you create new record through agent configuration view. |
| Include Nested Items | Yes | If the content synchronization is limited to the exact location you specify, or whether the child items of that location will be included in the synchronization as well. |
| Limit Entries to Folder | No | Optionally you can specify a folder in the library to synchronize. Example: "/HR-documents" |
| SharePoint Files Filter Query | No | Optionally you can specify additional filters to use when selecting files to be synchronized. Leave empty for no additional filtering. Example: PublicContent eq 'true' |
| Include SharePoint Pages | No | Optionally you can include SharePoint pages in the synchronization. |
| SharePoint Pages Filter Query | No | Optionally you can specify additional filters to use when selecting pages to be synchronized. Leave empty for no additional filtering. Example: PublicContent eq 'true' |
- Optionally you can validate your File Indexer Configuration by pressing Validate Connection from the command bar.
- Save
Copilot Studio Kit supports testing custom agents with Entra ID v2 (Azure Active Directory v2) service provider (with SSO) for end user authentication. To enable Copilot Studio Kit end user authentication on your application, please see instructions here.
- For the Copilot Studio Kit to be able to retrieve Conversation Transcript records from other Power Platform environments, the Microsoft Dataverse connection that is used when setting up solution must have
Readaccess on theConversationTranscripttable records in the target environments. - Only environments within the same tenant can be targeted.
- For enriching test results from Application Insights, app registration is required within the same tenant as the Application Insights resource
- Application Insights resource and the application can reside in a different tenant than the custom agent being tested
- For detailed instructions on how to register the application and get the required information for the Application Insights resource, please see here.
- Make sure that the custom agent is configured to send telemetry to Application Insights resource and that it is the same as in the agent configuration in the Copilot Studio Kit. For information on how to connect your custom Agent to Application Insights, please see here
Note
Configuring this requires at least the System Administrator security role on the environment, as well as specific permissions in the Azure Key Vault.
If you do not want to store secrets directly in Dataverse, you can choose to store them in Azure Key Vault. If you choose to do so, you then need to use environment variables of type secrets so that the Power CAT Copilot Studio Kit can fetch secrets from the key vault when running tests. Because each agent configuration may target a different agent and use a different secret, you need to create individual environment variables for each secret.
- Configure your Azure Key Vault by following this documentation: Configure Azure Key Vault.
- Go to make.powerapps.com
- Select the environment in which you want to install the Power CAT Copilot Studio Kit.
- Go to Solutions
- Select Common Data Services Default Solution
Note: you can choose to create or use your own custom solution as well. The idea here is to create environment variables that will be used by Power CAT Copilot Studio Kit to retrieve secrets from the Azure Key Vault. - Follow the steps in: Create a new environment variable for the Key Vault secret
- Once created, use the environment variable schema name (e.g., cr42e_Copilot1DirectLineSecret) in the agent Configuration record.