Batch evaluation fails for hosted agents deployed with SDK 2.2.0

[brendonmatheson]

• Package Name: azure-ai-projects
• Package Version: 2.2.0
• Operating System: Windows 11
• Python Version: 3.13

Describe the bug

SDK version 2.2.0 removed the container_protocol_versions field from HostedAgentDefinition in commit 1085ea6. This causes batch evaluations to fail for hosted agents deployed via the SDK because the server-side evaluation engine still reads ONLY container_protocol_versions, which is now always empty.

The error returned is:

Unsupported hosted agent protocols '' for agent 'myagent' version 'N'

To Reproduce

1. Deploy a hosted agent with container_configuration and protocol_versions:

from azure.ai.projects import AIProjectClient
from azure.ai.projects.models import (
    HostedAgentDefinition,
    ContainerConfiguration,
    ProtocolVersionRecord,
)

project = AIProjectClient(endpoint=endpoint, credential=credential, allow_preview=True)

agent = project.agents.create_version(
    agent_name="my-agent",
    definition=HostedAgentDefinition(
        protocol_versions=[ProtocolVersionRecord(protocol="responses", version="1.0.0")],
        cpu="1",
        memory="2Gi",
        container_configuration=ContainerConfiguration(image="myregistry.azurecr.io/myagent:latest"),
        environment_variables={"MY_VAR": "value"},
    ),
)

Note that the documentation still refers to container_protocol_versions: https://learn.microsoft.com/en-us/azure/foundry/agents/how-to/deploy-hosted-agent#create-a-hosted-agent-version

2. Wait for version to become active.

3. Run a batch evaluation targeting the agent via Foundry Portal

4. Evaluation fails with: Unsupported hosted agent protocols '' for agent 'my-agent' version '2'

Expected behavior

The batch evaluation run should succeed

Screenshots

Additional context

Except from the log file from the failed evaluations run:

2026-06-12 08:21:17 | ERROR    | evalrun_10a1c69a1b3b4d91825beed01dc8240a | Evaluation failed for run ID evalrun_10a1c69a1b3b4d91825beed01dc8240a with error: Unsupported hosted agent protocols '' for agent 'msft-hdf-agent' version '2'
Traceback (most recent call last):
  File "/app/evaluation/evaluation.py", line 98, in run_evaluation
    results, execution_path = await execute_evaluators_async(evaluation, input_path, output_path, logger, log_path, client=client)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/evaluation/eval_utils.py", line 1363, in execute_evaluators_async
    results = await asyncio.to_thread(
              ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda/lib/python3.12/asyncio/threads.py", line 25, in to_thread
    return await loop.run_in_executor(None, func_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda/lib/python3.12/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/evaluation/eval_utils.py", line 1166, in _execute_evaluators_blocking
    input_data = apply_target_on_data(input_data, evaluation, logger)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/utils/target/target_util.py", line 243, in apply_target_on_data
    raise ValidationErrorException(
exceptions.user_error.ValidationErrorException: Unsupported hosted agent protocols '' for agent 'msft-hdf-agent' version '2'

Querying the deployed agent version via agents.get_version() reveals two protocol-related fields in the service response:

{
  "metadata": {},
  "object": "agent.version",
  "id": "myagent:3",
  "version": "3",
  "definition": {
    "kind": "hosted",
    "container_protocol_versions": [],
    "cpu": "1",
    "memory": "2Gi",
    "environment_variables": {
      "AZURE_AI_MODEL_DEPLOYMENT_NAME": "gpt-4.1",
      "ENABLE_INSTRUMENTATION": "True",
      "ENABLE_SENSITIVE_DATA": "True"
    },
    "container_configuration": {
      "image": "myacr.azurecr.io/myagent:0.4.27"
    },
    "protocol_versions": [{"protocol": "responses", "version": "1.0.0"}]
  },
  "status": "active"
}

Deploying via azd yields an agent for which batch evaluations work. The difference is visible in the agents.get_version() response:

{
  "metadata": {
    "enableVnextExperience": "true"
  },
  "object": "agent.version",
  "id": "myagent:4",
  "version": "4",
  "definition": {
    "kind": "hosted",
    "image": "myacr.azurecr.io/ai-foundry-starter-basic/myagent@sha256:7cdc230e17d417f66d8d1cacbdb70b20b8c5faddb35b076863b49c165dd3f20c",
    "container_protocol_versions": [{"protocol": "responses", "version": "1.0.0"}],
    "cpu": "0.5",
    "memory": "1Gi",
    "environment_variables": {
      "AZURE_AI_MODEL_DEPLOYMENT_NAME": "gpt-4.1",
      "ENABLE_INSTRUMENTATION": "true",
      "ENABLE_SENSITIVE_DATA": "true"
    }
  },
  "status": "active"
}

Similarly calling POST /agents/{name}/versions?api-version=2025-05-15-preview directly with the flat schema produces a correctly populated definition:

{
  "metadata": {},
  "object": "agent.version",
  "id": "myagent:5",
  "version": "5",
  "definition": {
    "kind": "hosted",
    "image": "myacr.azurecr.io/myagent@sha256:6447b8aa3a80d13d89098a745fa5bca9978c8c98eed781598c7563365b0b4ebc",
    "container_protocol_versions": [{"protocol": "responses", "version": "1.0.0"}],
    "cpu": "1",
    "memory": "2Gi",
    "environment_variables": {
      "AZURE_AI_MODEL_DEPLOYMENT_NAME": "gpt-4.1",
      "ENABLE_INSTRUMENTATION": "true",
      "ENABLE_SENSITIVE_DATA": "true"
    }
  },
  "status": "active"
}

[kashifkhan]

Thank you for the feedback @brendonmatheson . Ive routed this to the right team that can help you further

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @bobogogo1990 @dargilco @glharper @howieleung @kingernupur @nick863 @trangevi @trrwilson.

[dargilco]

Thank you @brendonmatheson for the detailed report! I contacted the relevant service team. This is a known issue, and a fix is already underway.