fix(processor): retry 'model produced invalid content' and bump ResultGenerator max_tokens (Bug #43771)

Two-part fix for the Design step failure 'The model produced invalid content':

1. Add 'model produced invalid content' / 'invalid content' to the transient-error patterns recognised by _looks_like_rate_limit so that AzureOpenAIResponseClientWithRetry retries instead of failing.

2. Increase the ResultGenerator agent's max_tokens from 12_000 to 25_000 in OrchestratorBase to prevent truncation of large nested JSON schemas (the underlying cause of the 'invalid content' error).

ADO #43771

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Author: Shreyas-Microsoft

src/processor/src/libs/agent_framework/azure_openai_response_retry.py

@@ -78,6 +78,14 @@ def _looks_like_rate_limit(error: BaseException) -> bool:
     if isinstance(status, int) and 500 <= status < 600:
         return True
 
+    # "The model produced invalid content" is a transient error from Azure OpenAI
+    # when the model output fails content/schema validation — worth retrying.
+    if any(
+        s in msg
+        for s in ["model produced invalid content", "invalid content"]
+    ):
+        return True
+
     cause = getattr(error, "__cause__", None)
     if cause and cause is not error:
         return _looks_like_rate_limit(cause)

src/processor/src/libs/base/orchestrator_base.py

@@ -188,10 +188,12 @@ async def create_agents(
                 )
             elif agent_info.agent_name == "ResultGenerator":
                 # Structured JSON generation; deterministic and bounded.
+                # Use 25_000 to prevent truncation of complex nested JSON schemas
+                # which causes "model produced invalid content" errors.
                 builder = (
                     builder
                     .with_temperature(0.0)
-                    .with_max_tokens(12_000)
+                    .with_max_tokens(25_000)
                     .with_tool_choice("none")
                 )