Skip to content

feat: add Relay routing Decision API#8

Open
bbednarski9 wants to merge 4 commits into
NVIDIA-NeMo:mainfrom
bbednarski9:feature/relay-decision-api
Open

feat: add Relay routing Decision API#8
bbednarski9 wants to merge 4 commits into
NVIDIA-NeMo:mainfrom
bbednarski9:feature/relay-decision-api

Conversation

@bbednarski9

@bbednarski9 bbednarski9 commented Jun 30, 2026

Copy link
Copy Markdown

What

Summary

Adds the first, independently functional layer of the NeMo Relay integration:

  • a versioned JSON routing-request and routing-decision contract;
  • decision-only Random and LLM routing through the existing profile runtimes;
  • POST /v1/routing/decision with distinct malformed, unknown-profile, and unsupported-profile errors;
  • Relay-compatible route metadata, including the configured Switchyard target ID as backend_id.

This is the Decision API slice of a previous end-to-end implementation, split into a focused public-repository change.

Why

Relay needs Switchyard to select a configured target without dispatching that target itself. Keeping this contract and the two existing v2 routing profiles in a focused first PR makes the wire API, profile capability boundary, and server error behavior reviewable before ATOF state or Cascade routing are introduced.

How tested

Testing summary

  • cargo fmt --all --check
  • cargo clippy --workspace --all-targets -- -D warnings
  • cargo test -p switchyard-components-v2 (46 unit tests plus profile integration suites)
  • cargo test -p switchyard-server (21 server tests plus migration coverage)
  • cargo test --workspace --exclude switchyard-py
  • cargo check -p switchyard-py
  • Loopback classifier integration verifies LLM classification without calling the selected target.
  • uv run ruff check . — not applicable; no Python source changed.
  • uv run mypy switchyard — not applicable; no Python source changed.
  • uv run pytest tests/ — not applicable; the affected API is covered by Rust component/server tests.

Checklist

  • One class per file; filename = snake_case of the primary class. (N/A: Rust API change.)
  • New public symbols exported from switchyard/__init__.py.__all__ if intended for downstream use. (N/A: symbols are exported from the Rust crate root.)
  • Unit tests added for new components / bug fixes.
  • README / --help updated if customer-facing surface changed. (DESIGN.md documents the runtime contract.)
  • Commits signed off (Signed-off-by: Your Name <email>) per the DCO.

Notes for reviewers

Review guidelines

Suggested review order:

  1. decision.rs for the wire contract, protocol/materialization validation, and response conversion.
  2. profile.rs and registry.rs for the single-runtime capability path.
  3. Random/LLM profile adapters to verify that decision-only routing stops before selected-target dispatch.
  4. Server endpoint/error tests for Relay compatibility and status-code precedence.

The legacy Relay payload still includes decision_profile.router. Serde intentionally ignores that additive field; profile_id is authoritative. The response still includes router and all five route fields required by Relay.

Previous feedback addressed

  • Replaced the parallel DecisionRuntimeRegistry design with Profile::decide() on the same Arc<dyn Profile> already used for serving; configured runtimes are built once.
  • Removed typed request-side DecisionProfile.router while retaining legacy wire compatibility for the extra field.
  • Validates schema, attempt bounds, and inbound protocol before profile lookup, so malformed requests remain 400 even when the profile ID is unknown.
  • Rejects unknown inbound profiles rather than silently treating them as OpenAI Chat.
  • Validates actual user prompt material before an LLM classifier call, including adversarial empty/assistant-only shapes.
  • Replaced fallible test .expect() usage with typed error propagation.
  • Keeps malformed (400), unknown profile (404), and unsupported profile (422) failures distinct.

ATOF ingestion and Relay snapshots are intentionally deferred to #9; Relay-aware Cascade routing and unified session metadata are deferred to #11.

Follow-up fixes

  • Keeps Random and LLM decision-response conversion with their profile implementations; decision.rs now owns only shared contract and materialization helpers.
  • Validates required identity/protocol strings and rejects contradictory materialization modes or missing current_request.body before profile execution.
  • Adds focused envelope-validation coverage.

Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
@bbednarski9 bbednarski9 marked this pull request as ready for review July 1, 2026 04:03

- `run()` is the authoritative serving path. It owns the complete request lifecycle and is the
method a Switchyard server should call.
- `decide()` is the decision-only path. Supporting profiles reuse their request-side policy but

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants