feat: add Relay-aware Cascade routing#11
Open
bbednarski9 wants to merge 12 commits into
Open
Conversation
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
This was referenced Jul 1, 2026
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
df3a7e8 to
fd105d2
Compare
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Summary
Adds the third NeMo Relay integration layer on top of #8 and #9:
DecisionContextthat carries the typed request and one immutable, exact Relay snapshot through the existing profile registry;FeatureFreshness, explicit cold-default behavior, and no selected-target dispatch;(session_id, owner_id)lookup without parent, root, or session-only fallback;proxy_x_session_idandx-nemo-relay-session-id, including repeated-value, conflict, empty-value, invalid-UTF-8, and body/header mismatch handling.This is the Cascade and request-metadata slice of a previous end-to-end implementation, split into a focused public-repository change.
Why
The Decision API and router-neutral ATOF snapshots become end-to-end useful when an existing router can consume them safely. This PR lets Cascade route from Relay history while preserving its normal inference path, the single configured profile runtime, strict tenant/session isolation, and Relay's current body-only Decision identity behavior.
How tested
Testing summary
cargo fmt --all -- --checkcargo clippy --workspace --all-targets -- -D warningscargo check --workspace --all-targetscargo test -p switchyard-components-v2(70 unit tests plus 51 profile integration tests)cargo test -p switchyard-server(1 CLI, 10 migration, and 32 server tests)uv run ruff check .uv run mypy switchyard(160 source files)uv run pytest tests/ -q -o addopts=(1,880 passed, 50 skipped)http_postATOF ingestion followed by a body-only Decision request warms Cascade and returns a strong route while both selected target URLs are unreachable.cargo test -p switchyard-pyreaches the known macOS PyO3 cdylib link limitation (undefined _Py*symbols); the crate passes workspace check/clippy, and the binding behavior passes in the full 1,880-test Python suite.Checklist
snake_caseof the primary class. (N/A: Rust/Python functional API change.)switchyard/__init__.py.__all__if intended for downstream use. (N/A: Rust symbols are exported from the crate root; the existing Python module exposes the extended metadata type.)--helpupdated if customer-facing surface changed. (DESIGN.mddocuments decision-scoped resources and session metadata.)Signed-off-by: Your Name <email>) per the DCO.Notes for reviewers
Dependency and delta
This PR depends on #8 and #9. Because the author has read-only access to the upstream repository, this draft must target upstream
mainand temporarily contains both prerequisite commits.Review the #11 delta commits
2bb2abc0(Cascade integration),8425e3ab(Decision bearer auth and E2E coverage),c5701405(envelope-validation coverage), andae5a982a(shared Relay auth documentation), plus the bounded-memory/reporting fixes inherited from #9. Use the exact fork delta. After #9 merges, this branch will be rebased onto updatedmainso the PR contains only the Cascade/auth delta.Review guidelines
Suggested review order:
decision.rs,profile.rs, andregistry.rsfor the ownedDecisionContext, typed freshness/session metadata, and same-runtime dispatch.profiles/cascade.rsfor snapshot projection, cold/fresh behavior, source accounting, and the decision-only no-dispatch boundary.switchyard-serverfor exact canonical identity lookup and session-header/body reconciliation.session_idargument and matching alias semantics.Fresh snapshots use Cascade's existing override/scorer/classifier policy. Missing or turn-only snapshots select the configured picker default immediately, report
cascade_feature_cold_default, retain the existingfall_opensource, omit confidence, and do not call the classifier or selected target.Previous feedback addressed
Arc<dyn Profile>for inference and decisions; no parallel registry, duplicate runtime, or Cascade-owned event state is introduced.Option<FeatureFreshness>and an exhaustiveFreshmatch.(session_id, owner_id)lookup only. Parent/root scopes and session-only state cannot satisfy an owner-specific miss; padded owners normalize and blank owners match ownerless ATOF state.decision_profile.routerfield compatible whileprofile_idremains authoritative.run()signal extraction and selected-target dispatch path..expect(),.unwrap(), or panic paths in the integration code.Follow-up fixes