[None][test] Add Kimi-K2.5 disaggregated GSM8K accuracy test by Shixiaowei02 · Pull Request #15617 · NVIDIA/TensorRT-LLM

Shixiaowei02 · 2026-06-25T06:39:25Z

Description

This pull request adds a new integration test for the Kimi-K2.5 model with NVFP4 quantization, updates the GSM8K accuracy reference data, and ensures the new test is included in the test suite. The main changes are grouped below:

New integration test for Kimi-K2.5:

Added a new TestKimiK25 class in test_disaggregated_serving.py to test GSM8K accuracy for the moonshotai/Kimi-K2.5 model with NVFP4 quantization in a disaggregated server setup. This includes configuration for context and generation servers, and uses the launch_disaggregated_llm utility.

Test suite updates:

Registered the new test TestKimiK25::test_nvfp4 in the llm_function_core.txt test list to ensure it runs as part of the integration test suite.

Accuracy reference data update:

Updated gsm8k.yaml to add an entry for moonshotai/Kimi-K2.5 with NVFP4 quantization and the corresponding accuracy, aligning the reference data with the new test.

Test Coverage

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

Summary by CodeRabbit

New Features
- Added a new integration accuracy test for a Kimi K2.5 model configuration.
- Included the new test in the QA test list so it runs with the rest of the suite.
Bug Fixes
- Expanded GSM8K accuracy coverage with an additional validated model/configuration entry.

Shixiaowei02 · 2026-06-25T06:40:44Z

/bot run --add-multi-gpu-test --disable-fail-fast

coderabbitai · 2026-06-25T06:42:22Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 98f861ea-b27f-484f-9dcd-2a35c49ef2ca

📥 Commits

Reviewing files that changed from the base of the PR and between 70a7528 and cccaf96.

📒 Files selected for processing (3)

tests/integration/defs/accuracy/references/gsm8k.yaml
tests/integration/defs/accuracy/test_disaggregated_serving.py
tests/integration/test_lists/qa/llm_function_core.txt

📝 Walkthrough

Walkthrough

Adds a disaggregated serving accuracy test for moonshotai/Kimi-K2.5 using NVFP4 on GSM8K, records the matching accuracy reference, and adds the test to the QA selection list.

Changes

Kimi-K2.5 NVFP4 accuracy coverage

Layer / File(s)	Summary
Test, reference, and QA wiring `tests/integration/defs/accuracy/test_disaggregated_serving.py`, `tests/integration/defs/accuracy/references/gsm8k.yaml`, `tests/integration/test_lists/qa/llm_function_core.txt`	`TestKimiK25.test_nvfp4` sets up disaggregated context and generation servers for GSM8K with NVFP4, `gsm8k.yaml` adds the NVFP4 accuracy value, and `llm_function_core.txt` adds the test entry.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

NVIDIA/TensorRT-LLM#15544: Also updates an NVFP4 integration accuracy test path in disaggregated serving tests.

Suggested reviewers

xinhe-nv
jieli-matrix
QiJune

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is concise and clearly describes the new Kimi-K2.5 disaggregated GSM8K accuracy test.
Description check	✅ Passed	The description includes the change summary and checklist, but the Test Coverage section is left empty.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

tensorrt-cicd · 2026-06-25T06:46:34Z

PR_Github #55727 [ run ] triggered by Bot. Commit: cccaf96 Link to invocation

tensorrt-cicd · 2026-06-25T14:05:17Z

PR_Github #55727 [ run ] completed with state SUCCESS. Commit: cccaf96
/LLM/main/L0_MergeRequest_PR pipeline #44629 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Shixiaowei02 · 2026-06-26T07:43:02Z

/bot run --add-multi-gpu-test --disable-fail-fast

tensorrt-cicd · 2026-06-26T07:48:50Z

PR_Github #56033 [ run ] triggered by Bot. Commit: 5a0f38d Link to invocation

tensorrt-cicd · 2026-06-26T15:44:02Z

PR_Github #56033 [ run ] completed with state SUCCESS. Commit: 5a0f38d
/LLM/main/L0_MergeRequest_PR pipeline #44905 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Shixiaowei02 · 2026-06-29T09:46:15Z

/bot run --add-multi-gpu-test --disable-fail-fast

tensorrt-cicd · 2026-06-29T09:52:30Z

PR_Github #56345 [ run ] triggered by Bot. Commit: d100822 Link to invocation

tensorrt-cicd · 2026-06-29T15:23:42Z

PR_Github #56345 [ run ] completed with state SUCCESS. Commit: d100822
/LLM/main/L0_MergeRequest_PR pipeline #45193 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Shixiaowei02 · 2026-06-30T01:51:30Z

/bot run --add-multi-gpu-test --disable-fail-fast

Add TestKimiK25.test_nvfp4 to test_disaggregated_serving.py: a disaggregated (ctx-TP4 + gen-TP4) GSM8K accuracy test for Kimi-K2.5 (NVFP4), modeled on the sibling TestKimiK2 disagg test. GSM8K runs the text-only DeepSeek-V3 MLA path; trust_remote_code is set on both servers since the checkpoint ships custom HF modeling code via auto_map. The disagg harness resolves kv_cache_quant_algo=None, so add a bare (NVFP4) GSM8K reference entry for moonshotai/Kimi-K2.5 alongside the existing (NVFP4, FP8-KV) one (same accuracy), mirroring Kimi-K2-Thinking. Register the test in qa/llm_function_core.txt and in the pre-merge 8-GPU B200 stage of test-db/l0_dgx_b200.yml. Signed-off-by: Xiaowei Shi <39303645+Shixiaowei02@users.noreply.github.com>

Shixiaowei02 · 2026-07-01T06:15:45Z

/bot run --add-multi-gpu-test --disable-fail-fast

tensorrt-cicd · 2026-07-01T06:21:11Z

PR_Github #56859 [ run ] triggered by Bot. Commit: 438cf3a Link to invocation

tensorrt-cicd · 2026-07-01T12:55:22Z

PR_Github #56859 [ run ] completed with state SUCCESS. Commit: 438cf3a
/LLM/main/L0_MergeRequest_PR pipeline #45671 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Shixiaowei02 · 2026-07-02T09:49:17Z

/bot skip --comment "DGX_B200-8_GPUs-PyTorch-2/accuracy/test_disaggregated_serving.py::TestKimiK25::test_nvfp4 passed"

Shixiaowei02 · 2026-07-02T12:41:21Z

/bot skip --comment "DGX_B200-8_GPUs-PyTorch-2 has passed"

tensorrt-cicd · 2026-07-02T13:40:54Z

PR_Github #57185 [ skip ] triggered by Bot. Commit: 438cf3a Link to invocation

tensorrt-cicd · 2026-07-02T13:46:54Z

PR_Github #57185 [ skip ] completed with state SUCCESS. Commit: 438cf3a
Skipping testing for commit 438cf3a

Link to invocation

…15617) Signed-off-by: Xiaowei Shi <39303645+Shixiaowei02@users.noreply.github.com>

Shixiaowei02 requested review from a team as code owners June 25, 2026 06:39

github-actions Bot assigned Shixiaowei02 Jun 25, 2026

Shixiaowei02 requested review from bo-nv and chuangz0 June 25, 2026 06:39

Shixiaowei02 requested review from qiaoxj07 and reasonsolo June 25, 2026 06:54

jieli-matrix approved these changes Jun 26, 2026

View reviewed changes

Shixiaowei02 force-pushed the dev/xiaoweis/kimi-k25-disagg-acc branch 2 times, most recently from e654bf7 to 5a0f38d Compare June 26, 2026 07:42

Shixiaowei02 force-pushed the dev/xiaoweis/kimi-k25-disagg-acc branch 2 times, most recently from dca83e0 to d100822 Compare June 29, 2026 09:46

Shixiaowei02 force-pushed the dev/xiaoweis/kimi-k25-disagg-acc branch from d100822 to 629f8bc Compare June 30, 2026 01:51

Shixiaowei02 force-pushed the dev/xiaoweis/kimi-k25-disagg-acc branch from 629f8bc to 438cf3a Compare July 1, 2026 06:15

Shixiaowei02 enabled auto-merge (squash) July 2, 2026 09:49

Shixiaowei02 merged commit 5074112 into NVIDIA:main Jul 2, 2026
8 checks passed

Shixiaowei02 deleted the dev/xiaoweis/kimi-k25-disagg-acc branch July 2, 2026 15:44

coderabbitai Bot mentioned this pull request Jul 6, 2026

[None][test] Increase timeout for Kimi-K2.5 disaggregated nvfp4 test #15970

Merged

1 task

BrianLi23 pushed a commit to BrianLi23/TensorRT-LLM that referenced this pull request Jul 9, 2026

[None][test] Add Kimi-K2.5 disaggregated GSM8K accuracy test (NVIDIA#…

41746df

…15617) Signed-off-by: Xiaowei Shi <39303645+Shixiaowei02@users.noreply.github.com>

Uh oh!

Conversation

Shixiaowei02 commented Jun 25, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Test Coverage

PR Checklist

GitHub Bot Help

Summary by CodeRabbit

Uh oh!

Shixiaowei02 commented Jun 25, 2026

Uh oh!

coderabbitai Bot commented Jun 25, 2026

Walkthrough

Changes

Estimated code review effort

Possibly related PRs

Suggested reviewers

❌ Failed checks (1 warning)

Uh oh!

tensorrt-cicd commented Jun 25, 2026

Uh oh!

tensorrt-cicd commented Jun 25, 2026

Uh oh!

Shixiaowei02 commented Jun 26, 2026

Uh oh!

tensorrt-cicd commented Jun 26, 2026

Uh oh!

tensorrt-cicd commented Jun 26, 2026

Uh oh!

Shixiaowei02 commented Jun 29, 2026

Uh oh!

tensorrt-cicd commented Jun 29, 2026

Uh oh!

tensorrt-cicd commented Jun 29, 2026

Uh oh!

Shixiaowei02 commented Jun 30, 2026

Uh oh!

Shixiaowei02 commented Jul 1, 2026

Uh oh!

tensorrt-cicd commented Jul 1, 2026

Uh oh!

tensorrt-cicd commented Jul 1, 2026

Uh oh!

Shixiaowei02 commented Jul 2, 2026

Uh oh!

Shixiaowei02 commented Jul 2, 2026

Uh oh!

tensorrt-cicd commented Jul 2, 2026

Uh oh!

tensorrt-cicd commented Jul 2, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Shixiaowei02 commented Jun 25, 2026 •

edited by coderabbitai Bot

Loading