chore: CI proxy for #2953#2965
Closed
HuiyingLi wants to merge 3 commits into
Closed
Conversation
The DSpark offline cache path (precompute_dspark) loads the whole target on one process, so targets that do not fit on a single node (DeepSeek-V4-Flash, GLM-5.2) cannot be precomputed offline at all: the only workaround is the reduced-layer diagnostic load, whose cache is not usable for a real drafter. Add a distributed (multi-node) precompute entry point that loads such targets frozen through the same expert-parallel / FSDP2 path the online training recipe uses, runs them once over the dataset, and writes the identical DSpark cache that train_dspark consumes via recipe_args.cached_target_path. - Extract the DeepSeek V4 / GLM-5.2 frozen-target builders out of TrainDSparkRecipe into recipes/llm/_dspark_target_build.py so the recipe and the precompute share one implementation (no behavior change). - Add partition_cache_shards (contiguous, shard-aligned per-rank blocks: every shard is owned by exactly one rank, so ranks write global-indexed shards straight into a shared output directory with no merge step) and write_cache_shards_distributed (per-rank write loop that pads its forward count to the global max with dummy batches so the EP/FSDP collectives stay in lockstep) to offline_cache.py, sharing the shard accumulator with the single-process writer. - Add recipes/llm/precompute_dspark_dist.py: config-driven, torchrun-launched; rank zero gathers the DTensor-sharded embed_tokens / lm_head to full tensors and writes the target weights + manifest. - Unify the cache schema (compute_batch_cache, build_cache_manifest, manifest_mismatch_fields) in dspark_cache.py so both producers emit identical caches by construction. - Add example precompute configs for DeepSeek-V4-Flash and GLM-5.2 and document the two-phase offline flow in the speculative README. - Tests: partition and lockstep unit coverage, an entry-point suite, and a functional equivalence test that runs a real tiny Qwen3 target on two gloo ranks and asserts the distributed cache is bit-identical to the single-process reference. Closes #2948 Signed-off-by: khazic <khazzz1c@gmail.com>
…omplete marker Address PR review: the cache manifest recorded only tensor-shaping settings, so a rerun into an existing directory with a different dataset, split, seed, masking, or chat template passed the compatibility check, and an interrupted rerun left a contiguous readable mixture of old and new supervision. - build_cache_manifest now records the run's input identity (train_data_path, train_split, shuffle_seed, mask_reasoning_content, sha256 of the effective chat template); both precompute producers thread it through, so a different-input rerun fails the manifest compatibility check up front. - Staged publish: producers write the manifest with complete=false before the first shard and flip it to true only after every shard is written; consumers (read_manifest, CachedDSparkDataset) reject an incomplete cache, while the producers' own compat checks pass allow_incomplete to permit a same-config rerun into an interrupted directory. Legacy manifests without the field are still accepted. - Exempt the two command-only precompute configs from the example-YAML recipe lint (they are launched via python -m precompute_dspark_dist, not a recipe class), unblocking validate-nightly-recipes. Signed-off-by: khazic <khazzz1c@gmail.com>
HuiyingLi
requested review from
a team,
akoumpa,
athitten and
snowmanwwg
as code owners
Contributor
Author
|
/ok to test 92a07ba |
…y live CI (L0_Unit_Tests_CPU and _GPU) failed both tests: some other test earlier in the same pytest session leaks an initialized default process group (never destroyed), so test_make_sync_max_steps_identity_without_process_group's 'no group' precondition is false by the time it runs, and test_make_sync_max_steps_all_reduce_over_single_process_group's own init_process_group() then raises 'trying to initialize the default process group twice'. Passes in isolation; only fails as part of the full suite. Guard both with the same pytest.skip(...) idiom already used elsewhere in this test tree (test_train_eagle1_grad_accum.py, test_eagle3_token_cache.py) for tests that need a known-clean torch.distributed state: skip rather than force-destroy a group we don't own, since we don't know what still holds a reference to it. Signed-off-by: khazic <khazzz1c@gmail.com>
Contributor
Author
|
/ok to test 103a28e |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI-only proxy PR. Do not merge / do not review. Points at the exact head commit of #2953 (103a28e) so internal CI runs under the internal-contributor queue; results post back to #2953 via the shared SHA. Source of truth: #2953. Close once CI completes.