Skip to content

feat: per-step inline HF export for Megatron checkpoints#3045

Open
waleko wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
waleko:dev
Open

feat: per-step inline HF export for Megatron checkpoints#3045
waleko wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
waleko:dev

Conversation

@waleko

@waleko waleko commented Jul 2, 2026

Copy link
Copy Markdown

Summary

  • MegatronConfig gains an optional checkpoint: NotRequired[dict] passthrough, applied field-by-field onto megatron-bridge's CheckpointConfig in _create_checkpoint_config (hf_source_path defaults to the policy model name if unset).
  • Lets you opt into megatron-bridge's inline HF export (also_save_hf_checkpoint=True) at every checkpoint save, not just via a separate end-of-training conversion step. Produces iter_*/hf/ — a directly-loadable HF safetensors + tokenizer tree — alongside the native Megatron checkpoint.
  • DTensor backend is unaffected (this only touches megatron_cfg); it already has its own save_consolidated mechanism for full-model HF-shaped exports.

Usage

policy:
  megatron_cfg:
    checkpoint:
      also_save_hf_checkpoint: true
      hf_distributed_save: true       # spread the safetensors write across ranks
      ckpt_format: torch_dist         # required by also_save_hf_checkpoint
      non_persistent_ckpt_type: global  # required by also_save_hf_checkpoint

Test plan

  • Compiles cleanly, no signature-breaking changes (both new params are optional/keyword)
  • Verified live in a downstream fork's pipeline: also_save_hf_checkpoint=true produces a correct, directly-loadable iter_N/hf/ HF checkpoint at every save for a Megatron SFT run

Enable megatron-bridge's inline HF export at every checkpoint save (not
just the last). _create_checkpoint_config hardcoded CheckpointConfig with
no passthrough, so add a minimal opt-in: MegatronConfig gains an optional
"checkpoint" dict that _create_checkpoint_config applies onto
CheckpointConfig, defaulting hf_source_path to the policy model name.

Set policy.megatron_cfg.checkpoint = {also_save_hf_checkpoint: true,
hf_distributed_save: true, ckpt_format: torch_dist,
non_persistent_ckpt_type: global} to produce iter_*/hf/ on every save.

Signed-off-by: Alexander Kovrigin <alexander.kovrigin@jetbrains.com>
@waleko waleko requested review from a team as code owners July 2, 2026 15:37
@copy-pr-bot

copy-pr-bot Bot commented Jul 2, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-maintainers Waiting on maintainers to respond label Jul 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-request waiting-on-maintainers Waiting on maintainers to respond

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants