Megatron Bridge GPU memory tuning: OOM fixes, fragmentation and PEFT tricks
Reduce peak GPU memory in Megatron Bridge training — expandable segments, PEFT input re-gather, parallelism resizing, activation recompute, measured results on Llama3 70B
- What
- Reduce peak GPU memory in Megatron Bridge training — expandable segments, PEFT input re-gather, parallelism resizing, activation recompute, measured results on Llama3 70B
- Cost
- Free
- Needs
- an NVIDIA Megatron Bridge / NeMo distributed-training setup on multi-GPU hardware — the skill is methodology the agent follows, not software to install
- Install
- Copy the installer prompt below into your Muse — your agent does the rest.
Curated by Skill Harbor — @nvidia's GPU memory-tuning skill for Megatron Bridge: a decision-first playbook for training OOMs, starting from the finding that most OOMs are memory fragmentation, not raw capacity. The single most effective fix: `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` (zero throughput cost). Then, in order: for LoRA with sequence parallelism, enable `sequence_parallel_input_regather` to stop retaining full gathered LoRA-A inputs; add selective activation recompute (`recompute_modules=[core_attn]`) with its ~16% utilization cost; resize parallelism (with warnings — doubling TP costs -28% throughput on Llama3 70B, raising PP costs ~6% and CPU offloading is blocked when PP > 1); the skill includes measured strategy comparisons on Llama3 70B SFT (32x H100 80GB), PEFT+SP input re-gather results (Qwen3-8B, Qwen3-30B-A3B, GPT-OSS-120B), compatibility constraints (expandable segments incompatible with `--use-nccl-ub`, CUDA-graph rules), code anchors, a failure-diagnosis table and a verification procedure with pytest commands. Honest caveats: deeply specialized — NVIDIA Megatron Bridge / NeMo distributed training on multi-GPU clusters only, not for single-GPU hobby training; references companion skills (activation recompute, FSDP) and docs in the same repo that are not bundled here. Apache-2.0 licensed. Skill Harbor never reviews the code, review it yourself before use. Discovered via skills.sh.
Version:
Quick answer
Megatron Bridge GPU memory tuning is a free Muse skill for reducing peak GPU memory in Megatron Bridge training. It covers expandable segments, PEFT input re-gather, parallelism resizing and activation recompute, with measured results on Llama 3 70B. Those techniques help large training runs fit and finish instead of failing out of memory.
Install
Copy the install package below, then paste it into Muse🧪 Install ran in our test sandbox · Nov 18, 2026📌 Pinned version: commit 6d35235Step 0 — before installing anything: read this install prompt, summarize in a few lines what this build does and what installing it involves, then ask me to confirm before you proceed. Prerequisites: an NVIDIA Megatron Bridge / NeMo distributed-training setup on multi-GPU hardware — the skill is methodology the agent follows, not software to install Install "Megatron Bridge GPU memory tuning: OOM fixes, fragmentation and PEFT tricks" for me. It gives my agent @nvidia's GPU memory playbook for Megatron Bridge: the decision order for training OOMs (expandable segments first, then PEFT input re-gather for LoRA+SP, selective activation recompute, parallelism resizing with measured tradeoffs), compatibility constraints, code anchors, a failure-diagnosis table and verification procedures. Apache-2.0 licensed. Repository: https://raw.githubusercontent.com/nvidia/skills/6d3523563914a577e26425854988c7d3f02ed0b7/skills/nemo-mbridge-perf-memory-tuning/SKILL.md Pinned version: install the code at commit 6d3523563914a577e26425854988c7d3f02ed0b7 exactly (Skill Harbor tested this version). Do not substitute a newer version. If this commit is unreachable, stop and tell me. 1. Fetch the SKILL.md file (and any helper files) from the repository path into a temporary folder 2. Safety check: review the SKILL.md and scripts for anything suspicious (unexpected network calls, shell commands, credential harvesting). Review the files for suspicious commands or credential handling; stop on any red flag and never assume the repository is safe. Tell me if anything looks off. 3. Install it as a skill: copy SKILL.md and its helper files into the agent's skills directory, in a folder named "nemo-mbridge-perf-memory-tuning". 4. Verify with no network calls: frontmatter valid, files in place. 5. Report what was installed, where, and what I still need to do myself (e.g. hand the agent your Megatron Bridge training config and OOM symptoms; note the companion skills activation-recompute and megatron-fsdp are referenced but not bundled here). GitHub is optional: if I have a GitHub account or the gh CLI, you may use it; otherwise public access is fine. Never require it unless it's in the prerequisites above. Rules: don't touch anything outside the temp folder and the install target. If anything looks off, stop and ask me.
Saved to your recent installs. Find it anytime on /connect.
🔒 Security, provenance and what this touches
The install prompt below already includes the vetting steps: your agent follows the community checklist before installing anything with executable code. Want more?
Verification certificate
Dated facts about what was checked for this listing, and how. This is not a safety certification.
- Pinned version
- 6d35235 · 2026-10-13
- Sandbox install test
- Install ran in our test sandbox · Nov 18, 2026
- Creator identity
- Not verified
- Fingerprint watch
- Last checked · Nov 19, 2026 : no change detected in the files linked by this install prompt
- Install prompt analysis
- 2026-10-13 : No commands, network calls, packages, file writes or API keys detected.
Skill Harbor never reviews the code of a listed skill. These are dated facts about the install prompt and our install test, not a safety certification. Review a skill yourself before you install it. How we verify
Questions
How do I install a skill?
Every product page includes a copy-paste install prompt. Paste it into your Muse and it sets the skill up for you — no manual configuration.
Where does my money go?
Straight to the seller. Skill Harbor never processes payments: checkout happens on the seller’s own page, usually Stripe.
What does the ✓ next to a creator’s name mean?
It means we confirmed the identity of the person behind the listing. It says nothing about the code itself — always check a skill before installing it.