v0.18.4 Patch Release

What's Changed

• Update version by @sfc-gh-truwase in #7719
• Disable deterministic option in compile tests by @tohtana in #7720
• Fix SuperOffloadOptimizer_Stage3 crash due to missing param_names parameter by @ImaGoodFella in #7715
• [AMD][ROCm] Improve support of AMD by @k-artem in #7448
• fix typo by @stas00 in #7722
• Skip none in backward hook by @tohtana in #7725
• [Engine] Only scale gradients if scale_wrt_gas is True by @kashif in #7724
• Fix testcases that depends on triton by @k-artem in #7731
• Fix rare hang in DeepSpeed Async I/O wait by releasing the Python GIL by @xylian86 in #7727
• Fix #7733: Replace torch.sqrt with math.sqrt in scale_lr for sqrt method by @Rakshit-gen in #7735
• replace moe checkpoint dp_world_size with seq_dp_world_size by @wukong1992 in #7732
• [BUG] Fix UlyssesSPAttentionHF.register_with_transformers() crash with PEFT models by @Rakshit-gen in #7737
• Add core api update blog by @tohtana in #7738
• Fix Nebula checkpoint engine commit() API mismatch by @Rakshit-gen in #7740
• Fix DecoupledCheckpointEngine deadlock and improve reliability by @Rakshit-gen in #7742
• Fix OnebitLamb NaN propagation with empty parameters by @Rakshit-gen in #7736
• fix: remove premature MPI environment variable check in OpenMPIRunner by @leejianwoo-collab in #7751
• Enable python 3.11 and 3.12 tests by @loadams in #7007
• Add CI workflow to run tests on AWS by @tohtana in #7753
• Add fallback to BF16 support check by @tohtana in #7754
• Fix DeepCompile for PyTorch 2.8/2.9 compatibility by @tohtana in #7755
• Removed amp testcases by @k-artem in #7745
• fix: avoid IndexError in BF16_Optimizer.destroy() when using DummyOptim by @leejianwoo-collab in #7763

New Contributors

• @ImaGoodFella made their first contribution in #7715
• @k-artem made their first contribution in #7448
• @kashif made their first contribution in #7724
• @Rakshit-gen made their first contribution in #7735
• @leejianwoo-collab made their first contribution in #7751

Full Changelog: v0.18.3...v0.18.4

v0.18.3 Patch Release

What's Changed

• Update version.txt after release by @loadams in #7675
• [modal ci] fixes by @stas00 in #7676
• leaf modules: explain better by @stas00 in #7674
• disable nv-lightning-v100.yml cI by @stas00 in #7681
• allow seperate learning rate "muon_lr" and "adam_lr" for muon optimizer by @delock in #7658
• see_mem_usage: make always work by @stas00 in #7688
• make debug utils more resilient by @stas00 in #7690
• zero stage 1-2: don't pin memory if not configured by @stas00 in #7689
• modal ci: fix group concurrency by @stas00 in #7691
• Use pytorch utils to detect ninja by @Emrys-Merlin in #7687
• Update SECURITY.md to point to GitHub reporting rather than Microsoft by @loadams in #7692
• Add Qwen2.5 to AutoTP model list by @delock in #7696
• Trust intel server for XPU tests by @tohtana in #7698
• PyTorch-compatible backward API by @tohtana in #7665
• Add news about Ray x DeepSpeed Meetup by @PKUWZP in #7704
• Put Muon optimizer momentum buffer on GPU by @delock in #7648
• [ROCm] Relax tolerances for FP8 unit test for fp16 and bf16 cases by @rraminen in #7655
• Fix that ds_secondary_tensor may be dirty when loading the model or zero checkpoint for zero++. by @zhengchenyu in #7707
• fix: skip aio wait when swap tensors is empty by @xylian86 in #7712
• Low-precision master params/grads/optimizer states by @tohtana in #7700
• Enabled compiled autograd for backward pass by @deepcharm in #7667
• Wall clock timers API by @sfc-gh-truwase in #7714

New Contributors

• @Emrys-Merlin made their first contribution in #7687

Full Changelog: v0.18.2...v0.18.3

v0.18.2 Patch Release

What's Changed

• Update version after 0.18.1 release by @loadams in #7647
• Deduplicate fp32 weights under torch autocast and ZeRO3 by @eternalNight in #7651
• ulysses mpu: additional api by @stas00 in #7649
• ALST/UlyssesSP: more intuitive API wrt variable seqlen by @stas00 in #7656
• Fix misplaced overflow handling return in fused_optimizer.py by @rraminen in #7645
• [bug]: fixed comm_dtype in extra_large_param_to_reduce by @therealnaveenkamal in #7660
• UlyssesSP: TiledMLP doc - recomputes forward twice by @stas00 in #7664
• resolved a 0-dim tensor slicing bug from _get_state_without_padding by @therealnaveenkamal in #7659
• Fix typo in pytorch-profiler.md documentation by @kunheek in #7652
• README refresh by @sfc-gh-truwase in #7668

New Contributors

• @kunheek made their first contribution in #7652

Full Changelog: v0.18.1...v0.18.2

v0.18.1 Patch Release

What's Changed

• Add ZenFlow code for Stage 3 by @JoshWoo2003 in #7516
• [XPU][CI] recover xpu-max1100 workflow by @Liangliang-Ma in #7630
• Take **kwargs in init of DeepSpeedZeroOptimizer subclasses by @eternalNight in #7634
• add support for tensor learning rate (vs scalar) by @NirSonnenschein in #7633
• Fix illegal memory access with multi_tensor_apply size above INT_MAX by @wangyan-mms in #7639
• No Muon optimizer for embeding and lm_head layer by @delock in #7641
• z2: report param name and not zero id in assert by @stas00 in #7637
• z2: don't pass dtype to report_ipg_memory_usage by @stas00 in #7636
• Ulysses HF Accelerate integration by @stas00 in #7638
• Add DataStates-LLM: Asynchronous Checkpointing Engine Support by @mauryaavinash95 in #7166

New Contributors

• @JoshWoo2003 made their first contribution in #7516
• @wangyan-mms made their first contribution in #7639

Full Changelog: v0.18.0...v0.18.1

v0.18.0

What's Changed

• Update version.txt post 0.17.6 release by @loadams in #7572
• DeepCompile ZeRO-3: robust allgather for uneven shards; fix profiling… by @juyterman1000 in #7489
• logging: Also set log level of logger handlers by @eternalNight in #7576
• Deepcompile: Fix bugs when applying deepcompile to VLA-like models by @eternalNight in #7569
• Broadcast fp16 overflow in Z1 by @sfc-gh-truwase in #7580
• Deepcompile: Make size of activation to free configurable by @eternalNight in #7582
• SuperOffload Release by @xylian86 in #7559
• Include init file for superoffload folder by @nguyen599 in #7591
• disables ZeRO checkpoint loading path when stage=0 by @therealnaveenkamal in #7586
• Simplify leaf module hook by @tohtana in #7592
• Fix the universal checkpoint issue for stage3 when there are multiple subgroups. by @zhengchenyu in #7585
• Change current_device() to current_device_name() by @delock in #7600
• Fixed the problem of loading universal checkpoint error in multi-machine mode. by @zhengchenyu in #7601
• DeepCompile: Specify tensor aliasing in C++ op schema by @eternalNight in #7597
• DeepCompile: Fuse allgather and downcast by @eternalNight in #7588
• Add blog for SuperOffload by @xylian86 in #7594
• Add venv to .gitignore by @zhengchenyu in #7605
• Handle the case of DeepCompile's enabled but not activated by @tohtana in #7603
• DeepCompile: Fix IPG bucket clearing by @eternalNight in #7610
• Minor fix in the SuperOffload blog by @xylian86 in #7612
• Fixed the issue that universal checkpoint cannot be loaded for stage3 when world size expansion. by @zhengchenyu in #7599
• Fixed save_checkpoint race when consolidating NVMe offloaded tensors by @H1manshu21 in #7613
• [wall_clock_breakdown] always log stats when enabled by @stas00 in #7617
• DeepCompile: Use min_cut_rematerialization for partitioning joint graphs by @eternalNight in #7609
• Show mismatching values when DeepCompile test fails by @tohtana in #7618
• Improve leaf module interface (enable via config, relax matching criteria, add document, etc.) by @tohtana in #7604
• add print_dist util by @stas00 in #7621
• Super offload blog Chinese version by @delock in #7620
• Enable grad scaler for ZeRO-0 + torch.autocast path by @tohtana in #7619
• Blog of zenflow binding study by @delock in #7614
• Clarify document of leaf module config by @tohtana in #7623
• [TiledMLP] moe support by @stas00 in #7622
• Update email address by @sfc-gh-truwase in #7624

New Contributors

• @juyterman1000 made their first contribution in #7489
• @nguyen599 made their first contribution in #7591
• @zhengchenyu made their first contribution in #7585
• @H1manshu21 made their first contribution in #7613

Full Changelog: v0.17.6...v0.18.0

v0.17.6 Patch Release

What's Changed

• Update version.txt after 0.17.5 release by @loadams in #7502
• Support DeepSpeed offload and reload states with ZeRO1 and ZeRO2 by @LYMDLUT in #7421
• CI funding shout out to modal.com by @stas00 in #7503
• Fix assert when 'pp_int' object has no attribute 'custom_print_str' by @aeeeeeep in #7507
• Update TSC Committers by @PKUWZP in #7517
• Enabling Muon Optimizer in DeepSpeed by @PKUWZP in #7509
• Enable non-ZeRO mode by @sfc-gh-truwase in #7515
• Update README with ZenFlow release blog featured by PyTorch. by @Antlera in #7520
• Add riscv64 cpu support in deepspeed_shm_comm op by @heyujiao99 in #7519
• ZeRO3: Improve mismatch detection by @sfc-gh-truwase in #7525
• fix typo s/1014 /1024 by @digger-yu in #7528
• undo the revert by @stas00 in #7536
• [logging] less startup noise by @stas00 in #7526
• [doc] fixing moe tutorial by @stas00 in #7538
• docs typo: lrrt.md, reference to cycle_min_lr should be cycle_max_lr by @jakehemmerle in #7530
• fixed DeepSpeedCPULion with ZeRO-Offload bug by @qibin0506 in #7531
• Fix scaling and allgather with torch.autocast by @tohtana in #7534
• Fix zenflow_torch_adam.py by @stas00 in #7544
• Relax restrictions of torch.autocast integration by @tohtana in #7543
• Autotune ZenFlow affinity by @delock in #7506
• fix get_cuda_compile_flag by @mingjielu in #7521
• avoid setting device_id to init_process_group by @kaixuanliu in #7542
• Improve error message and reduce validation in autocast test by @tohtana in #7547
• Revert "Add index to HPU devices (#7497)" by @deepcharm in #7545
• [ALST tutorial] support bs>1 by @sfc-gh-sbekman in #7550
• [MoE] Fix misuse of num_experts as expert parallel group size (ep_size) by @Flakes342 in #7551
• Limit random seed range in tests by @tohtana in #7553
• Fix gradient buffer access for DeepCompile Z1/2 by @tohtana in #7548
• Move modal tests to tests/v1 by @tohtana in #7557
• Add dependency for deepcompile test by @tohtana in #7558
• deepcompile: Create dummy inputs using empty_strided by @eternalNight in #7564
• deepcompile: Record graph order using OrderedDict by @eternalNight in #7563
• deepcompile: Create a full list of no-copy ops by @eternalNight in #7562
• fix npu device_id AttributeError issue by @we1sper in #7560
• Make Muon optimizer easier to enable by @delock in #7555
• scripts: Check .is_cuda only in non-C++ files by @eternalNight in #7561
• [bugfix] fix partition context unpatch by @hjh0119 in #7566

New Contributors

• @LYMDLUT made their first contribution in #7421
• @aeeeeeep made their first contribution in #7507
• @heyujiao99 made their first contribution in #7519
• @jakehemmerle made their first contribution in #7530
• @qibin0506 made their first contribution in #7531
• @mingjielu made their first contribution in #7521
• @kaixuanliu made their first contribution in #7542
• @sfc-gh-sbekman made their first contribution in #7550
• @Flakes342 made their first contribution in #7551
• @we1sper made their first contribution in #7560
• @hjh0119 made their first contribution in #7566

Full Changelog: v0.17.5...v0.17.6

v0.17.5 Patch Release

What's Changed

• Update version.txt after v0.17.4 release by @loadams in #7460
• Update README.md by @PKUWZP in #7465
• Add getter APIs for TP/PP/DP ranks in DeepSpeedEngine by @WoosungMyung in #7427
• fix issues raised by Coverity scans by @NirSonnenschein in #7431
• Fix all-gather duplicate params and wrong dtype by @eternalNight in #7462
• fix #7188 by @lpnpcs in #7371
• add --bind_cores_to_rank to zero offload tutorial by @delock in #7474
• Add blog for ZenFlow by @Antlera in #7463
• Fix cpu CI by @sfc-gh-truwase in #7481
• fix deepspeed --venv_script by @stas00 in #7469
• Modal CI by @sfc-gh-truwase in #7289
• [UlyssesSPDataLoaderAdapter] fix iterator reset by @stas00 in #7472
• [TiledFusedLogitsLoss] support inference by @stas00 in #7477
• Fix pre-compile on cpu-only machines by @AlongWY in #7168
• Enable forked PRs by @sfc-gh-truwase in #7486
• fix xpu device_id AttributeError issue by @yao-matrix in #7488
• Add Zenflow code for Stage 1 & 2 by @Antlera in #7391
• Fix invalid f-strings by @cyyever in #7457
• Fix DeepCompile for PyTorch v2.8 by @tohtana in #7496
• Reduce performance impact of compiler.enable decorator by @deepcharm in #7498
• Add index to HPU devices by @deepcharm in #7497

New Contributors

• @WoosungMyung made their first contribution in #7427
• @eternalNight made their first contribution in #7462
• @lpnpcs made their first contribution in #7371
• @Antlera made their first contribution in #7463
• @AlongWY made their first contribution in #7168
• @yao-matrix made their first contribution in #7488
• @cyyever made their first contribution in #7457

Full Changelog: v0.17.4...v0.17.5

v0.17.4 Patch Release

What's Changed

• Update version.txt after 0.17.3 release. by @loadams in #7455
• Fix: UnboundLocalError for variable 'dim' about issue by @weeknan in #7449
• adding TiledFusedLogitsLoss by @stas00 in #7437
• TiledFusedLogitsLoss bug fix by @stas00 in #7459

New Contributors

• @weeknan made their first contribution in #7449

Full Changelog: v0.17.3...v0.17.4

v0.17.3 Patch Release

What's Changed

• [TiledMLP]: fix for bs>1 by @stas00 in #7412
• Update version.txt after v0.17.2 release. by @loadams in #7417
• Enable torch version dependent compilation of record_module and iter_params by @deepcharm in #7362
• [BUGFIX] Reset bucket.elements after reduction in ZeRO Stage 3 by @rahul713rk in #7418
• Align missing argument in AllReduceCoalescedHandle by @deepcharm in #7414
• Improvements to Communication Logger by @alexk101 in #7404
• trying to fix nv-accelerate-v100.yml CI job by @stas00 in #7424
• fix: Propagate strip_tensor_paddings by @saforem2 in #7426
• Use past_key_value when provided by @deepcharm in #7428
• set device_id in torch's init_process_group by @stas00 in #7266
• [Ulysses-ALST] add FA3 support by @stas00 in #7430
• TiledMLP + SequenceTiledCompute: improve the bs>1 use-case by @stas00 in #7422
• Remove unused yaml test configurations and update README by @loadams in #7441
• [ALST] fix typo in the url by @stas00 in #7444
• [ALST] fix typo in the url part2 by @stas00 in #7446
• Remove additional unused tests (human-eval) by @loadams in #7445
• Fix: Adapt Llama injection policy for newer transformers versions by @huanyuqu in #7443

New Contributors

• @rahul713rk made their first contribution in #7418
• @huanyuqu made their first contribution in #7443

Full Changelog: v0.17.2...v0.17.3

v0.17.2 Patch Release

What's Changed

• Update version after 0.17.1 release by @loadams in #7345
• s/UlyssesPlus/Arctic Long Sequence Training (ALST)/ by @stas00 in #7348
• Don't break set_start_method by @tjruwase in #7349
• Fix error of <glog/logging.h> by @Freed-Wu in #7351
• Improve padding util for compile by @tohtana in #7355
• Fix 404s by @tjruwase in #7363
• Fix tutorial title by @stas00 in #7365
• Restore real inputs for recompilation by @tohtana in #7356
• Fix(scheduler): WarmupLR inherits optimizer lr when not specified by @Flink-ddd in #7360
• sequence parallel default dtype by @stas00 in #7364
• Enable torch.autocast with ZeRO by @tohtana in #6993
• add Arctic Long Sequence Training paper reference by @stas00 in #7372
• Flops profiler support for F.interpolate by @sfc-gh-truwase in #7353
• Relax tolerances for FP8 unit test only for ROCm + FP16 by @rraminen in #7373
• Update latest news with DeepNVMe by @loadams in #7375
• Fix release of IPG buffer by @tohtana in #7376
• fix wandb.log() call by removing sync kwarg by @ned2 in #7383
• Fix dtype mismatch in TestParamPartitioningSkipInit by @tohtana in #7377
• Add support for ws=1 scenario by @NirSonnenschein in #7379
• fix(inference): Add missing dtype attribute to ParameterBase setter by @Flink-ddd in #7378
• add blog link by @stas00 in #7385
• fix broken url by @stas00 in #7390
• add support for CUDAtk12.9 by @loscrossos in #7394
• Fix unbound local error for return_val by @HollowMan6 in #7395
• Fix ZeRO stage 1 and add stage 2 support with DeepCompile by @tohtana in #7366
• Improve coverage of DeepCompile by @tohtana in #7386
• Added device detection to communication logging by @alexk101 in #7398
• fix: Add csrc/compile to include paths for DeepCompile builder by @HollowMan6 in #7401
• fix: DeepCompile for torch 2.8 by @HollowMan6 in #7402
• fix(comm): Expose GradBucket in deepspeed.comm API by @Flink-ddd in #7400
• fix: fix FileNotFoundError for build_win.bat by @gjj2828 in #7399
• fix: engine initializes optimizer attributes at the beginning by @HollowMan6 in #7410

New Contributors

• @Freed-Wu made their first contribution in #7351
• @Flink-ddd made their first contribution in #7360
• @ned2 made their first contribution in #7383
• @alexk101 made their first contribution in #7398
• @gjj2828 made their first contribution in #7399

Full Changelog: v0.17.1...v0.17.2