Training hangs on H20 GPU after 1000+ iterations (A100 used in original setup)

[gulizhoutao]

Hi, thank you for the great work and the open-sourced code.

I'm currently trying to reproduce your baseline results using an H20 GPU. However, I noticed that the original paper used an A100 GPU, and due to driver and compatibility constraints on my side, I’m using a slightly different version of PyTorch than the one specified in your environment.

The issue I'm encountering is that training proceeds normally for the first few hundred iterations, but then hangs (freezes) around iteration 1000–2000, without any error message or crash. The GPU usage flatlines, and no forward/backward steps are completed. I’ve verified that this is not due to out-of-memory issues or data loading bottlenecks.

I suspect this may be due to hardware or framework-level differences (e.g., CUDA kernel incompatibility between A100 and H20), but I wanted to check whether anyone else has encountered similar behavior or if you have any suggestions (e.g., known incompatibilities, or workaround options).

Here’s a brief summary of my setup:

GPU: NVIDIA H20

PyTorch: 2.6.0+cu122

CUDA: 12.2

Thanks in advance!

[QAQdev]

Thanks for your attention!

It is already an solved problem and my bad 😥😥😥 (I forgot to push the fix to github after rebuttal), I made a fix in 4927232

[gulizhoutao]

I was able to successfully run the experiments. However, the results I obtained are consistently lower than those reported in the paper. I’ve double-checked the configuration and tried to align the hyperparameters as closely as possible, but there still seems to be a noticeable gap.

I'm currently using a slightly different environment (e.g., different GPU and different versions of PyTorch), so I wonder if the performance might be sensitive to specific library versions, CUDA behaviors, or other subtle implementation details.

Additionally, are there any tricks or non-obvious steps (e.g., seed tuning, learning rate warm-up schedule, gradient clipping, etc.) that you found important for reproducing the results?

Any insights would be greatly appreciated. Thanks again for your valuable work!

[QAQdev]

Could you post your results here and let me have a check

[gulizhoutao]

SQA:62.2,GQA:59.2 for Stablelm-1.6B