How to convert a large model(e.g., with hundreds of billions of parameters) from Hugging Face to NeMo format ？ #14102

gaojingwei · 2025-07-02T03:22:37Z

gaojingwei
Jul 2, 2025

During the process of converting a model from Hugging Face to NeMo format, I encountered an issue: if the model is too large, it throws an "Cannot allocate memory" error. This does not happen with smaller models. My model has 600 billion parameters (approximately 1.2TB in size), and I'm running it on an A100 server with 2TB of RAM. The convert_state() completes without errors, but it fails during the save_nemo() step. The error message is shown in the screenshot below:

[rank0]:   File "/opt/NeMo/nemo/collections/llm/api.py", line 887, in import_ckpt
[rank0]:     output = io.import_ckpt(model=model, source=source, output_path=output_path, overwrite=overwrite, **kwargs)
[rank0]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/opt/NeMo/nemo/lightning/io/api.py", line 167, in import_ckpt
[rank0]:     ckpt_path = importer(overwrite=overwrite, output_path=output_path, **kwargs)
[rank0]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/opt/NeMo/nemo/lightning/io/connector.py", line 101, in __call__
[rank0]:     to_return = self.apply(_output_path, **kwargs)
[rank0]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/opt/NeMo/nemo/collections/llm/gpt/model/my_model.py", line 134, in apply
[rank0]:     self.nemo_save(output_path, trainer)
[rank0]:   File "/opt/NeMo/nemo/lightning/io/connector.py", line 207, in nemo_save
[rank0]:     trainer.save_checkpoint(output_path)
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/lightning/pytorch/trainer/trainer.py", line 1365, in save_checkpoint
[rank0]:     self.strategy.save_checkpoint(checkpoint, filepath, storage_options=storage_options)
[rank0]:   File "/opt/NeMo/nemo/lightning/pytorch/strategies/megatron_strategy.py", line 893, in save_checkpoint
[rank0]:     self.checkpoint_io.save_checkpoint(checkpoint, filepath, storage_options=storage_options)
[rank0]:   File "/opt/NeMo/nemo/utils/callbacks/dist_ckpt_io.py", line 132, in save_checkpoint
[rank0]:     async_request = self.checkpoint_io.save_checkpoint(checkpoint, path, storage_options)
[rank0]:                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/opt/NeMo/nemo/lightning/io/pl.py", line 201, in save_checkpoint
[rank0]:     async_save_request = dist_checkpointing.save(
[rank0]:                          ^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/opt/megatron-lm/megatron/core/dist_checkpointing/serialization.py", line 404, in save
[rank0]:     async_request = sharded_strategy.async_save(sharded_state_dict, checkpoint_dir)
[rank0]:                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/opt/megatron-lm/megatron/core/dist_checkpointing/strategies/fully_parallel.py", line 91, in async_save
[rank0]:     return self.base_strategy.async_save(sharded_state_dict, checkpoint_dir)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/opt/megatron-lm/megatron/core/dist_checkpointing/strategies/torch.py", line 744, in async_save
[rank0]:     ) = save_state_dict_async_plan(
[rank0]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/opt/megatron-lm/megatron/core/dist_checkpointing/strategies/state_dict_saver.py", line 159, in save_state_dict_async_plan
[rank0]:     storage_writer.prepare_write_data(central_plan, planner)
[rank0]:   File "/opt/megatron-lm/megatron/core/dist_checkpointing/strategies/filesystem_async.py", line 155, in prepare_write_data
[rank0]:     self.results_queue = _get_write_results_queue()
[rank0]:                          ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/opt/megatron-lm/megatron/core/dist_checkpointing/strategies/filesystem_async.py", line 39, in _get_write_results_queue
[rank0]:     _results_queue = ctx.Manager().Queue()
[rank0]:                      ^^^^^^^^^^^^^
[rank0]:   File "/usr/lib/python3.12/multiprocessing/context.py", line 57, in Manager
[rank0]:     m.start()
[rank0]:   File "/usr/lib/python3.12/multiprocessing/managers.py", line 562, in start
[rank0]:     self._process.start()
[rank0]:   File "/usr/lib/python3.12/multiprocessing/process.py", line 121, in start
[rank0]:     self._popen = self._Popen(self)
[rank0]:                   ^^^^^^^^^^^^^^^^^
[rank0]:   File "/usr/lib/python3.12/multiprocessing/context.py", line 282, in _Popen
[rank0]:     return Popen(process_obj)
[rank0]:            ^^^^^^^^^^^^^^^^^^
[rank0]:   File "/usr/lib/python3.12/multiprocessing/popen_fork.py", line 19, in __init__
[rank0]:     self._launch(process_obj)
[rank0]:   File "/usr/lib/python3.12/multiprocessing/popen_fork.py", line 66, in _launch
[rank0]:     self.pid = os.fork()
[rank0]:                ^^^^^^^^^
[rank0]: OSError: [Errno 12] Cannot allocate memory

The error is likely caused by os.fork() when creating a subprocess, as it duplicates the parent process's address space, leading to an out-of-memory issue.

I tried setting asynchronous saving to False, but I still got the same error.

trainer = self.nemo_setup(target, ckpt_async_save=False)

The NeMo Docker image version I'm using is 2504.

gaojingwei · 2025-07-03T08:07:53Z

gaojingwei
Jul 3, 2025
Author

I change all get_context('fork') to get_context('spawn‘)，the above error is disappeared，and get the new log

Global Checkpoint Save : Rank: 0 : Iteration: 0 : Start time: 1748312564.883s : Save duration: 176 s

but the process is killed by system becuase out of memory。

My question is why nemo_save() is taking up so much memory. What can I do to successfully complete the conversion?

0 replies

gaojingwei · 2025-07-07T02:58:26Z

gaojingwei
Jul 7, 2025
Author

To solve the memory issue, I tried increasing the swap by 512GB, but I received a new error:

[NeMo I 2025-07-04 07:06:48 nemo_logging:393] Global Checkpoint Save : Rank: 0 : Iteration: 0 : Start time: 1751612652.030s : Save duration: 155.982s

[a0528dee9e2e:822  :94:1148] Caught signal 7 (Bus error: nonexistent physical address)
[1751613504.356728] [a0528dee9e2e:822  :95]           debug.c:1300 UCX  WARN  ucs_debug_disable_signal: signal 8 was not set in ucs
[a0528dee9e2e:822  :95:1132] Caught signal 7 (Bus error: nonexistent physical address)
[1751613504.356729] [a0528dee9e2e:822  :94]        spinlock.c:29   UCX  WARN  ucs_recursive_spinlock_destroy() failed: busy
==== backtrace (tid:   1121) ====
 0  /opt/hpcx/ucx/lib/libucs.so.0(ucs_handle_error+0x2e4) [0x7fe207f08654]
 1  /opt/hpcx/ucx/lib/libucs.so.0(+0x3684c) [0x7fe207f0884c]
 2  /opt/hpcx/ucx/lib/libucs.so.0(+0x36abf) [0x7fe207f08abf]
 3  /usr/lib/x86_64-linux-gnu/libc.so.6(+0x45330) [0x7fe2135e7330] 
 4  /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so(+0x673968b) [0x7fe20014e68b]
 5  /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so(_ZNK2at18TensorIteratorBase15serial_for_eachEN3c1012function_refIFvPPcPKlllEEENS_5RangeE+0x4e2) [0x7fe1fae22ae2]
 6  /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so(+0x140dc33) [0x7fe1fae22c33]
 7  /usr/lib/x86_64-linux-gnu/libgomp.so.1(+0x2279f) [0x7fe212ba779f]
 8  /usr/lib/x86_64-linux-gnu/libc.so.6(+0x9caa4) [0x7fe21363eaa4]
 9  /usr/lib/x86_64-linux-gnu/libc.so.6(__clone+0x44) [0x7fe2136cba34]  
=================================

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

How to convert a large model(e.g., with hundreds of billions of parameters) from Hugging Face to NeMo format ？ #14102

Uh oh!

{{title}}

Uh oh!

Replies: 2 comments

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

How to convert a large model(e.g., with hundreds of billions of parameters) from Hugging Face to NeMo format ？ #14102

Uh oh!

gaojingwei Jul 2, 2025

Replies: 2 comments

Uh oh!

gaojingwei Jul 3, 2025 Author

Uh oh!

gaojingwei Jul 7, 2025 Author

gaojingwei
Jul 2, 2025

gaojingwei
Jul 3, 2025
Author

gaojingwei
Jul 7, 2025
Author