Is there any benchmark comparison with Megatron-LM ? #444
Description
Not sure BYTEPS how to handle distribute training cross thousands of GPUs, if NCCL only run within single Node. Especially without SHARP.
{{ message }}
Not sure BYTEPS how to handle distribute training cross thousands of GPUs, if NCCL only run within single Node. Especially without SHARP.