We want to train large amount of data. All data can not be downloaded at a time. So we are deciding to download in chunk by chunk.

But while running megatron_gpt_pretraining.py, I need to give the data files in config file. Does it read the data files sequentially and sends to training one by one? If I give the file names, at the latter part of the list, which are not present currently, will it work ?

Is there any better way to do it?