lora_finetune.py for continuous pretraining for DeepSeek-V3

[ghtaro]

Hi,

I am new to collosal AI and would like to continous-pretraining DeepSeek-V3 to domain-specific corpus.
I am wondering if lora_finetune.py can be used for that.

My idea is as follows:

• dataset: use DataCollatorForSupervisedDataset instead of RawConversationDataset
• loss: loss defined in lora_finetune.py seems to be loss on all the tokens passed to the model. Is my understanding correct? If, so I do not need to do anything on this.

It would be helpful if I am missing something.
Thank you very much for you help!