First checkpoint not being saved

[jwliu36]

Hi team,

I am trying to save all the checkpoints as well as the last.ckpt by setting save_last=True and save_top_k=-1. Additionally, I am also using AsyncCheckpointIO for async uploading checkpoints to local and S3 file paths.

However, I am running into issue where the first checkpoint-{epoch}-{step}.ckpt is not getting saved, but only last.ckpt is created. As the training job goes on, all subsequent checkpoint-{epoch}-{step}.ckpt would get saved into the same directory.

Can you point me to which method within ModelCheckpoint class that I may need to override?
Would it be _save_last_checkpoint: code ref or _should_skip_saving_checkpoint code ref? If it is other methods, please point me to the reference. Thank you!