Error while using custom DistributedSampler

[karthi0804]

I wanted to set shuffle to False. So, i tried

sampler = torch.utils.data.distributed.DistributedSampler(dataset, shuffle=False)
dataloader = DataLoader(dataset, batch_size=32, sampler=sampler)

I am getting an error

RuntimeError: Default process group has not been initialized, please make sure to call init_process_group.

Please anyone tell me how to use custom sampler.

[carmocca]

You should set shuffle=True in your DataLoader directly

Regarding the error, can you provide a script to reproduce it?

You can adapt https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pl_examples/bug_report_model.py

[karthi0804]

Thanks.

[jmlemercier]

I'm quite interested in a proper answer, as replace_sampler_ddp=False seems not to be the only proper workaround to DDP automatically shuffling the batches in the training DataLoader. If I set this option, and do not specify a DistributedSampler for the plain DataLoader I am using, I am getting the expected behaviour, that is, when shuffle=False, that every split of the batch corresponding to a single GPU uses the same data...

So the question is basically, from my side : "How to either being able to set shuffle=False for the training DataLoader when using accelerator='ddp', or how to properly use DistributedSampler (when setting replace_sampler_ddp=False) without having to go through the abstruse setup of a ProcessGroup / distributed application ?"

[awaelchli]

Adding on to the existing answer:

DataLoader(shuffle, sampler) are mutually exclusive, i.e., if you set shuffle=True you will get a RandomSampler and if it is set to False you get a SequentialSampler.

When using DDP, Lightning takes your dataloader and replaces it with the following
DataLoader(sampler=DistributedDampler(shuffle=True), ...), however ONLY if the sampler is not already a distributed sampler. If it is, no changes are done.

So OP can do the following:

def train_dataloader(self):
    return DataLoader(sampler=DistributedSampler(shuffle=False), ...)`

returning their own sampler. Note this needs to be done in a place where the distributed group is already initialized, so basically in any hook after (including) setup(). OP got this error

RuntimeError: Default process group has not been initialized, please make sure to call init_process_group.

Because they probably did the following:

dataloader = DataLoader(sampler=DistributedSampler(shuffle=False), ...) # fails here, because distributed not init yet
trainer = Trainer()
trainer.fit(dataloader)

[jmlemercier]

Many thanks for this answer. So actually if you'd want to modify the sampler you'd need to define the (Distributed)DataLoader(s) as a hook in the LightningModule model, correct ? (Thanks for the clear answer on the distributed instances thing)

[awaelchli]

You are welcome. Yes, that's correct. If you just put the line
sampler = DistributedSampler(dataset=..., shuffle=False) anywhere at the start of your script (doesn't have to be lightning) you will see the error

RuntimeError: Default process group has not been initialized, please make sure to call `init_process_group`.

because as the error says, the user would have to call this function init_process_group() from torch distrib utils.
Lightning does this for the user under the hood, but unfortunately we cannot prevent anyone from initializing a distributed sampler before Lightning has the chance to init the process group and run into this issue.

[jmlemercier]

Right many thanks