What's the relationship between number of gpu and batch size (global batch size))

[HuangChiEn]

Yesterday, I saw the code 🔗
, which given the comment at the last cell.

To improve the performance of the model, train with multi-GPU and a global batch size of 24. So if you use 8 GPUs with trainer.gpus=8, set model.train_ds.batch_size=3

In tensorflow, we need to calculate the global batch size in multi-gpu settings. Is the same concept also applied in the multi-gpu of trainer ?

If we setup trainer = Trainer(**{'devices':4, 'accelerator':'gpu', 'strategy':'ddp'}) with the batch size 64, will the model benefit with the larger batch size (256), or the model only benefit with batch size (64) ?

# sample code snippet :
tra_ld = DataLoader(train_dset, batch_size=32, ... )
trainer = Trainer(**{'devices':4, 'accelerator':'gpu', 'strategy':'ddp'})
trainer.fit(model, tra_ld)

[akihironitta]

Hi @HuangChiEn, in your case shown in the sample code, the actual batch size will be 128 (32 * 4).

It behaves differently depending on which strategy is used. You can read more about it

• in our docs: https://pytorch-lightning.readthedocs.io/en/stable/accelerators/gpu.html#batch-size
• in PyTorch forum: https://discuss.pytorch.org/t/do-dataparallel-and-distributeddataparallel-affect-the-batch-size-and-gpu-memory-consumption/97194

[HuangChiEn]

TKS for your explanation.