Shard non divisible batch size

[long21wt]

Hi,

I'd like to sample my batch according to Poisson sampling, therefore the batch size might result in an indivisible number of local devices to be shard.

• Normal Sampling
  • devices = 2, all batch size = 10 → OK
• Poisson Sampling
  • devices = 2, a single batch size = 10 → OK
  • devices = 2, a single batch size = 5 → Failed

Is there a workaround for my case?
Thanks

[cgarciae]

Hey @long21wt, when using batch sizes that are not divisible by the number of devices the solution is usually to use form some of padding + masking. Checkout our Processing the entire Datase guide for some pointers.

[long21wt]

Thanks, seems like padding is the easiest way.