How to set the number of parallels when using scatter

[kongxiangya]

For example, when we run 100 samples and use scatter, we add --parallel at runtime. In reality, 100 samples will be submitted simultaneously at runtime, resulting in insufficient system memory. Is there a parameter that can set the number of allowed parallels? For example, if set to 20, only 20 samples will be run at a time. When a sample is completed, other samples will be run, and overall, the running samples will be kept to be less than or equal to 20

If this configuration is not currently available, where would you suggest adding it and how to write it appropriately

thanks

[tetron]

There isn't an option for that, although it is something that has been discussed before.

When using cwltool --parallel it does do basic resource management and won't run more simultaneous tasks than available RAM or cores. So for the basic case, you might already be covered.

There's also a couple of possible versions of limiting parallelism. You could cap the total number of simultaneous processes, or cap the fan-out of a specific scatter step. These have different implications for scheduling and the order that things will be executed.

[kongxiangya]

There isn't an option for that, although it is something that has been discussed before.

When using cwltool --parallel it does do basic resource management and won't run more simultaneous tasks than available RAM or cores. So for the basic case, you might already be covered.

There's also a couple of possible versions of limiting parallelism. You could cap the total number of simultaneous processes, or cap the fan-out of a specific scatter step. These have different implications for scheduling and the order that things will be executed.

Thank you for your answer

When using cwltool --parallel it does do basic resource management and won't run more simultaneous tasks than available RAM or cores.

Does this mean that the`cwltool --parallel has already considered RAM or cores when delivering tasks, and will not submit when the limit is exceeded？

You could cap the total number of simultaneous processes, or cap the fan-out of a specific scatter step.

Is this an existing configuration? Could you please provide an example to illustrate？

[mr-c]

Does this mean that the`cwltool --parallel has already considered RAM or cores when delivering tasks, and will not submit when the limit is exceeded？

Yes, see

cwltool/cwltool/executors.py

Line 275 in 16393f0

def select_resources(

Note that currently, the amount of available memory and cores is only checked at the beginning. It is not updated during the workflow run.

cwltool/cwltool/executors.py

Lines 270 to 271 in 16393f0

	self.max_ram = int(psutil.virtual_memory().available / 2**20)
	self.max_cores = float(psutil.cpu_count())

You could cap the total number of simultaneous processes, or cap the fan-out of a specific scatter step.

Is this an existing configuration? Could you please provide an example to illustrate？

No, that's not an existing configuration option. But it would be relatively straightforward to implement.

[hongduosun]

Thanks for developing cwltool!
And do we have any plan to allow user-specified max_cores or max_ram?
This could be straightforward to implement but is there any side effect to not doing so?

[mr-c]

And do we have any plan to allow user-specified max_cores or max_ram?

I haven't personally planned on implementing that. If someone else wants to work on that, I'm happy to help them.

This could be straightforward to implement but is there any side effect to not doing so?

I don't understand the question, sorry.