Add worker ramping option to prevent system overload when starting many workers

[twiggy]

Problem Description

When running tests with many workers (e.g., pytest -n 50), all worker processes start simultaneously. This can cause several issues:

• System overload: Sudden spike in CPU, memory, and I/O usage can make the system unresponsive
• Resource contention: All workers competing for resources at once can actually slow down test execution
• CI/CD failures: Resource-constrained environments may fail when too many processes start at once
• Database/service overload: If tests connect to external services, simultaneous connections can overwhelm them

Proposed Solution

Add a --ramp option that gradually starts workers over a specified time period, similar to JMeter's ramp-up period for load testing.

Example Usage

# Start 10 workers over 10 seconds (one worker per second)
pytest -n 10 --ramp 10s

# Start 50 workers over 5 minutes
pytest -n 50 --ramp 5m

# Start 100 workers over 1 hour  
pytest -n 100 --ramp 1h

How it would work

With --ramp 10s and -n 10:

• Worker 0: starts immediately
• Worker 1: starts after 1 second
• Worker 2: starts after 2 seconds
• ... and so on
• Worker 9: starts after 9 seconds

Benefits

1. Prevents system overload by distributing the resource usage spike over time
2. Better for shared environments where sudden resource spikes affect other users/processes
3. Improves test reliability in resource-constrained CI/CD environments
4. Useful for load testing scenarios where gradual ramp-up is desired
5. Backward compatible - no impact when --ramp is not specified

Implementation Considerations

• Support common time formats: seconds (s), minutes (m), hours (h)
• Workers should delay test execution, not just process initialization
• Provide progress feedback during ramp-up period
• Should work with all existing distribution modes (--dist)

Use Cases

1. Large test suites: Organizations with thousands of tests that need many workers
2. Shared CI/CD environments: Where resource usage must be controlled
3. Database-heavy tests: Preventing connection pool exhaustion
4. Performance testing: Gradually increasing load on the system under test
5. Development machines: Preventing system freeze when running tests locally

Related Issues/Context

This feature is similar to:

• JMeter's ramp-up period for thread groups
• Kubernetes' rolling deployments
• Database connection pool gradual scaling

I've been able to bang this out using AI hopefully with enough tests and documentation and have used it against my own suite. Creating this issue as a holder. Hoping to create PR in the near future.

[amezin]

Usually you have one or a few fixtures that do some heavy work (pytest worker itself isn't that heavy) - for example, start a Docker container, and that actually causes problems when done concurrently. In such cases, I wrap the fixture with some form of mutex/semaphore - for example, https://github.com/tox-dev/filelock

[twiggy]

In our case it's not the machines running pytest, but the systems it's impacting. Think e2e tests or api tests. Our jenkins instance can handle the pytest, selenium grid the chrome load, but we don't generally want to waste cash scaling out our system... we don't all have deep pockets. We do some warmup, but we see the first few tests blast and fail. --lf reruns them or flakey, but this is helping a lot and just generally a good strat when doing parallelism. When I hit it all local on a laptop this also helps. Its more convenience I could put it in my test harness as well but maybe other would find it useful or haven't even really considered it if they are not load engineers...e2e tests can create less artificial load but can cause initial herd type things.

[RonnyPfannschmidt]

The primary concern for xdist is parallelism not concurrency

Using 50 workers seems to be on the excessive side

Id recommend picking more conservative values

[twiggy]

So e2e tests often take several minutes to complete. We do these parallel so we don't take hours for the suite to complete. This is a very common pattern with e2e testing.

[RonnyPfannschmidt]

Ramp up is not part of the current design

50 workers are not something thats accounted for

The primary concern for xdist is parallel not concurrent