`/dev/shm` exhaustion with async sandbox cleanup

[brians-neptune]

Description of the bug:

After upgrading to Bazel 8.1.1, I tried removing --experimental_sandbox_async_tree_delete_idle_threads=0 (which I had been using due to #20965). This resulted in builds filling up /dev/shm and then failing (along with other applications on my computer which rely on /dev/shm crashing, ie Chrome).

I see the memory usage climb rapidly during a build, and then after stopping it takes around 10 seconds to free the 32G it allocated (or proportionately less if I stop the build before it fills up).

Some .bazelrc settings which may be relevant:

common --noincompatible_sandbox_hermetic_tmp
common --sandbox_default_allow_network=false
common --incompatible_strict_action_env
common --heap_dump_on_oom
common --sandbox_base=/dev/shm/ --reuse_sandbox_directories
common --watchfs
common --sandbox_fake_hostname=true
common --sandbox_fake_username=true
common --spawn_strategy=linux-sandbox
common --strategy=CopyFile=standalone
common --strategy=CopyDirectory=standalone
common --strategy=TestRunnerSbom=local

Which category does this issue belong to?

Local Execution

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

No response

Which operating system are you running Bazel on?

Debian 12 bookworm

What is the output of bazel info release?

release 8.1.1

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse HEAD ?

Proprietary codebase

If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

[oquenchil]

The mitigation from the bug you linked helped a bit but it was still tight (#20965 (comment)).

I'd say that it's a legitimate use case to not have async deletion if your build and system cannot support it. In that case though the flag experimental_sandbox_async_tree_delete_idle_threads should be renamed to remove the experimental_* prefix.

Currently, there is 1 thread deleting files asynchronously and the value passed with the flag is the number of threads that will be used for deletion while the server is idle. The number of threads performing deletions during the build could potentially be bumped but this will take away resources from the running actions. I think if this will be better or worse will depend a lot on the build.

For now I'd suggest you leave synchronous deletion.