Profiling/Limiting Java Virtual Machine Memory use in Bazel?

[Ryang20718]

Wondering if there's some workflow to check for what's contributing to the bazel server memory use?

I've looked at https://bazel.build/advanced/performance/memory and https://bazel.build/rules/performance#memory-profiling
to narrow down that it's not a matter of our rules nor is it our tests that are causing the bazel server to use a lot of system memory.
Mainly whenever I run ps aux | grep bazel when I'm not running a bazel command, it reports high memory use of 12 GB+ even though max Heap size is 12. Wondering if there's some flag to debug and know where bazel is eating up all this memory/flags to help tune the jvm for bazel?

[sluongng]

--host_jvm_args is the flag you are looking for.

It's quite common for folks to have something like this in their .bazelrc file

startup --host_jvm_args=-Xmx4g

Keep in mind you probably want to have a different value for your CI server where you should have more resources than local laptop.

[Ryang20718]

yeah, we set --host_jvm_args=-Xmx12g but ideally it shouldn't hit that nor should it exceed it

[Ryang20718]

Was hoping if there's a guide/previous knowledge regarding what GC to use/jdk to use to optimize a lower memory footprint for the server itself

[sluongng]

The action workers(sandbox, process-wrapper) are not counted as part of the JVM memory. So it's quite typical to see the overall system memory to be above Bazel JVM.

Are you saying the bazel JVM is exceeding the memory limit or are you talking about host's memory?

How does your ps awwfux | grep bazel look like?

[Ryang20718]

Run ps aux --sort=-%mem | head -n 20
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
dbazel        400 47.7 38.0 18943376 12346796 ?   Ssl  18:53  68:57 bazel

Some of our relevant flags (still need to run ps awwfux on our server)

--host_jvm_args=-Xmx12G
--jobs=8

[Ryang20718]

With the default GC, bazel hits a peak of 5.8 GB of memory and barely releases

However, after the run, I ran ps awwfux --sort=-%mem | head -n 20 and I'm seeing the bazel server holding onto 11+ GB of sys memory

USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
github_+       1  0.0  0.0   2552   220 ?        Ss   07:23   0:00 tail -f /dev/null
github_+     356 44.9 36.0 18786916 11695040 ?   Ssl  07:[24](https://github.com/waabi-ai/av/actions/runs/5302428703/jobs/9597346174#step:19:26)  58:27 bazel --add-opens=java.base/java.lang=ALL-UNNAMED -Xverify:none -Djava.util.logging.config.file=/home/github_actions/.cache/bazel/_bazel_github_actions/049fd0d9a142b0eee346c643b8cf35e6/javalog.properties -Dcom.google.devtools.build.lib.util.LogHandlerQuerier.class=com.google.devtools.build.lib.util.SimpleLogHandler$HandlerQuerier -XX:-MaxFDLimit -Djava.library.path=/github/home/.cache/bazel/_bazel_github_actions/install/97c81a3bdd984debe0ff1b26c2dc04e0/embedded_tools/jdk/lib/jli:/github/home/.cache/bazel/_bazel_github_actions/install/97c81a3bdd984debe0ff1b26c2dc04e0/embedded_tools/jdk/lib:/github/home/.cache/bazel/_bazel_github_actions/install/97c81a3bdd984debe0ff1b26c2dc04e0/embedded_tools/jdk/lib/server:/github/home/.cache/bazel/_bazel_github_actions/install/97c81a3bdd984debe0ff1b26c2dc04e0/ -Dfile.encoding=ISO-8859-1 -Xmx12G -jar /github/home/.cache/bazel/_bazel_github_actions/install/97c81a3bdd984debe0ff1b26c2dc04e0/A-server.jar --max_idle_secs=10800 --noshutdown_on_low_sys_mem --connect_timeout_secs=30 --output_user_root=/github/home/.cache/bazel/_bazel_github_actions --install_base=/github/home/.cache/bazel/_bazel_github_actions/install/97c81a3bdd984debe0ff1b26c2dc04e0 --install_md5=97c81a3bdd984debe0ff1b26c2dc04e0 --output_base=/home/github_actions/.cache/bazel/_bazel_github_actions/049fd0d9a142b0eee346c643b8cf35e6 --workspace_directory=/__w/av/av --default_system_javabase= --failure_detail_out=/home/github_actions/.cache/bazel/_bazel_github_actions/049fd0d9a142b0eee346c643b8cf35e6/failure_detail.rawproto --expand_configs_in_place --idle_server_tasks --write_command_log --nowatchfs --nofatal_event_bus_exceptions --nowindows_enable_symlinks --client_debug=false --host_jvm_args=-Xmx12G --product_name=Bazel --option_sources=host_Ujvm_Uargs:/github/home/.bazelrc

WIth Serial GC --host_jvm_args=-XX:+UseSerialGC , it seems to be slightly better, but still not great. Ideally the jvm running bazel should release memory that isn't used back to the system if the bazel server isn't using it right?

bazel server was using 5.8 GB of memory at the peak and at the end of the run at 2.0 GB of memory usage.

USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
github_+       1  0.0  0.0   2552   436 ?        Ss   00:58   0:00 tail -f /dev/null
github_+     343 30.8 21.9 18316860 7116368 ?    Ssl  00:58  39:44 bazel --add-opens=java.base/java.lang=ALL-UNNAMED -Xverify:none -Djava.util.logging.config.file=/home/github_actions/.cache/bazel/_bazel_github_actions/049fd0d9a142b0eee346c643b8cf35e6/javalog.properties -Dcom.google.devtools.build.lib.util.LogHandlerQuerier.class=com.google.devtools.build.lib.util.SimpleLogHandler$HandlerQuerier -XX:-MaxFDLimit -Djava.library.path=/github/home/.cache/bazel/_bazel_github_actions/install/97c81a3bdd984debe0ff1b26c2dc04e0/embedded_tools/jdk/lib/jli:/github/home/.cache/bazel/_bazel_github_actions/install/97c81a3bdd984debe0ff1b26c2dc04e0/embedded_tools/jdk/lib:/github/home/.cache/bazel/_bazel_github_actions/install/97c81a3bdd984debe0ff1b26c2dc04e0/embedded_tools/jdk/lib/server:/github/home/.cache/bazel/_bazel_github_actions/install/97c81a3bdd984debe0ff1b26c2dc04e0/ -Dfile.encoding=ISO-8859-1 -Xmx12G -XX:+UseSerialGC -jar /github/home/.cache/bazel/_bazel_github_actions/install/97c81a3bdd984debe0ff1b26c2dc04e0/A-server.jar --max_idle_secs=10800 --noshutdown_on_low_sys_mem --connect_timeout_secs=30 --output_user_root=/github/home/.cache/bazel/_bazel_github_actions --install_base=/github/home/.cache/bazel/_bazel_github_actions/install/97c81a3bdd984debe0ff1b26c2dc04e0 --install_md5=97c81a3bdd984debe0ff1b26c2dc04e0 --output_base=/home/github_actions/.cache/bazel/_bazel_github_actions/049fd0d9a142b0eee346c643b8cf35e6 --workspace_directory=/__w/av/av --default_system_javabase= --failure_detail_out=/home/github_actions/.cache/bazel/_bazel_github_actions/049fd0d9a142b0eee346c643b8cf35e6/failure_detail.rawproto --expand_configs_in_place --idle_server_tasks --write_command_log --nowatchfs --nofatal_event_bus_exceptions --nowindows_enable_symlinks --client_debug=false --host_jvm_args=-Xmx12G --host_jvm_args=-XX:+UseSerialGC