Rust fails to build projects about half the time on Mac with VirtioFS enabled

[dslatkin]

When using the Rust dev container template which is customized almost entirely from this image, a freshly cargo inited project randomly will fail to build about half the time on my basic setup. Disabling VirtioFS lets the builds work reliably.

Supporting VirtioFS is greatly preferred since it presents significant file system speed improvements on Mac. It's been generally available (no longer "experimental") since Docker Desktop 4.15.0 (released Dec '22) and is the default since Docker Desktop 4.23 (Sep '23).

Reproduction

1. Make sure to use Mac with Docker Desktop
   • In my case it's on a MacBook Air 13-inch with M2
   • I've recreated with both Docker Desktop 4.23.0 and 4.24.0 (most up-to-date as of writing this)
2. Make sure VirtioOS is enabled in your Docker engine
3. Create a new project directory
4. Add the default Rust dev container definition
5. Build and open the project in the dev container
6. Run cargo init to initialize a new "hello world" project
7. Repeat the command rm -fr target && cargo build notice it fails some of the time
   • rm -fr target ensures the build happens every time for purposes of this repro

It usually takes about 5 successful builds at most until the first failure happens.

8. Reopen the folder locally
9. Switch to gRPC FUSE file implementation in Docker engine settings
10. The window should have disconnected when Docker restarts, reload the window to reconnect
11. Repeat the command rm -fr target && cargo build, notice it will no longer fail

I'm not able to find any pattern as to when builds will fail, it seems entirely random.

• I'm not using any special customizations to the container/image
• Docker has plenty of dedicated resources when reproducing this (10 GB RAM and ~100 GB storage)
• It does matter using Rosetta for x86/amd64 emulation, enabling and disabling it doesn't help

Logs

Successful builds

With VirtioFS disabled (gRPC FUSE enabled):

$ while rm -fr target/ && cargo build; do :; done
   Compiling example v0.1.0 (/workspaces/example)
    Finished dev [unoptimized + debuginfo] target(s) in 0.99s
   Compiling example v0.1.0 (/workspaces/example)
    Finished dev [unoptimized + debuginfo] target(s) in 0.99s
   Compiling example v0.1.0 (/workspaces/example)
    Finished dev [unoptimized + debuginfo] target(s) in 0.97s
   Compiling example v0.1.0 (/workspaces/example)
    Finished dev [unoptimized + debuginfo] target(s) in 1.00s
   Compiling example v0.1.0 (/workspaces/example)
    Finished dev [unoptimized + debuginfo] target(s) in 0.99s
   Compiling example v0.1.0 (/workspaces/example)
    Finished dev [unoptimized + debuginfo] target(s) in 0.99s
   Compiling example v0.1.0 (/workspaces/example)
    Finished dev [unoptimized + debuginfo] target(s) in 1.03s
   Compiling example v0.1.0 (/workspaces/example)
    Finished dev [unoptimized + debuginfo] target(s) in 0.98s
   Compiling example v0.1.0 (/workspaces/example)
    Finished dev [unoptimized + debuginfo] target(s) in 0.99s
   Compiling example v0.1.0 (/workspaces/example)
    Finished dev [unoptimized + debuginfo] target(s) in 0.99s
   Compiling example v0.1.0 (/workspaces/example)
    Finished dev [unoptimized + debuginfo] target(s) in 1.00s
   Compiling example v0.1.0 (/workspaces/example)
    Finished dev [unoptimized + debuginfo] target(s) in 0.99s
   ...

Failed builds

With VirtioFS enabled:

$ while rm -fr target/ && cargo build; do :; done
   Compiling example v0.1.0 (/workspaces/example)
    Finished dev [unoptimized + debuginfo] target(s) in 0.29s
   Compiling example v0.1.0 (/workspaces/example)
    Finished dev [unoptimized + debuginfo] target(s) in 0.31s
   Compiling example v0.1.0 (/workspaces/example)
error: linking with `cc` failed: exit status: 1
  |
  = note: LC_ALL="C" PATH="/usr/local/rustup/toolchains/1.72.1-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/bin:/vscode/vscode-server/bin/linux-arm64/e7e037083ff4455cf320e344325dacb480062c3c/bin/remote-cli:/usr/local/cargo/bin:/usr/local/cargo/bin:/usr/local/cargo/bin:/usr/local/cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/vscode/.local/bin" VSLANG="1033" "cc" "/tmp/rustc0jtGY9/symbols.o" "/workspaces/example/target/debug/deps/example-286854bdcea43dbf.16ymvqt8cxxyt04w.rcgu.o" "/workspaces/example/target/debug/deps/example-286854bdcea43dbf.1i5hqwsdxzrrmsga.rcgu.o" "/workspaces/example/target/debug/deps/example-286854bdcea43dbf.29r392z43xk4ppmb.rcgu.o" "/workspaces/example/target/debug/deps/example-286854bdcea43dbf.2eb8j4q71itcksuj.rcgu.o" "/workspaces/example/target/debug/deps/example-286854bdcea43dbf.3i82ksilnfvdli6v.rcgu.o" "/workspaces/example/target/debug/deps/example-286854bdcea43dbf.4gx83nd82i5seyfy.rcgu.o" "/workspaces/example/target/debug/deps/example-286854bdcea43dbf.155fsdtb1bfio2qj.rcgu.o" "-Wl,--as-needed" "-L" "/workspaces/example/target/debug/deps" "-L" "/usr/local/rustup/toolchains/1.72.1-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/usr/local/rustup/toolchains/1.72.1-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libstd-cca993492431c26c.rlib" "/usr/local/rustup/toolchains/1.72.1-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libpanic_unwind-767683484988fddb.rlib" "/usr/local/rustup/toolchains/1.72.1-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libobject-33d8ca36e0751c49.rlib" "/usr/local/rustup/toolchains/1.72.1-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libmemchr-feb6369d9fb8ea6a.rlib" "/usr/local/rustup/toolchains/1.72.1-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libaddr2line-da5e802c6b261da9.rlib" "/usr/local/rustup/toolchains/1.72.1-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libgimli-c11c22644b73fbb3.rlib" "/usr/local/rustup/toolchains/1.72.1-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_demangle-b538894d8056db90.rlib" "/usr/local/rustup/toolchains/1.72.1-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libstd_detect-5f75957d64265585.rlib" "/usr/local/rustup/toolchains/1.72.1-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libhashbrown-eb377e4796f89c63.rlib" "/usr/local/rustup/toolchains/1.72.1-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-7959bd6dc5a6dc9b.rlib" "/usr/local/rustup/toolchains/1.72.1-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libminiz_oxide-27e9cf0ad28ebc46.rlib" "/usr/local/rustup/toolchains/1.72.1-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libadler-9a4d8927e95998a8.rlib" "/usr/local/rustup/toolchains/1.72.1-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libunwind-7ce72c5f7491b638.rlib" "/usr/local/rustup/toolchains/1.72.1-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libcfg_if-88468c0375a0c64b.rlib" "/usr/local/rustup/toolchains/1.72.1-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/liblibc-ceea9f1fc4b03431.rlib" "/usr/local/rustup/toolchains/1.72.1-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/liballoc-96085dbefe195fa8.rlib" "/usr/local/rustup/toolchains/1.72.1-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_std_workspace_core-a8add003da8dc54d.rlib" "/usr/local/rustup/toolchains/1.72.1-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libcore-082d5e9d53946848.rlib" "/usr/local/rustup/toolchains/1.72.1-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libcompiler_builtins-21239b3d9ec3b48b.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "/usr/local/rustup/toolchains/1.72.1-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-o" "/workspaces/example/target/debug/deps/example-286854bdcea43dbf" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-nodefaultlibs"
  = note: /usr/bin/ld: cannot find /workspaces/example/target/debug/deps/example-286854bdcea43dbf.1i5hqwsdxzrrmsga.rcgu.o: No such file or directory
          collect2: error: ld returned 1 exit status
          

error: could not compile `example` (bin "example") due to previous error

Specs

Run on my host machine:

$ docker version
Client:
 Cloud integration: v1.0.35+desktop.4
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:28:49 2023
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.23.0 (120376)
 Engine:
  Version:          24.0.6
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.7
  Git commit:       1a79695
  Built:            Mon Sep  4 12:31:36 2023
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.22
  GitCommit:        8165feabfdfe38c65b599c4993d227328c231fca
 runc:
  Version:          1.1.8
  GitCommit:        v1.1.8-0-g82f18fe
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[samruddhikhandale]

Hi 👋

We had a similar issue opened sometime ago, see microsoft/vscode-dev-containers#1487. Turned out to be an issue with a specific version of VirtioFS.

To me, it looks like an issue with VirtioFS instead of the image? (which proves the point when cargo works with VirtioFS disabled). @dslatkin do you mind opening an issue with that team instead?

We are more than happy to accept any change required to the image to support VirtioFS out of the box. However, I am not sure what changes might be needed, hence, chatting and gaining more information from the respective team makes more sense. Let me know what you think, thanks!

[dslatkin]

@dslatkin do you mind opening an issue with that team instead?

Hi, thanks for your help! I'll go ahead and do that. I just opened an issue with the official Docker image for Rust as well since I can reproduce it there without needing an explicit dev container. I'm going to see if they recommend anything to try first or suggest I collect additional debugging information to share with VirtioFS before I open an issue there. I'll keep this issue updated.