TLS handshake error in project manager and asset library on custom Linux builds with `dev_build=yes`/`optimize=none` and TLS v1.3 enabled

[akien-mga]

Tested versions

• Reproducible in 4.5.beta (d89f4ab)
• Reproducible in 4.4.2.rc (e531f3e)
• Not reproducible in 4.3.1.rc (63c6165)

System information

Fedora Linux 42 (KDE Plasma Desktop Edition) on Wayland - X11 display driver, Multi-window, 2 monitors - OpenGL 3 (Compatibility) - AMD Radeon 780M Graphics (radeonsi, phoenix, LLVM 20.1.6, DRM 3.63, 6.15.3-200.fc42.x86_64) - AMD Ryzen 7 8845HS w/ Radeon 780M Graphics (16 threads) - 60.61 GiB memory

Issue description

When compiling Godot from source with dev_build=yes on Fedora 42, I get some mbedTLS "TLS handshake error" messages whenever TLS is used.

This is visible e.g. in the Project Manager when it checks for Godot updates:

ERROR: TLS handshake error: -27648
   at: _do_handshake (modules/mbedtls/stream_peer_mbedtls.cpp:88)
mbedtls error: returned -0x6c00

Or in the Asset Library (same error code):

According to @pythongirl325 this is also reproducible on Arch Linux: #107754 (comment)

I can reproduce this in 4.4 and 4.5, but not 4.3, so I suspect this is related to enabling support for TLS v1.3 in 4.4:

• [mbedTLS] Enable TLS 1.3 support #96394
  And so not directly linked to the update to Mbed TLS 3.6 per se, which happened in Godot 4.3 already, but without the PSA/TLSv1.3 stuff.

Toggling the network/tls/enable_tls_v1.3 editor setting doesn't seem to change the behavior, but I have a suspicion that this setting might be broken as the .3 might be interpreted as a feature override (at least it's styled as such in the editor settings).

I debugged this a bit more a few weeks ago but I can't find the chat logs about it. IIRC the problem was not dev_build=yes per se, but optimize=none which it enables. So mbedtls without optimization would end up being broken.

CC @Faless

Steps to reproduce

• On Fedora 42 or Arch Linux: scons dev_build=yes
• Open project manager and see errors

Minimal reproduction project (MRP)

n/a

[akien-mga]

I debugged this a bit more a few weeks ago but I can't find the chat logs about it. IIRC the problem was not dev_build=yes per se, but optimize=none which it enables. So mbedtls without optimization would end up being broken.

I confirmed this, scons optimize=none is enough to reproduce the issue.

Toggling the network/tls/enable_tls_v1.3 editor setting doesn't seem to change the behavior, but I have a suspicion that this setting might be broken as the .3 might be interpreted as a feature override (at least it's styled as such in the editor settings).

I tested some more:

• The setting does seem to work despite the dot in the name. I tried renaming it to enable_tls_v1_3 and saw similar behavior. I'd suggest we should do this rename anyway to be on the safe side.
• Disabling TLS v1.3 does prevent the issue in the editor for the Asset Library. It doesn't solve the issue for the Project Manager, which seems to imply that the Project Manager doesn't respect editor settings.

[akien-mga]

Depending on which $CFLAGS Arch Linux uses, #107754 may still be the same issue as this bug in the end, as they pass custom flags here:
https://gitlab.archlinux.org/archlinux/packaging/packages/godot/-/blob/main/PKGBUILD?ref_type=heads#L60-62

Which should override the one we set with optimize. I doubt they pass -O0 though, usually distros use -O2 or -O3.

Edit: I checked which flags Arch Linux uses in its /etc/makepkg.conf:

CFLAGS="-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions \
CXXFLAGS="$CFLAGS -Wp,-D_GLIBCXX_ASSERTIONS"

This is pretty standard and similar to our default for optimize=speed_trace (-O2).

Edit 2: Actually cflags/cxxflags don't override optimize, they're added before, not after. One needs to use optimize=custom to do let cflags/cxxflags take precedence.