Refactor build system (with SCons, probably… unless?)

[Repiteo]

Our current integration of SCons, the build system used by the repo, is built atop a foundation of hacks. The most recent example being godotengine/godot#101726, which applies a fix for a hack (godotengine/godot#98105) which was a fix for a hack (godotengine/godot#91847) which expanded on a workaround (godotengine/godot#79871) for a hack that existed before the repo became open source. This sort of domino-effect integration makes me sympathize with those that are frustrated with this build system, because anyone would be rightfully confused and overwhelmed with this system.

The more I look into SCons' documentation and dig into their codebase, the more I realize that we were never using this tool to its full potential. We have custom platform & module systems and setup that requires manual integration, when SCons has builtin equivalents of both. We bind functions to the main environment, when SCons supports creating a custom class entirely. We parse options for our builds after the environment is already built, when all of that could be pre-processed with either a dummy environment or no environment whatsoever. We have validators and converters for countless variables, but have never once integrated them as native parts of the SCons variables themselves. We haven't just reinvented the wheel, we've Ship-of-Theseus'd the entire build system!

So when people suggest that our current system should be completely redone to something more modern and accessible, I find myself wholeheartedly agreeing with them — SCons should absolutely be uprooted and replaced!

…With SCons! _{wait what?}

---

Hot take: SCons is a damn good build system & sticking with it is entirely reasonable. But sticking with it as it's currently implemented isn't sustainable long-term. As such, I propose a full refactor of our SCons integration.

Yes, that includes all 170+ SCons-specific scripts.
Yes, that includes all 100+ Python helper scripts.

If that scope sounds absurd… Well, yeah, it is. But this is a bandage that needs to be ripped off eventually, so I'd prefer to be proactive with it. Still, that's a tall order. What kind of benefits could even come from that which wouldn't be feasible with the system as it currently exists? Well:

1. Build multiple platforms/architectures simultaneously. This is the biggest one by far, and I'm 100% convinced it's impossible with the current implementation. SCons' environment system has full support for declaring specific options on individual environments, so declaring them early enough will allow for multiple builds to occur at once. Not one-after-the-other — at once. I say it's impossible right now, because we kinda walked ourselves into a corner by not respecting the original file layouts. The original version of the SConstruct script in the repo indicates that multi-building was a thing in this repo at some point (the FAQ page seems to still think this, which… no), but the amount of hacks put into the main environment itself got to a point that it's functionally inseparable — or, at least, the effort required would be on the same scale as a full refactor anyway.
2. Treat modules/platforms as Tools. godot-cpp has a bit of a head-start on this one, but we can seriously streamline a lot of the logistics of isolating our different platforms and modules by making them be parsed as the SCons-native Tool. It'd mean less hacky imports in SConstruct, as all the environment calls would be with the integrated methods and callbacks. This would perfectly play into the above change, as these systems would not play nice with a baseline environment that is constantly changing its options/arguments mid-construction.
3. Validate & convert arguments before an environment is created. Another really cool feature SCons has out-of-the-box is the ability to pass entirely custom validators and convertors to variable options. We've gotten a taste of these with stuff like BoolVariable and EnumVariable, but those are generic cases; we'd be able to make our own generic types or even specialized, per-option types to really consolidate the logic. While technically feasible with the current systems, it'd be extremely painful to try and retrofit onto the existing options, as we'd have to ensure that we aren't breaking backwards compatibility.
4. Breaking backwards compatibility. This is a bit weird to tout as a "feature", but whatever. A full refactor means that we won't have to walk on eggshells trying to maintain weird design decisions from half a decade ago. Out with the old, in with the new.

This is a non-exhaustive list, but I think you get the idea. I'm all aboard with breaking a few hundred eggs for this omlette, but I still wanna hear what other people think. Even if this is pipedream-territory, might as well list out all the nice-to-haves that everyone's sitting on.

---

For posterity — because talk of build system refactors often cause suggestions for alternatives — let's make it as unambiguous as possible what would be necessary to replace SCons as the build system. From where I sit, it would necessitate:

• Never requiring manual cleanup.
• Building for multiple platforms/architectures simultaneously.
• Capable of building everything, from fresh slate, with a single command.
• Automatically converts & validates passed arguments.
• Painless to download/setup the tool itself; bonus points if distros have it by default.
• Some sort of cache system, ideally builtin.
• A scripting system, or a seamless means of binding to one.

[huwpascoe]

Godot is the only project that still uses SCons, there's no future for that build-system
Godot and MongoDB uses SCons.

CMake

• Never requiring manual cleanup.
• Building for multiple platforms/architectures simultaneously.
• Capable of building everything, from fresh slate, with a single command.
• Automatically converts & validates passed arguments.
• Painless to download/setup the tool itself; bonus points if distros have it by default.
• Some sort of cache system, ideally builtin.
• A scripting system, or a seamless means of binding to one.

Plus

• compiler toolchains
• industry standard (used by android native SDK)
• well documented
• full testing suite, can use doctest
• extensive tooling and IDE integration

[AThousandShips]

I haven't called anyone out

Except:

Don't assume just because it works on my machine™ that everyone has the same experience. Don't dismiss other people's experience with "liar", that's incredibly close-minded. Be better.

[Calinou]

Regarding SCons rebuilds, it will happen if you use different parameters that don't use a different binary name (e.g. optimize=speed vs optimize=size). However, it shouldn't happen with target=editor versus target=template_debug or target=template_release because these use different binary name (and different object names for the .o/.obj files).

We could add the optimization level to the object names to avoid this, but I don't know if it can be done in a way that doesn't affect the binary name. Right now, the object names always match the binary name.

[huwpascoe]

Except:

Except the part where it was necessary in order to address that very behaviour, yes. I'm aware of the irony. 😛 But seriously, stop now.

---

However, it shouldn't happen with target=editor versus target=template_debug or target=template_release because these use different binary name

There they are, .windows.editor.dev.x86_64.lib... I should test to see what works and what doesn't.

[Faless]

We could add the optimization level to the object names to avoid this, but I don't know if it can be done in a way that doesn't affect the binary name. Right now, the object names always match the binary name.

That can be done by forcing the PROGSUFFIX (and LIBSUFFIX when generating android/ios lib), but we really shouldn't imo.

If you need that for custom builds, you can use extra_suffix, otherwise, we can just implement variant_dir, so you can do out-of-tree builds (build_o3/, build_os/, etc).

[huwpascoe]

After a lot of ...mostly reinstalling things, it's now down to only 150~ files per change - half of which are skipped because cache or something - which is much more acceptable. Hopefully the refactor can reduce this further.

[Ivorforce]

I don't have much to say with regards to build systems, but i will say this: The proposal to refactor our use of scons may mean a lot of work, but it is a lot of work that can be approached incrementally.
A switch to a different build system would have us maintain two at once until we can hard switch. That's a big risk (the "big rewrite just over the horizon" problem).

To the proposal itself: Fully agreed; there is a lot of unnecessary complexity, some of which could definitely be reduced through more idiomatic SCons use!

[Repiteo]

The proposal to refactor our use of scons may mean a lot of work, but it is a lot of work that can be approached incrementally.

Ehhh, not exactly. While we can make incremental improvements on smaller scales, there's the aforementioned issue of not breaking backwards compatibility, which comes up much more than you'd expect.

A switch to a different build system would have us maintain two at once until we can hard switch. That's a big risk (the "big rewrite just over the horizon" problem).

Maintaing two buildsystems at once wouldn't be a thing; it'd be an immediate hard switch or nothing. That's a brutal caveat, but I ultimately agree: that burden should be wholly isolated to the implementation PR.

[Ivorforce]

That isn't the way I usually see big changes go, but I guess it wouldn't be new for Godot!

[stuartcarnie]

My first recommendation would be to start separating the Python code that is responsible for code generation from the code responsible for building. The goal would be that we could call all the Python manually and directly from the shell without any dependency on scons. I've managed to do that on my branch to a degree, with stand-ins for the scons environment as a hack. Once we've separated the concerns of code generation from building, it would be easier to start experimenting with alternative build systems, such as CMake, on the main branch.

The Python for code generation will require inputs from the build system, but those can then be provided when invoked from alternative system or the shell.

[huwpascoe]

👍 Standard argv is the way

[stuartcarnie]

Yes – agree. That is how I'm doing it in my branch

[Repiteo]

Separating that code is a very reasonable first step. We already have separate files for build functions for at least half of the generators, but that could go a step further & have zero overlap with SCons by only passing pre-parsed arguments. It should be reasonably straightforward to integrate this, albeit a bit of a timesink; will look into a draft PR over the next couple of days, time permitting

[stuartcarnie]

That would be amazing. I would update my branch and CMake scripts to leverage the changes, which would be a good test case that we've separated. I can help with subsequent PRs too.

[AThousandShips]

I think that regardless of what we do it's going to be a major undertaking, with a lot of issues possibly popping up, however SCons serves us well

IMO spending the same amount of work on cleaning up and improving our use of SCons would be a much better use of our time, as suggested here, and I'd argue we'd spend less time and work on doing that than entirely swapping systems, even ignoring fixing bugs popping up down the line due to migrating

Not to mention the demand migrating systems places on people working with the buildsystem having to re-learn a different one

Having worked with CMake in the past I find SCons beautifully simple to use, even setting up relatively complicated options is pretty smooth, in the past I've had to constantly fight CMake to make it do what I want

But to me one of the greatest strengths of SCons is how self contained it is, sure it might not always offer as good solutions as other tools do, but it generally guarantees that it is available where ever you can run it, no pain because you can't find a version of a particular tool or library on some niche distro (or Windows, at all, CMake has been even more of a pain there IMO)

tl;dr; Sticking with SCons, and spending work on cleaning it up and improving things, is a safer and more realistic option IMO, it works from a known starting point we know works, meaning we're not stuck a few months after migrating with a breaking issue

[stuartcarnie]

That makes sense @AThousandShips – as I proposed earlier, if we were to separate the Python code responsible for generating with the code that is responsible for building, it would make it easier to explore other build tools without having to change or replace scons. The separation of concerns would benefit us in other ways, as we could then do things like manually run the scripts to generate code and verify the output without having to go through the build system.

[Repiteo]

@stuartcarnie We do something like that already with the shader builders, albeit in a limited scope; they have a dedicated pytest static check on CI. Granted, that implementation could use a makeover, but the precedent is there!

[huwpascoe]

It'd be good to have a sort-of-official repo with it's own issues page that provides cmake files. Then SCons can stay and there are alternative build options for anyone who wants it.

[AThousandShips]

Why? Why do extra work? Maintaining two parallel build systems is unreasonable work

If you want to make one you're welcome to do so

[LouChiSoft]

This might be getting ahead of myself here but is there a chance that this could be the groundworks for a Godot version 5? I know that Godot 4 isn't all that old but the way I see it is if you want to break compatibility with basically every version of Godot before it then making it a version 5 makes it clear to users that this is more than just some new features and performance improvements.

And since the build system is quite literally the foundation that everything else is built on top of it would seem a good place to start. A freshly reworked build system that better enables other compatibility breaking changes to be made on top of it. I see no reason why this couldn't be done in a "godot-next" branch while the main branch continues as normal.

P.S. No matter what happens the one thing I would love is better examples of how to use Sconstruct with Godot. As it stands I have had way more issues with Scons than I've ever had with CMake, and while most of that will be down to experience it still doesn't help that most of the documentation is just "look at another file that does the same thing". But that doesn't help when there isn't an example of it being done, e.g. linking a pre-compiled static library directly into Godot via modules rather than building from source.