How to compile certain cc_binary's with a different toolchain

[nickdesaulniers]

I have a codebase that's a mix of C++ for the host and target machines of different architectures (and some python just for the host).

Ideally, I'd like to continue using cc_library, cc_binary, and cc_test for target specific programs, but I'm looking for the most ergonomic way to say something to the effect of "this cc_binary should only be built for the target, and never the host."

I've mostly followed this tutorial. I'm also trying to mix that with the platforms concept.

In my BUILD.bazel, I tried adding:

constraint_setting(name = "mcpu_setting")
constraint_value(name = "my_cpu", constraint_setting = ":mcpu_setting")
platform(
    name = "my_platform",
    constraint_values = [":my_cpu"],
)
toolchain(
  ...
  toolchain = ":cc_my_toolchain",
  target_compatible_with = [
        ":my_cpu",
    ],
)

Building with bazel build //example:foo --toolchain_resolution_debug='@bazel_tools//tools/cpp:toolchain_type' --platforms=//bazel/toolchain/my_toolchain:my_platform, I can see my toolchain getting selected:

**ToolchainResolution:   Selected //bazel/toolchain/my_toolchain:cc_my_toolchain to run on execution platform //bazel/toolchain:clang**

But then I observe failures due to Python:

ERROR: ...external/rules_python+/python/BUILD.bazel:363:21: While resolving toolchains for target @@rules_python+//python:current_py_toolchain (c38586b): No matching toolchains found for types:
  @@bazel_tools//tools/python:toolchain_type

and adding --toolchain_resolution_debug='@@bazel_tools//tools/python:toolchain_type' shows that python rules know about lots of different platforms, but not my platform. At this point, I'm flabbergasted; I'm trying to build a cc_library here with no deps, who cares about python?

---

Building with --platforms=//bazel/toolchain/my_toolchain:my_platform is already way way too verbose to inflict on our developers. Surely there's a way to just specify in the cc_library rule to use a specific toolchain always?

target_compatible_with smells like that maybe might do what I want. Adding:

cc_libary(
  ...
  target_compatible_with = [
        "//bazel/toolchain/my_toolchain:my_cpu",
    ],

But then building without --platforms=//bazel/toolchain/my_toolchain:my_platform fails since the host compiler is selected:

ERROR: Analysis of target '//example:foo' failed; build aborted: Target //example:foo is incompatible and cannot be built, but was explicitly requested.
Dependency chain:
    //example:foo (be84d8)   <-- target platform (//bazel/toolchain:clang) didn't satisfy constraint //bazel/toolchain/my_toolchain:my_cpu

Rebuilding with --platforms=... causes the python failure above.

---

Is what I'm trying to do even possible with bazel? Is platforms the right way to do this, or something else that I'm missing? Thanks for taking a look!

[filmil]

Check out this video: https://www.youtube.com/watch?v=CrfQaZjuj_Y
I think it treats the issue you are having.

The issues you are observing are due to you either trying (1) to compile for the host platform, that your target is incompatible with; or (2) you try to compile everything for your target, for which you don't have the appropriate python toolchain configured.

Also note that, for example, cc_binary, has compatible_with = ... param.

And, as always, a minimal repro example would have been helpful, so take the above with a grain of salt.

[nickdesaulniers]

Thanks for the quick response!

Check out this video: https://www.youtube.com/watch?v=CrfQaZjuj_Y
I think it treats the issue you are having.

Hmm...it definitely covers similar ground that the platforms documentation covers. But I think my usage is slightly more advanced; I'm trying to combine the ideas from platforms and toolchains such that specific toolchains are used when cross compiling.

The issues you are observing are due to you either trying (1) to compile for the host platform, that your target is incompatible with; or (2) you try to compile everything for your target, for which you don't have the appropriate python toolchain configured.

Is it not possible to have one build where BOTH are built in one invocation of bazel? For example, I have cases where I need to cross compile a binary for a target architecture different than the host, but then test it on the host by feeding the target binary into another program compiled for the host to "test" the target binary.

Also, I'm not trying to compile everything; I'm invoking bazel build //example:foo which is a single cc_library with no-deps. I could maybe understand if what I was trying to build was python, but it's not. Maybe I need to define some fake platform for python just so the cross compilation of C++ can proceed (if so, seems like boilerplate or perhaps a misfeature of platforms)?

Also note that, for example, cc_binary, has compatible_with = ... param.

Whether I use target_compatible_with or simply compatible_with, I observe the same failures.

---

Also forgot to mention another thing I tried; cc_library can have a toolchains attr! But that doesn't seem to do what I'm looking for.

  cc_library(
    ...
    toolchains = [
        "//bazel/toolchain/my_toolchain:my_toolchain",
    ],

$ bazel build //example:foo
ERROR: example/BUILD.bazel:3:11: in toolchains attribute of cc_library rule //example:foo: '//bazel/toolchain/my_toolchain:my_toolchain' does not have mandatory providers: 'TemplateVariableInfo'

---

I guess what I'm looking for, what would be ideal is something like:

cc_binary(
  name = "foo",
   ...
   target_compatible_with = [
        "@platforms//cpu:arm",
   ],
)
cc_binary(
  name = "bar",
   ...
   target_compatible_with = [
        "@platforms//cpu:riscv",
   ],
)

Then have BOTH foo and bar build from one invocation of bazel, via distinct toolchains. I can do exactly that by hacking something up using rules, but I'm basically re-implementing a poor man's cc_binary all over again.

[filmil]

If you want to build the same target for multiple platforms in a single build invocation, I think you want to look into "platform transitions". They allow you to build a binary foo for some platform, then build a binary bar which is "same as foo but for a different platform". Transitions can be more general than that, but that's what I used them for.

This comes up in embedded development for example, as you may be aware - you have a library that you want to build for an embedded platform, but for testing you can also build it for use on the exec host. Unfortunately good examples are few and far between, but I think this may get you somewhere: https://github.com/bazel-contrib/bazel-lib/blob/main/docs/transitions.md. I do have a repo where I use this approach, but sadly it's not publishable. If you find yourself building for the same set of platforms all the time, someone cooked this up: https://stackoverflow.com/questions/72681255/how-to-properly-use-bazel-transitions-for-multiarch-build

[nickdesaulniers]

Ah, thanks for the tip. I was just reading the docs on transitions this morning, wondering if that's possibly what I'd need to use here. I had trouble wrapping my head around the syntax, but based on your confirmation it sounds like I should give them another shot. Thanks for confirming!

Unfortunately good examples are few and far between, but I think this may get you somewhere: https://github.com/bazel-contrib/bazel-lib/blob/main/docs/transitions.md

So I'd need to add aspect_bazel_lib as a dependency? I guess I can infer the usage from their tests, lol. I don't think they have an equivalent for cc_library, so I filed a feature request: bazel-contrib/bazel-lib#1065. Worst case, I could try to code up the rule locally, then go about upstreaming it.

If you find yourself building for the same set of platforms all the time, someone cooked this up: https://stackoverflow.com/questions/72681255/how-to-properly-use-bazel-transitions-for-multiarch-build

Yeah, I can kind of see how that might work for me. Let me give it a shot. Thank you so much for the links and discussion!

[fmeum]

You might be interested in https://github.com/fmeum/with_cfg.bzl

[nickdesaulniers]

You might be interested in https://github.com/fmeum/with_cfg.bzl

Thanks @fmeum , let me give that a shot.

Ok, so I tried:

# Module.bazel
bazel_dep(name = "with_cfg.bzl", version = "0.9.1")

# cc_toolchain_config.bzl
load("@with_cfg.bzl", "with_cfg")
xlibrary, _xlibrary = with_cfg(native.cc_library).set("platforms", "//example:my_platform").build()

# BUILD.bazel
load(":cc_toolchain_config.bzl", "xlibrary")
xlibrary( name = "...

But this fails

$ bazel build //example:foo
ERROR: example/BUILD.bazel:6:14: com.google.devtools.build.lib.packages.BuildFileNotFoundException: no such package '@@with_cfg.bzl+//example': BUILD file not found in directory 'example' of external repository @@with_cfg.bzl+. Add a BUILD file to a directory to mark it as a package.

Am I holding it wrong?

[fmeum]

You need to wrap all labels passed into with_cfg into Label(...) calls, otherwise they are resolved relative to the with_cfg repo.

[nickdesaulniers]

Ah, this might be an issue for the ergonomics I seek:

NOTE: There is currently no way to attach Starlark transitions to native rules. If you need to do this, contact [email protected] for help with figuring out workarounds.

So I'm guessing then I will still need a rule vs being able to just use cc_binary?

You might be interested in https://github.com/fmeum/with_cfg.bzl

Thanks @fmeum , let me give that a shot.

---

If you find yourself building for the same set of platforms all the time, someone cooked this up: https://stackoverflow.com/questions/72681255/how-to-properly-use-bazel-transitions-for-multiarch-build

I wired up exactly this, but I still hit that error relating to python.

ERROR: ...external/rules_python+/python/BUILD.bazel:363:21: While resolving toolchains for target @@rules_python+//python:current_py_toolchain (c38586b): No matching toolchains found for types:
  @@bazel_tools//tools/python:toolchain_type

it's as if as soon as I specify a non-host platform (explicilty, or via transitions), bazel errors because rules_python (which should be irrelevant for a cc_library) don't recognice my custom platform.

[filmil]

So I'm guessing then I will still need a rule vs being able to just use cc_binary?

FWIW there is now a starlark implementation of rules_cc, so this shouldn't be an issue for your cc_* targets if you are using rules_cc. See https://github.com/bazelbuild/rules_cc

[nickdesaulniers]

Is that a re-implementation from the native cc rules?