Skip to content

Commit 691149a

Browse files
bazel-iofmeum
andauthored
[7.2.0] Add a git merge driver for MODULE.bazel.lock (#22650)
Adds a `jq` script to `scripts/` that merges any number of `MODULE.bazel.lock` files without using Bazel or reading the corresponding `MODULE.bazel` files. The lockfile docs now have a section explaining the steps needed to set up this script as a custom merger driver for Git, which means that merge conflicts in `MODULE.bazel.lock` files will always be resolved automatically. Note that resolution may emit lockfiles with redundant information that will be dropped by subsequent Bazel invocations. When Bazel encounters an error during lockfile parsing that could be caused by a merge conflict, it emits a different error message with a link to the docs. This required fixing the following kind of server crash when a conflict marker occurs inside a `recordedFileInputs` object: ``` FATAL: bazel crashed due to an internal error. Printing stack trace: java.lang.RuntimeException: Unrecoverable error while evaluating node 'com.google.devtools.build.lib.bazel.bzlmod.BazelLockFileValue$$Lambda/0x000000f8011da998@314cd9ee' (requested by nodes 'RegistryKey{url=https://bcr.bazel.build/}') at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:557) at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:426) at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1403) at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387) at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312) at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843) at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808) at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188) Caused by: java.lang.IllegalArgumentException: the provided path should be absolute in the filesystem at com.google.common.base.Preconditions.checkArgument(Preconditions.java:143) at com.google.devtools.build.lib.rules.repository.RepoRecordedInput$RepoCacheFriendlyPath.createOutsideWorkspace(RepoRecordedInput.java:202) at com.google.devtools.build.lib.rules.repository.RepoRecordedInput$RepoCacheFriendlyPath.parse(RepoRecordedInput.java:222) at com.google.devtools.build.lib.rules.repository.RepoRecordedInput$File$1.parse(RepoRecordedInput.java:265) at com.google.devtools.build.lib.bazel.bzlmod.GsonTypeAdapterUtil$11.read(GsonTypeAdapterUtil.java:376) at com.google.devtools.build.lib.bazel.bzlmod.GsonTypeAdapterUtil$11.read(GsonTypeAdapterUtil.java:367) at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.read(TypeAdapterRuntimeTypeWrapper.java:41) at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:186) at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:145) at com.google.devtools.build.lib.bazel.bzlmod.DelegateTypeAdapterFactory$1.read(DelegateTypeAdapterFactory.java:133) at com.google.devtools.build.lib.bazel.bzlmod.LockFileModuleExtension_GsonTypeAdapter.read(LockFileModuleExtension_GsonTypeAdapter.java:171) at com.google.devtools.build.lib.bazel.bzlmod.LockFileModuleExtension_GsonTypeAdapter.read(LockFileModuleExtension_GsonTypeAdapter.java:17) at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.read(TypeAdapterRuntimeTypeWrapper.java:41) at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:187) at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:145) at com.google.devtools.build.lib.bazel.bzlmod.DelegateTypeAdapterFactory$1.read(DelegateTypeAdapterFactory.java:133) at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.read(TypeAdapterRuntimeTypeWrapper.java:41) at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:187) at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:145) at com.google.devtools.build.lib.bazel.bzlmod.DelegateTypeAdapterFactory$1.read(DelegateTypeAdapterFactory.java:133) at com.google.devtools.build.lib.bazel.bzlmod.BazelLockFileValue_GsonTypeAdapter.read(BazelLockFileValue_GsonTypeAdapter.java:129) at com.google.devtools.build.lib.bazel.bzlmod.BazelLockFileValue_GsonTypeAdapter.read(BazelLockFileValue_GsonTypeAdapter.java:15) at com.google.gson.Gson.fromJson(Gson.java:991) at com.google.gson.Gson.fromJson(Gson.java:956) at com.google.gson.Gson.fromJson(Gson.java:905) at com.google.gson.Gson.fromJson(Gson.java:876) at com.google.devtools.build.lib.bazel.bzlmod.BazelLockFileFunction.getLockfileValue(BazelLockFileFunction.java:93) at com.google.devtools.build.lib.bazel.bzlmod.BazelLockFileFunction.compute(BazelLockFileFunction.java:73) at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:468) ... 7 more ``` Alternatives considered: * Letting Bazel resolve the conflict would require building knowledge about particular version control systems and their conflict style into Bazel. It would also either require the user to resolve conflicts in `MODULE.bazel` first or deviate from the current behavior that the lockfile is not updated when any Bzlmod error is encountered. The jq script can be used as is by every VCS with merge driver support and resolves the conflict in `MODULE.bazel.lock` independently of `MODULE.bazel`. * Implementing the git merge driver as a `bazel mod` subcommand. This could be the source of intransparent slowdowns during regular git operations, which may even be triggered by other tools such as IDEs. The jq script is very fast. * Implementing the merger as a Go binary in buildtools would replace the ubiquitous jq tool with a special purpose binary while also not solving the problem that per-user action is required once to register a custom merge driver. Implements https://docs.google.com/document/d/1TjA7-M5njkI1F38IC0pm305S9EOmxcUwaCIvaSmansg/edit#heading=h.5mcn15i0e1ch RELNOTES: Git merge conflicts in `MODULE.bazel.lock` files can be resolved automatically. See https://bazel.build/external/lockfile#automatic-resolution for the required setup. Closes #22428. PiperOrigin-RevId: 640596606 Change-Id: I20659e3e53a7d8f2529f2ad5a3e7f258d7af026d Commit 3187250 Co-authored-by: Fabian Meumertzheim <[email protected]>
1 parent 0308f42 commit 691149a

File tree

8 files changed

+523
-15
lines changed

8 files changed

+523
-15
lines changed

MODULE.bazel

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -307,6 +307,30 @@ gvm.graalvm(
307307
)
308308
use_repo(gvm, "graalvm_toolchains")
309309

310+
http_file = use_repo_rule("@bazel_tools//tools/build_defs/repo:http.bzl", "http_file")
311+
312+
# DO NOT UPDATE the jq version, this is used to verify compatibility with old versions.
313+
http_file(
314+
name = "jq_linux_amd64",
315+
executable = True,
316+
integrity = "sha256-xrOn19PntwxvUbcGo7kL0BgzhGxU0yyjLwAn8AIm/20=",
317+
urls = ["https://github.com/jqlang/jq/releases/download/jq-1.5/jq-linux64"],
318+
)
319+
320+
http_file(
321+
name = "jq_macos_amd64",
322+
executable = True,
323+
integrity = "sha256-OG6SyYKlb+SFFGjXqTHfyilWDO4wag5mxqG9QGXT2sU=",
324+
urls = ["https://github.com/jqlang/jq/releases/download/jq-1.5/jq-osx-amd64"],
325+
)
326+
327+
http_file(
328+
name = "jq_windows_amd64",
329+
executable = True,
330+
integrity = "sha256-6+zYQLpH779mgihoF4zHIaFRBgk396xAbj0xvQFb3pQ=",
331+
urls = ["https://github.com/jqlang/jq/releases/download/jq-1.5/jq-win64.exe"],
332+
)
333+
310334
# =========================================
311335
# Other Bazel testing dependencies
312336
# =========================================

MODULE.bazel.lock

Lines changed: 79 additions & 8 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

scripts/BUILD

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,32 @@ sh_test(
4040
],
4141
)
4242

43+
filegroup(
44+
name = "jq",
45+
srcs = select({
46+
"@platforms//os:linux": ["@jq_linux_amd64//file"],
47+
"@platforms//os:macos": ["@jq_macos_amd64//file"],
48+
"@platforms//os:windows": ["@jq_windows_amd64//file"],
49+
}),
50+
)
51+
52+
sh_test(
53+
name = "bazel_lockfile_merge_test",
54+
size = "small",
55+
srcs = ["bazel_lockfile_merge_test.sh"],
56+
data = [
57+
"bazel-lockfile-merge.jq",
58+
"testenv.sh",
59+
":jq",
60+
"//src/test/shell:bashunit",
61+
"//src/test/tools/bzlmod:MODULE.bazel.lock",
62+
"@bazel_tools//tools/bash/runfiles",
63+
],
64+
env = {
65+
"JQ_RLOCATIONPATH": "$(rlocationpath :jq)",
66+
},
67+
)
68+
4369
filegroup(
4470
name = "srcs",
4571
srcs = glob(["**"]) + [

scripts/bazel-lockfile-merge.jq

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
# Merges an arbitrary number of MODULE.bazel.lock files.
2+
#
3+
# Input: an array of MODULE.bazel.lock JSON objects (as produced by `jq -s`).
4+
# Output: a single MODULE.bazel.lock JSON object.
5+
#
6+
# This script assumes that all files are valid JSON and have a numeric
7+
# "lockFileVersion" field. It will not fail on any such files, but only
8+
# preserves information for files with a version of 10 or higher.
9+
#
10+
# The first file is considered to be the base when deciding which values to
11+
# keep in case of conflicts.
12+
13+
# Like unique, but preserves the order of the first occurrence of each element.
14+
def stable_unique:
15+
reduce .[] as $item ([]; if index($item) == null then . + [$item] else . end);
16+
17+
# Given an array of objects, shallowly merges the result of applying f to each
18+
# object into a single object, with a few special properties:
19+
# 1. Values are uniquified before merging and then merged with last-wins
20+
# semantics. Assuming that the first value is the base, this ensures that
21+
# later occurrences of the base value do not override other values. For
22+
# example, when this is called with B A1 A2 and A1 contains changes to a
23+
# field but A2 does not (compared to B), the changes in A1 will be preserved.
24+
# 2. Object keys on the top level are sorted lexicographically after merging,
25+
# but are additionally split on ":". This ensures that module extension IDs,
26+
# which start with labels, sort as strings in the same way as they due as
27+
# structured objects in Bazel (that is, //python/extensions:python.bzl
28+
# sorts before //python/extensions/private:internal_deps.bzl).
29+
def shallow_merge(f):
30+
map(f) | stable_unique | add | to_entries | sort_by(.key | split(":")) | from_entries;
31+
32+
(
33+
# Ignore all MODULE.bazel.lock files that do not have the maximum
34+
# lockFileVersion.
35+
(map(.lockFileVersion) | max) as $maxVersion
36+
| map(select(.lockFileVersion == $maxVersion))
37+
| {
38+
lockFileVersion: $maxVersion,
39+
registryFileHashes: shallow_merge(.registryFileHashes),
40+
selectedYankedVersions: shallow_merge(.selectedYankedVersions),
41+
# Group extension results by extension ID across all lockfiles with
42+
# shallowly merged factors map, then shallowly merge the results.
43+
moduleExtensions: (map(.moduleExtensions | to_entries)
44+
| flatten
45+
| group_by(.key)
46+
| shallow_merge({(.[0].key): shallow_merge(.value)}))
47+
}
48+
)? //
49+
# We get here if the lockfiles with the highest lockFileVersion could not be
50+
# processed, for example because all lockfiles have lockFileVersion < 10.
51+
# In this case Bazel 7.2.0+ would ignore all lockfiles, so we might as well
52+
# return the first lockfile for the proper "mismatched version" error
53+
# message.
54+
.[0]

0 commit comments

Comments
 (0)