Platform package repository snapshots (and support for build metadata in package versions) #796
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dzuelke
force-pushed
the
platform-repo-snapshots
branch
2 times, most recently
from
452f93a to
a8d530f
Compare
Malax
approved these changes
May 8, 2025
There was a problem hiding this comment.
Looks good to me, I'm missing some context for the finer details but the code has comments to explain these. But I did not verify that the comments were correct.
The only thing that stood out to me are the inconsistencies in the test fixture .json files when it comes to trailing newlines. They probably should have them. I'll approve this PR nonetheless to avoid unnecessary re-review loops. Please consider fixing those before merging. :)
Really only expected_platform_composer.json should have no trailing newline at EOF because that is how they get generated and we imported them verbatim from the generator calls.
Less confusing if the two s3:// URLs are indented the same.
This handles packages such as 'php-8.4.6+build2', which will allow re-building of existing packages without overwriting them in a published repository (a pre-requisite for stable repository snapshots). Because semver ignores build metadata for version comparison, if multiple packages with the same version but different build metadata exists, mkrepo.sh will only include the "highest" build metadata variant (sorted using "natural", version-like sorting, so "+build11" is higher than "+build9" etc). GUS-W-18461573
easiest to tell apart which is getting installed using build metadata in the version string
just a bit tidier, they are not formulae
A hash is calculated based on the sorted list of formulae in support/build/. If a formula is renamed or added, the hash changes. During repository upload, mkrepo.sh will create a packages-${hash}.json in addition to packages.json (which still serves as the "bleeding edge" repository, as before).
In bin/compile, the same hash is used to calculate a repository URL.
Together with the previous changes for build metadata variants in package formulae versions, this allows stable repository snapshots, including re-builds of existing published package versions, that are not picked up by published buildpack releases, since they would reference a different snapshot hash in the URL to packages-${hash}.json.
CI now requires platform repo snapshots to be present on the develop bucket/prefix for pull request runs. This still allows manual workflow_dispatch runs without a snapshot, e.g. during development/debugging. Tags and main run against the default repo, as usual, which now uses snapshots. The Prepare Release workflow enforces the presence of a snapshot on the stable bucket/prefix as an additional measure.
GUS-W-18463967
dzuelke
force-pushed
the
platform-repo-snapshots
branch
from
a8d530f to
f4db1b4
Compare
|
Yes, well spotted @Malax, I tidied those up (also in any other fixtures that were missing them, except for all the |
…has nothing to do otherwise Useful e.g. if obsolete formulae (that were never built in the destination) got removed, say, on stack EOL. Or if the way the snapshot hash is computed ever changes, but not the formulae themselves. Both of these of course require manual copying of the packages-SNAPSHOT.json files in the source bucket, but their existence is already guaranteed by the CI rules that force snapshot usage on Pull Requests.
Now that we have snapshots, packages will never be removed; the platform-remove workflow is only for iterating during development.
We definitely do not want to cancel in-progress syncs for other stacks if one fails for whatever reason, that could leave a bucket/repo in a bad state.
Just internal release tooling in there.
dzuelke
force-pushed
the
platform-repo-snapshots
branch
from
f4db1b4 to
0f0a507
Compare
|
Sync (only to create snapshots on |
schneems
reviewed
May 9, 2025
|
|
||
| When updating the build for an existing, published version of a package (e.g. with updated compile options), it is recommended to use *build metadata* in the version string of the updated package in order to distinguish it from the existing, published version. For example, `php-8.4.6` would become `php-8.4.6+build2`. | ||
|
|
||
| Per Semver specifications, any build metadata in a version string is ignored during version comparison. This means it could not be guaranteed that `php-8.4.6+build2` is picked, if it exists in the same repository as `php-8.4.6`. For this reason, when [(Re-)generating Repositories](re-generating-repositories), only the package/version combination with the "highest" (compared using lenient version string comparison) build metadata will be included, meaning that in the example above, `php-8.4.6` would not be included in the generated repository, only `php-8.4.6+build2`. |
There was a problem hiding this comment.
This link is broken:
$ curl -I -s https://github.com/heroku/heroku-buildpack-php/blob/main/support/build/re-generating-repositories | grep HTTP
HTTP/2 404
$ curl -I -s https://github.com/heroku/heroku-buildpack-php/blob/0f0a5075ae34b471ca17bbc2c05bb88e8c01ca3e/support/build/re-generating-repositories | grep HTTP
HTTP/2 404
I think you need a # so it's an anchor link. That's what the link in the ### Repository Snapshots section uses.
This was referenced May 9, 2025
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This handles packages such as "
php-8.4.6+build2", which will allow re-building of existing packages without overwriting them in a published repository (a pre-requisite for stable repository snapshots).Because semver ignores build metadata for version comparison, if multiple packages with the same version but different build metadata exists,
mkrepo.shwill only include the "highest" build metadata variant (sorted using "natural", version-like sorting, so "+build11" is higher than "+build9" etc).For snapshots, a hash is calculated based on the sorted list of formulae in
support/build/. If a formula is renamed or added, the hash changes. During repository upload,mkrepo.shwill create apackages-${hash}.jsonin addition topackages.json(which still serves as the "bleeding edge" repository, as before).In
bin/compile, the same hash is used to calculate a repository URL.Together with the changes for build metadata variants in package formulae versions described above, this allows stable repository snapshots, including re-builds of existing published package versions, that are not picked up by published buildpack releases, since they would reference a different snapshot hash in the URL to
packages-${hash}.json.CI now requires platform repo snapshots to be present on the develop bucket/prefix for pull request runs. This still allows manual
workflow_dispatchruns without a snapshot, e.g. during development/debugging. Tags and main run against the default repo, as usual, which now uses snapshots. The Prepare Release workflow enforces the presence of a snapshot on the stable bucket/prefix as an additional measure.Syncing from develop to stable creates the snapshot in the stable bucket, too. A sync is rejected if the destination already has the calculated snapshot. This helps prevents accidental sync runs using e.g.
maininstead of the branch with the formulae changes; this concern is going away longer term once we no longer treat "all present packages" as the canonical form of a repository, but instead explicitly build repository snapshots from a known list of packages that are expected to be present.GUS-W-18461573
GUS-W-18463967