Skip to content

Commit 43f66bc

Browse files
authored
Install app dependencies into a virtual environment (#257)
App dependencies are now installed into a Python virtual environment (aka venv / virtualenv) instead of into a custom user site-packages location. This: 1. Avoids user site-packages compatibility issues with some packages when using relocated Python (see #253) 2. Improves parity with how dependencies will be installed when using Poetry in the future (since Poetry doesn't support `--user` installs) 3. Unblocks being able to move pip into its own layer (see #254) This approach is possible since pip 22.3+ supports a new `--python` / `PIP_PYTHON` option which can be used to make pip operate against a different environment to the one in which it is installed. This allows us to continuing keeping pip in a separate layer to the app dependencies (currently the Python layer, but in a later PR pip will be moved to its own layer). For a venv to work, it depends upon the `<venv_layer>/bin/python` script being earlier in `PATH` than the main Python installation. To achieve that with CNBs, the venv's layer name must be alphabetically after the Python layer name. In addition, lifecycle 0.20.1+ is required, since earlier versions didn't implement the spec correctly during the execution of later buildpacks - see: buildpacks/lifecycle#1393 Now that app dependencies are installed into a venv, we no longer need to make the system site-packages directory read-only to protect against later buildpacks installing into the wrong location. This has been split out of the Poetry PR for easier review. See also: - https://docs.python.org/3/library/venv.html - https://pip.pypa.io/en/stable/cli/pip/#cmdoption-python Closes #253. GUS-W-16616226.
1 parent f005cd3 commit 43f66bc

File tree

8 files changed

+93
-126
lines changed

8 files changed

+93
-126
lines changed

CHANGELOG.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
77

88
## [Unreleased]
99

10+
### Changed
11+
12+
- App dependencies are now installed into a virtual environment instead of user site-packages. ([#257](https://github.com/heroku/buildpacks-python/pull/257))
13+
1014
## [0.15.0] - 2024-08-07
1115

1216
### Changed

src/errors.rs

Lines changed: 18 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -171,11 +171,6 @@ fn on_python_layer_error(error: PythonLayerError) {
171171
"locating the pip wheel file bundled inside the Python 'ensurepip' module",
172172
&io_error,
173173
),
174-
PythonLayerError::MakeSitePackagesReadOnly(io_error) => log_io_error(
175-
"Unable to make site-packages directory read-only",
176-
"modifying the permissions on Python's 'site-packages' directory",
177-
&io_error,
178-
),
179174
// This error will change once the Python version is validated against a manifest.
180175
// TODO: (W-12613425) Write the supported Python versions inline, instead of linking out to Dev Center.
181176
// TODO: Decide how to explain to users how stacks, base images and builder images versions relate to each other.
@@ -196,6 +191,22 @@ fn on_python_layer_error(error: PythonLayerError) {
196191

197192
fn on_pip_dependencies_layer_error(error: PipDependenciesLayerError) {
198193
match error {
194+
PipDependenciesLayerError::CreateVenvCommand(error) => match error {
195+
StreamedCommandError::Io(io_error) => log_io_error(
196+
"Unable to create virtual environment",
197+
"running 'python -m venv' to create a virtual environment",
198+
&io_error,
199+
),
200+
StreamedCommandError::NonZeroExitStatus(exit_status) => log_error(
201+
"Unable to create virtual environment",
202+
formatdoc! {"
203+
The 'python -m venv' command to create a virtual environment did
204+
not exit successfully ({exit_status}).
205+
206+
See the log output above for more information.
207+
"},
208+
),
209+
},
199210
PipDependenciesLayerError::PipInstallCommand(error) => match error {
200211
StreamedCommandError::Io(io_error) => log_io_error(
201212
"Unable to install dependencies using pip",
@@ -207,8 +218,8 @@ fn on_pip_dependencies_layer_error(error: PipDependenciesLayerError) {
207218
StreamedCommandError::NonZeroExitStatus(exit_status) => log_error(
208219
"Unable to install dependencies using pip",
209220
formatdoc! {"
210-
The 'pip install' command to install the application's dependencies from
211-
'requirements.txt' failed ({exit_status}).
221+
The 'pip install -r requirements.txt' command to install the app's
222+
dependencies failed ({exit_status}).
212223
213224
See the log output above for more information.
214225
"},

src/layers/pip_dependencies.rs

Lines changed: 44 additions & 79 deletions
Original file line numberDiff line numberDiff line change
@@ -6,20 +6,18 @@ use libcnb::layer::UncachedLayerDefinition;
66
use libcnb::layer_env::{LayerEnv, ModificationBehavior, Scope};
77
use libcnb::Env;
88
use libherokubuildpack::log::log_info;
9-
use std::path::{Path, PathBuf};
9+
use std::path::PathBuf;
1010
use std::process::Command;
1111

1212
/// Creates a layer containing the application's Python dependencies, installed using pip.
1313
//
14-
// To do this we use `pip install --user` so that the dependencies are installed into the user
15-
// `site-packages` directory in this layer (set by `PYTHONUSERBASE`), rather than the system
16-
// `site-packages` subdirectory of the Python installation layer.
17-
//
18-
// Note: We can't instead use pip's `--target` option along with `PYTHONPATH`, since:
19-
// - Directories on `PYTHONPATH` take precedence over the Python stdlib (unlike the system or
20-
// user site-packages directories), which can cause hard to debug stdlib shadowing issues
21-
// if one of the app's transitive dependencies is an outdated stdlib backport package.
22-
// - `--target` has bugs, eg: <https://github.com/pypa/pip/issues/8799>
14+
// We install into a virtual environment since:
15+
// - We can't install into the system site-packages inside the main Python directory since
16+
// we need the app dependencies to be in their own layer.
17+
// - Some packages are broken with `--user` installs when using relocated Python, and
18+
// otherwise require other workarounds. eg: https://github.com/unbit/uwsgi/issues/2525
19+
// - PEP-405 style venvs are very lightweight and are also much more frequently
20+
// used in the wild compared to `--user`, and therefore the better tested path.
2321
//
2422
// This layer is not cached, since:
2523
// - pip is a package installer rather than a project/environment manager, and so does not
@@ -35,38 +33,59 @@ pub(crate) fn install_dependencies(
3533
env: &mut Env,
3634
) -> Result<PathBuf, libcnb::Error<BuildpackError>> {
3735
let layer = context.uncached_layer(
38-
layer_name!("dependencies"),
36+
// The name of this layer must be alphabetically after that of the `python` layer so that
37+
// this layer's `bin/` directory (and thus `python` symlink) is listed first in `PATH`:
38+
// https://github.com/buildpacks/spec/blob/main/buildpack.md#layer-paths
39+
layer_name!("venv"),
3940
UncachedLayerDefinition {
4041
build: true,
4142
launch: true,
4243
},
4344
)?;
44-
4545
let layer_path = layer.path();
46-
let layer_env = generate_layer_env(&layer_path);
46+
47+
log_info("Creating virtual environment");
48+
utils::run_command_and_stream_output(
49+
Command::new("python")
50+
.args(["-m", "venv", "--without-pip", &layer_path.to_string_lossy()])
51+
.env_clear()
52+
.envs(&*env),
53+
)
54+
.map_err(PipDependenciesLayerError::CreateVenvCommand)?;
55+
56+
let mut layer_env = LayerEnv::new()
57+
// Since pip is installed in a different layer (outside of this venv), we have to explicitly
58+
// tell it to perform operations against this venv instead of the global Python install.
59+
// https://pip.pypa.io/en/stable/cli/pip/#cmdoption-python
60+
.chainable_insert(
61+
Scope::All,
62+
ModificationBehavior::Override,
63+
"PIP_PYTHON",
64+
&layer_path,
65+
)
66+
// For parity with the venv's `bin/activate` script:
67+
// https://docs.python.org/3/library/venv.html#how-venvs-work
68+
.chainable_insert(
69+
Scope::All,
70+
ModificationBehavior::Override,
71+
"VIRTUAL_ENV",
72+
&layer_path,
73+
);
4774
layer.write_env(&layer_env)?;
75+
// Required to pick up the automatic PATH env var. See: https://github.com/heroku/libcnb.rs/issues/842
76+
layer_env = layer.read_env()?;
4877
env.clone_from(&layer_env.apply(Scope::Build, env));
4978

50-
log_info("Running pip install");
51-
79+
log_info("Running 'pip install -r requirements.txt'");
5280
utils::run_command_and_stream_output(
5381
Command::new("pip")
5482
.args([
5583
"install",
5684
"--no-input",
5785
"--progress-bar",
5886
"off",
59-
// Using `--user` rather than `PIP_USER` since the latter affects `pip list` too.
60-
"--user",
6187
"--requirement",
6288
"requirements.txt",
63-
// For VCS dependencies installed in editable mode, the repository clones must be
64-
// kept after installation, since their directories are added to the Python path
65-
// directly (via `.pth` files in `site-packages`). By default pip will store the
66-
// repositories in the current working directory (the app dir), but we want them
67-
// in the dependencies layer instead.
68-
"--src",
69-
&layer_path.join("src").to_string_lossy(),
7089
])
7190
.current_dir(&context.app_dir)
7291
.env_clear()
@@ -77,35 +96,10 @@ pub(crate) fn install_dependencies(
7796
Ok(layer_path)
7897
}
7998

80-
fn generate_layer_env(layer_path: &Path) -> LayerEnv {
81-
LayerEnv::new()
82-
// We set `PATH` explicitly, since lifecycle will only add the bin directory to `PATH` if it
83-
// exists - and we want to support the scenario of installing a debugging package with CLI at
84-
// run-time, when none of the dependencies installed at build-time had an entrypoint script.
85-
.chainable_insert(
86-
Scope::All,
87-
ModificationBehavior::Prepend,
88-
"PATH",
89-
layer_path.join("bin"),
90-
)
91-
.chainable_insert(Scope::All, ModificationBehavior::Delimiter, "PATH", ":")
92-
// Overrides the default user base directory, used by Python to compute the path of the user
93-
// `site-packages` directory. Setting this:
94-
// - Makes `pip install --user` install the dependencies into the current layer rather
95-
// than the user's home directory (which would be discarded at the end of the build).
96-
// - Allows Python to find the installed packages at import time.
97-
// See: https://docs.python.org/3/using/cmdline.html#envvar-PYTHONUSERBASE
98-
.chainable_insert(
99-
Scope::All,
100-
ModificationBehavior::Override,
101-
"PYTHONUSERBASE",
102-
layer_path,
103-
)
104-
}
105-
10699
/// Errors that can occur when installing the project's dependencies into a layer using pip.
107100
#[derive(Debug)]
108101
pub(crate) enum PipDependenciesLayerError {
102+
CreateVenvCommand(StreamedCommandError),
109103
PipInstallCommand(StreamedCommandError),
110104
}
111105

@@ -114,32 +108,3 @@ impl From<PipDependenciesLayerError> for libcnb::Error<BuildpackError> {
114108
Self::BuildpackError(BuildpackError::PipDependenciesLayer(error))
115109
}
116110
}
117-
118-
#[cfg(test)]
119-
mod tests {
120-
use super::*;
121-
122-
#[test]
123-
fn pip_dependencies_layer_env() {
124-
let mut base_env = Env::new();
125-
base_env.insert("PATH", "/base");
126-
base_env.insert("PYTHONUSERBASE", "this-should-be-overridden");
127-
128-
let layer_env = generate_layer_env(Path::new("/layer-dir"));
129-
130-
assert_eq!(
131-
utils::environment_as_sorted_vector(&layer_env.apply(Scope::Build, &base_env)),
132-
[
133-
("PATH", "/layer-dir/bin:/base"),
134-
("PYTHONUSERBASE", "/layer-dir"),
135-
]
136-
);
137-
assert_eq!(
138-
utils::environment_as_sorted_vector(&layer_env.apply(Scope::Launch, &base_env)),
139-
[
140-
("PATH", "/layer-dir/bin:/base"),
141-
("PYTHONUSERBASE", "/layer-dir"),
142-
]
143-
);
144-
}
145-
}

src/layers/python.rs

Lines changed: 0 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -11,8 +11,6 @@ use libcnb::layer_env::{LayerEnv, ModificationBehavior, Scope};
1111
use libcnb::Env;
1212
use libherokubuildpack::log::log_info;
1313
use serde::{Deserialize, Serialize};
14-
use std::fs::Permissions;
15-
use std::os::unix::prelude::PermissionsExt;
1614
use std::path::{Path, PathBuf};
1715
use std::process::Command;
1816
use std::{fs, io};
@@ -138,16 +136,6 @@ pub(crate) fn install_python_and_packaging_tools(
138136
)
139137
.map_err(PythonLayerError::BootstrapPipCommand)?;
140138

141-
// By default pip installs into the system site-packages directory if it is writeable by the
142-
// current user. Whilst the buildpack's own `pip install` invocations always use `--user` to
143-
// ensure app dependencies are installed into the user site-packages, it's possible other
144-
// buildpacks or custom scripts may forget to do so. By making the system site-packages
145-
// directory read-only, pip will automatically use user installs in such cases:
146-
// https://github.com/pypa/pip/blob/24.1.2/src/pip/_internal/commands/install.py#L662-L720
147-
let site_packages_dir = python_stdlib_dir.join("site-packages");
148-
fs::set_permissions(site_packages_dir, Permissions::from_mode(0o555))
149-
.map_err(PythonLayerError::MakeSitePackagesReadOnly)?;
150-
151139
Ok(())
152140
}
153141

@@ -369,7 +357,6 @@ pub(crate) enum PythonLayerError {
369357
BootstrapPipCommand(StreamedCommandError),
370358
DownloadUnpackPythonArchive(DownloadUnpackArchiveError),
371359
LocateBundledPip(io::Error),
372-
MakeSitePackagesReadOnly(io::Error),
373360
PythonArchiveNotFound { python_version: PythonVersion },
374361
}
375362

tests/fixtures/pip_editable_git_compiled/requirements.txt

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# This requirement uses a VCS URL and `-e` in order to test that:
22
# - Git from the stack image can be found (ie: the system PATH has been correctly propagated to pip).
3-
# - The editable mode repository clone is saved into the dependencies layer (via the `--src` option).
3+
# - The editable mode repository clone is saved into the dependencies layer.
44
#
55
# A C-based package is used instead of a pure Python package, in order to test that the
66
# Python headers can be found in the `include/pythonX.Y/` directory of the Python layer.

tests/fixtures/testing_buildpack/bin/build

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,6 @@
55
# - Python's sys.path is correct.
66
# - The correct version of pip was installed.
77
# - Both the package manager and Python can find the typing-extensions package.
8-
# - The system site-packages directory is protected against running 'pip install'
9-
# without having passed '--user'.
108
# - The typing-extensions package was installed into a separate dependencies layer.
119

1210
set -euo pipefail
@@ -20,5 +18,4 @@ python -c 'import pprint, sys; pprint.pp(sys.path)'
2018
echo
2119
pip --version
2220
pip list
23-
pip install --dry-run typing-extensions
2421
python -c 'import typing_extensions; print(typing_extensions)'

tests/mod.rs

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,7 @@ fn default_build_config(fixture_path: impl AsRef<Path>) -> BuildConfig {
5151
("PYTHONHOME", "/invalid"),
5252
("PYTHONPATH", "/invalid"),
5353
("PYTHONUSERBASE", "/invalid"),
54+
("VIRTUAL_ENV", "/invalid"),
5455
]);
5556

5657
config

0 commit comments

Comments
 (0)