Document design aspects: limited gem dependencies, no gem activation during Ruby loader initialization

[ci skip]

Author: FooBarWidget

Gemfile

@@ -1,14 +1,16 @@
+# Specifies Passenger *development* dependencies.
+# See also: doc/DesignAspects/LimitedGemDependencies.md
+
 source 'https://rubygems.org/'
 
 ruby '>= 2.5'
 
+gemspec
+
 group :development do
   gem 'json'
   gem 'mime-types', '~> 3.5.1'
   gem 'drake'
-  gem 'rack'
-  gem 'rackup', '>= 2.1'
-  gem 'rake'
   gem 'rspec', '~> 3.12.0'
   gem 'rspec-collection_matchers'
   gem 'webrick', '~> 1.8.1'

Gemfile.lock

@@ -1,3 +1,11 @@
+PATH
+  remote: .
+  specs:
+    passenger (6.0.25)
+      rack (>= 1.6.13)
+      rackup (>= 2.0.0)
+      rake (>= 12.3.3)
+
 GEM
   remote: https://rubygems.org/
   specs:
@@ -10,9 +18,8 @@ GEM
       mime-types-data (~> 3.2015)
     mime-types-data (3.2024.1001)
     rack (3.1.8)
-    rackup (2.1.0)
+    rackup (2.2.1)
       rack (>= 3)
-      webrick (~> 1.8)
     rake (13.2.1)
     rspec (3.12.0)
       rspec-core (~> 3.12.0)
@@ -41,9 +48,7 @@ DEPENDENCIES
   drake
   json
   mime-types (~> 3.5.1)
-  rack
-  rackup (>= 2.1)
-  rake
+  passenger!
   rspec (~> 3.12.0)
   rspec-collection_matchers
   webrick (~> 1.8.1)

doc/DesignAspects/LimitedGemDependencies.md

@@ -0,0 +1,39 @@
+# Limited gem dependencies
+
+## Context
+
+- We provide native packages (APT/YUM) for Passenger.
+- Passenger has a bunch of tools (e.g. `bin/passenger-status`, `helper-scripts/backtrace-sanitizer.rb`, `helper-scripts/prespawn`) that are written in Ruby. When Passenger is natively packaged, we ensure that these tools run in the "system" Ruby.
+- On Debian-based systems, the OS package repositories may supply multiple Ruby versions. So our native packaging can't assume a single Ruby version.
+
+This raises the question of how to deal with Passenger's own gem dependencies. An analysis of possible strategies and their implications/tradeoffs:
+
+1. If the OS package repositories supply dependent gems, then our native packages can depend on those directly.
+
+   Drawbacks:
+
+    - Limited selection of gems.
+
+2. We package the gem dependencies ourselves. In addition, any dependency with native extensions must be packaged separately, once for each Ruby version installable through the native OS package repositories.
+
+   Drawbacks:
+
+    - More work for us.
+
+3. We vendor the gem dependencies within the native packages. In addition, any dependency with native extensions must be vendored separately, once for each Ruby version installable through the native OS package repositories.
+
+## Decision
+
+We pick strategy 1.
+
+## Consequences
+
+- Any gems that the written-in-Ruby tools depend on, must either be available through OS repositories, or must be a Ruby standard library gem.
+  - All such dependencies must be specified in the Passenger gemspec.
+- Development-related Passenger Ruby code can depend on any gem, since they're not included in native packages.
+  - All such dependencies must be specified in the Passenger Gemfile.
+
+## See also
+
+- [No gem activation during Ruby loader initialization](NoGemActivationDuringRubyLoaderInitialization.md)
+- System Ruby runner (only used in Debian packages): `packaging/debian/debian_specs/passenger/passenger_system_ruby.c.erb`

doc/DesignAspects/NoGemActivationDuringRubyLoaderInitialization.md

@@ -0,0 +1,27 @@
+# No gem activation during Ruby loader initialization
+
+## Context
+
+Passenger loads a Ruby app using loader scripts (rack-loader.rb and rack-preloader.rb). These scripts perform these steps:
+
+1. Initialization, e.g., to setup communication with the Passenger Core.
+2. Activate Bundler.
+3. Load the application (config.ru).
+
+If step 1 activates any gems, then that may conflict with Bundler activation because Gemfile could specify a different gem version.
+
+## Decision
+
+All code callable by the Ruby loaders' initialization steps, must not activate any gems.
+
+## Consequences
+
+Any needed functionality that's typically provided by gems, must be reimplemented within the `PhusionPassenger` module.
+
+For example, the initialization step needs to parse JSON. Instead of activating the JSON gem, we reimplement a JSON parser in `PhusionPassenger::Utils::JSON`.
+
+### Caveat: standard library gems
+
+More and more parts of the Ruby standard library are becoming [gemified](https://stdgems.org/). This increases the chances of the initialization step activating a standard library gem. In principle, our strategy remains the same: we reimplement those libraries. For example, we reimplemented StringScanner (needed by our JSON implementation) in `PhusionPassenger::Utils::StringScanner`, but we limited its scope to just the functionality needed by the JSON implementation.
+
+If the standard library in question is a native extension (e.g., `etc`), then the strategy is to reimplement it in C++ as part of the [Passenger Agent binary](../../src/agent/README.md), as a separately callable tool. The Ruby code then spawns the written-in-C++ tool and parses its output.

passenger.gemspec

@@ -23,9 +23,13 @@ Gem::Specification.new do |s|
     "mailing_list_uri"  => "https://www.phusionpassenger.com/contact",
     "wiki_uri"          => "https://github.com/phusion/passenger/wiki"
   }
+
+  # Limit dependencies only to those available through OS package repositories.
+  # See doc/DesignAspects/LimitedGemDependencies.md
   s.add_dependency 'rake', '>= 12.3.3'
   s.add_dependency 'rack', '>= 1.6.13'
   s.add_dependency 'rackup', '>= 2.0.0'
+
   s.files = Dir[*PhusionPassenger::Packaging::GLOB] -
     Dir[*PhusionPassenger::Packaging::EXCLUDE_GLOB]
   s.executables = PhusionPassenger::Packaging::USER_EXECUTABLES +