Draft Changelog

The following items should be added to the changelog for the next Toil release, at which point the list should be cleared. Add new items at the bottom:

Version 9.x Draft

Changelog

Highlighted Features Added

• Added new unified plugin system (#5302)
• Toil now supports URL plugins. To provide a plugin, add toil_url_access_ at beginning of python package name. To register a new URL scheme use register_plugin("url_access", scheme, implementation_type_factory). (#5302)
• Updated real world example in quick start (#3534)

Breaking Changes

• Refactor URL access functions to use new URLAccess class instead of AbstractJobStore (#5301)
• Toil Docker container no longer ships s3am by default (#5321)
• Deleted deprecated addBatchSystemFactory, BATCH_SYSTEM_FACTORY_REGISTRY, and BATCH_SYSTEMS. Use add_batch_system_factory instead. (#5323)
• --importWorkersThreshold has been renamed to --importWorkersBatchsize (#5325)

CWL

• Fix a bug in Toil server mode CWL YAML parsing (#5252)
• Fix missing CWL badges (#5304)
• CWL worker import works again (#5256)
• CWL worker import test (and other non-conformance CWL tests) now run on CI again (#5256)
• Fixed a regression with tmpdir_prefix and nonexistent diectories (#5310)
• Mesos CI tests no longer reference CWL conformance test class by old name (#5329)
• Remove redundant listings when loading inputs to a job (#5099)
• Added --cwl-min-ram to CWL. This specifies a minimum memory allocation for all jobs. This is different than --defaultMemory, which only applies to jobs without explicitly set memory requirements. (#2862)

WDL

• Toil now deletes files that go out of scope at the ends of workflows and tasks. (#5222)
• toil-wdl-runner can again successfully finish a workflow and emit outputs when using --restart (#5247)
• wdl tests: use pytest fixtures to support parallel execution (#5235)
• Added lint checking to toil-wdl-runner (#4630)
  • Use --strict=True to error on lint warnings
• Toil can now handle coercing input Files to Strings and back in WDL workflows. (#5265)
• Allow relaxing quantifier checks with --quantCheck (#5297)
• toil-wdl-runner will bump container memory if below 4MiB (#5325)

Kubernetes

• Added command line options --kubernetesPodSecurityContext and --kubernetesSecurityContext to change the security context of pods and containers (#5179)

Dependencies

• Update to connexion 3 (#4927)

Misc

• Toil metrics published to Dockstore now are marked as coming from TOIL and not OTHER. (#5237)
• Toil metrics published to Dockstore are now "authenticated" with a public, non-secret token. (#5239)
• Dockstore publication tkinter dialog now opens over other windows on Mac. (#5239)
• Publishing metrics for the current workflow will no longer send an empty list of tasks also. (#5239)
• needs_aws_s3: also skip tests if there is a proxy error (#5250)
• Speed up CWL badge generation on CI (#5246)
• gitlab-ci: add collapsable sections (with recorded durations) (#5235)
• gitlab-ci: increase test threads to 4; standalone tests only need one test thread; use worksteal more often; upgrade cwltest, run conformance tests in parallel; enable caching of pip & mypy (#5235)
• Toil can now build correctly for pip installs from Git again.
• Toil now tests linting on the oldest supported Python for every PR (#5276)
• Added a way to turn Toil history database on and off with TOIL_HISTORY. (#5254)
• Added TOIL_JOB_HISTORY to force-enable job history tracking even though it's not ready for production use. (#5258)
• Toil tests no longer record history to the Toil history database. (#5254)
• Suppress spurious nvidia warnings (#5306)
• Use different Google Cloud example file since the Broad data is now requester-pays (#5307)
• Improve Google cloud download speed by polling for metadata once (#5311)
• Toil Docker build is now willing to hide system Python packages (#5320)
• Remove separate plugin test job since quick_test_offline runs them (#5334)
• Toil server mode CI tests now run on PRs. Toil server mode integration tests now run on PRs that touch the server code. Toil server mode CI tests are now updated for Connexion 3. (#5332)

Thank you to our contributors: