Skip to content

Version cache dirs independently from Pydra version #825

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
effigies opened this issue May 19, 2025 · 1 comment
Open

Version cache dirs independently from Pydra version #825

effigies opened this issue May 19, 2025 · 1 comment
Assignees
Labels
enhancement New feature or request

Comments

@effigies
Copy link
Contributor

via #149 (comment)

Since the run-cache now gets created /Users/<user>/Library/Caches/pydra/1.0a1/run-cache by default (by platformdirs on Mac, will be different on Windows and Linux), (need separate issue about versioning caches), we could specify a directory called /Users/<user>/Library/Caches/pydra/1.0a1/runs (or similar), which contains timestamped symlinks to the top-level cache directory that was submitted, e.g. /Users/<user>/Library/Caches/pydra/1.0a1/runs/20250520065430, which the last run could be picked up when a specific cache directory isn't specified by pydra crash

Just a quick note that I think we should version the caches in their own right, not based on the Pydra version.

You can see this style of thing in effect in uv, where each type of thing that might be cached has its own version format. Some need to be frequently bumped, others haven't:

$ ls .cache/uv 
archive-v0  CACHEDIR.TAG     git-v0          sdists-v9   simple-v16
builds-v0   environments-v2  interpreter-v4  simple-v15  wheels-v5

We could do something like .cache/pydra/tasks-v0 for now, and then when a change to the engine justifies a cache invalidation, we bump to tasks-v1.

@effigies effigies added the enhancement New feature or request label May 19, 2025
@effigies effigies moved this from Triage to v1.0 in Pydra Roadmap May 19, 2025
@effigies effigies self-assigned this May 19, 2025
@effigies effigies marked this as a duplicate of #824 May 19, 2025
@tclose
Copy link
Contributor

tclose commented May 19, 2025

Sounds good, although I would probably prefer separate sub-directories <user-cache-dir>/pydra/tasks-cache/v0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: v1.0
Development

No branches or pull requests

2 participants