Refactor `Dataset.map` to reuse cache files mapped with different `num_proc`#7434

Open

ringohoffman wants to merge 16 commits intohuggingface:mainfrom

ringohoffman:reuse-cache-on-different-num_proc

Commits on Mar 4, 2025

Refactor Dataset.map to reuse cache files mapped with different num_proc
ringohoffman
committed
Only give reprocessing message doing a partial remap
ringohoffman
committed
Update logging message to account for if a cache file will be written at all and written by the main process or not
ringohoffman
committed
Refactor string_to_dict to return None if there is no match instead of raising ValueError
ringohoffman
committed
Merge branch 'return-none-if-string_to_dict-no-match' into reuse-cache-on-different-num_proc
ringohoffman
committed
Simplify existing existing_cache_file_map with string_to_dict
ringohoffman
committed
Set initial value if there are already existing cache files
ringohoffman
committed

Commits on Mar 5, 2025

Commits on Mar 7, 2025

Commits on Mar 9, 2025

Merge branch 'return-none-if-string_to_dict-no-match' into reuse-cache-on-different-num_proc
ringohoffman
committed

Commits on Mar 12, 2025

Commits on Mar 14, 2025