Added cache dirs to load and file_utils

[gmongaras]

When adding "cache_dir" to datasets.load_dataset, the cache_dir gets lost in the function calls, changing the cache dir to the default path. This fixes a few of these instances.

[lhoestq]

hi ! the hf_hub_download cache_dir is a different cache directory than the one for datasets.

hf_hub_download uses the huggingface_hub cache which is located in by default in ~/.cache/huggingface/hub, while datasets uses a different cache for Arrow files and map() results ~/.cache/huggingface/datasets

[gmongaras]

Is there a way to change the default cache directory for both of these on calling load_dataset? Currently, cache_dir makes dealing with where I want files to go a bit confusing as the documentation doesn't mention it only relocates.../datasets and not .../hub.

[lhoestq]

You can set HF_HOME which is the common parent directory for those two caches. Or individually HF_DATASETS_CACHE and HF_HUB_CACHE

[gmongaras]

Got it. Can this be added to the documentation for load_dataset and related functions to avoid confusion with cache_dir?