[BUG]: Snapshot still not restored after the new version 4.0

[BenDes21]

Hi there, even after the update 4.0, snapshots cannot be restored ( I followed all the procedure ). Im still getting errors during the installation / import of custom nodes

Would be amazing if the runpod team can take a look of it @TimPietrusky @TimPietruskyRunPod

Here the log :

2025-04-26T13:23:43.847869931Z ==========
2025-04-26T13:23:43.847877134Z == CUDA ==
2025-04-26T13:23:43.847955570Z ==========
2025-04-26T13:23:43.851029485Z CUDA Version 12.6.3
2025-04-26T13:23:43.851722943Z Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2025-04-26T13:23:43.852386809Z This container image and its contents are governed by the NVIDIA Deep Learning Container License.
2025-04-26T13:23:43.852392842Z By pulling and using the container, you accept the terms and conditions of this license:
2025-04-26T13:23:43.852397229Z https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license
2025-04-26T13:23:43.852405722Z A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.
2025-04-26T13:23:43.863949260Z worker-comfyui: Starting ComfyUI
2025-04-26T13:23:43.864088015Z worker-comfyui: Starting RunPod Handler
2025-04-26T13:23:43.921961382Z Adding extra search path checkpoints /runpod-volume/models/checkpoints
2025-04-26T13:23:43.921982555Z Adding extra search path clip /runpod-volume/models/clip
2025-04-26T13:23:43.921995725Z Adding extra search path clip_vision /runpod-volume/models/clip_vision
2025-04-26T13:23:43.922037120Z Adding extra search path configs /runpod-volume/models/configs
2025-04-26T13:23:43.922086512Z Adding extra search path controlnet /runpod-volume/models/controlnet
2025-04-26T13:23:43.922105579Z Adding extra search path embeddings /runpod-volume/models/embeddings
2025-04-26T13:23:43.922120413Z Adding extra search path loras /runpod-volume/models/loras
2025-04-26T13:23:43.922171359Z Adding extra search path upscale_models /runpod-volume/models/upscale_models
2025-04-26T13:23:43.922177512Z Adding extra search path vae /runpod-volume/models/vae
2025-04-26T13:23:43.922203333Z Adding extra search path unet /runpod-volume/models/unet
2025-04-26T13:23:45.373320977Z [START] Security scan
2025-04-26T13:23:45.373343547Z [DONE] Security scan
2025-04-26T13:23:45.373349219Z ## ComfyUI-Manager: installing dependencies done.
2025-04-26T13:23:45.373365449Z ** ComfyUI startup time: 2025-04-26 13:23:45.373
2025-04-26T13:23:45.373531271Z ** Platform: Linux
2025-04-26T13:23:45.373564291Z ** Python version: 3.11.0rc1 (main, Aug 12 2022, 10:02:14) [GCC 11.2.0]
2025-04-26T13:23:45.373632803Z ** Python executable: /usr/bin/python
2025-04-26T13:23:45.373695727Z ** ComfyUI Path: /comfyui
2025-04-26T13:23:45.373763904Z ** ComfyUI Base Folder Path: /comfyui
2025-04-26T13:23:45.373836352Z ** User directory: /comfyui/user
2025-04-26T13:23:45.373896586Z ** ComfyUI-Manager config path: /comfyui/user/default/ComfyUI-Manager/config.ini
2025-04-26T13:23:45.373986483Z ** Log path: /comfyui/user/comfyui.log
2025-04-26T13:23:46.201041492Z --- Starting Serverless Worker |  Version 1.7.9 ---
2025-04-26T13:23:46.376919013Z {"requestId": null, "message": "Jobs in queue: 1", "level": "INFO"}
2025-04-26T13:23:46.376945899Z {"requestId": null, "message": "Jobs in progress: 1", "level": "INFO"}
2025-04-26T13:23:46.376947985Z {"requestId": "e0247faf-7f63-41fa-955f-c164db4ebe8c-u2", "message": "Started.", "level": "INFO"}
2025-04-26T13:23:46.586815256Z Prestartup times for custom nodes:
2025-04-26T13:23:46.586820898Z    0.0 seconds: /comfyui/custom_nodes/rgthree-comfy
2025-04-26T13:23:46.586826401Z    0.0 seconds: /comfyui/custom_nodes/comfyui-easy-use
2025-04-26T13:23:46.586830635Z    2.7 seconds: /comfyui/custom_nodes/ComfyUI-Manager
2025-04-26T13:23:48.094661968Z Checkpoint files will always be loaded safely.
2025-04-26T13:23:48.475476260Z Total VRAM 24138 MB, total RAM 515602 MB
2025-04-26T13:23:48.475519264Z pytorch version: 2.7.0+cu126
2025-04-26T13:23:48.475929408Z Set vram state to: NORMAL_VRAM
2025-04-26T13:23:48.476288335Z Device: cuda:0 NVIDIA RTX A5000 : cudaMallocAsync
2025-04-26T13:23:49.561577974Z Using pytorch attention
2025-04-26T13:23:50.688289463Z Python version: 3.11.0rc1 (main, Aug 12 2022, 10:02:14) [GCC 11.2.0]
2025-04-26T13:23:50.688317394Z ComfyUI version: 0.3.30
2025-04-26T13:23:50.691205894Z ComfyUI frontend version: 1.17.11
2025-04-26T13:23:50.691674278Z [Prompt Server] web root: /usr/local/lib/python3.11/dist-packages/comfyui_frontend_package/static
2025-04-26T13:23:51.154342583Z ### Loading: ComfyUI-Manager (V3.31.12)
2025-04-26T13:23:51.154477735Z [ComfyUI-Manager] network_mode: public
2025-04-26T13:23:51.190427815Z ### ComfyUI Revision: 3389 [a97f2f85] *DETACHED | Released on '2025-04-24'
2025-04-26T13:23:51.440016868Z [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/alter-list.json
2025-04-26T13:23:51.457025568Z /tmp/tmps5hoz00i/main.c:5:10: fatal error: Python.h: No such file or directory
2025-04-26T13:23:51.457050787Z     5 | #include <Python.h>
2025-04-26T13:23:51.457053322Z       |          ^~~~~~~~~~
2025-04-26T13:23:51.457054957Z compilation terminated.
2025-04-26T13:23:51.486519306Z Traceback (most recent call last):
2025-04-26T13:23:51.486557773Z   File "/comfyui/nodes.py", line 2128, in load_custom_node
2025-04-26T13:23:51.486563350Z     module_spec.loader.exec_module(module)
2025-04-26T13:23:51.486568047Z   File "<frozen importlib._bootstrap_external>", line 940, in exec_module
2025-04-26T13:23:51.486573626Z   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
2025-04-26T13:23:51.486577753Z   File "/comfyui/custom_nodes/comfyui-easy-use/__init__.py", line 15, in <module>
2025-04-26T13:23:51.486582456Z     importlib.import_module('.py.routes', __name__)
2025-04-26T13:23:51.486586567Z   File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module
2025-04-26T13:23:51.486590747Z     return _bootstrap._gcd_import(name[level:], package, level)
2025-04-26T13:23:51.486595236Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-26T13:23:51.486599503Z   File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
2025-04-26T13:23:51.486603736Z   File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
2025-04-26T13:23:51.486617703Z   File "<frozen importlib._bootstrap>", line 1128, in _find_and_load_unlocked
2025-04-26T13:23:51.486621870Z   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
2025-04-26T13:23:51.486626023Z   File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
2025-04-26T13:23:51.486629945Z   File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
2025-04-26T13:23:51.486633825Z   File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
2025-04-26T13:23:51.486637737Z   File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
2025-04-26T13:23:51.486641737Z   File "<frozen importlib._bootstrap_external>", line 940, in exec_module
2025-04-26T13:23:51.486727772Z   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
2025-04-26T13:23:51.486836234Z   File "/comfyui/custom_nodes/comfyui-easy-use/py/__init__.py", line 2, in <module>
2025-04-26T13:23:51.486844711Z     from .libs.sampler import easySampler
2025-04-26T13:23:51.486849478Z   File "/comfyui/custom_nodes/comfyui-easy-use/py/libs/sampler.py", line 10, in <module>
2025-04-26T13:23:51.486853725Z     from ..modules.brushnet.model_patch import add_model_patch
2025-04-26T13:23:51.486858741Z   File "/comfyui/custom_nodes/comfyui-easy-use/py/modules/brushnet/__init__.py", line 12, in <module>
2025-04-26T13:23:51.486871215Z     from .model import BrushNetModel, PowerPaintModel
2025-04-26T13:23:51.486876288Z   File "/comfyui/custom_nodes/comfyui-easy-use/py/modules/brushnet/model.py", line 21, in <module>
2025-04-26T13:23:51.486880431Z     from diffusers.models.modeling_utils import ModelMixin
2025-04-26T13:23:51.486891918Z   File "/usr/local/lib/python3.11/dist-packages/diffusers/models/modeling_utils.py", line 40, in <module>
2025-04-26T13:23:51.486896405Z     from ..quantizers import DiffusersAutoQuantizer, DiffusersQuantizer
2025-04-26T13:23:51.486900408Z   File "/usr/local/lib/python3.11/dist-packages/diffusers/quantizers/__init__.py", line 15, in <module>
2025-04-26T13:23:51.486904594Z     from .auto import DiffusersAutoQuantizer
2025-04-26T13:23:51.486909094Z   File "/usr/local/lib/python3.11/dist-packages/diffusers/quantizers/auto.py", line 22, in <module>
2025-04-26T13:23:51.486913645Z     from .bitsandbytes import BnB4BitDiffusersQuantizer, BnB8BitDiffusersQuantizer
2025-04-26T13:23:51.486944893Z   File "/usr/local/lib/python3.11/dist-packages/diffusers/quantizers/bitsandbytes/__init__.py", line 2, in <module>
2025-04-26T13:23:51.486955775Z     from .utils import dequantize_and_replace, dequantize_bnb_weight, replace_with_bnb_linear
2025-04-26T13:23:51.486960655Z   File "/usr/local/lib/python3.11/dist-packages/diffusers/quantizers/bitsandbytes/utils.py", line 32, in <module>
2025-04-26T13:23:51.486965110Z     import bitsandbytes as bnb
2025-04-26T13:23:51.486969875Z   File "/usr/local/lib/python3.11/dist-packages/bitsandbytes/__init__.py", line 15, in <module>
2025-04-26T13:23:51.486973975Z     from .nn import modules
2025-04-26T13:23:51.486978273Z   File "/usr/local/lib/python3.11/dist-packages/bitsandbytes/nn/__init__.py", line 21, in <module>
2025-04-26T13:23:51.487104390Z     from .triton_based_modules import (
2025-04-26T13:23:51.487121220Z   File "/usr/local/lib/python3.11/dist-packages/bitsandbytes/nn/triton_based_modules.py", line 6, in <module>
2025-04-26T13:23:51.487128524Z     from bitsandbytes.triton.dequantize_rowwise import dequantize_rowwise
2025-04-26T13:23:51.487133492Z   File "/usr/local/lib/python3.11/dist-packages/bitsandbytes/triton/dequantize_rowwise.py", line 18, in <module>
2025-04-26T13:23:51.487138077Z     @triton.autotune(
2025-04-26T13:23:51.487142884Z      ^^^^^^^^^^^^^^^^
2025-04-26T13:23:51.487150392Z   File "/usr/local/lib/python3.11/dist-packages/triton/runtime/autotuner.py", line 378, in decorator
2025-04-26T13:23:51.487161092Z     return Autotuner(fn, fn.arg_names, configs, key, reset_to_zero, restore_value, pre_hook=pre_hook,
2025-04-26T13:23:51.487165526Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-26T13:23:51.487169829Z   File "/usr/local/lib/python3.11/dist-packages/triton/runtime/autotuner.py", line 130, in __init__
2025-04-26T13:23:51.487174422Z     self.do_bench = driver.active.get_benchmarker()
2025-04-26T13:23:51.487179176Z                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-26T13:23:51.487183226Z   File "/usr/local/lib/python3.11/dist-packages/triton/runtime/driver.py", line 23, in __getattr__
2025-04-26T13:23:51.487187383Z     self._initialize_obj()
2025-04-26T13:23:51.487195423Z   File "/usr/local/lib/python3.11/dist-packages/triton/runtime/driver.py", line 20, in _initialize_obj
2025-04-26T13:23:51.487199743Z     self._obj = self._init_fn()
2025-04-26T13:23:51.487206089Z                 ^^^^^^^^^^^^^^^
2025-04-26T13:23:51.487213333Z   File "/usr/local/lib/python3.11/dist-packages/triton/runtime/driver.py", line 9, in _create_driver
2025-04-26T13:23:51.487217712Z     return actives[0]()
2025-04-26T13:23:51.487223766Z            ^^^^^^^^^^^^
2025-04-26T13:23:51.487230779Z   File "/usr/local/lib/python3.11/dist-packages/triton/backends/nvidia/driver.py", line 535, in __init__
2025-04-26T13:23:51.487235026Z     self.utils = CudaUtils()  # TODO: make static
2025-04-26T13:23:51.487239132Z                  ^^^^^^^^^^^
2025-04-26T13:23:51.487247576Z   File "/usr/local/lib/python3.11/dist-packages/triton/backends/nvidia/driver.py", line 89, in __init__
2025-04-26T13:23:51.487255969Z     mod = compile_module_from_src(Path(os.path.join(dirname, "driver.c")).read_text(), "cuda_utils")
2025-04-26T13:23:51.487260203Z           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-26T13:23:51.487265699Z   File "/usr/local/lib/python3.11/dist-packages/triton/backends/nvidia/driver.py", line 66, in compile_module_from_src
2025-04-26T13:23:51.487270861Z     so = _build(name, src_path, tmpdir, library_dirs(), include_dir, libraries)
2025-04-26T13:23:51.487274916Z          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-26T13:23:51.487278979Z   File "/usr/local/lib/python3.11/dist-packages/triton/runtime/build.py", line 36, in _build
2025-04-26T13:23:51.487283586Z     subprocess.check_call(cc_cmd, stdout=subprocess.DEVNULL)
2025-04-26T13:23:51.487292078Z   File "/usr/lib/python3.11/subprocess.py", line 413, in check_call
2025-04-26T13:23:51.487301363Z     raise CalledProcessError(retcode, cmd)
2025-04-26T13:23:51.487323683Z subprocess.CalledProcessError: Command '['/usr/bin/gcc', '/tmp/tmps5hoz00i/main.c', '-O3', '-shared', '-fPIC', '-Wno-psabi', '-o', '/tmp/tmps5hoz00i/cuda_utils.cpython-311-x86_64-linux-gnu.so', '-lcuda', '-L/usr/local/lib/python3.11/dist-packages/triton/backends/nvidia/lib', '-L/lib/x86_64-linux-gnu', '-I/usr/local/lib/python3.11/dist-packages/triton/backends/nvidia/include', '-I/tmp/tmps5hoz00i', '-I/usr/include/python3.11']' returned non-zero exit status 1.
2025-04-26T13:23:51.487440618Z Cannot import /comfyui/custom_nodes/comfyui-easy-use module for custom nodes: Command '['/usr/bin/gcc', '/tmp/tmps5hoz00i/main.c', '-O3', '-shared', '-fPIC', '-Wno-psabi', '-o', '/tmp/tmps5hoz00i/cuda_utils.cpython-311-x86_64-linux-gnu.so', '-lcuda', '-L/usr/local/lib/python3.11/dist-packages/triton/backends/nvidia/lib', '-L/lib/x86_64-linux-gnu', '-I/usr/local/lib/python3.11/dist-packages/triton/backends/nvidia/include', '-I/tmp/tmps5hoz00i', '-I/usr/include/python3.11']' returned non-zero exit status 1.
2025-04-26T13:23:51.532712266Z [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/custom-node-list.json
2025-04-26T13:23:51.593873269Z [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/extension-node-map.json
2025-04-26T13:23:51.607322123Z [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/model-list.json
2025-04-26T13:23:51.633598319Z [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/github-stats.json
2025-04-26T13:23:51.639009600Z �[92m[rgthree-comfy] Loaded 42 fantastic nodes. 🎉�[00m
2025-04-26T13:23:51.647378136Z �[94mConDelta Nodes: �[92mLoaded�[0m
2025-04-26T13:23:51.689402040Z /tmp/tmpb7nr957r/main.c:5:10: fatal error: Python.h: No such file or directory
2025-04-26T13:23:51.689435922Z     5 | #include <Python.h>
2025-04-26T13:23:51.689442072Z       |          ^~~~~~~~~~
2025-04-26T13:23:51.689446732Z compilation terminated.
2025-04-26T13:23:51.692422488Z Traceback (most recent call last):
2025-04-26T13:23:51.692438298Z   File "/usr/local/lib/python3.11/dist-packages/diffusers/utils/import_utils.py", line 820, in _get_module
2025-04-26T13:23:51.692455941Z     return importlib.import_module("." + module_name, self.__name__)
2025-04-26T13:23:51.692460724Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-26T13:23:51.692464891Z   File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module
2025-04-26T13:23:51.692469321Z     return _bootstrap._gcd_import(name[level:], package, level)
2025-04-26T13:23:51.692473764Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-26T13:23:51.692477698Z   File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
2025-04-26T13:23:51.692481624Z   File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
2025-04-26T13:23:51.692485618Z   File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
2025-04-26T13:23:51.692490348Z   File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
2025-04-26T13:23:51.692494581Z   File "<frozen importlib._bootstrap_external>", line 940, in exec_module
2025-04-26T13:23:51.692498594Z   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
2025-04-26T13:23:51.692502804Z   File "/usr/local/lib/python3.11/dist-packages/diffusers/loaders/single_file_model.py", line 24, in <module>
2025-04-26T13:23:51.692506841Z     from ..quantizers import DiffusersAutoQuantizer
2025-04-26T13:23:51.692511003Z   File "/usr/local/lib/python3.11/dist-packages/diffusers/quantizers/__init__.py", line 15, in <module>
2025-04-26T13:23:51.692515028Z     from .auto import DiffusersAutoQuantizer
2025-04-26T13:23:51.692519213Z   File "/usr/local/lib/python3.11/dist-packages/diffusers/quantizers/auto.py", line 22, in <module>
2025-04-26T13:23:51.692523203Z     from .bitsandbytes import BnB4BitDiffusersQuantizer, BnB8BitDiffusersQuantizer
2025-04-26T13:23:51.692528138Z   File "/usr/local/lib/python3.11/dist-packages/diffusers/quantizers/bitsandbytes/__init__.py", line 2, in <module>
2025-04-26T13:23:51.692547738Z     from .utils import dequantize_and_replace, dequantize_bnb_weight, replace_with_bnb_linear
2025-04-26T13:23:51.692556846Z   File "/usr/local/lib/python3.11/dist-packages/diffusers/quantizers/bitsandbytes/utils.py", line 32, in <module>
2025-04-26T13:23:51.692561373Z     import bitsandbytes as bnb
2025-04-26T13:23:51.692565823Z   File "/usr/local/lib/python3.11/dist-packages/bitsandbytes/__init__.py", line 15, in <module>
2025-04-26T13:23:51.692570146Z     from .nn import modules
2025-04-26T13:23:51.692577196Z   File "/usr/local/lib/python3.11/dist-packages/bitsandbytes/nn/__init__.py", line 21, in <module>
2025-04-26T13:23:51.692581626Z     from .triton_based_modules import (
2025-04-26T13:23:51.692590617Z   File "/usr/local/lib/python3.11/dist-packages/bitsandbytes/nn/triton_based_modules.py", line 6, in <module>
2025-04-26T13:23:51.692596556Z     from bitsandbytes.triton.dequantize_rowwise import dequantize_rowwise
2025-04-26T13:23:51.692601473Z   File "/usr/local/lib/python3.11/dist-packages/bitsandbytes/triton/dequantize_rowwise.py", line 18, in <module>
2025-04-26T13:23:51.692608770Z     @triton.autotune(
2025-04-26T13:23:51.692615163Z      ^^^^^^^^^^^^^^^^
2025-04-26T13:23:51.692619350Z   File "/usr/local/lib/python3.11/dist-packages/triton/runtime/autotuner.py", line 378, in decorator
2025-04-26T13:23:51.692623300Z     return Autotuner(fn, fn.arg_names, configs, key, reset_to_zero, restore_value, pre_hook=pre_hook,
2025-04-26T13:23:51.692629430Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-26T13:23:51.692636933Z   File "/usr/local/lib/python3.11/dist-packages/triton/runtime/autotuner.py", line 130, in __init__
2025-04-26T13:23:51.692640993Z     self.do_bench = driver.active.get_benchmarker()
2025-04-26T13:23:51.692647007Z                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-26T13:23:51.692654167Z   File "/usr/local/lib/python3.11/dist-packages/triton/runtime/driver.py", line 23, in __getattr__
2025-04-26T13:23:51.692658140Z     self._initialize_obj()
2025-04-26T13:23:51.692662085Z   File "/usr/local/lib/python3.11/dist-packages/triton/runtime/driver.py", line 20, in _initialize_obj
2025-04-26T13:23:51.692666022Z     self._obj = self._init_fn()
2025-04-26T13:23:51.692672362Z                 ^^^^^^^^^^^^^^^
2025-04-26T13:23:51.692676435Z   File "/usr/local/lib/python3.11/dist-packages/triton/runtime/driver.py", line 9, in _create_driver
2025-04-26T13:23:51.692681407Z     return actives[0]()
2025-04-26T13:23:51.692686360Z            ^^^^^^^^^^^^
2025-04-26T13:23:51.692690962Z   File "/usr/local/lib/python3.11/dist-packages/triton/backends/nvidia/driver.py", line 535, in __init__
2025-04-26T13:23:51.692697017Z     self.utils = CudaUtils()  # TODO: make static
2025-04-26T13:23:51.692704299Z                  ^^^^^^^^^^^
2025-04-26T13:23:51.692708937Z   File "/usr/local/lib/python3.11/dist-packages/triton/backends/nvidia/driver.py", line 89, in __init__
2025-04-26T13:23:51.692713365Z     mod = compile_module_from_src(Path(os.path.join(dirname, "driver.c")).read_text(), "cuda_utils")
2025-04-26T13:23:51.692717289Z           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-26T13:23:51.692721359Z   File "/usr/local/lib/python3.11/dist-packages/triton/backends/nvidia/driver.py", line 66, in compile_module_from_src
2025-04-26T13:23:51.692727652Z     so = _build(name, src_path, tmpdir, library_dirs(), include_dir, libraries)
2025-04-26T13:23:51.692734445Z          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-26T13:23:51.692742882Z   File "/usr/local/lib/python3.11/dist-packages/triton/runtime/build.py", line 36, in _build
2025-04-26T13:23:51.692749395Z     subprocess.check_call(cc_cmd, stdout=subprocess.DEVNULL)
2025-04-26T13:23:51.692754025Z   File "/usr/lib/python3.11/subprocess.py", line 413, in check_call
2025-04-26T13:23:51.692761502Z     raise CalledProcessError(retcode, cmd)
2025-04-26T13:23:51.692767965Z subprocess.CalledProcessError: Command '['/usr/bin/gcc', '/tmp/tmpb7nr957r/main.c', '-O3', '-shared', '-fPIC', '-Wno-psabi', '-o', '/tmp/tmpb7nr957r/cuda_utils.cpython-311-x86_64-linux-gnu.so', '-lcuda', '-L/usr/local/lib/python3.11/dist-packages/triton/backends/nvidia/lib', '-L/lib/x86_64-linux-gnu', '-I/usr/local/lib/python3.11/dist-packages/triton/backends/nvidia/include', '-I/tmp/tmpb7nr957r', '-I/usr/include/python3.11']' returned non-zero exit status 1.
2025-04-26T13:23:51.692788776Z The above exception was the direct cause of the following exception:
2025-04-26T13:23:51.692803926Z Traceback (most recent call last):
2025-04-26T13:23:51.692808226Z   File "/comfyui/nodes.py", line 2128, in load_custom_node
2025-04-26T13:23:51.692812326Z     module_spec.loader.exec_module(module)
2025-04-26T13:23:51.692818076Z   File "<frozen importlib._bootstrap_external>", line 940, in exec_module
2025-04-26T13:23:51.692826024Z   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
2025-04-26T13:23:51.692831726Z   File "/comfyui/custom_nodes/ComfyUI-FramePackWrapper/__init__.py", line 1, in <module>
2025-04-26T13:23:51.692835816Z     from .nodes import NODE_CLASS_MAPPINGS, NODE_DISPLAY_NAME_MAPPINGS
2025-04-26T13:23:51.692841756Z   File "/comfyui/custom_nodes/ComfyUI-FramePackWrapper/nodes.py", line 22, in <module>
2025-04-26T13:23:51.692849219Z     from .diffusers_helper.models.hunyuan_video_packed import HunyuanVideoTransformer3DModelPacked
2025-04-26T13:23:51.692853216Z   File "/comfyui/custom_nodes/ComfyUI-FramePackWrapper/diffusers_helper/models/hunyuan_video_packed.py", line 8, in <module>
2025-04-26T13:23:51.692857466Z     from diffusers.loaders import FromOriginalModelMixin
2025-04-26T13:23:51.692862048Z   File "<frozen importlib._bootstrap>", line 1231, in _handle_fromlist
2025-04-26T13:23:51.692867861Z   File "/usr/local/lib/python3.11/dist-packages/diffusers/utils/import_utils.py", line 810, in __getattr__
2025-04-26T13:23:51.692875081Z     module = self._get_module(self._class_to_module[name])
2025-04-26T13:23:51.692880455Z              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-26T13:23:51.692886158Z   File "/usr/local/lib/python3.11/dist-packages/diffusers/utils/import_utils.py", line 822, in _get_module
2025-04-26T13:23:51.692893925Z     raise RuntimeError(
2025-04-26T13:23:51.692902865Z RuntimeError: Failed to import diffusers.loaders.single_file_model because of the following error (look up to see its traceback):
2025-04-26T13:23:51.692908145Z Command '['/usr/bin/gcc', '/tmp/tmpb7nr957r/main.c', '-O3', '-shared', '-fPIC', '-Wno-psabi', '-o', '/tmp/tmpb7nr957r/cuda_utils.cpython-311-x86_64-linux-gnu.so', '-lcuda', '-L/usr/local/lib/python3.11/dist-packages/triton/backends/nvidia/lib', '-L/lib/x86_64-linux-gnu', '-I/usr/local/lib/python3.11/dist-packages/triton/backends/nvidia/include', '-I/tmp/tmpb7nr957r', '-I/usr/include/python3.11']' returned non-zero exit status 1.
2025-04-26T13:23:51.693046174Z Cannot import /comfyui/custom_nodes/ComfyUI-FramePackWrapper module for custom nodes: Failed to import diffusers.loaders.single_file_model because of the following error (look up to see its traceback):
2025-04-26T13:23:51.693063977Z Command '['/usr/bin/gcc', '/tmp/tmpb7nr957r/main.c', '-O3', '-shared', '-fPIC', '-Wno-psabi', '-o', '/tmp/tmpb7nr957r/cuda_utils.cpython-311-x86_64-linux-gnu.so', '-lcuda', '-L/usr/local/lib/python3.11/dist-packages/triton/backends/nvidia/lib', '-L/lib/x86_64-linux-gnu', '-I/usr/local/lib/python3.11/dist-packages/triton/backends/nvidia/include', '-I/tmp/tmpb7nr957r', '-I/usr/include/python3.11']' returned non-zero exit status 1.
2025-04-26T13:23:51.711233190Z ComfyUI-GGUF: Partial torch compile only, consider updating pytorch
2025-04-26T13:23:52.298798882Z generated new fontManager
2025-04-26T13:23:52.877720836Z ------------------------------------------
2025-04-26T13:23:52.877753866Z �[34mComfyroll Studio v1.76 : �[92m 175 Nodes Loaded�[0m
2025-04-26T13:23:52.877758590Z ------------------------------------------
2025-04-26T13:23:52.877762820Z ** For changes, please see patch notes at https://github.com/Suzie1/ComfyUI_Comfyroll_CustomNodes/blob/main/Patch_Notes.md
2025-04-26T13:23:52.877782433Z ** For help, please see the wiki at https://github.com/Suzie1/ComfyUI_Comfyroll_CustomNodes/wiki
2025-04-26T13:23:52.877789560Z ------------------------------------------
2025-04-26T13:23:54.432392135Z FETCH ComfyRegistry Data: 5/82
2025-04-26T13:23:54.483251478Z Adding /comfyui/custom_nodes to sys.path
2025-04-26T13:23:54.753840344Z 
�[36mEfficiency Nodes:�[0m Attempting to add Control Net options to the 'HiRes-Fix Script' Node (comfyui_controlnet_aux add-on)...�[92mSuccess!�[0m
2025-04-26T13:23:54.758448471Z Loaded Efficiency nodes from /comfyui/custom_nodes/efficiency-nodes-comfyui
2025-04-26T13:23:54.765464245Z Loaded ControlNetPreprocessors nodes from /comfyui/custom_nodes/comfyui_controlnet_aux
2025-04-26T13:23:54.766264560Z Could not find AdvancedControlNet nodes
2025-04-26T13:23:54.767587307Z Could not find AnimateDiff nodes
2025-04-26T13:23:54.768873503Z Could not find IPAdapter nodes
2025-04-26T13:23:54.778498109Z Could not find VideoHelperSuite nodes
2025-04-26T13:23:54.779973746Z Could not load ImpactPack nodes Could not find ImpactPack nodes
2025-04-26T13:23:54.937980669Z ### Loading: ComfyUI-Impact-Pack (V8.14)
2025-04-26T13:23:55.083996641Z [Impact Pack] Wildcards loading done.
2025-04-26T13:23:55.084092996Z ### Loading: ComfyUI-Impact-Subpack (V1.2.10)
2025-04-26T13:23:55.154506120Z Creating new Ultralytics Settings v0.0.6 file ✅
2025-04-26T13:23:55.154552890Z View Ultralytics Settings with 'yolo settings' or at '/root/.config/Ultralytics/settings.json'
2025-04-26T13:23:55.154558420Z Update Settings with 'yolo settings key=value', i.e. 'yolo settings runs_dir=path/to/dir'. For help see https://docs.ultralytics.com/quickstart/#ultralytics-settings.
2025-04-26T13:23:55.390370046Z No module named 'dill'
2025-04-26T13:23:55.390427248Z !!!!!
2025-04-26T13:23:55.390436665Z [ComfyUI-Impact-Subpack] If this error occurs, please check the following link:
2025-04-26T13:23:55.390441705Z 	https://github.com/ltdrdata/ComfyUI-Impact-Pack/blob/Main/troubleshooting/TROUBLESHOOTING.md
2025-04-26T13:23:55.390450725Z !!!!!
2025-04-26T13:23:55.391586852Z Traceback (most recent call last):
2025-04-26T13:23:55.391600268Z   File "/comfyui/nodes.py", line 2128, in load_custom_node
2025-04-26T13:23:55.391605132Z     module_spec.loader.exec_module(module)
2025-04-26T13:23:55.391609728Z   File "<frozen importlib._bootstrap_external>", line 940, in exec_module
2025-04-26T13:23:55.391615031Z   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
2025-04-26T13:23:55.391619278Z   File "/comfyui/custom_nodes/comfyui-impact-subpack/__init__.py", line 23, in <module>
2025-04-26T13:23:55.391623922Z     imported_module = importlib.import_module(".modules.{}".format(module_name), __name__)
2025-04-26T13:23:55.391628128Z                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-26T13:23:55.391633205Z   File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module
2025-04-26T13:23:55.391640365Z     return _bootstrap._gcd_import(name[level:], package, level)
2025-04-26T13:23:55.391649548Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-26T13:23:55.391653911Z   File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
2025-04-26T13:23:55.391659805Z   File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
2025-04-26T13:23:55.391667328Z   File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
2025-04-26T13:23:55.391671755Z   File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
2025-04-26T13:23:55.391675960Z   File "<frozen importlib._bootstrap_external>", line 940, in exec_module
2025-04-26T13:23:55.391679958Z   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
2025-04-26T13:23:55.391684125Z   File "/comfyui/custom_nodes/comfyui-impact-subpack/modules/subpack_nodes.py", line 3, in <module>
2025-04-26T13:23:55.391689542Z     from . import subcore
2025-04-26T13:23:55.391694255Z   File "/comfyui/custom_nodes/comfyui-impact-subpack/modules/subcore.py", line 135, in <module>
2025-04-26T13:23:55.391698415Z     raise e
2025-04-26T13:23:55.391720584Z   File "/comfyui/custom_nodes/comfyui-impact-subpack/modules/subcore.py", line 62, in <module>
2025-04-26T13:23:55.391729164Z     import dill._dill
2025-04-26T13:23:55.391737297Z ModuleNotFoundError: No module named 'dill'
2025-04-26T13:23:55.391756771Z Cannot import /comfyui/custom_nodes/comfyui-impact-subpack module for custom nodes: No module named 'dill'
2025-04-26T13:23:55.421460074Z Total VRAM 24138 MB, total RAM 515602 MB
2025-04-26T13:23:55.421494010Z pytorch version: 2.7.0+cu126
2025-04-26T13:23:55.421942850Z Set vram state to: NORMAL_VRAM
2025-04-26T13:23:55.422060159Z Device: cuda:0 NVIDIA RTX A5000 : cudaMallocAsync
2025-04-26T13:23:55.598220469Z �[36;20m[/comfyui/custom_nodes/comfyui_controlnet_aux] | INFO -> Using ckpts path: /comfyui/custom_nodes/comfyui_controlnet_aux/ckpts�[0m
2025-04-26T13:23:55.598447833Z �[36;20m[/comfyui/custom_nodes/comfyui_controlnet_aux] | INFO -> Using symlinks: False�[0m
2025-04-26T13:23:55.598660063Z �[36;20m[/comfyui/custom_nodes/comfyui_controlnet_aux] | INFO -> Using ort providers: ['CUDAExecutionProvider', 'DirectMLExecutionProvider', 'OpenVINOExecutionProvider', 'ROCMExecutionProvider', 'CPUExecutionProvider', 'CoreMLExecutionProvider']�[0m
2025-04-26T13:23:55.650528152Z Making the "web\extensions\FizzleDorf" folder
2025-04-26T13:23:55.650628538Z Update to javascripts files detected
2025-04-26T13:23:55.650704051Z Copying Folder here to satisfy init, eventually I'll have stuff in here..txt to extensions folder
2025-04-26T13:23:55.660600394Z Traceback (most recent call last):
2025-04-26T13:23:55.660615001Z   File "/comfyui/nodes.py", line 2128, in load_custom_node
2025-04-26T13:23:55.660618392Z     module_spec.loader.exec_module(module)
2025-04-26T13:23:55.660621311Z   File "<frozen importlib._bootstrap_external>", line 940, in exec_module
2025-04-26T13:23:55.660624125Z   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
2025-04-26T13:23:55.660626426Z   File "/comfyui/custom_nodes/comfyui_fizznodes/__init__.py", line 57, in <module>
2025-04-26T13:23:55.660629810Z     from .ScheduledNodes import (
2025-04-26T13:23:55.660631906Z   File "/comfyui/custom_nodes/comfyui_fizznodes/ScheduledNodes.py", line 4, in <module>
2025-04-26T13:23:55.660637378Z     import numexpr
2025-04-26T13:23:55.660640277Z ModuleNotFoundError: No module named 'numexpr'
2025-04-26T13:23:55.660701916Z Cannot import /comfyui/custom_nodes/comfyui_fizznodes module for custom nodes: No module named 'numexpr'
2025-04-26T13:23:55.704609602Z �[92m[tinyterraNodes] �[32mLoaded�[0m
2025-04-26T13:23:55.741695873Z �[32mInitializing ControlAltAI Nodes�[0m
2025-04-26T13:23:55.764370535Z 
�[36mEfficiency Nodes:�[0m Attempting to add Control Net options to the 'HiRes-Fix Script' Node (comfyui_controlnet_aux add-on)...�[92mSuccess!�[0m
2025-04-26T13:23:56.374594612Z /tmp/tmp8v85e_gs/main.c:5:10: fatal error: Python.h: No such file or directory
2025-04-26T13:23:56.374634744Z     5 | #include <Python.h>
2025-04-26T13:23:56.374640420Z       |          ^~~~~~~~~~
2025-04-26T13:23:56.374644831Z compilation terminated.
2025-04-26T13:23:56.378123536Z Traceback (most recent call last):
2025-04-26T13:23:56.378163748Z   File "/comfyui/nodes.py", line 2128, in load_custom_node
2025-04-26T13:23:56.378168895Z     module_spec.loader.exec_module(module)
2025-04-26T13:23:56.378173758Z   File "<frozen importlib._bootstrap_external>", line 940, in exec_module
2025-04-26T13:23:56.378179428Z   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
2025-04-26T13:23:56.378183480Z   File "/comfyui/custom_nodes/teacache/__init__.py", line 2, in <module>
2025-04-26T13:23:56.378188185Z     from .nodes_diffusers import NODE_CLASS_MAPPINGS as DIFFUSERS_NODE_CLASS, NODE_DISPLAY_NAME_MAPPINGS as DIFFUSERS_NODES_DISPLAY
2025-04-26T13:23:56.378193205Z   File "/comfyui/custom_nodes/teacache/nodes_diffusers.py", line 8, in <module>
2025-04-26T13:23:56.378197640Z     from .models.cogvideox.custom_cogvideox_transformer_3d import CogVideoXTransformer3DModel
2025-04-26T13:23:56.378205482Z   File "/comfyui/custom_nodes/teacache/models/cogvideox/custom_cogvideox_transformer_3d.py", line 32, in <module>
2025-04-26T13:23:56.378235398Z     from diffusers.models.modeling_utils import ModelMixin
2025-04-26T13:23:56.378247749Z   File "/usr/local/lib/python3.11/dist-packages/diffusers/models/modeling_utils.py", line 40, in <module>
2025-04-26T13:23:56.378252742Z     from ..quantizers import DiffusersAutoQuantizer, DiffusersQuantizer
2025-04-26T13:23:56.378257185Z   File "/usr/local/lib/python3.11/dist-packages/diffusers/quantizers/__init__.py", line 15, in <module>
2025-04-26T13:23:56.378261242Z     from .auto import DiffusersAutoQuantizer
2025-04-26T13:23:56.378265852Z   File "/usr/local/lib/python3.11/dist-packages/diffusers/quantizers/auto.py", line 22, in <module>
2025-04-26T13:23:56.378270659Z     from .bitsandbytes import BnB4BitDiffusersQuantizer, BnB8BitDiffusersQuantizer
2025-04-26T13:23:56.378275135Z   File "/usr/local/lib/python3.11/dist-packages/diffusers/quantizers/bitsandbytes/__init__.py", line 2, in <module>
2025-04-26T13:23:56.378279969Z     from .utils import dequantize_and_replace, dequantize_bnb_weight, replace_with_bnb_linear
2025-04-26T13:23:56.378289762Z   File "/usr/local/lib/python3.11/dist-packages/diffusers/quantizers/bitsandbytes/utils.py", line 32, in <module>
2025-04-26T13:23:56.378294892Z     import bitsandbytes as bnb
2025-04-26T13:23:56.378300765Z   File "/usr/local/lib/python3.11/dist-packages/bitsandbytes/__init__.py", line 15, in <module>
2025-04-26T13:23:56.378307987Z     from .nn import modules
2025-04-26T13:23:56.378317547Z   File "/usr/local/lib/python3.11/dist-packages/bitsandbytes/nn/__init__.py", line 21, in <module>
2025-04-26T13:23:56.378324782Z     from .triton_based_modules import (
2025-04-26T13:23:56.378334977Z   File "/usr/local/lib/python3.11/dist-packages/bitsandbytes/nn/triton_based_modules.py", line 6, in <module>
2025-04-26T13:23:56.378341714Z     from bitsandbytes.triton.dequantize_rowwise import dequantize_rowwise
2025-04-26T13:23:56.378346851Z   File "/usr/local/lib/python3.11/dist-packages/bitsandbytes/triton/dequantize_rowwise.py", line 18, in <module>
2025-04-26T13:23:56.378352169Z     @triton.autotune(
2025-04-26T13:23:56.378358021Z      ^^^^^^^^^^^^^^^^
2025-04-26T13:23:56.378366449Z   File "/usr/local/lib/python3.11/dist-packages/triton/runtime/autotuner.py", line 378, in decorator
2025-04-26T13:23:56.378370971Z     return Autotuner(fn, fn.arg_names, configs, key, reset_to_zero, restore_value, pre_hook=pre_hook,
2025-04-26T13:23:56.378374964Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-26T13:23:56.378378881Z   File "/usr/local/lib/python3.11/dist-packages/triton/runtime/autotuner.py", line 130, in __init__
2025-04-26T13:23:56.378383537Z     self.do_bench = driver.active.get_benchmarker()
2025-04-26T13:23:56.378397844Z                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-26T13:23:56.378403048Z   File "/usr/local/lib/python3.11/dist-packages/triton/runtime/driver.py", line 23, in __getattr__
2025-04-26T13:23:56.378407908Z     self._initialize_obj()
2025-04-26T13:23:56.378412564Z   File "/usr/local/lib/python3.11/dist-packages/triton/runtime/driver.py", line 20, in _initialize_obj
2025-04-26T13:23:56.378417437Z     self._obj = self._init_fn()
2025-04-26T13:23:56.378422524Z                 ^^^^^^^^^^^^^^^
2025-04-26T13:23:56.378427038Z   File "/usr/local/lib/python3.11/dist-packages/triton/runtime/driver.py", line 9, in _create_driver
2025-04-26T13:23:56.378431848Z     return actives[0]()
2025-04-26T13:23:56.378439594Z            ^^^^^^^^^^^^
2025-04-26T13:23:56.378449624Z   File "/usr/local/lib/python3.11/dist-packages/triton/backends/nvidia/driver.py", line 535, in __init__
2025-04-26T13:23:56.378454237Z     self.utils = CudaUtils()  # TODO: make static
2025-04-26T13:23:56.378461301Z                  ^^^^^^^^^^^
2025-04-26T13:23:56.378474368Z   File "/usr/local/lib/python3.11/dist-packages/triton/backends/nvidia/driver.py", line 89, in __init__
2025-04-26T13:23:56.378478318Z     mod = compile_module_from_src(Path(os.path.join(dirname, "driver.c")).read_text(), "cuda_utils")
2025-04-26T13:23:56.378482154Z           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-26T13:23:56.378487393Z   File "/usr/local/lib/python3.11/dist-packages/triton/backends/nvidia/driver.py", line 66, in compile_module_from_src
2025-04-26T13:23:56.378499838Z     so = _build(name, src_path, tmpdir, library_dirs(), include_dir, libraries)
2025-04-26T13:23:56.378512226Z          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-26T13:23:56.378520388Z   File "/usr/local/lib/python3.11/dist-packages/triton/runtime/build.py", line 36, in _build
2025-04-26T13:23:56.378527693Z     subprocess.check_call(cc_cmd, stdout=subprocess.DEVNULL)
2025-04-26T13:23:56.378531776Z   File "/usr/lib/python3.11/subprocess.py", line 413, in check_call
2025-04-26T13:23:56.378535963Z     raise CalledProcessError(retcode, cmd)
2025-04-26T13:23:56.378541747Z subprocess.CalledProcessError: Command '['/usr/bin/gcc', '/tmp/tmp8v85e_gs/main.c', '-O3', '-shared', '-fPIC', '-Wno-psabi', '-o', '/tmp/tmp8v85e_gs/cuda_utils.cpython-311-x86_64-linux-gnu.so', '-lcuda', '-L/usr/local/lib/python3.11/dist-packages/triton/backends/nvidia/lib', '-L/lib/x86_64-linux-gnu', '-I/usr/local/lib/python3.11/dist-packages/triton/backends/nvidia/include', '-I/tmp/tmp8v85e_gs', '-I/usr/include/python3.11']' returned non-zero exit status 1.
2025-04-26T13:23:56.378558447Z Cannot import /comfyui/custom_nodes/teacache module for custom nodes: Command '['/usr/bin/gcc', '/tmp/tmp8v85e_gs/main.c', '-O3', '-shared', '-fPIC', '-Wno-psabi', '-o', '/tmp/tmp8v85e_gs/cuda_utils.cpython-311-x86_64-linux-gnu.so', '-lcuda', '-L/usr/local/lib/python3.11/dist-packages/triton/backends/nvidia/lib', '-L/lib/x86_64-linux-gnu', '-I/usr/local/lib/python3.11/dist-packages/triton/backends/nvidia/include', '-I/tmp/tmp8v85e_gs', '-I/usr/include/python3.11']' returned non-zero exit status 1.
2025-04-26T13:23:56.635256128Z �[34mWAS Node Suite: �[0mCreated default conf file at `/comfyui/custom_nodes/was-node-suite-comfyui/was_suite_config.json`.�[0m
2025-04-26T13:23:57.341037991Z �[34mWAS Node Suite: �[0mOpenCV Python FFMPEG support is enabled�[0m
2025-04-26T13:23:57.341063581Z �[34mWAS Node Suite �[93mWarning: �[0m`ffmpeg_bin_path` is not set in `/comfyui/custom_nodes/was-node-suite-comfyui/was_suite_config.json` config file. Will attempt to use system ffmpeg binaries if available.�[0m
2025-04-26T13:23:57.482432842Z FETCH ComfyRegistry Data: 10/82
2025-04-26T13:23:58.000187683Z �[34mWAS Node Suite: �[0mFinished.�[0m �[32mLoaded�[0m �[0m220�[0m �[32mnodes successfully.�[0m
2025-04-26T13:23:58.000244142Z 	�[3m�[93m"Art is the bridge that connects imagination to reality."�[0m�[3m - Unknown�[0m
2025-04-26T13:23:58.002035535Z Import times for custom nodes:
2025-04-26T13:23:58.002145730Z    0.0 seconds: /comfyui/custom_nodes/websocket_image_save.py
2025-04-26T13:23:58.002171423Z    0.0 seconds: /comfyui/custom_nodes/ComfyUI-CustomScheduler
2025-04-26T13:23:58.002224043Z    0.0 seconds: /comfyui/custom_nodes/prompt_templates
2025-04-26T13:23:58.002313162Z    0.0 seconds: /comfyui/custom_nodes/cg-use-everywhere
2025-04-26T13:23:58.002323119Z    0.0 seconds: /comfyui/custom_nodes/skimmed_cfg
2025-04-26T13:23:58.002372029Z    0.0 seconds: /comfyui/custom_nodes/ComfyUI-FlexTools
2025-04-26T13:23:58.002401556Z    0.0 seconds: /comfyui/custom_nodes/sdxl_prompt_styler
2025-04-26T13:23:58.002458891Z    0.0 seconds: /comfyui/custom_nodes/sd-dynamic-thresholding
2025-04-26T13:23:58.002505608Z    0.0 seconds: /comfyui/custom_nodes/comfyui-detail-daemon
2025-04-26T13:23:58.002549774Z    0.0 seconds: /comfyui/custom_nodes/efficiency-nodes-comfyui
2025-04-26T13:23:58.002590268Z    0.0 seconds: /comfyui/custom_nodes/DistanceSampler
2025-04-26T13:23:58.002637440Z    0.0 seconds: /comfyui/custom_nodes/comfyui-sampler-lcm-alternative
2025-04-26T13:23:58.002691367Z    0.0 seconds: /comfyui/custom_nodes/GF_nodes
2025-04-26T13:23:58.002735863Z    0.0 seconds: /comfyui/custom_nodes/ComfyUI-ConDelta
2025-04-26T13:23:58.002779119Z    0.0 seconds: /comfyui/custom_nodes/comfyui-florence2
2025-04-26T13:23:58.002827983Z    0.0 seconds (IMPORT FAILED): /comfyui/custom_nodes/comfyui_fizznodes
2025-04-26T13:23:58.002874476Z    0.0 seconds: /comfyui/custom_nodes/Comfy_KepListStuff
2025-04-26T13:23:58.002923248Z    0.0 seconds: /comfyui/custom_nodes/ComfyUI-TiledDiffusion
2025-04-26T13:23:58.002965321Z    0.0 seconds: /comfyui/custom_nodes/comfyui-post-processing-nodes
2025-04-26T13:23:58.003021922Z    0.0 seconds: /comfyui/custom_nodes/comfyui-custom-scripts
2025-04-26T13:23:58.003066258Z    0.0 seconds: /comfyui/custom_nodes/comfyui-automaticcfg
2025-04-26T13:23:58.003091572Z    0.0 seconds: /comfyui/custom_nodes/comfyui_controlnet_aux
2025-04-26T13:23:58.003143109Z    0.0 seconds: /comfyui/custom_nodes/controlaltai-nodes
2025-04-26T13:23:58.003191875Z    0.0 seconds: /comfyui/custom_nodes/ComfyUI-ReplenishNodes
2025-04-26T13:23:58.003234231Z    0.0 seconds: /comfyui/custom_nodes/ComfyUI-GGUF
2025-04-26T13:23:58.003274701Z    0.0 seconds: /comfyui/custom_nodes/comfyui-various
2025-04-26T13:23:58.003316290Z    0.0 seconds: /comfyui/custom_nodes/comfyui_essentials
2025-04-26T13:23:58.003363384Z    0.0 seconds: /comfyui/custom_nodes/comfyui-videohelpersuite
2025-04-26T13:23:58.003406038Z    0.0 seconds (IMPORT FAILED): /comfyui/custom_nodes/ComfyUI-FramePackWrapper
2025-04-26T13:23:58.003456205Z    0.0 seconds (IMPORT FAILED): /comfyui/custom_nodes/teacache
2025-04-26T13:23:58.003494113Z    0.1 seconds: /comfyui/custom_nodes/ComfyUI-Manager
2025-04-26T13:23:58.003537840Z    0.1 seconds: /comfyui/custom_nodes/comfyui_tinyterranodes
2025-04-26T13:23:58.003591279Z    0.1 seconds: /comfyui/custom_nodes/comfyui-dynamicprompts
2025-04-26T13:23:58.003633328Z    0.1 seconds: /comfyui/custom_nodes/comfyui-kjnodes
2025-04-26T13:23:58.003681814Z    0.2 seconds: /comfyui/custom_nodes/rgthree-comfy
2025-04-26T13:23:58.003727196Z    0.2 seconds: /comfyui/custom_nodes/comfyui-impact-pack
2025-04-26T13:23:58.003781963Z    0.3 seconds (IMPORT FAILED): /comfyui/custom_nodes/comfyui-easy-use
2025-04-26T13:23:58.003813917Z    0.3 seconds: /comfyui/custom_nodes/comfyui-art-venture
2025-04-26T13:23:58.003857347Z    0.3 seconds (IMPORT FAILED): /comfyui/custom_nodes/comfyui-impact-subpack
2025-04-26T13:23:58.003902787Z    0.6 seconds: /comfyui/custom_nodes/flow2-wan-video
2025-04-26T13:23:58.003941779Z    1.1 seconds: /comfyui/custom_nodes/ComfyUI_Comfyroll_CustomNodes
2025-04-26T13:23:58.003982542Z    1.6 seconds: /comfyui/custom_nodes/ComfyUI_ExtraModels
2025-04-26T13:23:58.004037966Z    1.6 seconds: /comfyui/custom_nodes/was-node-suite-comfyui
2025-04-26T13:23:58.024997663Z Starting server
2025-04-26T13:23:58.025465544Z To see the GUI go to: http://127.0.0.1:8188
2025-04-26T13:23:58.062468462Z worker-comfyui - API is reachable
2025-04-26T13:23:58.066170599Z got prompt
2025-04-26T13:23:58.066793128Z invalid prompt: {'type': 'invalid_prompt', 'message': 'Cannot execute because node TeaCache does not exist.', 'details': "Node ID '#158'", 'extra_info': {}}
2025-04-26T13:23:58.120058081Z {"requestId": "e0247faf-7f63-41fa-955f-c164db4ebe8c-u2", "message": "Finished.", "level": "INFO"}
2025-04-26T13:24:00.461146738Z FETCH ComfyRegistry Data: 15/82
2025-04-26T13:24:03.460357235Z FETCH ComfyRegistry Data: 20/82

[TimPietrusky]

@BenDes21 thanks for reporting this! can you please share both the snapshot + workflow that you have used here?

[BenDes21]

@BenDes21 thanks for reporting this! can you please share both the snapshot + workflow that you have used here?

Hi there

Here the snapshot file

2025-04-26_15-28-14_snapshot.json

But I think the problem come from the dockerfile, with the help the google geminin 2.5, I was able to fix it. Don't remember exactly what I did but I can send you my dockerfile if you want

[TimPietrusky]

@BenDes21 i will look into this today, so hopefully gemini 2.5 will also try the problem :D but of course i would love to see the dockerfile, thank you!

[BenDes21]

@BenDes21 i will look into this today, so hopefully gemini 2.5 will also try the problem :D but of course i would love to see the dockerfile, thank you!

Thanks, I can send you the dockerfile on discord :)

Ah also, if you know how to fix this error who happen sometimes for an unknow reason it's will be awesome ::D

[TimPietrusky]

@BenDes21 awesome, thank you!

Ah also, if you know how to fix this error who happen sometimes for an unknow reason it's will be awesome ::D

this is the same thing you have reported in #103 right? that bug is also on my list for this week.

[BenDes21]

@BenDes21 awesome, thank you!

Ah also, if you know how to fix this error who happen sometimes for an unknow reason it's will be awesome ::D

this is the same thing you have reported in #103 right? that bug is also on my list for this week.

Yes this is the same, awesome, thanks a lot!

[TimPietrusky]

@BenDes21 would you mind to dm me on discord? would love to work directly with you to get this solved.

[BenDes21]

@BenDes21 would you mind to dm me on discord? would love to work directly with you to get this solved.

Hi there :) Yes sure!

[AIExplorer25]

Please let me know if you have any fix for the custom nodes, I am also facing same issue while running the worker, it says custom node import error. I have kept the snapshot.json file at root dir, and used the latest docker image 'runpod/worker-comfyui:5.0.0-base' as base image. I am in urgent need for the fix, please let me know if you have found any alternate solution

[artokun]

having the same issue. can't deploy a serverless without being able to install my custom_nodes

[TimPietrusky]

@BenDes21 @AIExplorer25 @artokun we are replacing "restore snapshot" in #118 with a super minimal dockerfile, which is using the comfy-cli to install custom nodes and download models in the correct location.

i'm also experimenting with an comfy-agent that can already convert a ComfyUI workflow into a serverless endpoint with custom nodes & models (will share more about this soon, dm me on our discord if you want to beta test the agent with your workflow)

[TimPietrusky]

🎉 This issue has been resolved in version 5.0.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀