[BUG] Startup on 3.1.0 consumes too much cpu

[twsouthwick]

Environment

Self-Hosted (Docker)

System

Docker version 26.1.1, build 4cf5afa

Version

3.1.0

Describe the problem

After launching the container with switching to 3.1.0, the memory/cpu consumption starts going through the roof. I hadn't put constraints on this container (now I do :)) and it was taking over my VM:

6e8b003b68a1   dashy                       190.58%   739.2MiB / 1GiB       72.19%    8.11kB / 1.68kB   897kB / 24.6kB    212

it will cycle through very quickly to take all the available memory/cpu.

Additional info

No response

Please tick the boxes

• You have explained the issue clearly, and included all relevant info
• You are using a supported version of Dashy
• You've checked that this issue hasn't already been raised
• You've checked the docs and troubleshooting guide
• You agree to the code of conduct

[CrazyWolf13]

Hi
I can't seem to reproduce your issue, neither lissy I guess.

Could you share a bit more?
RAM usage on dashy container available ram, cpu usage.

Does it ever go down, or just the first build?

[DerLokich]

Same thing. Updated today and RAM & CPU overloaded

[CrazyWolf13]

Same thing. Updated today and RAM & CPU overloaded

Could you provide more details?

Does dashy always consume such high amouts, just at start?
Can you share like some more info about ram and cpu usage of the container?
This can be viewed the easiest with portainer.

[DerLokich]

Same thing. Updated today and RAM & CPU overloaded

Could you provide more details?

Does dashy always consume such high amouts, just at start? Can you share like some more info about ram and cpu usage of the container? This can be viewed the easiest with portainer.

I just make repull to update dashy and right after start i see 4 processes "vue-cli-service build", that consume all avaible memory and CPU, so other containers were not avaible. Even console works with huge lags... Only hard reset helps and right after rebbot need to quick stop dashy

[CrazyWolf13]

Please be more specific.

Does the usage go down after some time, like 5 min at max?

How much RAM exactly?
How much have you assigned?

The first build of dashy takes a bit more ressources, but there aren't any more rebuilds neccessary later on.

[DerLokich]

Please be more specific.

Does the usage go down after some time, like 5 min at max?

How much RAM exactly? How much have you assigned?

The first build of dashy takes a bit more ressources, but there aren't any more rebuilds neccessary later on.

I've waited about 30 mins, but as other contaniers doesn't working, i do hard reset.
Host 2 Gb RAM, 2 vCPU, all avaible for dashy.

[CrazyWolf13]

@DerLokich

Just for testing, could you extend it to 3GB RAM ?

And would you mind sending a screenshot of the ressource usage from container start to like 10minutes?
After freshly pulling the image?

This can be done with portainer:

docker run -d -p 8000:8000 -p 9443:9443 --name portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ce:latest

Thanks!

[CrazyWolf13]

To @twsouthwick

1GB is a bit less for dashy, could you try with 2GB?
Dashy uses quite a big amount of ressources for the first build, but since V3, there are no more rebuilds neccessary on config changes.

While I got it to work with a raspberrypi which only has 1GB of RAM, memory management is different compared to like a VM or a big server running dashy.

[DerLokich]

@DerLokich

Just for testing, could you extend it to 3GB RAM ?

And would you mind sending a screenshot of the ressource usage from container start to like 10minutes? After freshly pulling the image?

This can be done with portainer:

docker run -d -p 8000:8000 -p 9443:9443 --name portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ce:latest

Thanks!

Can't extend, it's VPS in another country :).

Just for testing i tried to take screenshot in portainer, but VM stuck after eating all resources...
Only last output of htop avaible now:

[DerLokich]

Waited till now, so it's about 90 min, no effect. Hard reset VM.

[twsouthwick]

I've updated it to 3gb, and even with the resource constraints, my VM becomes unresponsive:

services:
  dashy:
    container_name: 'dashy'
    image: lissy93/dashy:3.1.0
    restart: no #unless-stopped
    deploy:
      resources:
        limits:
          cpus: '2'
          memory: 3gb

I'll try to gather some more details this weekendn

[twsouthwick]

Out of curiosity, what is the value of rebuilding the dashboard in the container? Why not just ship a prebuilt component with the container? I think historically dashy needed the config as part of the build, but now it doesn't need it, right? It makes the dashboard a fairly heavyweight system as it is.

[CrazyWolf13]

Out of curiosity, what is the value of rebuilding the dashboard in the container?

I think currently it needs to rebuild for other things like for example auth needs a rebuild, custom css, added files like icons, defaultView change and some other very specific things.

So in general file changes or additions not related to conf.yml still require a rebuild.

@Lissy93 Please correct me if I'm wrong.

[CrazyWolf13]

I'll try to gather some more details this weekendn

Thanks, that'd be awesome!

Because I really cannot reproduce this here and I think lissy has also never been able to reproduce this issue.

[bobodevil]

I've had the same issue as well. I thought it was my issue, so I rebuilt the container, but it just chewed up the CPU and memory again.

Edit: I had my watchtower auto update. I just read about the 3.0+ update.
changing -p 8080:8080 and /app/user-data/conf.yml fixed my issue.

[Lissy93]

I'll look into this, as somethings definatley not right here, while Node (used for build) is a bit heavy, it shouldn't ever be consuming that amount of resources.

So far, haven't been able to reproduce.

@DerLokich - it looks like build is being run multiple times, which is weird because it is only needed once.

[twsouthwick]

Looks like memory consumption isn't the issue - but it spikes to 20cpus and then the system sends a kill as it's taken all the cpus of the machine.

And the system sends a kill:

 WARN  A new version of sass-loader is available. Please upgrade for best experience.
error Command failed with signal "SIGKILL".
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
node:events:496
      throw er; // Unhandled 'error' event
      ^

Error: write EPIPE
    at target._send (node:internal/child_process:879:20)
    at target.send (node:internal/child_process:752:19)
    at callback (/app/node_modules/worker-farm/lib/child/index.js:32:17)
    at module.exports (/app/node_modules/terser-webpack-plugin/dist/worker.js:13:5)
    at handle (/app/node_modules/worker-farm/lib/child/index.js:44:8)
    at process.<anonymous> (/app/node_modules/worker-farm/lib/child/index.js:55:3)
    at process.emit (node:events:518:28)
    at emit (node:internal/child_process:951:14)
    at process.processTicksAndRejections (node:internal/process/task_queues:83:21)
Emitted 'error' event on process instance at:
    at node:internal/child_process:883:39
    at process.processTicksAndRejections (node:internal/process/task_queues:77:11) {
  errno: -32,
  code: 'EPIPE',
  syscall: 'write'
}

Node.js v20.11.1
node:events:496
      throw er; // Unhandled 'error' event
      ^

Error: write EPIPE
    at target._send (node:internal/child_process:879:20)
    at target.send (node:internal/child_process:752:19)
    at callback (/app/node_modules/worker-farm/lib/child/index.js:32:17)
    at module.exports (/app/node_modules/terser-webpack-plugin/dist/worker.js:13:5)
    at handle (/app/node_modules/worker-farm/lib/child/index.js:44:8)
    at process.<anonymous> (/app/node_modules/worker-farm/lib/child/index.js:55:3)
    at process.emit (node:events:518:28)
    at emit (node:internal/child_process:951:14)
    at process.processTicksAndRejections (node:internal/process/task_queues:83:21)
Emitted 'error' event on process instance at:
    at node:internal/child_process:883:39
    at process.processTicksAndRejections (node:internal/process/task_queues:77:11) {
  errno: -32,
  code: 'EPIPE',
  syscall: 'write'
}

[kzkvv]

I see logs and it looks like Dashy is building in docker while startup. It looks strange. It consumes almost all my VPS resources (I have a small VPS with 2 CPU and 2GB RAM. It also takes a lot of time, I've been waiting about 5 minutes and Dashy still doesn't start :(
Maybe I missunderstand the main idea, but I think any application should be built once, packaged once into docker-image. Once, not N-times during each (or first) startup. Or are there some technical reasons to perform build during container startup?

There is a part of my docker-compose with Dashy:

  dashy:
    image: lissy93/dashy
    container_name: dashy
    ports:
      - "4000:8080"
    # Set any environmental variables
    environment:
      - NODE_ENV=production
    # Specify your user ID and group ID. You can find this by running `id -u` and `id -g`
      - UID=1000
      - GID=1000
    restart: unless-stopped
    healthcheck:
      test: ['CMD', 'node', '/app/services/healthcheck']
      interval: 1m30s
      timeout: 10s
      retries: 3
      start_period: 40s

[dacagi]

which version works? anyone?

I am running latest (v3.1.1) on a Raspberry 4 (4GB RAM). Previously I was running v2.1.0 for a year without issues. Tested both ARM and x86/64 architectures.

This issue is about it consuming too many resources during initial startup, not that it doesn't work at all.

[dasunsrule32]

The issue happens when it's building the /dist dir. I've attempted mounting that to a folder on the local system so that it doesn't need to rebuild it each time and haven't had much success going that route as of yet.

It takes up about 2.5G of RAM during the build, when it's completed, it returns to normal usage.

[SimonWoidig]

The docker image has the following command/CMD set: [yarn,build-and-start] which is then passed to the docker-entrypoint.sh that just executes that. The command should rather be server.js which then gets executed by the entrypoint and just starts the built Dashy.
So in this current state, the image is not unusable but it doesn't work with the default command.
This becomes a problem in the Helm chart (https://github.com/vyrtualsynthese/selfhosted-helmcharts/tree/main/charts/dashy) because there is no possibility to change the executed command.

EDIT: seems that this issue was brought in via #1559. Before the PR associated with this issue was merged, the command [yarn,start] was executed instead. That runs node server which works the same as the workaround I mentioned.

[CrazyWolf13]

@SimonWoidig

No #1559 did not bring up that issue.

Before V3 yarn build-and-start was the default command, to ensure when for example a user mapps files into the container, they get picked up and dashy rebuilds, on config changes, on display-view changes etc.

in V3 config loading got dynamic, so rebuilding was no longer needed, a user suggested to change it to start instead of build-and-start -> #1543

Then we kinda got flooded with issues, and due to lissy being invited to Github Accelerator a while ago she barely had time to look into this, so we decided temporarly reverting the command back to it's original yarn build-and-start, as it was before V3 is the best idea.

I have since not heard back from lissy.

Have you got anything to add @Lissy93

[SimonWoidig]

@SimonWoidig

No #1559 did not bring up that issue.

Before V3 yarn build-and-start was the default command, to ensure when for example a user mapps files into the container, they get picked up and dashy rebuilds, on config changes, on display-view changes etc.

in V3 config loading got dynamic, so rebuilding was no longer needed, a user suggested to change it to start instead of build-and-start -> #1543

Then we kinda got flooded with issues, and due to lissy being invited to Github Accelerator a while ago she barely had time to look into this, so we decided temporarly reverting the command back to it's original yarn build-and-start, as it was before V3 is the best idea.

I have since not heard back from lissy.

Have you got anything to add @Lissy93

Sorry if I misunderstood. I just used git blame to see, when the command was changed. I didn't know about the config rebuilding and such.
I really like this project and I just wanted to find the root cause.
Thank you for your work and I understand if life gets a bit too busy. I just hope, we can get this fixed. And I apologize if I came off a bit harsh.

[DanielHabenicht]

Mounting /app/dist/ does not work as a workaround because some script tries to remove it before the build.
I must also support @thegamerx1 I have never before seen a docker image that builds the fronted on start (let alone on every restart!). It should be included in the pre-build image.
Also, this is a Dashboard - it should not take up close to 2 GB of RAM (even during start).

Hope it can get resolved, otherwise its really nice :)

[CrazyWolf13]

I started slowly shifting away to alternatives, sadly ....
https://github.com/awesome-selfhosted/awesome-selfhosted?tab=readme-ov-file#personal-dashboards

I have a feeling, this won't get fixed that quickly.

[hqoq]

Sadly, I'm having the exact same problem, my 2c2g VPS is getting stuck on builds and the VPS is completely unresponsive.

I have to hard reboot the vps.

Here is the docker-compose.yml I use:

services:
  dashy:
    image: lissy93/dashy
    container_name: dashy
    volumes:
      - ./data/item-icons:/app/user-data/item-icons/
      - ./data/conf.yml:/app/user-data/conf.yml
    environment:
      - NODE_ENV=production
      - UID=0
      - GID=0
    ports:
      - 127.0.0.1:8080:8080
    restart: unless-stopped

    healthcheck:
      test: ['CMD', 'node', '/app/services/healthcheck']
      interval: 60m
      timeout: 10s
      retries: 3
      start_period: 40s

docker daemon log:

`Jan 05 22:27:14 rush systemd[1]: Started docker.service - Docker Application Container Engine.
Jan 05 22:27:35 rush dockerd[103278]: time="2025-01-06T06:27:35.621496754Z" level=info msg="ignoring event" container=caa16f7af909dbeb6cc600ec193c9f69734eab05efa58b51e15737c925b9748d modu>
Jan 05 22:27:35 rush dockerd[103278]: time="2025-01-06T06:27:35.627536897Z" level=info msg="ignoring event" container=3da9c5a700489f0cd61de2d38631fa35253462b0b4105eefde0bdfa19c55d62a modu>
Jan 05 22:27:35 rush dockerd[103278]: time="2025-01-06T06:27:35.638528690Z" level=warning msg="ShouldRestart failed, container will not be restarted" container=caa16f7af909dbeb6cc600ec193>
Jan 05 22:27:35 rush dockerd[103278]: time="2025-01-06T06:27:35.644619358Z" level=warning msg="ShouldRestart failed, container will not be restarted" container=3da9c5a700489f0cd61de2d3863>
Jan 06 00:08:31 rush dockerd[103278]: time="2025-01-06T08:08:31.793058910Z" level=info msg="ignoring event" container=64cb2967563b39ae1372fc2b2b945f4ac9cd9763b910a32c556b097010b94e39 modu>
Jan 06 00:08:31 rush dockerd[103278]: time="2025-01-06T08:08:31.803500230Z" level=warning msg="ShouldRestart failed, container will not be restarted" container=64cb2967563b39ae1372fc2b2b9>
Jan 06 13:00:07 rush dockerd[103278]: time="2025-01-06T21:00:07.165283599Z" level=info msg="ignoring event" container=eed16614ce6022678a6123ada481779e078754e037f5743f78d50fa336401996 modu>
Jan 07 13:00:05 rush dockerd[103278]: time="2025-01-07T21:00:05.919925281Z" level=info msg="ignoring event" container=914d2b8c3d825766534b4ded90e0d6a0dd0faf656e3d0b5bebf6cfa6863ea9db modu>
Jan 07 18:08:02 rush dockerd[103278]: time="2025-01-08T02:08:02.239892668Z" level=error msg="Handler for GET /containers/json returned error: write unix /run/docker.sock->@: write: broken>
Jan 07 18:08:02 rush dockerd[103278]: 2025/01/08 02:08:02 http: superfluous response.WriteHeader call from go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*respWriterWrappe>
Jan 07 18:09:20 rush dockerd[103278]: time="2025-01-08T02:09:20.653529915Z" level=error msg="Could not send KILL signal to container process" container=d5761ee7072a3c02d3a1e6134e17cd8876c>
Jan 07 18:10:55 rush dockerd[103278]: time="2025-01-08T02:10:53.922993174Z" level=warning msg="Health check for container d5761ee7072a3c02d3a1e6134e17cd8876cce148dcda3bd61344ec96497b30b7 >
Jan 07 18:11:29 rush dockerd[103278]: time="2025-01-08T02:11:27.549564707Z" level=error msg="stream copy error: reading from a closed fifo"`

I hope it helps locate the issue.

[webysther]

The main problem is not CPU, is memory, limit memory to less than 2G broke and CPU limit don't break the app, this can be fixed on CPU side with:

services:
  dashy:
    image: ghcr.io/lissy93/dashy:latest
    deploy:
      resources:
        limits:
          cpus: '1.00'
          memory: 2G

Another important point: increase the CPU from 1 to 8 don't speedup too much the build, so keep in the lower at 1 core is far enough.

[webysther]

Please test with the latest version using the limits for CPU, I think this issue is resolved.

[webysther]

Looks like memory consumption isn't the issue - but it spikes to 20cpus and then the system sends a kill as it's taken all the cpus of the machine.

And the system sends a kill:

 WARN  A new version of sass-loader is available. Please upgrade for best experience.
error Command failed with signal "SIGKILL".
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
node:events:496
      throw er; // Unhandled 'error' event
      ^

Error: write EPIPE
    at target._send (node:internal/child_process:879:20)
    at target.send (node:internal/child_process:752:19)
    at callback (/app/node_modules/worker-farm/lib/child/index.js:32:17)
    at module.exports (/app/node_modules/terser-webpack-plugin/dist/worker.js:13:5)
    at handle (/app/node_modules/worker-farm/lib/child/index.js:44:8)
    at process.<anonymous> (/app/node_modules/worker-farm/lib/child/index.js:55:3)
    at process.emit (node:events:518:28)
    at emit (node:internal/child_process:951:14)
    at process.processTicksAndRejections (node:internal/process/task_queues:83:21)
Emitted 'error' event on process instance at:
    at node:internal/child_process:883:39
    at process.processTicksAndRejections (node:internal/process/task_queues:77:11) {
  errno: -32,
  code: 'EPIPE',
  syscall: 'write'
}

Node.js v20.11.1
node:events:496
      throw er; // Unhandled 'error' event
      ^

Error: write EPIPE
    at target._send (node:internal/child_process:879:20)
    at target.send (node:internal/child_process:752:19)
    at callback (/app/node_modules/worker-farm/lib/child/index.js:32:17)
    at module.exports (/app/node_modules/terser-webpack-plugin/dist/worker.js:13:5)
    at handle (/app/node_modules/worker-farm/lib/child/index.js:44:8)
    at process.<anonymous> (/app/node_modules/worker-farm/lib/child/index.js:55:3)
    at process.emit (node:events:518:28)
    at emit (node:internal/child_process:951:14)
    at process.processTicksAndRejections (node:internal/process/task_queues:83:21)
Emitted 'error' event on process instance at:
    at node:internal/child_process:883:39
    at process.processTicksAndRejections (node:internal/process/task_queues:77:11) {
  errno: -32,
  code: 'EPIPE',
  syscall: 'write'
}

Please make this test:

• Stop the container
• Move the data folder and create a empty one
• Start the container and wait the default page
• Get only the conf and import inside the screen, if show warning fix
• Move the icons

[webysther]

Based on #1836 to anyone that want to test, I think 1.5G works now, I hate not round numbers, so I never tested. Another point, for really vCPU constraint VM, set a 0.50 for CPU in limits I think still works, someone need to test both and put here the build time, but only will be fair if remove the container and data because now the project is using incremental build.