Skip to content

[Bug] When is heavy load mempool my Full Node disconnects from peers #19518

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
heget opened this issue Apr 18, 2025 · 8 comments
Open

[Bug] When is heavy load mempool my Full Node disconnects from peers #19518

heget opened this issue Apr 18, 2025 · 8 comments
Labels
bug Something isn't working

Comments

@heget
Copy link

heget commented Apr 18, 2025

What happened?

Hi,

I have noticed several times that something is wrong in the latest version, when the mempool is full, my fullnode is out of sync, or all connections to peers are lost.
Here in the log you can also see when the mempool is full and what starts as an error like: Broadcasting 7 removed transactions to 0 peers.
This really only happens when the mempool is full and I then have to close the entire GUI, otherwise it will not synchronize again.

Version

2.5.3.

What platform are you using?

Windows

What ui mode are you using?

GUI

Relevant log output

2025-04-14T11:05:14.381 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.97s, pre_validation: 0.31s, CLVM: 0.30s, post-process: 1.20s, cost: 3895678750, percent full: 35.415% header_hash: 018528f64a44cdcfdd3bc64be26b49cd8faf7e899adabe0cdd6862853fc4389b height: 6904615
2025-04-14T11:18:37.282 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.64s, pre_validation: 0.19s, CLVM: 0.19s, post-process: 1.01s, cost: 3865900007, percent full: 35.145% header_hash: 25618f3db590fde7de4031185877287a57fbe514bcf4a1f01e1b5dc0402b18ae height: 6904661
2025-04-14T11:22:05.490 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 3.45s, pre_validation: 0.44s, CLVM: 0.44s, post-process: 1.56s, cost: 4075237853, percent full: 37.048% header_hash: 1bdbf7c0a52507a367ac2e9c1b12b80703d6c40582f170004f4ada3d2baeebe4 height: 6904673
2025-04-14T11:25:58.776 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.84s, pre_validation: 0.38s, CLVM: 0.36s, post-process: 1.02s, cost: 3803109693, percent full: 34.574% header_hash: 1b6d63d06c0fb2c14b06763094fd6f693af9ecd8e7d528ef7b70931bd83f4b88 height: 6904681
2025-04-14T11:30:15.025 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.67s, pre_validation: 0.27s, CLVM: 0.27s, post-process: 0.95s, cost: 3728379183, percent full: 33.894% header_hash: ffbbf871dc4897b4b470943faf1c558e7f8bcabb256220f27d6feb66b6560e72 height: 6904699
2025-04-14T11:34:16.769 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.83s, pre_validation: 0.22s, CLVM: 0.22s, post-process: 1.14s, cost: 3843671742, percent full: 34.942% header_hash: 78f928932970e8c62025706394bbdc83691977c8bb6a740d5a71446b55f01bd2 height: 6904712
2025-04-14T11:39:21.986 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.84s, pre_validation: 0.20s, CLVM: 0.20s, post-process: 1.09s, cost: 3860122460, percent full: 35.092% header_hash: 571ddf27c85a7cbe77b761cba538c498deb923033947d63b29ae40d55f376e42 height: 6904724
2025-04-14T11:43:15.958 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.83s, pre_validation: 0.26s, CLVM: 0.26s, post-process: 1.11s, cost: 3858801285, percent full: 35.08% header_hash: f440738215dcaeccb90312eaa314242aa9e5420796816f464c794d2fea2b6605 height: 6904738
2025-04-14T11:51:37.359 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 3.06s, pre_validation: 0.61s, CLVM: 0.61s, post-process: 0.97s, cost: 3888293261, percent full: 35.348% header_hash: 999c5906d583770a9ccae87139c2774372701874a4e474561810c1d86568c762 height: 6904763
2025-04-14T11:58:09.292 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.98s, pre_validation: 0.52s, CLVM: 0.52s, post-process: 0.98s, cost: 3852218878, percent full: 35.02% header_hash: 0eca1f90e66c63f9a8b36d7d206aa834d196a7b50400b1e8739cabea60fbb911 height: 6904782
2025-04-14T12:02:53.230 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.62s, pre_validation: 0.19s, CLVM: 0.19s, post-process: 0.97s, cost: 3759961006, percent full: 34.181% header_hash: 7b34d842bc916ba65f98e7d4150aa4e73bc43bcdfbb7ef605ec92eb5223c5256 height: 6904797
2025-04-14T12:50:36.939 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.62s, pre_validation: 0.12s, CLVM: 0.12s, post-process: 1.00s, cost: 3762601371, percent full: 34.205% header_hash: fc302ee15c0a7e1aa371ba691465ca1ce19faffbf65c4b5315d496e372b687c8 height: 6904959
2025-04-14T13:02:07.494 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 3.25s, pre_validation: 0.39s, CLVM: 0.39s, post-process: 1.38s, cost: 3881799585, percent full: 35.289% header_hash: 1967f1b01d985ca1deefdae190a4622a5b2af636b20fc02175f2cc995e3a2a0b height: 6904998
2025-04-14T13:18:57.763 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 3.33s, pre_validation: 0.59s, CLVM: 0.59s, post-process: 1.19s, cost: 3900142350, percent full: 35.456% header_hash: 1e7424522d45f930c27c4a357f236e1715ccc09f5c13eed6a95de7c679dae137 height: 6905041
2025-04-14T13:21:58.453 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.84s, pre_validation: 0.20s, CLVM: 0.20s, post-process: 1.17s, cost: 3840204690, percent full: 34.911% header_hash: 4c8809db93adf53cdb3d00fb113cb3915e9a933b41b61b1072774b6e45730963 height: 6905056
2025-04-14T13:30:22.050 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 3.03s, pre_validation: 0.31s, CLVM: 0.31s, post-process: 1.19s, cost: 3818508589, percent full: 34.714% header_hash: 14d41bd2e437f86d8b77ad383477685f0244d553bdd4357fac775c60ce165d57 height: 6905084
2025-04-14T13:31:25.493 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 3.06s, pre_validation: 0.42s, CLVM: 0.42s, post-process: 1.19s, cost: 3833145941, percent full: 34.847% header_hash: ecb5ef5a14d055085963145debe973e8ac4a9dae484d506866377e49959f5fd4 height: 6905085
2025-04-14T13:32:55.367 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.86s, pre_validation: 0.55s, CLVM: 0.55s, post-process: 0.92s, cost: 3755208764, percent full: 34.138% header_hash: 3c2447001bf462a2f7fe980864a6a565ac1e202131b9bfa4d3f8daf1946ebc73 height: 6905086
2025-04-14T13:33:54.598 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 3.02s, pre_validation: 0.36s, CLVM: 0.36s, post-process: 1.11s, cost: 4429917832, percent full: 40.272% header_hash: 3e8e7375d0de70f329b1addff86488f0fbda4da8bcd1de50da91d1c2c8120b47 height: 6905088
2025-04-14T13:44:15.477 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 3.27s, pre_validation: 0.27s, CLVM: 0.25s, post-process: 1.14s, cost: 3813582072, percent full: 34.669% header_hash: b347070d78adc0c78ee044d0bdf6b34b07c71bc43aeb80f1d0bb0a7feb7e8192 height: 6905117
2025-04-14T13:46:01.651 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 5.67s, pre_validation: 0.20s, CLVM: 0.19s, post-process: 2.17s, cost: 7618466819, percent full: 69.259% header_hash: c1a5de7622ab0b4c09923fa63a536a0e9cdeb275d871245b11ae1c52df3212e7 height: 6905121
2025-04-14T14:23:18.527 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.70s, pre_validation: 0.19s, CLVM: 0.19s, post-process: 1.05s, cost: 3783787544, percent full: 34.398% header_hash: 80c309457333b60c4af5cc4986f9623e041ffc2bf32c00520151c1a9381ba8df height: 6905249
2025-04-14T14:24:02.884 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.84s, pre_validation: 0.36s, CLVM: 0.36s, post-process: 1.09s, cost: 3721313551, percent full: 33.83% header_hash: 7fd576e6f89cddd667a6a713eefdc0bc17855cb4a3663c46aa84f488a7f7d612 height: 6905250
2025-04-14T15:15:48.825 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.47s, pre_validation: 0.17s, CLVM: 0.17s, post-process: 0.89s, cost: 3844934387, percent full: 34.954% header_hash: 87c0beebd89531de0d371e2769153bfd8a074de69123237bbd8d712cd893ab45 height: 6905408
2025-04-14T15:17:00.539 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.77s, pre_validation: 0.34s, CLVM: 0.34s, post-process: 1.02s, cost: 3934506115, percent full: 35.768% header_hash: f8171708e2d7c76bdf333b25f940a3a4624d78fdc6eb2f446d61e43bb9961de6 height: 6905411
2025-04-14T15:23:11.238 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 3.05s, pre_validation: 0.28s, CLVM: 0.28s, post-process: 0.89s, cost: 3848253835, percent full: 34.984% header_hash: 0f71dea6eeda5b606950b70ecc2cec51108c6b8f1528a07ded2836440ad8a60a height: 6905428
2025-04-14T15:25:15.891 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.67s, pre_validation: 0.22s, CLVM: 0.22s, post-process: 1.05s, cost: 3804881329, percent full: 34.59% header_hash: f57d19ce911ec540672fdfd7fc0e74d4664a826656e1c45cf296585ff6d00739 height: 6905438
2025-04-14T15:25:55.947 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.53s, pre_validation: 0.09s, CLVM: 0.09s, post-process: 1.03s, cost: 3804671504, percent full: 34.588% header_hash: d725eb1d3e925a82d102a1d65cee400ccb3cb71891eeeb3329f85412b06ea8d3 height: 6905442
2025-04-14T15:26:49.841 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 5.12s, pre_validation: 0.41s, CLVM: 0.41s, post-process: 1.91s, cost: 7417715084, percent full: 67.434% header_hash: 30ce407a76291caa4368df4a2a4e4797f85e50987973c3ed0c333021de7367d5 height: 6905446
2025-04-14T15:29:24.984 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.64s, pre_validation: 0.22s, CLVM: 0.22s, post-process: 1.02s, cost: 3865576892, percent full: 35.142% header_hash: 4171406cfd51a014398d56db15b9e2e845a327007a86ea7ddc387d78a8ed5df8 height: 6905459
2025-04-14T15:30:13.304 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.89s, pre_validation: 0.27s, CLVM: 0.27s, post-process: 1.25s, cost: 3844536060, percent full: 34.95% header_hash: f4f2d39f65d29c601c6266f961d48ba39cc6928bbad9be9b696a64c30ac7f2ea height: 6905462
2025-04-14T16:25:54.729 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.73s, pre_validation: 0.38s, CLVM: 0.38s, post-process: 0.86s, cost: 3873661423, percent full: 35.215% header_hash: 09f0df7eec6761079ce65b92a96789b80ab93d56545a638c4d571462cadc00de height: 6905662
2025-04-14T16:26:45.402 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.69s, pre_validation: 0.23s, CLVM: 0.23s, post-process: 1.03s, cost: 3841218140, percent full: 34.92% header_hash: 544cf13087f68b227a19e7c707ee081edf3dac21817dd58e741999263452c5ed height: 6905664
2025-04-14T16:27:33.478 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.73s, pre_validation: 0.30s, CLVM: 0.30s, post-process: 1.02s, cost: 3799795588, percent full: 34.544% header_hash: 31248f8e249723d82a7d722598d29468005e28b74ffe450df94b331dd76ae08e height: 6905666
2025-04-14T16:40:23.378 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.72s, pre_validation: 0.25s, CLVM: 0.25s, post-process: 1.05s, cost: 3732557446, percent full: 33.932% header_hash: 53b9cebf89e34e61a8f46fe759aa306dae9da78ca8e47777c5df761b5fc4c910 height: 6905708
2025-04-14T16:40:56.768 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.61s, pre_validation: 0.28s, CLVM: 0.28s, post-process: 0.95s, cost: 3812164526, percent full: 34.656% header_hash: 2ce580e74ce492923beb969af7248077466875530f04542affdaeb1e00b3c8e9 height: 6905709
2025-04-14T16:42:19.160 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.61s, pre_validation: 0.34s, CLVM: 0.34s, post-process: 0.86s, cost: 3785235516, percent full: 34.411% header_hash: 46ce1f449456beea3f8120ca18b0cc377731bb3d24a3728da934c8204dae7f13 height: 6905713
2025-04-14T16:46:46.206 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.69s, pre_validation: 0.20s, CLVM: 0.20s, post-process: 1.08s, cost: 3814197680, percent full: 34.675% header_hash: c6cc1d2677daf395b78d009f9858d0c575a1f64ec75f829ce62f8d6b1eae1966 height: 6905728
2025-04-14T17:06:10.583 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.91s, pre_validation: 0.11s, CLVM: 0.09s, post-process: 1.36s, cost: 3706373765, percent full: 33.694% header_hash: 89353ec76a845c62b936578f692270180513cfde2af9764fe66eefd07c5932b7 height: 6905792
2025-04-14T17:06:40.351 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.45s, pre_validation: 0.20s, CLVM: 0.20s, post-process: 0.86s, cost: 3774332467, percent full: 34.312% header_hash: 2df980ac8214a079d2b2817dc6dce8866669609cf2cd885b9dd5f434712592a8 height: 6905794
2025-04-14T17:08:15.946 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.64s, pre_validation: 0.22s, CLVM: 0.22s, post-process: 1.03s, cost: 3778256004, percent full: 34.348% header_hash: c1c1d3026a8ba9d928ceec45d4642bcf49c37dd80619818a6c6dbf3f1303b1c8 height: 6905798
2025-04-14T17:11:51.597 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.58s, pre_validation: 0.19s, CLVM: 0.19s, post-process: 1.00s, cost: 3774355141, percent full: 34.312% header_hash: 7bc24abed69ab6758eb831b52039b0c541c664d58a8386673733c617309b9f5f height: 6905815
2025-04-14T17:18:43.450 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.92s, pre_validation: 0.34s, CLVM: 0.34s, post-process: 0.88s, cost: 4711252267, percent full: 42.83% header_hash: d9f9ceb01af10c5a00ed0a1b37b10484fa91cf054348fa142926050d0ed03a25 height: 6905835
2025-04-14T17:20:19.272 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.58s, pre_validation: 0.30s, CLVM: 0.30s, post-process: 0.84s, cost: 3855036088, percent full: 35.046% header_hash: ab7b123387b261d5c56f4c3db1d894c1ceda6edffe2579018e56015a8054df86 height: 6905844
2025-04-14T17:22:45.419 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.75s, pre_validation: 0.34s, CLVM: 0.34s, post-process: 0.83s, cost: 3770147641, percent full: 34.274% header_hash: ec4ad83bac45430593ee2df79046dec11696869acd538183aaf3d5e0b7ea71a6 height: 6905847
2025-04-14T17:24:26.941 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.58s, pre_validation: 0.39s, CLVM: 0.39s, post-process: 0.84s, cost: 3721373559, percent full: 33.831% header_hash: 78a04d0cdc84a6d1ce9263463ca27c1e4ac6f239f21226ba07cfcfd851efb8ca height: 6905850
2025-04-14T17:25:09.984 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.36s, pre_validation: 0.20s, CLVM: 0.20s, post-process: 0.80s, cost: 3815920660, percent full: 34.69% header_hash: c959b71f3195a25058a666c4cc8ab49d67db51cde24632eb42975eaf0748f8ad height: 6905852
2025-04-14T17:26:11.470 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.31s, pre_validation: 0.17s, CLVM: 0.17s, post-process: 0.83s, cost: 3785583516, percent full: 34.414% header_hash: 95c67bc0394f9dbaa4007cda5e62c966007f6a589537c99e204e30f7a1351d27 height: 6905853
2025-04-14T17:28:45.795 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.36s, pre_validation: 0.16s, CLVM: 0.16s, post-process: 0.81s, cost: 3724684620, percent full: 33.861% header_hash: b9f046a1cfd6abfe6b245f3153c1ce93e0228d8b34c496910c35ff56df21a62b height: 6905865
2025-04-15T02:07:35.125 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 3.39s, pre_validation: 0.47s, CLVM: 0.47s, post-process: 1.08s, cost: 6176797409, percent full: 56.153% header_hash: 60edb7420a4371efaeb5a256fc9d87d293120671c7e7f2595a17e5ab5a707c05 height: 6907574
2025-04-15T02:09:09.193 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.28s, pre_validation: 0.25s, CLVM: 0.25s, post-process: 0.80s, cost: 4000214755, percent full: 36.366% header_hash: 2f39004151dde9714483031500956ee0d122e916a615201ab63de2583455a47a height: 6907579
2025-04-15T02:11:08.720 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.14s, pre_validation: 0.62s, CLVM: 0.62s, post-process: 0.59s, cost: 3165075332, percent full: 28.773% header_hash: 928f5fc3f5b6a6b7fd7df560d63c859fe7798946e2a1d8850e8d643786287726 height: 6907582
2025-04-15T07:30:53.906 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.81s, pre_validation: 0.31s, CLVM: 0.31s, post-process: 1.03s, cost: 3811564171, percent full: 34.651% header_hash: 32d44e8f14712cbcfb0bb1d5f618f39f64b95d22f7e7aeffcae7585f4692ee89 height: 6908612
2025-04-15T07:31:32.795 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.59s, pre_validation: 0.19s, CLVM: 0.19s, post-process: 0.98s, cost: 3818484647, percent full: 34.713% header_hash: 5fbf88f6817971f342a0ca51683a3eb39286a77f4a0efaf0ffa03504d7249915 height: 6908614
2025-04-15T07:32:53.676 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.52s, pre_validation: 0.16s, CLVM: 0.15s, post-process: 0.98s, cost: 3870204612, percent full: 35.184% header_hash: 084873ec6c18157c37f5e72028d43dafa81b50c56059d4a9dd8cb4f7532eb05e height: 6908618
2025-04-15T07:34:52.750 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.61s, pre_validation: 0.33s, CLVM: 0.33s, post-process: 0.86s, cost: 3764422666, percent full: 34.222% header_hash: f32eff231cc88e9de13229004ba910338938851e6d18cd092ce010b789e1672c height: 6908622
2025-04-15T07:39:14.878 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 5.02s, pre_validation: 0.22s, CLVM: 0.22s, post-process: 1.88s, cost: 7704574864, percent full: 70.042% header_hash: 1f5ed8c6a71001aeb8c2f30e142f72413eae89918ddbfb65c4cc256f2a1470df height: 6908633
2025-04-15T07:40:11.970 2.5.3 full_node chia.full_node.full_node_api: ERROR    got weight proof request for unknown peak 314facde80401571348cc9b7792f11cecae9fcffa65b6aaa00cd74bf8a519b53
2025-04-15T08:08:20.595 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 5.12s, pre_validation: 0.38s, CLVM: 0.38s, post-process: 1.89s, cost: 7502973740, percent full: 68.209% header_hash: b5c6008cd7190ea6457dd4397eba41cc7d8c8aa1712d1c73f0ff986dfeeec148 height: 6908730
2025-04-15T08:21:05.456 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 3.34s, pre_validation: 0.61s, CLVM: 0.61s, post-process: 0.84s, cost: 3965928288, percent full: 36.054% header_hash: 8cccff37f9d07ee92b196526ae41e2af9e0b06dc0ec7036e6203610035c33f75 height: 6908760
2025-04-15T08:22:25.547 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.55s, pre_validation: 0.30s, CLVM: 0.30s, post-process: 0.84s, cost: 3721337553, percent full: 33.83% header_hash: e89097b1e88ea57c5654090db5c493143a9953d9362566eeb1cd911d7e8b5d77 height: 6908764
2025-04-15T08:23:45.339 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.36s, pre_validation: 0.14s, CLVM: 0.14s, post-process: 0.83s, cost: 3717263504, percent full: 33.793% header_hash: f7794c2d9f5868e3523393f07d06849f523b1f01f9dadc9129d43980b92f93dc height: 6908770
2025-04-15T08:25:10.333 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 5.17s, pre_validation: 0.36s, CLVM: 0.36s, post-process: 2.02s, cost: 7482476953, percent full: 68.023% header_hash: 5a27fbd1745fd1eab08d58963800ba787600f76b5a338acb9c015a528b724328 height: 6908773
2025-04-15T08:27:30.443 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 5.53s, pre_validation: 0.39s, CLVM: 0.39s, post-process: 1.95s, cost: 7433616458, percent full: 67.578% header_hash: 16e7f1f114e2b56a178b3aacaedc5bc3dcba09782bc7899e5a41215963b5c853 height: 6908777
2025-04-15T08:31:42.137 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.67s, pre_validation: 0.27s, CLVM: 0.27s, post-process: 1.03s, cost: 3799365622, percent full: 34.54% header_hash: 77b0d8dca3831d25767d6d9c91ae9b16722c97e6615829b73f64f86b52838af1 height: 6908787
2025-04-15T08:33:45.215 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 5.14s, pre_validation: 0.38s, CLVM: 0.38s, post-process: 1.94s, cost: 7502752168, percent full: 68.207% header_hash: 85801d4a554aa8754613fe8726dba849dc2b54ec0b22e9d5f69dafc48d678d1d height: 6908796
2025-04-15T08:37:27.468 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.53s, pre_validation: 0.22s, CLVM: 0.22s, post-process: 0.83s, cost: 3770433300, percent full: 34.277% header_hash: f86f7234fc9611cdb48d34ec898049e1b801f15dcb5ab2977304ffda0166f386 height: 6908808
2025-04-15T08:40:20.932 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.67s, pre_validation: 0.11s, CLVM: 0.11s, post-process: 0.92s, cost: 3732480895, percent full: 33.932% header_hash: 8ab016ff3ba9e228ba6be31dd996aff757e9f53043f04628f7895b179df293f4 height: 6908816
2025-04-15T08:42:13.440 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.03s, pre_validation: 0.31s, CLVM: 0.31s, post-process: 0.59s, cost: 3781379839, percent full: 34.376% header_hash: 5293d925cbbb49ec456b712d5b1e1df3ade6932650f8978d21f4be38956c87ad height: 6908820
2025-04-15T08:45:25.152 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 3.56s, pre_validation: 0.33s, CLVM: 0.33s, post-process: 1.58s, cost: 7421518153, percent full: 67.468% header_hash: 3ec84694f35864c06bc40518bf7c218cb4cbb3a050d2c62724a170dca65ecfec height: 6908826
2025-04-15T08:46:50.096 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 3.12s, pre_validation: 0.56s, CLVM: 0.56s, post-process: 1.11s, cost: 6933687542, percent full: 63.034% header_hash: b43d2942b577eacf5d6afff9adc8c2e472cf95226c79d7815367af2890dc46e4 height: 6908827
2025-04-15T09:04:59.744 2.5.3 full_node chia.full_node.mempool_manager: WARNING  updating the mempool using the slow-path. peak: 963b401b7055b303558175b603b497856e48042402cf3b59db20e00ace961c31 new-peak-prev: 9d443c0a791b710b66f75a6bab6d87a2ab42e4f48ab029cc742f8639e996817b coins: set
2025-04-15T09:08:43.314 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.08s, pre_validation: 0.61s, CLVM: 0.61s, post-process: 0.66s, cost: 3733488742, percent full: 33.941% header_hash: bffadf63170248423e7c129855480b0a94537a1740b2569f7f887ab366667a85 height: 6908890
2025-04-15T09:53:59.920 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.36s, pre_validation: 0.31s, CLVM: 0.31s, post-process: 1.25s, cost: 3684988096, percent full: 33.5% header_hash: 68f06c8a465689a36c2973b0ce8b722ffcb3398470b99e7021b6f789bab7b326 height: 6909014
2025-04-15T10:36:35.759 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 3.09s, pre_validation: 0.25s, CLVM: 0.25s, post-process: 1.17s, cost: 7443083750, percent full: 67.664% header_hash: 23779acc804e8c25aad25f475d31b43ddea7e69f8fdcfeaf74030735a21a3abd height: 6909163
2025-04-15T11:27:38.564 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.06s, pre_validation: 0.33s, CLVM: 0.31s, post-process: 0.56s, cost: 3690580622, percent full: 33.551% header_hash: 87380b5ff8058cf12d63ef0f2ae9aa7266852a022dc7161b321ec66c0992af50 height: 6909290
2025-04-15T11:27:44.833 2.5.3 full_node full_node_server        : WARNING  Connection failed to wss://89.58.33.71:8444/ws. Error: 403, message='Invalid response status', url='wss://89.58.33.71:8444/ws'
2025-04-15T11:52:17.013 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.05s, pre_validation: 0.62s, CLVM: 0.61s, post-process: 0.62s, cost: 3923532857, percent full: 35.668% header_hash: b315cd1db310fce9f2e1085deb7924503de87047aa2cbf0a01e6d8f8ab23cc34 height: 6909360
2025-04-15T11:58:44.502 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.03s, pre_validation: 0.62s, CLVM: 0.62s, post-process: 0.59s, cost: 3853999841, percent full: 35.036% header_hash: 82e3deae74488b2a1bc6fbb1847a0ab549ba6a8e130549e6cfb8116ff9091760 height: 6909373
2025-04-15T11:59:49.883 2.5.3 full_node chia.full_node.mempool_manager: WARNING  updating the mempool using the slow-path. peak: c3505eb347ccc748d99d403815e6d4bf344e30eb3ebdec1ca697c2561ccc10de new-peak-prev: 82e3deae74488b2a1bc6fbb1847a0ab549ba6a8e130549e6cfb8116ff9091760 coins: set
2025-04-15T12:00:43.860 2.5.3 full_node chia.full_node.mempool_manager: WARNING  updating the mempool using the slow-path. peak: 40392afc9cc2d0dd077036ebfcfc45adacde1fae9902b1982f436c87cd2bcbcb new-peak-prev: c3505eb347ccc748d99d403815e6d4bf344e30eb3ebdec1ca697c2561ccc10de coins: set
2025-04-15T12:52:45.009 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.09s, pre_validation: 0.25s, CLVM: 0.25s, post-process: 0.62s, cost: 4069416092, percent full: 36.995% header_hash: 092152b946b87754687922dc95b7dae2f1d31b7aa8b667b34a7cd1f578cc3e34 height: 6909538
2025-04-15T12:58:35.676 2.5.3 full_node chia.full_node.mempool_manager: WARNING  updating the mempool using the slow-path. peak: c64543117bcc6bd50c55879a331e417b92b9e2ce5654993fbacea09c69a364ac new-peak-prev: f1c63839fa442bf37b353d11fc2e4318d1222f99424acff278d3177d5ada7708 coins: set
2025-04-15T13:17:06.571 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 3.51s, pre_validation: 0.48s, CLVM: 0.48s, post-process: 1.42s, cost: 7426667631, percent full: 67.515% header_hash: 82f9aab475a7373d6cd7fd858ef920ad95d938ad9dac28a2104aa1d16f9d0ffc height: 6909613
2025-04-15T13:34:49.214 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 3.05s, pre_validation: 0.12s, CLVM: 0.12s, post-process: 1.31s, cost: 7380415668, percent full: 67.095% header_hash: eb03ca5f8e0e9fa6554bbf65770885e6c86c89ef3e5b9262c7c5dfa158425c96 height: 6909665
2025-04-15T13:39:19.318 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 3.28s, pre_validation: 0.34s, CLVM: 0.33s, post-process: 1.34s, cost: 7362038201, percent full: 66.928% header_hash: ecc505f238654381eff4b12f6b38fbe9e89226945d72f5c51d4e3140fac3d5a6 height: 6909681
2025-04-15T14:00:33.164 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.08s, pre_validation: 0.22s, CLVM: 0.22s, post-process: 0.81s, cost: 4355061720, percent full: 39.591% header_hash: 783a2d9a445b4e261416af7736670bcd07e79af83919eab95291f73e1ff55462 height: 6909748
2025-04-15T15:12:54.735 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.45s, pre_validation: 0.22s, CLVM: 0.22s, post-process: 1.16s, cost: 5019009825, percent full: 45.627% header_hash: 4ed40108041172c7f03813dc2fe477197796a602a1811ad04461364f233239d3 height: 6910020
2025-04-15T15:44:05.638 2.5.3 full_node chia.full_node.mempool  : WARNING  serializing block generator took 1.88 seconds
2025-04-15T16:02:42.528 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.06s, pre_validation: 0.30s, CLVM: 0.30s, post-process: 0.59s, cost: 3744695894, percent full: 34.043% header_hash: 953d32f83ff7ae4125c71abfbf8d4e09ccd5ae04a5e5cef1335e4826739e47d6 height: 6910169
2025-04-15T16:30:02.022 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.11s, pre_validation: 0.56s, CLVM: 0.56s, post-process: 0.73s, cost: 3945987873, percent full: 35.873% header_hash: 3ec50d09eb7493e4c68aae798bd5e74ac00db3c01df822793f80be643d631537 height: 6910242
2025-04-15T17:00:36.977 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.05s, pre_validation: 0.48s, CLVM: 0.48s, post-process: 0.78s, cost: 3755984139, percent full: 34.145% header_hash: d19fabdbc28c897a8f82ac1b03552fa98cdacc3019aea7763149102daefe83e3 height: 6910331
2025-04-15T20:28:19.781 2.5.3 full_node chia.full_node.mempool_manager: WARNING  updating the mempool using the slow-path. peak: c0cc9ee779763e13b89ae9095c9efd41fdd3aa1ddaa7c3d3d766262c62c80e50 new-peak-prev: 05ef65ae21e0e7e0e1c2e8d7d95f3249f5ec986fc075300c1398e048c63cea7c coins: set
2025-04-16T07:49:51.970 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.23s, pre_validation: 0.22s, CLVM: 0.22s, post-process: 0.86s, cost: 3772027043, percent full: 34.291% header_hash: fb8927858250a2b31baf535e53c6b79a2a0c39beda3c241a71e6a88cd10ef28a height: 6913126
2025-04-16T08:00:09.165 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.08s, pre_validation: 0.33s, CLVM: 0.31s, post-process: 0.62s, cost: 3870567797, percent full: 35.187% header_hash: 6bf1f1c2bd1f870f44de2eb1a480f0c54dd54e9c707de7db551a0fb90c864bff height: 6913161
2025-04-16T14:45:22.261 2.5.3 full_node chia.full_node.mempool_manager: WARNING  updating the mempool using the slow-path. peak: f44cc695031100ff056723caa3ac90e32175ae817161ae3dbf420ca20e8adc0c new-peak-prev: fb9d9ec7a3330f200964252b0e9d87b9cdb63588da698dfd2ad8bebf40789325 coins: set
2025-04-16T17:52:45.323 2.5.3 full_node chia.full_node.mempool_manager: WARNING  updating the mempool using the slow-path. peak: 78a7fe3e6ec27d66283889ec5a6b1a411bdb40b702c72814b81e2da1ebbe932a new-peak-prev: 0eeef69c965a93782654502409501186ae10fd8f1628db52eeded2037a5547e6 coins: set
2025-04-16T18:56:48.928 2.5.3 full_node chia.full_node.mempool_manager: WARNING  updating the mempool using the slow-path. peak: 2bc5da0a9def117255c7340166c3c0d5876908fe6a2011bbc48105a8e1720b93 new-peak-prev: fdb19810a25edf2c2407e7dd18195d4cd801869275b41d8a043fe7014b0f06ae coins: set
2025-04-16T19:15:30.447 2.5.3 full_node chia.full_node.mempool_manager: WARNING  updating the mempool using the slow-path. peak: 09b84f0a6861bcc0672af50521fe53a0a6094cd4a52fc01ea6ad0487a0df542e new-peak-prev: 8b921bdd0c43178a49a8b0256899714ecf850c52c8f4c45874de3f68daffd819 coins: set
2025-04-17T03:36:40.948 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 3.48s, pre_validation: 0.30s, CLVM: 0.30s, post-process: 1.19s, cost: 3650527035, percent full: 33.187% header_hash: 91c258c442991ad402a9882b449615bf57ff1cd6f10e0f972ca31fa717c18ed4 height: 6916972
2025-04-17T03:42:17.622 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 3.56s, pre_validation: 0.36s, CLVM: 0.36s, post-process: 1.22s, cost: 3694096859, percent full: 33.583% header_hash: 14481ce6cfa1bd00908a95f01f4e53f6712a41ecfbb5a0e50f25cc0b51a6fec0 height: 6916988
2025-04-17T03:42:46.140 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.48s, pre_validation: 0.17s, CLVM: 0.16s, post-process: 0.89s, cost: 2906719865, percent full: 26.425% header_hash: e0ae916b694ffeff2f59286f2cd523382ab9339c57abc40ce57abbccb9df5d0f height: 6916991
2025-04-17T17:37:33.453 2.5.3 full_node chia.full_node.mempool_manager: WARNING  pre_validate_spendbundle took 2.0215 seconds for 831eb476ce43ed5a3c4912c9d7783343625cba1f5ca295ffc21f0f74fbb8f437 (queue-size: 1)
2025-04-17T17:38:26.642 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 3 removed transactions to 0 peers took 2.7180s
2025-04-17T17:40:42.919 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 7 removed transactions to 0 peers took 2.7660s
2025-04-17T17:41:35.768 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 4 removed transactions to 0 peers took 1.6090s
2025-04-17T17:43:02.167 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 7 removed transactions to 0 peers took 1.6250s
2025-04-17T17:45:11.319 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 2 removed transactions to 0 peers took 1.7340s
2025-04-17T18:04:51.963 2.5.3 full_node chia.full_node.mempool_manager: WARNING  pre_validate_spendbundle took 2.1765 seconds for e308aa8915d7032a67ea17987f07f260ef64bb1e3e02056baaa7ae0e61a30e81 (queue-size: 0)
2025-04-17T18:06:43.051 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 5 removed transactions to 0 peers took 2.8440s
2025-04-17T18:11:14.901 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 6 removed transactions to 0 peers took 1.8590s
2025-04-17T18:12:47.266 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 11 removed transactions to 0 peers took 1.5310s
2025-04-17T18:13:32.936 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 7 removed transactions to 0 peers took 1.2970s
2025-04-17T18:13:35.078 2.5.3 full_node chia.full_node.mempool_manager: WARNING  pre_validate_spendbundle took 2.0423 seconds for 366d2c438435a61424ed35c80925cc80cf224bea4156171cc04f97f9975106df (queue-size: 0)
2025-04-17T18:14:15.997 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 5 removed transactions to 0 peers took 1.3750s
2025-04-17T18:14:57.267 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 5 removed transactions to 0 peers took 2.7180s
2025-04-17T18:16:00.650 2.5.3 full_node chia.full_node.mempool_manager: WARNING  pre_validate_spendbundle took 2.0504 seconds for bf4b3724cd35dd0ad94d2642eae46de162fc23e5213677058bb6e65dd50ac8c5 (queue-size: 0)
2025-04-17T18:16:09.899 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 5 removed transactions to 0 peers took 2.8430s
2025-04-17T18:16:44.956 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 1 removed transactions to 0 peers took 1.3910s
2025-04-17T18:18:05.029 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 3 removed transactions to 0 peers took 2.7660s
2025-04-17T18:18:41.144 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 1 removed transactions to 0 peers took 1.4370s
2025-04-17T18:19:47.986 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 3 removed transactions to 0 peers took 2.6250s
2025-04-17T18:21:59.606 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 3 removed transactions to 0 peers took 1.3280s
2025-04-17T18:22:46.947 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 4 removed transactions to 0 peers took 3.1410s
2025-04-17T18:23:33.715 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 2 removed transactions to 0 peers took 1.3750s
2025-04-17T18:25:41.468 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 4 removed transactions to 0 peers took 3.3280s
2025-04-17T18:28:24.811 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 8 removed transactions to 0 peers took 1.5780s
2025-04-17T18:29:18.027 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 5 removed transactions to 0 peers took 2.6410s
2025-04-17T18:29:28.817 2.5.3 full_node chia.full_node.mempool_manager: WARNING  pre_validate_spendbundle took 2.0834 seconds for e57204999a9c5e7bf5d59abba8b7033401e81f05ee128eec63c1d4d14a99d62b (queue-size: 0)
2025-04-17T18:31:16.227 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 6 removed transactions to 0 peers took 2.9370s
2025-04-17T18:33:47.211 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 4 removed transactions to 0 peers took 3.2350s
2025-04-17T18:35:26.067 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 8 removed transactions to 0 peers took 1.3900s
2025-04-17T18:36:07.224 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 2 removed transactions to 0 peers took 1.6560s
2025-04-17T18:38:15.568 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 10 removed transactions to 0 peers took 3.0940s
2025-04-17T18:38:51.085 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 2 removed transactions to 0 peers took 1.4220s
2025-04-17T18:42:46.953 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 5 removed transactions to 0 peers took 1.2500s
2025-04-17T18:45:17.466 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 3 removed transactions to 0 peers took 1.8430s
2025-04-17T18:46:01.385 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 7 removed transactions to 0 peers took 3.3900s
2025-04-17T18:46:05.043 2.5.3 full_node chia.full_node.mempool_manager: WARNING  pre_validate_spendbundle took 2.0048 seconds for 8da7bf808d39336a74e7389699b15eb2872068003256edc1ce38417dbec750d0 (queue-size: 1)
2025-04-17T18:47:23.523 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 2 removed transactions to 0 peers took 1.7500s
2025-04-17T18:49:26.380 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 5 removed transactions to 0 peers took 1.3440s
2025-04-17T18:52:05.794 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 5 removed transactions to 0 peers took 2.9850s
2025-04-17T18:53:32.428 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 3 removed transactions to 0 peers took 1.6410s
2025-04-17T18:55:45.147 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 5 removed transactions to 0 peers took 2.9680s
2025-04-17T18:57:32.844 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 10 removed transactions to 0 peers took 1.5150s
2025-04-17T19:01:04.599 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 5 removed transactions to 0 peers took 1.5930s
2025-04-17T19:01:36.907 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 1 removed transactions to 0 peers took 1.6410s
2025-04-17T19:01:44.791 2.5.3 full_node chia.full_node.mempool_manager: WARNING  pre_validate_spendbundle took 2.0779 seconds for e204900d6563e2a3290801323112b00884e565585dd3a0ba142652c4dd491e89 (queue-size: 1)
2025-04-17T19:01:45.979 2.5.3 full_node chia.full_node.mempool_manager: WARNING  pre_validate_spendbundle took 2.0644 seconds for a630ae4409bb769ffa6fd09fe518c52d6e5e8677d7b41672e44f67e009443155 (queue-size: 0)
2025-04-17T19:04:46.867 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 5 removed transactions to 0 peers took 1.6090s
2025-04-17T19:07:23.295 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 4 removed transactions to 0 peers took 1.6720s
2025-04-17T19:12:11.517 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 5 removed transactions to 0 peers took 1.8280s
2025-04-17T19:15:24.590 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 3 removed transactions to 0 peers took 1.7340s
2025-04-17T19:19:41.384 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 4 removed transactions to 0 peers took 1.7180s
2025-04-17T19:22:50.125 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 5 removed transactions to 0 peers took 1.7820s
2025-04-17T19:23:37.793 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 4 removed transactions to 0 peers took 1.7350s
2025-04-17T19:33:27.565 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 5 removed transactions to 0 peers took 1.7500s
2025-04-17T19:40:52.864 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 2 removed transactions to 0 peers took 1.7500s
2025-04-17T19:43:40.362 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 7 removed transactions to 0 peers took 1.2340s
2025-04-17T19:45:32.941 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 2 removed transactions to 0 peers took 1.3440s
2025-04-17T19:46:30.079 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 6 removed transactions to 0 peers took 2.7660s
2025-04-17T19:48:53.593 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 2 removed transactions to 0 peers took 3.0620s
2025-04-17T19:50:52.587 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 4 removed transactions to 0 peers took 2.7960s
2025-04-17T19:54:14.500 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 2 removed transactions to 0 peers took 3.1100s
2025-04-17T19:54:59.615 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 5 removed transactions to 0 peers took 2.7660s
2025-04-17T19:55:03.181 2.5.3 full_node chia.full_node.mempool_manager: WARNING  pre_validate_spendbundle took 2.7052 seconds for 724feca1200810cc27da1655c649183a6830f92c26aa28757641e6a924153a6c (queue-size: 0)
2025-04-17T19:56:30.080 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 3 removed transactions to 0 peers took 1.2810s
2025-04-17T19:59:56.450 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 3 removed transactions to 0 peers took 1.2660s
2025-04-17T20:00:49.650 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 4 removed transactions to 0 peers took 3.0000s
2025-04-17T20:03:43.479 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 4 removed transactions to 0 peers took 2.9370s
2025-04-17T20:04:42.374 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 5 removed transactions to 0 peers took 2.6720s
2025-04-17T20:06:21.661 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 8 removed transactions to 0 peers took 2.5780s
2025-04-17T20:06:23.742 2.5.3 full_node chia.full_node.mempool_manager: WARNING  pre_validate_spendbundle took 2.0348 seconds for 828f8bb903f9a49a5626a555eaf246eb861d5552a51785468ccbfaa5f48e1c93 (queue-size: 0)
2025-04-17T20:07:21.019 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 2 removed transactions to 0 peers took 1.2500s
2025-04-17T20:13:23.320 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 5 removed transactions to 0 peers took 1.7030s
2025-04-17T20:15:13.933 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 2 removed transactions to 0 peers took 2.5470s
2025-04-17T20:16:25.944 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 3 removed transactions to 0 peers took 1.5780s
2025-04-17T20:17:04.612 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 3 removed transactions to 0 peers took 2.8280s
2025-04-17T20:18:29.822 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 5 removed transactions to 0 peers took 2.8750s
2025-04-17T20:18:29.822 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.88s, pre_validation: 2.77s, CLVM: 2.77s, post-process: 0.09s, cost: 7905584904, percent full: 71.869% header_hash: 8f0390fbe862bd5f4923a94a0ef17acaa2c267ed92a3d6f9b28bc14389ad2fee height: 6920249
2025-04-17T20:21:22.210 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 6 removed transactions to 0 peers took 2.6410s
2025-04-17T20:24:13.523 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 2 removed transactions to 0 peers took 1.3280s
2025-04-17T20:33:46.731 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 3 removed transactions to 0 peers took 1.2340s
2025-04-17T20:35:54.563 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 5 removed transactions to 0 peers took 1.4060s
2025-04-17T20:40:06.168 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 6 removed transactions to 0 peers took 1.7190s
2025-04-17T20:40:56.886 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 3 removed transactions to 0 peers took 1.4060s
2025-04-17T20:44:56.894 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 5 removed transactions to 0 peers took 1.5930s
2025-04-17T20:48:50.953 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 7 removed transactions to 0 peers took 1.2350s
2025-04-17T20:49:44.883 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 4 removed transactions to 0 peers took 1.2190s
2025-04-17T20:51:48.683 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.55s, pre_validation: 0.31s, CLVM: 0.31s, post-process: 0.19s, cost: 2716574918, percent full: 24.696% header_hash: eb3b1a0fb432bbe831e5700ee48cf6e50a4d9596a047d42f7e667385018e3307 height: 6920358
2025-04-17T20:53:46.414 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 2 removed transactions to 0 peers took 1.7970s
2025-04-17T20:55:20.403 2.5.3 full_node full_node_server        : WARNING  cannot connect to 92.132.156.166, duplicate/self connection
2025-04-17T20:57:27.712 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 5 removed transactions to 0 peers took 1.7810s
2025-04-17T20:59:46.015 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 5 removed transactions to 0 peers took 1.7810s
2025-04-17T21:04:18.887 2.5.3 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'DOUBLE_SPEND', 'status': 3, 'txid': '0xaae0627cf1c6ccc1d9da80a9a23419c5388fb255bcc2d667ef9be75181ce7b73'}
2025-04-17T21:04:18.903 2.5.3 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'DOUBLE_SPEND', 'status': 3, 'txid': '0xaae0627cf1c6ccc1d9da80a9a23419c5388fb255bcc2d667ef9be75181ce7b73'}
2025-04-17T21:04:18.903 2.5.3 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'DOUBLE_SPEND', 'status': 3, 'txid': '0x437ab7f10bfd28409388c8a0242736759e8096fc11be11d75799297b472c3afe'}
2025-04-17T21:04:18.919 2.5.3 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'DOUBLE_SPEND', 'status': 3, 'txid': '0x2caabd54df7ece5d48669529ddf68198f846c09705c31fe6e27a238300744493'}
2025-04-17T21:04:18.919 2.5.3 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'DOUBLE_SPEND', 'status': 3, 'txid': '0x3fb988e838a18d21c7fb884774966bc6aaeee2da1a748ab40c8598de67b6165b'}
2025-04-17T21:04:18.919 2.5.3 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'DOUBLE_SPEND', 'status': 3, 'txid': '0xd8dc1d4ad54350116d40ef5d1472a59c345f6def648a0cebef0d3814697b9f1f'}
2025-04-17T21:04:18.935 2.5.3 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'DOUBLE_SPEND', 'status': 3, 'txid': '0x775beb7a2532e3425de96d1dcc39a4023971be950a7efb3ec58dbefd66aeae42'}
2025-04-17T21:04:18.935 2.5.3 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'DOUBLE_SPEND', 'status': 3, 'txid': '0x7e977d2d398652870c74759e7a7e9d619b9e117de48f43c9c636c8d282236d18'}
2025-04-17T21:04:18.935 2.5.3 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'DOUBLE_SPEND', 'status': 3, 'txid': '0x437ab7f10bfd28409388c8a0242736759e8096fc11be11d75799297b472c3afe'}
2025-04-17T21:04:18.951 2.5.3 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'DOUBLE_SPEND', 'status': 3, 'txid': '0x2caabd54df7ece5d48669529ddf68198f846c09705c31fe6e27a238300744493'}
2025-04-17T21:04:18.951 2.5.3 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'DOUBLE_SPEND', 'status': 3, 'txid': '0x3fb988e838a18d21c7fb884774966bc6aaeee2da1a748ab40c8598de67b6165b'}
2025-04-17T21:04:18.966 2.5.3 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'DOUBLE_SPEND', 'status': 3, 'txid': '0xd8dc1d4ad54350116d40ef5d1472a59c345f6def648a0cebef0d3814697b9f1f'}
2025-04-17T21:04:18.966 2.5.3 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'DOUBLE_SPEND', 'status': 3, 'txid': '0x775beb7a2532e3425de96d1dcc39a4023971be950a7efb3ec58dbefd66aeae42'}
2025-04-17T21:04:18.966 2.5.3 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'DOUBLE_SPEND', 'status': 3, 'txid': '0x7e977d2d398652870c74759e7a7e9d619b9e117de48f43c9c636c8d282236d18'}
2025-04-17T21:04:18.982 2.5.3 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'DOUBLE_SPEND', 'status': 3, 'txid': '0xaae0627cf1c6ccc1d9da80a9a23419c5388fb255bcc2d667ef9be75181ce7b73'}
2025-04-17T21:04:18.982 2.5.3 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'DOUBLE_SPEND', 'status': 3, 'txid': '0x437ab7f10bfd28409388c8a0242736759e8096fc11be11d75799297b472c3afe'}
2025-04-17T21:04:18.998 2.5.3 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'DOUBLE_SPEND', 'status': 3, 'txid': '0x2caabd54df7ece5d48669529ddf68198f846c09705c31fe6e27a238300744493'}
2025-04-17T21:04:18.998 2.5.3 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'DOUBLE_SPEND', 'status': 3, 'txid': '0x3fb988e838a18d21c7fb884774966bc6aaeee2da1a748ab40c8598de67b6165b'}
2025-04-17T21:04:18.998 2.5.3 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'DOUBLE_SPEND', 'status': 3, 'txid': '0xd8dc1d4ad54350116d40ef5d1472a59c345f6def648a0cebef0d3814697b9f1f'}
2025-04-17T21:05:59.918 2.5.3 wallet wallet_server              : ERROR    WebSocket Error: WSMessage(type=<WSMsgType.ERROR: 258>, data=ServerTimeoutError(), extra=None)
2025-04-17T21:05:59.920 2.5.3 wallet chia.wallet.wallet_node    : ERROR    Error syncing to PeerInfo(_ip=IPv4Address('85.206.78.126'), _port=8444)
Traceback (most recent call last):
  File "chia\wallet\wallet_node.py", line 1215, in new_peak_from_untrusted
  File "chia\wallet\wallet_node.py", line 1224, in long_sync_from_untrusted
  File "chia\wallet\wallet_node.py", line 1353, in fetch_and_update_weight_proof
Exception: weight proof response was none
2025-04-17T21:06:23.670 2.5.3 wallet chia.wallet.wallet_node    : WARNING  Peer None did not respond in time.
2025-04-17T21:06:23.670 2.5.3 wallet wallet_server              : WARNING  Banning 85.206.78.126 for 120 seconds
2025-04-17T21:06:33.786 2.5.3 wallet chia.wallet.wallet_node    : WARNING  Peer None did not respond in time.
2025-04-17T21:06:33.786 2.5.3 wallet wallet_server              : WARNING  Banning 104.6.229.69 for 120 seconds
@heget heget added the bug Something isn't working label Apr 18, 2025
@heget
Copy link
Author

heget commented Apr 23, 2025 •
edited
Loading

And today again repeating the same, mempool going to be full and my node lost sync.

Image

Image

2025-04-18T12:08:03.564 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.69s, pre_validation: 0.20s, CLVM: 0.20s, post-process: 0.88s, cost: 3845400863, percent full: 34.958% header_hash: de696ff443c1651aa4cb98a4afa9e4e857aba4c99934ec3fcb984ad1c44f479a height: 6923346
2025-04-18T12:09:00.556 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.77s, pre_validation: 0.27s, CLVM: 0.27s, post-process: 0.88s, cost: 3826100222, percent full: 34.783% header_hash: 47c58b48c8bd1f475bf417addc372109b7e81a6315e1f3c28657a29f526b97a3 height: 6923351
2025-04-18T12:10:05.829 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.83s, pre_validation: 0.38s, CLVM: 0.38s, post-process: 0.86s, cost: 3874946480, percent full: 35.227% header_hash: 7f0caa4dbae2e804217dda13d34e57af2c43d609ba03d96ddaf858acbb09f98a height: 6923353
2025-04-18T15:20:09.549 2.5.3 full_node chia.full_node.mempool_manager: WARNING  updating the mempool using the slow-path. peak: f29c5ab7cfe440d3e3bcfe53cb6417129b7a28280a905fb23ebd04d95c0f4d7f new-peak-prev: 821f4cb54d71d1e3af40cfe2026f524dd1ce84e98f7e749da185bde4cb77cf6d coins: set
2025-04-18T15:20:41.901 2.5.3 full_node chia.full_node.mempool_manager: WARNING  updating the mempool using the slow-path. peak: 3e318ab4b1003cc12fdcd85efc1ed33b532c45498a1753d5faa756d67f1fd332 new-peak-prev: 377c5b8cd1d671acc34947fcf46d3dac888ce6b75f6eb55c9455f49ffca8a439 coins: set
2025-04-18T19:36:40.117 2.5.3 full_node chia.full_node.full_node_api: ERROR    got weight proof request for unknown peak d68e979c5991e915d5a09a201fc23b2210420dc3c071ef74a902fddd02d6215e
2025-04-19T01:51:19.435 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.91s, pre_validation: 0.39s, CLVM: 0.39s, post-process: 0.92s, cost: 3715763245, percent full: 33.78% header_hash: 26439275ec6306bed598b47e65e837d4eb2c17eabbe023e16ea8122e2c87f314 height: 6925957
2025-04-19T01:55:52.496 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.95s, pre_validation: 0.30s, CLVM: 0.30s, post-process: 1.06s, cost: 3678012670, percent full: 33.436% header_hash: d717cc9599605b808d7999076d1ae8b7fc7b18d8d6efae5d44c3afa75f49245c height: 6925965
2025-04-19T01:56:39.496 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.16s, pre_validation: 0.20s, CLVM: 0.20s, post-process: 0.77s, cost: 2857284706, percent full: 25.975% header_hash: efab2c6a93a343ec05519374921c1d92ce6b89b66bc2ea0ac36ab84bb8a65287 height: 6925972
2025-04-19T06:02:18.270 2.5.3 full_node chia.full_node.full_node_api: ERROR    got weight proof request for unknown peak 803eccf49957bbb9bb86e10ebb0e554404b8ab4fa227061fcf6fd100c49ece77
2025-04-19T06:15:10.191 2.5.3 full_node chia.full_node.mempool_manager: WARNING  updating the mempool using the slow-path. peak: 4719a907ace0149b27a9cb7589e6ff6af44b02e7ffc4338bafed06effe83ef5b new-peak-prev: 6c5ae2f1fee7e7b4b62a0c0757ed2c42776220f3abfe028565e1159a46d194b5 coins: set
2025-04-19T11:53:24.872 2.5.3 full_node chia.full_node.full_node_api: ERROR    got weight proof request for unknown peak 71350a4b5c4d67c38e3aae53660ac8bff0eebdee05bf57c73b7260f0db5ef040
2025-04-19T14:24:17.201 2.5.3 full_node chia.full_node.full_node_api: ERROR    got weight proof request for unknown peak 1763f9eb5e44c98b0ceb4d5d38d91704b72b98339b6f824e47ec1456812fb90b
2025-04-19T15:00:47.802 2.5.3 full_node chia.full_node.mempool_manager: WARNING  updating the mempool using the slow-path. peak: 0a2e41ae626c686d55e23d81565e4db822b654e213a6c1837dffa31f77f9d9de new-peak-prev: dfd561411b4770fdf5dc42a902ea7dbb77e7c4fd6f3dcc074282bd4cceb68fdb coins: set
2025-04-19T15:00:59.282 2.5.3 full_node chia.full_node.mempool_manager: WARNING  updating the mempool using the slow-path. peak: 59efe973f8dab88114edcc5628a537f9d618a2f4bfd5a7803eb63e1752f3f099 new-peak-prev: dfd561411b4770fdf5dc42a902ea7dbb77e7c4fd6f3dcc074282bd4cceb68fdb coins: set
2025-04-19T20:59:37.401 2.5.3 full_node chia.full_node.full_node_api: ERROR    got weight proof request for unknown peak db7ae17240dd5a9438b0fd81f7ee74a4743b29d302f677de62aa91fc2696df66
2025-04-20T01:49:48.238 2.5.3 full_node chia.full_node.mempool_manager: WARNING  updating the mempool using the slow-path. peak: d89ca96fdc7a3f961d19636334cb39f20b117a40ea24c3717af6cb6d1ea4d6e5 new-peak-prev: c5450348fc11bc062235f95da6a15478118c4d02161f09fcb97a2ce2e209ce91 coins: set
2025-04-20T03:21:36.868 2.5.3 full_node chia.full_node.mempool_manager: WARNING  updating the mempool using the slow-path. peak: 3d230c4095a6a3ee6330b56a8a81f21c7cac30a6844caf78a90d7a8d850e83c4 new-peak-prev: 8390fdc4846f2819c8d2f96b899855fe27c7459c2b065106311b73d0803943d6 coins: set
2025-04-20T03:21:38.567 2.5.3 full_node chia.full_node.full_node: WARNING  Do not have sub slot 9a308569754fe432b8a54c66d1cec796d57389ff33c76e468b25d9c0db6696d5
2025-04-20T03:21:54.581 2.5.3 full_node chia.full_node.mempool_manager: WARNING  updating the mempool using the slow-path. peak: 393d38b1fe52f12ec169f8ffadf357dba674293e24eff64db850f52981be1e7b new-peak-prev: 8390fdc4846f2819c8d2f96b899855fe27c7459c2b065106311b73d0803943d6 coins: set
2025-04-20T03:21:54.689 2.5.3 full_node full_node_server        : ERROR    Exception: Error code: DOUBLE_SPEND [], PeerInfo(_ip=IPv4Address('154.61.62.190'), _port=8444). Traceback (most recent call last):
  File "chia\server\ws_connection.py", line 443, in wrapped_coroutine
  File "chia\full_node\full_node_api.py", line 574, in respond_unfinished_block
  File "chia\full_node\full_node.py", line 2407, in add_unfinished_block
chia.util.errors.ConsensusError: Error code: DOUBLE_SPEND []

2025-04-20T03:21:54.705 2.5.3 full_node full_node_server        : ERROR    Exception: Error code: DOUBLE_SPEND [] <class 'chia.util.errors.ConsensusError'>, closing connection PeerInfo(_ip=IPv4Address('154.61.62.190'), _port=8444). Traceback (most recent call last):
  File "chia\server\ws_connection.py", line 464, in _api_call
  File "asyncio\tasks.py", line 445, in wait_for
  File "chia\server\ws_connection.py", line 443, in wrapped_coroutine
  File "chia\full_node\full_node_api.py", line 574, in respond_unfinished_block
  File "chia\full_node\full_node.py", line 2407, in add_unfinished_block
chia.util.errors.ConsensusError: Error code: DOUBLE_SPEND []

2025-04-20T03:21:54.705 2.5.3 full_node full_node_server        : WARNING  Banning 154.61.62.190 for 600 seconds
2025-04-20T03:50:40.782 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting added transaction e023d2ad016b88d8bde7ff19815543b5a7cd728d7ad7630265e23f2d2aef0719 to 0 peers took 0.9370s
2025-04-20T09:42:07.723 2.5.3 harvester harvester_server        : ERROR    Exception: Value -1450200 does not fit into uint64, PeerInfo(_ip=IPv4Address('127.0.0.1'), _port=8447). Traceback (most recent call last):
  File "chia\server\ws_connection.py", line 443, in wrapped_coroutine
  File "chia\harvester\harvester_api.py", line 284, in new_signage_point_harvester
  File "chia_rs\struct_stream.py", line 82, in __init__
ValueError: Value -1450200 does not fit into uint64

2025-04-20T09:42:07.723 2.5.3 harvester harvester_server        : ERROR    Exception: Value -1450200 does not fit into uint64 <class 'ValueError'>, closing connection PeerInfo(_ip=IPv4Address('127.0.0.1'), _port=8447). Traceback (most recent call last):
  File "chia\server\ws_connection.py", line 464, in _api_call
  File "asyncio\tasks.py", line 445, in wait_for
  File "chia\server\ws_connection.py", line 443, in wrapped_coroutine
  File "chia\harvester\harvester_api.py", line 284, in new_signage_point_harvester
  File "chia_rs\struct_stream.py", line 82, in __init__
ValueError: Value -1450200 does not fit into uint64

2025-04-20T09:42:07.723 2.5.3 harvester harvester_server        : WARNING  Trying to ban localhost for 10, but will not ban
2025-04-20T10:55:42.601 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.09s, pre_validation: 0.27s, CLVM: 0.27s, post-process: 0.75s, cost: 3196460827, percent full: 29.059% header_hash: 6121e3ad6349a38bb5fbfb3276af17700d435dda3a54c392e7cd7821ebb3a967 height: 6932303
2025-04-20T11:07:51.115 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.12s, pre_validation: 0.11s, CLVM: 0.09s, post-process: 0.77s, cost: 3672296092, percent full: 33.385% header_hash: fbf2786d706b29a7678f9a298ef896cc7710e21a46ec0b6e80e728f843451ca5 height: 6932335
2025-04-20T18:49:14.920 2.5.3 full_node chia.full_node.mempool_manager: WARNING  updating the mempool using the slow-path. peak: cb206ce70e5884f69981d111ed86e9222f8a6413bad723e270ec14bd796bc375 new-peak-prev: 480efcb66808e91b938f24279737dac6c18295131d8c7168dee8eb84baf8ef53 coins: set
2025-04-20T19:50:13.264 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting added transaction f8636ad5224b807adda924fb9c10e046fd8bd803000b2fc302f8e5ccc6997a82 to 0 peers took 0.8910s
2025-04-21T02:51:06.898 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.58s, pre_validation: 0.41s, CLVM: 0.41s, post-process: 0.92s, cost: 3614630983, percent full: 32.86% header_hash: 00595474d07842d2e196953674ed2e182cea8111e659915fab490415d871975f height: 6935367
2025-04-21T02:57:23.551 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.20s, pre_validation: 0.16s, CLVM: 0.15s, post-process: 0.75s, cost: 3671996078, percent full: 33.382% header_hash: fa5910f11bb88c15905e72ae36c4ce45b66fcd64bdcde23ea9b412adb7d0f6cb height: 6935390
2025-04-21T02:58:29.725 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.59s, pre_validation: 0.28s, CLVM: 0.28s, post-process: 0.95s, cost: 3561206979, percent full: 32.375% header_hash: 3e067d7c5d731fdd06d41e9534d06d95ebf1f02f86a47b78281b6816979418e9 height: 6935393
2025-04-21T05:04:03.366 2.5.3 full_node chia.full_node.mempool_manager: WARNING  updating the mempool using the slow-path. peak: 16d97229fab44165bbb2730d6fadade9d3bf8795f1cf9c73ef7a4ad830a13bc6 new-peak-prev: 920b5929fc730074a4c08012c4e4eb9b602477c1de32d45e83c8168da18d9f13 coins: set
2025-04-21T14:43:44.392 2.5.3 full_node chia.full_node.mempool_manager: WARNING  updating the mempool using the slow-path. peak: 9a720e24cb9c0fffbc08204d7ef593df3a6a4df841e799d6eeb28008c2ce43a2 new-peak-prev: ca3542719f6d18539e6708b75c40263604ac6fa92c075aa89867ca7a52b7dd5f coins: set
2025-04-21T14:43:45.647 2.5.3 full_node chia.consensus.block_body_validation: ERROR    Err.UNKNOWN_UNSPENT: COIN ID: 915c7cff2837e37875edbdbd50d70f3809b8109cc9f596e8cd9774266b2ef722 fork_info: ForkInfo(fork_height=6937604, peak_height=6937604, peak_hash=<bytes32: 695201cf36e72193e7ee6742f93e699b6a6a56d84e24c9fd4a978457a88648f3>, additions_since_fork={}, removals_since_fork={}, block_hashes=[])
2025-04-21T14:43:45.647 2.5.3 full_node full_node_server        : ERROR    Exception: Error code: UNKNOWN_UNSPENT [], PeerInfo(_ip=IPv4Address('87.244.50.26'), _port=8444). Traceback (most recent call last):
  File "chia\server\ws_connection.py", line 443, in wrapped_coroutine
  File "chia\full_node\full_node_api.py", line 574, in respond_unfinished_block
  File "chia\full_node\full_node.py", line 2407, in add_unfinished_block
chia.util.errors.ConsensusError: Error code: UNKNOWN_UNSPENT []

2025-04-21T14:43:45.647 2.5.3 full_node full_node_server        : ERROR    Exception: Error code: UNKNOWN_UNSPENT [] <class 'chia.util.errors.ConsensusError'>, closing connection PeerInfo(_ip=IPv4Address('87.244.50.26'), _port=8444). Traceback (most recent call last):
  File "chia\server\ws_connection.py", line 464, in _api_call
  File "asyncio\tasks.py", line 445, in wait_for
  File "chia\server\ws_connection.py", line 443, in wrapped_coroutine
  File "chia\full_node\full_node_api.py", line 574, in respond_unfinished_block
  File "chia\full_node\full_node.py", line 2407, in add_unfinished_block
chia.util.errors.ConsensusError: Error code: UNKNOWN_UNSPENT []

2025-04-21T14:43:45.663 2.5.3 full_node full_node_server        : WARNING  Banning 87.244.50.26 for 600 seconds
2025-04-21T14:44:14.460 2.5.3 full_node chia.full_node.mempool_manager: WARNING  updating the mempool using the slow-path. peak: db43170b21c7c015e64aadc75203f12e03609d44014433017ee071fb161f1162 new-peak-prev: 695201cf36e72193e7ee6742f93e699b6a6a56d84e24c9fd4a978457a88648f3 coins: set
2025-04-21T18:14:51.133 2.5.3 full_node chia.full_node.mempool_manager: WARNING  updating the mempool using the slow-path. peak: 5ee8a6681bab4764b1fd1237b3b8f091c3320a9de35ed4922ffebb4df585d134 new-peak-prev: bd6c4d7b62724af3b7f10cde8a036b7840fbc25875fe957d36d76d984677b460 coins: set
2025-04-21T18:23:32.157 2.5.3 full_node chia.full_node.mempool_manager: WARNING  updating the mempool using the slow-path. peak: eede005b8da4652548c55dacdeea2f430010f188293f9b96aa520fc61797adcd new-peak-prev: 3801fa83574d6ec4e23e507e7ec741d44721e9dbf878c233be683b1d39aebd29 coins: set
2025-04-21T22:41:53.449 2.5.3 full_node chia.full_node.mempool_manager: WARNING  updating the mempool using the slow-path. peak: 3156b59a09703782b1bd09824e0abaeb010bdc7a15c3273d654c4ef17bfe6d6a new-peak-prev: 7858b63c3c4da931ad6eca3686253aa450d79d1729ff41b28de70dbffdfe4948 coins: set
2025-04-21T22:41:56.548 2.5.3 full_node chia.full_node.mempool_manager: WARNING  updating the mempool using the slow-path. peak: eadd170b1be53f801777d46fd4165c999f2e398840a193c02dbfd1688e1be2b6 new-peak-prev: 7858b63c3c4da931ad6eca3686253aa450d79d1729ff41b28de70dbffdfe4948 coins: set
2025-04-22T03:21:01.031 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.64s, pre_validation: 0.61s, CLVM: 0.61s, post-process: 0.78s, cost: 3287841333, percent full: 29.889% header_hash: 0ce6191b5610ae0b3266cef455339679715eeb6fb78a248f2cc6d47417a9e7f9 height: 6940020
2025-04-22T03:30:01.944 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.41s, pre_validation: 0.48s, CLVM: 0.48s, post-process: 0.72s, cost: 3715215971, percent full: 33.775% header_hash: 95f9be0f68c3d4411a77788866c0e541070e2c11509b03422aeaadb94a2c91ac height: 6940041
2025-04-22T10:04:33.432 2.5.3 full_node chia.full_node.mempool_manager: WARNING  updating the mempool using the slow-path. peak: 21e165d0f50f1bfd5067f48d1ea877a996c1a8d7bbe88719f720d356b49fb954 new-peak-prev: 3cb11403b585561d57cbb066a33c873c5ab5bb05ecd124663c8d79e3ab142ec9 coins: set
2025-04-22T10:04:42.112 2.5.3 full_node chia.full_node.mempool_manager: WARNING  updating the mempool using the slow-path. peak: 286db2d94c90ff83bf2b5ec0ce6242af2102de05e36340231bdeeb573a894e52 new-peak-prev: 3cb11403b585561d57cbb066a33c873c5ab5bb05ecd124663c8d79e3ab142ec9 coins: set
2025-04-22T16:16:44.545 2.5.3 full_node chia.full_node.mempool_manager: WARNING  updating the mempool using the slow-path. peak: 9c0ab881e38c17d926d7d152fa5e08f734bc90c517682a41dfe46322d98a6b48 new-peak-prev: 378863829970ade826df456609d12a4c9cb927d1e794f110a492dbf17193adfb coins: set
2025-04-22T21:34:15.526 2.5.3 full_node chia.full_node.mempool_manager: WARNING  updating the mempool using the slow-path. peak: 6d90d9888fa20039b76ca7d21d9e853544a5171afea4756f1e0e17ae7292592e new-peak-prev: d9db3020c750083cff8acca9172cd76c621fced3f8bbc3d5a22e85f1057ee1b4 coins: set
2025-04-22T21:34:44.512 2.5.3 full_node chia.full_node.mempool_manager: WARNING  updating the mempool using the slow-path. peak: db4c5faf9ce924785c04fa73055c03f9eb84ce84032cff0cd6d0fc824a81aeb5 new-peak-prev: 799632a5696100f560d8f78d490f76b8d002ea7628434dee76b987b89bce9998 coins: set
2025-04-23T02:45:52.008 2.5.3 full_node chia.full_node.mempool_manager: WARNING  updating the mempool using the slow-path. peak: 57bfaed3687d4bc7992c90c64a1fc64bdbed25f85b40d7d66e0e1236170f35e0 new-peak-prev: 3ab6a929ddc9425582cfca1a4c703ccba614385fdd8134103e1ceab793734c15 coins: set
2025-04-23T03:02:23.302 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 2 removed transactions to 0 peers took 1.2970s
2025-04-23T03:03:10.330 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 1 removed transactions to 0 peers took 1.2970s
2025-04-23T03:04:04.059 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 6 removed transactions to 0 peers took 2.6720s
2025-04-23T03:04:45.038 2.5.3 full_node chia.full_node.mempool_manager: WARNING  updating the mempool using the slow-path. peak: 597bf6dc859fe59ad7250be4625442bb0521ebb005d58ac54741561f263432ac new-peak-prev: 3db3894b6482a733d970dad8d08c115766a9895a684cc789bd97a70ecd699f06 coins: set
2025-04-23T03:05:03.036 2.5.3 full_node chia.full_node.mempool_manager: WARNING  pre_validate_spendbundle took 2.2893 seconds for e134bf27d448134a44b1c2f70b7cb96a1c4a73a9f7e32d0ad9597598d5a504f0 (queue-size: 0)
2025-04-23T03:07:32.813 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 3 removed transactions to 0 peers took 2.7500s
2025-04-23T03:08:13.097 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 3 removed transactions to 0 peers took 2.7190s
2025-04-23T03:10:39.775 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 7 removed transactions to 0 peers took 1.4220s
2025-04-23T03:14:33.149 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 8 removed transactions to 0 peers took 1.4530s
2025-04-23T03:17:53.558 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 5 removed transactions to 0 peers took 1.3750s
2025-04-23T03:23:42.764 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 3 removed transactions to 0 peers took 1.3910s
2025-04-23T03:25:53.774 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 16.89s, pre_validation: 0.30s, CLVM: 0.30s, post-process: 1.38s, cost: 3696162151, percent full: 33.601% header_hash: e11b2e914030b245460c6f4c0a448137bc8f669ec5929190290bae3fe5917aaf height: 6944714
2025-04-23T03:27:08.484 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 7 removed transactions to 0 peers took 1.4850s
2025-04-23T03:27:08.500 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 3.61s, pre_validation: 0.50s, CLVM: 0.50s, post-process: 1.19s, cost: 7939004722, percent full: 72.173% header_hash: 9226ae9be5ebb6f4bd28fceb1687120e391fc9b219131a27f79532468881fcf8 height: 6944716
2025-04-23T03:28:06.066 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.55s, pre_validation: 0.25s, CLVM: 0.25s, post-process: 0.92s, cost: 2953884288, percent full: 26.853% header_hash: c98188a644e2daed207a6093061c804e174c53d6f29c4358621aa0270dee2dbb height: 6944718
2025-04-23T03:28:21.432 2.5.3 full_node chia.full_node.mempool_manager: WARNING  pre_validate_spendbundle took 2.1576 seconds for 1a1c9fa1b163c3021886b2d90d6f725b1b0de491790ae66b6c7d818e39e1fc8b (queue-size: 0)
2025-04-23T03:32:03.955 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 3 removed transactions to 0 peers took 1.4370s
2025-04-23T03:32:53.902 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 3 removed transactions to 0 peers took 3.2190s
2025-04-23T03:33:35.332 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 6 removed transactions to 0 peers took 2.7650s
2025-04-23T03:35:50.333 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 7 removed transactions to 0 peers took 1.3900s
2025-04-23T03:36:31.140 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 4 removed transactions to 0 peers took 2.8910s
2025-04-23T03:37:26.928 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 3 removed transactions to 0 peers took 2.9840s
2025-04-23T03:37:58.678 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 2 removed transactions to 0 peers took 1.3750s
2025-04-23T03:39:26.335 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 4 removed transactions to 0 peers took 1.6250s
2025-04-23T03:44:01.696 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 2 removed transactions to 0 peers took 2.7190s
2025-04-23T03:45:12.969 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 5 removed transactions to 0 peers took 3.8750s
2025-04-23T03:49:58.339 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 6 removed transactions to 0 peers took 1.5470s
2025-04-23T03:54:55.189 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 7 removed transactions to 0 peers took 1.5780s
2025-04-23T03:55:31.701 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 1 removed transactions to 0 peers took 1.5780s
2025-04-23T03:56:20.830 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 4 removed transactions to 0 peers took 3.0310s
2025-04-23T03:56:55.138 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 2 removed transactions to 0 peers took 1.6560s
2025-04-23T03:58:31.011 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 4 removed transactions to 0 peers took 1.6720s
2025-04-23T04:02:12.423 2.5.3 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'DOUBLE_SPEND', 'status': 3, 'txid': '0xb26f822e86e8ee64f639f3160e65a7d241c0120dec3ed24496cc03ca90c3f8ec'}
2025-04-23T04:02:12.439 2.5.3 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'DOUBLE_SPEND', 'status': 3, 'txid': '0xeefe1d04a9ff1e4419bad544e33b31e1c00a2d651421daa0615ac07601e7d904'}
2025-04-23T04:02:12.439 2.5.3 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'DOUBLE_SPEND', 'status': 3, 'txid': '0xb26f822e86e8ee64f639f3160e65a7d241c0120dec3ed24496cc03ca90c3f8ec'}
2025-04-23T04:02:12.439 2.5.3 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'DOUBLE_SPEND', 'status': 3, 'txid': '0xeefe1d04a9ff1e4419bad544e33b31e1c00a2d651421daa0615ac07601e7d904'}
2025-04-23T04:02:12.454 2.5.3 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'DOUBLE_SPEND', 'status': 3, 'txid': '0xb26f822e86e8ee64f639f3160e65a7d241c0120dec3ed24496cc03ca90c3f8ec'}
2025-04-23T04:02:12.454 2.5.3 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'DOUBLE_SPEND', 'status': 3, 'txid': '0xeefe1d04a9ff1e4419bad544e33b31e1c00a2d651421daa0615ac07601e7d904'}
2025-04-23T04:14:31.994 2.5.3 wallet chia.wallet.wallet_node    : ERROR    Error syncing to PeerInfo(_ip=IPv4Address('76.183.176.76'), _port=8444)
Traceback (most recent call last):
  File "chia\wallet\wallet_node.py", line 1215, in new_peak_from_untrusted
  File "chia\wallet\wallet_node.py", line 1224, in long_sync_from_untrusted
  File "chia\wallet\wallet_node.py", line 1353, in fetch_and_update_weight_proof
Exception: weight proof response was none
2025-04-23T04:14:40.634 2.5.3 wallet chia.wallet.wallet_node    : WARNING  Peer None did not respond in time.
2025-04-23T04:14:40.634 2.5.3 wallet wallet_server              : WARNING  Banning 76.183.176.76 for 120 seconds
2025-04-23T04:14:41.614 2.5.3 wallet chia.wallet.wallet_node    : WARNING  Peer None did not respond in time.
2025-04-23T04:14:41.629 2.5.3 wallet wallet_server              : WARNING  Banning 76.183.176.76 for 120 seconds
2025-04-23T04:14:41.935 2.5.3 wallet chia.wallet.wallet_node    : WARNING  Peer None did not respond in time.
2025-04-23T04:14:41.951 2.5.3 wallet wallet_server              : WARNING  Banning 76.183.176.76 for 120 seconds
2025-04-23T04:14:42.331 2.5.3 wallet chia.wallet.wallet_node    : WARNING  Peer None did not respond in time.
2025-04-23T04:14:42.331 2.5.3 wallet wallet_server              : WARNING  Banning 76.183.176.76 for 120 seconds
2025-04-23T04:14:42.981 2.5.3 wallet chia.wallet.wallet_node    : WARNING  Peer None did not respond in time.
2025-04-23T04:14:42.981 2.5.3 wallet wallet_server              : WARNING  Banning 76.183.176.76 for 120 seconds
2025-04-23T04:14:44.205 2.5.3 wallet chia.wallet.wallet_node    : WARNING  Peer None did not respond in time.
2025-04-23T04:14:44.205 2.5.3 wallet wallet_server              : WARNING  Banning 24.237.156.167 for 120 seconds
2025-04-23T04:14:53.727 2.5.3 wallet chia.wallet.wallet_node    : WARNING  Peer None did not respond in time.
2025-04-23T04:14:53.727 2.5.3 wallet wallet_server              : WARNING  Banning 76.183.176.76 for 120 seconds
2025-04-23T06:48:01.486 2.5.3 wallet chia.wallet.wallet_node    : WARNING  bad header block response from Peer PeerInfo(_ip=IPv4Address('174.85.87.16'), _port=8444).
2025-04-23T07:48:15.996 2.5.3 wallet chia.wallet.wallet_node    : WARNING  Peer PeerInfo(_ip=IPv4Address('158.174.146.209'), _port=8444) did not respond in time.
2025-04-23T07:48:15.996 2.5.3 wallet wallet_server              : WARNING  Banning 158.174.146.209 for 120 seconds

@BrandtH22
Copy link
Contributor

Hey @heget ,

I am seeing a number of different things in these logs and have listed them below in the order I would troubleshoot (as fixing the prior ones might resolve the latter ones):

  • Block validation time warnings: the node is having trouble validating blocks that are only 30% full, this usually indicates that either the blockchain db is on too slow of a drive or a computer resource (cpu or memory) is being overloaded. Can you let us know: What type of drive is the db running on (HDD or SSD)? How is that drive connected (sata, usb, etc)? Are you seeing resource over-utilization in resource monitor? NOTE this issue could well lead to most of the other issues present
  • Unknown unspent errors: these could be from bad data being sent from peers or could be an indicator that the blockchain db has missing or invalid data. I would verify the item above first but once confirmed that the db is on a fast ssd and connection method with no other resource issues on the device then I would verify the db with chia db validate if that succeeds without issue then run chia db validate --validate-blocks. If either of these indicate invalid blocks or other issues with the db let us know and we can provide the instructions to get the official DB snapshot.
  • Banning multiple peers for various reasons: this could be related to the above, as we work to resolve the above issues make sure to delete the peers.dat and peers.sqlite files in the ~/.chia/mainnet/db/ directory with chia stopped. This will make sure that when chia starts your node will get a new set of peers (which would change what mempool you are rebuilding as you are getting the data from different sources)
  • Harvester value exceptions: these could well be due to the above db and peer issues so if they persist after resolving the others let's readdress
  • Broadcasting removed transaction warnings: these are expected when your mempool has a series of invalid transactions and is purging those invalid transactions to clear your own mempool. The lack of sending to the peers indicates your peers do not have those transactions in their mempools (likely since they have already purged them). I would expect these messages to reduce in volume as your node connects to more quality peers.
  • Weight proof response was none: This is expected and indicative that your node was connected to a peer on a fork. As it requests the weight proof your node is disconnected from that peer as it recognizes it is not on the correct chain

This all said let's start with db storage location and connection method then go from there

@heget
Copy link
Author

heget commented Apr 23, 2025 •
edited
Loading

Hi @BrandtH22 ,
My HW is good enough even if I have 1+ PT farm.
My setup Iam solo farmer.
CPU Ryzen 5 5500U (average usage 4-12%), RAM 32 GB, HDD SSD Nvme 2TB ultra fast.

As I see it, I think the problem will be with the connection to quality peers. As you suggested, I will try to delete those files.
Alternatively, if there is a list of really quality peers, I like to define them manually.
I am on a passive network, so no open port, so I always have only 8 outbound peers.
But as I say, I noticed that these problems only happen when the mempool on the network is increased.
Other times the farm runs without problems and beautiful clean logs!
And it didn't happen until this version 5.2.3.

And also I've validated the DB first phase:

Image

Second phase : done also success I hope ;)

Image

And I had only peers.dat in this db folder, peers.sqlite doesn't exist?

@lpaln
Copy link

lpaln commented Apr 23, 2025 •
edited
Loading

I arrived here after noticing the same "WARNING Broadcasting X removed transactions to 0 peers took Xs" filling my logs.

Following...

Also, not addressed here is this other error:
2025-04-23T04:14:31.994 2.5.3 wallet chia.wallet.wallet_node : ERROR Error syncing to PeerInfo(_ip=IPv4Address('76.183.176.76'), _port=8444)
Traceback (most recent call last):
File "chia\wallet\wallet_node.py", line 1215, in new_peak_from_untrusted
File "chia\wallet\wallet_node.py", line 1224, in long_sync_from_untrusted
File "chia\wallet\wallet_node.py", line 1353, in fetch_and_update_weight_proof
Exception: weight proof response was none

I show this error as well (different IP) when starting the Chia client. I'm not having any luck interpreting what this is trying to communicate.

@heget
Copy link
Author

heget commented Apr 24, 2025

So the last things didn't help, I have it again because there was a peek in the mempool overnight and that caused this disconnection from peers. Or is there a bug somewhere in the wallet peers, because I still have them connected...?

Image

Image

Image

And look on the time when mempool is filling and the time in log and see that starts in the same time.

2025-04-24T06:31:42.811 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 7 removed transactions to 0 peers took 1.7190s
2025-04-24T06:25:01.592 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 6 removed transactions to 0 peers took 1.8280s
2025-04-24T06:16:56.643 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 13 removed transactions to 0 peers took 1.8440s
2025-04-24T06:07:05.428 2.5.3 full_node chia.full_node.mempool_manager: WARNING  pre_validate_spendbundle took 2.0345 seconds for d0da4d61f0644d2e27540c13390b5e3316badba00ec4a6a3dc3165bbf89b6f9c (queue-size: 0)
2025-04-24T06:05:48.017 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 3 removed transactions to 0 peers took 2.6880s
2025-04-24T06:03:46.212 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 3 removed transactions to 0 peers took 3.1720s
2025-04-24T06:03:07.862 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 2 removed transactions to 0 peers took 3.1250s
2025-04-24T06:01:39.441 2.5.3 full_node chia.full_node.mempool_manager: WARNING  pre_validate_spendbundle took 2.5247 seconds for 518a758b490db83da901fd6aacce7bee988ca21a9842c8efc542eb5d6cc1003d (queue-size: 2)
2025-04-24T06:01:34.875 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 6 removed transactions to 0 peers took 3.0780s
2025-04-24T06:00:12.453 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 1 removed transactions to 0 peers took 1.5150s
2025-04-24T05:57:58.824 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 3 removed transactions to 0 peers took 1.3590s
2025-04-24T05:51:54.762 2.5.3 full_node chia.full_node.full_node: WARNING  Broadcasting 4 removed transactions to 0 peers took 1.7190s
2025-04-24T04:51:19.096 2.5.3 full_node chia.full_node.mempool_manager: WARNING  updating the mempool using the slow-path. peak: a4c87f6fad43846070a7d46de09a6f102003ba7c59681aecfa67ea7c517f6ea7 new-peak-prev: 981b871f926ea06c4e69e21dcfb94520c1f2016265df4f36432920c5d4a6e7a3 coins: set
2025-04-24T04:38:04.526 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 2.55s, pre_validation: 0.23s, CLVM: 0.23s, post-process: 0.95s, cost: 2915853396, percent full: 26.508% header_hash: cbb0bd7ccc2119ba8eb4350e26e656adbf09bf79c9343eab8bcbcc82eabb78d1 height: 6949403
2025-04-24T04:37:21.740 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 3.12s, pre_validation: 0.25s, CLVM: 0.25s, post-process: 1.17s, cost: 3668007160, percent full: 33.346% header_hash: 48c346f6492997e36573971e6fc8e886c28d1f2927a2f1f289bc58d2d1bf763b height: 6949401
2025-04-24T04:36:08.959 2.5.3 full_node chia.full_node.full_node: WARNING  Block validation: 3.14s, pre_validation: 0.12s, CLVM: 0.12s, post-process: 1.27s, cost: 3854638444, percent full: 35.042% header_hash: 5db00a5a861bc9ac3083abe0c2d759350cca1220ab89f53d6935e0bbaecbaf49 height: 6949397
2025-04-24T02:51:00.014 2.5.3 full_node chia.full_node.mempool_manager: WARNING  updating the mempool using the slow-path. peak: f0b2bd389ac43adb1b474d555dd0b98abb6b71d01be7090e8d55f3fcd8841890 new-peak-prev: 0dcb5cfe27a7bbe3a125e4afaad49a6bcad45d38f045d514fc69c5d6e3f08440 coins: set
2025-04-24T01:59:09.213 2.5.3 full_node chia.full_node.full_node_store: WARNING  Bad SES, expected SubEpochSummary { prev_subepoch_summary_hash: 90c6b5d34aa8326524c14b0b6993e7c7065d85cebd999641f386b7864daaf690, reward_chain_hash: 467331822b41216417e07f7ae8ebe815757103925cbb9fe421aea619c2f2ba07, num_blocks_overflow: 28, new_difficulty: Some(11520), new_sub_slot_iters: Some(578813952) } expected hash bedcb3387e3418f72d1917ca36824119cbb51f162f04a5caa16922cbdb9383bf, got ChallengeChainSubSlot { challenge_chain_end_of_slot_vdf: VDFInfo { challenge: f439b420f32c5b03cc645e25ecc904bf67b2523e42c40b655c1a7681bd6c769a, number_of_iterations: 578813952, output: ClassgroupElement { data: 00004f0c4a398d363888159a19de9ea3cf1fc293ef434d3cd1338e4d0ee807c7231e2f023095e917c9515892581057d8713a8986fc873e343f4e826e2d898ca7702f911379690b6e25e1a3d80cc4287863dca88ec2a323f463b1931e5fe70abff63f0100 } }, infused_challenge_chain_sub_slot_hash: Some(a28503812ec2f30a1dd4ecce8f940e8de5257695db2701a30d2e28e3b6e522aa), subepoch_summary_hash: Some(a2528622695f7b9be255423b0394152d97d96f921d8037257767417f421f191f), new_sub_slot_iters: Some(574619648), new_difficulty: Some(11072) }
2025-04-24T01:59:09.213 2.5.3 full_node chia.full_node.full_node_store: WARNING  Bad SES, expected SubEpochSummary { prev_subepoch_summary_hash: 90c6b5d34aa8326524c14b0b6993e7c7065d85cebd999641f386b7864daaf690, reward_chain_hash: 467331822b41216417e07f7ae8ebe815757103925cbb9fe421aea619c2f2ba07, num_blocks_overflow: 28, new_difficulty: Some(11520), new_sub_slot_iters: Some(578813952) } expected hash bedcb3387e3418f72d1917ca36824119cbb51f162f04a5caa16922cbdb9383bf, got ChallengeChainSubSlot { challenge_chain_end_of_slot_vdf: VDFInfo { challenge: f439b420f32c5b03cc645e25ecc904bf67b2523e42c40b655c1a7681bd6c769a, number_of_iterations: 578813952, output: ClassgroupElement { data: 00004f0c4a398d363888159a19de9ea3cf1fc293ef434d3cd1338e4d0ee807c7231e2f023095e917c9515892581057d8713a8986fc873e343f4e826e2d898ca7702f911379690b6e25e1a3d80cc4287863dca88ec2a323f463b1931e5fe70abff63f0100 } }, infused_challenge_chain_sub_slot_hash: Some(a28503812ec2f30a1dd4ecce8f940e8de5257695db2701a30d2e28e3b6e522aa), subepoch_summary_hash: Some(a2528622695f7b9be255423b0394152d97d96f921d8037257767417f421f191f), new_sub_slot_iters: Some(574619648), new_difficulty: Some(11072) }
2025-04-24T01:32:48.691 2.5.3 full_node chia.full_node.mempool_manager: WARNING  updating the mempool using the slow-path. peak: 636e6c1b7ec873da940e0e4a9fcc46a554202e16a44f167d60ef4c8b2a72ad21 new-peak-prev: 187c2d891ed673fca920e3a594774e60b906850151ce4d48ad28bb09b438fb71 coins: set
2025-04-23T23:03:06.917 2.5.3 full_node chia.full_node.full_node_api: ERROR    got weight proof request for unknown peak d3ee693caf67ffc076c3393369c9ca9205061d51d37511542e6c7609d396b82c
2025-04-23T22:32:18.463 2.5.3 full_node chia.full_node.mempool_manager: WARNING  updating the mempool using the slow-path. peak: 78c6872a49013fd8f8332f4e014ad655ffac31c5af9e968623f005d5e1e30e9e new-peak-prev: 793bc2b9335c6597e8e4c1c370cb12df2bda2e4cd871f3f6b006d0335c1e36d8 coins: set
2025-04-23T22:30:05.798 2.5.3 full_node chia.full_node.mempool_manager: WARNING  updating the mempool using the slow-path. peak: 235e40c9bd3b5a19ea1754391e1a07b230bcfb9c4b59619b5af99cf22f88d589 new-peak-prev: 0564ea59ae023d19c33728e049ca389b6c7603948d88511c49aa6275c9bae47b coins: not set

And it was all good till this time startin on 6:00 UTC.

@wjblanke
Copy link
Contributor

wjblanke commented May 1, 2025

Could you turn your log setting to INFO? That will give us more information. I am trying to figure out why your non-wallet peers are disconnecting from you. Maybe there is a rate limit issue with the high number of TXs.

@wjblanke
Copy link
Contributor

wjblanke commented May 7, 2025

Do you see CPU spikes when this is occurring? We are trying to determine if it is an IO issue.

@heget
Copy link
Author

heget commented May 7, 2025 •
edited
Loading

Hi, there are no CPU spikes, my CPU has a maximum load of up to 20%.
But I'm not sure about that, because whenever this happened, the peers were already disconnected, and therefore the CPU load was low.
And I also set logging to INFO.
Anyway, I tried to use Active IP (open 8444) with more peers. So we'll see what happens when the mempool cost is full again.
Now we just have to wait, the farm is running fine since the last outage.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@wjblanke @heget @BrandtH22 @lpaln