Investigate why distance from head is is more than expected

[BigLep]

This is a tracking issue for investigating why F3 participation post activation is less than what we observed in passive testing hours before.

We went from 5 epochs behind on average to ~8 epochs behind on average.

Before activation:

https://grafana.f3.eng.filoz.org/d/edsu1k5s7gtfkb/f3-passive-testing?orgId=1&var-network=mainnet&var-instance=ida.f3.eng.filoz.org%3A80&from=1745798400000&to=1745884800000&viewPanel=56

After activation:

https://grafana.f3.eng.filoz.org/d/edsu1k5s7gtfkb/f3-passive-testing?orgId=1&var-network=mainnet&var-instance=ida.f3.eng.filoz.org%3A80&from=1745928000000&to=1746014400000&viewPanel=56

(note: I'm not showing one contiguous graph since there is a bootstrap phase which dramatically scales up the y-axis).

[BigLep]

2025-04-30

• Differences between passive ad activation manifest
  • finalization - there is an extra step in path of finalization
  • network name - runtime may be slight different

We're going to look into both of these.

There is a gap between instance 5 to ~30. Looks like coordinated drop in participation.

Curio thread

The concern was that the distance from head is now 10 vs. in passive testing it was 9 5% of the time.

We're focused on "our ship" first (manifest differences).

2025-04-29

• Forest

  • Waiting to see if/what their problems are

• Curio

  • Confirming they are good

Going the path of getting observer setup so can see who isn't participating

• Top priority is to get observer running

Hypothesis 1 : instances upgraded to the retracted version that won't being activated by contract

• If that were the case, they should join passive testing if we restarted it

[masih]

Checked pubsub settings in lotus, in relation to network name change in the activation manifest. The only difference i see in terms of code path execution in lotus is how the list of allowed topics is compiled here.

[masih]

New metrics to measure time spent on checkpointing is deployed on test nodes. It initially shows the process to be slow but it fluctuates quite a bit. Letting it collect data for a while.

[BigLep]

Per filecoin-project/f3-activation-contract#22 (comment) , lets also capture a snapshot the minerIds that are participating so we can take diffs in future of further changes.

[masih]

Time spent checkpointing settled to a small value, and unlikely to be causing issues here. The initial delay in checkpointing was only observed during instance restart which is expected since the node was slightly behind on syncing the chain. After that checkpointing time reduced to a few milliseconds at 99th percentile.

[BigLep]

2025-05-01 standup update:
Pubsub and finalization don't appear to be the cause. They were investigated yesterday.

A key thread is drop in participation. That is the main thread we'll pull on. To do that we will...

Next steps

• (@masih) Get the participation monitoring in place: Add finality certificate capture to Observer #745
• (@masih) Setup mainnet Lotus node on k8s which always runs latest Lotus release (i.e., isn't being messed with - maybe only extra metrics, but not experimental code): https://github.com/filecoin-project/f3-passive-testing/issues/305
• (@rjan to setup) Get this of minerIds that aren't participating/sort by highest power (https://github.com/filecoin-project/f3-passive-testing/commit/07fdfa44f809ad5c67ddafc4e65c2178532903ae ), and then do manual reachouts to see about if/why they aren't participating. (This involves searching slack, datacap allocations, FF SP registry)

[masih]

As of 1824Z yesterday the participation in F3 observed by our test node increased by about 6%, to the level we observed during passive testing. This is great to see.

I am left to conclude that the root cause of unstable distance from head after activation was solely participation as we have not seen any evidence that suggests this could have been caused by the slight change in code path execution (which could not have been tested passively on mainnet).

Since the increase in participation yesterday, we are back to seeing -9 as the worst case distance from head, which is consistent with what we observed during extended passive testing on mainnet.

We continue to monitor the participation in F3 for a bit longer before closing this issue.