5 posts tagged with "Prysm"

Six Besu nodes, same version, same hardware, same config. Five held a flat JVM heap around 1.0 to 1.3 GB. The sixth climbed about 10 GB a day and was on track to be OOM-killed roughly 30 hours after a restart. The one thing different about it was the consensus client on the other side of the engine API.

This is a walkthrough of how StereumLabs AI, reading our fleet's metrics and logs, took that one anomalous node, traced it to a single method, and filed it upstream. Besu shipped a round of mitigations and closed the issue. A later devnet reproduction showed the underlying layers still pile up, the issue was reopened, and the fix that followed is now in review. The bug is operational: recoverable by a restart, no consensus impact, no double-sign, no state-root divergence. It is also the kind of cross-client interaction a single-node test will never surface, because it only appears when a live pairing lands in a specific state.

A Prysm node and a Lodestar node on the same chain, on identical hardware, both export beacon_reorgs_total. Over the last 90 days, the Prysm hosts in our Vienna NDC2 fleet incremented that counter 6,011 times. The Lodestar hosts incremented it zero times. Both numbers are correct readings of what each implementation chose to count.

This post is a 90-day reorg census across that fleet plus the smaller GCP comparator cohort: every consensus client (CC) paired with every execution client (EC), against the same Ethereum mainnet, with per-host normalization. The questions we answer: which Prometheus counter to trust for which question, why the same EC behind two different CCs produces very different reorg numbers, and why a "zero reorgs" reading on some clients is silence rather than safety.

Key findings at a glance:

• Prysm increments beacon_reorgs_total 8× more often than Lighthouse over 7 days. The gap shrinks to 1.6× over 90 days.
• Lodestar's beacon_reorgs_total is 0 for the entire 90-day window. Its decline-reason counter fires roughly 54 times per host per week.
• The same EC behind two CCs produces 2–5× different counts: Prysm + Besu reports 69 per host vs Prysm + Nethermind 280.
• Geth is the only EC in our fleet whose Prometheus reorg counter increments at all. Nethermind and Reth export the metric but it never increments; Besu, Erigon, and Ethrex don't export one at all.
• A single fixed beacon_reorgs_total alert threshold does not port between consensus clients. Re-baseline per CC × EC pair.

We measured gossip verification, KZG batch verification, data column reconstruction, and getBlobsV2 latency across 72 nodes running all six consensus clients. The verification speed spread between fastest and slowest client: 50x.

We measured CPU, memory, disk I/O, and network across all 36 CC×EC pairings on our non-supernode fleet — here's what the Fusaka hardfork actually did to hardware consumption.

Let's have a look at both versions of Prysm 7.1.1 and 7.1.2 and it's resource consumption