5 posts tagged with "Lodestar"

Over three days in August, Lodestar timed 21,530 newPayload calls to its Ethrex peer and reported a mean of 372.6 ms; Ethrex timed the same calls from its own end of the same socket and reported 164.8 ms. Both figures are right as far as the client reporting them is concerned, since the two stopwatches start and stop at different points, but the 208 ms between them is larger than most of the differences this series has spent six editions measuring between execution clients.

For six editions we compared execution clients through whichever consensus client happened to be watching them, and it took until the sixth to work out how much of what we were reading belonged to the watcher rather than the watched. Six consensus clients run against the same fleet here, no two of them measure a slot the same way, and one of them cannot produce a mean for the engine call at all.

Lodestar publishes 1,175 metric names and 162 histograms on our fleet, 62% more metric names than any other consensus client we run and twenty-seven times Teku's histogram count. Part five ended by asking whether the richest instrument answers the question in this series' title better than the sparsest one did.

It does, and by a wider margin than we expected. Lodestar decomposes a block's journey into timers that add up: arrival, the wait on the execution layer, import, head. The pieces sum to the whole within 4 milliseconds. One bucket boundary sits at 4.000 seconds, so the share of blocks not processed by the attestation deadline is a count rather than an interpolation. It is 5.1% of blocks on the best execution client here and 9.0% on the worst, while only 0.11 to 0.16% of blocks arrive after 4 seconds at all.

Then the same instrument turned around and told us something about itself. Five execution clients time their own engine-API handlers, and four of them publish a mean we can subtract. Lodestar's wall clock around newPayload runs 181 to 208 milliseconds longer than what the execution client on the other end says it spent. The four other consensus clients that publish a mean sit 6 to 50 milliseconds above their peer. That gap is not a client comparison, it is a measurement about Lodestar, and it took a second stopwatch at the execution client's end to find it.

A Prysm node and a Lodestar node on the same chain, on identical hardware, both export beacon_reorgs_total. Over the last 90 days, the Prysm hosts in our Vienna NDC2 fleet incremented that counter 6,011 times. The Lodestar hosts incremented it zero times. Both numbers are correct readings of what each implementation chose to count.

This post is a 90-day reorg census across that fleet plus the smaller GCP comparator cohort: every consensus client (CC) paired with every execution client (EC), against the same Ethereum mainnet, with per-host normalization. The questions we answer: which Prometheus counter to trust for which question, why the same EC behind two different CCs produces very different reorg numbers, and why a "zero reorgs" reading on some clients is silence rather than safety.

Key findings at a glance:

• Prysm increments beacon_reorgs_total 8× more often than Lighthouse over 7 days. The gap shrinks to 1.6× over 90 days.
• Lodestar's beacon_reorgs_total is 0 for the entire 90-day window. Its decline-reason counter fires roughly 54 times per host per week.
• The same EC behind two CCs produces 2–5× different counts: Prysm + Besu reports 69 per host vs Prysm + Nethermind 280.
• Geth is the only EC in our fleet whose Prometheus reorg counter increments at all. Nethermind and Reth export the metric but it never increments; Besu, Erigon, and Ethrex don't export one at all.
• A single fixed beacon_reorgs_total alert threshold does not port between consensus clients. Re-baseline per CC × EC pair.

We measured gossip verification, KZG batch verification, data column reconstruction, and getBlobsV2 latency across 72 nodes running all six consensus clients. The verification speed spread between fastest and slowest client: 50x.

We measured CPU, memory, disk I/O, and network across all 36 CC×EC pairings on our non-supernode fleet — here's what the Fusaka hardfork actually did to hardware consumption.