6 posts tagged with "Grandine"

We ran Optimum's mump2p gateway, version 1.1.1 at commit 66e391d, on 36 mainnet full nodes in a single datacenter for 11.5 days at the end of August, one gateway on every node, across the full six-by-six matrix of consensus against execution clients. Blocks arrived earlier on three of the six consensus clients. On a fourth about half the move is real, on a fifth the metric cannot say either way, and on the sixth the treated nodes got slower rather than faster. None of the numbers is statistically significant, and the reason is built into the design rather than into the data.

The apparent effect, before we looked hard, was 40 to 110 ms. We then spent a second pass attacking our own control group, because the treated nodes are bare metal in Vienna and the controls are GCP in Frankfurt, and a difference-in-differences absorbs a constant offset between two datacenters but never a drift. What survived that pass is smaller, is carried by fewer clients, and rests on a different kind of evidence than the control group provides. This post is about which parts held.

Over three days in August, Lodestar timed 21,530 newPayload calls to its Ethrex peer and reported a mean of 372.6 ms; Ethrex timed the same calls from its own end of the same socket and reported 164.8 ms. Both figures are right as far as the client reporting them is concerned, since the two stopwatches start and stop at different points, but the 208 ms between them is larger than most of the differences this series has spent six editions measuring between execution clients.

For six editions we compared execution clients through whichever consensus client happened to be watching them, and it took until the sixth to work out how much of what we were reading belonged to the watcher rather than the watched. Six consensus clients run against the same fleet here, no two of them measure a slot the same way, and one of them cannot produce a mean for the engine call at all.

This edition was nearly published with its central claim inverted. We ran three Prometheus metric-name searches against Grandine, across eleven substrings including payload, forkchoice, engine, execution, latency, delay and duration. The first returned nothing, the second two histograms and the third eleven, and not one of them was the engine-call timing we were looking for. The obvious conclusion was that Grandine is the sparse end of this series: a client that times block arrival and blob fetches and nothing else. A fact-check caught it before publication.

Grandine times both engine calls, per method, at more than one observation per slot. The metric is called ETH1_API_REQUEST_TIMES. It is in capitals, it contains none of the eleven substrings we searched for, and the engine method lives in a label rather than the name, so method="engine_newPayloadV4" sits inside a series whose name says only that some eth1 API was called. No name search reaches it.

Grandine is not the sparse end of this series, and it is not the rich end either: an inventory of its scrape job returns 70 histograms against Lodestar's 158 and Lighthouse's 143, and on total metric names it is the sparsest of the six at 330. The point is not where it ranks. The point is that a comparatively modest client still hid a per-method engine timer from three searches, which means the search was the problem, not the client.

What the metric shows once you find it: Besu's head-update call costs 162 ms against 4 to 7 ms for every other execution client, which is the fourth independent instrument to say so, and the middle of the execution-client field reorders again.

A Prysm node and a Lodestar node on the same chain, on identical hardware, both export beacon_reorgs_total. Over the last 90 days, the Prysm hosts in our Vienna NDC2 fleet incremented that counter 6,011 times. The Lodestar hosts incremented it zero times. Both numbers are correct readings of what each implementation chose to count.

This post is a 90-day reorg census across that fleet plus the smaller GCP comparator cohort: every consensus client (CC) paired with every execution client (EC), against the same Ethereum mainnet, with per-host normalization. The questions we answer: which Prometheus counter to trust for which question, why the same EC behind two different CCs produces very different reorg numbers, and why a "zero reorgs" reading on some clients is silence rather than safety.

Key findings at a glance:

• Prysm increments beacon_reorgs_total 8× more often than Lighthouse over 7 days. The gap shrinks to 1.6× over 90 days.
• Lodestar's beacon_reorgs_total is 0 for the entire 90-day window. Its decline-reason counter fires roughly 54 times per host per week.
• The same EC behind two CCs produces 2–5× different counts: Prysm + Besu reports 69 per host vs Prysm + Nethermind 280.
• Geth is the only EC in our fleet whose Prometheus reorg counter increments at all. Nethermind and Reth export the metric but it never increments; Besu, Erigon, and Ethrex don't export one at all.
• A single fixed beacon_reorgs_total alert threshold does not port between consensus clients. Re-baseline per CC × EC pair.

We measured gossip verification, KZG batch verification, data column reconstruction, and getBlobsV2 latency across 72 nodes running all six consensus clients. The verification speed spread between fastest and slowest client: 50x.

We measured CPU, memory, disk I/O, and network across all 36 CC×EC pairings on our non-supernode fleet — here's what the Fusaka hardfork actually did to hardware consumption.