9 posts tagged with "Reth"

Lodestar publishes 1,175 metric names and 162 histograms on our fleet, 62% more metric names than any other consensus client we run and twenty-seven times Teku's histogram count. Part five ended by asking whether the richest instrument answers the question in this series' title better than the sparsest one did.

It does, and by a wider margin than we expected. Lodestar decomposes a block's journey into timers that add up: arrival, the wait on the execution layer, import, head. The pieces sum to the whole within 4 milliseconds. One bucket boundary sits at 4.000 seconds, so the share of blocks not processed by the attestation deadline is a count rather than an interpolation. It is 5.1% of blocks on the best execution client here and 9.0% on the worst, while only 0.11 to 0.16% of blocks arrive after 4 seconds at all.

Then the same instrument turned around and told us something about itself. Five execution clients time their own engine-API handlers, and four of them publish a mean we can subtract. Lodestar's wall clock around newPayload runs 181 to 208 milliseconds longer than what the execution client on the other end says it spent. The four other consensus clients that publish a mean sit 6 to 50 milliseconds above their peer. That gap is not a client comparison, it is a measurement about Lodestar, and it took a second stopwatch at the execution client's end to find it.

Teku is the only client in this series that answers the question in the title directly. It splits a block's journey into eleven named stages and counts every one of them, so instead of inferring where the slot goes you can read it off: arrival, gossip validation, pre-state retrieval, the engine call, the state transition, the commit.

In the same three days, on the same six machines, Teku's own counters reported that the healthiest node in the fleet was one that had not committed a single block to its canonical chain.

Both of those are worth your attention, and they are the same story. Teku gives you more slot detail than any other client here, and the detail is carried in metric types that will mislead you at least three separate ways if you read them the way their names suggest.

How much machine does an Ethereum execution client need once it sits at the tip of mainnet? Our fleet runs every major execution client on its own host, all with 12 cores, five of six client sets with 16 GiB RAM, one consensus client per pairing on a separate machine. Same chain, same engine-API traffic, one telemetry pipeline. Averaged over the same 36 hours, the synced clients hold between 4.9 and 15.6 GiB of host memory, burn about half a CPU core, and write to disk at anywhere from 0.9 to 22 MB/s. The spread is the story: the client you pick decides whether your box idles or works.

A Geth node and an Erigon node sit in the same rack, follow the same Ethereum mainnet, take the same engine-API traffic from their paired consensus clients. Today the Geth host's root filesystem reports ~1,688 GB used. The Erigon host reports ~509 GB. That is a 3.3× spread, on the same chain, with no archive mode involved. Both are full nodes, both synced to mainnet head.

Disk is only half of it. The same pruning that bounds those footprints also runs on the engine-API hot path, and on some consensus-client pairings it pushes engine_newPayload past the 4-second sync-committee deadline. This post covers both halves: how big each client gets and whether you can see it pruning, then what that pruning costs in latency once a node is at the tip.

Key findings:

• 3.3× footprint spread on the same chain (root-FS used per host): Erigon ~509 GB, Ethrex ~598 GB, Nethermind ~1,225 GB, Besu ~1,261 GB, Reth ~1,352 GB (not synced), Geth ~1,688 GB.
• Five of six ECs grew 15 to 19 GB in the 7-day window (~2–3 GB/day, normal block ingestion). The cumulative spread comes from retention policy, not the past week's growth.
• Each EC exposes a different pruning surface, and Ethrex exposes none. Erigon's domain pruner fires ~501 k blocks events per host per week; Nethermind's Hybrid pruner runs continuously at ~1,497 nodes/sec/host; Geth path-mode prunes implicitly; Ethrex has no pruning metric at all.
• Reth is still in initial staged sync: its checkpoint sits ~70 to 130 k blocks behind the tip and is not advancing, so it is excluded from the synced comparisons.
• From Nimbus, three of five synced ECs cross the 4-second deadline on P99 engine_newPayload (Nethermind 4.44 s, Geth 4.67 s, Ethrex ≥5.00 s), but the ranking flips by consensus client. It is a pairing property, not a fixed EC property.
• Nethermind's spikes trace to two deliberate v1.37.1 choices. nethermind_pruning_time spikes past 5 s, over the deadline, entirely from the in-memory pruner.
• Sync-committee duties get hit before attestations. A late sync-committee message misses the next block's sync_aggregate outright; a late attestation still has one slot of grace.

A Prysm node and a Lodestar node on the same chain, on identical hardware, both export beacon_reorgs_total. Over the last 90 days, the Prysm hosts in our Vienna NDC2 fleet incremented that counter 6,011 times. The Lodestar hosts incremented it zero times. Both numbers are correct readings of what each implementation chose to count.

This post is a 90-day reorg census across that fleet plus the smaller GCP comparator cohort: every consensus client (CC) paired with every execution client (EC), against the same Ethereum mainnet, with per-host normalization. The questions we answer: which Prometheus counter to trust for which question, why the same EC behind two different CCs produces very different reorg numbers, and why a "zero reorgs" reading on some clients is silence rather than safety.

Key findings at a glance:

• Prysm increments beacon_reorgs_total 8× more often than Lighthouse over 7 days. The gap shrinks to 1.6× over 90 days.
• Lodestar's beacon_reorgs_total is 0 for the entire 90-day window. Its decline-reason counter fires roughly 54 times per host per week.
• The same EC behind two CCs produces 2–5× different counts: Prysm + Besu reports 69 per host vs Prysm + Nethermind 280.
• Geth is the only EC in our fleet whose Prometheus reorg counter increments at all. Nethermind and Reth export the metric but it never increments; Besu, Erigon, and Ethrex don't export one at all.
• A single fixed beacon_reorgs_total alert threshold does not port between consensus clients. Re-baseline per CC × EC pair.

Every execution client connects to different peers, sees a different slice of the network, and churns through connections at wildly different rates. We queried Prometheus metrics and Elasticsearch container logs across our entire fleet to find out what each EC actually experiences at the P2P layer, and what it means for Ethereum's network health.

We measured gossip verification, KZG batch verification, data column reconstruction, and getBlobsV2 latency across 72 nodes running all six consensus clients. The verification speed spread between fastest and slowest client: 50x.

We deployed all 6 Ethereum execution clients from scratch on bare metal and tracked exactly how long each one took to fully sync. The results range from impressive to alarming.

We measured CPU, memory, disk I/O, and network across all 36 CC×EC pairings on our non-supernode fleet — here's what the Fusaka hardfork actually did to hardware consumption.