In the same three days, on the same six machines, Teku's own counters reported that the healthiest node in the fleet was one that had not committed a single block to its canonical chain.

Both of those are worth your attention, and they are the same story. Teku gives you more slot detail than any other client here, and the detail is carried in metric types that will mislead you at least three separate ways if you read them the way their names suggest.

The eleven stages​

beacon_block_import_delay_counter_total carries a stage label with eleven values, and every stage fires once per imported block. Here is where a Geth-paired Teku spent its slots over 21,476 imports:

Two of those eleven are not stages in their own right: arrival is time from slot start until the block reached the node, which is network and gossip rather than processing, and total_processing_time is the sum of the rest. That leaves nine processing stages, and seven of them are close to noise. Five of the seven finish in under 50 milliseconds for better than 99% of blocks on every pairing. The other two miss that bar on the slower execution clients: completed runs from 99.31% under Geth down to 98.31% under Erigon, and begin_importing from 99.75% down to 98.86%. Pooled across the five pairings completed sits at 98.96%, so the 99% line does not hold fleet-wide for it. The shortfall is under a percentage point either way. Two stages carry the entire cost: processed, which sits at 100 to 250 ms for around 90% of blocks, and execution_payload_result_received, which is where the engine call is waited on.

That split is the useful part. processed barely moves across pairings, from 84.1% of blocks in the 100-to-250 ms bucket under Besu to 90.5% under Nethermind. Whatever it covers is work Teku does itself. execution_payload_result_received moves by a factor of nearly five. That is the execution client's contribution, isolated by the consensus client's own stopwatch.

One limit before the numbers: we have not verified from Teku's source what each stage boundary corresponds to internally. The stage names are Teku's, the ordering is Teku's, and the durations sum the way you would expect them to, but this is a relative comparison across pairings on one client's instrument rather than an audited decomposition.

What the engine wait costs, per execution client​

The two leftmost columns come from the block-import stage, the two rightmost from beacon_engine_requests_total{method="new_payloadV4"}. They are separate instruments inside the same process, and they agree at both extremes.

This is the fifth consensus client to put Ethrex at the fast end and Erigon at the slow end of newPayload, which is now the most heavily replicated result in this series. The middle of the field permutes again, and this time it permutes within a single instrument depending on which statistic you read. Ranked by the fastest bucket, Nethermind is second and Geth third. Ranked by the share of calls at a second or worse, Geth is second and Nethermind fourth. Nethermind is bimodal: a large fast mode paired with a fat tail. Geth has a slower mode and a thinner tail. Pick either column alone and you get a different middle.

Errors are rare and instructive: 40 failed newPayload calls across the five clients over three days, and every single one of them landed in the [5000,∞) bucket. Erigon 15, Nethermind 11, Besu 10, Ethrex 2, Geth 2. Nothing failed quickly. That pattern is what a fixed client-side timeout looks like, though we did not confirm Teku's timeout value or find an error-reason label, so treat the cause as inferred rather than measured.

Besu's head update, now on a fifth instrument​

forkchoiceUpdated again separates Besu from everyone else:

Four of five clients answer the head-update call in under 100 milliseconds better than 96% of the time. Besu does so 14.8% of the time, and its modal bucket is 100 to 300 ms, holding 77.5% of its calls.

Teku's fastest bucket is 100 ms wide, too coarse to resolve the peer values the earlier instruments reported for Besu: 4.3 to 7.4 ms under Grandine, 4.3 to 9.9 ms under Prysm, 5.8 to 14.5 ms under Lighthouse. Nimbus is the exception it was in part four, with peers running from 63 to 262 ms and two of them slower than its own Besu. So this instrument cannot reproduce the size of the gap. What it can do is place Besu, and only Besu, in a slower bucket than the rest of the field, which is consistent with the 162-to-200 ms figure the four earlier instruments measured directly. Five instruments have now timed this call and all five put Besu's head update above 100 ms. Four of them also put Besu behind every peer. The magnitude holds on five, the gap on four.

Worth repeating from part three: this cost is not attestation budget. forkchoiceUpdated does not sit between a block arriving and that block becoming attestable.

The counter that called a syncing node healthy​

Reth was excluded from parts one through four. Going into this edition its metrics said that was over.

Four first places and a tie at zero on the one counter whose name promises to catch this. By its consensus client's block-import and engine instrumentation, Reth was the healthiest and fastest execution client on the fleet.

Note the first row against the window's chain: Reth recorded 21,546 imports where the chain produced 21,531 blocks. The counter counts import events rather than canonical blocks, so it can exceed the block count, and on a node that is optimistically importing without validating there is nothing to pull it back down.

Here is Teku's log on that same host, at the same moment, and it wrote a variant of this line every twelve seconds:

2026-08-05 23:59:59.145 INFO - Syncing *** Slot: 14929198, Head slot: 14929197,
Waiting for execution layer sync, Connected peers: 100

That line appeared 21,600 times in 21,600 slots. Every slot in the window, without a single exception, on all three days. On the Geth pairing the corresponding line is Slot Event *** Slot: 14929198, Block: dcba1324..., Justified: 466536, Finalized: 466535, and Waiting for execution layer sync appears zero times.

Reth's own log agrees with Teku's log and disagrees with Teku's metrics:

2026-08-05T23:59:59.829405Z INFO reth_node_events::node: Committed stage progress
pipeline_stages=9/14 stage=MerkleExecute checkpoint=0 target=25588987 stage_progress=40.05%

It was running the staged-sync pipeline, which advanced from stage 4 of 14 at the start of the window to stage 9 of 14 at the end, against a pipeline target roughly 103,000 blocks behind the live chain head of 25,692,170. Canonical chain committed appears zero times in three days. The control that makes this airtight: the identical Reth build paired with Lighthouse, on the same fleet over the same window, logged that line 6,452 times.

The mechanism is optimistic sync, and it is doing what the specification asks of it. When the execution client cannot yet validate a payload, the consensus client may import the block optimistically and carry on. Teku's block-import counter records that as result="success", because from the consensus layer's point of view the import did succeed. The _syncing result value exists but stayed at zero, so it cannot be used as the guard either.

The timing consequence is the one that would have produced a false finding here. Reth answered 95.7% of payload requests in under 50 milliseconds, better than any synced client. An execution client that is not executing returns immediately. The fastest engine responses in the fleet came from the node doing none of the work, and every block-import and engine counter presented that as excellence.

The container log is how we caught it, and it is not the only thing that could have. Teku's beacon_node_syncing_active gauge read 1 on all 4,320 scrapes of the window on the Reth pairing, against at most 5 scrapes of 4,320 on any other pairing, and 0 on Geth and Besu. That gauge sits outside the eleven-stage instrument this whole post is built from, so a dashboard assembled around block import never shows it. What it will not do is name the cause: it also reached 1 on the Ethrex pairing, where Waiting for execution layer sync never appears in the log at all.

So the lesson is narrower than "metrics cannot tell you" and more useful. The gauge tells you the beacon node is off the tip. The log tells you the execution layer is why. Every counter that a slot-timing dashboard would plausibly carry rated a node that committed nothing the best in the fleet, and the one series that dissented is the one no block-import dashboard has on it.

Three ways the buckets will lie to you​

The instrument that produced everything above needs handling. All of these are properties of Teku 26.7.1 as deployed, and all of them are silent failures rather than errors.

One: there are three schemes, not one. arrival and total_processing_time use a ten-bucket coarse scheme running to [12000,∞). The other nine stages use a seven-bucket fine scheme running to [2000,∞). beacon_engine_requests_total uses a third, eight-bucket scheme running to [5000,∞). The single interval string common to all three is [500,1000), which makes it useless for telling them apart. There is one further wrinkle: at stage="total_processing_time" the scheme flips to fine when result="unknown_parent", so the scheme is a function of the stage and result pair rather than of either label alone.

Two: summing without a stage filter inflates elevenfold. Write sum by (interval) (beacon_block_import_delay_counter_total{...}) and you pool all eleven stages into one pseudo-histogram whose bins overlap. For the Geth pairing that totals 236,324 observations against 21,476 blocks. Because the fine scheme's [0,50) bin then holds 71.4% of everything, a median read off that chart lands around 25 ms. The true block-arrival median is in the 1,500-to-2,000 ms bucket. The chart is wrong by a factor of about seventy and looks entirely plausible.

Three: the top bucket is spelled with U+221E. It renders as [12000,∞), [2000,∞) and [5000,∞). A selector written interval="[12000,inf)" matches zero series and returns no error. Every slow block silently vanishes from the result, which presents as a clean tail.

One more trap, this one about method rather than about Teku. Establishing that these buckets are disjoint rather than cumulative cannot be done on one client. Reth's newPayload distribution falls monotonically across all eight bins, which is the exact signature a cumulative scheme produces. Tested on Reth alone the answer comes out backwards. It is Besu, Erigon and Geth, whose distributions rise to an interior mode before falling away, that settle it. Ethrex and Nethermind peak in the first bin like Reth, and give themselves away only later, with a step back up that no cumulative scheme can produce.

What to measure on your own Teku nodes​

• Alert on beacon_node_syncing_active, then read the logs to attribute the cause. Held at 1 for five minutes, this gauge fired on the Reth host and nowhere else in the window; there it read 1 at every scrape for three days, while every block-import and engine counter Teku publishes looked healthy or better than healthy. The gauge tells you the beacon node is behind, not why. For the why, grep Teku's log for Waiting for execution layer sync, then check that the execution client is committing blocks, which for Reth means Canonical chain committed. Run both checks rather than either one: in this window they disagreed in both directions, with the gauge reaching 1 on the Ethrex pairing where the Teku log line never appeared, and the log line appearing twice on the Besu pairing where the gauge never left 0.
• Always filter by stage before touching interval. Without it the counts are eleven times too large and the bins overlap. Add result="success" too unless you want unknown_parent injecting six foreign bucket strings into a ten-bucket histogram.
• Match the infinity bucket on ∞, not on inf. If a panel shows no blocks in the slowest bucket, check the selector before you believe it.
• Read processed and execution_payload_result_received separately. The first is Teku's own cost and barely moves; the second is the execution client's and moves by nearly five times. Charting only total_processing_time blends them.
• Do not expect quantiles. There is no le label, so histogram_quantile returns nothing. Bucket-bounded statements are the honest ceiling here: a modal bucket or a share above a threshold, not a median in milliseconds.

Coming next in the series​

Lodestar is the last client, and on surface area it is the opposite of Teku: 1,184 metric names and 162 genuine histograms against Teku's 410 and 6. Then the finale, where all six clients and their instruments go side by side, with an honest list of the claims from earlier editions that did not survive being measured a second way.

This edition adds one entry to that list, and it is ours. Between drafting and publishing, the Reth pairing was going to be reported as this series' first fully-synced Reth comparison on the strength of five metrics that all said so. The log said otherwise. Across five editions the pattern has held without exception: a client comparison is only as good as the instrument you read it through, and the failure mode is never an error message. It is a plausible number. Catching those is what StereumLabs AI does on our fleet, on the measurement stack we described here. If you run Ethereum infrastructure and want this lens on your own nodes, reach us at stereumlabs.com or contact@stereumlabs.com.

Methodology​

Numbers come from Teku's and the execution clients' own metrics on our NDC2 deployment (Vienna), queried on the Prometheus-cold datasource (uid aez9ck4wz05q8e), with the fleet labels documented in build your own dashboards. The window is 2026-08-03T00:00:00Z to 2026-08-06T00:00:00Z. Counts are exact counter deltas, M - M offset 3d evaluated at the closing anchor, rather than increase(...[3d]), which extrapolates and returns fractional values. The two agree to within 0.05%. NDC2 runs a six-by-six matrix of consensus and execution clients on identical 12-core bare-metal hosts, so each execution client appears six times, once per consensus-client pairing; every figure here is the Teku pairing specifically.

• Chain progress. The chain produced 21,531 blocks in the window's 21,600 slots, height 25,670,639 to 25,692,170. Four execution clients agree on both endpoints to within one block via chain_head_block (Geth), ethereum_blockchain_height (Nethermind, Besu) and head_height (Ethrex), the one-block spread being scrape timing rather than disagreement. Erigon publishes no head-height metric name at all; its head is the label value sync{stage="finish"}, which is a further instance of the part-four lesson that capability hides in labels.
• The stage decomposition is beacon_block_import_delay_counter_total{result="success"} grouped by stage and interval. Per-pairing import totals are Besu 21,465, Erigon 21,465, Ethrex 21,056, Geth 21,476, Nethermind 21,487. All eleven stages report identical counts within a pairing. The buckets are disjoint, established four ways: fine-scheme and coarse-scheme stages sum to bit-identical totals; those totals equal roughly one increment per block; a cumulative-from-below scheme would require the lowest bin to be the maximum, which fails for Besu, Geth and Erigon; and a cumulative le-style scheme would require the top bin to be the maximum, which fails everywhere. beacon_block_import_delay_latest also exists, per stage, but it is a gauge of the most recent block only.
• Engine-call figures are beacon_engine_requests_total grouped by method, interval and outcome, one uniform eight-bucket scheme across all five methods and both outcomes. Successful newPayload totals per pairing run from 21,668 to 22,088 against 21,531 blocks, slightly above one per block. The 40 errors quoted are newPayload only and exclude Reth's zero; Reth logged 11 engine errors on other methods, and Geth is the fleet minimum on engine errors overall with 4.
• Metric surface area per consensus client, anchor-evaluated as distinct metric names and _bucket metric names: Lodestar 1,184 / 162, Lighthouse 781 / 144, Prysm 590 / 65, Nimbus 531 / 25, Teku 410 / 6, Grandine 332 / 71. Teku is not the smallest surface; it is the histogram outlier.
• Sync was verified from container logs, not metrics. Geth logged Chain head was updated 21,555 times, from block 25,670,641 to 25,692,172. Nethermind logged Finished pre-warming caches for block 21,549 times, ending at 25,692,172. Besu logged new added block 25692172. Erigon logs a head update roughly every nine minutes rather than per block; its last one in the window reads head updated number=25692010 age=3s, and its Prometheus head stood at 25,692,169. Ethrex ran from block 25,670,642 to 25,692,172. Teku's Waiting for execution layer sync line appeared 0 times for Geth and Ethrex, 2 for Besu, 11 for Nethermind, 65 for Erigon and 21,600 for Reth. Teku's beacon_node_syncing_active gauge, scraped 4,320 times per host, summed to 4,320 on Reth, 5 on Erigon, 4 on Ethrex, 2 on Nethermind and 0 on Besu and Geth.
• Reth's exclusion. Canonical chain committed appears zero times on the Teku-paired Reth host across the window, against 6,452 occurrences on the Lighthouse-paired host running the same build over the same window. Its pipeline advanced from pipeline_stages=4/14 stage=Execution to pipeline_stages=9/14 stage=MerkleExecute with target 25,588,987. We did not confirm from Teku's source that optimistic import is what produces the success result on that pairing; the specification permits it, the log states the node was waiting on the execution layer, and the counter recorded successes, but the causal link is our reading rather than a measurement. Separately, reth_blockchain_tree_canonical_chain_height reads 0 on all six Reth hosts for the first two days of the window, so it is unusable as a sync indicator in either direction.
• The Ethrex shortfall. Ethrex imported 21,056 blocks against a chain that produced 21,531, and its own log shows 21,104 prewarm passes, so both instruments agree the pair observed roughly 450 fewer blocks. The deficit concentrates in three hours (109, 190 and 159 blocks against a normal 300 per hour) with no errors and no restart in the log. Teku's beacon_node_syncing_active on that host reached 1 in two of those three hours and in no other hour of the window, which narrows when the deficit accrued without establishing its cause. Ethrex was at the chain tip at both window ends. Every Ethrex percentage here therefore rests on a denominator about 2% smaller than its peers, which does not change its rank on any measure reported above but is worth knowing.
• Our fleet runs no live validators. It receives mirrored validator-client traffic, unchanged across this window.

Grandine times both engine calls, per method, at more than one observation per slot. The metric is called ETH1_API_REQUEST_TIMES. It is in capitals, it contains none of the eleven substrings we searched for, and the engine method lives in a label rather than the name, so method="engine_newPayloadV4" sits inside a series whose name says only that some eth1 API was called. No name search reaches it.

Grandine is not the sparse end of this series, and it is not the rich end either: an inventory of its scrape job returns 70 histograms against Lodestar's 158 and Lighthouse's 143, and on total metric names it is the sparsest of the six at 330. The point is not where it ranks. The point is that a comparatively modest client still hid a per-method engine timer from three searches, which means the search was the problem, not the client.

What the metric shows once you find it: Besu's head-update call costs 162 ms against 4 to 7 ms for every other execution client, which is the fourth independent instrument to say so, and the middle of the execution-client field reorders again.

The metric a keyword search cannot find​

Here is what the three failed searches looked for, against what the job publishes:

The second search is the instructive one. Searching for engine does return an engine-API histogram, beacon_engine_getBlobsV2_request_duration_seconds, which is how we came to believe Grandine timed the blob fetch and nothing else on that side. The third widens to eleven histograms, mostly PeerDAS and KZG timers, and still misses the payload call. Only the inventory finds it, because ETH1_API_REQUEST_TIMES shares no substring with any word a person would think to type.

It carries method="engine_newPayloadV4", "engine_forkchoiceUpdatedV3", "engine_getPayloadV5", "engine_getBlobsV2" and "engine_exchangeCapabilities". Alongside it sits MUTATOR_BLOCK_PROCESSING_TIMES, a per-slot consensus-side block-processing timer. Name searches can reach some of what a client publishes; what they cannot do is establish that something is absent.

The generalisable point for anyone auditing their own stack: a metric's name tells you what its author called it, not what it measures. Capability hides in two places a substring search does not reach, unconventional casing and label values, and both appear here at once. Inventory the scrape job, then read the labels.

What Grandine's engine timing shows​

With the metric located, Grandine answers the question this series has been asking of every client. Over the window, per pairing:

Two things carry over from earlier editions and one is new.

Carrying over: Ethrex fastest and Erigon slowest on newPayload, which now holds under four consensus clients. And the middle reorders again, with Nethermind ahead of Geth here where Prysm had Geth ahead of Nethermind, which is part three's finding that the middle of the field does not rank across pairings, arriving on schedule with a fourth instrument.

New, and worth its own section: Besu's head-update call.

Besu's head update, now on four instruments​

forkchoiceUpdated is the engine call almost no dashboard shows. Four consensus clients on our fleet time it, on four separate sets of execution-client hosts, and they agree:

All four rows are the same three days, 2026-07-27 to 07-30, so nothing here is being compared across windows.

Besu's own figure is the stable one: 162 to 200 ms on four instruments and four sets of hosts, at roughly 22,000 calls each. The gap to its peers is not equally stable. On three of the four instruments Besu costs 13 to 46 times what the other execution clients cost, and on Nimbus it does not, because under Nimbus both Ethrex and Nethermind also read in the hundreds of milliseconds, which no other instrument reproduces. So four instruments carry the magnitude and three carry the gap. Even at three, it is the most reproducible client-specific result in this series. The mechanism remains bounded but unproven, and the pruning census is where the plausible cause sits, since that is where Besu was found doing storage work on the engine-API hot path.

We also repeat the limit stated in part three: this cost is not attestation budget. forkchoiceUpdated does not sit between a block arriving and the block becoming attestable. It is engine-layer work your node pays for and your dashboard probably does not show you.

The arrival histogram, and the number we are not quoting​

Grandine's arrival histogram, beacon_block_gossip_slot_start_delay_time, recorded 21,596 to 21,717 observations per pairing against the 21,521 blocks these 21,600 slots produced. It counts gossip block receptions, not slots, so the surplus of 75 to 196 is a node hearing some blocks more than once and the total cannot tell you that every slot is represented. It also contains a trap worth more attention than the numbers it yields:

Between 0.51% and 0.88% of observations exceed the top finite bucket of 10 s. The histogram can only record them as "past 10 s"; sum/count weights them at their true magnitude, and each of them averages many minutes past slot start rather than seconds. That is enough to move the mean by an order of magnitude, and by different amounts per pairing, so a dashboard panel built on sum/count invents a 2.3x spread between execution clients out of nothing but how many stragglers each one saw.

What we will not do is turn the median column into a finding. Three quarters of the data, 74.9 to 75.5%, sits in a single bucket spanning 1 s to 2.5 s: 77.6 to 78.3% of observations fall at or below 2.5 s and only 2.7% below 1 s. So histogram_quantile is interpolating the median across that one bucket and the nine milliseconds separating the five pairings reflects bucket occupancy rather than arrival timing. This series has an arrival control built from Lighthouse, Nimbus and Prysm, whose buckets or gauges can carry it. Grandine's cannot, and saying so is more useful than a fourth data point that would not mean what the first three mean.

The driver effect: what replicates is a split, not an order​

Part three found that the same execution client, same version, on equally loaded identical machines, runs materially slower under some consensus clients than others, and left the mechanism open. We now have a second window and finer granularity, and the finding needs both a confirmation and a correction.

Confirmed: there are two tiers, and the gap between them holds. Split those twelve-hour sub-windows across both three-day windows and the boundary holds in 12 of 12: the slowest of Grandine, Teku and Lighthouse is below the fastest of Nimbus, Lodestar and Prysm every single time. Grandine is the fastest driver in all twelve.

Corrected: the six-way ordering in the table above is an artifact of aggregating three days. Teku and Lighthouse trade places in 2 of the 12 sub-windows, and so do Lodestar and Prysm, because inside each tier the clients sit within a few percent of each other. Part three read an ordering; what the data supports is a split, and we would have overstated it without checking the sub-windows. The magnitudes also fell unevenly between the two windows, by 1.3% for Nimbus and 16.5% for Lighthouse, so the earlier reading that "every value came down 10 to 20%" was too tidy as well. Whatever moved them all downward was not a change in what we ask of the fleet: the mirrored validator-client load it serves was unchanged across both windows.

Three limits on the split itself, because a tier boundary is the kind of finding that is easy to draw around whatever the data happened to do. We defined the boundary after seeing these twelve windows, so "holds in 12 of 12" is a description of the data it was drawn on and not an out-of-sample test. It is also resolution-dependent: at six-hour granularity the two tiers overlap in 2 of 24 sub-windows, once by 2 ms and once by 36 ms. And each driver here is one consensus client on one execution-client host, so a driver effect and a host effect cannot be separated at a 62 ms tier gap, which is why part three's host-equivalence checks matter and why we still call the mechanism open.

And the two execution clients disagree about the drivers​

Erigon publishes its own slot timeline, block_consumer_delay, with the moment it had the block body, execution start and execution end. Same version everywhere, same window:

Nethermind's fastest driver is Grandine and its slowest is Prysm. For Erigon the slowest is Lighthouse, and the fastest is Lodestar by 6 ms over Nimbus, which is thin enough that the top of this column is a tie rather than a winner: split it by day and Lodestar leads by 23 ms, then the two land within 0.3 ms, then Nimbus leads by 3.5 ms. What survives is the crossing itself. Grandine, first for Nethermind in every one of twelve sub-windows, sits fourth of six here on every one of the three days. Lodestar and Nimbus, fifth and fourth for Nethermind, occupy the top two places here on all three.

How strongly the two disagree depends on which reading of Erigon's timeline you take, and we checked two: on the execution span the rank correlation with Nethermind's order is -0.60, and on absolute execution-end time it is -0.14. Neither is significant at six drivers, and the range between them is the point. There is no consistent relationship, in either direction, between how a driver treats Nethermind and how it treats Erigon.

So part three's open question closes in the only direction the data supports. The driver matters, and it is not a property of the driver: it is specific to the pair. There is no gentle consensus client and no demanding one.

Two limits on this table. The Lighthouse row is reconstructed from the two days of the three that reported, because a single scrape returning no value propagates through avg_over_time and voids the whole window; on both reporting days it was the slowest of the six. And the span column subtracts two separately computed medians, which approximates the typical execution window rather than being the median of a duration. The three fastest spans sit within 42 ms of each other, so what we read from this table is that the two orderings cross, not the position of any single driver in it.

What to measure on your own Grandine nodes​

• Inventory before you conclude. count by (__name__) ({__name__=~".+_bucket", job="grandine"}) returns 70 histograms; drop the _bucket filter and you get all 330 metric names it publishes. The keyword searches that sent us wrong returned 0, 2 and 11. If you have ever concluded that a client does not expose something, that is the query to rerun.
• Both engine calls are there: ETH1_API_REQUEST_TIMES{method="engine_newPayloadV4"} and {method="engine_forkchoiceUpdatedV3"}. Watch the second one too; on one execution client it is the larger cost.
• Do not chart sum/count on either default-bucket histogram. On the arrival metric that reads ten times the median. Use histogram_quantile, and know that with 75% of the data in the single 1-to-2.5 s bucket the interpolated quantile is coarse.
• Neither histogram can answer the deadline question. No bucket boundary sits near 4 s. For a share-of-late-blocks figure you need a histogram bucketed with the deadline in mind, which on our fleet means Nimbus or Prysm.
• Do not carry a client comparison across pairings. The driver changes an execution client's measured speed by a wide margin, in tiers that hold and an order that does not, and differently for different execution clients.

Coming next in the series​

Teku and Lodestar remain, and both already appear as drivers above: Teku sits in the fast tier for Nethermind and is second-slowest for Erigon, Lodestar the reverse. Then the finale, where all six clients and all of their instruments go side by side, with an honest list of the claims from earlier editions that did not survive being measured a second way. This edition contributes two entries to that list, both ours: the reading of part three that the driver effect was an ordering, and the premise this post was drafted with.

The pattern in all four editions is that a client comparison is only as good as the instrument you read it through, and that the instrument is easy to misread in ways that look like client differences: a mean poisoned by stragglers, a quantile interpolated across an over-full bucket, an ordering that only exists at one aggregation level, a capability hidden behind a name. Catching those is what StereumLabs AI does on our fleet, on the measurement stack we described here, and this edition is a fair sample of how often it catches us. If you run Ethereum infrastructure and want this lens on your own nodes, reach us at stereumlabs.com or contact@stereumlabs.com.

Methodology​

Numbers come from Grandine's and the execution clients' own metrics on our NDC2 deployment (Vienna), queried on the Prometheus-cold datasource (uid aez9ck4wz05q8e), with the fleet labels documented in build your own dashboards. The window is 2026-07-27T00:00:00Z to 2026-07-30T00:00:00Z, with increase(...[3d]) and avg_over_time(...[3d]) evaluated at the closing anchor so figures are stable and reproducible rather than drifting with query time. No execution-client version changed inside it and Grandine held 2.0.5 throughout.

• How the instrumentation was established. count(count by (__name__) ({__name__=~".+_bucket", job="grandine", deployment="NDC2"})) returns 70; the same query without the _bucket filter returns 330 distinct metric names. Run per job across the six consensus clients, evaluated at the window's closing anchor rather than at query time, it gives 158 histograms for Lodestar, 143 for Lighthouse, 70 for Grandine, 65 for Prysm, 25 for Nimbus and 6 for Teku, and 1,160 / 778 / 330 / 591 / 535 / 411 total metric names respectively, which is the basis for saying Grandine is neither the sparse nor the rich end. These counts drift by a few series with query time, which is why we pin them to the anchor like every other figure here. Counts are of metric families exposed on this deployment and include non-timing histograms, so they are a rough measure of surface area rather than of timing coverage. The engine timing is ETH1_API_REQUEST_TIMES (_bucket, _count, _sum), seconds, with a method label taking engine_newPayloadV4, engine_forkchoiceUpdatedV3, engine_getPayloadV5, engine_getBlobsV2 and engine_exchangeCapabilities, six series per method, one per pairing. Finite bucket edges for both this and the arrival histogram are 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5 and 10 s. The three name searches that missed it are reproduced in the table above; none of them can match a name that is all-caps and carries its subject in a label.
• Engine-call figures. Means are increase(_sum[3d]) / increase(_count[3d]) per ec_client, which is exact. The newPayload 90th percentiles are histogram_quantile over windowed buckets and are interpolated inside the 0.25-to-1 s bucket for Geth, Nethermind and Besu, so treat them as approximate; Erigon's falls in the 1-to-2.5 s bucket. We do not quote quantiles for forkchoiceUpdated, whose 4-to-7 ms values sit below the second bucket edge. Call counts are 21,883 (Ethrex), 22,198 (Nethermind), 22,534 (Geth), 22,933 (Besu) and 23,981 (Erigon) against 21,600 slots carrying 21,521 blocks, so slightly more than one call per block.
• The four-instrument Besu comparison uses execution_layer_request_times{method="forkchoice_updated"} on Lighthouse v8.2.1, engine_api_request_duration_seconds{request="forkchoiceUpdated"} on Nimbus v26.7.0, forkchoice_updated_v1_latency_milliseconds on Prysm v7.1.7 and ETH1_API_REQUEST_TIMES{method="engine_forkchoiceUpdatedV3"} on Grandine 2.0.5, all as windowed means over this edition's window rather than part three's, so all four rows cover the same three days.
• The arrival histogram. Medians are histogram_quantile(0.5, ...); means are increase(_sum[3d]) / increase(_count[3d]). Share at or below the 10 s bucket: 99.124% Nethermind, 99.139% Geth, 99.156% Ethrex, 99.240% Besu, 99.486% Erigon. Share at or below 2.5 s: 77.58% Ethrex, 77.88% Besu, 77.96% Nethermind, 78.23% Geth, 78.29% Erigon, which is why the interpolated median is coarse and why we do not read a cross-client spread from it. Counts are 21,596 to 21,717 per pairing.
• The driver comparison uses avg_over_time(nethermind_new_payload_execution_time[3d]) for ec_version="1.39.2" grouped by cc_client, the execution client's own timer with no consensus client in the measurement. Sub-window stability is the same query at [12h] on a 12-hour step across both three-day windows, twelve points per driver. The tier boundary, defined as the maximum of Grandine, Teku and Lighthouse against the minimum of Nimbus, Lodestar and Prysm, holds in all twelve, and at six-hour granularity in 22 of 24. The boundary was chosen after inspecting these windows, so that is a description of this data rather than an out-of-sample test. Within-tier order changes in 2 of 12 sub-windows for Teku against Lighthouse and 2 of 12 for Lodestar against Prysm. Window-to-window change per driver is Nimbus -1.3%, Lodestar -7.4%, Grandine -9.9%, Prysm -10.1%, Teku -14.8%, Lighthouse -16.5%.
• Erigon's timeline is block_consumer_delay for ec_version="v3.5.2", a summary with client-computed quantiles; we take avg_over_time of the quantile="0.5" series for type="body_download", "pre_execution" and "post_execution", which is the average over the window of Erigon's own reported median rather than a median over the window. The Lighthouse pairing has one scrape with no value inside the window, which voids a three-day avg_over_time, so its row is the mean of the two daily values that did report, 1.835 and 1.907 s for body, 2.589 and 2.623 s for execution start, 3.377 and 3.368 s for execution end. Per-day spans for the other five are Lodestar 0.508/0.516/0.531, Nimbus 0.531/0.516/0.528, Prysm 0.565/0.556/0.559, Grandine 0.593/0.569/0.582 and Teku 0.699/0.688/0.709 s. Spearman rank correlation against Nethermind's driver order is -0.60 using the span and -0.14 using absolute execution-end time; with six drivers neither is significant, and we report the range rather than a single figure. We have not verified from Erigon's source what each type boundary corresponds to internally, so this table is a relative comparison across drivers on one metric rather than an absolute decomposition.
• MUTATOR_BLOCK_PROCESSING_TIMES, Grandine's per-slot block-processing timer, gives Geth 229, Besu 248, Erigon 254, Ethrex 332 and Nethermind 333 ms over the window. That ordering differs from the newPayload ordering on the same client, and because we cannot say from the metric alone which phases it spans, we report its existence and this disagreement as a caution rather than building a finding on it.
• Sync was verified from logs, not metrics. At both ends of the window each execution client's container log showed it at the chain tip, block 25,620,466 on 2026-07-27 and 25,641,987 on 2026-07-30, via Geth's Chain head was updated, Besu's block-add lines, Erigon's head updated, Nethermind's Received ForkChoice and Ethrex's Prewarm pass for block. Reth is excluded: no Canonical chain committed line appears in its log across the window, Prysm's new_payload_valid_node_count for the Reth pairing increases at zero per hour against about 300 for the other five, and Grandine issued 86,350 newPayload calls to it against about 22,000 for each of the others. On identical 12-core hosts; consensus and validator processes run on separate machines. Our fleet runs no live validators; it receives mirrored validator-client traffic, and that mirroring was unchanged across both windows, so it is not a candidate explanation for the between-window shift.

That triangulation is the finding, and it changes how we read our own earlier editions. Two results survive all three instruments: Ethrex is the fastest execution client on newPayload and Erigon the slowest, and Besu spends around 200 ms on forkchoiceUpdated while it is a single-digit or low-double-digit call for the rest. Almost nothing else does. The middle of the field, Geth against Nethermind against Besu, permutes depending on which client you ask. The reason is not the metric definition, and it is not the hardware either. At a fixed version, the same execution client's own internal timer varies by 160 ms depending on which consensus client is driving it, on machines whose load and database size are indistinguishable. That is larger than the margins we would be ranking on.

What Prysm measures​

Prysm times both engine calls at one entry per slot, with newPayload buckets running from 25 ms to 4,000 ms, which brackets the range these calls live in. Over the window:

*bucket-interpolated, since the top finite bucket is 4,000 ms.

On an average block even the slowest of these fits inside the budget: Erigon's 516 ms is about 13% of the 4-second attestation deadline against Ethrex's 4%. The tail is what bites, and at the 99th percentile every client is into the seconds, the same mechanism part one found. Stack a 3-second tail onto a 1.8-second arrival and the slot is gone.

Read on its own, that table looks like a ranking. The rest of this post is about why only its first and last rows are one.

Three instruments, one question​

Because all three clients publish a complete per-slot histogram of the same engine call, and because our fleet runs each consensus client against every execution client, the same question can be asked three times over the same three days on identical execution-client versions. The only thing that changes is which consensus client is holding the stopwatch and driving the execution client.

Ethrex is first in all three columns and Erigon last in all three. Those two results are worth trusting: three instruments, three host sets, about 21,500 observations each.

Everything between them moves. Lighthouse orders the middle Nethermind, Geth, Besu. Nimbus orders it Geth, Besu, Nethermind. Prysm orders it Geth, Nethermind, Besu. All three permutations appear, and no two agree.

Why the middle does not rank​

It would be convenient to blame the metric definitions, and we nearly did. It does not hold up, for two reasons we can measure.

The definitional effect is small and we can size it without leaving one host. Lighthouse publishes both quantities itself, on the same machine, in the same window: the gauge that brackets its execution step and the histogram of the full engine round trip. Going from one to the other costs 13 ms for Besu, 22 ms for Geth, 34 ms for Nethermind, 42 ms for Ethrex and 76 ms for Erigon. It reorders nothing: Nethermind stays 36 ms ahead of Geth on both of Lighthouse's own rulers, on all three days separately.

What does move is the pairing, and the execution clients measure it themselves. Nethermind 1.39.2 publishes its own internal payload-execution time, a number no consensus client is in the loop for. Across our six pairings, same version, same window:

One client, one version, one metric, no consensus client in the measurement, spanning 160 ms, or 80% of the fastest value.

The obvious suspect is the machine, and we checked it, because on this fleet it is checkable. The six hosts are identical 12-core bare metal in the same rack with sub-millisecond ping between them. Over this window they ran 5.2 to 6.7% CPU and 5.1 to 7.3% disk utilisation, so neither resource was anywhere near saturation on any of them. They read 1.8 to 2.1 MB/s from disk, held 8.9 to 9.9 GiB of page cache, and their databases are within 0.1% of the same size, 1.1506 against 1.1516 TiB. Nothing in that spread of load explains an 80% spread in execution time, and the ordering does not follow it: the Lighthouse-paired host is the second least busy and the third slowest.

So the number tracks which consensus client is asking, not the box it runs on. What we cannot do from timing data alone is say why. The candidate we would test next is handover timing against speculative execution: these clients pre-warm state from the mempool before the payload arrives, so a consensus client that hands the block over at a slightly different moment, or that pipelines newPayload differently against its blob and forkchoice traffic, leaves a different amount of that work already done. That is a hypothesis, not a result, and it is the kind of thing the next editions can test now that we know to look.

For anyone reading a client comparison, including this one, the practical consequence does not depend on the mechanism. Two execution clients are only comparable when the same consensus client drove both of them. That is why the table in the previous section is a fair ranking under Prysm and not a fair ranking full stop, and why differences of the size of Ethrex against Erigon, a factor of three, survive changing the driver while differences of 10 or 15% do not.

The engine call almost nobody looks at​

Every slot, a consensus client makes at least two engine calls: newPayload to hand over the block and forkchoiceUpdated to name the head. Dashboards, benchmarks and the first two editions of this series all watch the first one. The second is where the one client-specific anomaly on our fleet lives, and unlike the middle-of-field ordering, it survives triangulation:

Besu's row is the flat one: 197, 193, 208 ms, on three instruments and three pairings, at about 21,500 calls each. Under Prysm it is roughly 30x the next client and 443 ms at the 90th percentile against their 23 ms, and the shape of the distribution carries it rather than a few outliers: only 14% of Besu's calls come in under 100 ms, 58% under 200 ms, and just 0.5% exceed a second.

The other four rows are the cautionary tale in miniature. Nethermind's forkchoiceUpdated reads 6 ms under Lighthouse, 4 ms under Prysm and 255 ms under Nimbus. We are not going to pretend to explain that from timing data alone; the call counts are about one per slot under all three, so it is not a difference in how often the call is made. It is a reminder that a number this instrument-dependent is not a client property, and that Besu's stability across the same three instruments is what makes its 200 ms worth reporting.

Why Besu, we can bound but not settle. The same Besu is the one that in our pruning census does storage work on the engine-API hot path, and forkchoiceUpdated is where a client updates its canonical head, so head-update bookkeeping touching storage is the plausible mechanism. We have not lined up individual slow calls against Besu's storage activity, so this is a measured cost without a confirmed cause.

What the 200 ms does and does not cost​

It is tempting to add the two calls up and crown a winner. We did, in a draft of this post, and the fleet's own data talked us out of it. Summing the means is arithmetically fine, since both histograms carry the same one-per-slot denominator: Besu spends about 650 ms per slot across the two calls Prysm times against Erigon's 522 ms, so Besu has the larger engine-API bill.

But the bill is not the deadline. Prysm publishes chain_service_processing_milliseconds, its own total block-processing time, at the same one observation per block:

Every client sits 57 to 94 ms above its own newPayload mean, Besu included at +70 ms. The 208 ms head-update call is therefore measurably outside the region Prysm spends importing a block, which is both why it costs no attestation budget and why it does not make Besu the worst client for duty timing. On the quantity that governs when a block becomes attestable, the order does not reorder at all: Erigon is still last. Besu has the largest engine-API bill and the fourth-longest wait to attestability, and both statements are worth knowing.

One caveat on the off-path finding: our fleet does not propose blocks. A proposing node calls forkchoiceUpdated with payload attributes to start block assembly, and there a slow head update would sit on a path that matters.

Arrival, execution-client-independent for the third time​

Seventeen milliseconds separate fastest from slowest, on clients whose newPayload latencies span 345 ms. Arrival happens before the execution client is asked for anything, so it cannot carry the execution client's signature, and across three editions it never has. If your blocks arrive late, that is peering and proposers, and our peering deep dive is where to take it.

What to measure on your own nodes​

• Watch both engine calls. Whichever client you run, it publishes both: new_payload_v1_latency_milliseconds and forkchoice_updated_v1_latency_milliseconds on Prysm, execution_layer_request_times{method=...} on Lighthouse, engine_api_request_duration_seconds{request=...} on Nimbus. Watching only newPayload hides a call that on one client here costs 200 ms.
• Watch the tail, not the mean. Every client in this comparison is comfortable on an average block and into the seconds at the 99th percentile.
• Compare execution clients under one consensus client, or not at all. A 160 ms spread at fixed version across our own pairings is the reason we will not rank the middle of the field. If you are choosing between two clients whose published numbers are within 10 or 15%, ask what consensus client produced them, and rerun the comparison on your own stack before acting on it.
• Separate the bill from the deadline. Engine-API wall time and time-to-attestable are different quantities that rank clients differently. Pair the engine histograms with your client's block-processing metric, and with attestation inclusion distance from rated.network or beaconcha.in as the outcome.
• Do not read attestation_inclusion_delay_slots as your own duty performance. It came out identical across all five pairings because it describes the network, not this node.

Coming next in the series​

Three consensus clients down, and the useful surprise is that they overlap more than we assumed: all three time the engine API completely, which turns the series from a tour of instruments into a way to cross-check results. Teku, Lodestar and Grandine are next, and all three already appear as drivers in the table above. What we will be asking of them is not only what they instrument, but which of our findings survive being measured somewhere else.

Then the finale, which is the one we are now looking forward to: six consensus clients, six execution clients, every instrument against every other, and an honest accounting of which claims from the earlier editions of this series hold up under that comparison and which do not. This edition already found two that need revisiting. We would rather publish that list than quietly not mention it.

Holding hardware and consensus client fixed while varying the execution client is the cut that makes execution cost legible, and cross-checking the same question against a second and third instrument is what tells you which parts of it were about the client at all. That work, including catching a node that only looks synced and catching our own overreach, is what StereumLabs AI does on our fleet, on the measurement stack we described here. The same data supports doing it yourself on matched versions and identical hardware. If you run Ethereum infrastructure and want this lens on your own nodes, reach us at stereumlabs.com or contact@stereumlabs.com.

Methodology​

Numbers come from the three consensus clients' engine-API and block-timing histograms on our NDC2 deployment (Vienna), queried on the Prometheus-cold datasource (uid aez9ck4wz05q8e), with the fleet labels documented in build your own dashboards. The window is the three days from 2026-07-24T00:00:00Z to 2026-07-27T00:00:00Z, with increase(...[3d]) evaluated at the closing anchor so figures are stable and reproducible rather than drifting with query time. The window sits inside a period when no execution-client version changed on these pairings; the last bump was on 2026-07-22.

• Prysm's metrics. new_payload_v1_latency_milliseconds and forkchoice_updated_v1_latency_milliseconds, histograms in milliseconds with finite buckets at 25, 50, 100, 200, 500, 1000, 2000 and 4000 ms; block_arrival_latency_milliseconds with finite buckets at 100 to 24000 ms; chain_service_processing_milliseconds as a summary. Values are for cc_client="prysm-super", cc_version="v7.1.7", deployment="NDC2" grouped by ec_client. Means are increase(_sum[3d]) / increase(_count[3d]), which is exact. Percentiles are histogram_quantile over windowed buckets and are bucket-interpolated; the 99th percentiles land in the 2000-to-4000 ms band and are approximate.
• The three-instrument comparison uses execution_layer_request_times{method="new_payload"} on cc_client="lighthouse-super" (v8.2.1), engine_api_request_duration_seconds{request="newPayload"} on cc_client="nimbus-super" (v26.7.0), and Prysm's new_payload_v1_latency_milliseconds, all in the same window for the same execution-client versions. The Lighthouse and Nimbus metrics are in seconds and converted. Coverage is comparable: increase(_count[3d]) gives about 21,571 (Lighthouse), 21,577 (Nimbus) and 21,565 (Prysm) per pairing against 21,600 slots. The same three metrics with the forkchoice_updated, forkchoiceUpdated and forkchoice_updated_v1 selectors give the second table, at about 22,266, 21,577 and 21,562 calls respectively.
• The definitional effect is measured within Lighthouse on one host: avg_over_time(beacon_block_delay_execution_time[3d]) against the mean of execution_layer_request_times{method="new_payload"}, giving Besu 390 to 403 ms, Geth 289 to 311, Nethermind 241 to 275, Ethrex 97 to 139 and Erigon 427 to 503. No consensus client can observe an execution client's internal work, since the engine API returns no timing, so both of these are consensus-side wall clocks that differ in how much of the call they bracket. An earlier draft of this post described the Lighthouse gauge as an internal execution timer and attributed the Geth-Nethermind ordering difference to that distinction. Both were wrong, and the two paragraphs below are why.
• Pairing variance is measured with the execution clients' own timers, which no consensus client is in the loop for. avg_over_time(nethermind_new_payload_execution_time[3d]) for ec_version="1.39.2" across the six pairings gives 195, 239, 263, 311, 341 and 355 ms, for the Grandine, Teku, Lighthouse, Nimbus, Lodestar and Prysm pairings respectively, on 4,320 samples each. The Lighthouse-to-Prysm difference of 92 ms is the dominant term in what an earlier draft of this post read as a measurement-definition effect.
• Ruling out the machine. Over the same window, on the same six execution-client hosts, node_cpu_seconds_total gives 5.2 to 6.7% busy, node_disk_io_time_seconds_total gives 5.1 to 7.3% utilisation, node_disk_read_bytes_total gives 1.83 to 2.13 MB/s, node_memory_Cached_bytes gives 8.9 to 9.9 GiB, and used filesystem bytes span 1.1506 to 1.1516 TiB, a 0.1% difference. All six are identical 12-core bare metal in one rack with sub-millisecond ping. Neither CPU nor disk was near saturation on any host, the load ordering does not match the latency ordering, and the database sizes are equal, so we attribute the spread to the pairing rather than to the hardware and leave the mechanism open.
• On resolvability. Within a single instrument the differences we report are resolvable: Lighthouse's gauge holds 4,320 samples per pairing over the window with standard deviations near 494 ms (Geth) and 352 ms (Nethermind), and Prysm's histogram is a complete per-slot census. Those samples are 60 seconds apart and serially correlated, so a naive independent-samples standard error understates the uncertainty. We do not rest anything on it: the cross-instrument comparisons in this post are limited by between-pairing variance of 92 to 160 ms, which is larger than the effects a significance test would be resolving, so we report the extremes and the reproducible Besu result and decline the middle.
• Besu's forkchoiceUpdated distribution under Prysm, from the raw buckets: about 14% of calls under 100 ms, 58% under 200 ms, 97.6% under 500 ms and 0.5% over 1 s, with an interpolated median near 183 ms. The four other clients' p90 values of 22.8 to 23.2 ms are first-bucket interpolation artifacts, since 97 to 99% of their calls land under the 25 ms bucket edge, and should not be read as those clients agreeing with each other.
• The engine-API bill against the deadline. chain_service_processing_milliseconds on Prysm, one observation per block like the two engine histograms, gives Ethrex 265, Geth 415, Nethermind 445, Besu 512 and Erigon 573 ms, so each client sits 57 to 94 ms above its own newPayload mean and the 208 ms forkchoiceUpdated is outside that region. Summing the two engine means is valid as an expected per-slot total because both denominators are the same, but it covers only the two calls these clients time per block, not all engine traffic. Our fleet does not propose blocks, so the payload-attribute variant of forkchoiceUpdated is not exercised here.
• attestation_inclusion_delay_slots returned 3.586 slots for all five synced pairings, identical to three decimals, the signature of a network-wide quantity rather than a per-node one. Our fleet runs no live validators, so no metric here depends on attestation signing.
• Sync was verified from logs, not metrics. At both ends of the window each execution client's own container log showed it at the chain tip: Geth's Chain head was updated number 25,598,917 and 25,620,466, Erigon's head updated, Besu's block-add lines, Nethermind's Received ForkChoice: 25598917, and Ethrex's Prewarm pass for block 25598918. Reth is excluded on three independent grounds: zero VALID payload responses against 40,556 optimistic ones on this pairing, reth_sync_checkpoint{stage="Execution"} at 16,340,084 against a chain past 25.6M, and no Canonical chain committed line in the three-day window. Its Received new payload lines do reach tip block numbers, which is the trap: receiving a payload is not importing it. On identical 12-core hosts; consensus and validator processes run on separate machines.

That is not a gap in our data. It is what Nimbus exposes. Where Lighthouse breaks the path to attestable into arrival, consensus verification and execution verification, Nimbus publishes a single histogram of block-arrival delay. The good news is that this histogram is complete, counting every block, not the once-a-minute sample Lighthouse's gauges gave us. The catch is that it sees only the network-and-proposer part of the slot, so the 3.7x spread that mattered under Lighthouse is simply not in the data Nimbus reports.

Two instruments, two views of the same slot​

The slot is the same one Lighthouse showed us: a block is proposed elsewhere, gossiped across the network, observed locally somewhere around 2 seconds in, then verified by the consensus and execution layers before it can be attested to, all against the 4-second line. What changes between consensus clients is which parts of that path they put a number on.

Lighthouse gives the granular gauges (beacon_block_delay_observed_slot_start, _consensus_verification_time, _execution_time, _attestable_slot_start) plus the head_slot_start_exceeded counter, sampled about once a minute. Nimbus gives one histogram, beacon_block_delay, of arrival delay, complete to the block. Neither is the full picture. Lighthouse shows you the execution slice but only in samples; Nimbus shows you arrival in full but stops there. The consensus client you run decides which half of the trade you get.

What Nimbus shows well: the arrival distribution​

For the part it does measure, Nimbus measures it completely. Where Lighthouse sampled its block-delay gauges about once a minute, Nimbus's histogram counts every block it times, so the mean and the count past 4 seconds are exact rather than the conservative floors that sampling forced on us in the first edition. Over the window, per pairing:

The shape is healthy and close for everyone: blocks arrive around 2 seconds in on the mean, the 90th percentile sits near 3.5 seconds, and over three days each pairing logged 37 to 101 blocks past the 4-second line, out of twelve to nineteen thousand timed, between about a third and a half of a percent. (A note on the percentiles: because well under 1% of blocks cross 4 seconds and the bucket below it is 2 seconds wide, an interpolated 99th percentile sits just under 4 seconds for every client, an artifact of bucket width, not a per-client reading. The mean and the past-4-second count are exact.) Those late-arriving blocks are a network-and-proposer story, not an execution-client one: a block that shows up at 4.5 seconds was slow to reach the node, and no execution layer makes it earlier.

The blind spot: arrival cannot rank execution clients​

Look down the table again. The five clients run from 1.94 to 2.05 seconds on the mean and from 37 to 101 blocks past 4 seconds, and that order does not match the execution ranking. The clearest tell is Erigon. Under Lighthouse, on the same NDC2 fleet and the same five execution clients (at slightly different versions, spelled out in the methodology), Erigon was the worst by a wide margin: the disk-bound newPayload tail that pushed its would-fail rate to 3.7x Geth's. Under Nimbus, Erigon has among the fewest late arrivals on the fleet, 40 in three days, second only to Besu. The execution outlier is unremarkable here, and the client with the most late arrivals, Nethermind, was middle of the pack on execution.

That is the whole point. If Nimbus's histogram were picking up the execution cost, Erigon's heavy newPayload tail would put it near the top here, the way it topped the would-fail count under Lighthouse. Instead it sits near the bottom. There is a genuine spread in the arrival numbers, but it is a host-and-network spread, not an execution one, and the giveaway is that it is not stable: in the previous deployment of this fleet the same metric put Besu at the top of the late-arrival count, and here Besu is at the bottom. That pairing even moved to a different Nimbus host between the two deployments, and a patch bump to Besu cannot change when Nimbus first hears a block over gossip, so what moved the number is the machine and where it sits in the network, not the execution client. Arrival is decided by a host's network position, peer set and local load, before the execution client is asked to verify anything, so it cannot carry the execution client's signature.

So an operator running Nimbus who asks "which execution client is eating my attestation timing" gets no answer from Nimbus. The histogram will show all of them arriving around 2 seconds and will look reassuring, while a 3.7x difference in how often the timing budget blows sits entirely in the slice Nimbus does not report.

Where the execution slice went, and where to find it​

Nimbus has not hidden the execution time out of carelessness; it records block delay at arrival and leaves engine-API timing to the execution layer. So the number you want is on the execution-layer side of the engine API, and the coverage there is itself uneven. On our fleet, Nethermind exposes nethermind_new_payload_execution_time and Reth exposes reth_consensus_engine_beacon_new_payload_latency, both timing the engine call directly. The others surface per-block execution time mainly in their logs rather than as a clean metric: Geth's import lines carry an elapsed field, Ethrex's [METRIC] BLOCK entries a per-block time. There is no single cross-client newPayload panel to read; you assemble it from a different source per client, which is the same instrumentation-coverage gap one layer down, and the cut Lighthouse handed us for free by timing newPayload itself.

Some consensus clients do time the engine call themselves. Prysm, for one, publishes a per-execution-client newPayload latency histogram, the view Nimbus does not give, which a later edition will use. Nimbus does publish a second histogram, validator_monitor_beacon_block_delay_seconds, which is populated on our fleet, but it tracks block delay for monitored validators, an arrival-side measurement again, not the engine call, so it does not rescue the blind spot.

What to measure on your own Nimbus nodes​

• Use the arrival histogram for what it is good at: proposer and network health. beacon_block_delay gives you a complete distribution of how late blocks reach this node. A drifting mean or a fattening 90th percentile is a peering or connectivity signal, and it is comparable across your pairings because arrival does not depend on the execution client. Our peering deep dive is where to take that.
• Do not read the per-execution-client arrival spread as an execution-client ranking. As the Erigon inversion shows, the spread is host and network position, and it reshuffles when hosts change. If you switch execution client to chase a 0.1-second arrival difference, you will not get it, because you are not changing the thing that sets arrival.
• For the execution slice, instrument the execution layer directly. Pull your execution client's own newPayload or block-processing timing and watch its tail, not its average. That is the lever the Lighthouse edition found, and on Nimbus it is the only place that lever is visible.
• Watch the outcome, not only the inputs. Attestation inclusion distance and attestation effectiveness, which most operators already track through rated.network or beaconcha.in, are what a blown timing budget shows up as. If those stay clean, a slightly fat arrival tail or a slow newPayload is not yet costing you duties; if they slip, the arrival histogram and the execution-layer timing together tell you which half of the slot to chase.

Coming next in the series​

Two consensus clients down, and they already disagree on what a slot timing metric even is: Lighthouse times the execution slice but only samples it, Nimbus measures arrival in full but stops there. The next editions take the same question to Teku, Prysm, Lodestar and Grandine, and the pattern we are testing is that each one shows a different slice of the same slot. Where a client's instrumentation hides something its neighbours expose, that is the finding, and we will say so.

The comparison holds the hardware and the consensus client fixed and varies only the execution client. Lining clients up on equal footing across windows of uneven data quality, and catching when a node only looks synced, is an analysis in its own right rather than a lookup on a static panel, and it is what StereumLabs AI does on our fleet, on the measurement stack we described here. The same data lets you do this yourself: line specific clients up on matched versions and identical hardware, with StereumLabs AI as the analyst, instead of eyeballing two editions run weeks apart. If you run Ethereum infrastructure and want this lens on your own nodes, reach us at stereumlabs.com or contact@stereumlabs.com.

Methodology​

Numbers come from Nimbus's beacon_block_delay histogram on our NDC2 deployment (Vienna), queried on the Prometheus-cold datasource (uid aez9ck4wz05q8e), with the fleet labels documented in build your own dashboards. Per-client values are for cc_client="nimbus-super", cc_version="multiarch-v26.6.0", deployment="NDC2" grouped by ec_client, over the three days from 2026-06-27T00:00:00Z to 2026-06-30T00:00:00Z, with increase(...[3d]) evaluated at the closing anchor so the counts are stable and reproducible rather than drifting with query time. This is the current deployment, which by this window had been running about six days, long enough for every mainstream execution client to be synced and steady throughout.

• Versions differ between the two editions, and we state it. The Lighthouse edition's window ran Besu 26.6.0, Erigon v3.4.3 and Ethrex 16.0.0; this Nimbus window runs Besu 26.6.1, Erigon v3.4.4 and Ethrex 17.0.0, with Geth v1.17.3 and Nethermind 1.38.1 identical in both (all confirmed from the datasource). Arrival is consensus-side and fixed before the execution client runs, so these differences do not move the arrival numbers here; they would matter for a direct execution-time comparison. Comparing specific execution clients head-to-head therefore wants matched versions on identical hardware, which is the version-pinned alignment StereumLabs and StereumLabs AI do across windows of version churn, and which you can run on your own pairings with the same data.
• The metric is Nimbus's beacon_block_delay histogram (_bucket/_sum/_count), with buckets at 2, 4, 6, 8, 10, 12 and 14 seconds, in seconds. Nimbus records this delay when it first receives a block over gossip, before it hands the block to the execution layer to verify, so it is a block-arrival measurement. The cross-check that it is arrival and not a post-execution timestamp: its magnitude (around 2 s mean here) is in the range of Lighthouse's observed-arrival gauge on the same fleet, sits well below the execution-inclusive time-to-attestable, and does not move with execution-client speed.
• The maths. The mean is increase(_sum[3d]) / increase(_count[3d]); the count past 4 seconds is increase(_bucket{le="+Inf"}[3d]) - increase(_bucket{le="4.0"}[3d]), exact because 4 seconds is a bucket boundary; quantiles are histogram_quantile over the same windowed buckets. Because the buckets are 2 seconds wide, the median and the 99th percentile are bucket-interpolated and approximate; the mean and the past-4-seconds count are not.
• What the histogram counts, and why the per-client tail is not an execution ranking. Nimbus records this delay for blocks it received and timed on gossip, so the per-pairing counts (around 12,000 to 19,000 over roughly 21,600 slots) are a subset of all slots, and the share past 4 seconds is conditional on a block being timed rather than a per-slot miss rate. The per-client tail does show a spread (37 to 101 blocks past 4 seconds), but it tracks the host, not the execution client: it does not match the execution ranking, and the order is not stable across deployments. In the previous deployment of this fleet the same metric put Besu at the top of the late-arrival count; here Besu is at the bottom. That reshuffle is the signature of network position, not a client property.
• Arrival is execution-client-independent, and we checked it both ways. The mean spans only 1.94 to 2.05 s across the five synced pairings, and the ordering does not match the execution ranking we measured under Lighthouse: Erigon, the Lighthouse would-fail outlier at 3.7x, has among the fewest late arrivals here. That non-correlation, together with the cross-deployment reshuffle, is the evidence that the spread is host and network position, not the execution client.
• Sync was verified from logs, not metrics. For each pairing we read the execution client's own container logs across the window and confirmed it was importing at the chain tip: Geth's Imported new potential chain segment and Chain head was updated at the tip block, Besu's canonical block updates, Erigon's head validated and Forkchoice Commit with initialCycle false, Ethrex's [METRIC] BLOCK lines, Nethermind's Valid. Result of New Block at the tip. Reth was still in staged sync (Executing stage 4/14 after six days on this deployment), so it is excluded; its Received new payload lines reach the tip number but only record that the consensus client offered the head, not that Reth had imported it. On identical 12-core hosts; consensus and validator processes run on separate machines.
• Nimbus exposes no execution-side block-delay metric. The Lighthouse gauges beacon_block_delay_execution_time, _consensus_verification_time, _attestable_slot_start and the counter _head_slot_start_exceeded_total return no series for cc_client="nimbus-super". This is the basis for the claim that Nimbus cannot rank execution clients on timing from its own metrics; the execution slice has to come from the execution client's own newPayload instrumentation.

This is the first of a series. We run all six execution clients paired with all six consensus clients on identical bare metal, and each consensus client reports slot timing differently. We start with Lighthouse because it instruments the block-delay breakdown more completely than any other CC on the fleet. Later editions take the same question to Prysm, Teku, Nimbus, Lodestar and Grandine.

The 4-second deadline​

Attestations for a slot are due one-third of the way through it, at the 4-second mark, and the sync-committee message rides the same clock. Cross that line and the head vote is late: you lose the timely-head reward, and if late production then misses the inclusion windows for the source and target flags, those erode too, down to zero for the slot in the worst case. The attestation can still be included late, so this is a reward-erosion deadline, not a hard cutoff. So the question for an operator is not "did my node import the block" but "did it import the block, verify it, and make it attestable before 4 seconds were up." Everything below is measured against that line.

Where the 12 seconds go​

A block does not exist at your node at slot start. It is proposed elsewhere, gossiped across the network, and only then observed locally. Lighthouse records each step. Averaged over the window, the path to attestable looks like this:

The arithmetic is close to additive: time-to-attestable is roughly arrival plus consensus verification plus execution-layer verification. For Geth that is 1.87 + 0.17 + 0.20 = 2.24 s, against a measured attestable time of 2.28 s; for Erigon, 1.66 + 0.17 + 0.46 = 2.29 s against 2.31 s. The decomposition holds for every client, with a 30-to-40 ms residual for the availability and import steps Lighthouse times separately. The pieces:

Two of those three rows are effectively fixed. Arrival is a network property: it ranges only from 1.66 to 1.87 s across the six pairings, and it does not track execution-client speed (Erigon, the slowest EC here, sees the earliest arrivals). Consensus verification is the same Lighthouse code in every pairing, around 165 ms throughout. The one row you choose is execution.

Execution is your lever: newPayload latency by client​

On a normal block the execution slice is small, 100 to 460 ms, comfortably inside the budget. Arrival is not your problem; this slice is. The risk lives in the tail, and the proof of the tail is the complete counter below, not the sampled magnitudes. For a sense of size, execution-layer verification on a single block reaches a sampled p99 of about 1.2 s on Geth, 1.5 s on Besu, and 2.1 s on Erigon (a once-a-minute sample, so a conservative floor). The mechanism is plain: a slow newPayload tail landing on a block that already arrived late is what tips an otherwise marginal slot past 4 seconds. Stack a 2.1 s Erigon execution tail onto a 1.7 s arrival and you are near 3.8 s with consensus and import still to come.

It shows up in Lighthouse's own counter. beacon_block_delay_head_slot_start_exceeded_total, which Lighthouse describes as firing when "the duration between the start of the block's slot and the current time will result in failed attestations," counts the slots where the node landed too late to attest cleanly:

*p99 of the once-a-minute samples, a conservative floor. The tail claim rests on the complete would-fail counter, not this column.

The extremes line up: the fastest-tail clients (Ethrex, Geth) sit lowest at around 150 to 170 a day, and Erigon, with the heaviest tail, sits at 618, about 3.7x as often, on the same hardware behind the same Lighthouse with the same arrival times. The four mainstream clients span 150 to 228 a day, about 1.5x; Erigon's disk-bound tail is what stretches that to 3.7x. In slot terms, 618 a day is about 8.6% of slots flagged on Erigon against 2.3% on Geth. The middle is not strictly ordered by the average, and that is the point: Nethermind's average execution is below Geth's, yet it lands late more often. The mean is comfortable for every client; the tail is what sets the miss rate, which is why the p99 column matters more than the average. The control holds at the tail too: arrival's p99 is 3.3 to 3.5 s across all six pairings and does not order by execution-client speed (Geth's arrival tail is the highest, Erigon's among the lowest), so the would-fail gap is the execution layer's to own.

Why Erigon has the highest newPayload latency tail​

This is the same Erigon that, in our pruning census, runs its pruning on the engine-API hot path, and the same Erigon that in the hardware-footprint census sustains the fleet's highest disk writes (about 1.9 TB a day). Heavy, continuous disk work is the most likely driver of the tail of a newPayload call: most are fast, but the ones that land during a flush or compaction are slow, and those are the blocks that miss. We have not yet lined up individual slow-newPayload timestamps against Erigon's compaction windows; a later edition will. The trade Erigon makes for the smallest disk footprint on the fleet shows up again here, on the timing budget.

To be fair across the board: on a typical block every one of these clients is comfortably attestable around 2 seconds, half the budget to spare. None of them is "too slow" in the average case. The difference is entirely in how often the tail bites, and that is what the per-day counter captures. Besu sits one step down in the same regime: the second-heaviest tail (p99 ~1.5 s, 297 ms average) for the same disk reason, just biting less often than Erigon's. It is a gradient, not one villain.

The honest version of "blame your EL"​

Look at a single late block and the execution client usually looks innocent. In Lighthouse's DEBUG log, a delayed head block typically reads like observed_delay_ms: 7194, consensus_time_ms: 236, execution_time_ms: 80: the block showed up 7 seconds into the slot, and the local node handled it in a few hundred milliseconds. The lateness was the proposer's or the network's, and no execution client would have saved that slot.

So why does the execution client move the daily count by 3.7x? Because it is not about the catastrophically late blocks, it is about the marginal ones. Across thousands of slots, a meaningful fraction arrive late enough that a few hundred extra milliseconds of execution is the difference between attestable-in-time and not. A faster execution layer does not fix a 7-second-late block, but it rescues the slots that were close. That is the lever you hold.

What to measure on your own nodes​

• Pull beacon_block_delay_observed_slot_start per pairing. If your blocks arrive late on average, that is a peering or proposer-connectivity problem, not an execution-client one, and changing EC will not help. Our peering deep dive is the place to start on that.
• Pull beacon_block_delay_execution_time and watch its p99, not its average. The average is fine for everyone; the tail is where duties are lost.
• Track beacon_block_delay_head_slot_start_exceeded_total as a rate. It is the closest single number to "how often is my timing budget blown," and it is comparable across your own pairings, as long as you pin one consensus-client version per comparison so a mid-window upgrade does not pool two code paths.

Coming next in the series​

Lighthouse gives the cleanest breakdown, which is why it is first. The other consensus clients expose this timing differently, and that difference is itself a finding: the next editions take the same slot-timing question to Prysm, Teku, Nimbus, Lodestar and Grandine, and where their instrumentation hides things Lighthouse shows, we will say so.

This comparison holds the hardware and the consensus client fixed and varies only the execution client, which is the cut that makes the execution layer's contribution legible. Lining clients up on truly equal footing across windows of uneven data quality is an analysis in its own right, not a lookup on a static panel, and it is what StereumLabs AI does on our fleet, on the measurement stack we described here. When one pairing stops matching its siblings, that divergence is a finding, sometimes an upstream bug report. If you run Ethereum infrastructure and want this lens on your own nodes, reach us at stereumlabs.com or contact@stereumlabs.com.

Methodology​

Numbers come from Lighthouse's beacon_block_delay_* metrics on our NDC2 deployment (Vienna), queried on the Prometheus-cold datasource (uid aez9ck4wz05q8e), with the fleet labels documented in build your own dashboards. Per-client values are for cc_client="lighthouse-super", cc_version="v8.1.3", deployment="NDC2" grouped by ec_client. The would-fail counter rates are increase(...[3d]) evaluated at 2026-06-23T00:00:00Z, so they are stable and reproducible rather than drifting with the query time. The fleet has since moved to Lighthouse v8.2.0, so pinning one CC version per comparison is what reproduces this window. Pinning deployment="NDC2" also keeps the bare-metal Vienna hosts from being pooled with a second Geth set on GCP, which sees the same block-arrival time (~1.88 s) but lands late about 303 times a day against the NDC2 host's 168, a platform effect of the cloud host rather than the network or the client.

• Components are the Lighthouse gauges beacon_block_delay_observed_slot_start (arrival), beacon_block_delay_consensus_verification_time, beacon_block_delay_execution_time (execution-layer verification), and beacon_block_delay_attestable_slot_start (time to attestable). All are in milliseconds, confirmed against Lighthouse's DEBUG block-delay logs which label the same fields _ms.
• Sampling. The component gauges are scraped about once every 66 seconds, so each value is one sampled block rather than a full per-block stream. Averages over three days are unbiased; the sampled quantile_over_time p99 and max_over_time worst-case are conservative floors, since between-scrape spikes are missed. We do not lean on them: the tail is carried by the complete would-fail counter. A precise percentile would need a histogram, which Lighthouse does not expose for these gauges (Nimbus does, so the Nimbus edition can use it).
• The would-fail-attestations figure is increase(beacon_block_delay_head_slot_start_exceeded_total[3d]) / 3, per ec_client. Lighthouse's own help text defines this counter as triggering "when the duration between the start of the block's slot and the current time will result in failed attestations." Unlike the block-delay gauges, this is a monotonic counter, so the per-day counts are complete rather than sampled; only the execution average and p99 columns are the once-a-minute samples. It fires from block timing regardless of validators (it increments on our fleet with none attached), so it is Lighthouse's lateness heuristic, not an observed miss count, and it is a binary flag rather than an inclusion-or-penalty outcome: a flagged slot can still attest late. Read the counts as relative risk between pairings, not as absolute missed duties. Attaching live validators would not move these numbers: the counter times block import, which is independent of attestation signing, aggregation and local block production.
• Arrival is EC-independent. beacon_block_delay_observed_slot_start ranges 1.66 to 1.87 s across the six pairings and does not order by execution-client speed, which is why we attribute the would-fail differences to execution rather than to the network or host.
• Reth is excluded from the comparison: it was resyncing throughout (v2.3.0, finish stage at zero, zero VALID forkchoice responses over 24 h), so its block-delay metrics are not those of a synced node; the sync-speed census covers from-scratch sync times client by client. On identical 12-core hosts; consensus and validator processes run on separate machines.

The fleet, and what we measured​

Each execution client runs six times, once per consensus-client partner (Prysm, Lighthouse, Teku, Nimbus, Lodestar, Grandine, all supernodes on their own hosts; Fusaka's hardware impact is a separate post). The EC host runs the execution client, a node exporter, and a log shipper, nothing else. So host-level telemetry is, to a close approximation, the execution client's footprint, measured the same way for all six clients. Per-client numbers below average across the six pairings.

Versions in the June window: Geth v1.17.3, Nethermind 1.38.0, Besu 26.6.0, Erigon v3.4.3, Ethrex 15.0.0. Sync state was verified from container logs, not just metrics; all five were importing blocks at the live tip (block ~25,294,900) during the window. Reth ran v2.2.0 then but was mid-resync, so its row is taken from v1.11.3 in April, the last time it was synced on this fleet (see below).

RAM is host memory in use (total minus available). Throughput values are decimal MB/s. CPU is in cores of the 12 each host has. P2P traffic is the host's public interface; the internal interface (engine API, monitoring) is excluded. Reth note: the version on our hosts during the June window was mid-resync, so Reth is measured over four days ending 2026-04-23, its last fully-synced window on this fleet, running v1.11.3. The chain head was about 0.4M blocks lower then, so read Reth as same-regime, not same-day. More below.

RAM: 5 to 16 GiB, and one client out of envelope​

Five of the six fit a 16 GiB host. Nethermind is the lightest at 4.9 GiB average, Geth the steadiest heavyweight at 7.9 GiB with a peak only half a GiB above its average, and Reth sits just above it at 8.5 GiB. The peak column is where sizing decisions live: Besu spikes to 9.7 GiB, Reth to 12.7, and Erigon's commit and prune cycles push the host to 13.6 GiB. On 16 GiB that still works, but those three leave the least slack.

Ethrex does not fit. Its process alone held 15.3 GiB resident, more than the usable RAM of the 16 GiB hosts the other five run on, so its set lives on 64 GiB machines where it averaged 15.6 GiB and peaked at 17.6 GiB. No judgement attached: it is the youngest client of the six and the same fleet measured it executing blocks in 66 ms at the tip. But if you plan an Ethrex node today, plan more than 16 GiB.

The consensus partner barely matters for the established clients: across the six pairings, per-pairing RAM varies by one to four percent for Geth, Nethermind, Besu and Erigon. Ethrex moves more, 15.7 to 19.1 GiB depending on the partner over a 7-day view.

The RSS trap​

If you put these six clients on one RSS dashboard, Erigon looks like it is about to die: its process reports 12.9 GiB resident on a host whose memory in use is 6.0 GiB. Both numbers are correct. Erigon (and Reth) serve state from memory-mapped files, and resident set size counts every mapped page currently in RAM, almost all of which the kernel can reclaim on demand. Ethrex's 15.3 GiB RSS, by contrast, sits within a few hundred MiB of its host's missing memory: that allocation is anonymous and cannot be reclaimed.

Same metric name, opposite meanings. Alert on host available memory, not on the client process RSS, or an mmap-based client will page you forever while a genuinely full host stays quiet. The rest of each 16 GiB box fills with page cache (7 to 10 GiB here), which is the system working as intended, not a leak. We hit the same class of problem with disk metrics in the pruning census and with a heap that did lie in the Besu case study.

CPU: half a core at the tip​

Every synced client averaged between 0.42 and 0.55 of one core. Twelve cores sit mostly idle once a node follows the chain; the busiest half-hour of any client was Erigon's 2.86 cores. CPU is a sync-time resource, and Reth shows both sides of it: 0.42 cores synced, against the 1.22 cores its hosts pull right now while they resync the chain. If you size a machine for an execution client, you buy cores for the initial sync and for recovery after downtime, not for steady state. Our sync-speed census measures that phase client by client.

Disk I/O: where the spread gets serious​

This is the widest spread in the census, and it is architectural. Geth keeps its working state in those 7.9 GiB of RAM and touches disk at under 1 MB/s in each direction, the quietest profile of the six. Nethermind, Besu, Ethrex and Reth stay in the low single digits (Reth at 2.0 read, 4.0 write). Erigon inverts the picture: its layered domain files and continuous background pruning write a sustained 22.7 MB/s, about 1.9 TB per day, plus 8 MB/s of reads, in exchange for the smallest disk footprint of the fleet (519 GB vs Geth's 1,731 GB).

That write rate is a budget item. A consumer 2 TB NVMe rated for 1,200 TBW reaches its endurance rating in roughly 20 months of Erigon steady state, before counting the initial sync. Enterprise drives shrug at it, but on hobbyist hardware the cheapest-disk client is also the one that consumes disks fastest. Disk capacity itself is the other axis, and we covered it in depth in the pruning post; current footprints run Erigon 519 GB, Ethrex 588 GB, Nethermind 1,222 GB, Besu 1,265 GB and Geth 1,731 GB, with synced Reth (v1.11.3, April) around 1,399 GB.

Network: a home connection is plenty​

P2P traffic, measured at the host's public interface, stays under 2 MB/s combined for every client. Ethrex moves the most (0.82 MB/s down, 0.74 up, about 4 TB per month both directions combined), Geth uploads the most of the rest (0.68 MB/s, serving peers from its long-lived position in the network), Besu is the quietest at about 0.6 TB per month, with Reth's April figures landing in the middle near 2 TB. On an uncapped line none of this matters; on a metered connection the gap between 0.6 and 4 TB per month picks your client for you. Two notes on reading these numbers: the interface sees discovery chatter, transport overhead and peers that never complete a handshake, so it reads higher than a client's own P2P accounting (Geth's internal counters claim roughly half of what its interface carries), and how each client builds the peer set behind this traffic is covered in the peering deep dive.

Reth, and why its row is dated differently​

Reth's row needs the asterisk because the version on our hosts during the June window was not following the tip. It has been resyncing across two releases (v2.1.0, then v2.2.0) since late April, and another deploy lands today, which resets the back-fill again. So rather than print back-fill load as if it were steady state, we took Reth's numbers from the last window it was demonstrably synced: four days ending 2026-04-23, on v1.11.3, with forkchoiceUpdated answered VALID and the pipeline's finish stage tracking the head block for block. Whether the v2.x line stays behind because of the client or our own deploy cadence is a separate question we have not closed, so we make no claim about it here.

Synced, Reth is unremarkable in the best way: 8.5 GiB average RAM, half a core, 2 MB/s of reads and 4 of writes. That is close to Geth on memory and well under Erigon on disk, which corrects something we expected going in, that Reth's mmap-heavy design would write like Erigon. It does not.

The current back-fill is also a clean illustration of two things this series keeps returning to. First, the cost of syncing: the same hosts that sip 0.42 cores when synced pull 1.22 cores and sustain 22 MB/s reads and 23 MB/s writes while catching up. Second, why we read logs and not just dashboards. Checking the resync this week, Reth's stage checkpoints had all jumped to 25236949 and a metrics panel would have called it done, but the logs showed only execution, hashing and merkle finished, with the final transaction-lookup and history-index stages still grinding at checkpoint=0, about 93,000 blocks behind the tip, every live payload answered SYNCING. We will refresh all six clients on a single window once the new Reth follows the tip again.

Sizing takeaways​

Condensed into hardware requirements for a mainnet execution client in mid-2026:

• 16 GiB RAM fits five of six clients with room to spare; Nethermind and Besu leave the most headroom, Erigon the least. Ethrex currently needs more.
• Cores are for syncing. Half a core follows mainnet; buy 8+ for the day you resync.
• Watch TBW, not just GB. Erigon saves you a terabyte of capacity and spends your drive's write endurance for it. Geth does the opposite.
• Bandwidth is a non-issue unless your line is metered; then 0.6 to 4 TB per month separates the clients.
• Alert on host available memory. Process RSS lies for mmap-based clients.

That table at the top is what StereumLabs AI works with on our fleet: identical hosts, one variable, every client pairing observed the same way. This post shows that comparison under the conditions the fleet happened to be in, which is not fully like-for-like: Reth's row comes from an earlier window and the hosts were freshly rebuilt. Lining the clients up on truly equal footing means choosing comparable periods out of a history with uneven data quality and consistency, which is an analysis in its own right, not a lookup on a static panel. That is what StereumLabs AI is built for, and it is how we pinned Reth's last synced window for this post. When one node stops matching its siblings, that divergence is a finding, sometimes an upstream bug report. If you run Ethereum infrastructure and want this lens on your own nodes, reach us at stereumlabs.com or contact@stereumlabs.com.

Methodology​

Numbers come from node_exporter on the dedicated EC hosts of our NDC2 deployment (Vienna), queried on the Prometheus-cold datasource (uid aez9ck4wz05q8e), with the fleet labels documented in build your own dashboards. Per-client values average the six CC pairings (avg by (ec_client)); a second Geth set on GCP was excluded as a different platform.

• Window: 36 hours ending 2026-06-11 ~15:00 UTC, e.g. avg_over_time((node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes)[36h:30m]). We chose 36 h over our usual 7 d because most EC hosts were rebooted 2.2 days earlier; counter resets at the reboot depress 7-day increase() figures (Erigon's write rate reads 12 MB/s over 7 d but holds 19 to 23 MB/s in every post-reboot hour).
• Reth's window: four days ending 2026-04-23, the last period Reth was synced on this fleet (v1.11.3), measured the same way and averaged across the same six pairings. Confirmed synced by forkchoiceUpdated answered VALID with zero SYNCING and the pipeline finish stage advancing one block per slot; the v2.1.0 then v2.2.0 line deployed since has answered SYNCING continuously. Chain head was ~24.94M then against ~25.29M for the June five.
• RAM is MemTotal - MemAvailable in GiB; peaks are max_over_time over the same window. Process RSS via process_resident_memory_bytes where clients expose it (Besu, Ethrex, Erigon).
• CPU from increase(node_cpu_seconds_total{mode!="idle"}[36h]) per host; the busiest-half-hour figure is the max of 30-minute rates.
• Disk from node_disk_read_bytes_total / node_disk_written_bytes_total on the data device. Network from node_network_*_bytes_total on the host's public P2P interface only; each host's second, internal interface (engine API from the CC host, Prometheus scrapes, log shipping) is excluded. We cross-checked the interface assignment against Geth's and Nethermind's own P2P byte counters: the host interface reads higher than client-internal accounting because discovery, failed handshakes and transport overhead never reach the client's counter, and we report the host number because it is what your line carries. Binary byte values converted to decimal MB/s.
• Hardware: every EC host has 12 cores; 16 GiB RAM except the Ethrex set (64 GiB). Consensus clients (supernodes) run on separate hosts and are not part of these numbers.
• Sync verification per client from container logs at measurement time (import lines at the live tip for Geth, Besu, Erigon, Ethrex; Nethermind additionally via its own head metric at block 25,294,914; Reth's April window as above). Metrics alone can read "synced" while a client back-fills, which is what Reth's June resync showed and why we check logs; see the pruning post for the long version.

This is a walkthrough of how StereumLabs AI, reading our fleet's metrics and logs, took that one anomalous node, traced it to a single method, and filed it upstream. Besu shipped a round of mitigations and closed the issue. A later devnet reproduction showed the underlying layers still pile up, the issue was reopened, and the fix that followed is now in review. The bug is operational: recoverable by a restart, no consensus impact, no double-sign, no state-root divergence. It is also the kind of cross-client interaction a single-node test will never surface, because it only appears when a live pairing lands in a specific state.

One odd node out of six is a signal, not noise​

We run every major execution client paired with every major consensus client, on identical bare metal, with one telemetry pipeline across the whole set. The point of that layout is differential: when six Besu nodes share a version, a host spec, and a config, and differ only in their paired consensus client, any divergence between them isolates the variable for you. So one of six otherwise-identical Besu nodes leaking heap while its five siblings stay flat is not a mystery to explain away. It is a starting point.

StereumLabs AI sits on top of that telemetry. It read the divergence, formed a root-cause hypothesis, and walked it down to a single method. The steps below are ordinary observability: fleet Prometheus, JVM GC metrics, container logs, and one flight-recorder window. Nothing exotic.

The hunt​

Isolate. A cross-pair comparison over fleet Prometheus showed five flat Besu nodes and one climbing. The leaking node was the Besu paired with Prysm v7.1.3. That alone narrowed the cause to something about that pairing, not Besu in general.

Characterise. The garbage-collection signature said live leak, not allocation pressure. After every collection, Old Gen occupancy still climbed about 0.4 GB/h, and the leaking node had the lowest allocation rate of the six. Post-collection Old Gen used equalled live data: the collector had nothing left to reclaim. Heap was being retained, not churned.

Corroborate. The leaking node emitted roughly seven times the log volume of the median Besu pair, dominated by one engine new-payload hot path: "block already present". It logged 53,756 engine_newPayloadV4 calls in 17.5 hours, against a one-per-slot baseline near 5,250, for blocks it already had.

Confirm. A Java Flight Recorder window settled it. About 84 percent of all CPU samples sat inside one method, LayeredKeyValueStorage.isClosed(), with most of the rest in the layered store's comparator. The chain head had not advanced in over 18 hours, despite tens of thousands of new-payload calls.

The root cause, in plain terms​

The cause sat in the seam between the two clients; the symptom showed up only on Besu.

On the consensus side, the paired Prysm had fallen behind and was catching up. It fed Besu a run of blocks through engine_newPayload without sending a forkchoiceUpdated between them. Many of those calls timed out from its HTTP client, so it retried, which is where the loud "block already present" log volume came from. The Prysm-side issue that drops it out of sync, and into this catch-up, is tracked in OffchainLabs/prysm#16096. Consensus clients that send a forkchoice update first, such as Lighthouse and Nimbus, take a different path on Besu and do not trigger it.

On the Besu side sat the part that had not been seen before. For each new block applied this way, Besu freezes a Bonsai LayeredKeyValueStorage and, on the no-forkchoice path, never closes it, so eviction falls to the garbage collector. Under sustained catch-up the layers pile up faster than the collector frees them. The repeat "block already present" calls do not add layers themselves, but every state access now has to check whether its layer is closed, and that check delegates straight up the parent chain:

public boolean isClosed() {
  return parent.isClosed();
}

That one line runs on every state access. With a flat layer stack it is free. With a thousand-plus accumulated layers it becomes a walk up the entire parent chain on every access, which is why it rose to about 84 percent of CPU. Block validation could then no longer finish inside the engine timeout, so the chain head stalled, which made Prysm time out and retry more, which deepened the stack further. A self-reinforcing loop, with the heap curve in the chart above as its outward sign.

What shipped, what reopened it, and the fix​

We filed this as besu-eth/besu#10498. Besu triaged it P2 and landed a set of mitigations:

• PR #10508 caches a layer's closed-state so the parent-chain walk can short-circuit.
• Three allocation cleanups (#10523, #10526, #10527) replace per-call exception objects with stackless singletons, cutting the GC pressure the stall generated.
• PR #10559 caches the validated engine JWT so a consensus-client reconnect storm no longer forces a Besu restart.

Those merged and the issue was closed on 2026-05-28. The direct structural fix, PR #10509, which would have capped the layer stack and returned SYNCING past a bound, was closed on 2026-05-27 without merging.

It came back. On a mainnet nightly node running 26.6-develop with Prysm 7.1.4, the heap climb returned, and Besu's maintainer reopened the issue on 2026-06-05. The isClosed() cache from #10508 only short-circuits once a layer is closed, and these layers are never closed, so a deep open chain still recurses fully on every call. The caches relieve the surrounding pressure. They do not stop the layers from accumulating.

To pin the mechanism down we built a Kurtosis devnet reproduction: a Besu 26.6.0 target paired with Prysm, plus two controls, the same Besu paired with Lighthouse and a Prysm paired with Geth. Stopping the target's Prysm, letting the network run ahead, then feeding the new blocks back through engine_newPayloadV4 with no forkchoice update between calls drove the target from 58 to 1,362 retained layers while the control Besu stayed at 0. Replays of already-present blocks stayed flat over 5,000 calls, which isolated the growth to new blocks applied without an intervening forkchoice update. isClosed() was again the dominant cost, and engine_newPayload latency tracked the depth: about 3.5 ms against an already-present block near the top of the stack, rising to a median around 245 ms and up to 1.4 s at depth 1,300 to 1,500. On that evidence the issue was reopened, and this time the fix came quickly.

Besu opened PR #10600, which returns SYNCING when the parent world state is not immediately cached. That routes a run of no-forkchoice new-payload calls into backward sync, the same path Lighthouse and Nimbus already take, instead of the trie-log replay that builds layers. Besu validated it against our Kurtosis file: on the unpatched 26.6.0 image, 30 new-payload calls in 437 ms with no forkchoice update between them left 30 unflushable Bonsai layers; on the patched build the same test left one layer, answered 27 of the calls with SYNCING, and recovered through backward sync in about two minutes. A companion change, PR #10603, makes LayeredKeyValueStorage.isClosed() itself O(1), so the CPU hot path is gone even if a deep chain ever forms. Both are in review as of this writing, so the issue stays open until they land, but the fix is written and confirmed against the same reproduction that reopened it.

Credit to both teams. The Prysm-side sync loss is a known issue with a fix in progress, and Besu has been fast and precise on every round, turning our reproduction into a validated fix. Neither client is at fault in isolation; the bug lives in the seam between them, on a path that only a specific live pairing reaches.

Why this kind of bug needs a fleet​

A single Besu node, tested on its own with a healthy consensus client, will not show this. It needs a paired client that has fallen behind and is feeding new blocks through new-payload without forkchoice updates, sustained long enough for the layer stack to grow. It does not need an attacker: one restart that drops a consensus client into catch-up is enough. The operational bite is serious for the operator who hits it, a Besu that OOMs in about a day with the paired client churning peers and missing duties around it, but the path to seeing it at all runs through fleet-scale, differential observation.

That is what StereumLabs AI does on our fleet: it reads the telemetry across every client pairing, notices when one node stops matching its siblings, and turns the anomaly into something a client team can act on. If you run Ethereum clients at scale, you can point it at your own nodes and logs. Reach us at stereumlabs.com or contact@stereumlabs.com.

Disk is only half of it. The same pruning that bounds those footprints also runs on the engine-API hot path, and on some consensus-client pairings it pushes engine_newPayload past the 4-second sync-committee deadline. This post covers both halves: how big each client gets and whether you can see it pruning, then what that pruning costs in latency once a node is at the tip.

Key findings:

• 3.3× footprint spread on the same chain (root-FS used per host): Erigon ~509 GB, Ethrex ~598 GB, Nethermind ~1,225 GB, Besu ~1,261 GB, Reth ~1,352 GB (not synced), Geth ~1,688 GB.
• Five of six ECs grew 15 to 19 GB in the 7-day window (~2–3 GB/day, normal block ingestion). The cumulative spread comes from retention policy, not the past week's growth.
• Each EC exposes a different pruning surface, and Ethrex exposes none. Erigon's domain pruner fires ~501 k blocks events per host per week; Nethermind's Hybrid pruner runs continuously at ~1,497 nodes/sec/host; Geth path-mode prunes implicitly; Ethrex has no pruning metric at all.
• Reth is still in initial staged sync: its checkpoint sits ~70 to 130 k blocks behind the tip and is not advancing, so it is excluded from the synced comparisons.
• From Nimbus, three of five synced ECs cross the 4-second deadline on P99 engine_newPayload (Nethermind 4.44 s, Geth 4.67 s, Ethrex ≥5.00 s), but the ranking flips by consensus client. It is a pairing property, not a fixed EC property.
• Nethermind's spikes trace to two deliberate v1.37.1 choices. nethermind_pruning_time spikes past 5 s, over the deadline, entirely from the in-memory pruner.
• Sync-committee duties get hit before attestations. A late sync-committee message misses the next block's sync_aggregate outright; a late attestation still has one slot of grace.

What we measured: the StereumLabs Ethereum execution-client fleet​

The bare-metal NDC2 fleet in Vienna runs the full pairing matrix: every consensus client (Grandine, Lighthouse, Lodestar, Nimbus, Prysm, Teku) paired with every execution client (Besu, Erigon, Ethrex, Geth, Nethermind, Reth). The GCP comparator cohort runs only Geth on the EC side. Host counts: Geth 13 (7 NDC2 + 6 GCP), Besu 7, all others 6. Versions current at query time: Geth v1.17.3, Nethermind 1.37.2 (PruningMode=Hybrid), Reth v2.2.0, Besu 26.5.0, Erigon v3.4.1, Ethrex 13.0.0. Hardware setup is documented in the StereumLabs measurement stack post.

All counters come from our prometheus-cold datasource over a rolling 7-day window ending around 2026-05-28 12:00 UTC. Per-host averages divide each EC's totals by its live host count. Queries are in the methodology section. (Host ages range 7 to 16 days, per the caveat above; Besu and Ethrex hosts are youngest at ~8 days.)

Every EC exposes its own pruning vocabulary on Prometheus. No two share names, and one exposes nothing at all:

• Geth (path-mode): pathdb_gc_node_count, pathdb_history_state_bytes_data, pathdb_history_state_time. No explicit prune-event counter; pruning is structural to path-mode.
• Nethermind (Hybrid): nethermind_pruned_persisted_nodes_count, nethermind_deep_pruned_persisted_nodes_count, nethermind_pruning_time, nethermind_state_db_in_pruning_writes, nethermind_pruning_cutoff_blocknumber. The most complete surface, with an explicit continuous-vs-deep split.
• Besu (Bonsai): besu_pruner_trie_log_added_to_prune_queue_total, besu_pruner_trie_log_pruned_from_queue_total, besu_pruner_trie_log_pruned_orphan_total, plus the besu_executors_ethscheduler_chaindatapruner_* thread-pool view. Trie-log queue only.
• Erigon: prune_seconds_count (per type label), domain_prune_size, domain_prunable, domain_pruning_progress, domain_prune_took_bucket. Domain-based, the cleanest cross-cutting surface.
• Reth: reth_pruner_duration_seconds, reth_pruner_segments_duration_seconds, reth_pruner_segments_highest_pruned_block. Per-segment timing; only SenderRecovery is populated here.
• Ethrex: no pruning metric at all. Operators derive pruning state from datadir_size_bytes.

So "did my client prune in the last hour" is answerable for some ECs and unanswerable for others, before we look at a single number.

How big are the datadirs?​

The numbers below come from node_filesystem_* (root-FS used per host), not the EC-internal size metrics. Those internal metrics each report only a slice of the datadir and understate it by hundreds of GB; the methodology has the per-client gap analysis (it is what produced the wrong 26 GB Erigon figure). On these hosts the EC datadir lives on / and the CC runs elsewhere, so root-FS is EC datadir plus ~5 to 10 GB of OS and logs.

Every EC's own size metric understates its real disk use. Erigon's db_table_size_bytes sees 26 GB of a 509 GB datadir; Besu's RocksDB metric reports 0.

A head-block metric is not a reliable sign that an EC is actually synced (a client can follow the consensus layer's forkchoice head optimistically while still syncing), so we confirmed each one from its own container logs, not just its metrics. Five of six are importing at the tip in real time: Geth logs Imported new potential chain segment, Nethermind Synced Chain Head to 25213…, Besu AbstractEngineNewPayload | Imported #25,213,…, Erigon Post-Forkchoice prune … initialCycle=false, and Ethrex [METRIC] BLOCK 25213… exec … 99% BOTTLENECK (slow per block, but genuinely executing at the tip). Reth is the exception and gets its own section below; read its 1,352 GB as mid-sync state.

The 3.3× spread comes from retention architecture: Erigon drops account, storage, and block-body history once fork-choice no longer needs it and keeps no ancient store, while Geth path-mode adds a continuously growing ancient store on top of its bounded state journal. The per-week growth is the same ~16 GB across the synced clients (normal block ingestion); only the starting footprint differs. The per-client sections below break down each mechanism. It is the storage-side version of what the EC sync-speed comparison found on the same fleet: same hardware, same chain, very different footprints.

Geth: path-mode prunes implicitly​

Geth v1.17.3 runs path-based storage. Historical trie nodes for each state transition flow into a bounded "state history" journal and drop out as new history is written. There is no prune event; the boundedness is the prune. Three metrics show it:

• pathdb_gc_node_count: ~300 M nodes garbage-collected per host since startup (291 M to 301 M across our 13 hosts), growing monotonically as path-mode GC trims old tries.
• pathdb_history_state_bytes_data: 1.33 GB per host, barely moving. The state-history journal, bounded by --history.state (default 90,000 blocks).
• pathdb_history_state_time p99: 510 µs to 1.29 ms per host (one GCP host at 1.66 ms). Time to write a single state-history block.

Geth exposes no per-cycle prune counter and no cutoff block. The only health signal is pathdb_history_state_time p99 climbing over hours, which means the pruner is falling behind writes, though it will not tell you whether the pruner or the writes are the slow part.

Geth's +16.2 GB/week is ancient-store growth, not state. The ancient store holds frozen headers, bodies, and receipts that path-mode never touches, and at ~1,688 GB it is the bulk of Geth's footprint. Bounding total Geth disk needs separate --history.transactions and --history.logs flags, left at defaults here.

Nethermind: the most complete pruning surface​

Nethermind 1.37.2 with PruningMode=Hybrid is the only EC that exposes both continuous and deep pruning as distinct counters. The series that matter:

• nethermind_pruned_persisted_nodes_count: ~1.13 billion increments in 7 days, ~1,497 nodes/sec/host (range 1,478 to 1,517).
• nethermind_deep_pruned_persisted_nodes_count: full-prune count. Zero on every host.
• nethermind_state_db_in_pruning_writes: 0.33 to 0.41, so the state DB is in a pruning write roughly a third of the time.
• nethermind_pruning_cutoff_blocknumber: 0 across every host. On Hybrid, that means the deep-prune branch has never fired.
• nethermind_pruning_time, nethermind_pruning_cutoff_timestamp: cycle duration and cutoff in epoch seconds.

So the continuous in-memory pruner does all the work in our window and the deep prune has not been needed. On nethermind_db_size the state DB stays roughly flat, but the full datadir (receipts, indexes, logs) still grows ~16 GB/week, so monitor root-FS for capacity, not nethermind_db_size. That continuous pruner is also what makes engine_newPayload spike on this version, covered in the latency section below.

Nethermind is the easiest EC to alert on: rate(nethermind_pruned_persisted_nodes_count[5m]) + rate(nethermind_deep_pruned_persisted_nodes_count[5m]) from the public Nethermind dashboard gives pruning throughput in nodes/sec. That rate dropping while disk grows means the pruner stalled.

Reth: still in initial sync, so its pruning is not yet observable​

Reth v2.2.0 has not finished its initial staged sync on any of our six hosts, and still had not as of this writing. Its container logs show the sync::stages pipeline still running rather than importing at the tip, and the metrics agree: the staged-sync checkpoint sits ~75 to 135 k blocks behind the tip and has not advanced in 12 hours on any host, the final pipeline stages (Finish, TransactionLookup, the history indexes) are still at 0, reth_blockchain_tree_canonical_chain_height is 0, and every newPayload returns SYNCING, never VALID. The process is alive (CPU and disk I/O are non-zero, with static-file rewrites both adding and reclaiming tens of GB), it just is not catching up. We cannot say from one fleet whether the stall is the client or something in our deployment, so we are not drawing any conclusion about Reth beyond "not comparable here."

So Reth's pruning is genuinely unmeasured. Its pruner has only ever run the one startup pass on the SenderRecovery segment (reth_pruner_duration_seconds_count is 0 or 1 per host). The ~1,352 GB on disk is mid-sync state, not a steady-state footprint, and Reth's documented engine backpressure (reth_consensus_engine_beacon_backpressure_active) never activates because the node never reaches steady state. We will rerun the comparison once these hosts finish syncing.

Besu: trie-log pruning is the only window into Bonsai cleanup​

Besu 26.5.0 runs Bonsai. Its pruning surface is narrower than Nethermind's and focused on the trie-log queue, the per-block diffs Bonsai keeps for reorg handling and historical queries:

• besu_pruner_trie_log_pruned_from_queue_total: ~352 k trie logs pruned in 7 days, ~50,343 per host per day.
• besu_pruner_trie_log_added_to_prune_queue_total: logs enqueued. The queue-minus-pruned difference is the backlog.
• besu_pruner_trie_log_pruned_orphan_total: logs marked for deletion with no matching entry. A corruption signal; expect near zero.
• besu_executors_ethscheduler_chaindatapruner_*: thread-pool view, confirms the pruner scheduler is alive.

Besu exposes no Bonsai state size, no cutoff block, and no total disk figure (besu_rocksdb_rocks_db_files_size_bytes returns 0 here, an instrumentation gap). You read total Besu disk from node_filesystem_*. And the 50 k logs/day figure does not convert to bytes, since trie-log size varies per block with transaction mix, so it is a heartbeat, not a capacity input.

Erigon: the smallest footprint in the fleet, by a wide margin​

Erigon v3.4.1 runs domain-based pruning across three named domains: blocks (block bodies and headers), state (account and storage state), and bor (Polygon-specific; always 0 on Ethereum mainnet). The Prometheus surface follows that decomposition:

• prune_seconds_count{type=blocks}: ~501 k prune events per host in 7 days (460 k to 528 k), about one every 1.2 seconds.
• prune_seconds_count{type=state}: ~50,775 events per host in 7 days, about one every 12 seconds.
• prune_seconds averages: ~1 ms per blocks prune, ~12 ms per state prune.
• domain_prune_size, domain_prunable, domain_pruning_progress: per-domain queue gauges, for confirming the backlog stays bounded.

Erigon's root-FS sits at ~509 GB, ~3× below the heaviest EC. Its weekly growth (+17.7 GB) matches everyone else's; only the cumulative footprint is smaller, because of the aggressive history retention. (The 26 GB an earlier draft quoted was the MDBX-only db_table_size_bytes; the snapshot and history files make up the rest.)

A Caplin standalone host (Erigon's built-in CL) shows the same prune_seconds_count series under job="caplin", and its root-FS sits at 525 GB, only ~25 GB above a plain Erigon host despite carrying the CL in the same binary. Details in the Erigon + Caplin standalone vs classic post.

Ethrex: pruning is invisible from Prometheus​

Ethrex 13.0.0 exposes no pruning metric. list_prometheus_metric_names for ethrex_.* returns 31 series (peer counts, sync stages, discovery), none describing pruning, retention, or reclamation. The only disk signal is datadir_size_bytes, which matched node_filesystem_* to within ~100 GB and grew +15.3 GB/week per host.

Root-FS sits at ~598 GB, smaller than every EC except Erigon, so some retention bound is clearly in effect. But you cannot see when Ethrex prunes, how much it reclaims, or whether the pruner stalled. The Ethrex dashboard has one disk panel ("DB Size On Disk Total") and no pruning section.

For operators that is the tradeoff: a young Rust client with the smallest metrics surface and no first-class pruning observability. It is fixable by adding instrumentation to the existing metrics endpoint, not a storage-layer limitation. We are happy to coordinate the metric shape with the Ethrex team.

Pruning vs the 4-second sync-committee deadline​

The footprints above are the static picture. The operational question is what the pruner does to the engine-API hot path while a node is at the tip. Sync-committee messages and attestations both need to be ready ~4 seconds into the slot. A P99 engine_newPayload above that ceiling means a tail of slots where the CC builds duties against a stale head. Measured from each Nimbus host on engine_api_request_duration_seconds_bucket{request="newPayload"}, 7-day window:

Three of the five synced clients sit over 4 s from Nimbus. Ethrex's value is at the histogram's top bucket (5.00 s), so the true P99 is higher; we write ≥5.00 s, and it is a slow-block-validation problem rather than pruning per se. The order tracks the per-client mechanisms above: Besu prunes off-thread, Erigon prunes in a fast stage, and Nethermind, Geth, and Ethrex all touch the engine thread during pruning. Reth's 0.23 s is the SYNCING return path, not block execution, so it is excluded.

The ranking flips by observer​

Three CCs export a newPayload histogram: Nimbus (engine_api_request_duration_seconds), Lighthouse (execution_layer_request_times), and Prysm/Lodestar (new_payload_v1_latency_milliseconds). Teku does not. Each calls notifyNewPayload at a different point relative to gossip validation, so the same EC reads differently:

The same EC lands on opposite sides of the 4 s line depending on the CC watching. Nethermind breaches on Nimbus only; Besu breaches on Lighthouse.

Across the synced clients:

• Nethermind: 4.44 s on Nimbus, ~1 s on Lighthouse and Prysm. Nimbus calls newPayload later in the slot, into Nethermind's in-memory pruning window between blocks.
• Besu: 1.07 s on Nimbus, ~4 to 5 s on Lighthouse. The Bonsai path meets the Lighthouse calling pattern differently.
• Geth: over 3.4 s on every CC. Its PBSS commit cost lands on the engine thread whenever the call arrives.
• Ethrex: 3.55 s to ≥5.00 s on every CC. Different cause (slow block validation, not pruning), same outcome for duty timing.

So "is my EC fast enough?" depends on the pairing, not the EC alone. The rule: if any CC↔EC pair holds a P99 above 4 s, that pair is the one to investigate for late sync-committee messages. Re-measure with your own CC's histogram first. Latency that swings this much by consensus client is a recurring pattern on this fleet; we saw the same shape in blob verification latency across consensus clients.

Why Nethermind ships this trade-off (the v1.37.1 PRs)​

Nethermind 1.37.2 inherited two 1.37.1 changes that trade engine-thread serialization for lower queue latency:

1. nethermind#10479, "Run newPayload inline" (Feb 2026). Enables AllowSynchronousContinuations, so newPayload runs on the engine thread when the queue is empty. It saves a context switch, but any in-flight prune on that thread now extends newPayload directly. This is why a Nethermind behind a CC that calls at the wrong moment hits 4 to 7 s spikes.
2. nethermind#10247, "Bump default pruning cache by 512MB" (Jan 2026). Pruning.CacheMb 1280 → 1792, Pruning.DirtyCacheMb 1024 → 1536. The PR's own note: "does take a bunch longer; but also is between blocks so should be ok." On our fleet that assumption breaks; the longer cycle does not always finish between blocks under load.
3. nethermind#10227 added the two knobs operators now need: Pruning.PruneDelayMilliseconds (default 75 ms, gap between prune iterations) and Merge.PostBlockGcDelayMs (default SecondsPerSlot/8 = 1500 ms).

Fleet evidence: nethermind_pruning_time baselines at ~700 ms with regular spikes to 4,000 to 6,300 ms, and nethermind_pruning_cutoff_blocknumber = 0 for the whole window. No deep prune ran; every spike is the in-memory pruner alone. The Nethermind pairing turning up as the slow one is a repeat finding on this fleet: our Teku version comparison flagged persistently elevated block-import delays on the Teku + Nethermind pairing across three releases, so this is not purely a Nimbus artifact.

24 h on one Nethermind host. The baseline is harmless; the spikes are not. The 5.06 s cycle alone overruns the slot budget, and a newPayload landing in that window inherits the delay.

Open Nethermind tracker items:

• nethermind#10557 (open, stability): production validator infrastructure issue for this pattern.
• nethermind#6893 (open): FCU/newPayload timeouts with Lighthouse after restart.
• nethermind#8433 (open since 2025-03): memory-config recommendations, no conclusion.
• nethermind#9149 (closed via PR #9588): proposed pausing in-memory pruning; the merged PR only marks finalized blocks, it does not pause.

In-flight PRs that may help in a later release: nethermind#11539 (hot-path allocations), nethermind#10952 (PersistenceManager prune leak), nethermind#11636 (double execution in newPayloadWithWitness).

Where each client runs pruning relative to the engine thread​

Sources: Reth's backpressure is a first-class metric (reth_consensus_engine_beacon_backpressure_active) with a documented --engine.persistence-backpressure-threshold knob. Besu's separate-process choice is in besu#4476, with PR #4409 dropping p95 block processing from 2.98 s to 0.81 s. Erigon's bounded stage is what remained after erigon#6584 closed as not-planned. Geth's pattern is geth#28317.

This is a design ranking, not a measured leaderboard: the cross-CC table shows the measured order moves with the observer, and Reth's rank-1 design is untested because its hosts never synced.

What an operator can tune today​

Starting points to test on a canary node, not blind production changes. We applied the Nethermind set on our own nodes; the rest come from each client's docs and source, linked inline so you can check the exact key names and defaults yourself.

Nethermind 1.37.2 has the highest measured impact, so start here. The lever is making each prune iteration yield to the engine thread, not growing the cache:

# Pruning
PruneDelayMilliseconds = 150   # default 75; the main lever, more yield to the engine thread
DirtyCacheMb = 1280            # default 1536; smaller dirty cache = shorter prune cycles

# Merge
PostBlockGcDelayMs = 900       # default SecondsPerSlot/8 = 1500

Pruning.PruneDelayMilliseconds and Merge.PostBlockGcDelayMs were added in nethermind#10227 (defaults 75 ms and SecondsPerSlot/8); the Pruning.CacheMb 1280→1792 and Pruning.DirtyCacheMb 1024→1536 default bumps are nethermind#10247. Do not raise Pruning.CacheMb to fix the latency: a bigger cache makes each prune cycle take longer (that PR's own note: "does take a bunch longer"), the opposite of what you want when the prune collides with newPayload. Shorter, more frequent iterations with a higher PruneDelayMilliseconds keep the engine thread responsive. These keys live in Nethermind's JSON/.cfg config or --Pruning.PruneDelayMilliseconds-style flags, not YAML. Validate against your own P99 first.

Geth v1.17.x (command-line options): --cache 8192 (default 4096) if memory allows, and raise --cache.trie (default 15, the percentage of --cache given to the trie). Do not reach for --gcmode archive on a validator: it stops pruning but turns the node into a multi-TB growing archive, a different deployment, not a knob. Ancient-store growth is bounded separately by --history.transactions and --history.logs (both default 2,350,000 blocks), and path-mode state by --history.state (default 90,000).

Reth v2.2.0: no synced host here, so no measured advice. By design it gates pruning behind engine backpressure (--engine.persistence-backpressure-threshold, default 16, introduced in Reth 2.0); confirm against your own synced node's P99.

Besu 26.5.0 (limit-trie-logs docs): keep --bonsai-limit-trie-logs-enabled=true (default in 26.5+); the prune batch is --bonsai-trie-logs-pruning-window-size (default 30000). Besu was the slowest from the Lighthouse observer (~4.9 s), so test against your actual CC.

Erigon v3.4.x (options): little to tune; accept the ~2 s baseline. --prune.mode defaults to full; archive does the opposite of saving disk (it keeps all state history), so only use it if you need archive data.

Ethrex 13.0.0: not validator-grade in our measurements (≥3.5 s on every CC); track upstream and raise your CC's per-EL timeout if you must run it.

Why sync-committee duties get hit before attestations​

Both duties share the slot-start + ~4 s deadline. But an attestation has one slot of inclusion grace via the next aggregator, while a sync-committee message is aggregated into the next block's sync_aggregate and a late one misses that inclusion outright. So the same newPayload tail that dents attestation effectiveness dents sync-committee participation more, in percentage terms. That matches what we and other Nethermind operators see: sync-committee periods feel disproportionately bad during pruning windows, even when attestation effectiveness barely moves.

Cross-client cheat sheet​

By the question you are actually asking:

How much disk does my EC use? Use (node_filesystem_size_bytes − node_filesystem_avail_bytes) on the datadir mount. The EC-internal size metrics all understate it (see methodology); node_exporter is the portable answer.

Did my EC prune in the last 5 minutes? Yes on Erigon (rate(prune_seconds_count[5m])), Nethermind (rate(nethermind_pruned_persisted_nodes_count[5m])), Besu (rate(besu_pruner_trie_log_pruned_from_queue_total[5m])). No on Geth (no event counter), Reth (one event total, so any rate is noise), Ethrex (no metric).

Has the deep/full prune fired? Only Nethermind tells you, via nethermind_pruning_cutoff_blocknumber. It is 0 here, so the deep prune has not run.

Is this EC fast enough for my CC? Compute P99 engine_newPayload from your CC's histogram and compare to 4 s. The answer is per pairing, not per EC (see the latency section); re-measure on your own stack before trusting any ranking.

Why did my pruner stop? Nethermind: nethermind_state_db_in_pruning_writes hits zero while disk grows. Erigon: domain_prunable rises while domain_prune_size stalls. Besu: the added-minus-pruned trie-log backlog grows. Geth: pathdb_history_state_time p99 climbs over hours. Reth and Ethrex: no usable signal, check the config or the disk.

This mirrors what the reorg accounting post found on the consensus side, where beacon_reorgs_total reads 0 on Lodestar and 6,011 on Prysm over 90 days, and what the EC P2P peering deep dive found in how differently each client reports the network. One shared concept, six instrumentation choices, and on Ethrex, silence.

How Glamsterdam history expiry will change EC pruning​

EIP-4444 (history expiry) lets execution clients drop pre-expiry headers, bodies, and receipts instead of keeping them forever in ancient stores. Glamsterdam is the activation vehicle, in devnet stabilisation per the EF's April 2026 Checkpoint, with no fixed mainnet slot yet. Two consequences for the numbers here:

• Geth's and Ethrex's footprints are mostly historical retention. Once operators flip on expiry, the ~1,688 GB Geth and ~598 GB Ethrex numbers should fall toward Erigon's ~509 GB. Read today's figures as the pre-expiry baseline.
• The observability gap gets more expensive. A stalled pruner you can see (Erigon, Nethermind, Besu) is a quick fix; a stalled pruner on a client with no pruning metric (Ethrex) is invisible until the disk fills. If you cannot answer "is my EC pruning" today, you will not be able to answer "is it observing the post-expiry bounds" after the fork. The instrumentation needs to land first.

Frequently asked questions about Ethereum EC pruning​

Why is Erigon's datadir 3.3× smaller than Geth's on the same chain?​

Storage architecture. Erigon prunes account, storage, and block-body history once fork-choice no longer needs it and keeps no ancient store. Geth path-mode keeps a bounded ~1.33 GB state journal plus a continuously growing ancient store. Both serve the same engine-API traffic; they just retain different amounts of history. EIP-4444 should narrow the gap.

How do I check if my Geth path-mode node is pruning correctly?​

Watch two metrics: pathdb_history_state_bytes_data should stay roughly flat at your --history.state window (default 90,000 blocks), and pathdb_history_state_time p99 should stay under ~2 ms. There is no prune-cycle counter; the bounded journal is the prune. Ancient-store growth is separate and bounded by --history.transactions and --history.logs.

Why does Nethermind's pruning_cutoff_blocknumber stay at 0?​

On PruningMode=Hybrid, Nethermind runs a continuous in-memory pruner and only escalates to a disk-mode deep prune when needed. The deep prune has not been needed in our window, so the cutoff and nethermind_deep_pruned_persisted_nodes_count stay at 0 while the continuous pruner does the work at ~1,497 nodes/sec/host. It keeps the state DB flat on nethermind_db_size, but the full datadir still grows ~16 GB/week, so monitor root-FS for capacity.

Why does Nethermind miss sync-committee duties during pruning?​

Its 1.37.1 inline-newPayload change runs engine_newPayload on the engine thread, so an in-flight in-memory prune now serializes with it. nethermind_pruning_time spikes to 4 to 6 s, which pushes P99 past 4 s on Nimbus pairings. On Lighthouse and Prysm it stays ~1 s, so it is the Nimbus calling pattern meeting the inline path. Mitigate with PruneDelayMilliseconds up and DirtyCacheMb down (see the tuning section).

Why is the same EC fast on one CC and slow on another?​

Each CC calls notifyNewPayload at a different point in the slot (Prysm early, Lighthouse mid, Nimbus late), so a CC that calls late meets more pruning contention. Nethermind is the clearest case: 4.44 s on Nimbus, ~1 s on Lighthouse and Prysm, same EC and load. The rule: if any CC↔EC pair holds a P99 above 4 s, investigate that pair, and measure with your own CC's histogram.

Is Reth synced in this comparison?​

No. Every Reth host is still in initial staged sync: the checkpoint sits ~70 to 130 k blocks behind the tip and has not advanced in 12 hours, every newPayload returns SYNCING, and the final pipeline stages are still at 0. So its disk figure is mid-sync state, its 0.23 s newPayload is the SYNCING return path, and its pruning is unmeasured. We will re-run once these hosts finish syncing.

How do I monitor Ethrex pruning without Prometheus metrics?​

Watch datadir_size_bytes and its 7-day delta. If it grows faster than mainnet block data implies (roughly 30 to 50 GB/month per client at current activity), investigate. There is no prune-event metric today; ask for one upstream, or write to contact@stereumlabs.com.

Will EIP-4444 history expiry change these numbers?​

Yes, for the cumulative footprint. Geth's ~1,688 GB and Ethrex's ~598 GB are mostly historical block data that EIP-4444 lets clients drop; Erigon at ~509 GB already approximates the post-expiry state. (Reth is excluded; its hosts are still syncing.)

Methodology​

All queries ran against our prometheus-cold datasource over windows ending around 2026-05-28 12:00 UTC.

Per-EC root-FS used (the ground-truth for on-disk size), 7-day rolling average, and its 7-day net delta:

avg by (ec_client) (
  avg_over_time(
    (node_filesystem_size_bytes{mountpoint="/", fstype!="rootfs", role="ec"}
     - node_filesystem_avail_bytes{mountpoint="/", fstype!="rootfs", role="ec"})[7d:1h]
  )
)

avg by (ec_client) (
  (node_filesystem_size_bytes{mountpoint="/", fstype!="rootfs", role="ec"}
   - node_filesystem_avail_bytes{mountpoint="/", fstype!="rootfs", role="ec"})
  -
  (node_filesystem_size_bytes{mountpoint="/", fstype!="rootfs", role="ec"} offset 7d
   - node_filesystem_avail_bytes{mountpoint="/", fstype!="rootfs", role="ec"} offset 7d)
)

Per-EC pruning-event rates, plus the Geth path-mode and Nethermind in-pruning signals:

sum by (ec_client) (rate(besu_pruner_trie_log_pruned_from_queue_total[7d])) * 86400
sum by (ec_client) (rate(nethermind_pruned_persisted_nodes_count[1h]))
sum by (ec_client, type) (increase(prune_seconds_count[7d]))
pathdb_gc_node_count
avg_over_time(nethermind_state_db_in_pruning_writes[7d])
nethermind_pruning_cutoff_blocknumber

Reth sync-state check (confirming the hosts have not finished syncing, so their disk and newPayload numbers are mid-sync):

reth_blockchain_tree_canonical_chain_height
reth_consensus_engine_beacon_new_payload_valid
max by (instance) (reth_sync_checkpoint)          # vs chain_head_block; flat over 12h = no progress

Every EC's sync state was cross-checked against its container logs in Elasticsearch (block-import lines such as Geth's Imported new potential chain segment or Besu's Imported #… for the synced clients, versus Reth's running sync::stages pipeline), not metrics alone, because a head-block metric can reflect the forkchoice head an EC is following optimistically while it is still syncing.

P99 engine_newPayload from each CC observer, 7-day rolling, and the Nethermind prune-cycle latency it tracks:

# Nimbus
histogram_quantile(0.99, sum by (cc_client, ec_client, le) (rate(engine_api_request_duration_seconds_bucket{request="newPayload"}[7d])))
# Lighthouse
histogram_quantile(0.99, sum by (cc_client, ec_client, le) (rate(execution_layer_request_times_bucket{method="new_payload"}[7d])))
# Prysm/Lodestar (Teku does not export this histogram in our window)
histogram_quantile(0.99, sum by (cc_client, ec_client, le) (rate(new_payload_v1_latency_milliseconds_bucket[7d])))

nethermind_pruning_time

Host counts, taken from count by (ec_client) (count by (ec_client, instance) (up{role="ec", job!="node_exporter"})): Geth 13 (7 NDC2 + 6 GCP comparator), Besu 7, Erigon 6, Ethrex 6, Nethermind 6, Reth 6. The job!="node_exporter" filter avoids double-counting, since every EC host also runs node_exporter under role="ec". The two Grandine custom-image hosts (cc_version="unstable-465c727") are kept because the EC underneath them is a production image.

Why we use node_exporter, not the EC-internal size metrics. Each EC-internal metric reports only a slice of the datadir. This is the gap that produced the wrong 26 GB Erigon figure:

Geth lands within ~60 GB and Ethrex within ~100 GB; the rest understate by hundreds of GB, and Besu by everything. So we use node_filesystem_*, which adds only ~5 to 10 GB of OS and logs on top of the datadir.

Top-bucket caveat. Ethrex's ≥5.00 s (and Reth's 4.00 s on Prysm) sit at the histogram's top bucket, so the true P99 is in the +Inf tail. We do not rank a capped value against a measured one. The read-this-first caveats (no validators, per-CC-observer) apply to every latency figure; where a single P99 is unattributed it is from Nimbus.

The window covers post-Fusaka mainnet conditions only. Pre-Fusaka data (before December 3, 2025) is in cold storage but not included here because PeerDAS introduced significant fork-choice timing changes that mix populations.

Dashboard panel definitions for every metric used here are linked from the EC-specific dashboards in our public catalog at /docs/dashboards/list/catalog. The per-client dashboards on grafana.stereumlabs.com carry the pruning panels above; Ethrex is the exception and ships without a pruning section.

If you maintain an Ethereum execution client and want to clarify what we measured (the Besu RocksDB size series, Reth's intended pruner cadence, the Ethrex Prometheus surface, or Nethermind's inline-newPayload path), write to contact@stereumlabs.com. We will append a correction with attribution.

This post is a 90-day reorg census across that fleet plus the smaller GCP comparator cohort: every consensus client (CC) paired with every execution client (EC), against the same Ethereum mainnet, with per-host normalization. The questions we answer: which Prometheus counter to trust for which question, why the same EC behind two different CCs produces very different reorg numbers, and why a "zero reorgs" reading on some clients is silence rather than safety.

Key findings at a glance:

• Prysm increments beacon_reorgs_total 8× more often than Lighthouse over 7 days. The gap shrinks to 1.6× over 90 days.
• Lodestar's beacon_reorgs_total is 0 for the entire 90-day window. Its decline-reason counter fires roughly 54 times per host per week.
• The same EC behind two CCs produces 2–5× different counts: Prysm + Besu reports 69 per host vs Prysm + Nethermind 280.
• Geth is the only EC in our fleet whose Prometheus reorg counter increments at all. Nethermind and Reth export the metric but it never increments; Besu, Erigon, and Ethrex don't export one at all.
• A single fixed beacon_reorgs_total alert threshold does not port between consensus clients. Re-baseline per CC × EC pair.

What we measured: the StereumLabs Ethereum client fleet​

The bare-metal NDC2 fleet in Vienna carries the full pairing matrix: every CC family (Grandine, Lighthouse, Lodestar, Nimbus, Prysm, Teku) crossed with every EC family (Besu, Erigon, Ethrex, Geth, Nethermind, Reth). Each CC family runs on 6 hosts, one per EC pairing. The GCP comparator cohort runs only Geth on the EC side, paired with one supernode variant of each CC (introduced for PeerDAS custody-group experiments). The full setup is documented in the StereumLabs measurement stack post.

All counters in this post come from our prometheus-cold datasource over rolling 7-day, 30-day, and 90-day windows ending around 2026-05-26. Per-host averages divide each family's total reorg count by its host count. Grandine currently carries two extra non-production hosts on a custom image; those are excluded from every comparison below. Queries are reproduced in the methodology section.

The reorg-relevant metric vocabulary, by client, is heterogeneous:

That table alone tells most of the story. Before we look at a number, we already know that "is my client reorging more than it used to" is a question with six different correct answers.

What each Ethereum consensus client reports as a reorg​

Across all six CCs, 6 production hosts each, here is what beacon_reorgs_total accumulated over our 7-, 30-, and 90-day windows:

• Prysm: 1,373 reorgs in 7d (229 per host), 502 per host over 30d, 1,002 per host over 90d
• Teku: 282 in 7d (47 per host), 209 per host over 30d, 759 per host over 90d
• Nimbus: 210 in 7d (35 per host), 172 per host over 30d, 672 per host over 90d
• Lighthouse: 171 in 7d (29 per host), 159 per host over 30d, 638 per host over 90d
• Grandine: 153 in 7d (26 per host), 163 per host over 30d, 668 per host over 90d
• Lodestar: 0 in 7d, 0 over 30d, 0 over 90d

A few observations:

• Prysm increments roughly 8× more often than Lighthouse over 7 days. Over 30 days the multiple drops to 3.2×, over 90 days to 1.6×. Either Prysm had a spike in recent days (it did, see below), or Prysm's counter responds to short-lived events that other clients smooth out across longer windows. Both apply here.
• Grandine, Lighthouse, Nimbus, and Teku land within a 1.8× band over 7 days (Grandine 26, Teku 47 per host) and tighten to a 1.2× band over 90 days. That is the tightest cluster in the dataset.
• Lodestar increments the counter zero times. Not "low": exactly zero, across 90 days and 6 hosts. The Lodestar section below describes what it does instead.

The 7-day-to-90-day collapse on Prysm is the first practical lesson: short reorg windows can mislead. In this single 7-day window Prysm registered two sustained spikes that no other CC registered and that no EC-side counter registered either. An alert calibrated to a Lighthouse-shaped baseline would have fired twice on Prysm during this week alone, both times without a chain event behind it.

Cross-CC divergence at this scale is not unique to reorgs. The same six consensus clients showed a 50× spread in blob gossip verification latency on identical hardware. Different metric, different mechanism, same conclusion: a shared metric name does not guarantee a shared definition.

What each execution client reports: Geth, Nethermind, Reth, and the rest​

The EC side is simpler, and emptier. Host counts vary by EC: Geth runs on 13 hosts (7 NDC2 + 6 GCP comparator), Besu on 7, all others on 6.

• Geth: chain_reorg_executes increments. 7d: 414 reorgs (32 per host); 90d: 7,383 (568 per host).
• Nethermind: nethermind_reorganizations is exposed, scraped, and stays at 0 across the 90-day window.
• Reth: reth_blockchain_tree_reorgs is exposed, scraped, and stays at 0 across the 90-day window.
• Besu, Erigon, Ethrex: no Prometheus reorg counter exported at all.

Geth is the only EC in our fleet whose counter increments. Nethermind and Reth carry the metric in their vocabulary, both series are registered and scraped successfully, but neither has incremented in 90 days on any host. We did not review either client's source to determine whether the increment path is unreachable on mainnet, intentionally gated, or implements a stricter definition of "reorg" than Geth does. That distinction is for the client teams to clarify. (Neither Nethermind nor Reth is deployed in our GCP cohort, so the 90-day window covers only the NDC2 hosts for those two clients.)

Besu, Erigon, and Ethrex do not export a Prometheus reorg counter at all. We have not in this round audited the DEBUG-level logs of those three clients for reorg events; it is possible that the events are visible there even though no counter is exposed. What we can say is that no Prometheus alert rule lives over them for this question today.

EC-side observability gaps are an existing pattern on this fleet. The EC sync-speed comparison found that initial sync time ranged from 1.5 hours to over 8 days on identical hardware across the same six clients, partly because each one logs and exports its sync progress differently. Reorg counters are the steady-state analogue of that observation.

The CC × EC pairing matrix: where reorg counts diverge most​

The strongest finding in this dataset is not at the CC level or the EC level. It is in the CC × EC pairings.

The Lighthouse + Erigon cell is a data gap: two hosts are up in that pairing, but beacon_reorgs_total returned no series for them in the queried window. Either the counter has not registered on those hosts yet (a recent deployment), or there is a pairing-specific instrumentation gap. We flag the gap rather than impute a number.

Three patterns emerge from the matrix:

1. Most CCs split their pairings into two tiers. For Lighthouse, Nimbus, Teku and Grandine, the same metric counts roughly twice as often when paired with Geth, Nethermind or Reth than when paired with Besu or Ethrex. Nimbus, for example, sits at 17–18 per host with Besu and Ethrex and at 35–36 with the other four ECs. The split looks like an EC-side property propagating into the CC counter, not a CC property.
2. Prysm + Besu and Teku + Reth are real outliers. Prysm + Besu is 69 per host, while every other Prysm pairing lands between 114 (Ethrex) and 280 (Nethermind), a 1.6–4× spread that is not visible in any other CC row. Teku + Reth sits at 100 per host, while every other Teku pairing lands between 18 and 38, a 2.6–5.5× elevation.
3. Prysm operates on a different scale. Even the lowest Prysm pairing (Besu, 69) is higher than the highest pairing of any other CC except Teku + Reth.

The same metric (beacon_reorgs_total) responds to the paired EC's behaviour, not just to the chain. Block import timing, attestation late-arrival rates, and engine-API forkchoice update latency on the EC side feed back into the CC's fork-choice loop, and the counters reflect that. We previously characterised the same EC↔CC coupling from a different angle in the Erigon + Caplin standalone vs classic comparison, where collapsing both layers into one binary moved EVM execution by ~25% and MDBX commits by ~2× on the same engine-API timing surface, observed from the inside. We are not yet in a position to attribute the Prysm + Besu reduction or the Teku + Reth elevation to a specific mechanism; the pairings are flagged here as findings for follow-up.

Practical takeaway for operators: if you swap your EC, expect your CC reorg numbers to move. If you alert on a fixed rate(beacon_reorgs_total[15m]) threshold, that threshold needs to be re-baselined after any EC change. Otherwise you are paging yourself for a re-baselined client behavior rather than a chain event.

The Prysm reorg-rate spike: a counter responding to what?​

Across 7 days, Prysm increments beacon_reorgs_total 1,373 times. Across 90 days, 6,011. That means 23% of Prysm's 90-day total accumulated in the last 7 days. The other CCs do not show the same concentration: Lighthouse's 7-day-to-90-day ratio is 4%, Teku's is 6%, Grandine's is 4%, Nimbus's is 5%.

The hourly time series confirms two distinct sustained spikes within the 7-day window. One lasted roughly 15 hours, the second roughly 36 hours. During both, Prysm's fleet-wide rate jumped from a baseline near 0.0014 reorgs/sec (across all 6 Prysm hosts) to a sustained 0.005–0.010 reorgs/sec, a 4× to 7× elevation. No other CC counter and no EC counter registered the spikes. Geth's chain_reorg_executes stayed at its normal pulse pattern (around 0.0034 reorgs/sec across its 13 hosts) throughout both windows.

We checked the obvious explanation first: a Prysm version change during the window. The Prometheus cc_version label held at v7.1.3 continuously across all 6 Prysm hosts for the full 7 days. The spikes are not the artifact of a release.

We also cannot point to a corresponding mainnet event for either window. Public block explorers do not record deep reorgs in those periods, and our EC-side counters are quiet through them.

One hypothesis that fits the data: Prysm's beacon_reorgs_total may increment on every fork-choice head re-attribution, including transient ones that are subsequently re-overridden by the next attestation update, while the other CCs that share the metric name appear to increment only when a head change crosses some stability threshold. We have not audited any client's source to prove this. The observation is restricted to: Prysm spikes, others stay flat. Either way, a Prysm reorg-rate spike on its own is not evidence of a chain anomaly. It is consistent with the fork-choice loop chewing on late blocks under transient peering or attestation delay, with the increment policy determining whether the counter responds.

Prysm's instrumentation also shifts between releases. We previously tracked one such transition in detail in the Prysm v7.1.1 vs v7.1.2 resource report, where block-processing latency on one EC pairing changed by 78% across a single point release. A rerun of this reorg census against future Prysm releases could land in different places.

Reorg depth histograms: Lodestar vs Prysm​

Two CCs in our fleet populate a reorg-depth histogram: Lodestar (via beacon_fork_choice_reorg_distance_bucket) and Prysm (via reorg_depth_bucket, a Prysm-specific name). The other four CCs do not export an equivalent series over the queried window. Lighthouse increments beacon_reorgs_total 167 times in 7 days but does not appear to populate the matching histogram. We mention it as an instrumentation observation, not a finding about Lighthouse's correctness.

Lodestar's histogram, summed 7-day rate per bucket (sum by (le) (rate(beacon_fork_choice_reorg_distance_bucket[7d]))):

Every observation Lodestar's histogram records sits in the 1–2 slot bucket. This is consistent with Lodestar evaluating candidate reorgs internally, classifying their distance, then declining to execute them. It lines up with the 324 headBlockIsTimely decline events on the same fleet over the same window.

Prysm's histogram, same 7-day rate:

Prysm's distribution has weight all the way out to the 16–32 slot bucket. Mainnet did not see a deep canonical reorg in this window. Public explorers confirm the chain stayed within single-slot reorgs throughout, and Geth's EC-side counter agrees. So Prysm's reorg_depth_bucket is recording something that is not canonical-chain depth. More likely, it is the distance along Prysm's internal fork-choice graph from the previous head to a new one, including head transitions that were never canonical. That is a reasonable thing to measure, but it is not the same thing Lodestar is measuring.

Two clients can both label a histogram "reorg depth" and be measuring different things. The bucket boundaries are themselves a hint about what each team expected to see. Lodestar's spec-aligned 1, 2, 3, 5, 7, 10, 20, 30, 50, 100 versus Prysm's 1, 2, 4, 8, 16, 32 reflect different priors about the distribution.

Lodestar's silent counting: zero reorgs, but the evaluation pipeline runs​

Lodestar reports beacon_reorgs_total = 0 across every host, every window. But Lodestar is not idle. In the same 7-day window the depth histogram (beacon_fork_choice_reorg_distance_bucket, covered above) does receive observations, and beacon_fork_choice_not_reorged_reason_total{reason="headBlockIsTimely"} accumulates 324 increments across the 6 Lodestar hosts (54 per host) and 50 increments on the single Lodestar supernode. Both series carry non-trivial weight; only the headline counter stays zero.

The reason label name lines up with the consensus specs' should_override_forkchoice_update helper (validator spec). The simplified flow:

1. The validator sees a late block at the start of slot N+1.
2. It considers proposing on the parent of that late block instead of building on it (a "late-block reorg" override).
3. Safety checks run, including: is the head block timely enough that overriding it would not damage the chain?
4. If the head block is timely, the spec says do not override.

The label name suggests Lodestar's counter increments on that branch, but we have not verified against the source. The 7-day data on every Lodestar host:

• beacon_reorgs_total: 0
• beacon_fork_choice_not_reorged_reason_total: 54 per host (non-supernode) or 50 (supernode), all under the single label value headBlockIsTimely, no other reason labels populated
• beacon_fork_choice_reorg_distance_bucket: populated, all observations in the 1–2 slot bucket

Taken together, the three series show a CC that is actively running the late-block reorg evaluation pipeline (computing candidate distances, recording decline reasons) but declining to act on the result in this window. Lodestar's published metrics make it look like Lodestar never executes a reorg on mainnet. Whether that is a true statement about Lodestar's behavior or a missing increment on beacon_reorgs_total requires source-level confirmation, which we are requesting from the Lodestar team alongside this post. In the meantime, beacon_reorgs_total is not a portable observability primitive across CCs: a Lodestar operator who alerts on it gets no signal at all.

Cross-client reorg alerting cheat sheet​

For an operator runbook, by question:

Did our CC re-attribute its head in the last 5 minutes? Use rate(beacon_reorgs_total[5m]). Alert on sustained elevation, but only after re-baselining per CC and per EC pair. Ignore brief Prysm spikes and any reading on Lodestar.

Did the EC see a canonical chain rewrite? Use chain_reorg_executes on Geth only. Alert on any non-zero rate during sustained head movement. Treat a zero reading on Nethermind or Reth as an instrumentation gap, not as safety.

How deep was the latest reorg? Use beacon_fork_choice_reorg_distance_bucket if you run Lodestar, or reorg_depth_bucket if you run Prysm. Alert on histogram weight in buckets greater than 2 slots. Do not compare Lodestar's histogram to Prysm's; they record different things.

Did the proposer-boost reorg path trigger? Use beacon_late_block_attempted_reorgs on Prysm or beacon_safe_reorgs_total on Nimbus. Alert on sudden surges. Absence of the metric on the other CCs is not absence of the event.

Why didn't the CC reorg when we expected it to? Use beacon_fork_choice_not_reorged_reason_total on Lodestar (the only CC in our fleet that exports it). Alert on reason-label changes over time. Cross-client comparison is not possible with this metric today.

If you need a single rule that works across the fleet, use rate(chain_reorg_executes[15m]) on Geth EC hosts, with per-host thresholds calibrated to that host's 90-day baseline. Geth is the only EC in our fleet whose counter both exists and increments.

How Glamsterdam and ePBS may change reorg accounting​

The Glamsterdam hardfork is expected to enshrine proposer-builder separation (ePBS) via EIP-7732. At time of writing (May 2026), it is still in devnet stabilization per the Ethereum Foundation's April 2026 Checkpoint, with no fixed mainnet activation slot. Under ePBS, the slot gains an explicit commit-reveal cycle: the proposer commits to a builder-signed execution header in-slot, the builder reveals the payload, and a payload-timeliness committee (PTC) votes on whether the reveal arrived in time. If the PTC says no, the slot's execution payload is empty.

That mechanism creates new transient head states that today's reorg counters do not contemplate. We are not in a position to predict exact numbers, but two structural points are worth flagging now rather than after the fork ships:

• Whichever client increment policy you have today carries into Glamsterdam. A counter that increments on every fork-choice head re-attribution will see a higher post-Glamsterdam baseline than one that only increments on canonical rewrites. Operators should re-baseline alert thresholds after the fork, not before.
• The CC-side counter becomes the operational signal. With execution payloads decoupled from proposer commitments, "the chain reorged" and "the execution payload changed" partially decouple. EC-side counters like chain_reorg_executes will under-report what a validator operator cares about, and the relevant counters move to the CC side and to relay/builder observability.

The pre-Glamsterdam takeaway from our 90-day census: if you cannot answer "what does my client mean by reorg today," you will not be able to answer "did my client behave correctly under ePBS" once the fork lands.

Frequently asked questions about Ethereum reorg monitoring​

Why does Prysm report more reorgs than other Ethereum consensus clients?​

Prysm's beacon_reorgs_total appears to increment on transient fork-choice head re-attributions in addition to canonical reorgs. The other CCs that share the metric name (Lighthouse, Teku, Nimbus, Grandine) appear to increment only when a head change persists. The cross-client gap shrinks from 7× over 7 days to 1.5× over 90 days, suggesting the bigger Prysm count comes from short-lived events the other counters smooth over. We have not verified this against source code.

Does Lodestar really not reorg on the Ethereum mainnet?​

Lodestar's beacon_reorgs_total shows exactly zero across every host and every queried window. But Lodestar's depth histogram is populated and its beacon_fork_choice_not_reorged_reason_total{reason="headBlockIsTimely"} counter fires around 54 times per host per week (about every 3 hours per host). From outside, that looks like Lodestar evaluates candidate reorgs and declines to execute them. Source-level confirmation is pending from the Lodestar team.

Which Ethereum execution clients export a Prometheus reorg counter?​

In our fleet: Geth (chain_reorg_executes, increments normally), Nethermind (nethermind_reorganizations, registered and scraped but never increments), Reth (reth_blockchain_tree_reorgs, same as Nethermind). Besu, Erigon, and Ethrex do not export a Prometheus reorg counter at all. Operators have to derive reorg events from logs. This patchy coverage is one of the operational consequences of Ethereum client diversity: the same observability question gets six different answers, and three of those answers are silence.

How should I alert on reorgs across different Ethereum consensus clients?​

Per-host baselines, per CC × EC pair. A single fixed rate(beacon_reorgs_total[5m]) threshold does not port between clients: the same EC behind two different CCs produces reorg counts that differ by 2–3×. For one EC-side rule that works fleet-wide, alert on rate(chain_reorg_executes[15m]) for Geth hosts; the same metric does not exist on the other ECs. Skip alerting on Lodestar beacon_reorgs_total entirely. It is always zero.

Will Ethereum's Glamsterdam upgrade change reorg counting?​

Probably yes. Glamsterdam is expected to enshrine proposer-builder separation (ePBS) via EIP-7732, which introduces commit-reveal cycles and payload-timeliness committees. Those mechanisms create new transient head states that today's counters do not contemplate. Plan to re-baseline alert thresholds after the fork. EC-side counters like chain_reorg_executes will under-report what validator operators care about, since execution payloads decouple from proposer commitments under ePBS.

Methodology​

All queries ran against our prometheus-cold datasource over windows ending around 2026-05-26 12:00 UTC.

Per-CC family total:

sum by (cc_client) (increase(beacon_reorgs_total{cc_version!="unstable-465c727"}[7d]))

Per-host averages divide each family's total by the live production-host count (6 NDC2 hosts per CC family; the cc_version!="unstable-465c727" filter excludes the two Grandine custom-image hosts from the comparison).

Per-pairing average per host:

avg by (cc_client, ec_client) (increase(beacon_reorgs_total{cc_version!="unstable-465c727"}[7d]))

EC-side Geth counter:

sum by (ec_client) (increase(chain_reorg_executes[7d]))

Histogram bucket rates:

sum by (le) (rate(beacon_fork_choice_reorg_distance_bucket[7d]))
sum by (le, cc_client) (rate(reorg_depth_bucket[7d]))

Lodestar's decline reason:

sum by (cc_client, reason) (increase(beacon_fork_choice_not_reorged_reason_total[7d]))

Host counts as of 2026-05-26, taken from count by (cc_client) (count by (cc_client, instance) (up{role="cc", job!="node_exporter"})) and the EC equivalent. Filtering out job="node_exporter" is necessary because every CC and EC host runs node_exporter under the same role label, which otherwise double-counts.

• CC (NDC2): Grandine 8 (6 stable v2.0.4 + 2 custom-image unstable-465c727), Lighthouse 6, Lodestar 6, Nimbus 6, Prysm 6, Teku 6.
• CC supernodes (GCP): one host per CC family.
• EC (NDC2 + GCP): Geth 13 (7 NDC2 + 6 GCP comparator), Besu 7, Erigon 6, Ethrex 6, Nethermind 6, Reth 6.

The two Grandine custom-image hosts (unstable-465c727) are excluded from every comparison in the post; comparisons in this analysis run across the same 6 production hosts per CC family.

Lighthouse + Erigon shows up in the up query (two hosts), but beacon_reorgs_total returns no series for that specific pairing in the queried window. Likely an instrumentation or registration gap, not zero reorgs. Numbers in the per-pairing matrix divide totals by the matching count(up), so pairings with missing counter series do not contribute to the comparison.

The 90-day window covers the post-Fusaka mainnet conditions only. Pre-Fusaka data (before December 3, 2025) is in cold storage but not included here because PeerDAS introduced sufficient changes to fork-choice timing that combining the two periods would mix populations. The Fusaka hardware impact post characterizes that boundary in detail.

Dashboard panel definitions for every metric used here are linked from the consensus-client dashboards in our public catalog at /docs/dashboards/list/catalog. The per-client dashboards on grafana.stereumlabs.com each carry the reorg panels described above. We do not yet ship a cross-client comparison dashboard for these metrics, so the cheat sheet above stands in for one.

If you operate one of the clients we measured and want to clarify what your counter actually measures (particularly the Lodestar beacon_reorgs_total definition and the Nethermind / Reth increment paths), write to contact@stereumlabs.com. We will append a correction with attribution.

This post is the technical introduction to that platform: the bare-metal fleet, the metrics and logs pipeline, the label conventions that make the pairings comparable, and the in-house AI workflow that turns the resulting telemetry into the blog posts you are reading.

RockLogic publishes a separate case study on the business side of this workflow: how the same "AI on own data" pattern keeps customer telemetry inside the perimeter while still producing useful answers. This post is the technical view from the other side of the same workflow.

The measurement question​

Every Ethereum client team publishes their own benchmarks. Every cloud provider publishes their own reference architectures. Every node operator has a folder of opinions. What is rarer is a fleet where all six execution clients and all seven consensus clients run on identical hardware, identical scenarios, identical scrape cadence, and identical label conventions, with the raw data kept long enough to ask new questions about old runs.

StereumLabs exists to fill that gap. The technical choices below are not the only ones that could work, but they are the ones that keep comparability cheap, reproducibility verifiable, and post-hoc questions answerable without re-running anything.

The hardware base​

The bare-metal anchor lives in our Vienna NDC2 data center, run end-to-end by RockLogic. Bare metal cuts out three moving parts that contaminate cloud measurements: instance families that change underneath you, noisy neighbours that blur tail latencies, and IOPS guarantees that come with footnotes. For a measurement product, all three are problems we prefer not to have.

The bare-metal fleet is paired with a smaller cloud footprint in GCP. The cloud cohort exists for one specific reason: to surface, at indicator level, how much of what we measure is the client and how much is the environment. Without any cloud comparator, every measurement we publish is implicitly "what Erigon does on a bare-metal NVMe box in Vienna". With one, we can flag when a result changes meaningfully across environments. The GCP cohort is small and currently focused on Geth and a subset of pairings. It is sized to catch direction-of-effect differences, not to characterise the full performance surface of any commercial cloud. Where a finding turns on the comparator (such as the inbound-firewall observation discussed later in this post), we say so explicitly.

A typical EC plus CC pairing occupies two VMs: one for the execution client, one for the consensus client. Splitting the roles across two hosts is deliberate. It means node_exporter on each VM gives us clean per-layer resource consumption, with no need to attribute CPU or RAM to one or the other after the fact.

The math is straightforward: six EC families times six CC families is 36 pairings, plus the standalone Caplin host that runs both layers in one binary, plus the GCP cohort that mirrors a subset of those pairings. The full fleet sits at roughly 90 hosts across both deployments. The 36 pairing matrix below is the bare-metal NDC2 core; the GCP comparator widens the picture but does not change its shape.

The data pipeline​

The pipeline has three rails: metrics, logs, and dashboards. Each rail is boring on its own. The interesting part is how they are wired together so that a question asked today can be answered against data captured weeks ago.

Metrics​

Every EC and CC process exposes a /metrics endpoint in Prometheus text format. Every host runs node_exporter for OS-level counters. Scrape cadence is 15 seconds, which is a deliberate compromise: short enough to catch most state transitions, long enough that cardinality stays manageable.

We run two Prometheus tiers:

• prometheus-free: short retention, mirrored subset of metrics, available to Free-tier subscribers with a 7-day data delay.
• prometheus-cold: full metric set, no retention cap. This is where we run avg_over_time(...[7d:1h]) queries when we write a blog post.

Cold storage is what makes most of the StereumLabs blog series possible. The Caplin standalone analysis from last week pulled metrics that were captured before Caplin v3.3.10 had been added to the fleet, and ran the same queries against the new host. No re-runs, no warm-up periods, no "we'll publish in two weeks once we have data".

Logs​

Logs travel a parallel path on the same diagram above: Filebeat streams logs from every container into Elasticsearch, where Kibana and the MCP server (Model Context Protocol, the read-only query gateway our AI workflow uses; covered later in this post) consume them in the same way Grafana consumes Prometheus. Logs are the second half of what we measure, and frequently the half that surfaces things metrics cannot. The classic example: the per-block head updated line in Erigon contains four fields (execution time, commit time, age, mgas/s) that no /metrics endpoint exposes. Without log capture, the Caplin standalone vs classic comparison would have been a far shallower piece.

Logs land in Elasticsearch with structured fields for container_name, host, ec_client, cc_client, and the rest of the StereumLabs label vocabulary. Fleet-wide ingest currently runs around 30,000 log lines per second with debug logging enabled across every container.

A side effect of capturing everything is that we can write log-derived percentile tables that ask questions the original log format was not designed for. We do not need to instrument the client. We just need it to log enough.

Dashboards​

Grafana sits on top of both Prometheus and Elasticsearch. Dashboards are the public face of the platform: panels with info icons that link back to the definitions page, time-aligned charts for cross-client comparison, lifecycle badges (active, experimental, legacy) so you know whether to cite a result. Plan-tier-gated access controls how much retention and how many users you get.

The naming convention is Category – Metric (Scope), units are SI, and per-core normalization is always labeled. The reason we are strict about it: the only way cross-client comparisons stay legible is if a panel titled "CPU – Utilization (EC process)" means the same thing on every dashboard regardless of which client it is plotting.

Custom labels: the comparability layer​

Every metric series in prometheus-cold carries a fixed set of StereumLabs labels:

• ec_client: execution client name (besu, erigon, ethrex, geth, nethermind, reth).
• ec_version: pinned version string.
• cc_client: consensus client name (grandine, lighthouse, lodestar, nimbus, prysm, teku, plus caplin when running standalone).
• cc_version: pinned version string.
• role: ec or cc, so a query can target the layer regardless of which process the metric came from.
• deployment: provider code (NDC2 for bare-metal Vienna, GCP for cloud).
• location: provider-specific region or zone identifier.

The result is that a query like

avg_over_time(
  node_cpu_seconds_total{role="ec", ec_client="erigon", deployment="NDC2"}[7d:1h]
)

returns exactly the slice you would ask a human for in plain English: "the 7-day Erigon EC CPU average on bare metal". No prior knowledge of the underlying scrape topology needed.

Most of the unglamorous engineering effort goes into keeping this label vocabulary consistent. It is also what made the EC P2P peering deep dive feasible: that post compared roughly 90 hosts across six EC families and two deployment classes in one chart. Without consistent labels, the same comparison would have required a custom ETL job for each section.

Neutrality and how we keep runs comparable​

Neutral measurement is a methodological commitment, not a marketing claim. The concrete rules we enforce inside the fleet:

• Start from defaults. Every client runs vendor defaults unless a documented, client-team-recommended deviation is required for stability or scenario correctness. Each deviation is annotated in the run manifest.
• Equalize budgets. Every VM in a comparison cohort gets the same vCPU, RAM, and disk allocation. The Caplin standalone post explicitly called out the asymmetry where it existed, because pretending it did not is the fastest way to publish noise as signal.
• Pin versions. Versions are explicit in the labels, in the manifest, and in the dashboard headers. When a client team ships a new release, we add a new host. We do not silently upgrade the existing one underneath the data.
• Reject skew. NTP is enforced across the fleet. Runs with material clock drift during the measurement window are excluded from comparisons and flagged in the run notes.
• Document trade-offs. Pruning modes, cache sizes, peer caps: anything that affects interpretation is in the run notes, not the appendix of an internal wiki.

A run manifest is the concrete artifact that carries all of this. It records the pinned client versions, the host specs (vCPU, RAM, disk, hypervisor), the start and end of the measurement window, any deviations from defaults with their justifications, and the acceptance checks that gate publication. Together with the panel-definition links from the dashboard each chart was drawn against, the manifest is what makes a published result re-runnable later. Enterprise subscribers get the full manifest attached to every dashboard view; public posts cite the relevant subset in their methodology notes.

The hard part of this work is rarely the infrastructure itself. Most of the effort goes into the rules that keep runs comparable: label vocabulary, version pinning, deviation logging, scenario manifests, deciding what counts as "enough" data. The RockLogic AI case study makes a parallel observation: hooking a model up to the data was an afternoon; defining what a useful answer looks like took weeks. Spinning up Prometheus and Grafana is well-documented and quick. Turning that into a substrate where a researcher in 2027 can re-run today's query against the same definitions and get a comparable result is the ongoing commitment, and it is what subscribers are paying for.

How this fits alongside other measurement work​

StereumLabs is not the only public effort to characterize Ethereum client behavior, and not the first. A non-exhaustive list of work that we read, learn from, and consider complementary:

• MigaLabs' Armiarma: a libp2p crawler focused on Ethereum's CL network, producing peer-set and gossipsub data. MigaLabs also publishes broader network analysis at migalabs.es. Their network-wide view is wider than ours; our per-pairing resource view is more granular.
• The EF's ethpandaops: Kurtosis-based reproducible Ethereum devnets (ethereum-package), the ethereum-metrics-exporter, checkpointz, mainnet monitoring tooling, and related infrastructure. Their toolset covers a much wider operator surface than ours; our addition is the controlled cross-client comparison on identical hardware.
• Probe-Lab: P2P network measurement across libp2p ecosystems (Ethereum, IPFS, Filecoin and others) using their own tooling stack (Parsec, Nebula, Hermes, Ukla). Different protocol layers, complementary insights.
• MEV-Boost relay observability via projects like relayscan.io and mevboost.pics: payload-flow and relay-bid visibility that sits one level above what we measure.
• Client-team published benchmarks: Nethermind, Sigma Prime, the Erigon team and others publish their own perf work. Our value-add is running their releases head-to-head on neutral hardware rather than each team optimising their own demo setup.

Our niche is the cross-product: every EC and CC implementation, paired with every other, on identical hardware under identical scenarios, with the raw telemetry retained long enough to ask new questions of old runs. Where another effort goes wider or deeper on one axis, we cite it; where our data complements theirs, we say so.

What this stack has produced​

What matters is what the platform makes visible. A non-exhaustive recap of recent output:

• The Caplin standalone vs classic Erigon comparison found that the monolithic node executes EVM blocks 12 to 31% faster at the median but pays a 2x penalty on MDBX commits, ending up roughly tied at end-to-end p50 across 50,000 sampled blocks.
• The EC P2P peering deep dive found that Reth maintains over 5x the peer count of Besu under stock defaults on our fleet, and that in our GCP cohort the default cloud firewall rules silently blocked DevP2P inbound on Geth. An operator using the same provider defaults would inherit that artifact unless they opened the port explicitly.
• The EC sync speed comparison quantified the gap between the fastest and slowest initial sync at over an order of magnitude on identical hardware.
• The Nimbus v26.3.1 block-building post characterised how each EC behaves when Nimbus drives block-building duties, using a shadow setup that mirrors 1,000 validator pubkeys across five EC pairings over a 48-hour window.
• The Teku cross-version analysis traced how resource consumption shifted across three Teku releases.
• A two-week AI-assisted security-audit campaign (May 4 to May 18, 2026) used the fleet as a live test-bed, producing 54 finding documents, 6 Ethereum Foundation Bug Bounty submissions, 13 paste-ready upstream PR drafts across 7 client projects, and 24 Kurtosis devnet PoCs. Several findings (gossip-rejection spikes, engine-API timeout storms, P99 latency divergences) surfaced first as anomalies in the Prometheus and Elasticsearch stack. A dedicated post will cover the campaign.

None of these required new instrumentation. They required asking new questions of data that was already in prometheus-cold and Elasticsearch. That is the payoff of the upfront effort to keep everything labelled, retained, and reproducible.

The AI workflow: closing the loop​

This is where the post connects back to the RockLogic case study.

Every StereumLabs blog post you read carries stereumlabs-ai as a co-author. The label is literal. Anthropic Claude has read-only MCP access to our Prometheus and Elasticsearch instances, drives the queries, reads the results, drafts the prose, and proposes the tables. A human editor reviews, corrects, and approves before publication. The case study describes that workflow from a business angle: data stays in-house, output is cited and reproducible, weeks of analysis collapse into hours. This post is the technical view of the same loop from the other side.

The wiring looks like this:

A few properties of that loop that are worth being explicit about:

• Raw telemetry stays on EU infrastructure. The MCP (Model Context Protocol) server runs inside the same perimeter as Prometheus and Elasticsearch. What crosses the boundary to the Claude API is a curated query slice and its result, not the underlying logs or metric series.
• Citations are mandatory and auditable. Every number that lands in a published post is required to come back from a query the model can name and a result the editor can re-execute. The editor can re-run the cited PromQL or Elasticsearch query before publication to confirm the number, and reject the claim if it does not reproduce. "The system claims" is not a publishable sentence.
• Context is trimmed aggressively. A 7-day avg_over_time query returns one number, not a histogram. A head updated log extraction returns a summary table, not 50,000 lines. The model is given the smallest input that answers the question. The trimming is a structural mitigation, not a guarantee of correctness; it makes hallucinated numbers much harder to slip past a citation check, but it does not eliminate them.
• Human review is the final filter. Stefan Kobrc (Founder, RockLogic) is the named editor on every StereumLabs post. The editor's job is to re-run a sample of the cited queries, sanity-check the conclusions against domain knowledge, and reject claims that the data does not support. When a correction lands after publication, it is logged in the post's revision history rather than silently rewritten.
• Format compliance is enforced. The post you are reading uses the same frontmatter, the same table style, and the same link conventions as every other StereumLabs post because the workflow checks for it before output.

The same pattern is what we offer to customers in the RockLogic case study. We run it on ourselves first, on our own infrastructure with our own data, before pointing it at anyone else's.

What is harder than it looks, and what we cannot fully control​

Five observations from running this platform. The first three are operational costs we accept. The last two are biases worth flagging so that readers can weigh them against their own deployment.

Metric heterogeneity across clients is the dominant operational cost. Caplin v3.3.10 ships a comprehensive libp2p_rcmgr_* family but no current libp2p peers gauge. Prysm exposes connected_libp2p_peers{agent="..."} with per-implementation breakdowns. Lighthouse, Lodestar, Nimbus, Teku, and Grandine each use different metric names for what is conceptually the same thing. Every cross-client query is implicitly a translation layer. We maintain that layer as part of the dashboard panel definitions and as part of the AI workflow's instruction set.

Log format chaos is the second cost. Some clients log structured JSON, some log key=value, some log unstructured prose, some log all three depending on subsystem. Stripping ANSI color codes is a normal preprocessing step. Writing a parser like the one for Erigon's head updated line is the easy part; validating it against edge cases is what takes time.

Keeping runs comparable is the third cost, and the largest. The same conclusion the AI case study reaches about its own workflow: the hard part is not the technology but the editorial spine. For us, that means resisting the temptation to publish numbers from runs that were "almost equivalent". Almost is where bias lives.

Single-site bias. All bare-metal hosts sit in one Vienna data center (NDC2) and share an upstream BGP path and ASN-level peer reachability. That means peer-set composition, geographic latency to other mainnet nodes, and inbound discoverability are correlated across the fleet. Findings about peer count, peer diversity, or P2P churn carry an implicit "as seen from one network position". We add the GCP cohort to break that correlation in one direction, but two environments are an indicator, not a characterisation. Reproducing a finding from a different site or ASN is one good way to test how site-bound it is.

Scrape cadence has a tail-latency floor. 15-second Prometheus scrapes are sufficient for resource averages and steady-state behaviour, but they are too coarse for tail-latency analysis. GC pauses, individual slot timings (Ethereum slots are 12 seconds, so aliasing is possible on slot-aligned metrics), and sub-second spikes are visible only through log-derived percentiles, not through the metrics graph. Where we publish p99 numbers, they come from log parsing, not from histogram_quantile on 15s buckets. The relevant posts call this out where it affects interpretation.

Plans, access, and what is public​

Three subscription tiers, structured around what we can responsibly support at each price point:

The tiers map to different reader profiles. Free is the right level for ecosystem observers, students, and curious operators who want to read the published numbers and verify our claims against a delayed but otherwise full dataset. Pro is sized for client teams, research groups, consultants, competitive-intelligence shops, and press who need real-time access and unlimited retention across the fleet but do not need their own cohorts. Enterprise is for institutional operators and infrastructure providers who want to commission custom runs against their own questions, ingest data into their own pipelines via exports, and get a contracted scope of work with named contacts. Prices are EUR, excluding VAT, per the public plans page.

The Free tier exists because we think the public Ethereum ecosystem benefits from having neutral measurements visible to anyone. Paid tiers fund the bare-metal fleet, datacenter, hardware refresh, on-call rotation, and ongoing engineering effort that keeps the data reproducible week over week.

The blog and the public dashboards are open without subscription. Every post links back to the definitions, the methodology, and the scope page, so a critical reader can trace any claim back to the query that produced it.

Who this is for​

• Client teams comparing your own release against the rest of the field on identical hardware, without having to stand up the rest of the field yourself.
• Institutional node operators sizing infrastructure decisions against measurements taken at production-equivalent scale.
• Researchers running cross-client studies who need a reproducible substrate that is documented down to the firmware version.
• Security researchers and bug-bounty hunters using cross-client production observability to surface anomalies that source review alone misses.
• Builders of derivative tooling (indexers, RPC providers, MEV searchers) who want to know how the client they depend on behaves under load.

If you are in one of the above categories and want a custom dimension of the fleet measured against your specific question, Enterprise scope custom runs are designed for that. Start with the question, not with the SKU. Write to contact@stereumlabs.com describing what you want to learn, and we will scope it together, either inside the current scenario matrix or by spinning up a new VM cohort.

Summary​

Where to go next​

If you read one thing next, make it the Caplin standalone vs classic Erigon comparison. It is the most recent end-to-end example of this stack answering a concrete question, and it shows what the manifest, the labels, and the AI workflow combined produce.

After that:

• Browse the blog archive for previous analyses across the EC and CC fleet.
• Follow RockLogic on X for new posts as they ship.
• For the methodology in depth, read Purpose and Scope and the Client Metrics List.
• For the business-side companion to this post, see the RockLogic AI on own data case study.

If you want to talk about a custom measurement, a Pro or Enterprise account, or a question you would like the fleet pointed at, write to contact@stereumlabs.com. Tell us the question, not the SKU, and we will scope it with you.

Most operators run those two layers as two separate binaries that talk over the engine API. Caplin v3.3.10 collapses them back into a single Erigon binary. We added a Caplin standalone host to the StereumLabs fleet on April 8, 2026, and let it run alongside the classic Erigon plus CC pairings under identical mainnet conditions. This post reports what we measured: where the monolithic architecture wins, where it pays a tax, and where it leaves observability holes.

Setup​

The standalone deployment runs on a single bare-metal VM in our Vienna NDC2 cluster. Caplin runs as part of the Erigon binary with the consensus and execution layers sharing one executable, one MDBX database, and one combined libp2p plus devp2p networking stack. The host has 16 vCPU, 31.3 GB RAM, and 1.9 TB of NVMe storage, running Ubuntu 22.04.

The classic Erigon combos run as two separate binaries on two VMs per pairing. We split EC and CC by design across two hosts so that node-exporter measurements give us clean per-layer resource consumption. The EC side gets 12 vCPU, 15.6 GB RAM, and 1.9 TB. The CC side gets 8 vCPU, 11.7 GB RAM, and 527.8 GB. Combined, that is 20 vCPU, 27.3 GB RAM, and roughly 2.4 TB of disk per pairing.

That asymmetry matters. The standalone has 20% fewer vCPUs and 21% less disk, but 15% more RAM concentrated on one host. Any direct comparison has to account for it. We summed the resource consumption across the CC and EC VMs of each classic pairing and compared the total to the standalone single host.

We measured resource and stability metrics over a 7 day window using avg_over_time(...[7d:1h]) against our cold Prometheus datasource. Block processing latency comes from parsing 50,165 head updated log lines extracted via Elasticsearch, covering the trailing 24 hours across the standalone and all six Erigon plus CC pairings.

All six classic Erigon EC instances are paired with the same CC versions: Lighthouse v8.1.3, Lodestar v1.42.0, Nimbus v26.3.1, Prysm v7.1.3, Teku 26.4.0, and Grandine 2.0.4. Both the standalone and the classic ECs run Erigon v3.3.10. For the StereumLabs label conventions and metric availability per client, see Client metrics scope.

Resource footprint​

The first question is whether one binary holding two layers consumes more or less than two binaries holding one layer each.

Caplin standalone lands at or near the median of the Erigon plus CC pairings on every dimension except network, where it consumes well under half what the classic combos do. That gap is structural. In our split deployment, every engine_forkchoiceUpdated, engine_newPayload, and engine_getBlobsV2 call traverses the network between the CC and EC VMs. In the standalone binary, those calls become in-process function invocations that never touch a NIC.

Memory deserves a closer look. Caplin's container log reports Rss=26.1GB Pss=26.1GB Anonymous=11.7GB Swap=3.7GB, which seems to contradict the 13 GB we measured from node_memory_MemTotal_bytes minus node_memory_MemAvailable_bytes. Both numbers are correct. MDBX is memory-mapped, and roughly 14.5 GB of Erigon's RSS is shared-clean page cache that the kernel can reclaim instantly under pressure. MemAvailable correctly counts those pages as free. The 13 GB figure represents the working set that cannot be evicted without forcing reads from disk.

The 3.83 GB swap is also nothing unusual. Every classic Erigon EC swaps between 2.4 and 2.8 GB on its own, and the paired CC adds another 0 to 2.1 GB. Combined, the Erigon plus CC swap usage ranges from 2.5 to 4.6 GB, putting Caplin's 3.83 GB right in the middle. This is normal MDBX cold-page eviction behavior.

The peer story​

Caplin standalone runs two separate P2P networks from a single binary. devp2p for execution layer peers, libp2p for the consensus layer. Each has its own peer count.

On the execution side, devp2p reports 64 connected peers, split evenly across the eth/68 and eth/69 protocol versions. The classic Erigon EC instances on the same fleet report 42 to 43 peers each. Same Erigon binary, same network, same configuration philosophy, but the standalone consistently maintains roughly 50% more EC peers. The likely cause is the larger CPU and memory budget on the standalone host. For more on how the various ECs behave at the P2P layer, see our EC P2P peering deep dive.

On the consensus side, the picture is harder to extract. Caplin v3.3.10 does not expose a current libp2p peer count gauge in Prometheus. The metric appears only in the periodic container log line P2P app=caplin peers=127. We had to switch to log parsing to recover the value.

Caplin sits in the upper-middle band, below the aggressive peer collectors (Grandine, Lodestar, Lighthouse) but well above the more conservative implementations.

A side benefit of Prysm's connected_libp2p_peers{agent="..."} metric is that we can ask the Prysm fleet how many Caplin nodes it sees in the wider network. The answer is sobering: across all five working Prysm pairings combined, only 7 peers identify as agent="erigon/caplin". Caplin nodes remain rare in the mainnet peer graph. Some of those 7 may even be our own standalone connecting to multiple Prysm instances on the same datacenter network. Standalone Caplin is still a young deployment pattern.

Block processing: where Caplin wins​

To compare execution performance precisely, we parsed every head updated log line from the trailing 24 hours on the standalone and all six classic Erigon EC instances. The line format gives us four numeric fields per imported block: arrival age relative to slot start, EVM execution time, MDBX commit time, and the moving average gas throughput.

After parsing, we have between 5,113 and 7,174 samples per host, totaling roughly 50,000 blocks. The dataset is large enough that the percentile estimates are stable.

EVM execution time​

This is the time from the block being received and verified to its transactions being fully executed by the EVM, before the database commit phase.

At the median, Caplin executes blocks 12% faster than the next-fastest pairing (Lodestar) and 31% faster than the slowest (Teku). At the 90th percentile the lead is similar. At the 99th percentile the standalone still leads, though by a smaller margin.

The same story shows up in throughput. Mgas per second is the metric Erigon reports as a 12-block moving average inside the head updated line.

Caplin processes 14% to 30% more gas per second than any classic Erigon pairing across the percentile range. The Erigon binary is identical on every host, so the speedup is some combination of three things. The standalone has four extra vCPUs (16 vs 12). It avoids engine API marshaling because the CL hands payloads to the EL through in-process calls instead of JSON-RPC. And state lookups during execution skip serialization across the engine API boundary. Our data cannot fully separate the architectural contribution from the hardware contribution, but both push in the same direction.

Block processing: where Caplin pays​

Now the unflattering side. After execution, Erigon writes state changes to MDBX. In a classic deployment, this commit phase covers only EL state. In the standalone, the same phase also has to persist consensus-layer updates from the imported block, including beacon block headers, attestation aggregations, sync committee participation, and validator state changes.

Caplin's commit takes more than twice as long. The penalty is consistent across percentiles and time of day. The most plausible explanation is workload size: the standalone has to write more state to MDBX during the same block boundary because it owns both layers. Whether that happens as one MDBX transaction, two, or several sequential ones is something we did not verify at the source level. The metric is the total time the commit phase reports, regardless of how many transactions sit inside it.

End-to-end: a wash at the median, slightly long-tailed​

Adding execution and commit gives the total time to fully process a new head block. This is the number that matters for downstream signaling, RPC freshness, and validator readiness.

At the 50th percentile end-to-end, Caplin is essentially tied with the fastest classic pairings. Lodestar plus Erigon edges it out by 25 ms, which is well within sampling noise. Caplin beats every other pairing by 7 to 129 ms.

At the 99th percentile, the picture inverts slightly. The standalone's longer commit pushes its tail above some of the classic pairings, even though its execution tail is among the fastest. The total p99 spreads from 5485 ms (Lodestar plus Erigon) to 5821 ms (Nimbus plus Erigon), with Caplin at 5752 ms.

So the honest framing is this: the monolithic node executes faster, commits slower, and ends up roughly equivalent at the median. The execution win is real but the commit penalty cancels most of it. Caplin's advantage is that it gets there with one binary instead of two.

Block age, the time from slot start to when the EC first observes the block via gossip, is identical across all hosts: p50 of 3 to 4 seconds, p99 of 9 to 12 seconds. The network is not the differentiator here. Whatever happens after a block arrives is what we are measuring.

Stability and log hygiene​

Over the 7 day window, the Caplin standalone process did not restart once. Its current uptime is 9.7 days. Internal RPC failure counter (rpc_failure) reads zero. The single warning we found in the entire week was a forkchoice was invalid message at 06:53 UTC on May 8, an isolated event with no follow-up symptoms.

We compared container log volumes and warning rates across the standalone and all six classic Erigon EC hosts. The classic ECs ship between 123,000 and 154,000 log lines per week. The standalone ships 100,744. Less log volume on a setup that handles both layers seems counterintuitive until you look at what the classic ECs are saying.

The dominant Erigon log pattern that appears in classic ECs but not in the standalone is [NewPayload] Handling new payload, fired once per engine API call from the CC. On the Teku-paired Erigon EC we measured 28 of these in 5 minutes. Extrapolated, that accounts for roughly 336 lines per hour, or about 56,000 lines per week of pure engine API chatter. Take that out and the classic ECs fall close to Caplin's volume.

Zero ERROR-level lines and zero panics across the entire Erigon family for 7 days. Both architectures are equally calm in steady state. The only meaningful difference is that the standalone has fewer log lines to ignore.

Database and storage efficiency​

The Caplin standalone reports an MDBX chaindata size of 34.34 GB. Five of the six classic Erigon EC pairings sit between 20.05 GB (Lodestar pair, recently resynced) and 39.94 GB (Prysm pair, with extra unprune'd cache), with most clustering around 34.4 GB.

That is striking. The standalone holds the entire EL state plus the entire CL beacon state, including beacon blocks, attestations, sync committees, validator state, and randao mixes. Yet its database is the same size as an EC-only Erigon. Either MDBX's append-only B-tree representation is exceptionally efficient for the beacon-state schema, or the standalone backfill is not yet complete. Caplin v3.3.10 does not expose a backfill progress metric, so we cannot tell from outside the process which case applies.

This is the second observability gap worth flagging. Operators upgrading or restarting a Caplin standalone have no externally visible signal that historical CL state has finished downloading.

Caplin's CL-side health​

The consensus side of Caplin exposes a different metric set than dedicated CCs, but the few metrics that exist look healthy:

The attestation aggregation quality at the 50th percentile is 99.5%, meaning Caplin is folding nearly every unique-validator attestation it sees into the largest available aggregate. Attestation block processing time of 65 ms is faster than most CC implementations report for similar work.

Block proposal metrics (block_producer_delay) read NaN. The Caplin standalone host is one of the few in our fleet that runs without attached validator keys, so it never proposes blocks. The classic Erigon plus CC pairings on Vienna NDC2 do have validators attached on their CC sides, but that workload is on the CC, not on the EC, and it does not influence the EC commit numbers we measured. Comparing block proposal performance directly between the architectures requires a future investigation that runs Caplin standalone with attached validators. The Nimbus block-building work in our Nimbus v26.3.1 post gives a sense of what that comparison would look like once we have the data.

Architectural trade-offs that don't show up in metrics​

Two operational properties of the standalone are real but not measurable from the outside.

The first is failure-domain coupling. With separate binaries (whether on one VM or two), an EC restart leaves the CC running and able to continue gossiping, observing the chain head, and serving its own RPC clients. The CC may flag the EC as offline, but it does not crash. In the standalone, an EC restart is a CC restart by definition. There is no degraded mode where one layer keeps operating while the other recovers.

The second is upgrade granularity. With separate binaries, you can update the EC while leaving the CC stable, or vice versa. This matters when one layer ships a critical fix and the other is in a regression window. The standalone bundles the upgrade. There is no path to upgrade only the EL or only the CL while keeping the other version pinned.

Neither point is a defect, both are intentional consequences of the design choice. They are worth weighing against the resource and stability gains.

Who is this for?​

Erigon has a strong reputation as the most efficient archive-class Ethereum execution client. Its compact MDBX storage and pipelined sync turn full historical state into a workable handful of terabytes where many other clients require multiples of that. We documented some of that elsewhere in Execution client sync speed. Caplin standalone extends that strength by adding the consensus layer to the same binary. The operational unit becomes a single binary holding everything: full historical EVM state, beacon chain history, and a complete view of Ethereum from genesis to tip.

That makes Caplin standalone a particularly good match for:

• Archive node operators wanting the smallest possible disk and CPU footprint for a complete node.
• RPC providers prioritizing operational simplicity, predictable resource ceilings, and a single point of telemetry.
• Indexers, subgraph operators, and data extractors that need to query both EL state and CL beacon data from the same host without engineering two-binary coordination.
• MEV searchers and analytics platforms consuming real-time chain state plus historical data.
• Solo stakers who are comfortable accepting failure-domain coupling in exchange for a smaller hardware bill and simpler runbooks.

It is less appropriate for:

• High-availability validator infrastructure where you want the CC to keep functioning while the EC is being restarted, replaced, or upgraded.
• Operators with strict change-management policies that pin EL and CL upgrades on independent cadences, particularly during regression windows.
• Multi-CC redundancy patterns where running a different CC alongside the primary protects against bugs in one implementation. Caplin replaces the CC role, it cannot run alongside one.

For pure staking-only setups without external RPC consumers, the case for the standalone is genuinely close. The resource efficiency is real, the stability so far has been very good, and the operational surface area is smaller. The case against it comes down to one question: how much do you value being able to recover one layer while the other keeps working? If the answer is "a lot", the split deployment still has a meaningful edge. If the answer is "less than I value the simpler operations", the standalone earns its place.

Summary​

We will revisit this comparison once Caplin v3.3.10 has accumulated enough peer adoption to escape the seven-peers-in-our-Prysm-fleet floor, and once we can measure block proposal performance with attached validators on a standalone host. Until then, the standalone holds up well as a production candidate, the trade-offs are exactly where the architecture says they should be, and the audience it fits best is broader than the staking-only crowd usually assumes.

Methodology notes​

Resource and stability metrics queried from our cold Prometheus datasource using avg_over_time(...[7d:1h]) instant queries evaluated at the end of the 7 day window. Host-level metrics filtered with device!~"lo|veth.*|docker.*|br.*" to exclude virtual interfaces. Process-level metrics filtered by job to separate Caplin and Erigon scrapes from node_exporter.

Block processing latencies extracted from Filebeat-shipped container logs via Elasticsearch. We parsed head updated log lines using regex matching for execution=, commit=, age=, and mgas/s= fields. Sample sizes are 5,113 to 7,174 blocks per host over 24 hours. ANSI color codes stripped before parsing.

Peer counts on the consensus side use a mix of Prometheus gauges and log-extracted values. Caplin standalone CL peers come from the P2P app=caplin peers=N log line emitted once per minute, since v3.3.10 does not expose a current libp2p peer gauge. Cross-fleet Caplin visibility uses Prysm's connected_libp2p_peers{agent="erigon/caplin"}.

For the StereumLabs label conventions and how to point your own dashboards at our data, see Build your own dashboards. For prior context on EC behavior under similar conditions, see our posts on EC P2P peering, EC sync speed, and Teku cross-version analysis.

If you want a custom version of this analysis for your own fleet or a deeper look at any single dimension, write to us at contact@stereumlabs.com.

Why P2P peering matters​

When people talk about Ethereum client diversity, the conversation usually centers around "what percentage of the network runs Geth?" But there's a deeper question that rarely gets asked: what does each client actually see?

Every execution client maintains its own set of P2P connections through the DevP2P protocol. These peers are the node's window into the Ethereum network. Through them, the node receives new transactions, propagates blocks, and participates in the state sync protocol. If your peers are slow, your node sees transactions late. If your peers all run the same client, a single bug could blind you.

We set out to answer a set of straightforward questions using our NDC2 bare-metal fleet in Vienna and GCP cloud instances (roughly 90 hosts running all 6 execution clients paired with all 6 consensus clients):

• How many peers does each EC maintain?
• Who are those peers? What clients do they run?
• How fast does each EC cycle through its peer set?
• Does the consensus client you pair with your EC influence its P2P behavior?
• What happens when DevP2P inbound ports are blocked?
• What does all of this mean for Ethereum's network health?

The fleet and the data sources​

Our analysis combines two data sources: Prometheus metrics (7-day rolling averages via avg_over_time(...[7d:1h]) queries against our prometheus-cold datasource) and Elasticsearch container logs shipped by Filebeat from every EC container on the fleet. We've previously covered how these clients differ in sync speed and block building. This time, we're looking at how they differ in who they talk to.

Each EC exposes peer data differently. Some have rich Prometheus gauges; some log periodic peer reports; some give us almost nothing. Let's focus on what we actually found.

Peer counts: from 25 to 130​

The first and most basic question: how many peers does each EC maintain?

The spread is striking: Reth maintains over 5x the connections Besu does. Every client except Erigon runs at or near its configured maximum, which means the peer limit is the binding constraint. Erigon is the outlier, consistently operating below its limit, and we'll see why when we look at its protocol behavior.

What happens when DevP2P inbound ports are blocked?​

Our GCP cloud instances ran Geth with the same configuration as our NDC2 bare-metal fleet, but with one critical difference: the GCP default firewall did not have port 30303 (DevP2P) opened for inbound traffic. This is a very common situation: most cloud providers (GCP, AWS, Azure) block all inbound traffic by default, and operators who don't explicitly open port 30303 TCP+UDP end up in exactly this state. The Geth process itself has no idea, it starts normally, makes outbound connections, and follows the chain.

The effect on peering is dramatic:

With inbound blocked, Geth receives zero incoming DevP2P connections. Its 15 peers are entirely self-initiated outbound connections. With the port open (NDC2), the same Geth configuration receives 34 additional inbound connections, reaching the full 50-peer capacity.

The outbound count is essentially identical (~15-16 both ways), confirming this isn't a Geth software or configuration difference. It's purely a network access issue. Geth initiates the same number of outbound connections regardless of whether it can receive inbound ones, but other nodes on the network simply can't reach it.

What makes this finding more nuanced is the consensus layer comparison. The CC-side libp2p peers are nearly identical: 158 on NDC2 vs 161 on GCP. The CCs use different ports and protocols (libp2p with QUIC and TCP), and those traverse NATs and firewalls without issues. This means an operator whose CC shows healthy peer counts has no obvious signal that their EC is running with 70% fewer connections than it could have. Everything looks fine on the consensus side while the execution layer is quietly degraded.

Who are the peers?​

This is where things get interesting. Let's focus on the data we actually have: two ECs expose per-client peer breakdowns in Prometheus, and one logs it periodically.

Besu: 75% of peers are Reth​

Besu exposes besu_peers_peer_count_by_client, giving us a full breakdown:

Out of Besu's 24 peers on average, 18 are Reth nodes. Only about 2 are Geth, and Erigon barely registers. This is a remarkably skewed distribution. With three quarters of its connections going to a single client, a Reth-specific networking bug would effectively isolate Besu nodes from most of their peers.

Ethrex: a broader but Geth-heavy view​

Ethrex exposes ethrex_p2p_peer_clients with more granular labels:

Ethrex sees a much more diverse peer set. Geth leads at 43%, but the remaining 57% is spread across five other clients. Notably, Ethrex picks up "exotic" peers that other clients don't: gnode (likely a Geth fork or private deployment), various Nimbus execution layer variants, and even mempool scrapers. Ethrex appears to be the most permissive in accepting connections from non-standard clients.

Nethermind: the most transparent​

Nethermind doesn't expose per-client peer counts in Prometheus, but it does something even better: it logs a full diversity report every 5 minutes in its container output. These log lines are goldmines:

Peers: 49 | node diversity: Reth (37%), Geth (22%), Nethermind (20%), Unknown (14%), Erigon (4%), Besu (2%)

We pulled these from Elasticsearch across all 6 Nethermind instances (each paired with a different CC) and found something unexpected: the peer composition varies dramatically between instances in the same datacenter.

The Nimbus-paired Nethermind instance sees 70% Nethermind peers, while the Lighthouse-paired one sees a relatively healthy mix with Reth leading at 37%. Same client, same datacenter, same network, vastly different peer views.

Does the consensus client affect EC peering?​

This is one of the more subtle findings from the analysis. The short answer: the CC does not affect how many peers the EC gets, but it does affect which peers the EC keeps.

We checked peer counts per CC pairing across every EC on the fleet. The numbers are remarkably uniform:

The CC pairing has essentially zero influence on peer count. Every Geth instance gets ~50 peers regardless of whether it runs with Lighthouse, Nimbus, or Teku. The same holds for every other EC.

But peer composition is a different story. The Nethermind diversity logs show a 50-percentage-point swing in Nethermind peer concentration (20% with Lighthouse vs 70% with Nimbus). The Besu Prometheus data tells a similar story at smaller scale: the Grandine-paired Besu gets 18.6 Reth peers while the Nimbus-paired one gets 17.8, with corresponding shifts in Nethermind and Geth counts.

The mechanism isn't direct. The CC doesn't participate in DevP2P at all. But the CC influences the EC's startup timing, engine API call patterns, CPU and memory pressure on the shared host, and the timing of when the EC first reaches out to the discovery network. Early peer bonding creates path dependency: the peers you discover first tend to persist, and the ones that persist shape which new peers you discover next through the Kademlia routing table. Different CCs create subtly different boot conditions, and those conditions cascade into measurably different peer equilibria.

This has a practical implication: two operators running the same EC and the same CC version might still see different peer compositions, simply because their nodes booted at different moments or under different system load. Peer diversity is partly a matter of luck.

Protocol adoption: eth68 vs eth69​

Nethermind also logs the protocol version split among its peers, and Erigon reports its "GoodPeers" broken down by protocol:

Nethermind's view: eth69 dominates at 80-96% depending on the instance. The Prysm-paired node sees the highest eth69 adoption (96%), while Grandine and Lighthouse-paired nodes see around 80%. This makes sense: Nethermind and Reth (its top peers) both adopted eth69 early.

Erigon's behavior is peculiar: every pairing shows exactly 32 eth68 GoodPeers. The eth69 count varies (9-32), but eth68 is always hard-capped at 32. The standalone Caplin instance gets 32+32=64 good peers, but CC-paired instances get only ~42. This hard cap on eth68 peers likely explains why Erigon consistently runs below its connection limit: it's artificially constraining its usable peer set by protocol version.

Peer churn: who cycles and who holds​

How fast does each EC rotate through its peer connections? This matters for network resilience: too little churn means stale, possibly slow peers accumulate; too much churn means constant overhead from handshakes and state negotiation.

The spread here is staggering: a factor of roughly 50,000x between the most and least churning clients.

Besu has the most aggressive peer rotation relative to its pool size: it disconnects at 14x the rate it connects. With only 25 slots, it's constantly cycling. Average peer sessions are very short.

Nethermind receives the most incoming connection attempts (3.1/s) but maintains a steady 50. Most incoming connections are rejected because the limit is reached, not because of quality issues.

Reth is the most sought-after peer on the network. It rejects ~3 connections per second, tracks 7,400 peers in its peer table (the largest DHT view of any EC on the fleet), and has 540 backed-off peers at any given time. Its 130 connected peers are stable and long-lived.

Ethrex has the most stable peering of any client we measured: essentially zero disconnection rate. Once it connects a peer, it keeps it indefinitely. This is a double-edged sword: stable peering means reliable gossip propagation, but no rotation means stale or slow peers are never pruned.

Every EC sees a different Ethereum​

One of the most striking findings from this analysis is that no two execution clients see the same network. Even running on the same hardware in the same datacenter:

• Besu's 24 peers are 75% Reth. Its Ethereum is a Reth-dominated network.
• Ethrex's 105 peers are 43% Geth. Its Ethereum looks more like the "true" network composition.
• Nethermind's view changes depending on which CC it runs with. One instance sees 70% Nethermind peers; another sees 37% Reth.
• Erigon's protocol-level constraints give it a smaller effective peer set than its connection count suggests.
• Reth casts the widest net (7,400 tracked peers, 130 connected) but aggressively rejects newcomers.

This per-client perspective gap is a form of network fragmentation that isn't visible in aggregate statistics. The overall network might be 55% Geth, but individual nodes don't experience that average. They experience their own biased sample.

What this means for Ethereum's network health​

Client diversity at the EC layer is worse than it looks​

Aggregate client diversity numbers (e.g., "Geth runs on 55% of nodes") describe the network as a whole, but individual nodes experience a much more concentrated version. When we hear that no single client exceeds 55% network share, it sounds like the network has a safety margin. But our data shows that at the individual node level, the picture is far less balanced.

A Besu node with 75% Reth peers doesn't live in a "55% Geth" network. It lives in a "75% Reth" network. If Reth pushes a release with a networking regression, that Besu node doesn't lose 20% of its peers (proportional to Reth's global share). It loses 75% of them. Similarly, a Nimbus-paired Nethermind node with 70% Nethermind peers is exposed to a Nethermind-specific issue at 3-4x the rate the aggregate numbers would suggest.

This matters in concrete scenarios:

Client-specific bugs during chain events. When a contentious or complex fork activates (as has happened multiple times in Ethereum's history), clients occasionally disagree on block validity for a short window. During these periods, nodes need diverse peers to hear both sides of the fork quickly. A Besu node whose 18 Reth peers all follow one fork branch would receive almost no blocks from the other branch for critical minutes.

Cascading failures from bad releases. If a popular client ships a broken release that causes resource exhaustion or crashes, nodes heavily peered with that client lose their connections simultaneously. On a healthy, diverse peer set, losing one client type drops you from 50 to 40 peers. On a Besu node with 75% Reth, it drops you from 24 to 6. Six peers is below the threshold where gossip propagation remains reliable.

Transaction propagation and MEV. Transactions are propagated through the EC peer mesh. If your peers are disproportionately one client, your node's mempool is shaped by that client's transaction acceptance logic, ordering behavior, and propagation timing. For validators, this can mean seeing fewer unique transactions and missing MEV opportunities. We've previously shown how EC choice affects block building output when paired with the same CC, and peer composition is one of the contributing factors: a node with a thin, homogeneous peer set simply sees fewer transactions to include.

Consensus finality during network partitions. If a network-level event (e.g., a BGP hijack, a cloud provider outage) takes out a large fraction of one client's nodes, the remaining network's ability to finalize depends on which clients can still communicate. Nodes with homogeneous peer sets are more likely to end up on the isolated side of a partition.

The fundamental issue is that DevP2P's discovery mechanism (Kademlia DHT with distance-based routing) is optimized for finding any peers, not diverse peers. Unlike the consensus layer's gossipsub protocol, which has explicit scoring and mesh-balancing mechanisms, the execution layer has no built-in incentive for client diversity. Peers are peers, regardless of what software they run. This means diversity at the node level is largely a side effect of the overall network composition and the node's own discovery path, and our data shows that path can lead to surprisingly concentrated outcomes.

Besu's low peer limit amplifies the problem​

With a default of 25 peers and 75% of those being Reth, a Besu node has roughly 6 non-Reth connections to the Ethereum network. That's not just thin, it's fragile. At 6 connections, the loss of even 2-3 peers (through normal churn) can temporarily leave the node with only 3-4 diverse connections. Gossip propagation at that level is unreliable.

For comparison, Ethrex with 105 peers and 43% Geth still has 60 non-Geth peers even if every Geth node disappeared. That's still a functioning, well-connected node.

Ethrex's zero-churn peering needs attention​

While stable connections improve gossip reliability in the short term, never pruning peers means degraded connections accumulate. If a peer goes slow but doesn't fully disconnect, Ethrex keeps it, potentially dragging down block and transaction propagation latency over time. Every other EC has some form of peer rotation that naturally filters out underperforming connections.

Blocked inbound ports are an invisible problem​

Our GCP finding (zero inbound peers due to a default firewall that didn't have port 30303 opened) is likely representative of a large number of cloud-hosted Ethereum nodes. Operators who deploy to GCP, AWS, or Azure without explicitly opening DevP2P ports end up with only 15 outbound connections instead of 50, and their CC peer count gives no warning. A significant portion of the network may be operating with degraded EC connectivity without anyone noticing, which reduces the overall density of the gossip mesh and slows transaction propagation at scale.

The consensus layer shows what's possible​

For comparison, the consensus layer's gossipsub mesh shows much better diversity. From our Lighthouse nodes' block gossip mesh:

No single client dominates more than 37%, and the gossipsub scoring system actively maintains diversity. The execution layer's DevP2P protocol has no equivalent mechanism. This is consistent with what we observed in our blob performance analysis, where data column gossip delay was uniform across all EC pairings, confirming that the CL's P2P layer operates largely independent of the EC underneath. If DevP2P were to adopt some form of diversity-aware peer scoring (even a soft preference for connecting to underrepresented clients), the per-node diversity picture could improve substantially without requiring any consensus-level changes.

Which EC failure would hurt Ethereum the most?​

We can combine our peer dependency data with public network share estimates to model what would actually happen if a specific EC suffered a critical failure (a consensus bug, a bad release causing crashes, or a networking regression that takes all instances offline).

Current approximate EC network shares (from ethernodes.org and clientdiversity.org, April 2026):

Before we model scenarios, an important nuance: an EC going offline does not automatically mean the validator goes offline. Professional staking operators (Coinbase, Lido's curated set, institutional stakers) typically run multiple ECs with failover at the validator client or signer level. If Geth crashes, their validator client can switch to a Nethermind or Besu backup within seconds. For these operators, an EC failure causes a brief blip, not a prolonged outage.

Solo stakers, home stakers, and smaller operators are a different story. Most run a single EC paired with a single CC. For them, an EC failure means their validator goes offline until they manually intervene: restart the client, wait for a patch, or migrate to a different EC entirely (which means syncing from scratch, a process that can take anywhere from 1.5 hours to 8+ days depending on the client).

The real-world impact of an EC failure therefore depends on the mix of professional vs solo operators running that client. The scenarios below model the P2P cascading effects, which hit everyone regardless of their failover setup.

Scenario 1: Geth fails. This is the most impactful scenario by raw numbers. Roughly half of all execution client instances go offline. On our fleet, Ethrex would lose 43% of its peers in one stroke. Nethermind instances peered heavily with Geth (the Lighthouse-paired one sees 22% Geth) lose a significant chunk too. For validators, the impact is split: professional operators with EC failover switch to their backup within seconds and keep attesting. Solo stakers running only Geth go dark until the issue is resolved. Whether finality breaks depends on how many of Geth's ~50% share consists of solo stakers without failover. If even a third of them can't switch quickly, that's ~17% of validators going offline, which combined with normal churn could push the network close to the 1/3 offline threshold where finality stalls. Even short of that threshold, the gossip mesh takes a massive hit: surviving ECs lose a large fraction of their peers simultaneously, transaction propagation slows across the board, and block proposals from surviving validators may include fewer transactions because the mempool is suddenly thinner.

Scenario 2: Reth fails. At ~12% network share, a Reth failure seems manageable at first glance. But our data reveals a cascading dependency that makes this worse than the headline number suggests. Besu nodes on the fleet derive 75% of their peers from Reth. If Reth goes down, the average Besu node drops from 24 peers to 6. At 6 peers, gossip propagation becomes unreliable: blocks arrive late, transactions get missed, and the node may struggle to stay in sync. In effect, a Reth failure doesn't just remove 12% of the network's EC instances. It functionally degrades another 11% (Besu's share), potentially pushing Besu nodes below the peer threshold needed for reliable chain following. Besu validators don't go offline in the strict sense, their EC is still running, but they start missing attestations and producing delayed block proposals because their view of the chain is degraded. That turns a 12% EC outage into something closer to a 20-23% effective network degradation, well beyond what the simple market share numbers would predict.

Scenario 3: Nethermind fails. At ~21% share, a Nethermind failure is the most concerning for the professional staking segment, since Nethermind has been the primary beneficiary of the "move away from Geth" push (Coinbase, stakefish, and others have publicly migrated large validator sets to Nethermind). These operators likely have failover, but the sheer scale of simultaneous failover events could create its own pressure: backup ECs suddenly handling double the engine API load, discovery networks flooded with reconnection attempts, and a temporary peer diversity shock as the second-largest client disappears from the gossip mesh. The Nethermind-to-Nethermind peering concentration we observed (up to 70% on some instances) means that even surviving Nethermind nodes, those that haven't crashed but are peered with ones that did, lose most of their connections and need to re-discover peers.

The counterintuitive finding: By raw network share, Geth is the obvious answer for maximum damage. But Reth's failure is disproportionately dangerous relative to its 12% market share because of the peer dependency it has created with Besu. A 12% client taking down its own nodes plus functionally crippling an 11% client through peer starvation is a 2x amplification factor that no aggregate diversity statistic captures.

This peer dependency amplification is invisible in the standard client diversity charts. The usual framing ("no client above 33%") assumes that each client's failure is contained to its own nodes. Our data shows that's not the case at the P2P layer. Client failures cascade through the peer graph, and the pattern of those cascades depends on the specific, measurable peer composition that each client maintains. This is exactly the kind of second-order effect that requires fleet-level observability to detect.

This is just the surface​

This analysis is a short snapshot of one aspect of our fleet data. At StereumLabs, we continuously collect Prometheus metrics and container logs across all 6 consensus clients and all 6 execution clients, in every pairing, on both bare-metal (NDC2 Vienna) and cloud (GCP) infrastructure.

We use this data for version comparison reports (like our Teku cross-version analysis and Prysm resource comparison), hardfork impact analysis, security audits, and deep dives like this one. The P2P peering layer alone has months of historical data waiting to be mined: how does peer diversity change over time? Do software updates shift the composition? What happens during hardfork activations? How do different geographic locations affect peering patterns?

The dashboards are there, the data is there, and there is much more to find. If you're a researcher, an Ethereum client team, or an infrastructure operator and you want to dig deeper, get in touch. You can also watch our EthCC[9] talk for a broader overview of how we use AI-powered observability across the Ethereum staking stack.

What are blobs?​

Blobs (Binary Large Objects) are a data type attached to Ethereum transactions, introduced with EIP-4844 ("Proto-Danksharding") and expanded with the Fusaka hardfork (December 2025), which activated PeerDAS (Peer Data Availability Sampling).

Before blobs, Layer 2 rollups posted their transaction data into execution layer calldata, competing for the same gas as every other transaction. Blobs created a separate, cheaper data channel for rollup data. Each blob carries about 128 KB and lives on the consensus layer (beacon chain), not in execution layer state, so it does not permanently bloat the blockchain.

From blobs to data columns: PeerDAS​

Fusaka took this further with PeerDAS. Instead of every node downloading every full blob, blobs are split into 128 data columns via erasure coding. Each node only needs to custody a few of these columns, typically 4 custody groups for a standard node. This reduces bandwidth requirements while maintaining data availability guarantees through KZG commitments (mathematical proofs). We measured the hardware impact of PeerDAS on our fleet in a separate post.

The lifecycle of a blob under PeerDAS:

1. A rollup submits a transaction with blob data attached
2. The execution client receives the transaction and holds the blob in its blob pool
3. At block proposal time, the consensus client can fetch blobs from its local EC via the getBlobsV2 Engine API call
4. After the block is published, nodes verify the data column sidecars they receive via gossip, including KZG proof and inclusion proof verification
5. If a node does not receive all its custody columns via gossip in time, it can attempt reconstruction from the columns it does have (if it has enough)

Each of these steps has measurable performance characteristics, and they vary across consensus client implementations. That is what we measured.

Blobs and MEV-Boost​

Most validators today use MEV-Boost to outsource block building to external builders who compete on profitability. Blob transactions carry their own fee market (blob gas), so builders have an incentive to pack as many blob transactions as possible into their blocks.

When a validator proposes a block via MEV-Boost, the flow adds a step: the CC receives a blinded block from the relay, reveals it, and then gets the full execution payload including all blobs. The CC must then compute and broadcast all data column sidecars from these blobs before the attestation deadline. This means the proposer's CC must have full data availability (all 128 custody groups) to serve the data column sidecars. If the CC is slow at blob processing, the block's data columns reach the network late, which increases the chance of missed attestations by other validators who depend on timely gossip.

In practice, MEV-Boost blocks tend to be blob-heavy because builders optimize for maximum gas revenue. A CC that handles blobs well under normal conditions but degrades at 6 blobs per slot will perform worse on MEV-Boost blocks than on locally built blocks.

Test setup​

Our StereumLabs fleet at the NDC2 datacenter in Vienna, Austria runs all six major consensus clients, each paired with all six major execution clients: 72 nodes total (36 CC×EC pairings on bare metal, plus 6 GCP-hosted supernodes not covered here). Each CC is connected to a validator client that assigns validator duties. The validator clients themselves are not part of the StereumLabs metrics pipeline, but their effect is visible indirectly: because validators are attached, every CC should operate as a supernode (128 custody groups) to guarantee full data availability for block proposals. Whether each CC actually does this is one of the things we checked. For details on methodology and infrastructure, see the StereumLabs documentation. The fleet is operated by RockLogic GmbH.

Observation window: April 24, 2026, 11:00 to 12:00 UTC. All histogram percentiles and rates in this post are computed over this 1-hour window. Counter-based totals (like getBlobsV2 request counts) are cumulative since last process restart. Longer time ranges and historical comparisons can be explored directly on the StereumLabs dashboards. If you need a custom analysis or report for a specific time range, client pairing, or metric, reach out to us at contact@stereumlabs.com.

All nodes run on identical bare-metal hardware in Vienna. Performance differences reflect client implementation, not infrastructure.

Finding 1: Custody groups and the validator client effect​

Since every CC on our fleet has a validator client attached, each CC should automatically upgrade to 128 custody groups (full supernode) to ensure complete data availability for block proposals. The question is: do they actually do this?

The beacon_custody_groups metric is only exposed by Lodestar, Grandine, and Teku. Lodestar and Grandine both report 128 custody groups, as expected for nodes with validators attached. Teku reports only 4 custody groups across all 6 EC pairings, despite having a validator client connected. This is unexpected and may indicate that Teku 26.4.0 does not auto-upgrade custody group count when validator duties are assigned, or that a separate CLI flag is required.

Lighthouse does not expose beacon_custody_groups, but a different metric tells the story: peers_per_custody_group_count shows Lighthouse tracking 104 to 106 custody groups (out of 128) across all EC pairings. That is consistent with supernode behavior. The missing 22-24 groups likely just have no peers at the moment rather than being uncustodied.

Nimbus and Prysm do not expose any custody-related metrics at all. We cannot confirm their actual custody group count from Prometheus. However, Nimbus's low data column processing rate (0.23 sidecars/s received, comparable to Teku's 0.29 req/s at 4 custody groups) suggests it may also not be running at full supernode level. Prysm's higher processing rate (3.7 req/s) is more consistent with supernode-level traffic, though still lower than Grandine (11.1 req/s) and Lighthouse (7.5 req/s).

The custody group count has a direct impact on everything that follows in this post. A node custodying 128 groups downloads, verifies, and stores 32x more data column sidecars than a node custodying 4 groups. This has to be considered when comparing absolute processing rates and latency tails across clients.

Finding 2: Gossip verification speed varies 50x between clients​

When a data column sidecar arrives via gossip, the consensus client must verify it before accepting it. This includes KZG proof and inclusion proof checks. Verification speed determines how quickly a node can process incoming data columns and meet attestation deadlines.

Nimbus uses data_column_sidecar_validation_duration rather than the beacon_data_column_sidecar_gossip_verification_seconds metric that other clients expose. The values are not perfectly 1:1 comparable but represent the same operation.

Operator perspective: At p95, Lodestar takes nearly half a second to verify a single data column. In a blob-heavy slot with 6 blobs (which expand to many data columns), this verification backlog can delay attestation timing. Attestations need to be published within the first 4 seconds of a slot, and each 469ms verification eats into that window.

Researcher perspective: Lodestar is the only consensus client written in TypeScript/Node.js. The cryptographic operations (KZG proofs, hash computations) are heavily optimized in compiled languages: Rust for Lighthouse/Grandine, Go for Prysm, Java with JNI for Teku, Nim for Nimbus. A JIT-compiled runtime faces a hard ceiling here.

Finding 3: KZG batch verification confirms the pattern​

KZG (Kate-Zaverucha-Goldberg) commitments are the mathematical proofs that guarantee data availability without requiring every node to download every blob. CCs batch-verify multiple KZG proofs at once for efficiency.

Same pattern: Teku leads, Lodestar trails by an order of magnitude. We looked at Teku's performance evolution across three versions in a separate post. Worth noting here: Grandine performs better here than in gossip verification, which indicates its KZG implementation (likely Rust-native) holds up well even under the supernode load of 128 custody groups.

Finding 4: Prysm struggles with data column collection​

Prysm's verification speed is competitive (4.2ms p50). Its problem sits elsewhere: it fails to process the majority of incoming data column sidecars.

Prysm processes only 43% of data column sidecar requests successfully. More than half fail, forcing the client into a costly fallback: data column reconstruction.

Teku's low absolute rate (0.29 req/s) reflects its standard 4-custody-group configuration. It receives fewer data columns. Its 85% success rate may be partially inflated by sampling effects at low volume.

Finding 5: Reconstruction is expensive, and Prysm needs it constantly​

When a node does not collect all required data columns via gossip, it can attempt to reconstruct missing columns from the ones it has, provided it has at least 50% of the 128 columns. Reconstruction uses erasure coding and is computationally expensive.

Prysm triggers reconstruction roughly once every 3 seconds, orders of magnitude more than any other client. With a 379ms median reconstruction latency, this adds real computational overhead.

Teku and Grandine never need reconstruction. They collect their custody columns via gossip every time, consistent with their high processing success rates.

Finding 6: Execution client choice affects blob serving speed​

Since Fusaka, consensus clients can fetch blobs directly from their local execution client via the engine_getBlobsV2 API. This is faster than waiting for gossip and helps the CC complete data availability checks sooner after a new block arrives.

We measured the getBlobsV2 response latency from Teku's perspective (the only CC that exposes per-EC latency histograms for this call):

Reth serves blobs 6.5x faster than Erigon at p50. Reth and Ethrex are both Rust-based, newer execution clients with storage designs built after blobs existed. Geth and Erigon have blob pool implementations that predate the PeerDAS access pattern, which likely explains the gap. EC implementation differences also show up in sync speed, where we measured a range from 1.5 hours to 8+ days on the same hardware.

Finding 7: Nimbus barely uses getBlobsV2​

Total getBlobsV2 request counts from the Geth EC perspective (Geth is paired with every CC, so this gives a consistent comparison):

Two outliers:

Nimbus makes 16x fewer getBlobsV2 requests than other clients. It relies primarily on P2P gossip to obtain blob data and uses the getBlobsV2 API as a fallback only. Whether this is intentional (gossip-first strategy) or because the feature is still in early adoption is not clear from the metrics alone. We observed similar architectural differences when investigating Nimbus block building behavior.

Lodestar makes 63% more requests than the average and has a 5.9% miss rate, 10x higher than other clients. With 128 custody groups (supernode behavior), Lodestar aggressively tries to fetch blobs from the EL. The high miss rate suggests blobs have already expired from the EC's blob pool by the time Lodestar requests them, pointing to a timing issue in Lodestar's fetch pipeline.

Data column gossip arrival timing​

For context, we also measured when data column sidecars arrive relative to slot start. This is a pure network metric, independent of client implementation:

Data columns typically arrive 1.4 to 1.6 seconds after the slot begins, with 95% arriving within 3.2 to 3.6 seconds. This is consistent across all EC pairings. Gossip arrival is purely a P2P network phenomenon, not influenced by the local EC.

CCs therefore have roughly 0.4 to 2.6 seconds between data column arrival and the attestation deadline (4 seconds into the slot) to complete verification. At Teku's 3.9ms verification speed, that is plenty. At Lodestar's 469ms p95, it gets tight.

Summary table​

🟢 = best in class · 🟠 = notable outlier · 🔴 = potential concern · †indirect via peers_per_custody_group_count

Takeaways​

The 50x spread in gossip verification latency between Teku (3.9ms p50) and Lodestar (469ms p95) is the headline number from this analysis. It feeds directly into data availability timing models: how long does a CC need between receiving a data column and publishing its attestation? At 3.9ms the answer is "no measurable impact". At 469ms, each verification burns a chunk of the ~2.6 seconds between typical gossip arrival and the attestation deadline. PeerDAS security models that assume uniform verification times across clients need to account for this spread, especially since Lodestar is the only TypeScript-based CC and carries a runtime disadvantage for the cryptographic operations involved (KZG proofs, hashing).

The custody group situation turned out more nuanced than expected. All nodes on our fleet have validator clients attached and should therefore custody all 128 groups. Lodestar, Grandine, and Lighthouse do this correctly. Teku 26.4.0 reports only 4 custody groups despite having validators assigned, which raises a question about its auto-upgrade behavior or whether a specific CLI flag is needed. Nimbus and Prysm do not expose custody metrics at all, making it impossible to verify from Prometheus alone. For operators, this is worth checking manually. For anyone modeling PeerDAS network-level data availability, the default custody behavior is not consistent across implementations.

Prysm's 43% processing success rate with constant reconstruction (0.33/s at 379ms p50 latency) points to a problem upstream of verification. The KZG verification itself is fast (3.5ms p50), so the bottleneck is in the gossip collection or processing pipeline. If reconstruction were to fail on top of that, the node would miss data availability deadlines.

On the EC side, the choice of execution client has a measurable effect on blob serving latency via getBlobsV2. Reth and Ethrex respond in 6 to 8ms at p50. Geth and Erigon take 34 to 41ms, with Erigon reaching 201ms at p95. Over thousands of attestations, this compounds. It is a new performance dimension that did not exist before Fusaka and is worth considering when choosing an EC pairing, particularly for validators where attestation timing matters.

Nimbus's minimal use of getBlobsV2 (16x fewer requests than other clients) remains an open question. It may be a deliberate gossip-first architecture, or it may reflect early-stage getBlobsV2 adoption. The operational impact depends on whether gossip alone is sufficient to meet data availability deadlines reliably. Our earlier analysis of Nimbus block building behavior showed a similar pattern of Nimbus taking its own path on feature adoption.

We deployed all 6 Ethereum execution clients from scratch on bare metal and tracked exactly how long each one took to fully sync. The results range from impressive to alarming.

Overview​

On April 8, 2026 at 19:41 UTC, we deployed the entire NDC2 bare metal fleet from scratch. All 36 execution client instances (6 ECs × 6 consensus client pairings) started within a 12 second window. The chain tip was at block ~24,837,000 (slot ~14,071,100).

Each EC is paired with all 6 consensus clients (Grandine 2.0.4, Lighthouse v8.1.3, Lodestar v1.41.1, Nimbus multiarch-v26.3.1, Prysm v7.1.3, Teku 26.4.0). All nodes run on identical bare metal hardware with NVMe storage at our NDC2 datacenter in Vienna. Every EC runs on its own dedicated host, separate from the consensus client.

Bare metal vs. cloud​

This analysis focuses on our NDC2 bare metal fleet, where each node has dedicated CPU, RAM, and NVMe storage with no resource contention. We also run the same client combinations on Google Cloud Platform (GCP) instances, which are available to StereumLabs users for side by side comparison. The two environments serve different purposes: bare metal gives us a controlled, reproducible baseline where hardware is never the variable, while GCP reflects the reality many operators face with shared infrastructure, variable I/O performance, and cloud specific constraints like burst credits and network throttling. Sync behavior can differ significantly between these environments, so we encourage users to explore both datasets on our platform.

Results​

Detailed sync breakdown​

Nethermind 1.36.2: 1h 22min​

Nethermind was the fastest client by a wide margin. Its container logs show a clean three phase sync:

Phase 1, FastHeaders (19:42 to 20:59, ~1h 17min): Nethermind set a pivot at block 24,837,019 and downloaded all headers. The sync mode transitioned through UpdatingPivot, FastHeaders, SnapSync, and finally StateNodes.

Phase 2, StateNodes (20:59 to 21:04, ~5min): World state download at the pivot point.

Phase 3, Block processing (21:04 onward): First "Block throughput" log appeared at 21:04 UTC, showing live block processing at 107 MGas/s and 850 tps. From this point, Nethermind was fully operational.

08 Apr 21:04:39 | Block throughput 107.72 MGas/s | 850.4 tps | blobs 12

Total: ~1 hour 22 minutes from cold start to processing live blocks.

Erigon v3.3.10: 13 hours​

Erigon uses its unique staged sync pipeline. The sync Prometheus metric with per stage labels provides precise tracking:

• Headers, Bodies, and Senders: Jumped to block 24,767,999 within ~20 minutes. Erigon had pre existing frozen block data and only needed recent segments.
• Execution stage: Progressed in 5,000 block batches (24,768K to 24,773K to 24,778K and so on), advancing ~5K blocks every 60 to 90 minutes.
• Finish stage: Reached the chain tip at ~08:40 UTC on April 9.

Two Erigon instances (Lodestar and Teku pairings) required unscheduled process restarts 11 to 12 hours after deployment, indicating memory pressure during the execution stage.

Total: ~13 hours.

Besu 26.2.0: 37 hours (13h snap + 24h backfill)​

Besu performed a pivot based snap sync from block 0:

Phase 1, Snap sync (19:42 to Apr 9 08:57, ~13h 15min): Besu downloaded headers backward from pivot 24,837,019 to block 0 while simultaneously fetching world state. The ethereum_blockchain_height metric starts at block 15,537,394, which is the first post Merge block (The Merge occurred at block 15,537,393 on September 15, 2022). Besu only needs to forward execute post Merge blocks since pre Merge PoW data is handled differently.

Sync completed successfully with pivot block 24839885

Phase 2, Backward sync (Apr 9 08:57 to Apr 10 08:54, ~24h): After pivot sync completed, Besu backfilled pre Merge block bodies and receipts in a separate backward pass.

Starting a new backward sync session
...
Current backward sync session is done

Total: ~13h 15min to live operations, ~37h for complete historical data.

Reth v1.11.3: 5 days 18 hours​

Reth runs a sequential 14 stage pipeline from scratch. Container logs with Finished stage messages provide exact timing for every stage:

The Execution stage consumed 93% of the total sync time, replaying all 24.8 million blocks sequentially. Headers downloaded in 8 minutes and bodies in under 4 hours, but the actual block execution took over 5 days.

Total: 5 days 18 hours. Pipeline finished April 14, 13:47 UTC.

Geth v1.17.2: 7 days 15 hours​

Geth performed a snap sync from genesis with parallel header and state downloads:

Headers (19:42 to ~01:00 Apr 9, ~5h): Geth's header download progressed at ~1.25M headers/hour. Early logs showed "Syncing beacon headers downloaded=512 left=24,836,569 eta=4h20m". Geth explicitly rejected live CC blocks during this phase: "Ignoring payload while snap syncing".

State download (parallel with headers, ~1.5h): State download started simultaneously and progressed rapidly, showing "state download synced=0.15% eta=1h24m" at minute 10.

State healing (dominant bottleneck): After headers and initial state were downloaded within hours, state healing ran for days. This is the phase where Geth patches gaps and inconsistencies in the downloaded state trie.

Snap sync complete: April 16, 10:32 UTC.

Switching from snap-sync to full-sync    reason="snap-sync complete"

Total: ~7 days 15 hours. State healing was the multi day bottleneck.

Ethrex 9.0.0: Not completed (8+ days)​

Ethrex operates in dual mode: it immediately processes live blocks via the engine API while running P2P snap sync in the background. This means the paired CC tracks the chain head within seconds of startup, but the EC lacks complete historical state.

Live blocks (immediate): At 19:42:32 UTC, Ethrex received its first engine payload for block 24,837,020, just 26 seconds after startup.

P2P snap sync (ongoing): Snap sync started at 19:42:35. After ~3 hours it reached the "Requesting Bytecodes" phase. However, a restart at some point reset all progress. As of April 16, the snap sync status showed:

P2P Snap Sync | elapsed 00h 00m 07s | peers 7 | step Downloading Headers

The sync had effectively restarted from scratch with minimal peer connectivity.

Total: Not completed after 8+ days. The P2P snap sync is effectively stalled.

Peak resource usage during sync​

These are maximum values observed on each EC host during the first 6 hours after deployment (the peak sync burst period), measured via node exporter. Since each EC runs on its own dedicated host with no other significant services, these numbers reflect EC resource consumption directly.

Nethermind was the heaviest on disk I/O: 561 MB/s reads and 533 MB/s writes during its snap sync burst. It traded high resource intensity for the fastest sync by far.

Geth was CPU intensive (9.7 cores) but had the lowest memory footprint (5.6 GB) of any EC during sync.

Erigon consumed the most RAM among established clients (14.1 GB) despite being mid pack on sync speed, consistent with its known memory pressure pattern during staged execution.

Ethrex had the highest RAM usage overall (19.7 GB) and very high disk writes (484 MB/s), driven by simultaneous live block processing and background snap sync.

Reth was the most resource efficient during the initial 6 hour window: lowest CPU (2.9 cores), moderate RAM (9.4 GB), minimal I/O. However, this is misleading because Reth's pipeline was still in the Bodies download stage during this window, with the heavy Execution stage starting later and running for 5+ days.

Key observations​

1. Sync speed varies by orders of magnitude​

The gap between the fastest (Nethermind, 1.5h) and the slowest completed sync (Geth, 7.5 days) is a factor of 120×. For operators, this means the difference between being back online during a lunch break versus being offline for more than a week.

2. "In sync" does not mean synced​

This is the most operationally dangerous finding. Reth, Geth, Ethrex, and Besu all reported themselves as ready to serve the consensus client while their historical sync was far from complete. Operators monitoring only CC metrics (head slot, attestation inclusion) would see a healthy node while the EC was still days from full sync.

3. NVMe storage is essential for fast sync​

Nethermind's 561 MB/s peak disk reads demonstrate why: its snap sync strategy saturates fast storage to finish quickly. On slower drives (SATA SSD or HDD), the sync time would scale proportionally, potentially turning Nethermind's 1.5 hours into many hours.

4. Erigon's staged sync is predictable but slow​

Erigon's 5K block batch processing through the execution stage is the primary bottleneck. The staged approach provides clear progress visibility and checkpointing, but at the cost of being 10× slower than Nethermind. Two instances also required restarts due to memory pressure during this phase.

5. Reth and Geth need days for historical execution​

Both clients must replay the entire chain history. Reth does this sequentially through its Execution stage (5+ days), while Geth's bottleneck is state trie reconciliation after its initial snap download. These are fundamentally different bottlenecks but produce similar multi day timelines.

6. Ethrex syncing requires patience​

Ethrex delivers solid performance once fully synced, as our previous analyses have shown. The challenge is getting there. Its P2P snap sync is still maturing, and a restart during our test reset all progress. Peer connectivity (7 peers vs. 90+ for other clients) suggests the sync P2P discovery needs further work. The LambdaClass team is actively iterating, and upcoming Ethrex versions are expected to improve the sync experience considerably. For operators willing to invest patience (or use alternative bootstrapping methods), Ethrex is absolutely viable in production once it reaches the chain tip. We offer tooling and guidance for Ethrex sync on StereumLabs for those who want to get started today.

Relation to block building performance​

In our previous analysis of Nimbus block building across 5 ECs, we found that Erigon's slow block execution (423 to 567ms per imported block) causes it to deliver empty block payloads. This sync investigation confirms the pattern: Erigon's execution pipeline is the consistent bottleneck, whether processing historical blocks during sync or building new blocks for proposals.

Reth was excluded from that analysis because it had not completed its initial sync at the time. This post explains why: its full pipeline takes nearly 6 days.

Methodology​

• All data sourced from Prometheus (prometheus-cold datasource) and Elasticsearch (Filebeat container log shipping) on the StereumLabs Grafana instance.
• EC sync completion determined from EC container logs: Finished stage (Reth), Snap sync complete (Geth), Sync completed successfully / Backward sync session is done (Besu), Block throughput first appearance (Nethermind), P2P Snap Sync status messages (Ethrex), sync metric per stage progression (Erigon).
• Peak resource usage measured via node_exporter metrics with max_over_time(...[6h]) queries at each EC's dedicated host.
• All nodes run on NDC2 bare metal hardware (Vienna), eliminating cloud induced variance.

Overview​

We run 5 Nimbus multiarch-v26.3.1 beacon nodes on our NDC2 bare-metal fleet in Vienna, each paired with a different execution client:

A sixth node (Nimbus + Reth v1.11.3) is also deployed, but Reth has not yet completed its initial sync and is therefore excluded from this analysis.

All 5 nodes use Nimbus' built-in validator_monitor feature to passively observe the same set of 1,000 validator pubkeys on-chain. The validator monitor tracks attestation inclusion, vote correctness, block proposals, and timing data for these validators without requiring the signing keys to be locally attached.

The full analysis covers a 48-hour window ending April 13, 2026. Data sources: Prometheus (prometheus-cold) for metrics and Elasticsearch (Filebeat) for both EC container logs and Nimbus CC container logs.

Block building: the headline finding​

When one of the 1,000 monitored validators is selected as block proposer, Nimbus triggers engine_forkchoiceUpdatedV3 with payload attributes on its paired EC, asking it to build a block. The EC then constructs the execution payload iteratively, improving it over several seconds until Nimbus calls engine_getPayloadV4 to retrieve the result.

Over 48 hours, block building was triggered for approximately 13 unique blocks. The results differ dramatically between execution clients.

Build latency vs. payload output​

Nimbus logs the exact timestamps for Requesting engine payload and Received engine payload, giving us the end-to-end build latency and resulting gas_used for every payload. This CC-side perspective is consistent across all ECs and fills in data even when the EC's own logs lack detail.

The most counterintuitive finding: Erigon is the fastest responder (479ms avg) yet delivers 0 gas. Its transaction pool is apparently unable to supply transactions within the build window, so it returns an empty block quickly rather than spending time filling it. Besu takes 67ms longer on average but uses that time to pack 23.5 Mgas into the payload.

For high-gas blocks (slots with blob transactions), latency increases across all ECs: Besu peaks at 848ms for a 60M gas block, Ethrex at 728ms for the same block. This confirms that build latency scales with payload complexity.

Per-block distribution​

The averages above compress a wide range of behavior into single numbers. Plotting every individual block build reveals the full distribution:

Three distinct patterns emerge:

Standard blocks (500-650ms, 7-18 Mgas): The main cluster where most builds land. Besu, Ethrex, and Nethermind consistently occupy the upper band (14-18 Mgas), while Geth sits lower (7-15 Mgas). All four ECs overlap in latency, confirming that response time differences between them are marginal for regular blocks.

Blob-heavy blocks (700-850ms, 38-60 Mgas): A few outlier slots where blob transactions push gas usage to 38-60M. These blocks take noticeably longer to build across all ECs, with Besu and Ethrex reaching the gas limit (60M) while Geth tops out around 40-48M. The latency increase is proportional to payload complexity.

Erigon (480-520ms, 0 Mgas): Every single Erigon build sits flat on the x-axis at 0 gas, forming a tight horizontal cluster. Regardless of whether the same slot produced a 17M gas block on Besu or a 60M gas block on Ethrex, Erigon delivered an empty payload. This pattern is consistent across all 12 blocks with no exceptions.

The scatter also reveals a handful of cached payloads (sub-100ms) where all ECs returned near-instantly, likely from a previously prepared payload that Nimbus retrieved before the build window elapsed.

Cross-EC payload comparison​

By examining the Received engine payload log from all 5 Nimbus CC instances, we can compare the actual gas_used delivered by each EC for the same blocks:

Besu, Ethrex, and Nethermind are the strongest block builders, routinely filling blocks to 15-30% gas utilization. Ethrex is particularly consistent, delivering 14-18M gas for standard blocks. Geth produces lighter payloads. Erigon's payloads are effectively empty.

Besu: the most aggressive builder​

Besu produced 661 block improvement iterations across its built blocks. Its logs show the full iterative building process with reward tracking:

New proposal for payloadId 0x36f36d block 24869114
  gas used 17,624,773  transactions 366  reward 2.22 finney
  is better than the previous one by 103.24 szabo

For block #24,869,114, Besu's best build reached 366 transactions, 17.6M gas, and a 2.22 finney block reward. It logged every improvement step, including the marginal reward increase between iterations.

Geth: clean lifecycle, strong output​

Geth logged 214 payload updates and provides the clearest view of the block building lifecycle:

The build started with 6 transactions and grew to 349 transactions over ~10.5 seconds, with each update taking 25-67ms. Geth logs Starting work on payload, then multiple Updated payload entries (with txs, gas, fees, elapsed time), and finally Stopping work on payload reason=delivery when Nimbus retrieves the result.

Ethrex: strong builder, detailed execution breakdown​

Ethrex does not log block building progress in its own container logs at the default log level. However, Nimbus' CC logs reveal it is one of the strongest builders in the fleet, delivering 21.8 Mgas on average with one block hitting 100% gas utilization (60M gas). Each payload carries extra_data: ethrex 9.0.0.

What Ethrex does log is an excellent per-block execution breakdown when processing incoming blocks:

BLOCK EXECUTION THROUGHPUT (24869233): 0.366 Ggas/s
  TIME SPENT: 55 ms. Gas Used: 0.020 (33%), #Txs: 134
  block validation: 1% | exec(w/merkle): 91% | merkle-only: 2% | store: 7%

This level of transparency into where execution time is spent (91% in execution+merkle, 7% in storage, 1% in validation) is unique among the tested ECs and valuable for performance profiling.

Nethermind: strong builder, quiet logs​

Nethermind's own logs only show production requests (Production Request 24869114 PayloadId: 0x2321052e5846b8a2) without per-iteration detail at the default log level. However, Nimbus' CC logs reveal the full picture:

INF Received engine payload  slot=14103195 payload="(block_number: 24869114,
  gas_used: 17257515, gas_limit: 60000000, extra_data: Nethermind v1.36.2, ...)"

For block #24,869,114, Nethermind delivered 17.3M gas, competitive with Besu's 17.6M and Ethrex's 16.9M. Across all 13 payloads in 48h, Nethermind consistently produced blocks in the 9-17M gas range.

Erigon: severely impaired block building​

Erigon's own logs show it actively attempts to build blocks but consistently fails to include transactions:

76% of Erigon's 54 block build iterations produced completely empty blocks with 0 transactions and 0 gas. When it did include transactions, the count was dramatically lower than other clients: a maximum of 36 transactions versus 300+ for Besu and Geth.

Built block  height=24869114  txs=0  executionRequests=0
  gas used %=0.000  time=782.686ms

The Nimbus CC logs confirm this: across 12 blocks, every single payload from Erigon contained 0 or near-0 gas. This is not a latency issue. At 479ms average, Erigon is the fastest EC to return a payload. The problem is upstream: Erigon's block execution latency (423-567ms per imported block) delays transaction pool maintenance, so when Nimbus asks for a payload, Erigon has nothing to put in it.

Attestation pipeline: per-EC timing differences​

While all 5 nodes observe the same 1,000 validators and see identical on-chain results (99.996% attestation inclusion rate), how quickly each node observes attestation events varies by EC pairing. Three pipeline stages were measured:

Besu is the fastest for raw attestation observation. However, Geth leads in the third stage (attestation appearing inside aggregates), which is the most relevant for actual inclusion. Erigon is consistently the slowest in this final stage, adding 18ms of latency versus Geth. Ethrex performs well across all three stages.

Attestation volume: Erigon's observation gap​

All 5 nodes should see roughly the same number of attestations for the monitored validators. Over 48 hours, they do not:

Erigon misses ~45,000 attestations that the other 4 nodes see within the same 48-hour window. This is not a network issue; all nodes are on the same bare-metal fleet in Vienna. Erigon's slower block processing causes attestations to expire before the node can process them.

Vote accuracy: source, target, and head​

The 1,000 monitored validators' three Casper FFG vote types show dramatically different difficulty levels:

Head vote accuracy is the hardest duty, with ~210x more misses than source. It requires the node to see the latest block before the slot boundary, making it the most sensitive to processing speed.

Head vote: diurnal pattern​

Over 48 hours, head vote accuracy shows a clear cyclic pattern:

The rate fluctuates between 98.64% and 99.54%, likely driven by network congestion patterns. All 5 nodes report identical values at each time point, confirming this reflects on-chain truth, not individual node performance.

Block proposals via gossip​

Over the monitoring period, the nodes observed 27 block proposals from the 1,000 monitored validators via gossip. In the last 48 hours, 12 block proposals were detected via the validator_monitor_block_hit_total counter. All proposals arrived via the gossip network, confirming the validators are actively proposing on-chain.

Nimbus CC logs as an observability source​

One of the practical takeaways from this analysis: Nimbus' consensus client logs are a valuable observability layer for block building, regardless of how detailed the execution client's own logs are.

Nimbus logs three key events per block building cycle:

1. Requesting engine payload — timestamp, slot, beacon head, fee recipient
2. Received engine payload — full payload contents: gas_used, gas_limit, block_number, extra_data
3. Block proposal included — slot, validator ID

For Ethrex and Nethermind, whose default log levels don't expose per-iteration block building details, the Nimbus CC logs were the only way to determine that both clients are strong builders (17-22M gas avg). This CC-side perspective also revealed the build latency data that showed Erigon's problem is not response time but empty transaction pools.

EC log verbosity at default levels​

An interesting infrastructure finding: the default log levels produce vastly different volumes across execution clients.

Nethermind produces ~9x more log output than Besu at default log levels. All 6 ECs in our fleet (including Reth, which is syncing) are included here since this is a general infrastructure observation.

Summary​

Key takeaways​

1. Erigon's block building is severely impaired. 76% of built blocks were empty, and even when it did include transactions, the count was a fraction of what other clients achieved. Paradoxically, Erigon responds fastest (479ms) but with nothing in the payload. The root cause is its slow block execution (423-567ms) cascading into transaction pool readiness.

2. Besu, Ethrex, and Nethermind are all strong block builders. Besu leads on raw gas output (23.5M avg) and iteration count (661 per block). Ethrex is the most consistent (14-18M gas for standard blocks, 60M for full blocks). Nethermind is competitive at 21.2M avg despite minimal logging.

3. Nimbus' CC logs are a valuable observability source. The Received engine payload log provides payload gas_used, build latency, and block details for every EC, making it the most reliable cross-client comparison tool, especially for ECs like Ethrex and Nethermind whose own logs lack block building detail at default levels.

4. Erigon misses ~11% of attestation observations due to slower block processing. This volume gap is unique to Erigon; the other 4 ECs are within 1% of each other.

5. Head vote accuracy (~99.1%) shows a diurnal pattern visible across all nodes identically, driven by network congestion rather than client behavior.

6. Log volume varies 9x between the most and least verbose EC at default log levels. Nethermind's high verbosity (5,661/hr) is an infrastructure sizing consideration.

Methodology notes​

All Prometheus data sourced from the prometheus-cold datasource in the StereumLabs Grafana instance. Validator monitor metrics use the validator_monitor_* family with cc_client="nimbus" and validator="total" label selectors. 48-hour data uses increase(...[48h]) instant queries and histogram_quantile over rate(...[48h]) for delay distributions.

EC container logs are stored in Elasticsearch (Filebeat 9.3.0 → Elasticsearch 9.3.0) and queried via the container.image.name field to isolate each execution client's output. Block building logs were identified by client-specific patterns: Besu's New proposal for payloadId, Geth's Updated payload/Starting work on payload/Stopping work on payload, and Erigon's Built block/Building block.

Additionally, Nimbus CC container logs (statusim/nimbus-eth2:multiarch-v26.3.1) were analyzed for engine API interactions: Requesting engine payload (build request timestamps), Received engine payload (payload contents including gas_used), and Block proposal included (on-chain confirmation). This CC-side perspective provides build latency measurements and fills in payload details for ECs whose own logs lack that information at default log levels (notably Ethrex and Nethermind).

All nodes run on NDC2 bare-metal hardware (Vienna), eliminating cloud-induced variance.

For details on our label conventions and how to build your own dashboards against our data, see Build your own dashboards.

Overview​

We compared three Teku consensus client versions on our NDC2 bare-metal fleet in Vienna, each paired with all 6 execution clients (Besu, Erigon, Ethrex, Geth, Nethermind, Reth). Each version was measured over a 14-day window during its active deployment period using avg_over_time(...[14d:1h]) instant queries against our Prometheus-cold datasource.

The full report is available as a PDF download at the bottom of this post.

Fleet-level headline numbers​

The two architectural changes — LevelDB → RocksDB in 26.2.0 and the jemalloc memory allocator in 26.3.0 — drove the majority of the performance shifts:

CPU utilization​

Metric: rate(process_cpu_seconds_total{job="teku"}[5m]) — Teku JVM process CPU in CPU-seconds per second.

CPU dropped 38% from 25.12.0 to 26.2.0. RocksDB's block cache and bloom filters reduce per-lookup processing compared to LevelDB. Version 26.3.0 added another 11% reduction through jemalloc's more efficient allocation patterns reducing GC-driven CPU spikes. The Nethermind pairing consistently shows the lowest Teku CPU usage across all three versions.

Memory & JVM analysis​

Process RSS memory​

Metric: process_resident_memory_bytes{job="teku"} — total physical memory consumed by the Teku JVM process.

RSS peaked in 26.2.0 (+5.5% vs 25.12.0), consistent with RocksDB's larger in-memory structures (block cache, memtables, bloom filters). Version 26.3.0 clawed back roughly half through jemalloc's reduced memory fragmentation, leaving a net +2.8%.

JVM heap memory​

Metric: jvm_memory_used_bytes{area="heap"} — Java heap utilization.

Heap grew steadily, reflecting increased in-heap state caching from the RocksDB JNI bridge and the natural growth of Ethereum's state tree. All values remain well within Teku's default 8 GB max heap.

JVM native memory​

Native (off-heap) memory rose modestly with RocksDB and stabilized with jemalloc.

JVM garbage collection​

Metric: rate(jvm_gc_collection_seconds_sum[5m]) — fraction of time spent in GC per second. This is a critical Teku-specific metric because GC pauses directly impact block processing latency.

GC overhead was flat from 25.12.0 to 26.2.0. The introduction of jemalloc in 26.3.0 is the standout: fleet-wide GC time dropped 36%. jemalloc reduces heap fragmentation, meaning fewer large-object promotions to old gen and fewer full GC cycles. For a Java client like Teku, this matters directly — GC pauses are a primary contributor to block import latency.

Storage engine & disk I/O​

Host-level disk I/O​

Metrics: rate(node_disk_read_bytes_total[5m]) and rate(node_disk_written_bytes_total[5m]) — host-level metrics that include both Teku CC and its paired EC. Since EC versions didn't change across measurement windows, deltas are attributable to the Teku version change.

All values in KB/s.

Read throughput dropped 79% (1,518 → 310 KB/s) and write throughput fell 61% (3,258 → 1,270 KB/s). RocksDB's block cache and SST-based design is far more read-efficient than LevelDB. The Besu pairing saw the most dramatic read improvement (−85%).

RocksDB internal metrics (26.2.0+ only)​

These metrics are only available for versions using RocksDB. Version 25.12.0 ran LevelDB which does not expose equivalent counters.

Write amplification improved in 26.3.0: both raw write rate and compaction writes decreased ~11%. The slight read increase likely reflects the partial sidecar import feature doing more background reads.

Block import performance​

Metric: beacon_block_import_delay_latest — most recent block import delay in milliseconds. This captures end-to-end latency from receiving a block to completing its import into the beacon state.

Version 26.2.0 was slightly slower fleet-wide — likely early-stage RocksDB tuning and the concurrent DAS backfiller adding load. Version 26.3.0 brought a strong recovery, achieving the lowest fleet-wide latency at 297 ms. The Ethrex pairing improved most dramatically (−73%), while the Nethermind pairing shows persistently elevated import times that warrant EC-side investigation.

P2P networking & peer connectivity​

All peer metrics improved steadily. Discovery live nodes grew from ~161 to ~176, suggesting the Discv5 layer is maintaining more active ENR records. Gossip throughput peaked in 26.3.0 — consistent with faster block processing enabling quicker re-gossip.

Data availability sampling (DAS) & PeerDAS​

Metric: rate(beacon_data_column_sidecar_processing_requests_total[5m]) — DAS workload rate.

DAS processing rates show significant per-pairing variance. The zero values in 25.12.0 for Erigon/Ethrex reflect sync issues that resolved in later versions. Version 26.3.0 allows nodes with >50% custody requirements to begin importing blocks after downloading only 50% of sidecars — an architectural improvement that doesn't compromise data availability guarantees.

Open file descriptors — the trade-off​

File descriptors doubled — the most visible trade-off of RocksDB. Its LSM-tree architecture holds many SST files open simultaneously for efficient reads. Growth continued from 26.2.0 to 26.3.0 as databases accumulated more SST files through compaction.

Summary & recommendations​

Key takeaways​

1. Upgrade to 26.3.0 is mandatory. Beyond the SSZ serialization bug fix, 26.3.0 delivers the best performance profile across nearly every dimension.

2. Verify file descriptor limits. With 26.3.0 averaging 1,308 open FDs (and likely higher under peak load), ensure systems allow at least 65,536 file descriptors.

3. Monitor RocksDB compaction. Add storage_compact_write_bytes and storage_bytes_written to your dashboards. Abnormal compaction spikes can indicate database health issues.

4. Consider DAS backfiller tuning. If 26.x nodes show elevated CPU during backfill, --Xp2p-reworked-sidecar-custody-sync-batch-size=1 can throttle the backfiller.

5. Watch the Nethermind pairing. Block import delays are persistently higher with Nethermind across 26.x versions — this warrants investigation on the EL side.

Download the full report​

The complete analysis with additional detail on JVM thread pools, executor queue depths, and RocksDB internal counters is available as a styled PDF:

📄 Download full report (PDF)

Methodology notes​

All data sourced from the prometheus-cold datasource (Org 6) in the StereumLabs Grafana instance. Query pattern: avg by (ec_client) (avg_over_time(metric{cc_client="teku", cc_version="...", role="cc", job="teku"}[14d:1h])) evaluated as instant queries at the end of each 14-day window. Rate metrics use rate(...[5m]) inside the subquery. Host-level metrics filtered with device!~"lo|veth.*|docker.*|br.*" to exclude virtual interfaces. All nodes run on NDC2 bare-metal (Vienna), eliminating cloud noise.

For details on our label conventions and how to build your own dashboards against our data, see Build your own dashboards.

On April 2, 2026 we presented StereumLabs at EthCC[9] in Cannes. The talk covered what we've built, why AI on raw metrics alone produces useless output, and concrete Fusaka/PeerDAS runtime data from our bare-metal fleet.

This post is a written companion to that talk. If you prefer watching, the recording is embedded below. The slide deck is available as a PDF download.

Video​

The StereumLabs platform​

StereumLabs is our observability and analytics platform for Ethereum staking infrastructure. We run every relevant client combination on dedicated bare-metal hardware: 6 execution layer clients (Geth, Nethermind, Besu, Erigon, Reth, Ethrex), 6 consensus layer clients (Lighthouse, Prysm, Teku, Nimbus, Lodestar, Grandine), plus the standalone Erigon + Caplin pairing. That's 37 combinations, monitored 24/7 with 90-day rolling metrics.

All nodes run on isolated bare metal. No shared cloud instances, no noisy-neighbor effects. When we measure performance differences between clients, the data is reproducible and directly comparable.

The platform provides 20+ dashboards covering resource consumption (CPU, RAM, disk, network), client-specific metrics (attestation rates, block processing times, peer counts, GC behavior), and system logs. Client development teams already have free access to the dashboards. The project is supported by an Ethereum Foundation grant.

But dashboards have limits. And that's where AI comes in.

AI Chatbot: natural language meets live data​

We built an AI chatbot connected directly to our full monitoring dataset. Instead of navigating dashboards and writing queries, users ask questions in plain English:

• "Compare disk growth between Geth and Erigon over the last 30 days"
• "How did the Prysm update from v7.1.1 to v7.1.2 affect resource usage?"
• "Which consensus client uses the most bandwidth as a supernode?"

The result is a structured analysis with actual numbers, across all EL pairings, in seconds.

The Instruction Set​

Here's what most people get wrong about AI in infrastructure monitoring: the model alone doesn't produce useful results. If you point a language model at raw Prometheus metrics, it doesn't know which queries to run, what normal ranges look like, or how to interpret differences between client architectures.

That's why we've built a continuously evolving Instruction Set. It encodes which metrics matter for which client combination, what normal ranges look like per pairing, how to interpret architectural differences (Go vs. Java GC behavior, Rust memory models), and which queries to run in which order to build a meaningful analysis.

Without the Instruction Set, the AI produces generic answers. With it, it produces the kind of analysis that would take an experienced engineer hours to assemble manually. We expand it continuously as we encounter new patterns, new client versions, and new edge cases. It's built from years of running these clients professionally.

Proof: Prysm v7.1.1 to v7.1.2​

When the Prysm team shipped v7.1.2, we asked the chatbot one question and got a full resource impact analysis across all 6 EL pairings:

This analysis would normally take hours of manual work. The chatbot produced it from a single question. The full report is published on our blog: Prysm v7.1.1 & 7.1.2 resources.

AI Alerting: from "something is wrong" to "here's why"​

The chatbot is great for proactive analysis. But what about when things go wrong at 3am? That's where AI Alerting comes in.

Two-stage architecture​

Stage 1: Near-real-time threshold alerts. Monitors hard thresholds like attestation rate, disk usage, peer count, and missed blocks. Fires within seconds. No AI inference delay, no additional cost. If the AI layer is slow or unavailable, the basic alert still arrives.

Stage 2: AI root-cause analysis. When a threshold alert fires, a webhook triggers the AI. It pulls relevant metrics and logs, correlates the data against our neutral baseline from all 37 client combinations, and delivers a root-cause analysis with actionable next steps. Delivered in 5 to 15 seconds.

The result: operators don't just get "attestation rate dropped below 95%." They get: "Your attestation rate dropped because Geth's peer count fell to 3, likely due to a network partition. Your Prysm instance is healthy. Recommended action: check firewall rules and restart the EL client."

Built-in baseline: instant context​

What makes our alerting especially useful is the neutral baseline dataset from our own fleet. When an alert fires, the AI automatically compares against data from all 37 client combinations.

Every alert answers three questions: Is this happening across the Ethereum network right now? Is it specific to this client version? Or is it unique to your local environment?

That distinction between "the whole network is seeing elevated block processing times after a fork" and "your Geth instance is the only one with this problem" is the difference between waiting it out and taking immediate action.

Security monitoring​

The same two-stage architecture applies to security events:

• SSH login checks — authorized key? Expected source IP? Expected time window?
• Service restart analysis — when an execution client restarts, the AI verifies that fee recipient addresses haven't been changed. A compromised operator could redirect staking rewards without anyone noticing for days.
• Configuration drift detection — unauthorized processes, unexpected port openings, validator key access patterns.

Traditional monitoring tells you "Geth restarted." Our AI layer tells you "Geth restarted, fee recipient address changed from 0xABC to 0xDEF, this was not initiated through the operator's usual deployment pipeline, severity: critical."

For operators staking millions in ETH, the difference between detecting a compromised reward address in minutes versus days is the difference between a security incident and a financial disaster.

Fusaka + PeerDAS: what the hardfork did to hardware​

A significant portion of the talk covered our Fusaka hardfork measurements. We compared two 14-day windows (before and after the December 3, 2025 activation) across all 36 non-supernode client pairings.

The fleet-level headline numbers:

Notable client outliers: Nimbus CPU jumped +257% (most compute-intensive PeerDAS implementation), Lighthouse was the only consensus client to reduce CPU (-13%), and Besu saw the largest memory drop (-35%).

The worst pairing post-fork: Nimbus + Reth at 16.78% CPU.

The full analysis with per-client breakdowns, heatmaps, daily trends, and PromQL queries is available as a dedicated blog post: Fusaka hardfork: hardware impact on non-supernodes.

Deployment models​

One thing we hear constantly from professional operators: "I'm interested, but I can't send my metrics to your cloud." That's why StereumLabs supports multiple deployment options:

• SaaS — hosted by us in our ISO 27001 certified environment. Best for smaller operators and researchers.
• Alerting-as-a-Service (pull model) — you expose a Prometheus endpoint, we scrape it. Your data never enters our systems. It only meets our baseline data at the AI inference layer.
• On-premise — the entire StereumLabs stack deployed on your infrastructure. Your own API keys. Data never leaves your network.

The key message: your infrastructure data doesn't become someone else's competitive intelligence.

Current status​

Get in touch​

We're looking for node operators who want to try the AI chatbot, client development teams interested in automated cross-EL impact analysis, and staking protocols looking for monitoring and security standards across their operator ecosystem.

Reach out at contact@stereumlabs.com or visit stereumlabs.com.

Slides​

The full slide deck from the talk is available for download:

📄 Download slides (PDF)

What is Fusaka?​

Ethereum's Fusaka hardfork activated on December 3, 2025 at 21:49 UTC (slot 13,164,544). It was the second hard fork of 2025 after Pectra (May 2025) and arguably the most consequential upgrade since the Merge. The name combines "Fulu" (consensus layer, named after a star) and "Osaka" (execution layer, named after the host city of Devcon 2025).

The headline feature is PeerDAS (Peer Data Availability Sampling, EIP-7594) — a fundamental change in how Ethereum verifies blob data. Instead of every node downloading and verifying every blob, nodes now only need to sample small slices, verifying that the full data exists without actually possessing all of it. Vitalik Buterin called it "literally sharding" — Ethereum reaching consensus on blocks without requiring any single node to see more than a tiny fraction of the data.

Beyond PeerDAS, Fusaka shipped approximately 12 additional EIPs:

• EIP-7935 — raises the default block gas limit, targeting ~60 million gas
• EIP-7825 — introduces a per-transaction gas cap of 16.78 million to prevent single-transaction DoS attacks
• EIP-7892 — the Blob Parameter Only (BPO) mechanism, allowing blob capacity increases between hard forks
• EIP-7951 — secp256r1 precompile for device-native signing and passkeys
• EOF (EVM Object Format) — a cleaner, more efficient programming structure for smart contracts
• EIP-7918 — stabilizes blob fees
• EIP-7742 — new opcodes including CLZ (count leading zeros) for more efficient cryptographic operations

Two scheduled BPO forks followed:

• BPO-1 (~December 9–10): blob target/max raised from 6/9 to 10/15
• BPO-2 (~December 23 – January 7): blob target/max raised to 14/21

We wanted to know: what did all this actually do to the hardware running our nodes?

Methodology​

Fleet setup​

StereumLabs runs approximately 90 hosts split across GCP cloud instances and NDC2 bare-metal nodes in Vienna, Austria. Each non-supernode host runs one consensus client (CC) paired with one execution client (EC), covering all 36 possible combinations of:

• Consensus clients: Grandine, Lighthouse, Lodestar, Nimbus, Prysm, Teku
• Execution clients: Besu, Erigon, Ethrex, Geth, Nethermind, Reth

Data source​

All metrics come from our prometheus-cold datasource (UID aez9ck4wz05q8e, Org 6), which covers all available metrics without retention or delay restrictions. Node-level system metrics are collected via prometheus-node-exporter at a 15-second scrape interval.

Comparison windows​

We compared two 14-day windows:

Both windows use avg_over_time(...[14d:1h]) — a 14-day average at 1-hour subquery resolution, computed as an instant query at the boundary timestamp. This smooths out transient spikes while preserving meaningful shifts.

Filtering​

Supernodes are excluded via the label filter cc_client!~".*-super". Only instances with role=~"cc|ec" are included. Network metrics exclude loopback and virtual interfaces (device!~"lo|veth.*|docker.*|br.*").

Fleet-level results​

Here's what happened across the entire non-supernode fleet:

Let's dig into each metric.

Network bandwidth: the biggest win​

The most striking result is the 60% drop in network receive bandwidth — from 2.20 MiB/s to 0.87 MiB/s on average. This is PeerDAS in action: nodes now verify only sampled slices of blob data rather than downloading entire blobs.

Daily trend​

Looking at the daily time series, the decline actually began before the fork itself — around November 28–29 — suggesting some nodes started running fork-compatible client versions with PeerDAS-like optimizations ahead of the December 3 activation. On the fork day itself, the fleet-average RX was already down to 0.26 MiB/s.

The post-fork trend shows a brief recovery as nodes settled into the new protocol, stabilizing around 1.4 MiB/s by mid-December — still a 36% reduction from the pre-decline baseline of ~2.2 MiB/s.

Network transmit bandwidth also decreased, though more modestly — from 0.20 to 0.18 MiB/s (−13%).

CPU utilization: moderate increase, one clear outlier​

Fleet-average CPU rose from 4.66% to 6.08% — a 30% relative increase, but in absolute terms still very modest. This is the expected trade-off: PeerDAS introduces data availability sampling routines (compute) in exchange for reduced data transfer (bandwidth).

The fork-day spike​

The daily time series reveals two notable spikes:

• December 4: 14.82% — the day after fork activation. All clients simultaneously processed new consensus rules, PeerDAS bootstrapping, and EOF-related state transitions. This spike was transient and resolved within 24 hours.
• December 10: 10.09% — coincides with BPO-1 raising the blob target/max from 6/9 to 10/15. The higher blob capacity required additional sampling work.

After settling, the new baseline sits around 5.5–5.7% — roughly 1 percentage point above pre-fork levels.

Per-consensus-client breakdown​

Not all consensus clients handled Fusaka equally:

On the positive side, Lighthouse was the only consensus client to decrease CPU usage (−13%), suggesting either an efficient PeerDAS implementation or concurrent optimizations shipped in its fork-compatible release.

Lodestar remained essentially flat (+1%), and Prysm (+11%), Teku (+5%), and Grandine (+21%) showed moderate increases — all within comfortable bounds for typical node hardware.

Per-execution-client breakdown​

Execution clients saw a more uniform increase, consistent with the raised gas limit (EIP-7935) and new EVM opcodes (EOF):

Besu (+101%) and Geth (+64%) saw the largest relative increases. For Besu, the doubling (from a very low 2.58% baseline) likely reflects the new gas limit activating code paths that were previously idle. Geth's increase is consistent with heavier block processing under the raised 60M gas ceiling.

Nethermind bucked the trend entirely with a −26% decrease — a surprising result that may indicate a coincidental version update with CPU optimizations during the measurement window.

The full picture: CPU heatmap across all 36 pairings​

This heatmap shows the percentage change in CPU utilization for every CC×EC combination. Red means higher CPU after Fusaka, green means lower:

The Nimbus row is unmistakable — deep red across all EC pairings, with nimbus + ethrex showing an extreme +1060% (from 0.83% to 9.63%, though the low baseline suggests the pairing may have been partially offline pre-fork). The Nethermind column is notably green across most CC pairings, reinforcing that Nethermind itself saw a concurrent optimization.

Memory: a modest but welcome decrease​

Average memory usage dropped from 6.06 GiB to 5.59 GiB (−8%) across the fleet. This aligns perfectly with PeerDAS's design: nodes no longer hold entire blobs in memory, only the sampled slices they need for verification.

Daily trend​

The CPU + memory daily trend chart above also shows the memory trajectory (orange line). There's a sharp drop on December 4 (from 6.22 to 5.26 GiB) — the first full day after the fork — followed by a gradual recovery over the next two weeks as caches and new protocol state accumulated. By December 18, memory was back to 6.27 GiB, suggesting the initial drop was partly transient (e.g., cleared blob caches from the pre-fork protocol).

The 14-day average still shows a net decrease because the early post-fork days pull the average down significantly.

Per-consensus-client breakdown​

Nimbus (−20%) and Grandine (−17%) saw the largest memory reductions. Interestingly, Nimbus traded memory savings for CPU — a classic compute-vs-memory trade-off in its PeerDAS implementation.

Prysm was the only consensus client to increase memory (+6%), suggesting it keeps more sampling state in memory than peers. Given Prysm's CPU increase was moderate (+11%), this is a reasonable engineering choice.

Per-execution-client breakdown​

Besu saw a dramatic −35% memory drop — from 6.59 to 4.31 GiB. This is the largest single-client change in the dataset and likely reflects aggressive cache invalidation during the fork transition. Reth moved in the opposite direction (+13%), possibly due to its database engine accumulating more state under the new protocol.

Disk I/O: reads halved, writes slightly up​

Read rate​

Disk reads dropped 53% — from 2.58 to 1.21 MiB/s. This is directly attributable to PeerDAS: nodes no longer need to retrieve full blobs from disk for verification. The reduction was visible across nearly all pairings.

Some notable per-pairing changes:

• teku + ethrex went from 26.18 MiB/s (an extreme outlier pre-fork) to 2.86 MiB/s — a 89% drop
• lighthouse + reth dropped from 1.71 to 0.55 MiB/s (−68%)
• grandine + reth moved in the opposite direction — from 0.08 to 3.23 MiB/s — suggesting a change in Grandine's blob retrieval strategy for PeerDAS

Write rate​

Disk writes increased 11% — from 1.73 to 1.92 MiB/s. The increase is modest and likely comes from:

1. PeerDAS sampling metadata — nodes now store sampling proofs and column indices
2. EOF state changes — the EVM Object Format introduces new bytecode validation and storage patterns
3. Higher gas limit — more transactions per block means more state writes

This is a small overhead relative to the substantial read savings.

Full 36-pairing matrices​

For the detail-oriented, here are the complete CPU and memory matrices across all CC×EC combinations.

CPU utilization — before Fusaka (%)​

CPU utilization — after Fusaka (%)​

Memory usage — before Fusaka (GiB)​

Memory usage — after Fusaka (GiB)​