Where the slot goes: what held across six windows

Besu's forkchoiceUpdated call has now been timed by six consensus clients across four windows, and by Besu itself. Every one of them puts it above 100 ms. On five of the six the peers sit in single or low double digits; on the sixth they do not, and that row is in the table below. The absolute numbers move around, the instruments disagree about each other, the software underneath changed three times, and the result keeps turning up.

That pattern, rather than any single measurement, is what this series has to report. The companion post deals with the instruments themselves and the ways they mislead; this one deals with the handful of results that showed up whichever instrument was pointed at them.

Read this first

• Nothing here ranks clients. These are measured values on named quantities. A client at one end of the engine-timing axis is usually at the other end of the disk or memory axis, and the last section puts those side by side so nothing here reads as a league table.
• Absolute values are not comparable across windows or instruments. Ethrex's newPayload mean reads 119 ms on one instrument and 372.6 ms on another. What travels is the shape of the field, not the numbers.
• Six windows, five software states. Late June to mid August, with Ethrex going 16.0.0 to 23.0.0 and Erigon v3.4.3 to v3.5.4 underneath. That is on purpose, and the companion post explains why.
• Reth appears in none of the comparisons. In each edition's window, the Reth host paired with that edition's client was mid staged-sync and validated nothing. Three of the six had caught up by the second week of August. Two editions caught this after their own metrics said the opposite.

Three results that turned up every time

Three things survived being measured through every instrument that could measure them: the cost of Besu's head update, the ends of the newPayload field, and the independence of block arrival from the execution client. Each is below with every measurement we published, including the ones taken through instruments we now know are biased.

Besu's head update

Instrument	Window	Besu	Its peers, same instrument
Lighthouse	07-24 to 07-27	197 ms	6 to 14 ms
Nimbus	07-24 to 07-27	193 ms	63 to 261 ms
Prysm	07-24 to 07-27	208 ms	4 to 10 ms
Lighthouse	07-27 to 07-30	187 ms	5.8 to 14.5 ms
Nimbus	07-27 to 07-30	187 ms	63 to 262 ms
Prysm	07-27 to 07-30	200 ms	4.3 to 9.9 ms
Grandine	07-27 to 07-30	162 ms	4.3 to 7.4 ms
Teku	08-03 to 08-06	modal bucket 100 to 300 ms	over 96% under 100 ms
Lodestar	08-07 to 08-10	190 ms	34 to 45 ms
Besu's own handler	08-07 to 08-10	140 to 174 ms	peers' own handlers, 1.9 to 21 ms

Parts one and two did not publish this call at all. Part three measured it on three instruments in its own window, part four measured it on four in the next one, and the top three rows are the same three instruments a week apart.

Two columns in that table behave quite differently. Besu's holds between 162 and 208 ms across five instruments and four windows, and Besu's own server-side handler reads 140 to 174 ms, which overlaps the bottom of that band rather than sitting inside it; the difference between the two is the bracket each consensus client puts around the call. The peer column holds nothing at all, running from 4 ms under Prysm through 34 to 45 ms under Lodestar to 261 ms under Nimbus, which is the same instrument bracket the companion post measures, turning up here in a results table. The peers move with whichever stopwatch is reading them, in other words, and Besu largely does not, because most of its 190 ms is being spent inside Besu.

The three instruments that were read twice give the tightest version of that. Lighthouse reported 197 ms and then 187, Nimbus 193 and then 187, Prysm 208 and then 200, a week apart and on different Besu builds; the week-to-week movement is therefore smaller than the spread between instruments reading the same week.

Nimbus is the one row where the separation disappears rather than the magnitude. Its peers read 63 to 262 ms, which puts two of them above its own Besu figure, so on that instrument Besu is not distinguishable from the field. The magnitude replicates on six instruments and the separation on five.

The ends of the newPayload field

Instrument	Window	Ethrex	Erigon
Lighthouse	ending 06-23	low end, with Geth	618 would-fail attestations a day against 150 to 170
Prysm	07-24 to 07-27	171 ms mean	516 ms mean
Grandine	07-27 to 07-30	119 ms mean	568 ms mean
Teku	08-03 to 08-06	87.6% of calls under 100 ms	2.6%
Lodestar	08-07 to 08-10	372.6 ms mean	579.1 ms mean

Five instruments, five windows, three Ethrex versions and three Erigon versions, and Ethrex sits at the low end and Erigon at the high end every time.

Look at the Ethrex column: 171 ms under Prysm, 119 under Grandine, 372.6 under Lodestar. Those are the same client on the same fleet doing the same job, read through three different stopwatches in three different weeks. Part two is missing from this table because that edition reported Nimbus could not measure this. It can, and the companion post carries that correction.

What this does not say is that Erigon is slow at executing blocks. Erigon runs its pruning on the engine-API hot path, which the pruning census documented before this series started, and it stores a third of what Geth stores. The last section puts those next to each other.

Arrival does not move with the execution client

Block arrival is the largest single piece of the slot and it happens before the execution client is asked for anything. Four instruments could measure it across all their pairings, and all four found it flat:

Instrument	What it found
Lighthouse	1.66 to 1.87 s
Nimbus	a narrow band, the edition's centrepiece
Prysm	flat, the third instrument to find it
Lodestar	1,968.6 to 1,996.7 ms, a 28 ms spread

Lodestar's row is the tightest of the four, at a 28 ms spread across the same five pairings whose engine wait spans 207 ms, which puts a fairly firm bound on the idea that the execution client has anything to do with how quickly a block turns up.

Two instruments could not carry the comparison at all. Grandine's arrival histogram puts 77.6 to 78.3% of its observations at or below 2.5 s, so an interpolated median lands inside a bucket holding three quarters of the data and the nine milliseconds separating its pairings is bucket occupancy rather than timing; Teku published arrival for one pairing. Neither appears in the table above, which is the honest way to count them: four instruments carried this result, not six.

What did not hold

The middle of the field. Every edition that could resolve more than the two ends produced a different middle: Geth sat ahead of Nethermind under Prysm and the two swapped under Grandine, while under Lodestar three statistics drawn from a single histogram put the middle three clients in three different orders, by mean, by the share inside the first bucket and by the share in the tail. The middle is not a stable property of the execution client on this fleet, and we no longer report one.

The driver effect as an ordering. Part three found that the same execution client runs at different speeds depending on which consensus client drives it and reported a six-way rank, which dissolves once the window is split into twelve-hour pieces. Part four replaced it with a two-tier split that holds on the metric it was defined on, across eighteen sub-windows in three windows, but that travels less well than the count suggests: of the four execution clients that time themselves the tier membership holds on two, and it flips when read on Nethermind's second engine timer rather than its first.

That there is no gentle consensus client and no demanding one. That was part four's phrasing, and it goes further than the data does. Grandine sits at the low end of all four execution-client columns that can measure it and Nimbus at the high end of all three it appears in, though two of Grandine's four cells rest on a call population that is not one per block, as part six noted when it published them. Spearman between the driver columns runs from -0.10 to 0.60 over the five or six drivers each pair shares, none of which is significant at that many, so the defensible reading is that a driver's effect is partly shared across execution clients and partly specific to the pair, with neither half accounting for the other.

Anything inside about 100 ms across pairings. At a fixed Nethermind version, nethermind_new_payload_execution_time reads 195 ms under one consensus client and 355 ms under another. Every consensus client runs its own execution-client hosts, so a cross-pairing difference smaller than that spread is not attributable to the execution client at all.

The four-second question

Attestations are due a third of the way into the slot. The question an operator has is what share of blocks are not ready by then, and whether the six instruments can answer it turns out to depend on one design decision each: where the histogram's bucket edges fall.

Instrument	Blocks that arrived late	Blocks not processed in time
Lodestar	yes, le=4.0 on arrival	yes, le=4.0 on processed and on head
Nimbus	yes, le=4.0 on beacon_block_delay	no
Prysm	yes, le=4000 on block_arrival_latency_milliseconds	no
Teku	yes, via adjacent [3000,4000) and [4000,5000) intervals	no
Lighthouse	as an outcome count, not a distribution	as an outcome count
Grandine	no. Its buckets step 1, 2.5, 5, 10 s and skip the deadline	no

Five of the six can tell you how many blocks turned up late, and Teku manages it without putting an le bucket on any slot timer at all, because the interval strings on its arrival counter happen to break at 4000 ms.

Only Lodestar can tell you how many blocks turned up in time and were still not ready, and on that instrument 0.11 to 0.16% of blocks arrived after 4 seconds while 5.1 to 9.0% were still unprocessed at 4 seconds, with the execution client moving the second figure by a factor of 1.77.

Those two columns put the practical result of the whole series in one place. Almost none of the deadline pressure on this fleet comes from blocks turning up late; nearly all of it comes from what happens after they turn up, and four of the six instruments cannot see that part at the resolution the deadline needs.

Two limits apply to all of this. Our fleet runs no live validators, so processed is not the same event as attestable and none of these figures are attestation-failure rates, and 4 seconds is the deadline under the current attestation schedule rather than a fixed property of the protocol.

Four axes, and why none of them is a ranking

The timing numbers above are one cut through these clients, and our earlier censuses measured the same six on axes that have nothing to do with the engine API, where they do not line up at all.

Execution client	Stored bytes	RAM average	Disk write	Engine-call timing
Erigon	509 GB	6.0 GiB	22.7 MB/s	high end
Ethrex	598 GB	15.6 GiB	1.1 MB/s	low end
Nethermind	1,225 GB	4.9 GiB	1.1 MB/s	between the ends
Besu	1,261 GB	5.6 GiB	1.5 MB/s	between the ends, plus the head-update cost
Geth	1,688 GB	7.9 GiB	0.9 MB/s	between the ends

Erigon sits at the high end of engine-call timing while storing a third of what Geth stores, Ethrex sits at the low end of engine-call timing while wanting three times Nethermind's memory, and Geth writes the least to disk per second while keeping the largest footprint on it. None of those columns survives being turned into an ordering, and an operator picking a client is picking which of them matters on the hardware they have. The disk figures come from the pruning census and the memory and CPU figures from the hardware footprint census.

What we still cannot explain

Why Besu's head update costs what it does. The pruning census found Besu doing storage work on the engine-API hot path, which is the plausible cause and remains unproven. Lining individual slow calls up against storage activity needs a trace, not a dashboard.

Nethermind under Nimbus. Nethermind's own forkchoiceUpdated handler runs 1.9 to 3.8 ms under five drivers and 44.0 ms under Nimbus. Part three found the same shape from Nimbus's side and could not explain it. It is not an artifact of Nimbus's histogram, because Nethermind reports it about itself.

Where Lodestar's 200 ms goes. The queue averages 0.33 ms and reading the reply 0.14 ms, and no pairing on the fleet used the SSZ engine transport, so none of the obvious in-band candidates accounts for it. The gap scales with request size rather than with the execution client on the other end, which points at request serialisation without establishing it, and settling that needs a profile.

Coming next

This is the last edition of the series. The companion post carries the instrument reference and the calibration table, each per-client edition carries its own section on what to measure, and the fleet labels all of these queries use are documented in build your own dashboards.

Running six consensus clients against six execution clients on identical hardware, for seven weeks, is what StereumLabs AI does on our fleet, on the measurement stack we described here. If you run Ethereum infrastructure and want this lens on your own nodes, reach us at stereumlabs.com or contact@stereumlabs.com.

Methodology

Every figure in the replication tables was published in the edition it is attributed to, and each edition states its own window, versions and queries. All six ran on the Prometheus-cold datasource (uid aez9ck4wz05q8e) against the NDC2 deployment in Vienna, a six-by-six matrix of consensus and execution clients on identical 12-core bare-metal hosts, with fleet labels documented in build your own dashboards.

• The six windows and their execution-client versions. Part one, three days ending 2026-06-23: Geth v1.17.3, Nethermind 1.38.1, Besu 26.6.0, Erigon v3.4.3, Ethrex 16.0.0. Part two, 06-27 to 06-30: Besu 26.6.1, Erigon v3.4.4, Ethrex 17.0.0, the other two unchanged. Part three, 07-24 to 07-27, and part four, 07-27 to 07-30: Geth v1.17.4, Nethermind 1.39.2, Besu 26.7.0, Erigon v3.5.2, Ethrex 22.0.0. Part five, 08-03 to 08-06: Geth v1.17.5, Besu 26.7.1, Erigon v3.5.4, Ethrex 23.0.0, Nethermind still 1.39.2. Part six, 08-07 to 08-10: Nethermind 1.39.3, the rest unchanged. Each edition confirmed no version changed inside its own window.
• Besu's head update. The first three rows are part three's own measurements on its window; the next four are part four's, which read all four instruments over the same three days; the Teku row is part five's and the Lodestar row part six's. Parts one and two did not measure this call; their rows in the table come from part four, which read execution_layer_request_times{method="forkchoice_updated"} and engine_api_request_duration_seconds{request="forkchoiceUpdated"} on its own window. Besu's own handler is besu_rpc_request_time{methodName="engine_forkchoiceUpdatedV3"}, measured in part six's window against the equivalent server-side timers on Erigon, Ethrex and Nethermind.
• The newPayload table mixes statistics on purpose, because the instruments do not offer the same one. Part one reports a counter of would-fail attestations rather than a latency. Parts three, four and six report means from _sum and _count. Part five reports the share of calls in an interval bucket, because Teku publishes no sum for the call. The column headings say which.
• Arrival. Lighthouse from beacon_block_delay_observed_slot_start, a gauge averaged over its window. Nimbus and Prysm from their arrival histograms. Lodestar from lodestar_gossip_block_elapsed_time_till_received, means computed from _sum and _count, giving Besu 1,968.6, Erigon 1,970.3, Ethrex 1,978.0, Geth 1,988.0 and Nethermind 1,996.7 ms. Grandine's arrival histogram places 77.6 to 78.3% of observations at or below 2.5 s, so an interpolated median sits inside a bucket holding three quarters of the data and cannot separate pairings. Teku published arrival for one pairing.
• The driver comparison and its tier boundary are from parts three, four and six. Part six is the widest cut, using four execution clients that time themselves rather than one, and its Spearman coefficients between driver columns run -0.10, 0.00, 0.09, 0.49, 0.50 and 0.60 over the five or six drivers each pair shares. None is significant at that many drivers, which is the reason the conclusion is a shape rather than a coefficient.
• The four axes. Stored bytes and their per-host ranges come from the pruning census, measured on that post's own window with the client versions it names, which are older than any window in this series. RAM, CPU and disk-write figures come from the hardware footprint census. Neither was remeasured for this post, so read those columns as the state of the fleet at those dates rather than at this one. Ethrex's memory figure is from a 64 GiB host while the others ran on smaller boxes.
• Reth. In each edition's window, the Reth host paired with that edition's consensus client was running the staged-sync pipeline and returned SYNCING to every payload. By 2026-08-07 the Lighthouse-, Nimbus- and Teku-paired Reth hosts had reached the chain tip; the Prysm-, Grandine- and Lodestar-paired ones had not. No Reth figure appears in any comparison here.
• Our fleet runs no live validators. It receives mirrored validator-client traffic. Nothing here is an attestation-performance measurement, and the four-second shares are not attestation-failure rates.