Where the slot goes: Lodestar, and the 200 ms between two stopwatches

Lodestar publishes 1,175 metric names and 162 histograms on our fleet, 62% more metric names than any other consensus client we run and twenty-seven times Teku's histogram count. Part five ended by asking whether the richest instrument answers the question in this series' title better than the sparsest one did.

It does, and by a wider margin than we expected. Lodestar decomposes a block's journey into timers that add up: arrival, the wait on the execution layer, import, head. The pieces sum to the whole within 4 milliseconds. One bucket boundary sits at 4.000 seconds, so the share of blocks not processed by the attestation deadline is a count rather than an interpolation. It is 5.1% of blocks on the best execution client here and 9.0% on the worst, while only 0.11 to 0.16% of blocks arrive after 4 seconds at all.

Then the same instrument turned around and told us something about itself. Five execution clients time their own engine-API handlers, and four of them publish a mean we can subtract. Lodestar's wall clock around newPayload runs 181 to 208 milliseconds longer than what the execution client on the other end says it spent. The four other consensus clients that publish a mean sit 6 to 50 milliseconds above their peer. That gap is not a client comparison, it is a measurement about Lodestar, and it took a second stopwatch at the execution client's end to find it.

Read this first

• Reth is excluded for the sixth edition running, and this time the consensus client caught it unaided. Lodestar recorded 21,530 newPayload results on the Reth pairing and every one of them was SYNCING. Its payload-verification histogram counted zero observations in three days. Its data-column engine counter has no success_resolved series at all. Teku's block-import counters, in part five, rated the same node the healthiest in the fleet.
• Lodestar's own sync-status gauge does not catch it. lodestar_sync_status reads the same value on all six pairings at all 4,320 scrapes, including the one whose execution client validated nothing. Part five's advice to alert on a beacon-node syncing gauge was Teku-specific, and we say so below.
• Lodestar's engine timing is not comparable with other clients' engine timing. Its wall clock exceeds the execution client's own handler time by about 200 ms on newPayload and about 40 ms on forkchoiceUpdated. Within this post everything is measured on one instrument, so the comparison is fair. Carried across posts it is not.
• Lodestar essentially never recorded a newPayload under 100 ms, and that is a fact about its own wall clock. One observation below the 100 ms bucket edge in 107,649 calls across the five synced clients, on the engine request histogram, and zero in the 107,262 payloads the gossip-path verification histogram recorded. The execution clients were far quicker than that: on these same Lodestar-paired hosts Ethrex's own handler put 83.2% of its calls at or under 100 ms and Nethermind's put 34.8%. Lodestar's bracket adds about 195 ms, which the two-stopwatch section below measures. Quote means and the 300 ms bucket, not a fast-bucket share.
• Two metrics that share the lodestar_gossip_block_ prefix count different populations. The elapsed-time histograms count every imported block, 21,528 to 21,530 per pairing. The interval histograms count gossip-path blocks only, 21,425 to 21,484. And the payload-verification histogram silently drops payloads answered SYNCING, which is why its Reth count is zero rather than missing.
• Window and versions. Three days, 2026-08-07 to 2026-08-10, on Lodestar v1.45.0 paired with Geth v1.17.5, Nethermind 1.39.3, Besu 26.7.1, Erigon v3.5.4 and Ethrex 23.0.0, on NDC2 bare metal in Vienna. No client version changed inside the window on any of the fleet's 36 pairings. Every execution client except Reth was confirmed at the chain tip from its own container log at both ends.

The slot, end to end, and it adds up

Lodestar times a block's journey at seven points, some against the slot clock and some against the moment the block arrived. Here is where a Geth-paired node spent its slots:

Stage	What it is	Geth pairing
elapsed_time_till_received	slot start to block in hand	1,988 ms
before the engine call, derived	Lodestar's own pre-engine work	89 ms
execution_payload_verification_time	the wait on the execution client	483 ms
between the answer and import, derived	queueing and commit preparation	36 ms
block_block_import_time	writing the block to the chain	32 ms
elapsed_time_till_processed	slot start to processed	2,630 ms

The useful property is the last row. Add the five above it and you get 2,628 ms against a separately measured 2,630 ms. Across the five synced pairings that residual runs from 3.9 ms under to 2.5 ms over. The decomposition closes, which means you can attribute a slot rather than infer it.

Compare what part five could do with Teku's eleven named stages: report which bucket held the mode. Teku's stages are more finely named, and its instrument cannot add them up, because every one of them is a counter of bucket occupancy with no sum and no le label. Lodestar's are Prometheus histograms with _sum and _count, so means are exact and quantiles are available. More names is not what made the difference. Metric type is.

The two derived rows come from subtracting published timers. Lodestar reports received_to_execution_payload_verification at 571.8 ms and received_to_block_import at 607.5 ms on this pairing, both measured from the moment the block arrived, so the pre-engine 89 ms is the first of those minus the engine wait and the 36 ms is the second minus the first.

Two limits before the cross-client numbers. We have not verified from Lodestar's source what each boundary corresponds to internally, so this is a relative comparison on one client's instrument rather than an audited decomposition. And the 89 ms is arithmetic rather than a published timer. It runs 69 to 91 ms across pairings, which is the same size as Lodestar's state-transition timer at 72 to 91 ms, but whether those two overlap is not something the metrics can settle.

The four-second question, without interpolation

lodestar_gossip_block_elapsed_time_till_processed has a bucket boundary at 4.000 seconds. So does the arrival histogram. Nimbus and Prysm bucket arrival with the deadline in mind too, and part two counted blocks past 4 seconds on that basis. What no other client in this series can count without interpolating is the second column, blocks processed by the deadline:

Execution client	Blocks arrived by 4 s	Blocks processed by 4 s	Missed the deadline
Ethrex 23.0.0	99.861%	94.881%	5.119%
Besu 26.7.1	99.842%	94.617%	5.383%
Geth v1.17.5	99.865%	93.038%	6.962%
Erigon v3.5.4	99.889%	92.833%	7.167%
Nethermind 1.39.3	99.875%	90.956%	9.044%

Read the two middle columns together. Between 0.111% and 0.158% of blocks arrive after the 4-second mark, and that number is the same whichever execution client is attached, because arrival happens before the execution client is asked for anything. Then between 5.1% and 9.0% of blocks are still not processed at 4 seconds. Substantially all of the deadline pressure on this fleet is on the processing side of arrival, and the execution client moves it by a factor of 1.77.

Two things this table is not. It is not an attestation-failure rate: our fleet runs no live validators, processed is not the same event as attestable, and a late block is late for the whole network rather than for one node. And 4 seconds is the deadline under the current attestation schedule; the number to watch will move when the schedule does. What the table does give you is the same quantity measured five ways with one denominator, which is the cut that makes execution cost legible.

What the engine wait costs, and why the middle still will not rank

lodestar_gossip_execution_payload_verification_time is Lodestar's wait on the execution layer, one observation per gossip-path block, thirteen finite buckets from 50 ms to 4 s.

Execution client	Median (interpolated)	Mean (exact)	90th pct (interpolated)	Under 300 ms	Under 1 s
Ethrex	231 ms	373 ms	537 ms	76.5%	94.4%
Geth	325 ms	483 ms	810 ms	45.9%	92.8%
Nethermind	359 ms	554 ms	1,142 ms	40.9%	87.7%
Besu	410 ms	467 ms	735 ms	19.6%	95.8%
Erigon	466 ms	580 ms	1,017 ms	11.0%	89.7%

Ethrex fastest and Erigon slowest on the central statistics, which is the sixth consensus-client instrument to say so and the most heavily replicated result in this series. Ethrex's mean sits 207 ms below Erigon's, and that ordering survives everything we have thrown at it.

The middle does what it has done on all five earlier instruments, and this time without leaving the metric. Rank by median and it is Geth, Nethermind, Besu. Rank by mean and Besu moves from fourth to second, because its distribution is tighter: 95.8% of its calls finish inside a second, the best of the five, while only 19.6% finish inside 300 ms, the second worst. Rank by 90th percentile and Nethermind falls from third to last. Three statistics of one histogram, three different orderings of the same five clients. The median disagrees with the other two about the middle, and the mean and the 90th percentile disagree with each other about which client is last.

Besu is also the one client whose "under 1 s" share beats Ethrex's, so even the extremes are extremes on the statistics this series has been using rather than on every statistic. We would rather say that than round it off.

Both quantile columns are linear interpolations inside buckets 200 to 300 ms wide, so their orderings carry less weight than the mean's. Four of the five medians, Geth's, Nethermind's, Besu's and Erigon's, land inside the single 0.3 to 0.5 s bucket. On the 90th percentile only Ethrex has a bucket to itself: Besu at 735 ms and Geth at 810 ms share the 0.7 to 1.0 s bucket, and Erigon at 1,017 ms and Nethermind at 1,142 ms share the 1.0 to 1.3 s bucket, so which of those two is last is a product of the interpolation. The histogram counts how many calls fell inside a bucket, not where inside it, and the mean is the one column here computed from _sum and _count and free of that assumption.

One property of the instrument bears on all of these numbers. The engine histogram's finite buckets start at 50 ms and the request histogram's start at 100 ms, and across 107,649 newPayload calls on five synced execution clients, one landed below 100 ms. It was on the Erigon pairing, which is also the pairing whose SYNCING count for the window is one, and the verification histogram, which records nothing for a payload answered SYNCING, has zero below 100 ms in its 107,262 observations. Under Teku, in these same three days, 87.4% of Ethrex's calls came in under 100 ms. The next two sections are about why.

Besu's head update, now measured by Besu

forkchoiceUpdated separates Besu from the field for the sixth time:

Execution client	Lodestar's wall clock	Under 100 ms
Erigon	34 ms	98.60%
Ethrex	42 ms	96.76%
Nethermind	45 ms	96.08%
Geth	45 ms	96.57%
Besu	190 ms	13.50%

The 190 ms sits inside the 162-to-200 ms band that Lighthouse, Nimbus, Prysm and Grandine measured directly, and two of those instruments confirm the shape on their own sets of hosts over these same three days. Lighthouse's histogram reads Besu at 180.0 ms against 6.4 to 14.0 ms for the rest. Teku's coarser counter puts 18.18% of Besu's head updates under 100 ms and 97.31% to 99.93% of every other execution client's.

Then this window let us do something the previous five could not. Five execution clients time their own engine-API handlers server-side, with no consensus client in the measurement at all:

Execution client	Its own forkchoiceUpdated handler	Drivers covered
Nethermind	1.9 to 3.8 ms	five of six; the Nimbus pairing reads 44.0 ms
Geth	2.0 to 11.5 ms	six of six, median rather than mean
Erigon	3.1 to 21.0 ms	six of six
Ethrex	3.3 to 6.2 ms	six of six
Besu	140 to 174 ms	five of six; the Nimbus pairing was out of sync

Besu says it too. Every consensus client that has timed this call has been reporting a cost that Besu's own JSON-RPC handler confirms it pays, from 140 ms under Grandine to 174 ms under Lighthouse, against 2 to 21 ms for its four peers on the same fleet in the same three days. The magnitude now holds on six consensus-client instruments and on the execution client's own.

What we still cannot do is say why. The pruning census remains where the plausible cause sits, since that is where Besu was found doing storage work on the engine-API hot path, and we have not lined up individual slow calls against storage activity. Nor is this attestation budget: forkchoiceUpdated does not sit between a block arriving and that block becoming attestable, a limit stated in part three and still true here.

One more thing that table shows. Nethermind's handler runs 1.9 to 3.8 ms under five drivers and 44.0 ms under Nimbus, which is a smaller version of the anomaly part three found from Nimbus's side and could not explain. It is not a measurement artifact of Nimbus's histogram: Nethermind reports it about itself.

The 200 milliseconds between two stopwatches

Put the consensus client's wall clock next to the execution client's own handler time and the difference is transport, serialisation and whatever else the caller brackets. On our fleet that difference is small, and on one client it is not.

Execution client	Lodestar's newPayload	The client's own handler	Gap
Ethrex	372.6 ms	164.8 ms	207.7 ms
Besu	466.9 ms	270.8 ms	196.1 ms
Nethermind	557.7 ms	363.4 ms	194.3 ms
Erigon	579.1 ms	397.8 ms	181.3 ms

Four execution clients, four independent implementations of the server-side timer, one answer: about 195 ms. The same subtraction for the other consensus clients that publish a mean:

Consensus client	newPayload gap, across its execution clients
Lighthouse v8.2.1	5.9 to 34.3 ms
Grandine 2.0.6	6.8 to 31.7 ms
Prysm v7.1.8	9.9 to 43.4 ms
Nimbus v26.7.0	18.8 to 49.9 ms
Lodestar v1.45.0	181.3 to 207.7 ms

Denominators match to the call: Lodestar counted 21,530 newPayload requests to Nethermind over the window and Nethermind counted 21,529. So this is a per-call difference, not an accounting difference.

We can narrow where it is not. Lodestar's engine queue-wait timer averages 0.19 to 0.33 ms, and its response-stream timer 0.12 to 0.14 ms, so neither queuing nor reading the reply accounts for any of it. No pairing on the fleet used the SSZ engine transport, so it is not a case of one client sending a compact encoding while another sends JSON. And the gap scales with the size of the request rather than with the execution client: on forkchoiceUpdated, which carries no payload body, Lodestar's gap is 30.7 to 41.9 ms against Lighthouse's 3.6 to 6.3 ms over the same four execution clients. A payload-sized request costs Lodestar about five times what a head-update request costs it, on the same connection to the same process.

That points at request-side serialisation, and we are stopping short of saying so, because a metric that measures the absence of two things is not a measurement of the third. Lodestar is the one client here written in TypeScript, and encoding a full block body to JSON in that runtime is the obvious candidate. Confirming it needs a profile, not a dashboard.

The practical consequence does not wait on the mechanism. If you run Lodestar, about 195 ms of what your newPayload panel reads is not your execution client, and swapping the execution client will not move it. Against the five Lodestar means on this fleet that is between 31% and 56% of the number on the panel. And every comparison in this series that reads one consensus client's engine histogram against another's has been carrying an offset of this kind. Ours included: the Ethrex figures in part four came from Grandine, whose gap in this window is 11.6 ms, and the ones here come from Lodestar, whose gap is 207.7 ms.

Lodestar is the fastest driver, and the slowest

Part three found that the same execution client runs at different speeds depending on which consensus client is driving it. Part four corrected the reading from a six-way ordering to a two-tier split, on Nethermind's own payload-execution timer. This window has four execution clients timing themselves, so the question can be asked four times at once.

Driven by	Nethermind	Ethrex	Besu	Erigon
Grandine	167 ms (1)	83 ms (2)	210 ms (1)	467 ms (2)
Lighthouse	197 ms (2)	163 ms (3)	401 ms (5)	532 ms (3)
Teku	224 ms (3)	83 ms (1)	401 ms (4)	550 ms (4)
Prysm	310 ms (4)	182 ms (6)	336 ms (3)	660 ms (6)
Nimbus	312 ms (5)	179 ms (5)	not available	619 ms (5)
Lodestar	341 ms (6)	165 ms (4)	271 ms (2)	398 ms (1)

Identical execution-client and consensus-client versions on all 36 pairings, and each number is the execution client's own timer with no consensus client in the loop. Two of Grandine's four cells rest on a call population that is not one per block: it sent Besu 33,009 calls against the window's 21,506 blocks, and its Erigon cell covers 10,453 calls where every other driver sent about 21,530. So read Grandine's first-or-second placings with that in mind rather than as a like-for-like win. The Nethermind column is a time average of a gauge over 4,320 scrapes rather than a call-weighted mean, so call count is context there rather than a denominator.

Lodestar is sixth of six for Nethermind and first of six for Erigon. It makes Nethermind spend twice what Grandine makes it spend, and makes Erigon spend three fifths of what Prysm makes it spend. Lighthouse does the reverse trick, second of six for Nethermind and last of five for Besu.

That is the part of part four's conclusion that survives four execution clients: a driver's rank is not transferable between them. Spearman between the Nethermind and Besu columns is 0.00 across the five drivers they share, which is no rank agreement at all. What does not survive is the stronger version we wrote, that there is no gentle consensus client and no demanding one. Grandine is first or second on all four columns and Nimbus is fifth on all three it appears in, and across the six pairs of columns Spearman runs from -0.10 to 0.60 rather than sitting at zero. At five or six drivers none of those is significant, and the honest reading is a driver effect that is partly shared across execution clients and partly specific to the pair, rather than one or the other.

Part four's tier boundary survives on the metric it was defined on. Splitting these three days into six 12-hour sub-windows of nethermind_new_payload_execution_time, the slowest of Grandine, Teku and Lighthouse stays below the fastest of Nimbus, Lodestar and Prysm in 6 of 6, which takes it to 18 of 18 across three windows. Inside the tiers the order moves as before: Teku and Lighthouse swap in 2 of 6, and all three orderings of the slow tier appear.

It travels less well than that makes it sound, in two ways. Of the four execution clients that time themselves, the tier membership holds on two. On Ethrex it holds by 1.5 ms, with Lighthouse at 163.3 ms clearing Lodestar at 164.8 ms. On Besu and Erigon it breaks outright: Lighthouse and Teku, both nominally fast tier, are the two slowest drivers of Besu, and Lodestar is the fastest driver of Erigon by 70 ms. The split is also sensitive to which of Nethermind's two engine timers you read. On nethermind_new_payload_execution_time, which brackets block execution, Teku is third and firmly in the fast tier. On nethermind_json_rpc_call_duration_micros, which brackets the whole JSON-RPC call, Teku costs 340 ms against Nimbus at 324 ms and lands in the slow tier.

Those 117 ms are worth naming, because they are not the gap the previous section is about. That one measures what happens outside the execution client's JSON-RPC handler; this one is the parsing and dispatch inside it, and the two are disjoint segments of the same call. On that inner segment the ordering inverts: Nimbus costs 11.8 ms, Lighthouse 15.8, Lodestar 22.0, Grandine 24.5, Prysm 61.7 and Teku 116.6. The client with the largest outer gap has close to the smallest inner one.

Reth, and the three counters that caught it

Going into part five the Reth pairing looked healthy on five of Teku's counters and was mid staged-sync. This edition has the same node, still mid staged-sync, and Lodestar's instrument does not fall for it.

Lodestar counter, lodestar_ prefix dropped	Reth pairing	The other five
execution_engine_notify_new_payload_result_total{result="VALID"}	series absent	21,516 to 21,530
execution_engine_notify_new_payload_result_total{result="SYNCING"}	21,530	0 to 13
execution_engine_notify_forkchoice_update_result_total{result="VALID"}	series absent	21,538 to 21,540
gossip_execution_payload_verification_time_count	0	21,425 to 21,484
data_column_engine_result_total{result="success_resolved"}	series absent	14,376 to 17,485
sync_status	indistinguishable	one value, 4,320 scrapes each

The first five rows are what part five wanted and did not have. The payload-verification histogram is the one to put on a dashboard, and the reason it works is a design decision rather than an accident: it records an observation when the execution client returns a verdict, so a payload answered SYNCING produces no observation at all. On the Nethermind pairing it recorded 21,434 observations against the 21,447 gossip-path blocks its sibling interval histograms counted, and the 13 missing are the 13 SYNCING results Nethermind returned. On the Reth pairing the shortfall is all of them.

The sixth row is the one that should worry you. lodestar_sync_status held one value on every pairing at every scrape of the window, including this one. Lodestar's own log agrees with its gauge and disagrees with its counters, and does so in the most quietly misleading way we have seen in six editions:

Aug-09 23:59:53.000 info: Synced - slot: 14957997 - head: 0x6cbd…b869 -
exec-block: syncing(25720866 0xfcca…) - finalized: 0x6c06…a9b9:467435 - peers: 200

The line begins with the word Synced. The block number in it is the live chain tip, to the block. Only the word syncing inside the exec-block field is different from the healthy pairings, where the same field reads valid. On every one of the 12,649 slot lines from this host that reached our log store, spanning both ends of the window, it read syncing, and it never once read valid. On Geth, Besu, Erigon and Ethrex the ratio was 100% valid; on Nethermind, 12,713 of 12,717.

Reth's own log closes it. Canonical chain committed appears zero times across the three days. Committed stage progress pipeline_stages=4/14 stage=Execution runs from checkpoint 17,817,783 at the start of the window to 21,914,946 at the end, against a pipeline target of 25,645,374 and a live chain head of 25,720,864. The control that makes it airtight: the identical Reth build paired with Lighthouse, on the same fleet over the same window, logged Canonical chain committed 13,894 times.

And the trap that caught part five is still armed here. On Lodestar's slot-timing histograms, the ones this whole post is built from, the Reth pairing is the best node on the fleet: 2,333 ms to processed against 2,493 to 2,718 for the synced clients, and 97.76% of blocks processed inside 4 seconds against 90.96 to 94.88%. A node that verifies nothing is quick. The richest instrument in this series catches the syncing node in the counters that name the execution client's verdict, and flatters it in the metrics that measure how long the slot took.

So the honest generalisation, after two clients that each caught this and each missed it: there is no portable answer. Teku's beacon-node syncing gauge fired and its block-import counters did not. Lodestar's engine result counters fire and its sync-status gauge does not. The category that works is the one that records what the execution client said about a payload, and you have to find it per client.

What to measure on your own Lodestar nodes

• Alert on lodestar_execution_engine_notify_new_payload_result_total{result="SYNCING"} rising past a threshold, not on the VALID series going flat. This is the trap inside the fix. On the Reth pairing the VALID series does not exist at all, since a labelled counter is only published once it has been incremented, so a rate-to-zero or threshold rule on it evaluates to no data and stays quiet. Make the SYNCING rule a rate over a threshold rather than one that fires on any increment, because healthy pairings do return the occasional SYNCING. Over the window it counted 21,530 on the Reth pairing, a sustained 294 to 301 per hour for three days, against 13 on Nethermind and 1 on Erigon in total, and the busiest single hour on a healthy pairing stayed under ten. So increase(lodestar_execution_engine_notify_new_payload_result_total{result="SYNCING"}[1h]) > 100 sits in the gap between the two populations.

  Then catch the VALID series never appearing with a per-node set difference:

  count by (ec_client, instance) (lodestar_execution_engine_notify_new_payload_result_total)
    unless on (instance)
  count by (instance) (lodestar_execution_engine_notify_new_payload_result_total{result="VALID"})

  Do not reach for absent() here. It yields a sample only when its whole selector is empty, so on any Prometheus that scrapes more than one Lodestar node the healthy pairings keep it quiet forever. At our closing anchor absent(lodestar_execution_engine_notify_new_payload_result_total{result="VALID"}) returned no data, while the unless form above returned the Reth pairing alone. Its one limit is that a node which stops being scraped empties both sides and falls silent, so keep an up rule beside it. Add the rate of lodestar_gossip_execution_payload_verification_time_count dropping to zero, which does work as a flat-line rule because that series is present on all six pairings and sits at 0 only on Reth. All three flag the Reth pairing and nothing else.

• Do not alert on lodestar_sync_status. It read identically on the syncing pairing and the five healthy ones for all 4,320 scrapes of the window. Confirm from the log instead, where the field to grep is exec-block:, not the word Synced that precedes it.

• Chart lodestar_gossip_block_elapsed_time_till_processed_bucket at le="4.0" as your deadline panel. The boundary is on a bucket edge, so the share is a count with no interpolation. Pair it with lodestar_gossip_block_elapsed_time_till_received_bucket at the same edge: on our fleet the arrival column was flat at 99.84 to 99.89% while the processed column moved by four percentage points, which tells you whether to go and look at your peering or at your execution client. Use the full name on both: elapsed_time_till_received on its own also matches lodestar_data_column_elapsed_time_till_received_seconds and lodestar_sync_unknown_block_elapsed_time_till_received, which count different populations.

• Do not compare Lodestar's engine numbers with another client's. Its wall clock runs about 200 ms above the execution client's own handler on newPayload and about 40 ms above on forkchoiceUpdated. Inside one Lodestar fleet the comparison is fair; across clients it is an offset larger than most of the differences you would be ranking on.

• Read the engine wait three ways before you rank anything. Median, mean and 90th percentile of lodestar_gossip_execution_payload_verification_time put the middle three execution clients in three different orders on our fleet. Pick the statistic that matches what you care about, then say which one it was.

• Watch the denominators inside one metric family. lodestar_gossip_block_elapsed_time_till_received counts every imported block; lodestar_gossip_block_received_to_block_import, which shares its prefix, counts gossip-path blocks only. On our pairings that is 21,529 against 21,452, a 0.4% difference that will not show up as an error and will not cancel when you subtract one from the other.

Coming next in the series

That is all six consensus clients. The finale puts every instrument side by side: what each one can measure, what each one hides, which of the six can answer the deadline question and which cannot, and a list of the claims from earlier editions that did not survive being measured a second way.

That list stood at four entries. This edition adds a fifth and revises one that is already on it. The new entry is ours from part five: we recommended alerting on a beacon-node syncing gauge as the general guard against an optimistically-syncing execution client, and on Lodestar that gauge does not discriminate at all. The revision is to part four's driver finding, on both halves. Its two-tier split replicates on the metric it was defined on and on one of the three other execution clients that time themselves, not on all three, and not on Nethermind's own second timer. And its conclusion that there is no gentle consensus client and no demanding one was too strong: the driver effect is partly shared across execution clients and partly specific to the pair.

Six editions in, the pattern has not changed: a client comparison is only as good as the instrument you read it through, and the richest instrument on the fleet turned out to be the one furthest from the execution client's own reading. No amount of metric surface catches an offset like that on its own; it took a second stopwatch at the other end of the call. Catching those is what StereumLabs AI does on our fleet, on the measurement stack we described here. If you run Ethereum infrastructure and want this lens on your own nodes, reach us at stereumlabs.com or contact@stereumlabs.com.

Methodology

Numbers come from Lodestar's and the execution clients' own metrics on our NDC2 deployment (Vienna), queried on the Prometheus-cold datasource (uid aez9ck4wz05q8e), with the fleet labels documented in build your own dashboards. The window is 2026-08-07T00:00:00Z to 2026-08-10T00:00:00Z. Every figure is an exact counter delta, M - M offset 3d with the subtraction outside the aggregation, evaluated at the closing anchor rather than at query time; a cross-check with increase(...[3d]) agrees to within 0.2%. NDC2 runs a six-by-six matrix of consensus and execution clients on identical 12-core bare-metal hosts, so each execution client appears six times, once per consensus-client pairing.

• Window selection. count by (ec_client, ec_version) (up{job="lodestar", deployment="NDC2"}) at 12-hour steps over the preceding fortnight puts the last execution-client change at Nethermind 1.39.2 to 1.39.3, which landed at 2026-08-06T16:30Z, seven and a half hours before the window opens. All 36 pairings held identical execution-client and consensus-client versions throughout, checked at 12-hour steps, and changes(process_start_time_seconds[3d]) returns zero across every job that publishes it, so no process restarted inside the window. resets() is zero on every counter quoted.
• Chain progress. The chain produced 21,506 blocks in the window's 21,600 slots, height 25,699,358 to 25,720,864, slots 14,936,398 to 14,957,997. Geth, Nethermind, Besu and Ethrex agree on both endpoints to within one block, the spread being scrape timing; Erigon's head is the label value sync{stage="finish"} and matches.
• The decomposition is lodestar_gossip_block_elapsed_time_till_received, ..._till_processed, ..._till_become_head, lodestar_gossip_block_received_to_execution_payload_verification, lodestar_gossip_execution_payload_verification_time, lodestar_gossip_block_received_to_block_import and lodestar_gossip_block_block_import_time, all Prometheus histograms in seconds. Means are (sum - sum offset 3d) / (count - count offset 3d), which is exact. The elapsed-time family has six finite buckets at 0.5, 1, 2, 4, 6 and 12 s; the interval family has thirteen from 0.05 to 4 s. Arrival means are Besu 1,968.6, Erigon 1,970.3, Ethrex 1,978.0, Geth 1,988.0, Nethermind 1,996.7 ms, a 28 ms spread on clients whose engine wait spans 207 ms. Counting Lighthouse, Nimbus and Prysm, that is the fourth instrument in this series able to make the comparison and the fourth to find arrival independent of the execution client; Grandine's buckets were too coarse to carry it and part five reported arrival for one pairing only. Import time is 29.6 to 32.5 ms and the state-transition timer lodestar_stfn_process_block_seconds is 71.8 to 90.6 ms. The additive residual between the parts and elapsed_time_till_processed is -3.9 to +2.5 ms, per pairing Nethermind -3.9, Besu -2.5, Geth -1.8, Ethrex -1.1 and Erigon +2.5.
• Denominators. lodestar_import_block_by_source_total gives 21,425 to 21,485 gossip imports and 46 to 104 by root per pairing, totalling 21,529 to 21,531. That tracks the 21,528 to 21,530 the elapsed-time histograms count to within two observations rather than matching it: the two agree on Besu, Ethrex and Geth, and the counter runs two ahead on Erigon and one ahead on Nethermind. The interval histograms count the gossip subset only. lodestar_gossip_execution_payload_verification_time_count counts payloads that received a verdict: on Nethermind it is 21,434 against the 21,447 gossip-path blocks the interval histograms count, and the 13-observation shortfall equals the 13 SYNCING results that pairing recorded.
• Engine-call figures are lodestar_execution_engine_http_client_request_time_seconds with a routeId label taking notifyNewPayload, forkchoiceUpdated, getPayload, getBlobsV2 and getClientVersion, six finite buckets at 0.1, 0.5, 1, 2, 5 and 15 s. Counts are 21,529 to 21,531 for notifyNewPayload and 21,539 to 21,541 for forkchoiceUpdated, one per block. Below the 100 ms bucket edge the request histogram records one observation in 107,649 notifyNewPayload calls across the five synced pairings, on the Erigon pairing, whose notify_new_payload_result_total delta contains a single SYNCING result; the verification histogram records zero in its 107,262 observations, consistent with its dropping payloads answered SYNCING. We have not tied the one call to the one SYNCING result at slot level, so that pairing of facts is suggestive rather than established. Errors over the window are two, both Nethermind. This metric and the gossip-path verification histogram are separate instruments inside the same process and their newPayload means agree to within 4 ms on all five pairings.
• Server-side engine timing is rpc_request_duration_seconds{method="engine_newPayloadV4"} on Ethrex, besu_rpc_request_time{methodName="engine_newPayloadV4"}, nethermind_json_rpc_call_duration_micros{method="engine_newPayloadV4"} and engine_new_payload on Erigon, all with _sum and _count so the means are exact. Geth publishes rpc_duration_engine_newPayloadV4_success as a decaying summary with quantiles and no sum, so it appears only as a median snapshot in the forkchoiceUpdated table and not in the gap calculation. execution_engine_ssz_rest_requests_total on Nethermind is zero for all six drivers, so every pairing used the JSON engine transport.
• The two-stopwatch gap subtracts those server-side means from the consensus client's own mean for the same call on the same pair of hosts, so it is a difference of two measurements of one event and does not depend on host equivalence. Consensus-side means come from lodestar_execution_engine_http_client_request_time_seconds, execution_layer_request_times (Lighthouse), new_payload_v1_latency_milliseconds (Prysm), engine_api_request_duration_seconds (Nimbus) and ETH1_API_REQUEST_TIMES (Grandine). Teku publishes no sum for this call, so it has no row. Lodestar's contributing sub-timers are lodestar_engine_http_processor_queue_job_wait_time_seconds at 0.19 to 0.33 ms and lodestar_execution_engine_http_client_stream_time_seconds at 0.12 to 0.14 ms.
• The driver comparison uses each execution client's own timer at fixed version: nethermind_new_payload_execution_time as avg_over_time([3d]) on 4,320 samples per pairing, and exact histogram means for Ethrex, Besu and Erigon. Sub-window stability is the same Nethermind query at [12h] on six non-overlapping 12-hour steps. The tier boundary, defined in part four as the maximum of Grandine, Teku and Lighthouse against the minimum of Nimbus, Lodestar and Prysm, holds in all six; it was chosen after inspecting earlier windows, so this is a replication rather than an out-of-sample test. Applied to the three-day means of the other three execution clients it holds on Ethrex, where the fast group tops out at 163.3 ms against a slow-group minimum of 164.8 ms, and fails on Besu (401.2 against 270.8) and Erigon (550.3 against 397.8). Spearman between the six pairs of driver columns is Nethermind against Besu 0.00, Besu against Ethrex -0.10, Nethermind against Erigon 0.09, Ethrex against Erigon 0.49, Besu against Erigon 0.50 and Nethermind against Ethrex 0.60, computed over the five or six drivers each pair shares, where none is significant. Nethermind's two engine timers differ by 11.8 ms (Nimbus), 15.8 (Lighthouse), 22.0 (Lodestar), 24.5 (Grandine), 61.7 (Prysm) and 116.6 (Teku). The six Ethrex hosts ran 4.4 to 7.7% CPU and 4.1 to 9.0% disk utilisation over the window, so none was near saturation, but the load ordering tracks the latency ordering closely enough that we read it as a consequence of the effect rather than a cause of it.
• Comparisons with Teku use beacon_engine_requests_total{outcome="success"} evaluated on this window rather than part five's, so every Teku figure quoted here covers the same three days as the Lodestar figures. For method="new_payloadV4" the share in the [0,100) interval is Ethrex 87.43% of 21,579 calls, Nethermind 41.68% of 21,749, Geth 40.01% of 21,570, Besu 6.64% of 21,779 and Erigon 4.07% of 21,919. For method="forkchoice_updatedV3" it is Ethrex 99.93% of 21,671, Geth 99.93% of 21,669, Nethermind 99.80% of 21,733, Erigon 97.31% of 21,789 and Besu 18.18% of 21,709. Lighthouse's forkchoiceUpdated means in this window are Besu 180.0, Nethermind 6.4, Ethrex 12.4, Erigon 12.6 and Geth 14.0 ms. Metric surface area, anchor-evaluated as distinct metric names and _bucket names per scrape job: Lodestar 1,175 / 162, Lighthouse 725 / 129, Prysm 588 / 65, Nimbus 505 / 21, Teku 410 / 6, Grandine 327 / 70. These counts move by much more than a series or two, and the cause is a restart rather than drift. A metric name is only published once one of its series has been incremented, so a client's surface rebuilds after a restart. Lodestar's distinct names climbed from 1,138 at the window start to 1,175 at the close and its _bucket names from 155 to 162; Lighthouse's climbed from 661 to 725 and from 112 to 129, a 15% move on the histograms. The headline ratio moves with them: Lodestar publishes 62% more metric names than Lighthouse at the closing anchor and 72% more at the window start, which is why these are pinned like every other figure here. Part five, anchored at 2026-08-06T00:00Z, published Lodestar 1,184 / 162 and Lighthouse 781 / 144; those were correct at that anchor, as these are at this one. All of these counts include non-timing histograms.
• Sync was verified from container logs, not metrics. Geth logged Chain head was updated number=25,699,361 at 00:00:01 on the opening day and number=25,720,867 at 00:00:00 on the closing one. Besu logged Managing new added block 25699361 and 25720866. Erigon logged [5/5 Execution] parallel done blk=25699361 and blk=25720866, plus 12,545 head validated lines; the head updated line quoted in earlier editions does not appear in v3.5.4, which is one more instance of a log format changing under a version bump. Nethermind logged Finished pre-warming caches for block 25699361 and 25720867. Ethrex logged Prewarm pass for block 25699362 and 25720867. Lodestar's own slot line read exec-block: valid on 12,868 of 12,868 ingested lines for Geth, 12,541 of 12,541 for Besu, 12,578 of 12,578 for Erigon, 12,848 of 12,848 for Ethrex and 12,713 of 12,717 for Nethermind. Log shipping had an outage of roughly 26 hours starting on the first day of the window; inside the covered periods the rate is 1,800 slot lines per six hours, one per slot, so these are complete counts of an incomplete sample and are quoted as ratios rather than as slot totals.
• Reth's exclusion. Canonical chain committed appears zero times on the Lodestar-paired Reth host across the window, against 13,894 occurrences on the Lighthouse-paired host running the same build over the same window. Its pipeline ran pipeline_stages=4/14 stage=Execution throughout, advancing from checkpoint 17,817,783 to 21,914,946 against target 25,645,374. Lodestar's slot line read exec-block: syncing on all 12,649 ingested lines and valid on none. Its getBlobsV2 traffic to that host was 290,118 calls against 23,280 to 64,484 elsewhere, and lodestar_data_column_engine_result_total recorded 290,118 null_response and no success_resolved series. We have not confirmed from Lodestar's source that optimistic import is what produces this pattern; the specification permits it and every counter is consistent with it, but the causal link is our reading.
• One pairing outside this edition was also out of sync. The Nimbus-paired Besu stood at block 20,200,353 at the closing anchor and recorded 1,757 newPayload observations against about 21,524 on Nimbus's other pairings, with no counter reset. It is excluded from the Besu column of the driver table and from Nimbus's gap range, and it is not part of any figure attributed to Lodestar.
• Our fleet runs no live validators. It receives mirrored validator-client traffic, unchanged across this window, so nothing here depends on attestation signing and no figure in this post is an attestation-performance measurement.