Where the slot goes: six instruments, calibrated

Over three days in August, Lodestar timed 21,530 newPayload calls to its Ethrex peer and reported a mean of 372.6 ms; Ethrex timed the same calls from its own end of the same socket and reported 164.8 ms. Both figures are right as far as the client reporting them is concerned, since the two stopwatches start and stop at different points, but the 208 ms between them is larger than most of the differences this series has spent six editions measuring between execution clients.

For six editions we compared execution clients through whichever consensus client happened to be watching them, and it took until the sixth to work out how much of what we were reading belonged to the watcher rather than the watched. Six consensus clients run against the same fleet here, no two of them measure a slot the same way, and one of them cannot produce a mean for the engine call at all.

Read this first

• This is not a benchmark and there is no winner in it. We measure a live fleet and report what the instruments show. Where clients differ we say by how much on which quantity, and we put the timing numbers next to the disk and memory numbers from our pruning and hardware footprint censuses, because a client at one end of one axis is usually at the other end of another.
• Consensus-side engine timings are not comparable between clients. Each client's wall clock around the call includes its own transport and serialisation. Measured against the execution client's own server-side handler, that bracket costs 6 ms on one client and 208 ms on another.
• There is no portable way to spot a syncing execution client. Teku's node-syncing gauge separates a staged-sync peer cleanly while Teku's block-import counters call the same node the healthiest on the fleet. Lodestar is the mirror image: its engine result counters separate it and its sync-status gauge does not.
• Metric type matters more than metric count. Lodestar publishes 1,175 metric names and Grandine 327, but what decides whether you can answer a question is whether the timer is a Prometheus histogram with _sum, _count and le labels or a counter carrying its latency in a string label, which no quantile function can read.
• Six claims from earlier editions did not survive. They are below with the mechanism behind each, because the mechanisms repeat and the individual mistakes do not.

What each client lets you see

All six run against the same six execution clients on identical hardware, so what differs between these rows is the instrumentation rather than the workload.

	Arrival	Engine call	Named stages	4 s boundary	Metric names
Lighthouse	gauge, latest value	histogram	11, all gauges	on the outcome counter	725
Nimbus	histogram	histogram	3	on arrival	505
Prysm	histogram	histogram	3	on arrival	588
Grandine	histogram, coarse	histogram	2	no	327
Teku	counter, interval labels	counter, interval labels	11, on one label	on arrival	410
Lodestar	histogram	histogram	7, and they add up	on arrival, processed and head	1,175

The last column turns out to predict little of the rest. Grandine publishes 327 metric names, the fewest here, and answers the engine question cleanly; Teku publishes 410, names eleven stages of a block's journey on a single label in a way nobody else does, and cannot produce a median from any of them; Lodestar publishes 1,175 and is the only client whose decomposition closes, in the sense that its parts sum to its separately measured whole within 4 ms.

What separates those rows is metric type rather than metric count. A histogram with _sum and _count gives an exact mean and an interpolated quantile, whereas a gauge gives you the most recent block and nothing about the distribution, which is why Lighthouse's eleven stages support an average over a window but not a percentile. A counter carrying its range in a string label gives bucket occupancy and no mean at all.

Two rows in that table came out the opposite way round from what we expected. Lighthouse names eleven stages, as many as Teku does, but every one of them is a gauge holding the most recent block, so the decomposition that looks richest on paper is also the one no quantile can be taken from. Teku puts no le bucket on any of its slot timers and can still answer the four-second question, because its arrival counter happens to carry adjacent intervals of [3000,4000) and [4000,5000); the boundary is exact by accident of bucket choice rather than by design.

The two stopwatches

Four of the six execution clients time their own engine-API handlers and publish a _sum and a _count, so the mean is exact and no consensus client is in the measurement. Put that next to what the consensus client on the other end reports for the same call over the same three days, and the difference is transport, serialisation, and whatever else the caller brackets.

Consensus client	newPayload gap to the execution client's own handler
Lighthouse v8.2.1	5.9 to 34.3 ms
Grandine 2.0.6	6.9 to 31.7 ms
Prysm v7.1.8	9.9 to 43.4 ms
Nimbus v26.7.0	18.9 to 49.9 ms
Lodestar v1.45.0	181.3 to 207.7 ms
Teku 26.7.1	no row, see below

Teku has no row because its engine metric is a counter with no sum, which is the property from the capability table turning up as a practical limit rather than a theoretical one.

Lodestar's row held against four independent server-side implementations: Ethrex reported 164.8 ms where Lodestar reported 372.6, Besu 270.8 against 466.9, Nethermind 363.4 against 557.7 and Erigon 397.8 against 579.1, so four separate codebases with four separate timer implementations put the gap between 181 and 208 ms. On all four of those pairings the two sides counted within one call of each other, so this is a per-call difference and not an accounting difference. That is not true everywhere in the table: on forkchoiceUpdated Lighthouse counts 22,229 against Besu's 22,414, a 0.83% gap, and Nimbus runs five or six calls behind on newPayload.

The gap scales with the size of the request rather than with the execution client on the other end. On forkchoiceUpdated, which carries no payload body, Lodestar's gap runs 30.7 to 41.9 ms against Lighthouse's 3.6 to 6.3 ms over the same four peers, so a payload-sized request costs Lodestar roughly five times what a head-update request costs it on the same connection to the same process. Lodestar is the one client here written in TypeScript, and encoding a full block body to JSON in that runtime is the obvious candidate, but we are not claiming it: having ruled out the queue and the response read, we have bounded where the time is not spent rather than shown where it is. Settling that needs a profile rather than another dashboard.

For an operator this means that a Lodestar newPayload panel reading 400 ms is reporting between a third and a half of a number that has nothing to do with the execution client, and that changing the execution client will not move that part of it. It also means every cross-client engine comparison in this series, ours included, was carrying an offset of this kind without anyone knowing.

The table has four limits worth stating, since it is the figure from this series most likely to be quoted out of context.

The two sides may not be bracketing the same event at all. One measures from writing the request to reading the response, the other measures its own handler from entry to exit, and the difference between those two definitions sits inside every number in the table without any metric on either side being able to separate it out. For Lodestar's 190 ms that hardly matters, since no plausible definitional difference is that large, but for the rows at 6 to 50 ms it may be most of what we are looking at.

Nor did we measure the wire. Neither side publishes a transport timer, so the gap bounds network time and serialisation together rather than measuring either of them.

The units are inferred rather than declared. None of these metrics carries a unit annotation we could read, so we assigned units from the metric names and then checked each execution client's mean against the four or five independent consensus-side observations of the same host; every scaled value landed inside the range its counterparts measured, which corroborates the assignment without proving it.

Coverage is 18 pairings of a possible 30. Reth accounts for five, excluded everywhere in this series. Geth accounts for five more: it publishes its engine timing as a decaying summary with quantiles, no sum and no usable count, so it contributes no server-side row at all. Grandine's Erigon pairing was unscrapable for more than half the window. And Nimbus against Besu is out because that host sat five and a half million blocks behind, so the 1,757 calls it did record are not a sample of normal operation. Teku has no row anywhere, because a counter with no sum cannot produce a mean to subtract.

All 36 cells, both calls, with both means and both call counts, are in engine-stopwatch-calibration-2026-08.csv, which is the file to reach for if you are comparing engine metrics between consensus clients.

One row in it turns out not to be about the caller at all. On forkchoiceUpdated, Nimbus reads 60.9 to 196.5 ms above the server-side handler, an order of magnitude more than Lighthouse, Prysm or Grandine on the same call, and its Nethermind pairing carries most of that. Nethermind agrees from its side: its own handler reads 43.98 ms for the Nimbus pairing against 1.90 to 3.84 ms for the other five, so both ends of that one pairing are unusual at the same time. Part three found the same shape looking from Nimbus and could not explain it either.

Spotting a syncing execution client

Each edition of this series excluded its Reth pairing, because in each edition's window that host was mid staged-sync and validating nothing. Three of the six had caught up by the second week of August. The three that had not, paired with Prysm, Grandine and Lodestar, make a useful control, because a consensus client attached to a node that validates nothing should look obviously broken and mostly does not.

Under optimistic sync the consensus client imports the block anyway and carries on, which is what the specification asks of it, so the block-import counters record a success. An execution client that is not executing answers instantly, so the engine timers record the quickest responses on the fleet. Two editions in a row, the node doing none of the work came out ahead on every timing metric its consensus client published.

Whether a client can tell you any of this depends on one thing, which is whether anything it publishes carries the execution client's verdict on a payload as a label value.

	Signal that separates a non-validating peer	Tested at the extreme
Lodestar	lodestar_execution_engine_notify_new_payload_result_total{result="SYNCING"}	yes
Prysm	new_payload_optimistic_node_count against ..._valid_node_count	yes
Lighthouse	execution_layer_payload_status{status="syncing"}	no, its Reth was healthy
Grandine	none. Call volume only	by inference
Teku	none in this window. Its node-syncing gauge worked in the last one	see below
Nimbus	none	no

Lodestar counted 21,530 SYNCING newPayload results on the affected pairing against 0 to 13 on the others, and its VALID series does not exist there at all, since a labelled counter is only published once something has incremented it; its forkchoice counter agrees at 21,553. The gauge is what fails here, with lodestar_sync_status reading 3, meaning synced, on all six pairings including that one. Its log sets the sharpest trap in the series, because the slot line opens with the word Synced and carries the live chain tip to the block, and the only thing separating it from a healthy pairing is the word syncing inside the exec-block: field.

Prysm separates the pairing cleanly and gives you the share along with it. new_payload_optimistic_node_count read 21,529 on the affected pairing against 0 on Geth and Ethrex, while its valid counterpart read 0 against 21,529. Those same counters turned up something no edition had reported, namely 3,278 optimistic responses on Prysm's Besu pairing, 15.2% of the window, on a host that was otherwise sitting at the chain tip.

Lighthouse publishes a metric of the right shape in execution_layer_payload_status{method="new_payload"}, which carries valid and syncing, and the syncing series fired 5 times on Erigon and once on Nethermind, so the counter is demonstrably live. Its own Reth pairing was healthy through this window, so the extreme case never reached it. Its two node-level gauges, optimistic_sync and sync_eth2_synced, read identically across all six pairings and carry no execution-layer information at all.

Grandine publishes no payload-status label anywhere in its job, so the only thing separating its affected pairing is call volume: 4.06 newPayload calls per forkchoiceUpdated against 1.07 to 1.53 elsewhere, which is what repeated redelivery to a node that never returns a verdict looks like. Its head-tracking and error counters are no help, since its error count on that pairing sits mid-range, below Besu's.

Teku cannot encode this on its engine counter at all. beacon_engine_requests_total has two outcome values and no more, success and error, so a SYNCING response is filed as a success and the affected pairing is indistinguishable from Geth on both columns. Its beacon_node_syncing_active gauge does work, and part five measured it separating a staged-sync pairing at all 4,320 scrapes of that window. In this window every Teku pairing was at the chain tip, including its Reth, so the gauge correctly read 0 everywhere and the capability could not be retested.

Nimbus carries an engine-response counter whose status label holds the HTTP code, not the payload status, and it reads 200 for every response on every pairing. Its beacon_head_execution_number does catch a node that has fallen behind, which is how its degraded Besu pairing shows up at 5.5 million blocks back, but that is a statement about how far the node has drifted rather than about whether the execution client is validating.

So three of the six carry the execution client's verdict in a label, one carries it only in call volume, and two do not carry it in any form. The split does line up with metric count on these six, since the three that carry it are also the three publishing the most names, though six clients are nowhere near enough to tell a pattern from a coincidence, and the capability table above shows that same count predicting little else: Grandine answers the engine question cleanly on the fewest names here, while Teku names eleven stages on 410 and cannot take a mean from any of them.

Five ways an instrument misled us

Every edition turned up at least one of these, and the shapes recur across clients even where the metric names have nothing in common.

Capability hiding in a label value. Three metric-name searches across eleven substrings told us Grandine timed no engine call, when in fact it times both. The metric is ETH1_API_REQUEST_TIMES, in capitals, with the method sitting in a method label, so nothing a person would think to grep for gets anywhere near it. A metric name records what its author called the thing, not what the thing measures.

Latency in a string. Teku's beacon_block_import_delay_counter_total and beacon_engine_requests_total are both counters, with the duration held in a label called interval whose values look like [1000,1500). There is no le label, so histogram_quantile returns an empty result and a hint, and any quantile has to be computed by hand from the bins.

Bucket schemes that do not match. Those two Teku metrics carry three different sets of interval strings between them, and the only value common to all three is [500,1000), so summing across stages without a filter pools eleven populations into one pseudo-histogram with overlapping bins. For the Geth pairing that comes to 236,324 observations against 21,476 blocks, and a median that reads about 25 ms where block arrival sits between 1,500 and 2,000 ms.

A bucket edge you cannot select. Teku's unbounded bucket is [12000,∞), written with a Unicode infinity sign, so a selector written interval="[12000,inf)" matches nothing and returns no error while doing it. On a dashboard that presents as an absence of slow blocks.

Denominators inside one metric family. Lodestar's lodestar_gossip_block_elapsed_time_till_received counts every imported block, while lodestar_gossip_block_received_to_block_import, which shares its prefix, counts only the ones that arrived by gossip. On our pairings that is 21,529 against 21,452, so subtracting one from the other leaves a 0.4% residue that does not cancel and that nothing in the output mentions.

None of these five produces an error. Each returns a number that looks like the number that was asked for.

What we got wrong

Six claims from the first five editions did not survive later measurement. Those posts stay up as published, with the corrections collected here rather than edited into them.

Part one said Lighthouse exposes no histogram for engine-call timing, so a precise percentile was out of reach. The histogram exists: execution_layer_request_times{method="new_payload"} is complete at roughly 21,571 observations per pairing over three days, which makes the conservative sampling caveat that claim produced unnecessary. That edition's core result, that the execution client moved the would-fail-attestation count by 3.7x on identical hardware, does not rest on it either way.

Part two said Nimbus publishes no execution timing and cannot rank execution clients even in principle. That was the edition's title as well as its thesis, and it is wrong: engine_api_request_duration_seconds{request="newPayload"} exists per ec_client and is complete at about 21,577 observations per pairing. The true version is narrower and sits in that edition's own comparison table, where Lighthouse's granular block-delay gauges and its exceeded-slot counter return nothing for Nimbus, while Nimbus's own beacon_block_delay histogram is complete and is what the edition went on to measure. We took the absence of another client's gauges for the absence of the capability.

Both of those come from the same mechanism, which caught us a third time in part four with Grandine: deciding that a client does not expose something because a search for the name we expected turned up nothing.

Part three read the driver effect as a six-way ordering. The underlying effect holds, in that the same execution client does run at different speeds depending on which consensus client is driving it, but the ordering does not: split into twelve-hour sub-windows it dissolves into the two-tier split part four went on to report.

Part four concluded there is no gentle consensus client and no demanding one. That claims more than the data supports. Across four execution clients that time themselves, Spearman between driver columns runs from -0.10 to 0.60. Grandine sits at the low end on all four and Nimbus at the high end on all three it appears in, with the caveat part six attached to the first of those: two of Grandine's four cells do not carry one call per block, its Besu pairing at 33,009 calls against the window's 21,506 blocks and its Erigon pairing at 10,453. A driver's effect is partly shared across execution clients and partly specific to the pair. Part four's tier boundary also holds on only one of the three other execution clients that can test it, and it flips when read on Nethermind's second engine timer instead of its first.

Part five said no consensus-client counter can tell you your execution client is at the chain tip. That goes further than the evidence does. Teku's own beacon_node_syncing_active read 1 on all 4,320 scrapes of that window on the staged-sync pairing and 0 on two of the five others, so a counter did separate the affected node. What it reports is the beacon node's own state rather than the execution client's, and it goes to 1 for reasons that have nothing to do with the execution layer, which is why the section above pairs it with a log check. The metric was in the inventory we ran for that same post, and we did not connect it.

Part five then recommended alerting on a beacon-node syncing gauge as the general guard. That does not carry across clients: lodestar_sync_status held one value on all six Lodestar pairings, including the one whose execution client validated nothing for three days.

Three of those six are a claim about absence made from a search rather than an inventory, two are a claim about generality made from a single client or a single window, and one is an aggregation artifact that a finer cut dissolved.

Why we did not remeasure everything on one window

The six editions ran on six windows between late June and mid August. Execution-client versions moved underneath them: Ethrex from 16.0.0 to 23.0.0, Erigon from v3.4.3 to v3.5.4, Nethermind from 1.38.1 to 1.39.3. Part two flagged this in its own methodology and printed both version sets.

The tidy fix would be to pin one window, hold every version fixed and rerun all six clients against it, and we are not going to do that, because a frozen run answers a narrower question than the one an operator has. It tells you what six clients did on one Tuesday under one set of builds, and it cannot tell you whether the thing you are looking at survives a version bump, a fleet upgrade or a week when the network is busier. Our fleet keeps running and keeps being upgraded, which is a nuisance if you want a benchmark and the whole point if you want a census.

The drift is therefore in the data deliberately. A result that turns up on six instruments across seven weeks and three Ethrex generations has been tested against conditions that moved underneath it, whereas a result that turns up once and never returns is worth knowing about for the same reason; a frozen run would have filed the second of those as a finding. The companion post separates the two.

What the drift costs is absolute comparison across windows, and we make none. Ethrex's newPayload mean reads 119 ms under Grandine in one window and 372.6 ms under Lodestar in another, a difference larger than most of what this series measured between execution clients, and the calibration above accounts for most of it. Neither figure is a property of Ethrex on its own.

What to take from this

• Never conclude a client lacks instrumentation from a metric-name search. Inventory the scrape job with count by (__name__) ({job="X"}) and read the label values. Half our corrections trace to skipping this.
• Check the metric type before you pick a query function. A _total suffix on something that behaves like a timer means the latency is in a label, and quantile functions will return nothing.
• Do not compare engine timings across consensus clients without subtracting the bracket. On our fleet that bracket runs from 6 ms to 208 ms depending on which client holds the stopwatch.
• Find the sync signal for your own client and test it against a node you know is behind. The signals that work are the ones recording what the execution client said about a payload, rather than the ones describing your node's own state, and every client names them differently.

Coming next

A companion post follows next week and takes the findings rather than the instruments: what appeared on every instrument in every window, what appeared once and did not return, and what the timing numbers look like beside the disk, memory and pruning numbers from our earlier censuses.

Catching the failures above is what StereumLabs AI does on our fleet, on the measurement stack we described here. If you run Ethereum infrastructure and want this lens on your own nodes, reach us at stereumlabs.com or contact@stereumlabs.com.

Methodology

The calibration, capability and sync-signal figures come from one window, 2026-08-07T00:00:00Z to 2026-08-10T00:00:00Z, on the Prometheus-cold datasource (uid aez9ck4wz05q8e), with the fleet labels documented in build your own dashboards. Every value is an exact counter delta of the form sum by (X) (M) - sum by (X) (M offset 3d) with the subtraction outside the aggregation, evaluated instant at the closing anchor rather than at query time. No increase() and no rate(), because both extrapolate. Chain height ran 25,699,358 to 25,720,864 over the window's 21,600 slots. NDC2 is a six-by-six matrix of consensus and execution clients on identical 12-core bare-metal hosts in Vienna, so each execution client appears six times, once per consensus-client pairing.

• Consensus-side engine timers, established by inventorying each job rather than by guessing names: execution_layer_request_times{method="new_payload"} (Lighthouse), engine_api_request_duration_seconds{request="newPayload"} (Nimbus), new_payload_v1_latency_milliseconds (Prysm, already milliseconds, carries ec_client and no method label, and the v1 in the name is legacy while the fleet runs V4 payloads), ETH1_API_REQUEST_TIMES{method="engine_newPayloadV4"} (Grandine, capitalised), lodestar_execution_engine_http_client_request_time_seconds{routeId="notifyNewPayload"} (Lodestar), and beacon_engine_requests_total{method="new_payloadV4"} (Teku, a counter with no sum).
• Server-side engine timers: rpc_request_duration_seconds{method="engine_newPayloadV4"} (Ethrex), besu_rpc_request_time{methodName="engine_newPayloadV4"}, nethermind_json_rpc_call_duration_micros{method="engine_newPayloadV4"} and engine_new_payload (Erigon). Geth publishes rpc_duration_engine_newPayloadV4_success as a decaying summary: it has quantiles, no sum, and its _count is a reservoir sample size rather than a cumulative counter, reading 1 at the anchor with three-day deltas of -3, 0, 0, 1, 1 and 1. It contributes neither a mean nor a denominator.
• The gaps subtract the server-side mean from the consensus client's own mean for the same call on the same pair of hosts, so each is a difference between two measurements of one event and does not depend on host equivalence. Per-pairing call counts on the two sides agree to within one on all four Lodestar cells, which is what the headline gap rests on. Elsewhere they diverge by up to nine calls on newPayload and, on Lighthouse against Besu on forkchoiceUpdated, by 185. Units carry no annotation on any of these metrics; we inferred them from the names and checked each execution client's mean against the four or five independent consensus-side observations of the same host, where every scaled value landed inside the range its counterparts measured.
• The capability matrix comes from count by (__name__) ({job="X", deployment="NDC2"}) per job followed by reading label values, and from group by (le) on each candidate histogram to list its finite edges. Metric-name counts at the anchor are Lodestar 1,175, Lighthouse 725, Prysm 588, Nimbus 505, Teku 410, Grandine 327. These move with a fleet restart rather than drifting slowly, because a metric name is only published once one of its series has been incremented, which is why they are pinned to the anchor.
• The four-second column is group by (le) on every block-arrival and processed timer per job, plus count by (interval) on Teku's counters. Exact 4-second edges: beacon_block_delay (Nimbus), block_arrival_latency_milliseconds at le=4000 (Prysm), and lodestar_gossip_block_elapsed_time_till_received, ..._till_processed and ..._till_become_head (Lodestar). Teku has no le series matching 4 or 4000 anywhere, but beacon_block_import_delay_counter_total{stage="arrival"} carries adjacent [3000,4000) and [4000,5000) intervals. Lighthouse answers the question through beacon_block_delay_head_slot_start_exceeded_total, a counter of outcomes rather than a distribution.
• The sync signals were tested against the three Reth pairings that were mid staged-sync in this window, paired with Prysm, Grandine and Lodestar. The other three Reth hosts, paired with Lighthouse, Nimbus and Teku, had reached the chain tip by this window and read 25,720,861 to 25,720,864 with all pipeline stages equal. Teku's beacon_node_syncing_active therefore reads 0 on all six of its pairings here, which is correct for this window; the 4,320-of-4,320 reading quoted above is from part five's window, 2026-08-03 to 2026-08-06, when its Reth pairing was still in staged sync.
• Two pairings are excluded from the calibration table, and both belong to clients it reports on, so their ranges rest on three cells rather than four. Grandine's Erigon beacon node was unscrapable for most of the window, with up averaging 0.442, and its execution client sat 12,035 blocks behind head at the anchor. The Nimbus-paired Besu host stood at block 20,200,353 against a chain head of 25,720,864.
• What we did not measure. There is no transport timer on either side of the engine call, so the gap bounds network and serialisation together without separating them. We also did not establish that the two sides bracket the same event, and no metric here can settle that.
• The corrections were each re-derived against this datasource before publication. The Lighthouse and Nimbus histograms named above were confirmed present and complete at roughly 21,571 and 21,577 observations per pairing over three days in their editions' own windows.
• Our fleet runs no live validators. It receives mirrored validator-client traffic, so nothing here is an attestation-performance measurement.