Where the slot goes: Grandine, and the metrics a name search cannot find

This edition was nearly published with its central claim inverted. We ran three Prometheus metric-name searches against Grandine, across eleven substrings including payload, forkchoice, engine, execution, latency, delay and duration. The first returned nothing, the second two histograms and the third eleven, and not one of them was the engine-call timing we were looking for. The obvious conclusion was that Grandine is the sparse end of this series: a client that times block arrival and blob fetches and nothing else. A fact-check caught it before publication.

Grandine times both engine calls, per method, at more than one observation per slot. The metric is called ETH1_API_REQUEST_TIMES. It is in capitals, it contains none of the eleven substrings we searched for, and the engine method lives in a label rather than the name, so method="engine_newPayloadV4" sits inside a series whose name says only that some eth1 API was called. No name search reaches it.

Grandine is not the sparse end of this series, and it is not the rich end either: an inventory of its scrape job returns 70 histograms against Lodestar's 158 and Lighthouse's 143, and on total metric names it is the sparsest of the six at 330. The point is not where it ranks. The point is that a comparatively modest client still hid a per-method engine timer from three searches, which means the search was the problem, not the client.

What the metric shows once you find it: Besu's head-update call costs 162 ms against 4 to 7 ms for every other execution client, which is the fourth independent instrument to say so, and the middle of the execution-client field reorders again.

Read this first

• We got this wrong first, and the mechanism matters. A name-pattern search cannot establish that a client lacks instrumentation. ETH1_API_REQUEST_TIMES is all-caps, carries the engine method in a method label, and matches none of the substrings you would think to grep for. The reliable form is an inventory per scrape job, count by (__name__) ({job="grandine"}), and then reading labels. We now do that first, and the finale of this series will re-derive every "client X instruments Y" statement in earlier editions the same way.
• Two of Grandine's histograms use default Prometheus buckets, and that limits them. Both the arrival histogram and ETH1_API_REQUEST_TIMES step 1 s, 2.5 s, 5 s, 10 s at the top. Neither has a boundary at the 4-second attestation deadline, and for forkchoiceUpdated values of 4 to 7 ms the bottom buckets at 5 and 10 ms are coarser than the quantity being measured. We quote means where the buckets cannot support quantiles, and say which is which.
• We are not quoting a cross-client arrival spread from Grandine. Its arrival histogram puts 77.6 to 78.3% of observations at or below 2.5 s, so an interpolated median lands inside a bucket holding three quarters of the data. The nine milliseconds separating the five pairings is arithmetic on bucket occupancy, not a timing measurement, and we say so rather than adding it to this series' arrival control.
• Reth is excluded, for the fourth edition running, and Grandine's own counters show why more sharply than before: it logged 86,350 newPayload calls for the Reth pairing against about 22,000 for the other five, on a chain that produced 21,521 blocks in the window. Repeated re-delivery to a node that never returns VALID is what that looks like. Its 34 ms mean is a SYNCING artifact.
• Window and versions. Three days, 2026-07-27 to 2026-07-30, on Grandine 2.0.5 paired with Geth v1.17.4, Nethermind 1.39.2, Besu 26.7.0, Erigon v3.5.2 and Ethrex 22.0.0, on NDC2 bare metal in Vienna. No execution-client version changed inside the window; the last bump was 2026-07-23. Every execution client was confirmed at the chain tip from its own container log at both ends. This window begins where part three's ended.

The metric a keyword search cannot find

Here is what the three failed searches looked for, against what the job publishes:

What we searched	Histograms returned	Engine-call timing found
.*(payload|forkchoice|execution).*	0	no
.*(engine|latency|delay|arrival|import).*	2	no, the two are block arrival and the blob fetch
.*(latency|delay|duration|milliseconds|seconds).*	11	no
count by (__name__) ({__name__=~".+_bucket", job="grandine"})	70	yes

The second search is the instructive one. Searching for engine does return an engine-API histogram, beacon_engine_getBlobsV2_request_duration_seconds, which is how we came to believe Grandine timed the blob fetch and nothing else on that side. The third widens to eleven histograms, mostly PeerDAS and KZG timers, and still misses the payload call. Only the inventory finds it, because ETH1_API_REQUEST_TIMES shares no substring with any word a person would think to type.

It carries method="engine_newPayloadV4", "engine_forkchoiceUpdatedV3", "engine_getPayloadV5", "engine_getBlobsV2" and "engine_exchangeCapabilities". Alongside it sits MUTATOR_BLOCK_PROCESSING_TIMES, a per-slot consensus-side block-processing timer. Name searches can reach some of what a client publishes; what they cannot do is establish that something is absent.

The generalisable point for anyone auditing their own stack: a metric's name tells you what its author called it, not what it measures. Capability hides in two places a substring search does not reach, unconventional casing and label values, and both appear here at once. Inventory the scrape job, then read the labels.

What Grandine's engine timing shows

With the metric located, Grandine answers the question this series has been asking of every client. Over the window, per pairing:

Execution client	newPayload mean	newPayload 90th pct	forkchoiceUpdated mean
Ethrex 22.0.0	119 ms	168 ms	6.9 ms
Nethermind 1.39.2	239 ms	501 ms	4.3 ms
Geth v1.17.4	279 ms	492 ms	7.4 ms
Besu 26.7.0	376 ms	764 ms	162 ms
Erigon v3.5.2	568 ms	1,147 ms	5.6 ms

Two things carry over from earlier editions and one is new.

Carrying over: Ethrex fastest and Erigon slowest on newPayload, which now holds under four consensus clients. And the middle reorders again, with Nethermind ahead of Geth here where Prysm had Geth ahead of Nethermind, which is part three's finding that the middle of the field does not rank across pairings, arriving on schedule with a fourth instrument.

New, and worth its own section: Besu's head-update call.

Besu's head update, now on four instruments

forkchoiceUpdated is the engine call almost no dashboard shows. Four consensus clients on our fleet time it, on four separate sets of execution-client hosts, and they agree:

Instrument	Besu	Everyone else	Besu's multiple
Lighthouse	187 ms	5.8 to 14.5 ms	13x to 32x
Nimbus	187 ms	63 to 262 ms	0.7x to 3.0x
Prysm	200 ms	4.3 to 9.9 ms	20x to 46x
Grandine	162 ms	4.3 to 7.4 ms	22x to 37x

All four rows are the same three days, 2026-07-27 to 07-30, so nothing here is being compared across windows.

Besu's own figure is the stable one: 162 to 200 ms on four instruments and four sets of hosts, at roughly 22,000 calls each. The gap to its peers is not equally stable. On three of the four instruments Besu costs 13 to 46 times what the other execution clients cost, and on Nimbus it does not, because under Nimbus both Ethrex and Nethermind also read in the hundreds of milliseconds, which no other instrument reproduces. So four instruments carry the magnitude and three carry the gap. Even at three, it is the most reproducible client-specific result in this series. The mechanism remains bounded but unproven, and the pruning census is where the plausible cause sits, since that is where Besu was found doing storage work on the engine-API hot path.

We also repeat the limit stated in part three: this cost is not attestation budget. forkchoiceUpdated does not sit between a block arriving and the block becoming attestable. It is engine-layer work your node pays for and your dashboard probably does not show you.

The arrival histogram, and the number we are not quoting

Grandine's arrival histogram, beacon_block_gossip_slot_start_delay_time, recorded 21,596 to 21,717 observations per pairing against the 21,521 blocks these 21,600 slots produced. It counts gossip block receptions, not slots, so the surplus of 75 to 196 is a node hearing some blocks more than once and the total cannot tell you that every slot is represented. It also contains a trap worth more attention than the numbers it yields:

Execution client	Median	Mean of the same data	Ratio
Erigon	1.938 s	8.8 s	4.5x
Geth	1.939 s	19.2 s	9.9x
Besu	1.943 s	18.0 s	9.3x
Nethermind	1.943 s	19.9 s	10.2x
Ethrex	1.947 s	19.2 s	9.9x

Between 0.51% and 0.88% of observations exceed the top finite bucket of 10 s. The histogram can only record them as "past 10 s"; sum/count weights them at their true magnitude, and each of them averages many minutes past slot start rather than seconds. That is enough to move the mean by an order of magnitude, and by different amounts per pairing, so a dashboard panel built on sum/count invents a 2.3x spread between execution clients out of nothing but how many stragglers each one saw.

What we will not do is turn the median column into a finding. Three quarters of the data, 74.9 to 75.5%, sits in a single bucket spanning 1 s to 2.5 s: 77.6 to 78.3% of observations fall at or below 2.5 s and only 2.7% below 1 s. So histogram_quantile is interpolating the median across that one bucket and the nine milliseconds separating the five pairings reflects bucket occupancy rather than arrival timing. This series has an arrival control built from Lighthouse, Nimbus and Prysm, whose buckets or gauges can carry it. Grandine's cannot, and saying so is more useful than a fourth data point that would not mean what the first three mean.

The driver effect: what replicates is a split, not an order

Part three found that the same execution client, same version, on equally loaded identical machines, runs materially slower under some consensus clients than others, and left the mechanism open. We now have a second window and finer granularity, and the finding needs both a confirmation and a correction.

Driven by	2026-07-24 to 07-27	2026-07-27 to 07-30
Grandine	195 ms	175 ms
Teku	239 ms	204 ms
Lighthouse	263 ms	219 ms
Nimbus	311 ms	307 ms
Lodestar	341 ms	316 ms
Prysm	355 ms	319 ms

Confirmed: there are two tiers, and the gap between them holds. Split those twelve-hour sub-windows across both three-day windows and the boundary holds in 12 of 12: the slowest of Grandine, Teku and Lighthouse is below the fastest of Nimbus, Lodestar and Prysm every single time. Grandine is the fastest driver in all twelve.

Corrected: the six-way ordering in the table above is an artifact of aggregating three days. Teku and Lighthouse trade places in 2 of the 12 sub-windows, and so do Lodestar and Prysm, because inside each tier the clients sit within a few percent of each other. Part three read an ordering; what the data supports is a split, and we would have overstated it without checking the sub-windows. The magnitudes also fell unevenly between the two windows, by 1.3% for Nimbus and 16.5% for Lighthouse, so the earlier reading that "every value came down 10 to 20%" was too tidy as well. Whatever moved them all downward was not a change in what we ask of the fleet: the mirrored validator-client load it serves was unchanged across both windows.

Three limits on the split itself, because a tier boundary is the kind of finding that is easy to draw around whatever the data happened to do. We defined the boundary after seeing these twelve windows, so "holds in 12 of 12" is a description of the data it was drawn on and not an out-of-sample test. It is also resolution-dependent: at six-hour granularity the two tiers overlap in 2 of 24 sub-windows, once by 2 ms and once by 36 ms. And each driver here is one consensus client on one execution-client host, so a driver effect and a host effect cannot be separated at a 62 ms tier gap, which is why part three's host-equivalence checks matter and why we still call the mechanism open.

And the two execution clients disagree about the drivers

Erigon publishes its own slot timeline, block_consumer_delay, with the moment it had the block body, execution start and execution end. Same version everywhere, same window:

Driven by	Body available	Execution starts	Execution ends	Span
Lodestar	2.004 s	2.565 s	3.084 s	0.518 s
Nimbus	1.829 s	2.473 s	2.997 s	0.525 s
Prysm	1.671 s	2.171 s	2.731 s	0.560 s
Grandine	1.736 s	2.176 s	2.758 s	0.582 s
Teku	1.702 s	2.349 s	3.048 s	0.699 s
Lighthouse	1.871 s	2.606 s	3.373 s	0.767 s

Nethermind's fastest driver is Grandine and its slowest is Prysm. For Erigon the slowest is Lighthouse, and the fastest is Lodestar by 6 ms over Nimbus, which is thin enough that the top of this column is a tie rather than a winner: split it by day and Lodestar leads by 23 ms, then the two land within 0.3 ms, then Nimbus leads by 3.5 ms. What survives is the crossing itself. Grandine, first for Nethermind in every one of twelve sub-windows, sits fourth of six here on every one of the three days. Lodestar and Nimbus, fifth and fourth for Nethermind, occupy the top two places here on all three.

How strongly the two disagree depends on which reading of Erigon's timeline you take, and we checked two: on the execution span the rank correlation with Nethermind's order is -0.60, and on absolute execution-end time it is -0.14. Neither is significant at six drivers, and the range between them is the point. There is no consistent relationship, in either direction, between how a driver treats Nethermind and how it treats Erigon.

So part three's open question closes in the only direction the data supports. The driver matters, and it is not a property of the driver: it is specific to the pair. There is no gentle consensus client and no demanding one.

Two limits on this table. The Lighthouse row is reconstructed from the two days of the three that reported, because a single scrape returning no value propagates through avg_over_time and voids the whole window; on both reporting days it was the slowest of the six. And the span column subtracts two separately computed medians, which approximates the typical execution window rather than being the median of a duration. The three fastest spans sit within 42 ms of each other, so what we read from this table is that the two orderings cross, not the position of any single driver in it.

What to measure on your own Grandine nodes

• Inventory before you conclude. count by (__name__) ({__name__=~".+_bucket", job="grandine"}) returns 70 histograms; drop the _bucket filter and you get all 330 metric names it publishes. The keyword searches that sent us wrong returned 0, 2 and 11. If you have ever concluded that a client does not expose something, that is the query to rerun.
• Both engine calls are there: ETH1_API_REQUEST_TIMES{method="engine_newPayloadV4"} and {method="engine_forkchoiceUpdatedV3"}. Watch the second one too; on one execution client it is the larger cost.
• Do not chart sum/count on either default-bucket histogram. On the arrival metric that reads ten times the median. Use histogram_quantile, and know that with 75% of the data in the single 1-to-2.5 s bucket the interpolated quantile is coarse.
• Neither histogram can answer the deadline question. No bucket boundary sits near 4 s. For a share-of-late-blocks figure you need a histogram bucketed with the deadline in mind, which on our fleet means Nimbus or Prysm.
• Do not carry a client comparison across pairings. The driver changes an execution client's measured speed by a wide margin, in tiers that hold and an order that does not, and differently for different execution clients.

Coming next in the series

Teku and Lodestar remain, and both already appear as drivers above: Teku sits in the fast tier for Nethermind and is second-slowest for Erigon, Lodestar the reverse. Then the finale, where all six clients and all of their instruments go side by side, with an honest list of the claims from earlier editions that did not survive being measured a second way. This edition contributes two entries to that list, both ours: the reading of part three that the driver effect was an ordering, and the premise this post was drafted with.

The pattern in all four editions is that a client comparison is only as good as the instrument you read it through, and that the instrument is easy to misread in ways that look like client differences: a mean poisoned by stragglers, a quantile interpolated across an over-full bucket, an ordering that only exists at one aggregation level, a capability hidden behind a name. Catching those is what StereumLabs AI does on our fleet, on the measurement stack we described here, and this edition is a fair sample of how often it catches us. If you run Ethereum infrastructure and want this lens on your own nodes, reach us at stereumlabs.com or contact@stereumlabs.com.

Methodology

Numbers come from Grandine's and the execution clients' own metrics on our NDC2 deployment (Vienna), queried on the Prometheus-cold datasource (uid aez9ck4wz05q8e), with the fleet labels documented in build your own dashboards. The window is 2026-07-27T00:00:00Z to 2026-07-30T00:00:00Z, with increase(...[3d]) and avg_over_time(...[3d]) evaluated at the closing anchor so figures are stable and reproducible rather than drifting with query time. No execution-client version changed inside it and Grandine held 2.0.5 throughout.

• How the instrumentation was established. count(count by (__name__) ({__name__=~".+_bucket", job="grandine", deployment="NDC2"})) returns 70; the same query without the _bucket filter returns 330 distinct metric names. Run per job across the six consensus clients, evaluated at the window's closing anchor rather than at query time, it gives 158 histograms for Lodestar, 143 for Lighthouse, 70 for Grandine, 65 for Prysm, 25 for Nimbus and 6 for Teku, and 1,160 / 778 / 330 / 591 / 535 / 411 total metric names respectively, which is the basis for saying Grandine is neither the sparse nor the rich end. These counts drift by a few series with query time, which is why we pin them to the anchor like every other figure here. Counts are of metric families exposed on this deployment and include non-timing histograms, so they are a rough measure of surface area rather than of timing coverage. The engine timing is ETH1_API_REQUEST_TIMES (_bucket, _count, _sum), seconds, with a method label taking engine_newPayloadV4, engine_forkchoiceUpdatedV3, engine_getPayloadV5, engine_getBlobsV2 and engine_exchangeCapabilities, six series per method, one per pairing. Finite bucket edges for both this and the arrival histogram are 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5 and 10 s. The three name searches that missed it are reproduced in the table above; none of them can match a name that is all-caps and carries its subject in a label.
• Engine-call figures. Means are increase(_sum[3d]) / increase(_count[3d]) per ec_client, which is exact. The newPayload 90th percentiles are histogram_quantile over windowed buckets and are interpolated inside the 0.25-to-1 s bucket for Geth, Nethermind and Besu, so treat them as approximate; Erigon's falls in the 1-to-2.5 s bucket. We do not quote quantiles for forkchoiceUpdated, whose 4-to-7 ms values sit below the second bucket edge. Call counts are 21,883 (Ethrex), 22,198 (Nethermind), 22,534 (Geth), 22,933 (Besu) and 23,981 (Erigon) against 21,600 slots carrying 21,521 blocks, so slightly more than one call per block.
• The four-instrument Besu comparison uses execution_layer_request_times{method="forkchoice_updated"} on Lighthouse v8.2.1, engine_api_request_duration_seconds{request="forkchoiceUpdated"} on Nimbus v26.7.0, forkchoice_updated_v1_latency_milliseconds on Prysm v7.1.7 and ETH1_API_REQUEST_TIMES{method="engine_forkchoiceUpdatedV3"} on Grandine 2.0.5, all as windowed means over this edition's window rather than part three's, so all four rows cover the same three days.
• The arrival histogram. Medians are histogram_quantile(0.5, ...); means are increase(_sum[3d]) / increase(_count[3d]). Share at or below the 10 s bucket: 99.124% Nethermind, 99.139% Geth, 99.156% Ethrex, 99.240% Besu, 99.486% Erigon. Share at or below 2.5 s: 77.58% Ethrex, 77.88% Besu, 77.96% Nethermind, 78.23% Geth, 78.29% Erigon, which is why the interpolated median is coarse and why we do not read a cross-client spread from it. Counts are 21,596 to 21,717 per pairing.
• The driver comparison uses avg_over_time(nethermind_new_payload_execution_time[3d]) for ec_version="1.39.2" grouped by cc_client, the execution client's own timer with no consensus client in the measurement. Sub-window stability is the same query at [12h] on a 12-hour step across both three-day windows, twelve points per driver. The tier boundary, defined as the maximum of Grandine, Teku and Lighthouse against the minimum of Nimbus, Lodestar and Prysm, holds in all twelve, and at six-hour granularity in 22 of 24. The boundary was chosen after inspecting these windows, so that is a description of this data rather than an out-of-sample test. Within-tier order changes in 2 of 12 sub-windows for Teku against Lighthouse and 2 of 12 for Lodestar against Prysm. Window-to-window change per driver is Nimbus -1.3%, Lodestar -7.4%, Grandine -9.9%, Prysm -10.1%, Teku -14.8%, Lighthouse -16.5%.
• Erigon's timeline is block_consumer_delay for ec_version="v3.5.2", a summary with client-computed quantiles; we take avg_over_time of the quantile="0.5" series for type="body_download", "pre_execution" and "post_execution", which is the average over the window of Erigon's own reported median rather than a median over the window. The Lighthouse pairing has one scrape with no value inside the window, which voids a three-day avg_over_time, so its row is the mean of the two daily values that did report, 1.835 and 1.907 s for body, 2.589 and 2.623 s for execution start, 3.377 and 3.368 s for execution end. Per-day spans for the other five are Lodestar 0.508/0.516/0.531, Nimbus 0.531/0.516/0.528, Prysm 0.565/0.556/0.559, Grandine 0.593/0.569/0.582 and Teku 0.699/0.688/0.709 s. Spearman rank correlation against Nethermind's driver order is -0.60 using the span and -0.14 using absolute execution-end time; with six drivers neither is significant, and we report the range rather than a single figure. We have not verified from Erigon's source what each type boundary corresponds to internally, so this table is a relative comparison across drivers on one metric rather than an absolute decomposition.
• MUTATOR_BLOCK_PROCESSING_TIMES, Grandine's per-slot block-processing timer, gives Geth 229, Besu 248, Erigon 254, Ethrex 332 and Nethermind 333 ms over the window. That ordering differs from the newPayload ordering on the same client, and because we cannot say from the metric alone which phases it spans, we report its existence and this disagreement as a caution rather than building a finding on it.
• Sync was verified from logs, not metrics. At both ends of the window each execution client's container log showed it at the chain tip, block 25,620,466 on 2026-07-27 and 25,641,987 on 2026-07-30, via Geth's Chain head was updated, Besu's block-add lines, Erigon's head updated, Nethermind's Received ForkChoice and Ethrex's Prewarm pass for block. Reth is excluded: no Canonical chain committed line appears in its log across the window, Prysm's new_payload_valid_node_count for the Reth pairing increases at zero per hour against about 300 for the other five, and Grandine issued 86,350 newPayload calls to it against about 22,000 for each of the others. On identical 12-core hosts; consensus and validator processes run on separate machines. Our fleet runs no live validators; it receives mirrored validator-client traffic, and that mirroring was unchanged across both windows, so it is not a candidate explanation for the between-window shift.