Where the slot goes: Teku, and the counter that called a syncing node healthy

Teku is the only client in this series that answers the question in the title directly. It splits a block's journey into eleven named stages and counts every one of them, so instead of inferring where the slot goes you can read it off: arrival, gossip validation, pre-state retrieval, the engine call, the state transition, the commit.

In the same three days, on the same six machines, Teku's own counters reported that the healthiest node in the fleet was one that had not committed a single block to its canonical chain.

Both of those are worth your attention, and they are the same story. Teku gives you more slot detail than any other client here, and the detail is carried in metric types that will mislead you at least three separate ways if you read them the way their names suggest.

Read this first

• Reth is excluded again, and this time its block-import and engine metrics argued hard for inclusion. Teku recorded 21,546 successful block imports on the Reth pairing, the highest of all six, with zero newPayload errors and the fastest engine responses in the fleet. Its container log shows Reth ran the staged-sync pipeline for the entire window and committed no canonical blocks. Teku's own log says Waiting for execution layer sync on all 21,600 slots. One gauge did dissent, beacon_node_syncing_active, and it sits outside the instrument this post is built from. The section below has the details, because the mechanism generalises well beyond our fleet.
• Neither of Teku's slot metrics is a Prometheus histogram. beacon_block_import_delay_counter_total and beacon_engine_requests_total are counters. The latency lives in a string label called interval holding values like [1000,1500), there is no le label anywhere, and histogram_quantile returns an empty result with an explicit hint. Quantiles have to be computed by hand from the bins.
• Three incompatible bucket schemes coexist across those two metrics, and the only interval string common to all three is [500,1000). Summing across them without filtering by stage inflates the count elevenfold and moves the apparent median by a factor of roughly seventy.
• The unbounded bucket uses a Unicode infinity sign. It is [12000,∞) with U+221E, not [12000,inf). A selector written with inf matches nothing and silently drops the entire slow tail, which reads on a dashboard as an absence of slow blocks.
• Window and versions. Three days, 2026-08-03 to 2026-08-06, on Teku 26.7.1 paired with Geth v1.17.5, Nethermind 1.39.2, Besu 26.7.1, Erigon v3.5.4 and Ethrex 23.0.0, on NDC2 bare metal in Vienna. No client version changed inside the window. Every execution client except Reth was confirmed at the chain tip from its own container log at both ends.

The eleven stages

beacon_block_import_delay_counter_total carries a stage label with eleven values, and every stage fires once per imported block. Here is where a Geth-paired Teku spent its slots over 21,476 imports:

Stage	Modal bucket	Blocks in it
arrival	1,000 to 1,500 ms	7,387 of 21,476
gossip_validation	under 50 ms	21,385
pre-state_retrieved	under 50 ms	21,387
begin_importing	under 50 ms	21,422
transaction_prepared	under 50 ms	21,474
execution_payload_result_received	under 50 ms	18,738
data_availability_checked	under 50 ms	21,441
processed	100 to 250 ms	19,307
transaction_committed	under 50 ms	21,471
completed	under 50 ms	21,328
total_processing_time	under 500 ms	19,954

Two of those eleven are not stages in their own right: arrival is time from slot start until the block reached the node, which is network and gossip rather than processing, and total_processing_time is the sum of the rest. That leaves nine processing stages, and seven of them are close to noise. Five of the seven finish in under 50 milliseconds for better than 99% of blocks on every pairing. The other two miss that bar on the slower execution clients: completed runs from 99.31% under Geth down to 98.31% under Erigon, and begin_importing from 99.75% down to 98.86%. Pooled across the five pairings completed sits at 98.96%, so the 99% line does not hold fleet-wide for it. The shortfall is under a percentage point either way. Two stages carry the entire cost: processed, which sits at 100 to 250 ms for around 90% of blocks, and execution_payload_result_received, which is where the engine call is waited on.

That split is the useful part. processed barely moves across pairings, from 84.1% of blocks in the 100-to-250 ms bucket under Besu to 90.5% under Nethermind. Whatever it covers is work Teku does itself. execution_payload_result_received moves by a factor of nearly five. That is the execution client's contribution, isolated by the consensus client's own stopwatch.

One limit before the numbers: we have not verified from Teku's source what each stage boundary corresponds to internally. The stage names are Teku's, the ordering is Teku's, and the durations sum the way you would expect them to, but this is a relative comparison across pairings on one client's instrument rather than an audited decomposition.

What the engine wait costs, per execution client

Execution client	Result under 50 ms	Result at 1 s or worse	newPayload under 100 ms	newPayload at 1 s or worse
Ethrex 23.0.0	93.6%	1.9%	87.6%	2.6%
Geth v1.17.5	87.3%	2.1%	37.5%	3.0%
Nethermind 1.39.2	74.8%	5.6%	42.2%	7.3%
Besu 26.7.1	56.1%	4.7%	7.6%	6.7%
Erigon v3.5.4	20.4%	9.4%	2.6%	12.3%

The two leftmost columns come from the block-import stage, the two rightmost from beacon_engine_requests_total{method="new_payloadV4"}. They are separate instruments inside the same process, and they agree at both extremes.

This is the fifth consensus client to put Ethrex at the fast end and Erigon at the slow end of newPayload, which is now the most heavily replicated result in this series. The middle of the field permutes again, and this time it permutes within a single instrument depending on which statistic you read. Ranked by the fastest bucket, Nethermind is second and Geth third. Ranked by the share of calls at a second or worse, Geth is second and Nethermind fourth. Nethermind is bimodal: a large fast mode paired with a fat tail. Geth has a slower mode and a thinner tail. Pick either column alone and you get a different middle.

Errors are rare and instructive: 40 failed newPayload calls across the five clients over three days, and every single one of them landed in the [5000,∞) bucket. Erigon 15, Nethermind 11, Besu 10, Ethrex 2, Geth 2. Nothing failed quickly. That pattern is what a fixed client-side timeout looks like, though we did not confirm Teku's timeout value or find an error-reason label, so treat the cause as inferred rather than measured.

Besu's head update, now on a fifth instrument

forkchoiceUpdated again separates Besu from everyone else:

Execution client	forkchoiceUpdated under 100 ms	Modal bucket
Geth	99.88%	under 100 ms
Ethrex	99.80%	under 100 ms
Nethermind	99.74%	under 100 ms
Erigon	96.54%	under 100 ms
Besu	14.84%	100 to 300 ms

Four of five clients answer the head-update call in under 100 milliseconds better than 96% of the time. Besu does so 14.8% of the time, and its modal bucket is 100 to 300 ms, holding 77.5% of its calls.

Teku's fastest bucket is 100 ms wide, too coarse to resolve the peer values the earlier instruments reported for Besu: 4.3 to 7.4 ms under Grandine, 4.3 to 9.9 ms under Prysm, 5.8 to 14.5 ms under Lighthouse. Nimbus is the exception it was in part four, with peers running from 63 to 262 ms and two of them slower than its own Besu. So this instrument cannot reproduce the size of the gap. What it can do is place Besu, and only Besu, in a slower bucket than the rest of the field, which is consistent with the 162-to-200 ms figure the four earlier instruments measured directly. Five instruments have now timed this call and all five put Besu's head update above 100 ms. Four of them also put Besu behind every peer. The magnitude holds on five, the gap on four.

Worth repeating from part three: this cost is not attestation budget. forkchoiceUpdated does not sit between a block arriving and that block becoming attestable.

The counter that called a syncing node healthy

Reth was excluded from parts one through four. Going into this edition its metrics said that was over.

What Teku's metrics said about Reth	Value	Rank of six
Successful block imports	21,546	best
newPayload errors	0	best
Failed payload executions	0	best
Payload results under 50 ms	95.7%	best
Blocks recorded as failed_execution_payload_execution_syncing	0	tied, all six

Four first places and a tie at zero on the one counter whose name promises to catch this. By its consensus client's block-import and engine instrumentation, Reth was the healthiest and fastest execution client on the fleet.

Note the first row against the window's chain: Reth recorded 21,546 imports where the chain produced 21,531 blocks. The counter counts import events rather than canonical blocks, so it can exceed the block count, and on a node that is optimistically importing without validating there is nothing to pull it back down.

Here is Teku's log on that same host, at the same moment, and it wrote a variant of this line every twelve seconds:

2026-08-05 23:59:59.145 INFO - Syncing *** Slot: 14929198, Head slot: 14929197,
Waiting for execution layer sync, Connected peers: 100

That line appeared 21,600 times in 21,600 slots. Every slot in the window, without a single exception, on all three days. On the Geth pairing the corresponding line is Slot Event *** Slot: 14929198, Block: dcba1324..., Justified: 466536, Finalized: 466535, and Waiting for execution layer sync appears zero times.

Reth's own log agrees with Teku's log and disagrees with Teku's metrics:

2026-08-05T23:59:59.829405Z INFO reth_node_events::node: Committed stage progress
pipeline_stages=9/14 stage=MerkleExecute checkpoint=0 target=25588987 stage_progress=40.05%

It was running the staged-sync pipeline, which advanced from stage 4 of 14 at the start of the window to stage 9 of 14 at the end, against a pipeline target roughly 103,000 blocks behind the live chain head of 25,692,170. Canonical chain committed appears zero times in three days. The control that makes this airtight: the identical Reth build paired with Lighthouse, on the same fleet over the same window, logged that line 6,452 times.

The mechanism is optimistic sync, and it is doing what the specification asks of it. When the execution client cannot yet validate a payload, the consensus client may import the block optimistically and carry on. Teku's block-import counter records that as result="success", because from the consensus layer's point of view the import did succeed. The _syncing result value exists but stayed at zero, so it cannot be used as the guard either.

The timing consequence is the one that would have produced a false finding here. Reth answered 95.7% of payload requests in under 50 milliseconds, better than any synced client. An execution client that is not executing returns immediately. The fastest engine responses in the fleet came from the node doing none of the work, and every block-import and engine counter presented that as excellence.

The container log is how we caught it, and it is not the only thing that could have. Teku's beacon_node_syncing_active gauge read 1 on all 4,320 scrapes of the window on the Reth pairing, against at most 5 scrapes of 4,320 on any other pairing, and 0 on Geth and Besu. That gauge sits outside the eleven-stage instrument this whole post is built from, so a dashboard assembled around block import never shows it. What it will not do is name the cause: it also reached 1 on the Ethrex pairing, where Waiting for execution layer sync never appears in the log at all.

So the lesson is narrower than "metrics cannot tell you" and more useful. The gauge tells you the beacon node is off the tip. The log tells you the execution layer is why. Every counter that a slot-timing dashboard would plausibly carry rated a node that committed nothing the best in the fleet, and the one series that dissented is the one no block-import dashboard has on it.

Three ways the buckets will lie to you

The instrument that produced everything above needs handling. All of these are properties of Teku 26.7.1 as deployed, and all of them are silent failures rather than errors.

One: there are three schemes, not one. arrival and total_processing_time use a ten-bucket coarse scheme running to [12000,∞). The other nine stages use a seven-bucket fine scheme running to [2000,∞). beacon_engine_requests_total uses a third, eight-bucket scheme running to [5000,∞). The single interval string common to all three is [500,1000), which makes it useless for telling them apart. There is one further wrinkle: at stage="total_processing_time" the scheme flips to fine when result="unknown_parent", so the scheme is a function of the stage and result pair rather than of either label alone.

Two: summing without a stage filter inflates elevenfold. Write sum by (interval) (beacon_block_import_delay_counter_total{...}) and you pool all eleven stages into one pseudo-histogram whose bins overlap. For the Geth pairing that totals 236,324 observations against 21,476 blocks. Because the fine scheme's [0,50) bin then holds 71.4% of everything, a median read off that chart lands around 25 ms. The true block-arrival median is in the 1,500-to-2,000 ms bucket. The chart is wrong by a factor of about seventy and looks entirely plausible.

Three: the top bucket is spelled with U+221E. It renders as [12000,∞), [2000,∞) and [5000,∞). A selector written interval="[12000,inf)" matches zero series and returns no error. Every slow block silently vanishes from the result, which presents as a clean tail.

One more trap, this one about method rather than about Teku. Establishing that these buckets are disjoint rather than cumulative cannot be done on one client. Reth's newPayload distribution falls monotonically across all eight bins, which is the exact signature a cumulative scheme produces. Tested on Reth alone the answer comes out backwards. It is Besu, Erigon and Geth, whose distributions rise to an interior mode before falling away, that settle it. Ethrex and Nethermind peak in the first bin like Reth, and give themselves away only later, with a step back up that no cumulative scheme can produce.

What to measure on your own Teku nodes

• Alert on beacon_node_syncing_active, then read the logs to attribute the cause. Held at 1 for five minutes, this gauge fired on the Reth host and nowhere else in the window; there it read 1 at every scrape for three days, while every block-import and engine counter Teku publishes looked healthy or better than healthy. The gauge tells you the beacon node is behind, not why. For the why, grep Teku's log for Waiting for execution layer sync, then check that the execution client is committing blocks, which for Reth means Canonical chain committed. Run both checks rather than either one: in this window they disagreed in both directions, with the gauge reaching 1 on the Ethrex pairing where the Teku log line never appeared, and the log line appearing twice on the Besu pairing where the gauge never left 0.
• Always filter by stage before touching interval. Without it the counts are eleven times too large and the bins overlap. Add result="success" too unless you want unknown_parent injecting six foreign bucket strings into a ten-bucket histogram.
• Match the infinity bucket on ∞, not on inf. If a panel shows no blocks in the slowest bucket, check the selector before you believe it.
• Read processed and execution_payload_result_received separately. The first is Teku's own cost and barely moves; the second is the execution client's and moves by nearly five times. Charting only total_processing_time blends them.
• Do not expect quantiles. There is no le label, so histogram_quantile returns nothing. Bucket-bounded statements are the honest ceiling here: a modal bucket or a share above a threshold, not a median in milliseconds.

Coming next in the series

Lodestar is the last client, and on surface area it is the opposite of Teku: 1,184 metric names and 162 genuine histograms against Teku's 410 and 6. Then the finale, where all six clients and their instruments go side by side, with an honest list of the claims from earlier editions that did not survive being measured a second way.

This edition adds one entry to that list, and it is ours. Between drafting and publishing, the Reth pairing was going to be reported as this series' first fully-synced Reth comparison on the strength of five metrics that all said so. The log said otherwise. Across five editions the pattern has held without exception: a client comparison is only as good as the instrument you read it through, and the failure mode is never an error message. It is a plausible number. Catching those is what StereumLabs AI does on our fleet, on the measurement stack we described here. If you run Ethereum infrastructure and want this lens on your own nodes, reach us at stereumlabs.com or contact@stereumlabs.com.

Methodology

Numbers come from Teku's and the execution clients' own metrics on our NDC2 deployment (Vienna), queried on the Prometheus-cold datasource (uid aez9ck4wz05q8e), with the fleet labels documented in build your own dashboards. The window is 2026-08-03T00:00:00Z to 2026-08-06T00:00:00Z. Counts are exact counter deltas, M - M offset 3d evaluated at the closing anchor, rather than increase(...[3d]), which extrapolates and returns fractional values. The two agree to within 0.05%. NDC2 runs a six-by-six matrix of consensus and execution clients on identical 12-core bare-metal hosts, so each execution client appears six times, once per consensus-client pairing; every figure here is the Teku pairing specifically.

• Chain progress. The chain produced 21,531 blocks in the window's 21,600 slots, height 25,670,639 to 25,692,170. Four execution clients agree on both endpoints to within one block via chain_head_block (Geth), ethereum_blockchain_height (Nethermind, Besu) and head_height (Ethrex), the one-block spread being scrape timing rather than disagreement. Erigon publishes no head-height metric name at all; its head is the label value sync{stage="finish"}, which is a further instance of the part-four lesson that capability hides in labels.
• The stage decomposition is beacon_block_import_delay_counter_total{result="success"} grouped by stage and interval. Per-pairing import totals are Besu 21,465, Erigon 21,465, Ethrex 21,056, Geth 21,476, Nethermind 21,487. All eleven stages report identical counts within a pairing. The buckets are disjoint, established four ways: fine-scheme and coarse-scheme stages sum to bit-identical totals; those totals equal roughly one increment per block; a cumulative-from-below scheme would require the lowest bin to be the maximum, which fails for Besu, Geth and Erigon; and a cumulative le-style scheme would require the top bin to be the maximum, which fails everywhere. beacon_block_import_delay_latest also exists, per stage, but it is a gauge of the most recent block only.
• Engine-call figures are beacon_engine_requests_total grouped by method, interval and outcome, one uniform eight-bucket scheme across all five methods and both outcomes. Successful newPayload totals per pairing run from 21,668 to 22,088 against 21,531 blocks, slightly above one per block. The 40 errors quoted are newPayload only and exclude Reth's zero; Reth logged 11 engine errors on other methods, and Geth is the fleet minimum on engine errors overall with 4.
• Metric surface area per consensus client, anchor-evaluated as distinct metric names and _bucket metric names: Lodestar 1,184 / 162, Lighthouse 781 / 144, Prysm 590 / 65, Nimbus 531 / 25, Teku 410 / 6, Grandine 332 / 71. Teku is not the smallest surface; it is the histogram outlier.
• Sync was verified from container logs, not metrics. Geth logged Chain head was updated 21,555 times, from block 25,670,641 to 25,692,172. Nethermind logged Finished pre-warming caches for block 21,549 times, ending at 25,692,172. Besu logged new added block 25692172. Erigon logs a head update roughly every nine minutes rather than per block; its last one in the window reads head updated number=25692010 age=3s, and its Prometheus head stood at 25,692,169. Ethrex ran from block 25,670,642 to 25,692,172. Teku's Waiting for execution layer sync line appeared 0 times for Geth and Ethrex, 2 for Besu, 11 for Nethermind, 65 for Erigon and 21,600 for Reth. Teku's beacon_node_syncing_active gauge, scraped 4,320 times per host, summed to 4,320 on Reth, 5 on Erigon, 4 on Ethrex, 2 on Nethermind and 0 on Besu and Geth.
• Reth's exclusion. Canonical chain committed appears zero times on the Teku-paired Reth host across the window, against 6,452 occurrences on the Lighthouse-paired host running the same build over the same window. Its pipeline advanced from pipeline_stages=4/14 stage=Execution to pipeline_stages=9/14 stage=MerkleExecute with target 25,588,987. We did not confirm from Teku's source that optimistic import is what produces the success result on that pairing; the specification permits it, the log states the node was waiting on the execution layer, and the counter recorded successes, but the causal link is our reading rather than a measurement. Separately, reth_blockchain_tree_canonical_chain_height reads 0 on all six Reth hosts for the first two days of the window, so it is unusable as a sync indicator in either direction.
• The Ethrex shortfall. Ethrex imported 21,056 blocks against a chain that produced 21,531, and its own log shows 21,104 prewarm passes, so both instruments agree the pair observed roughly 450 fewer blocks. The deficit concentrates in three hours (109, 190 and 159 blocks against a normal 300 per hour) with no errors and no restart in the log. Teku's beacon_node_syncing_active on that host reached 1 in two of those three hours and in no other hour of the window, which narrows when the deficit accrued without establishing its cause. Ethrex was at the chain tip at both window ends. Every Ethrex percentage here therefore rests on a denominator about 2% smaller than its peers, which does not change its rank on any measure reported above but is worth knowing.
• Our fleet runs no live validators. It receives mirrored validator-client traffic, unchanged across this window.