Where the slot goes: Prysm, and what survives three instruments

The first two editions of this series each had one instrument. Under Lighthouse we used its block-delay gauges and found the execution client moved the would-fail-attestation rate up to 3.7x. Under Nimbus we used its arrival histogram and found it could not separate execution clients at all. Prysm gives us fine-grained histograms of both engine calls, one entry per slot, which is what we came for. It also gave us something we did not plan on: enough overlap with the other two clients to put the same question to three independent instruments on three separate pairings, in the same three days, on identical execution-client versions.

That triangulation is the finding, and it changes how we read our own earlier editions. Two results survive all three instruments: Ethrex is the fastest execution client on newPayload and Erigon the slowest, and Besu spends around 200 ms on forkchoiceUpdated while it is a single-digit or low-double-digit call for the rest. Almost nothing else does. The middle of the field, Geth against Nethermind against Besu, permutes depending on which client you ask. The reason is not the metric definition, and it is not the hardware either. At a fixed version, the same execution client's own internal timer varies by 160 ms depending on which consensus client is driving it, on machines whose load and database size are indistinguishable. That is larger than the margins we would be ranking on.

Read this first

• All three consensus clients time the engine API. Prysm publishes new_payload_v1_latency_milliseconds and forkchoice_updated_v1_latency_milliseconds. Lighthouse publishes execution_layer_request_times{method="new_payload"|"forkchoice_updated"}, and Nimbus publishes engine_api_request_duration_seconds{request="newPayload"|"forkchoiceUpdated"}, all as complete histograms at about one entry per slot. That is more overlap than we assumed when this series started, and the first two editions read differently as a result. They stand as published; we will put every client's instrumentation side by side, and say plainly what we got wrong along the way, in the finale where that comparison belongs.
• No consensus client can see an execution client's internal work. The engine API returns no timing, so every consensus-side number, including Lighthouse's beacon_block_delay_execution_time, is a wall clock around the client's own engine call. Where clients differ is how much of the call they bracket, which is worth 13 to 76 ms on our fleet, not the hundreds of milliseconds that separate host from host.
• Cross-pairing rankings inside about 100 ms are not reliable, including ours. At a fixed execution-client version, nethermind_new_payload_execution_time reads 195 ms under one consensus client and 355 ms under another. That is not a hardware story: the six machines sit in the same rack with sub-millisecond ping, run 5 to 7% CPU, 5 to 7% disk utilisation and databases within 0.1% of the same size. Comparing two execution clients that were driven by different consensus clients inherits that 160 ms spread, which is why we report the extremes and Besu's forkchoiceUpdated and decline to rank the middle.
• Reth is excluded, for the third edition running. Three independent signals agree: Prysm logged zero VALID payload responses from it against 40,556 optimistic ones, its own reth_sync_checkpoint puts the Execution stage at block 16,340,084 while the chain was past 25.6M, and its container log shows no canonical block commit in the window. It was mid staged-sync, so its fast-looking latency is a SYNCING artifact. See the sync-speed census.
• Coverage is complete, tail figures are interpolated. Prysm recorded about 21,565 newPayload and 21,562 forkchoiceUpdated calls per pairing across 21,600 slots. Means and counts are exact. The newPayload buckets stop at 4,000 ms, so 99th percentiles are bucket-interpolated and approximate.
• One metric that looks like the outcome is not. attestation_inclusion_delay_slots returned 3.59 slots for all five pairings, identical to three decimals, because it measures the inclusion delay of attestations in the blocks the node processes, a network-wide property. It cannot separate execution clients.
• Window and versions. Three days, 2026-07-24 to 2026-07-27, Prysm v7.1.7 (Lighthouse v8.2.1 and Nimbus v26.7.0 for the cross-checks) paired with Geth v1.17.4, Nethermind 1.39.2, Besu 26.7.0, Erigon v3.5.2 and Ethrex 22.0.0, on NDC2 bare metal in Vienna. Every execution client was confirmed at the chain tip from its own container logs at both ends of the window.

What Prysm measures

Prysm times both engine calls at one entry per slot, with newPayload buckets running from 25 ms to 4,000 ms, which brackets the range these calls live in. Over the window:

Execution client	Mean	90th pct	99th pct*
Ethrex 22.0.0	171 ms	407 ms	~1.94 s
Geth v1.17.4	326 ms	822 ms	~2.30 s
Nethermind 1.39.2	373 ms	963 ms	~3.16 s
Besu 26.7.0	442 ms	937 ms	~3.04 s
Erigon v3.5.2	516 ms	1,136 ms	~3.30 s

*bucket-interpolated, since the top finite bucket is 4,000 ms.

On an average block even the slowest of these fits inside the budget: Erigon's 516 ms is about 13% of the 4-second attestation deadline against Ethrex's 4%. The tail is what bites, and at the 99th percentile every client is into the seconds, the same mechanism part one found. Stack a 3-second tail onto a 1.8-second arrival and the slot is gone.

Read on its own, that table looks like a ranking. The rest of this post is about why only its first and last rows are one.

Three instruments, one question

Because all three clients publish a complete per-slot histogram of the same engine call, and because our fleet runs each consensus client against every execution client, the same question can be asked three times over the same three days on identical execution-client versions. The only thing that changes is which consensus client is holding the stopwatch and driving the execution client.

Execution client	Lighthouse	Nimbus	Prysm
Ethrex	139 ms	225 ms	171 ms
Nethermind	275 ms	359 ms	373 ms
Geth	311 ms	338 ms	326 ms
Besu	403 ms	343 ms	442 ms
Erigon	503 ms	535 ms	516 ms

Ethrex is first in all three columns and Erigon last in all three. Those two results are worth trusting: three instruments, three host sets, about 21,500 observations each.

Everything between them moves. Lighthouse orders the middle Nethermind, Geth, Besu. Nimbus orders it Geth, Besu, Nethermind. Prysm orders it Geth, Nethermind, Besu. All three permutations appear, and no two agree.

Why the middle does not rank

It would be convenient to blame the metric definitions, and we nearly did. It does not hold up, for two reasons we can measure.

The definitional effect is small and we can size it without leaving one host. Lighthouse publishes both quantities itself, on the same machine, in the same window: the gauge that brackets its execution step and the histogram of the full engine round trip. Going from one to the other costs 13 ms for Besu, 22 ms for Geth, 34 ms for Nethermind, 42 ms for Ethrex and 76 ms for Erigon. It reorders nothing: Nethermind stays 36 ms ahead of Geth on both of Lighthouse's own rulers, on all three days separately.

What does move is the pairing, and the execution clients measure it themselves. Nethermind 1.39.2 publishes its own internal payload-execution time, a number no consensus client is in the loop for. Across our six pairings, same version, same window:

Driven by	Nethermind's own newPayload time
Grandine	195 ms
Teku	239 ms
Lighthouse	263 ms
Nimbus	311 ms
Lodestar	341 ms
Prysm	355 ms

One client, one version, one metric, no consensus client in the measurement, spanning 160 ms, or 80% of the fastest value.

The obvious suspect is the machine, and we checked it, because on this fleet it is checkable. The six hosts are identical 12-core bare metal in the same rack with sub-millisecond ping between them. Over this window they ran 5.2 to 6.7% CPU and 5.1 to 7.3% disk utilisation, so neither resource was anywhere near saturation on any of them. They read 1.8 to 2.1 MB/s from disk, held 8.9 to 9.9 GiB of page cache, and their databases are within 0.1% of the same size, 1.1506 against 1.1516 TiB. Nothing in that spread of load explains an 80% spread in execution time, and the ordering does not follow it: the Lighthouse-paired host is the second least busy and the third slowest.

So the number tracks which consensus client is asking, not the box it runs on. What we cannot do from timing data alone is say why. The candidate we would test next is handover timing against speculative execution: these clients pre-warm state from the mempool before the payload arrives, so a consensus client that hands the block over at a slightly different moment, or that pipelines newPayload differently against its blob and forkchoice traffic, leaves a different amount of that work already done. That is a hypothesis, not a result, and it is the kind of thing the next editions can test now that we know to look.

For anyone reading a client comparison, including this one, the practical consequence does not depend on the mechanism. Two execution clients are only comparable when the same consensus client drove both of them. That is why the table in the previous section is a fair ranking under Prysm and not a fair ranking full stop, and why differences of the size of Ethrex against Erigon, a factor of three, survive changing the driver while differences of 10 or 15% do not.

The engine call almost nobody looks at

Every slot, a consensus client makes at least two engine calls: newPayload to hand over the block and forkchoiceUpdated to name the head. Dashboards, benchmarks and the first two editions of this series all watch the first one. The second is where the one client-specific anomaly on our fleet lives, and unlike the middle-of-field ordering, it survives triangulation:

Execution client	Lighthouse	Nimbus	Prysm
Besu	197 ms	193 ms	208 ms
Erigon	14 ms	63 ms	6 ms
Geth	11 ms	69 ms	10 ms
Ethrex	8 ms	261 ms	7 ms
Nethermind	6 ms	255 ms	4 ms

Besu's row is the flat one: 197, 193, 208 ms, on three instruments and three pairings, at about 21,500 calls each. Under Prysm it is roughly 30x the next client and 443 ms at the 90th percentile against their 23 ms, and the shape of the distribution carries it rather than a few outliers: only 14% of Besu's calls come in under 100 ms, 58% under 200 ms, and just 0.5% exceed a second.

The other four rows are the cautionary tale in miniature. Nethermind's forkchoiceUpdated reads 6 ms under Lighthouse, 4 ms under Prysm and 255 ms under Nimbus. We are not going to pretend to explain that from timing data alone; the call counts are about one per slot under all three, so it is not a difference in how often the call is made. It is a reminder that a number this instrument-dependent is not a client property, and that Besu's stability across the same three instruments is what makes its 200 ms worth reporting.

Why Besu, we can bound but not settle. The same Besu is the one that in our pruning census does storage work on the engine-API hot path, and forkchoiceUpdated is where a client updates its canonical head, so head-update bookkeeping touching storage is the plausible mechanism. We have not lined up individual slow calls against Besu's storage activity, so this is a measured cost without a confirmed cause.

What the 200 ms does and does not cost

It is tempting to add the two calls up and crown a winner. We did, in a draft of this post, and the fleet's own data talked us out of it. Summing the means is arithmetically fine, since both histograms carry the same one-per-slot denominator: Besu spends about 650 ms per slot across the two calls Prysm times against Erigon's 522 ms, so Besu has the larger engine-API bill.

But the bill is not the deadline. Prysm publishes chain_service_processing_milliseconds, its own total block-processing time, at the same one observation per block:

Execution client	Block processing, mean	Above its own newPayload
Ethrex	265 ms	+94 ms
Geth	415 ms	+89 ms
Nethermind	445 ms	+72 ms
Besu	512 ms	+70 ms
Erigon	573 ms	+57 ms

Every client sits 57 to 94 ms above its own newPayload mean, Besu included at +70 ms. The 208 ms head-update call is therefore measurably outside the region Prysm spends importing a block, which is both why it costs no attestation budget and why it does not make Besu the worst client for duty timing. On the quantity that governs when a block becomes attestable, the order does not reorder at all: Erigon is still last. Besu has the largest engine-API bill and the fourth-longest wait to attestability, and both statements are worth knowing.

One caveat on the off-path finding: our fleet does not propose blocks. A proposing node calls forkchoiceUpdated with payload attributes to start block assembly, and there a slow head update would sit on a path that matters.

Arrival, execution-client-independent for the third time

Execution client	Mean block arrival
Geth	1,833 ms
Besu	1,836 ms
Nethermind	1,840 ms
Ethrex	1,844 ms
Erigon	1,850 ms

Seventeen milliseconds separate fastest from slowest, on clients whose newPayload latencies span 345 ms. Arrival happens before the execution client is asked for anything, so it cannot carry the execution client's signature, and across three editions it never has. If your blocks arrive late, that is peering and proposers, and our peering deep dive is where to take it.

What to measure on your own nodes

• Watch both engine calls. Whichever client you run, it publishes both: new_payload_v1_latency_milliseconds and forkchoice_updated_v1_latency_milliseconds on Prysm, execution_layer_request_times{method=...} on Lighthouse, engine_api_request_duration_seconds{request=...} on Nimbus. Watching only newPayload hides a call that on one client here costs 200 ms.
• Watch the tail, not the mean. Every client in this comparison is comfortable on an average block and into the seconds at the 99th percentile.
• Compare execution clients under one consensus client, or not at all. A 160 ms spread at fixed version across our own pairings is the reason we will not rank the middle of the field. If you are choosing between two clients whose published numbers are within 10 or 15%, ask what consensus client produced them, and rerun the comparison on your own stack before acting on it.
• Separate the bill from the deadline. Engine-API wall time and time-to-attestable are different quantities that rank clients differently. Pair the engine histograms with your client's block-processing metric, and with attestation inclusion distance from rated.network or beaconcha.in as the outcome.
• Do not read attestation_inclusion_delay_slots as your own duty performance. It came out identical across all five pairings because it describes the network, not this node.

Coming next in the series

Three consensus clients down, and the useful surprise is that they overlap more than we assumed: all three time the engine API completely, which turns the series from a tour of instruments into a way to cross-check results. Teku, Lodestar and Grandine are next, and all three already appear as drivers in the table above. What we will be asking of them is not only what they instrument, but which of our findings survive being measured somewhere else.

Then the finale, which is the one we are now looking forward to: six consensus clients, six execution clients, every instrument against every other, and an honest accounting of which claims from the earlier editions of this series hold up under that comparison and which do not. This edition already found two that need revisiting. We would rather publish that list than quietly not mention it.

Holding hardware and consensus client fixed while varying the execution client is the cut that makes execution cost legible, and cross-checking the same question against a second and third instrument is what tells you which parts of it were about the client at all. That work, including catching a node that only looks synced and catching our own overreach, is what StereumLabs AI does on our fleet, on the measurement stack we described here. The same data supports doing it yourself on matched versions and identical hardware. If you run Ethereum infrastructure and want this lens on your own nodes, reach us at stereumlabs.com or contact@stereumlabs.com.

Methodology

Numbers come from the three consensus clients' engine-API and block-timing histograms on our NDC2 deployment (Vienna), queried on the Prometheus-cold datasource (uid aez9ck4wz05q8e), with the fleet labels documented in build your own dashboards. The window is the three days from 2026-07-24T00:00:00Z to 2026-07-27T00:00:00Z, with increase(...[3d]) evaluated at the closing anchor so figures are stable and reproducible rather than drifting with query time. The window sits inside a period when no execution-client version changed on these pairings; the last bump was on 2026-07-22.

• Prysm's metrics. new_payload_v1_latency_milliseconds and forkchoice_updated_v1_latency_milliseconds, histograms in milliseconds with finite buckets at 25, 50, 100, 200, 500, 1000, 2000 and 4000 ms; block_arrival_latency_milliseconds with finite buckets at 100 to 24000 ms; chain_service_processing_milliseconds as a summary. Values are for cc_client="prysm-super", cc_version="v7.1.7", deployment="NDC2" grouped by ec_client. Means are increase(_sum[3d]) / increase(_count[3d]), which is exact. Percentiles are histogram_quantile over windowed buckets and are bucket-interpolated; the 99th percentiles land in the 2000-to-4000 ms band and are approximate.
• The three-instrument comparison uses execution_layer_request_times{method="new_payload"} on cc_client="lighthouse-super" (v8.2.1), engine_api_request_duration_seconds{request="newPayload"} on cc_client="nimbus-super" (v26.7.0), and Prysm's new_payload_v1_latency_milliseconds, all in the same window for the same execution-client versions. The Lighthouse and Nimbus metrics are in seconds and converted. Coverage is comparable: increase(_count[3d]) gives about 21,571 (Lighthouse), 21,577 (Nimbus) and 21,565 (Prysm) per pairing against 21,600 slots. The same three metrics with the forkchoice_updated, forkchoiceUpdated and forkchoice_updated_v1 selectors give the second table, at about 22,266, 21,577 and 21,562 calls respectively.
• The definitional effect is measured within Lighthouse on one host: avg_over_time(beacon_block_delay_execution_time[3d]) against the mean of execution_layer_request_times{method="new_payload"}, giving Besu 390 to 403 ms, Geth 289 to 311, Nethermind 241 to 275, Ethrex 97 to 139 and Erigon 427 to 503. No consensus client can observe an execution client's internal work, since the engine API returns no timing, so both of these are consensus-side wall clocks that differ in how much of the call they bracket. An earlier draft of this post described the Lighthouse gauge as an internal execution timer and attributed the Geth-Nethermind ordering difference to that distinction. Both were wrong, and the two paragraphs below are why.
• Pairing variance is measured with the execution clients' own timers, which no consensus client is in the loop for. avg_over_time(nethermind_new_payload_execution_time[3d]) for ec_version="1.39.2" across the six pairings gives 195, 239, 263, 311, 341 and 355 ms, for the Grandine, Teku, Lighthouse, Nimbus, Lodestar and Prysm pairings respectively, on 4,320 samples each. The Lighthouse-to-Prysm difference of 92 ms is the dominant term in what an earlier draft of this post read as a measurement-definition effect.
• Ruling out the machine. Over the same window, on the same six execution-client hosts, node_cpu_seconds_total gives 5.2 to 6.7% busy, node_disk_io_time_seconds_total gives 5.1 to 7.3% utilisation, node_disk_read_bytes_total gives 1.83 to 2.13 MB/s, node_memory_Cached_bytes gives 8.9 to 9.9 GiB, and used filesystem bytes span 1.1506 to 1.1516 TiB, a 0.1% difference. All six are identical 12-core bare metal in one rack with sub-millisecond ping. Neither CPU nor disk was near saturation on any host, the load ordering does not match the latency ordering, and the database sizes are equal, so we attribute the spread to the pairing rather than to the hardware and leave the mechanism open.
• On resolvability. Within a single instrument the differences we report are resolvable: Lighthouse's gauge holds 4,320 samples per pairing over the window with standard deviations near 494 ms (Geth) and 352 ms (Nethermind), and Prysm's histogram is a complete per-slot census. Those samples are 60 seconds apart and serially correlated, so a naive independent-samples standard error understates the uncertainty. We do not rest anything on it: the cross-instrument comparisons in this post are limited by between-pairing variance of 92 to 160 ms, which is larger than the effects a significance test would be resolving, so we report the extremes and the reproducible Besu result and decline the middle.
• Besu's forkchoiceUpdated distribution under Prysm, from the raw buckets: about 14% of calls under 100 ms, 58% under 200 ms, 97.6% under 500 ms and 0.5% over 1 s, with an interpolated median near 183 ms. The four other clients' p90 values of 22.8 to 23.2 ms are first-bucket interpolation artifacts, since 97 to 99% of their calls land under the 25 ms bucket edge, and should not be read as those clients agreeing with each other.
• The engine-API bill against the deadline. chain_service_processing_milliseconds on Prysm, one observation per block like the two engine histograms, gives Ethrex 265, Geth 415, Nethermind 445, Besu 512 and Erigon 573 ms, so each client sits 57 to 94 ms above its own newPayload mean and the 208 ms forkchoiceUpdated is outside that region. Summing the two engine means is valid as an expected per-slot total because both denominators are the same, but it covers only the two calls these clients time per block, not all engine traffic. Our fleet does not propose blocks, so the payload-attribute variant of forkchoiceUpdated is not exercised here.
• attestation_inclusion_delay_slots returned 3.586 slots for all five synced pairings, identical to three decimals, the signature of a network-wide quantity rather than a per-node one. Our fleet runs no live validators, so no metric here depends on attestation signing.
• Sync was verified from logs, not metrics. At both ends of the window each execution client's own container log showed it at the chain tip: Geth's Chain head was updated number 25,598,917 and 25,620,466, Erigon's head updated, Besu's block-add lines, Nethermind's Received ForkChoice: 25598917, and Ethrex's Prewarm pass for block 25598918. Reth is excluded on three independent grounds: zero VALID payload responses against 40,556 optimistic ones on this pairing, reth_sync_checkpoint{stage="Execution"} at 16,340,084 against a chain past 25.6M, and no Canonical chain committed line in the three-day window. Its Received new payload lines do reach tip block numbers, which is the trap: receiving a payload is not importing it. On identical 12-core hosts; consensus and validator processes run on separate machines.