11 posts tagged with "observability"

Over three days in August, Lodestar timed 21,530 newPayload calls to its Ethrex peer and reported a mean of 372.6 ms; Ethrex timed the same calls from its own end of the same socket and reported 164.8 ms. Both figures are right as far as the client reporting them is concerned, since the two stopwatches start and stop at different points, but the 208 ms between them is larger than most of the differences this series has spent six editions measuring between execution clients.

For six editions we compared execution clients through whichever consensus client happened to be watching them, and it took until the sixth to work out how much of what we were reading belonged to the watcher rather than the watched. Six consensus clients run against the same fleet here, no two of them measure a slot the same way, and one of them cannot produce a mean for the engine call at all.

Lodestar publishes 1,175 metric names and 162 histograms on our fleet, 62% more metric names than any other consensus client we run and twenty-seven times Teku's histogram count. Part five ended by asking whether the richest instrument answers the question in this series' title better than the sparsest one did.

It does, and by a wider margin than we expected. Lodestar decomposes a block's journey into timers that add up: arrival, the wait on the execution layer, import, head. The pieces sum to the whole within 4 milliseconds. One bucket boundary sits at 4.000 seconds, so the share of blocks not processed by the attestation deadline is a count rather than an interpolation. It is 5.1% of blocks on the best execution client here and 9.0% on the worst, while only 0.11 to 0.16% of blocks arrive after 4 seconds at all.

Then the same instrument turned around and told us something about itself. Five execution clients time their own engine-API handlers, and four of them publish a mean we can subtract. Lodestar's wall clock around newPayload runs 181 to 208 milliseconds longer than what the execution client on the other end says it spent. The four other consensus clients that publish a mean sit 6 to 50 milliseconds above their peer. That gap is not a client comparison, it is a measurement about Lodestar, and it took a second stopwatch at the execution client's end to find it.

Teku is the only client in this series that answers the question in the title directly. It splits a block's journey into eleven named stages and counts every one of them, so instead of inferring where the slot goes you can read it off: arrival, gossip validation, pre-state retrieval, the engine call, the state transition, the commit.

In the same three days, on the same six machines, Teku's own counters reported that the healthiest node in the fleet was one that had not committed a single block to its canonical chain.

Both of those are worth your attention, and they are the same story. Teku gives you more slot detail than any other client here, and the detail is carried in metric types that will mislead you at least three separate ways if you read them the way their names suggest.

This edition was nearly published with its central claim inverted. We ran three Prometheus metric-name searches against Grandine, across eleven substrings including payload, forkchoice, engine, execution, latency, delay and duration. The first returned nothing, the second two histograms and the third eleven, and not one of them was the engine-call timing we were looking for. The obvious conclusion was that Grandine is the sparse end of this series: a client that times block arrival and blob fetches and nothing else. A fact-check caught it before publication.

Grandine times both engine calls, per method, at more than one observation per slot. The metric is called ETH1_API_REQUEST_TIMES. It is in capitals, it contains none of the eleven substrings we searched for, and the engine method lives in a label rather than the name, so method="engine_newPayloadV4" sits inside a series whose name says only that some eth1 API was called. No name search reaches it.

Grandine is not the sparse end of this series, and it is not the rich end either: an inventory of its scrape job returns 70 histograms against Lodestar's 158 and Lighthouse's 143, and on total metric names it is the sparsest of the six at 330. The point is not where it ranks. The point is that a comparatively modest client still hid a per-method engine timer from three searches, which means the search was the problem, not the client.

What the metric shows once you find it: Besu's head-update call costs 162 ms against 4 to 7 ms for every other execution client, which is the fourth independent instrument to say so, and the middle of the execution-client field reorders again.

The first two editions of this series each had one instrument. Under Lighthouse we used its block-delay gauges and found the execution client moved the would-fail-attestation rate up to 3.7x. Under Nimbus we used its arrival histogram and found it could not separate execution clients at all. Prysm gives us fine-grained histograms of both engine calls, one entry per slot, which is what we came for. It also gave us something we did not plan on: enough overlap with the other two clients to put the same question to three independent instruments on three separate pairings, in the same three days, on identical execution-client versions.

That triangulation is the finding, and it changes how we read our own earlier editions. Two results survive all three instruments: Ethrex is the fastest execution client on newPayload and Erigon the slowest, and Besu spends around 200 ms on forkchoiceUpdated while it is a single-digit or low-double-digit call for the rest. Almost nothing else does. The middle of the field, Geth against Nethermind against Besu, permutes depending on which client you ask. The reason is not the metric definition, and it is not the hardware either. At a fixed version, the same execution client's own internal timer varies by 160 ms depending on which consensus client is driving it, on machines whose load and database size are indistinguishable. That is larger than the margins we would be ranking on.

In the first edition of this series we measured, under Lighthouse, where a 12-second slot goes against the 4-second attestation deadline, and found that the execution client you pick is the lever: it shifts how often a node lands late enough to fail attestations by about 1.5x across the mainstream clients, and 3.7x once Erigon's disk-bound tail is counted. This edition runs the same six execution clients on the same bare-metal fleet, and asks the same question of Nimbus. The answer is the finding: Nimbus cannot tell you which execution client is costing you, because the one timing it reports is block arrival, and arrival is the part the execution client does not touch.

That is not a gap in our data. It is what Nimbus exposes. Where Lighthouse breaks the path to attestable into arrival, consensus verification and execution verification, Nimbus publishes a single histogram of block-arrival delay. The good news is that this histogram is complete, counting every block, not the once-a-minute sample Lighthouse's gauges gave us. The catch is that it sees only the network-and-proposer part of the slot, so the 3.7x spread that mattered under Lighthouse is simply not in the data Nimbus reports.

An Ethereum slot is 12 seconds, and your attestation is due 4 seconds in. By the time your execution client sees a block to verify, most of that 4-second budget is already gone: on our fleet the block arrives, on average, 1.7 to 1.9 seconds into the slot, and that number barely moves whichever execution client you run. What the execution client changes is the slice after arrival. Under Lighthouse, that slice runs from about 100 ms (Ethrex) to 460 ms (Erigon) on a normal block. Across the mainstream clients it shifts how often the node lands late enough that attestations would fail by about 1.5x, and by 3.7x once Erigon's disk-bound tail is in the picture, on identical hardware.

This is the first of a series. We run all six execution clients paired with all six consensus clients on identical bare metal, and each consensus client reports slot timing differently. We start with Lighthouse because it instruments the block-delay breakdown more completely than any other CC on the fleet. Later editions take the same question to Prysm, Teku, Nimbus, Lodestar and Grandine.

Six Besu nodes, same version, same hardware, same config. Five held a flat JVM heap around 1.0 to 1.3 GB. The sixth climbed about 10 GB a day and was on track to be OOM-killed roughly 30 hours after a restart. The one thing different about it was the consensus client on the other side of the engine API.

This is a walkthrough of how StereumLabs AI, reading our fleet's metrics and logs, took that one anomalous node, traced it to a single method, and filed it upstream. Besu shipped a round of mitigations and closed the issue. A later devnet reproduction showed the underlying layers still pile up, the issue was reopened, and the fix that followed is now in review. The bug is operational: recoverable by a restart, no consensus impact, no double-sign, no state-root divergence. It is also the kind of cross-client interaction a single-node test will never surface, because it only appears when a live pairing lands in a specific state.

A Prysm node and a Lodestar node on the same chain, on identical hardware, both export beacon_reorgs_total. Over the last 90 days, the Prysm hosts in our Vienna NDC2 fleet incremented that counter 6,011 times. The Lodestar hosts incremented it zero times. Both numbers are correct readings of what each implementation chose to count.

This post is a 90-day reorg census across that fleet plus the smaller GCP comparator cohort: every consensus client (CC) paired with every execution client (EC), against the same Ethereum mainnet, with per-host normalization. The questions we answer: which Prometheus counter to trust for which question, why the same EC behind two different CCs produces very different reorg numbers, and why a "zero reorgs" reading on some clients is silence rather than safety.

Key findings at a glance:

• Prysm increments beacon_reorgs_total 8× more often than Lighthouse over 7 days. The gap shrinks to 1.6× over 90 days.
• Lodestar's beacon_reorgs_total is 0 for the entire 90-day window. Its decline-reason counter fires roughly 54 times per host per week.
• The same EC behind two CCs produces 2–5× different counts: Prysm + Besu reports 69 per host vs Prysm + Nethermind 280.
• Geth is the only EC in our fleet whose Prometheus reorg counter increments at all. Nethermind and Reth export the metric but it never increments; Besu, Erigon, and Ethrex don't export one at all.
• A single fixed beacon_reorgs_total alert threshold does not port between consensus clients. Re-baseline per CC × EC pair.

Ethereum runs on two layers: an execution client (EC) handles the EVM, transactions, and world state, and a consensus client (CC) handles Proof-of-Stake fork choice and validator duties. Six EC implementations and seven CC implementations exist in production today, paired in dozens of combinations across the network. StereumLabs runs all of them, side by side, on identical hardware in our Vienna data center, and publishes the numbers.

This post is the technical introduction to that platform: the bare-metal fleet, the metrics and logs pipeline, the label conventions that make the pairings comparable, and the in-house AI workflow that turns the resulting telemetry into the blog posts you are reading.

RockLogic publishes a separate case study on the business side of this workflow: how the same "AI on own data" pattern keeps customer telemetry inside the perimeter while still producing useful answers. This post is the technical view from the other side of the same workflow.