9 posts tagged with "Besu"

In the first edition of this series we measured, under Lighthouse, where a 12-second slot goes against the 4-second attestation deadline, and found that the execution client you pick is the lever: it shifts how often a node lands late enough to fail attestations by about 1.5x across the mainstream clients, and 3.7x once Erigon's disk-bound tail is counted. This edition runs the same six execution clients on the same bare-metal fleet, and asks the same question of Nimbus. The answer is the finding: Nimbus cannot tell you which execution client is costing you, because the one timing it reports is block arrival, and arrival is the part the execution client does not touch.

That is not a gap in our data. It is what Nimbus exposes. Where Lighthouse breaks the path to attestable into arrival, consensus verification and execution verification, Nimbus publishes a single histogram of block-arrival delay. The good news is that this histogram is complete, counting every block, not the once-a-minute sample Lighthouse's gauges gave us. The catch is that it sees only the network-and-proposer part of the slot, so the 3.7x spread that mattered under Lighthouse is simply not in the data Nimbus reports.

An Ethereum slot is 12 seconds, and your attestation is due 4 seconds in. By the time your execution client sees a block to verify, most of that 4-second budget is already gone: on our fleet the block arrives, on average, 1.7 to 1.9 seconds into the slot, and that number barely moves whichever execution client you run. What the execution client changes is the slice after arrival. Under Lighthouse, that slice runs from about 100 ms (Ethrex) to 460 ms (Erigon) on a normal block. Across the mainstream clients it shifts how often the node lands late enough that attestations would fail by about 1.5x, and by 3.7x once Erigon's disk-bound tail is in the picture, on identical hardware.

This is the first of a series. We run all six execution clients paired with all six consensus clients on identical bare metal, and each consensus client reports slot timing differently. We start with Lighthouse because it instruments the block-delay breakdown more completely than any other CC on the fleet. Later editions take the same question to Prysm, Teku, Nimbus, Lodestar and Grandine.

How much machine does an Ethereum execution client need once it sits at the tip of mainnet? Our fleet runs every major execution client on its own host, all with 12 cores, five of six client sets with 16 GiB RAM, one consensus client per pairing on a separate machine. Same chain, same engine-API traffic, one telemetry pipeline. Averaged over the same 36 hours, the synced clients hold between 4.9 and 15.6 GiB of host memory, burn about half a CPU core, and write to disk at anywhere from 0.9 to 22 MB/s. The spread is the story: the client you pick decides whether your box idles or works.

Six Besu nodes, same version, same hardware, same config. Five held a flat JVM heap around 1.0 to 1.3 GB. The sixth climbed about 10 GB a day and was on track to be OOM-killed roughly 30 hours after a restart. The one thing different about it was the consensus client on the other side of the engine API.

This is a walkthrough of how StereumLabs AI, reading our fleet's metrics and logs, took that one anomalous node, traced it to a single method, and filed it upstream. Besu shipped a round of mitigations and closed the issue. A later devnet reproduction showed the underlying layers still pile up, the issue was reopened, and the fix that followed is now in review. The bug is operational: recoverable by a restart, no consensus impact, no double-sign, no state-root divergence. It is also the kind of cross-client interaction a single-node test will never surface, because it only appears when a live pairing lands in a specific state.

A Geth node and an Erigon node sit in the same rack, follow the same Ethereum mainnet, take the same engine-API traffic from their paired consensus clients. Today the Geth host's root filesystem reports ~1,688 GB used. The Erigon host reports ~509 GB. That is a 3.3× spread, on the same chain, with no archive mode involved. Both are full nodes, both synced to mainnet head.

Disk is only half of it. The same pruning that bounds those footprints also runs on the engine-API hot path, and on some consensus-client pairings it pushes engine_newPayload past the 4-second sync-committee deadline. This post covers both halves: how big each client gets and whether you can see it pruning, then what that pruning costs in latency once a node is at the tip.

Key findings:

• 3.3× footprint spread on the same chain (root-FS used per host): Erigon ~509 GB, Ethrex ~598 GB, Nethermind ~1,225 GB, Besu ~1,261 GB, Reth ~1,352 GB (not synced), Geth ~1,688 GB.
• Five of six ECs grew 15 to 19 GB in the 7-day window (~2–3 GB/day, normal block ingestion). The cumulative spread comes from retention policy, not the past week's growth.
• Each EC exposes a different pruning surface, and Ethrex exposes none. Erigon's domain pruner fires ~501 k blocks events per host per week; Nethermind's Hybrid pruner runs continuously at ~1,497 nodes/sec/host; Geth path-mode prunes implicitly; Ethrex has no pruning metric at all.
• Reth is still in initial staged sync: its checkpoint sits ~70 to 130 k blocks behind the tip and is not advancing, so it is excluded from the synced comparisons.
• From Nimbus, three of five synced ECs cross the 4-second deadline on P99 engine_newPayload (Nethermind 4.44 s, Geth 4.67 s, Ethrex ≥5.00 s), but the ranking flips by consensus client. It is a pairing property, not a fixed EC property.
• Nethermind's spikes trace to two deliberate v1.37.1 choices. nethermind_pruning_time spikes past 5 s, over the deadline, entirely from the in-memory pruner.
• Sync-committee duties get hit before attestations. A late sync-committee message misses the next block's sync_aggregate outright; a late attestation still has one slot of grace.

Every execution client connects to different peers, sees a different slice of the network, and churns through connections at wildly different rates. We queried Prometheus metrics and Elasticsearch container logs across our entire fleet to find out what each EC actually experiences at the P2P layer, and what it means for Ethereum's network health.

We deployed all 6 Ethereum execution clients from scratch on bare metal and tracked exactly how long each one took to fully sync. The results range from impressive to alarming.

How does each execution client behave when Nimbus asks it to build a block? We monitored 1,000 validator pubkeys across 5 EC pairings for 48 hours and found that block building performance varies dramatically, with one client producing near-empty blocks while the other four packed in millions of gas.

We measured CPU, memory, disk I/O, and network across all 36 CC×EC pairings on our non-supernode fleet — here's what the Fusaka hardfork actually did to hardware consumption.