Ethereum runs on two layers: an execution client (EC) handles the EVM, transactions, and world state, and a consensus client (CC) handles Proof-of-Stake fork choice and validator duties. Six EC implementations and seven CC implementations exist in production today, paired in dozens of combinations across the network. StereumLabs runs all of them, side by side, on identical hardware in our Vienna data center, and publishes the numbers.
\nThis post is the technical introduction to that platform: the bare-metal fleet, the metrics and logs pipeline, the label conventions that make the pairings comparable, and the in-house AI workflow that turns the resulting telemetry into the blog posts you are reading.
\nRockLogic publishes a separate case study on the business side of this workflow: how the same \"AI on own data\" pattern keeps customer telemetry inside the perimeter while still producing useful answers. This post is the technical view from the other side of the same workflow.
\nEvery Ethereum client team publishes their own benchmarks. Every cloud provider publishes their own reference architectures. Every node operator has a folder of opinions. What is rarer is a fleet where all six execution clients and all seven consensus clients run on identical hardware, identical scenarios, identical scrape cadence, and identical label conventions, with the raw data kept long enough to ask new questions about old runs.
\nStereumLabs exists to fill that gap. The technical choices below are not the only ones that could work, but they are the ones that keep comparability cheap, reproducibility verifiable, and post-hoc questions answerable without re-running anything.
\nThe bare-metal anchor lives in our Vienna NDC2 data center, run end-to-end by RockLogic. Bare metal cuts out three moving parts that contaminate cloud measurements: instance families that change underneath you, noisy neighbours that blur tail latencies, and IOPS guarantees that come with footnotes. For a measurement product, all three are problems we prefer not to have.
\n| Component | Detail | Why it matters |
|---|---|---|
| CPU | AMD EPYC 9654P (96 cores, single socket) | 96 cores let us isolate one client per VM with substantial headroom, so a busy client cannot crowd out its co-tenants |
| Memory | ECC DDR5 | Bit-flip detection is a measurement-quality concern, not a paranoia tax. Silent corruption skews log-derived percentiles |
| Storage | ZFS raidz1 on Solidigm D7-P5520 NVMe | Predictable IOPS, durable through single-disk failure, ARC behavior that we can describe rather than guess at |
| Network | 2x Broadcom P225P (each dual-port 25 GbE) | Four 25 GbE ports per host. Enough headroom to separate measurement traffic from client P2P without crosstalk |
| Hypervisor | Proxmox VE 9.1 | Open source, scriptable, lets us pin CPU sets per VM. Snapshots make scenario rollbacks cheap |
| OS in guests | Ubuntu 22.04 LTS | Boring, well-understood, matches what most operators run |
| Time sync | NTP | Clocks synchronised across the fleet so head updated timestamps are comparable across hosts |
The bare-metal fleet is paired with a smaller cloud footprint in GCP. The cloud cohort exists for one specific reason: to surface, at indicator level, how much of what we measure is the client and how much is the environment. Without any cloud comparator, every measurement we publish is implicitly \"what Erigon does on a bare-metal NVMe box in Vienna\". With one, we can flag when a result changes meaningfully across environments. The GCP cohort is small and currently focused on Geth and a subset of pairings. It is sized to catch direction-of-effect differences, not to characterise the full performance surface of any commercial cloud. Where a finding turns on the comparator (such as the inbound-firewall observation discussed later in this post), we say so explicitly.
\nA typical EC plus CC pairing occupies two VMs: one for the execution client, one for the consensus client. Splitting the roles across two hosts is deliberate. It means node_exporter on each VM gives us clean per-layer resource consumption, with no need to attribute CPU or RAM to one or the other after the fact.
\nThe math is straightforward: six EC families times six CC families is 36 pairings, plus the standalone Caplin host that runs both layers in one binary, plus the GCP cohort that mirrors a subset of those pairings. The full fleet sits at roughly 90 hosts across both deployments. The 36 pairing matrix below is the bare-metal NDC2 core; the GCP comparator widens the picture but does not change its shape.
\n![\"Fleet](\"https://docs.stereumlabs.com/assets/images/stereumlabs-fleet-matrix-317b3288b9ca6c5122bf381226790fbc.svg\")
The pipeline has three rails: metrics, logs, and dashboards. Each rail is boring on its own. The interesting part is how they are wired together so that a question asked today can be answered against data captured weeks ago.
\nEvery EC and CC process exposes a /metrics endpoint in Prometheus text format. Every host runs node_exporter for OS-level counters. Scrape cadence is 15 seconds, which is a deliberate compromise: short enough to catch most state transitions, long enough that cardinality stays manageable.
We run two Prometheus tiers:
\nprometheus-free: short retention, mirrored subset of metrics, available to Free-tier subscribers with a 7-day data delay.prometheus-cold: full metric set, no retention cap. This is where we run avg_over_time(...[7d:1h]) queries when we write a blog post.Cold storage is what makes most of the StereumLabs blog series possible. The Caplin standalone analysis from last week pulled metrics that were captured before Caplin v3.3.10 had been added to the fleet, and ran the same queries against the new host. No re-runs, no warm-up periods, no \"we'll publish in two weeks once we have data\".
\nLogs travel a parallel path on the same diagram above: Filebeat streams logs from every container into Elasticsearch, where Kibana and the MCP server (Model Context Protocol, the read-only query gateway our AI workflow uses; covered later in this post) consume them in the same way Grafana consumes Prometheus. Logs are the second half of what we measure, and frequently the half that surfaces things metrics cannot. The classic example: the per-block head updated line in Erigon contains four fields (execution time, commit time, age, mgas/s) that no /metrics endpoint exposes. Without log capture, the Caplin standalone vs classic comparison would have been a far shallower piece.
Logs land in Elasticsearch with structured fields for container_name, host, ec_client, cc_client, and the rest of the StereumLabs label vocabulary. Fleet-wide ingest currently runs around 30,000 log lines per second with debug logging enabled across every container.
A side effect of capturing everything is that we can write log-derived percentile tables that ask questions the original log format was not designed for. We do not need to instrument the client. We just need it to log enough.
\nGrafana sits on top of both Prometheus and Elasticsearch. Dashboards are the public face of the platform: panels with info icons that link back to the definitions page, time-aligned charts for cross-client comparison, lifecycle badges (active, experimental, legacy) so you know whether to cite a result. Plan-tier-gated access controls how much retention and how many users you get.
The naming convention is Category – Metric (Scope), units are SI, and per-core normalization is always labeled. The reason we are strict about it: the only way cross-client comparisons stay legible is if a panel titled \"CPU – Utilization (EC process)\" means the same thing on every dashboard regardless of which client it is plotting.
Every metric series in prometheus-cold carries a fixed set of StereumLabs labels:
ec_client: execution client name (besu, erigon, ethrex, geth, nethermind, reth).ec_version: pinned version string.cc_client: consensus client name (grandine, lighthouse, lodestar, nimbus, prysm, teku, plus caplin when running standalone).cc_version: pinned version string.role: ec or cc, so a query can target the layer regardless of which process the metric came from.deployment: provider code (NDC2 for bare-metal Vienna, GCP for cloud).location: provider-specific region or zone identifier.The result is that a query like
\navg_over_time(
node_cpu_seconds_total{role=\"ec\", ec_client=\"erigon\", deployment=\"NDC2\"}[7d:1h]
)
returns exactly the slice you would ask a human for in plain English: \"the 7-day Erigon EC CPU average on bare metal\". No prior knowledge of the underlying scrape topology needed.
\nMost of the unglamorous engineering effort goes into keeping this label vocabulary consistent. It is also what made the EC P2P peering deep dive feasible: that post compared roughly 90 hosts across six EC families and two deployment classes in one chart. Without consistent labels, the same comparison would have required a custom ETL job for each section.
\nNeutral measurement is a methodological commitment, not a marketing claim. The concrete rules we enforce inside the fleet:
\nA run manifest is the concrete artifact that carries all of this. It records the pinned client versions, the host specs (vCPU, RAM, disk, hypervisor), the start and end of the measurement window, any deviations from defaults with their justifications, and the acceptance checks that gate publication. Together with the panel-definition links from the dashboard each chart was drawn against, the manifest is what makes a published result re-runnable later. Enterprise subscribers get the full manifest attached to every dashboard view; public posts cite the relevant subset in their methodology notes.
\nThe hard part of this work is rarely the infrastructure itself. Most of the effort goes into the rules that keep runs comparable: label vocabulary, version pinning, deviation logging, scenario manifests, deciding what counts as \"enough\" data. The RockLogic AI case study makes a parallel observation: hooking a model up to the data was an afternoon; defining what a useful answer looks like took weeks. Spinning up Prometheus and Grafana is well-documented and quick. Turning that into a substrate where a researcher in 2027 can re-run today's query against the same definitions and get a comparable result is the ongoing commitment, and it is what subscribers are paying for.
\nStereumLabs is not the only public effort to characterize Ethereum client behavior, and not the first. A non-exhaustive list of work that we read, learn from, and consider complementary:
\nethereum-package), the ethereum-metrics-exporter, checkpointz, mainnet monitoring tooling, and related infrastructure. Their toolset covers a much wider operator surface than ours; our addition is the controlled cross-client comparison on identical hardware.Our niche is the cross-product: every EC and CC implementation, paired with every other, on identical hardware under identical scenarios, with the raw telemetry retained long enough to ask new questions of old runs. Where another effort goes wider or deeper on one axis, we cite it; where our data complements theirs, we say so.
\nWhat matters is what the platform makes visible. A non-exhaustive recap of recent output:
\nNone of these required new instrumentation. They required asking new questions of data that was already in prometheus-cold and Elasticsearch. That is the payoff of the upfront effort to keep everything labelled, retained, and reproducible.
This is where the post connects back to the RockLogic case study.
\nEvery StereumLabs blog post you read carries stereumlabs-ai as a co-author. The label is literal. Anthropic Claude has read-only MCP access to our Prometheus and Elasticsearch instances, drives the queries, reads the results, drafts the prose, and proposes the tables. A human editor reviews, corrects, and approves before publication. The case study describes that workflow from a business angle: data stays in-house, output is cited and reproducible, weeks of analysis collapse into hours. This post is the technical view of the same loop from the other side.
The wiring looks like this:
\nA few properties of that loop that are worth being explicit about:
\navg_over_time query returns one number, not a histogram. A head updated log extraction returns a summary table, not 50,000 lines. The model is given the smallest input that answers the question. The trimming is a structural mitigation, not a guarantee of correctness; it makes hallucinated numbers much harder to slip past a citation check, but it does not eliminate them.The same pattern is what we offer to customers in the RockLogic case study. We run it on ourselves first, on our own infrastructure with our own data, before pointing it at anyone else's.
\nFive observations from running this platform. The first three are operational costs we accept. The last two are biases worth flagging so that readers can weigh them against their own deployment.
\nMetric heterogeneity across clients is the dominant operational cost. Caplin v3.3.10 ships a comprehensive libp2p_rcmgr_* family but no current libp2p peers gauge. Prysm exposes connected_libp2p_peers{agent=\"...\"} with per-implementation breakdowns. Lighthouse, Lodestar, Nimbus, Teku, and Grandine each use different metric names for what is conceptually the same thing. Every cross-client query is implicitly a translation layer. We maintain that layer as part of the dashboard panel definitions and as part of the AI workflow's instruction set.
Log format chaos is the second cost. Some clients log structured JSON, some log key=value, some log unstructured prose, some log all three depending on subsystem. Stripping ANSI color codes is a normal preprocessing step. Writing a parser like the one for Erigon's head updated line is the easy part; validating it against edge cases is what takes time.
Keeping runs comparable is the third cost, and the largest. The same conclusion the AI case study reaches about its own workflow: the hard part is not the technology but the editorial spine. For us, that means resisting the temptation to publish numbers from runs that were \"almost equivalent\". Almost is where bias lives.
\nSingle-site bias. All bare-metal hosts sit in one Vienna data center (NDC2) and share an upstream BGP path and ASN-level peer reachability. That means peer-set composition, geographic latency to other mainnet nodes, and inbound discoverability are correlated across the fleet. Findings about peer count, peer diversity, or P2P churn carry an implicit \"as seen from one network position\". We add the GCP cohort to break that correlation in one direction, but two environments are an indicator, not a characterisation. Reproducing a finding from a different site or ASN is one good way to test how site-bound it is.
\nScrape cadence has a tail-latency floor. 15-second Prometheus scrapes are sufficient for resource averages and steady-state behaviour, but they are too coarse for tail-latency analysis. GC pauses, individual slot timings (Ethereum slots are 12 seconds, so aliasing is possible on slot-aligned metrics), and sub-second spikes are visible only through log-derived percentiles, not through the metrics graph. Where we publish p99 numbers, they come from log parsing, not from histogram_quantile on 15s buckets. The relevant posts call this out where it affects interpretation.
Three subscription tiers, structured around what we can responsibly support at each price point:
\n| Plan | Price | Data delay | Retention | Grafana users | Custom runs | Exports |
|---|---|---|---|---|---|---|
| Free | EUR 0 | 7 days | 90 days | 1 | no | no |
| Pro | EUR 3,200 / month, billed annually | near real-time | unlimited | 5 | no | no |
| Enterprise | custom | near real-time | unlimited | custom | yes | yes |
The tiers map to different reader profiles. Free is the right level for ecosystem observers, students, and curious operators who want to read the published numbers and verify our claims against a delayed but otherwise full dataset. Pro is sized for client teams, research groups, consultants, competitive-intelligence shops, and press who need real-time access and unlimited retention across the fleet but do not need their own cohorts. Enterprise is for institutional operators and infrastructure providers who want to commission custom runs against their own questions, ingest data into their own pipelines via exports, and get a contracted scope of work with named contacts. Prices are EUR, excluding VAT, per the public plans page.
\nThe Free tier exists because we think the public Ethereum ecosystem benefits from having neutral measurements visible to anyone. Paid tiers fund the bare-metal fleet, datacenter, hardware refresh, on-call rotation, and ongoing engineering effort that keeps the data reproducible week over week.
\nThe blog and the public dashboards are open without subscription. Every post links back to the definitions, the methodology, and the scope page, so a critical reader can trace any claim back to the query that produced it.
\nIf you are in one of the above categories and want a custom dimension of the fleet measured against your specific question, Enterprise scope custom runs are designed for that. Start with the question, not with the SKU. Write to contact@stereumlabs.com describing what you want to learn, and we will scope it together, either inside the current scenario matrix or by spinning up a new VM cohort.
\n| Dimension | What StereumLabs does |
|---|---|
| Hardware anchor | 🟢 Bare-metal EPYC 9654P fleet in Vienna NDC2, with GCP cloud comparator |
| Coverage | 🟢 6 EC × 6 CC pairings on bare-metal (36 hosts) plus 1 standalone Caplin plus a GCP comparator cohort; ~90 hosts total |
| Metrics retention | 🟢 prometheus-cold keeps everything, scrape every 15s |
| Log capture | 🟢 Filebeat to Elasticsearch from every container |
| Comparability | 🟢 Fixed label vocabulary across all series |
| Neutrality | 🟢 Vendor defaults except for documented, justified deviations |
| Reproducibility | 🟢 Definitions, manifests, and procedures published per dashboard |
| AI workflow | 🟢 Claude via MCP against own data; raw telemetry stays in EU; mandatory citations |
| Public access | 🟢 Free tier with 7-day delay; Pro and Enterprise for full retention |
| Source of insight | 🟢 Weekly blog series on EC and CC behavior, archive growing since launch |
| Closed surface | 🟡 Custom run authoring is Enterprise-only |
| Coverage gap | 🟡 Pre-merge clients and L2 sequencers out of current scope |
If you read one thing next, make it the Caplin standalone vs classic Erigon comparison. It is the most recent end-to-end example of this stack answering a concrete question, and it shows what the manifest, the labels, and the AI workflow combined produce.
\nAfter that:
\nIf you want to talk about a custom measurement, a Pro or Enterprise account, or a question you would like the fleet pointed at, write to contact@stereumlabs.com. Tell us the question, not the SKU, and we will scope it with you.
", "url": "https://docs.stereumlabs.com/blog/stereumlabs-introduced-measurement-stack", "title": "StereumLabs introduced: the stack behind our Ethereum client measurements", "summary": "A technical companion to the RockLogic 'AI on own data' case study. We open the hood on the StereumLabs measurement platform: the bare-metal fleet, the metrics and logs pipeline, the label conventions that make 36 execution-client + consensus-client pairings comparable, and the in-house AI workflow that turns the resulting telemetry into the blog posts you read here.", "date_modified": "2026-05-20T00:00:00.000Z", "author": { "name": "Stefan Kobrc" }, "tags": [ "StereumLabs", "methodology", "bare-metal", "Proxmox", "Prometheus", "Elasticsearch", "Grafana", "Filebeat", "MCP", "observability", "reproducibility", "security-audit", "bug-bounty" ] }, { "id": "https://docs.stereumlabs.com/blog/erigon-caplin-standalone-vs-classic-combos", "content_html": "Before The Merge in September 2022, an Ethereum node was a single Proof-of-Work client. Geth, Parity, Erigon, Nethermind, or Besu, each handling everything from networking and the EVM to mining and chain selection inside one binary. The Merge split that role in two: an execution layer (EL) for the EVM, transactions, mempool, and world state, and a consensus layer (CL) for Proof-of-Stake fork choice, finality, attestations, and block proposals. The two halves talk to each other over the engine API, a JSON-RPC channel authenticated with a JWT secret.
\nMost operators run those two layers as two separate binaries that talk over the engine API. Caplin v3.3.10 collapses them back into a single Erigon binary. We added a Caplin standalone host to the StereumLabs fleet on April 8, 2026, and let it run alongside the classic Erigon plus CC pairings under identical mainnet conditions. This post reports what we measured: where the monolithic architecture wins, where it pays a tax, and where it leaves observability holes.
\n![\"Caplin](\"https://docs.stereumlabs.com/assets/images/erigon-caplin-thumbnail-a69721318032392cc76b912f28c352c9.png\")
The standalone deployment runs on a single bare-metal VM in our Vienna NDC2 cluster. Caplin runs as part of the Erigon binary with the consensus and execution layers sharing one executable, one MDBX database, and one combined libp2p plus devp2p networking stack. The host has 16 vCPU, 31.3 GB RAM, and 1.9 TB of NVMe storage, running Ubuntu 22.04.
\nThe classic Erigon combos run as two separate binaries on two VMs per pairing. We split EC and CC by design across two hosts so that node-exporter measurements give us clean per-layer resource consumption. The EC side gets 12 vCPU, 15.6 GB RAM, and 1.9 TB. The CC side gets 8 vCPU, 11.7 GB RAM, and 527.8 GB. Combined, that is 20 vCPU, 27.3 GB RAM, and roughly 2.4 TB of disk per pairing.
\n| Profile | Architecture | CPU | RAM | Disk |
|---|---|---|---|---|
| Caplin standalone | EL plus CL combined in one Erigon binary | 16 vCPU | 31.3 GB | 1.9 TB |
| Classic Erigon plus CC | Two separate binaries (EL and CL) | 20 vCPU | 27.3 GB | 2.4 TB |
That asymmetry matters. The standalone has 20% fewer vCPUs and 21% less disk, but 15% more RAM concentrated on one host. Any direct comparison has to account for it. We summed the resource consumption across the CC and EC VMs of each classic pairing and compared the total to the standalone single host.
\nWe measured resource and stability metrics over a 7 day window using avg_over_time(...[7d:1h]) against our cold Prometheus datasource. Block processing latency comes from parsing 50,165 head updated log lines extracted via Elasticsearch, covering the trailing 24 hours across the standalone and all six Erigon plus CC pairings.
All six classic Erigon EC instances are paired with the same CC versions: Lighthouse v8.1.3, Lodestar v1.42.0, Nimbus v26.3.1, Prysm v7.1.3, Teku 26.4.0, and Grandine 2.0.4. Both the standalone and the classic ECs run Erigon v3.3.10. For the StereumLabs label conventions and metric availability per client, see Client metrics scope.
\nThe first question is whether one binary holding two layers consumes more or less than two binaries holding one layer each.
\n![\"Resource](\"https://docs.stereumlabs.com/assets/images/erigon-caplin-resources-9c4cef815e7147c707822545effcb393.png\")
| Metric | Caplin standalone | Erigon plus CC range | Erigon plus CC median |
|---|---|---|---|
| CPU (vCPU cores) | 1.21 | 0.68 to 2.20 | 1.20 |
| RAM used (GB) | 13.0 | 11.35 to 14.32 | 13.09 |
| Disk read (MB/s) | 5.0 | 5.17 to 17.01 | 5.82 |
| Disk write (MB/s) | 18.1 | 14.69 to 31.42 | 16.07 |
| Network RX (MB/s) | 0.73 | 0.77 to 2.04 | 1.75 |
| Network TX (MB/s) | 0.73 | 0.46 to 1.70 | 1.23 |
| Swap usage (GB) | 3.83 | 2.47 to 4.58 | 3.61 |
Caplin standalone lands at or near the median of the Erigon plus CC pairings on every dimension except network, where it consumes well under half what the classic combos do. That gap is structural. In our split deployment, every engine_forkchoiceUpdated, engine_newPayload, and engine_getBlobsV2 call traverses the network between the CC and EC VMs. In the standalone binary, those calls become in-process function invocations that never touch a NIC.
Memory deserves a closer look. Caplin's container log reports Rss=26.1GB Pss=26.1GB Anonymous=11.7GB Swap=3.7GB, which seems to contradict the 13 GB we measured from node_memory_MemTotal_bytes minus node_memory_MemAvailable_bytes. Both numbers are correct. MDBX is memory-mapped, and roughly 14.5 GB of Erigon's RSS is shared-clean page cache that the kernel can reclaim instantly under pressure. MemAvailable correctly counts those pages as free. The 13 GB figure represents the working set that cannot be evicted without forcing reads from disk.
The 3.83 GB swap is also nothing unusual. Every classic Erigon EC swaps between 2.4 and 2.8 GB on its own, and the paired CC adds another 0 to 2.1 GB. Combined, the Erigon plus CC swap usage ranges from 2.5 to 4.6 GB, putting Caplin's 3.83 GB right in the middle. This is normal MDBX cold-page eviction behavior.
\nCaplin standalone runs two separate P2P networks from a single binary. devp2p for execution layer peers, libp2p for the consensus layer. Each has its own peer count.
\nOn the execution side, devp2p reports 64 connected peers, split evenly across the eth/68 and eth/69 protocol versions. The classic Erigon EC instances on the same fleet report 42 to 43 peers each. Same Erigon binary, same network, same configuration philosophy, but the standalone consistently maintains roughly 50% more EC peers. The likely cause is the larger CPU and memory budget on the standalone host. For more on how the various ECs behave at the P2P layer, see our EC P2P peering deep dive.
\nOn the consensus side, the picture is harder to extract. Caplin v3.3.10 does not expose a current libp2p peer count gauge in Prometheus. The metric appears only in the periodic container log line P2P app=caplin peers=127. We had to switch to log parsing to recover the value.
| CC | libp2p peer count |
|---|---|
| Grandine | 201 |
| Lodestar | 200 |
| Lighthouse | 198 to 200 |
| Teku | 98 to 100 |
| Caplin standalone | 127 |
| Nimbus | 81 |
| Prysm | ~73 |
Caplin sits in the upper-middle band, below the aggressive peer collectors (Grandine, Lodestar, Lighthouse) but well above the more conservative implementations.
\nCaplin v3.3.10 ships a comprehensive libp2p_rcmgr_* metric family covering connection lifetime counters, but no current peers gauge. Operators have to scrape the P2P app=caplin peers=N log line, which is emitted once per minute. This is the most significant observability gap we encountered.
A side benefit of Prysm's connected_libp2p_peers{agent=\"...\"} metric is that we can ask the Prysm fleet how many Caplin nodes it sees in the wider network. The answer is sobering: across all five working Prysm pairings combined, only 7 peers identify as agent=\"erigon/caplin\". Caplin nodes remain rare in the mainnet peer graph. Some of those 7 may even be our own standalone connecting to multiple Prysm instances on the same datacenter network. Standalone Caplin is still a young deployment pattern.
To compare execution performance precisely, we parsed every head updated log line from the trailing 24 hours on the standalone and all six classic Erigon EC instances. The line format gives us four numeric fields per imported block: arrival age relative to slot start, EVM execution time, MDBX commit time, and the moving average gas throughput.
After parsing, we have between 5,113 and 7,174 samples per host, totaling roughly 50,000 blocks. The dataset is large enough that the percentile estimates are stable.
\nThis is the time from the block being received and verified to its transactions being fully executed by the EVM, before the database commit phase.
\n![\"Block](\"https://docs.stereumlabs.com/assets/images/erigon-caplin-execution-time-49337bd5c00c71c79d172e5150af1efd.png\")
| Host | n | p50 (ms) | p90 (ms) | p99 (ms) |
|---|---|---|---|---|
| Caplin standalone | 7142 | 865 | 1955 | 5233 |
| Erigon plus Lodestar | 7157 | 981 | 2049 | 5239 |
| Erigon plus Lighthouse | 7118 | 1014 | 2102 | 5396 |
| Erigon plus Grandine | 7141 | 1026 | 2161 | 5268 |
| Erigon plus Nimbus | 7164 | 1037 | 2208 | 5587 |
| Erigon plus Prysm | 6847 | 1039 | 2170 | 5532 |
| Erigon plus Teku | 5113 | 1135 | 2637 | 5371 |
At the median, Caplin executes blocks 12% faster than the next-fastest pairing (Lodestar) and 31% faster than the slowest (Teku). At the 90th percentile the lead is similar. At the 99th percentile the standalone still leads, though by a smaller margin.
\nThe same story shows up in throughput. Mgas per second is the metric Erigon reports as a 12-block moving average inside the head updated line.
\n| Host | p50 (Mgas/s) | p90 (Mgas/s) | p99 (Mgas/s) |
|---|---|---|---|
| Caplin standalone | 33.6 | 49.4 | 89.6 |
| Erigon plus Lodestar | 29.3 | 43.0 | 80.7 |
| Erigon plus Lighthouse | 28.3 | 42.0 | 74.8 |
| Erigon plus Grandine | 28.2 | 41.4 | 75.1 |
| Erigon plus Nimbus | 27.7 | 41.3 | 73.9 |
| Erigon plus Prysm | 27.7 | 41.1 | 77.4 |
| Erigon plus Teku | 25.8 | 38.7 | 93.3 |
Caplin processes 14% to 30% more gas per second than any classic Erigon pairing across the percentile range. The Erigon binary is identical on every host, so the speedup is some combination of three things. The standalone has four extra vCPUs (16 vs 12). It avoids engine API marshaling because the CL hands payloads to the EL through in-process calls instead of JSON-RPC. And state lookups during execution skip serialization across the engine API boundary. Our data cannot fully separate the architectural contribution from the hardware contribution, but both push in the same direction.
\nNow the unflattering side. After execution, Erigon writes state changes to MDBX. In a classic deployment, this commit phase covers only EL state. In the standalone, the same phase also has to persist consensus-layer updates from the imported block, including beacon block headers, attestation aggregations, sync committee participation, and validator state changes.
\n![\"Commit](\"https://docs.stereumlabs.com/assets/images/erigon-caplin-commit-time-230cdc8b657eb1e5388b104118bd9844.png\")
| Host | p50 (ms) | p90 (ms) | p99 (ms) |
|---|---|---|---|
| Erigon plus Nimbus | 118 | 163 | 234 |
| Erigon plus Lighthouse | 120 | 167 | 240 |
| Erigon plus Lodestar | 121 | 166 | 246 |
| Erigon plus Teku | 121 | 172 | 297 |
| Erigon plus Grandine | 123 | 172 | 294 |
| Erigon plus Prysm | 126 | 175 | 274 |
| Caplin standalone | 262 | 362 | 519 |
Caplin's commit takes more than twice as long. The penalty is consistent across percentiles and time of day. The most plausible explanation is workload size: the standalone has to write more state to MDBX during the same block boundary because it owns both layers. Whether that happens as one MDBX transaction, two, or several sequential ones is something we did not verify at the source level. The metric is the total time the commit phase reports, regardless of how many transactions sit inside it.
\nAdding execution and commit gives the total time to fully process a new head block. This is the number that matters for downstream signaling, RPC freshness, and validator readiness.
\n![\"End](\"https://docs.stereumlabs.com/assets/images/erigon-caplin-end-to-end-f442f8a764bcfdb11256db556234f479.png\")
| Host | Exec p50 + Commit p50 (ms) |
|---|---|
| Erigon plus Lodestar | 1102 |
| Caplin standalone | 1127 |
| Erigon plus Lighthouse | 1134 |
| Erigon plus Grandine | 1149 |
| Erigon plus Nimbus | 1155 |
| Erigon plus Prysm | 1165 |
| Erigon plus Teku | 1256 |
At the 50th percentile end-to-end, Caplin is essentially tied with the fastest classic pairings. Lodestar plus Erigon edges it out by 25 ms, which is well within sampling noise. Caplin beats every other pairing by 7 to 129 ms.
\nAt the 99th percentile, the picture inverts slightly. The standalone's longer commit pushes its tail above some of the classic pairings, even though its execution tail is among the fastest. The total p99 spreads from 5485 ms (Lodestar plus Erigon) to 5821 ms (Nimbus plus Erigon), with Caplin at 5752 ms.
\nSo the honest framing is this: the monolithic node executes faster, commits slower, and ends up roughly equivalent at the median. The execution win is real but the commit penalty cancels most of it. Caplin's advantage is that it gets there with one binary instead of two.
\nBlock age, the time from slot start to when the EC first observes the block via gossip, is identical across all hosts: p50 of 3 to 4 seconds, p99 of 9 to 12 seconds. The network is not the differentiator here. Whatever happens after a block arrives is what we are measuring.
\nOver the 7 day window, the Caplin standalone process did not restart once. Its current uptime is 9.7 days. Internal RPC failure counter (rpc_failure) reads zero. The single warning we found in the entire week was a forkchoice was invalid message at 06:53 UTC on May 8, an isolated event with no follow-up symptoms.
We compared container log volumes and warning rates across the standalone and all six classic Erigon EC hosts. The classic ECs ship between 123,000 and 154,000 log lines per week. The standalone ships 100,744. Less log volume on a setup that handles both layers seems counterintuitive until you look at what the classic ECs are saying.
\n![\"Container](\"https://docs.stereumlabs.com/assets/images/erigon-caplin-log-volume-b602fcc4f8526d39ea780d13b3be3c75.png\")
| Host | Lines / 7d | WARN | ERROR | Panic | WARN rate |
|---|---|---|---|---|---|
| Caplin standalone | 100,744 | 1 | 0 | 0 | 0.001% |
| Erigon plus Lodestar | 123,020 | 16 | 0 | 0 | 0.013% |
| Erigon plus Nimbus | 137,857 | 2 | 0 | 0 | 0.001% |
| Erigon plus Prysm | 139,585 | 365 | 0 | 0 | 0.261% |
| Erigon plus Lighthouse | 140,865 | 22 | 0 | 0 | 0.016% |
| Erigon plus Grandine | 141,026 | 66 | 0 | 0 | 0.047% |
| Erigon plus Teku | 154,006 | 13 | 0 | 0 | 0.008% |
The dominant Erigon log pattern that appears in classic ECs but not in the standalone is [NewPayload] Handling new payload, fired once per engine API call from the CC. On the Teku-paired Erigon EC we measured 28 of these in 5 minutes. Extrapolated, that accounts for roughly 336 lines per hour, or about 56,000 lines per week of pure engine API chatter. Take that out and the classic ECs fall close to Caplin's volume.
Zero ERROR-level lines and zero panics across the entire Erigon family for 7 days. Both architectures are equally calm in steady state. The only meaningful difference is that the standalone has fewer log lines to ignore.
\nThe Caplin standalone reports an MDBX chaindata size of 34.34 GB. Five of the six classic Erigon EC pairings sit between 20.05 GB (Lodestar pair, recently resynced) and 39.94 GB (Prysm pair, with extra unprune'd cache), with most clustering around 34.4 GB.
That is striking. The standalone holds the entire EL state plus the entire CL beacon state, including beacon blocks, attestations, sync committees, validator state, and randao mixes. Yet its database is the same size as an EC-only Erigon. Either MDBX's append-only B-tree representation is exceptionally efficient for the beacon-state schema, or the standalone backfill is not yet complete. Caplin v3.3.10 does not expose a backfill progress metric, so we cannot tell from outside the process which case applies.
\nThis is the second observability gap worth flagging. Operators upgrading or restarting a Caplin standalone have no externally visible signal that historical CL state has finished downloading.
\nThe consensus side of Caplin exposes a different metric set than dedicated CCs, but the few metrics that exist look healthy:
\n| Metric | Caplin value |
|---|---|
aggregate_quality_50 | 0.995 |
attestation_block_processing_time | 65 ms |
active_validators_count | 913,144 |
committee_size | 438 |
current_epoch | 446,409 |
The attestation aggregation quality at the 50th percentile is 99.5%, meaning Caplin is folding nearly every unique-validator attestation it sees into the largest available aggregate. Attestation block processing time of 65 ms is faster than most CC implementations report for similar work.
\nBlock proposal metrics (block_producer_delay) read NaN. The Caplin standalone host is one of the few in our fleet that runs without attached validator keys, so it never proposes blocks. The classic Erigon plus CC pairings on Vienna NDC2 do have validators attached on their CC sides, but that workload is on the CC, not on the EC, and it does not influence the EC commit numbers we measured. Comparing block proposal performance directly between the architectures requires a future investigation that runs Caplin standalone with attached validators. The Nimbus block-building work in our Nimbus v26.3.1 post gives a sense of what that comparison would look like once we have the data.
Two operational properties of the standalone are real but not measurable from the outside.
\nThe first is failure-domain coupling. With separate binaries (whether on one VM or two), an EC restart leaves the CC running and able to continue gossiping, observing the chain head, and serving its own RPC clients. The CC may flag the EC as offline, but it does not crash. In the standalone, an EC restart is a CC restart by definition. There is no degraded mode where one layer keeps operating while the other recovers.
\nThe second is upgrade granularity. With separate binaries, you can update the EC while leaving the CC stable, or vice versa. This matters when one layer ships a critical fix and the other is in a regression window. The standalone bundles the upgrade. There is no path to upgrade only the EL or only the CL while keeping the other version pinned.
\nNeither point is a defect, both are intentional consequences of the design choice. They are worth weighing against the resource and stability gains.
\nErigon has a strong reputation as the most efficient archive-class Ethereum execution client. Its compact MDBX storage and pipelined sync turn full historical state into a workable handful of terabytes where many other clients require multiples of that. We documented some of that elsewhere in Execution client sync speed. Caplin standalone extends that strength by adding the consensus layer to the same binary. The operational unit becomes a single binary holding everything: full historical EVM state, beacon chain history, and a complete view of Ethereum from genesis to tip.
\nThat makes Caplin standalone a particularly good match for:
\nIt is less appropriate for:
\nFor pure staking-only setups without external RPC consumers, the case for the standalone is genuinely close. The resource efficiency is real, the stability so far has been very good, and the operational surface area is smaller. The case against it comes down to one question: how much do you value being able to recover one layer while the other keeps working? If the answer is \"a lot\", the split deployment still has a meaningful edge. If the answer is \"less than I value the simpler operations\", the standalone earns its place.
\n| Dimension | Result |
|---|---|
| EVM execution speed | 🟢 Caplin +12 to +31% faster at p50 |
| Gas throughput | 🟢 Caplin +14 to +30% higher across percentiles |
| MDBX commit time | 🔴 Caplin 2x slower (more state per block boundary) |
| End-to-end head update at p50 | ⚪ Roughly equivalent to fastest classic pair |
| End-to-end at p99 | ⚪ Mid-pack, slightly long-tailed |
| EL peer count | 🟢 Caplin +50% vs classic Erigon EC |
| CL peer count | 🟢 Upper-middle band (127 vs 73 to 201) |
| Resource footprint | 🟢 At or below median of combo sums; minus 40 to 60% network |
| Log volume | 🟢 18 to 35% less than classic ECs |
| Warning rate (7d) | 🟢 0.001% (tied with Nimbus pairing for lowest) |
| Stability | 🟢 0 restarts, 9.7d continuous uptime |
| libp2p peer gauge | 🔴 Not exposed in Prometheus, log-only |
| Backfill progress | 🔴 Not exposed |
| Failure-domain isolation | 🔴 Single binary for both layers |
| Update granularity | 🔴 EL and CL upgrade together |
We will revisit this comparison once Caplin v3.3.10 has accumulated enough peer adoption to escape the seven-peers-in-our-Prysm-fleet floor, and once we can measure block proposal performance with attached validators on a standalone host. Until then, the standalone holds up well as a production candidate, the trade-offs are exactly where the architecture says they should be, and the audience it fits best is broader than the staking-only crowd usually assumes.
\nResource and stability metrics queried from our cold Prometheus datasource using avg_over_time(...[7d:1h]) instant queries evaluated at the end of the 7 day window. Host-level metrics filtered with device!~\"lo|veth.*|docker.*|br.*\" to exclude virtual interfaces. Process-level metrics filtered by job to separate Caplin and Erigon scrapes from node_exporter.
Block processing latencies extracted from Filebeat-shipped container logs via Elasticsearch. We parsed head updated log lines using regex matching for execution=, commit=, age=, and mgas/s= fields. Sample sizes are 5,113 to 7,174 blocks per host over 24 hours. ANSI color codes stripped before parsing.
Peer counts on the consensus side use a mix of Prometheus gauges and log-extracted values. Caplin standalone CL peers come from the P2P app=caplin peers=N log line emitted once per minute, since v3.3.10 does not expose a current libp2p peer gauge. Cross-fleet Caplin visibility uses Prysm's connected_libp2p_peers{agent=\"erigon/caplin\"}.
For the StereumLabs label conventions and how to point your own dashboards at our data, see Build your own dashboards. For prior context on EC behavior under similar conditions, see our posts on EC P2P peering, EC sync speed, and Teku cross-version analysis.
\nIf you want a custom version of this analysis for your own fleet or a deeper look at any single dimension, write to us at contact@stereumlabs.com.
", "url": "https://docs.stereumlabs.com/blog/erigon-caplin-standalone-vs-classic-combos", "title": "Erigon's monolithic node: 25% faster execution, 2x slower commits", "summary": "Caplin v3.3.10 is the first standalone setup in our fleet, running a complete Ethereum node from a single Erigon binary. We compared it head-to-head with all six classic Erigon plus CC pairings across 50,000 blocks and seven days of operational data.", "date_modified": "2026-05-13T00:00:00.000Z", "author": { "name": "Stefan Kobrc" }, "tags": [ "Erigon", "Caplin", "consensus client", "execution client", "standalone", "performance", "MDBX", "architecture", "archive node" ] }, { "id": "https://docs.stereumlabs.com/blog/ec-p2p-peering-behavior-analysis", "content_html": "Every execution client connects to different peers, sees a different slice of the network, and churns through connections at wildly different rates. We queried Prometheus metrics and Elasticsearch container logs across our entire fleet to find out what each EC actually experiences at the P2P layer, and what it means for Ethereum's network health.
\n![\"EC](\"https://docs.stereumlabs.com/assets/images/thumbnail-814e56f998d713e2f1884d032ef851f7.png\")
When people talk about Ethereum client diversity, the conversation usually centers around \"what percentage of the network runs Geth?\" But there's a deeper question that rarely gets asked: what does each client actually see?
\nEvery execution client maintains its own set of P2P connections through the DevP2P protocol. These peers are the node's window into the Ethereum network. Through them, the node receives new transactions, propagates blocks, and participates in the state sync protocol. If your peers are slow, your node sees transactions late. If your peers all run the same client, a single bug could blind you.
\nWe set out to answer a set of straightforward questions using our NDC2 bare-metal fleet in Vienna and GCP cloud instances (roughly 90 hosts running all 6 execution clients paired with all 6 consensus clients):
\nOur analysis combines two data sources: Prometheus metrics (7-day rolling averages via avg_over_time(...[7d:1h]) queries against our prometheus-cold datasource) and Elasticsearch container logs shipped by Filebeat from every EC container on the fleet. We've previously covered how these clients differ in sync speed and block building. This time, we're looking at how they differ in who they talk to.
Each EC exposes peer data differently. Some have rich Prometheus gauges; some log periodic peer reports; some give us almost nothing. Let's focus on what we actually found.
\nAll peer counts and diversity percentages in this post are 7-day averages from the fleet unless noted otherwise. Individual snapshots can vary, as we'll see in the churn section.
The first and most basic question: how many peers does each EC maintain?
\n![\"EC](\"https://docs.stereumlabs.com/assets/images/peer_counts-b2c540b1883aa02564b2f5de2b5cce23.png\")
| EC | Avg peers | Configured limit | % of limit |
|---|---|---|---|
| Reth v2.1.0 | 130 | ~130 | 100% |
| Ethrex v10.0.0 | 105 | high (uncapped?) | — |
| Nethermind 1.36.2 | 50 | 50 | 100% |
| Geth v1.17.2 | 50 | ~50 | 99% |
| Erigon v3.3.10 | 42 | ~50 | 84% |
| Besu 26.4.0 | 24 | 25 | 95% |
The spread is striking: Reth maintains over 5x the connections Besu does. Every client except Erigon runs at or near its configured maximum, which means the peer limit is the binding constraint. Erigon is the outlier, consistently operating below its limit, and we'll see why when we look at its protocol behavior.
\nBesu's default --max-peers=25 is the lowest of any EC. If you're running Besu, consider raising this to at least 50. With only 25 peers, a single-client bug affecting your dominant peer type can severely degrade your view of the network.
Our GCP cloud instances ran Geth with the same configuration as our NDC2 bare-metal fleet, but with one critical difference: the GCP default firewall did not have port 30303 (DevP2P) opened for inbound traffic. This is a very common situation: most cloud providers (GCP, AWS, Azure) block all inbound traffic by default, and operators who don't explicitly open port 30303 TCP+UDP end up in exactly this state. The Geth process itself has no idea, it starts normally, makes outbound connections, and follows the chain.
\nThe effect on peering is dramatic:
\n![\"Geth](\"https://docs.stereumlabs.com/assets/images/gcp_vs_ndc2-c968ad8395b0de50b8481bdfcfc067cc.png\")
| Metric | Port 30303 open (NDC2) | Port 30303 blocked (GCP) |
|---|---|---|
| Total peers | 50 | 15 |
| Inbound peers | 34 | 0 |
| Outbound peers | 16 | 15 |
With inbound blocked, Geth receives zero incoming DevP2P connections. Its 15 peers are entirely self-initiated outbound connections. With the port open (NDC2), the same Geth configuration receives 34 additional inbound connections, reaching the full 50-peer capacity.
\nThe outbound count is essentially identical (~15-16 both ways), confirming this isn't a Geth software or configuration difference. It's purely a network access issue. Geth initiates the same number of outbound connections regardless of whether it can receive inbound ones, but other nodes on the network simply can't reach it.
\nWhat makes this finding more nuanced is the consensus layer comparison. The CC-side libp2p peers are nearly identical: 158 on NDC2 vs 161 on GCP. The CCs use different ports and protocols (libp2p with QUIC and TCP), and those traverse NATs and firewalls without issues. This means an operator whose CC shows healthy peer counts has no obvious signal that their EC is running with 70% fewer connections than it could have. Everything looks fine on the consensus side while the execution layer is quietly degraded.
\nThis is not a \"cloud is worse than bare-metal\" finding. Our GCP instances simply had port 30303 blocked by the default firewall, and we expect many cloud-hosted Ethereum nodes to be in the same situation. If you run a node on any cloud provider, check whether TCP and UDP port 30303 are open for inbound traffic. If not, your EC is operating with outbound-only connections, roughly 70% fewer peers, a thinner mempool view, and slower block/transaction propagation. Your CC peer count will look healthy and give no indication of the problem.
This is where things get interesting. Let's focus on the data we actually have: two ECs expose per-client peer breakdowns in Prometheus, and one logs it periodically.
\nBesu exposes besu_peers_peer_count_by_client, giving us a full breakdown:
![\"Besu](\"https://docs.stereumlabs.com/assets/images/besu_peer_composition-be826dd89483476fcdab2040e465878b.png\")
Out of Besu's 24 peers on average, 18 are Reth nodes. Only about 2 are Geth, and Erigon barely registers. This is a remarkably skewed distribution. With three quarters of its connections going to a single client, a Reth-specific networking bug would effectively isolate Besu nodes from most of their peers.
\nEthrex exposes ethrex_p2p_peer_clients with more granular labels:
![\"Ethrex](\"https://docs.stereumlabs.com/assets/images/ethrex_peer_composition-2aebbc2df32b4548b7efff410f40ade9.png\")
Ethrex sees a much more diverse peer set. Geth leads at 43%, but the remaining 57% is spread across five other clients. Notably, Ethrex picks up \"exotic\" peers that other clients don't: gnode (likely a Geth fork or private deployment), various Nimbus execution layer variants, and even mempool scrapers. Ethrex appears to be the most permissive in accepting connections from non-standard clients.
Nethermind doesn't expose per-client peer counts in Prometheus, but it does something even better: it logs a full diversity report every 5 minutes in its container output. These log lines are goldmines:
\nPeers: 49 | node diversity: Reth (37%), Geth (22%), Nethermind (20%), Unknown (14%), Erigon (4%), Besu (2%)
We pulled these from Elasticsearch across all 6 Nethermind instances (each paired with a different CC) and found something unexpected: the peer composition varies dramatically between instances in the same datacenter.
\n![\"Nethermind](\"https://docs.stereumlabs.com/assets/images/nethermind_diversity_by_pairing-f94959736a0cdc8e058aea3b2fb09fea.png\")
| CC pairing | Top peer | 2nd peer | 3rd peer |
|---|---|---|---|
| Lighthouse | Reth 37% | Geth 22% | Nethermind 20% |
| Grandine | Nethermind 46% | Reth 26% | Unknown 14% |
| Lodestar | Nethermind 47% | Reth 20% | Geth 18% |
| Teku | Nethermind 57% | Reth 18% | Geth 12% |
| Prysm | Nethermind 62% | Reth 27% | Erigon 6% |
| Nimbus | Nethermind 70% | Reth 12% | Unknown 6% |
The Nimbus-paired Nethermind instance sees 70% Nethermind peers, while the Lighthouse-paired one sees a relatively healthy mix with Reth leading at 37%. Same client, same datacenter, same network, vastly different peer views.
\nThis is one of the more subtle findings from the analysis. The short answer: the CC does not affect how many peers the EC gets, but it does affect which peers the EC keeps.
\nWe checked peer counts per CC pairing across every EC on the fleet. The numbers are remarkably uniform:
\n| EC | Lowest CC pairing | Highest CC pairing | Spread |
|---|---|---|---|
| Geth | 49.3 (Lighthouse) | 49.6 (Teku) | 0.3 |
| Erigon | 41.8 (Teku) | 41.9 (Lighthouse) | 0.1 |
| Nethermind | 50.0 (Lighthouse) | 50.3 (Grandine) | 0.3 |
| Ethrex | 104.3 (Prysm) | 106.3 (Lighthouse) | 2.0 |
| Besu | 23.6 (Lighthouse) | 24.3 (Prysm) | 0.7 |
The CC pairing has essentially zero influence on peer count. Every Geth instance gets ~50 peers regardless of whether it runs with Lighthouse, Nimbus, or Teku. The same holds for every other EC.
\nBut peer composition is a different story. The Nethermind diversity logs show a 50-percentage-point swing in Nethermind peer concentration (20% with Lighthouse vs 70% with Nimbus). The Besu Prometheus data tells a similar story at smaller scale: the Grandine-paired Besu gets 18.6 Reth peers while the Nimbus-paired one gets 17.8, with corresponding shifts in Nethermind and Geth counts.
\nThe mechanism isn't direct. The CC doesn't participate in DevP2P at all. But the CC influences the EC's startup timing, engine API call patterns, CPU and memory pressure on the shared host, and the timing of when the EC first reaches out to the discovery network. Early peer bonding creates path dependency: the peers you discover first tend to persist, and the ones that persist shape which new peers you discover next through the Kademlia routing table. Different CCs create subtly different boot conditions, and those conditions cascade into measurably different peer equilibria.
\nThis has a practical implication: two operators running the same EC and the same CC version might still see different peer compositions, simply because their nodes booted at different moments or under different system load. Peer diversity is partly a matter of luck.
\nNethermind also logs the protocol version split among its peers, and Erigon reports its \"GoodPeers\" broken down by protocol:
\n![\"Protocol](\"https://docs.stereumlabs.com/assets/images/protocol_versions-b5a7f132e9bfcb4940bd42cb13e8c22f.png\")
Nethermind's view: eth69 dominates at 80-96% depending on the instance. The Prysm-paired node sees the highest eth69 adoption (96%), while Grandine and Lighthouse-paired nodes see around 80%. This makes sense: Nethermind and Reth (its top peers) both adopted eth69 early.
\nErigon's behavior is peculiar: every pairing shows exactly 32 eth68 GoodPeers. The eth69 count varies (9-32), but eth68 is always hard-capped at 32. The standalone Caplin instance gets 32+32=64 good peers, but CC-paired instances get only ~42. This hard cap on eth68 peers likely explains why Erigon consistently runs below its connection limit: it's artificially constraining its usable peer set by protocol version.
\nErigon's eth68 hard cap at 32 means it's effectively running with a smaller usable peer set than its connection count suggests. If you're running Erigon and noticing slow block propagation, this protocol-level constraint could be a contributing factor.
How fast does each EC rotate through its peer connections? This matters for network resilience: too little churn means stale, possibly slow peers accumulate; too much churn means constant overhead from handshakes and state negotiation.
\n![\"Peer](\"https://docs.stereumlabs.com/assets/images/churn_rates-2181b0f165ebd54b0cd1bd5a1b56b7ff.png\")
| EC | Key churn metric | Rate | Interpretation |
|---|---|---|---|
| Nethermind | Incoming connection attempts | 3.10/s | High inbound demand, rejecting most (at limit) |
| Reth | \"Too many peers\" rejections | 3.08/s | Constantly turning away new connections |
| Nethermind | Outgoing connections | 0.65/s | Moderate active peer seeking |
| Nethermind | Total disconnects | 0.34/s | Healthy rotation (~1,200/hour) |
| Besu | Disconnections | 0.12/s | Aggressive pruning despite few peers |
| Besu | New connections | 0.009/s | Very slow peer acquisition |
| Ethrex | Disconnections | 0.00006/s | Near zero, essentially static peer set |
The spread here is staggering: a factor of roughly 50,000x between the most and least churning clients.
\nBesu has the most aggressive peer rotation relative to its pool size: it disconnects at 14x the rate it connects. With only 25 slots, it's constantly cycling. Average peer sessions are very short.
\nNethermind receives the most incoming connection attempts (3.1/s) but maintains a steady 50. Most incoming connections are rejected because the limit is reached, not because of quality issues.
\nReth is the most sought-after peer on the network. It rejects ~3 connections per second, tracks 7,400 peers in its peer table (the largest DHT view of any EC on the fleet), and has 540 backed-off peers at any given time. Its 130 connected peers are stable and long-lived.
\nEthrex has the most stable peering of any client we measured: essentially zero disconnection rate. Once it connects a peer, it keeps it indefinitely. This is a double-edged sword: stable peering means reliable gossip propagation, but no rotation means stale or slow peers are never pruned.
\nOne of the most striking findings from this analysis is that no two execution clients see the same network. Even running on the same hardware in the same datacenter:
\n![\"Each](\"https://docs.stereumlabs.com/assets/images/network_views_compared-77543ebb78c412028d68770a43c71910.png\")
This per-client perspective gap is a form of network fragmentation that isn't visible in aggregate statistics. The overall network might be 55% Geth, but individual nodes don't experience that average. They experience their own biased sample.
\nAggregate client diversity numbers (e.g., \"Geth runs on 55% of nodes\") describe the network as a whole, but individual nodes experience a much more concentrated version. When we hear that no single client exceeds 55% network share, it sounds like the network has a safety margin. But our data shows that at the individual node level, the picture is far less balanced.
\nA Besu node with 75% Reth peers doesn't live in a \"55% Geth\" network. It lives in a \"75% Reth\" network. If Reth pushes a release with a networking regression, that Besu node doesn't lose 20% of its peers (proportional to Reth's global share). It loses 75% of them. Similarly, a Nimbus-paired Nethermind node with 70% Nethermind peers is exposed to a Nethermind-specific issue at 3-4x the rate the aggregate numbers would suggest.
\nThis matters in concrete scenarios:
\nClient-specific bugs during chain events. When a contentious or complex fork activates (as has happened multiple times in Ethereum's history), clients occasionally disagree on block validity for a short window. During these periods, nodes need diverse peers to hear both sides of the fork quickly. A Besu node whose 18 Reth peers all follow one fork branch would receive almost no blocks from the other branch for critical minutes.
\nCascading failures from bad releases. If a popular client ships a broken release that causes resource exhaustion or crashes, nodes heavily peered with that client lose their connections simultaneously. On a healthy, diverse peer set, losing one client type drops you from 50 to 40 peers. On a Besu node with 75% Reth, it drops you from 24 to 6. Six peers is below the threshold where gossip propagation remains reliable.
\nTransaction propagation and MEV. Transactions are propagated through the EC peer mesh. If your peers are disproportionately one client, your node's mempool is shaped by that client's transaction acceptance logic, ordering behavior, and propagation timing. For validators, this can mean seeing fewer unique transactions and missing MEV opportunities. We've previously shown how EC choice affects block building output when paired with the same CC, and peer composition is one of the contributing factors: a node with a thin, homogeneous peer set simply sees fewer transactions to include.
\nConsensus finality during network partitions. If a network-level event (e.g., a BGP hijack, a cloud provider outage) takes out a large fraction of one client's nodes, the remaining network's ability to finalize depends on which clients can still communicate. Nodes with homogeneous peer sets are more likely to end up on the isolated side of a partition.
\nThe fundamental issue is that DevP2P's discovery mechanism (Kademlia DHT with distance-based routing) is optimized for finding any peers, not diverse peers. Unlike the consensus layer's gossipsub protocol, which has explicit scoring and mesh-balancing mechanisms, the execution layer has no built-in incentive for client diversity. Peers are peers, regardless of what software they run. This means diversity at the node level is largely a side effect of the overall network composition and the node's own discovery path, and our data shows that path can lead to surprisingly concentrated outcomes.
\nWith a default of 25 peers and 75% of those being Reth, a Besu node has roughly 6 non-Reth connections to the Ethereum network. That's not just thin, it's fragile. At 6 connections, the loss of even 2-3 peers (through normal churn) can temporarily leave the node with only 3-4 diverse connections. Gossip propagation at that level is unreliable.
\nFor comparison, Ethrex with 105 peers and 43% Geth still has 60 non-Geth peers even if every Geth node disappeared. That's still a functioning, well-connected node.
\nWhile stable connections improve gossip reliability in the short term, never pruning peers means degraded connections accumulate. If a peer goes slow but doesn't fully disconnect, Ethrex keeps it, potentially dragging down block and transaction propagation latency over time. Every other EC has some form of peer rotation that naturally filters out underperforming connections.
\nOur GCP finding (zero inbound peers due to a default firewall that didn't have port 30303 opened) is likely representative of a large number of cloud-hosted Ethereum nodes. Operators who deploy to GCP, AWS, or Azure without explicitly opening DevP2P ports end up with only 15 outbound connections instead of 50, and their CC peer count gives no warning. A significant portion of the network may be operating with degraded EC connectivity without anyone noticing, which reduces the overall density of the gossip mesh and slows transaction propagation at scale.
\nFor comparison, the consensus layer's gossipsub mesh shows much better diversity. From our Lighthouse nodes' block gossip mesh:
\n| CC client | Mesh peers | Share |
|---|---|---|
| Prysm | 2.2 | 37% |
| Lighthouse | 1.6 | 27% |
| Teku | 1.1 | 19% |
| Nimbus | 0.5 | 9% |
| Lodestar | 0.4 | 6% |
No single client dominates more than 37%, and the gossipsub scoring system actively maintains diversity. The execution layer's DevP2P protocol has no equivalent mechanism. This is consistent with what we observed in our blob performance analysis, where data column gossip delay was uniform across all EC pairings, confirming that the CL's P2P layer operates largely independent of the EC underneath. If DevP2P were to adopt some form of diversity-aware peer scoring (even a soft preference for connecting to underrepresented clients), the per-node diversity picture could improve substantially without requiring any consensus-level changes.
\nWe can combine our peer dependency data with public network share estimates to model what would actually happen if a specific EC suffered a critical failure (a consensus bug, a bad release causing crashes, or a networking regression that takes all instances offline).
\nCurrent approximate EC network shares (from ethernodes.org and clientdiversity.org, April 2026):
\n| EC | Network share |
|---|---|
| Geth (incl. forks) | ~50% |
| Nethermind | ~21% |
| Reth | ~12% |
| Besu | ~11% |
| Erigon | ~5% |
| Ethrex | ~1% |
Before we model scenarios, an important nuance: an EC going offline does not automatically mean the validator goes offline. Professional staking operators (Coinbase, Lido's curated set, institutional stakers) typically run multiple ECs with failover at the validator client or signer level. If Geth crashes, their validator client can switch to a Nethermind or Besu backup within seconds. For these operators, an EC failure causes a brief blip, not a prolonged outage.
\nSolo stakers, home stakers, and smaller operators are a different story. Most run a single EC paired with a single CC. For them, an EC failure means their validator goes offline until they manually intervene: restart the client, wait for a patch, or migrate to a different EC entirely (which means syncing from scratch, a process that can take anywhere from 1.5 hours to 8+ days depending on the client).
\nThe real-world impact of an EC failure therefore depends on the mix of professional vs solo operators running that client. The scenarios below model the P2P cascading effects, which hit everyone regardless of their failover setup.
\nScenario 1: Geth fails. This is the most impactful scenario by raw numbers. Roughly half of all execution client instances go offline. On our fleet, Ethrex would lose 43% of its peers in one stroke. Nethermind instances peered heavily with Geth (the Lighthouse-paired one sees 22% Geth) lose a significant chunk too. For validators, the impact is split: professional operators with EC failover switch to their backup within seconds and keep attesting. Solo stakers running only Geth go dark until the issue is resolved. Whether finality breaks depends on how many of Geth's ~50% share consists of solo stakers without failover. If even a third of them can't switch quickly, that's ~17% of validators going offline, which combined with normal churn could push the network close to the 1/3 offline threshold where finality stalls. Even short of that threshold, the gossip mesh takes a massive hit: surviving ECs lose a large fraction of their peers simultaneously, transaction propagation slows across the board, and block proposals from surviving validators may include fewer transactions because the mempool is suddenly thinner.
\nScenario 2: Reth fails. At ~12% network share, a Reth failure seems manageable at first glance. But our data reveals a cascading dependency that makes this worse than the headline number suggests. Besu nodes on the fleet derive 75% of their peers from Reth. If Reth goes down, the average Besu node drops from 24 peers to 6. At 6 peers, gossip propagation becomes unreliable: blocks arrive late, transactions get missed, and the node may struggle to stay in sync. In effect, a Reth failure doesn't just remove 12% of the network's EC instances. It functionally degrades another 11% (Besu's share), potentially pushing Besu nodes below the peer threshold needed for reliable chain following. Besu validators don't go offline in the strict sense, their EC is still running, but they start missing attestations and producing delayed block proposals because their view of the chain is degraded. That turns a 12% EC outage into something closer to a 20-23% effective network degradation, well beyond what the simple market share numbers would predict.
\nScenario 3: Nethermind fails. At ~21% share, a Nethermind failure is the most concerning for the professional staking segment, since Nethermind has been the primary beneficiary of the \"move away from Geth\" push (Coinbase, stakefish, and others have publicly migrated large validator sets to Nethermind). These operators likely have failover, but the sheer scale of simultaneous failover events could create its own pressure: backup ECs suddenly handling double the engine API load, discovery networks flooded with reconnection attempts, and a temporary peer diversity shock as the second-largest client disappears from the gossip mesh. The Nethermind-to-Nethermind peering concentration we observed (up to 70% on some instances) means that even surviving Nethermind nodes, those that haven't crashed but are peered with ones that did, lose most of their connections and need to re-discover peers.
\nThe counterintuitive finding: By raw network share, Geth is the obvious answer for maximum damage. But Reth's failure is disproportionately dangerous relative to its 12% market share because of the peer dependency it has created with Besu. A 12% client taking down its own nodes plus functionally crippling an 11% client through peer starvation is a 2x amplification factor that no aggregate diversity statistic captures.
\nThis peer dependency amplification is invisible in the standard client diversity charts. The usual framing (\"no client above 33%\") assumes that each client's failure is contained to its own nodes. Our data shows that's not the case at the P2P layer. Client failures cascade through the peer graph, and the pattern of those cascades depends on the specific, measurable peer composition that each client maintains. This is exactly the kind of second-order effect that requires fleet-level observability to detect.
\nThis analysis is a short snapshot of one aspect of our fleet data. At StereumLabs, we continuously collect Prometheus metrics and container logs across all 6 consensus clients and all 6 execution clients, in every pairing, on both bare-metal (NDC2 Vienna) and cloud (GCP) infrastructure.
\nWe use this data for version comparison reports (like our Teku cross-version analysis and Prysm resource comparison), hardfork impact analysis, security audits, and deep dives like this one. The P2P peering layer alone has months of historical data waiting to be mined: how does peer diversity change over time? Do software updates shift the composition? What happens during hardfork activations? How do different geographic locations affect peering patterns?
\nThe dashboards are there, the data is there, and there is much more to find. If you're a researcher, an Ethereum client team, or an infrastructure operator and you want to dig deeper, get in touch. You can also watch our EthCC[9] talk for a broader overview of how we use AI-powered observability across the Ethereum staking stack.
\navg_over_time(...[7d:1h]) instant queries against our prometheus-cold datasource (UID aez9ck4wz05q8e, Grafana Org 6)._msearch proxy against Filebeat container log indices. Filtered by container.image.name per EC version.We measured gossip verification, KZG batch verification, data column reconstruction, and getBlobsV2 latency across 72 nodes running all six consensus clients. The verification speed spread between fastest and slowest client: 50x.
\n![\"Blob](\"https://docs.stereumlabs.com/assets/images/thumbnail-63b432c46e6b519bf6c89c76437e56e6.png\")
Blobs (Binary Large Objects) are a data type attached to Ethereum transactions, introduced with EIP-4844 (\"Proto-Danksharding\") and expanded with the Fusaka hardfork (December 2025), which activated PeerDAS (Peer Data Availability Sampling).
\nBefore blobs, Layer 2 rollups posted their transaction data into execution layer calldata, competing for the same gas as every other transaction. Blobs created a separate, cheaper data channel for rollup data. Each blob carries about 128 KB and lives on the consensus layer (beacon chain), not in execution layer state, so it does not permanently bloat the blockchain.
\nFusaka took this further with PeerDAS. Instead of every node downloading every full blob, blobs are split into 128 data columns via erasure coding. Each node only needs to custody a few of these columns, typically 4 custody groups for a standard node. This reduces bandwidth requirements while maintaining data availability guarantees through KZG commitments (mathematical proofs). We measured the hardware impact of PeerDAS on our fleet in a separate post.
\nThe lifecycle of a blob under PeerDAS:
\ngetBlobsV2 Engine API callEach of these steps has measurable performance characteristics, and they vary across consensus client implementations. That is what we measured.
\nMost validators today use MEV-Boost to outsource block building to external builders who compete on profitability. Blob transactions carry their own fee market (blob gas), so builders have an incentive to pack as many blob transactions as possible into their blocks.
\nWhen a validator proposes a block via MEV-Boost, the flow adds a step: the CC receives a blinded block from the relay, reveals it, and then gets the full execution payload including all blobs. The CC must then compute and broadcast all data column sidecars from these blobs before the attestation deadline. This means the proposer's CC must have full data availability (all 128 custody groups) to serve the data column sidecars. If the CC is slow at blob processing, the block's data columns reach the network late, which increases the chance of missed attestations by other validators who depend on timely gossip.
\nIn practice, MEV-Boost blocks tend to be blob-heavy because builders optimize for maximum gas revenue. A CC that handles blobs well under normal conditions but degrades at 6 blobs per slot will perform worse on MEV-Boost blocks than on locally built blocks.
\nOur StereumLabs fleet at the NDC2 datacenter in Vienna, Austria runs all six major consensus clients, each paired with all six major execution clients: 72 nodes total (36 CC×EC pairings on bare metal, plus 6 GCP-hosted supernodes not covered here). Each CC is connected to a validator client that assigns validator duties. The validator clients themselves are not part of the StereumLabs metrics pipeline, but their effect is visible indirectly: because validators are attached, every CC should operate as a supernode (128 custody groups) to guarantee full data availability for block proposals. Whether each CC actually does this is one of the things we checked. For details on methodology and infrastructure, see the StereumLabs documentation. The fleet is operated by RockLogic GmbH.
\nObservation window: April 24, 2026, 11:00 to 12:00 UTC. All histogram percentiles and rates in this post are computed over this 1-hour window. Counter-based totals (like getBlobsV2 request counts) are cumulative since last process restart. Longer time ranges and historical comparisons can be explored directly on the StereumLabs dashboards. If you need a custom analysis or report for a specific time range, client pairing, or metric, reach out to us at contact@stereumlabs.com.
\n| Consensus Client | Version | Execution Clients |
|---|---|---|
| Grandine | 2.0.4 | Besu, Erigon, Ethrex, Geth, Nethermind, Reth |
| Lighthouse | v8.1.3 | Besu, Erigon, Ethrex, Geth, Nethermind, Reth |
| Lodestar | v1.41.1 | Besu, Erigon, Ethrex, Geth, Nethermind, Reth |
| Nimbus | v26.3.1 | Besu, Erigon, Ethrex, Geth, Nethermind, Reth |
| Prysm | v7.1.3 | Besu, Erigon, Ethrex, Geth, Nethermind, Reth |
| Teku | 26.4.0 | Besu, Erigon, Ethrex, Geth, Nethermind, Reth |
All nodes run on identical bare-metal hardware in Vienna. Performance differences reflect client implementation, not infrastructure.
\nMetrics are averaged across all EC pairings per CC unless otherwise noted. All histograms are computed over a 1-hour rolling window.
Since every CC on our fleet has a validator client attached, each CC should automatically upgrade to 128 custody groups (full supernode) to ensure complete data availability for block proposals. The question is: do they actually do this?
\n![\"Custody](\"https://docs.stereumlabs.com/assets/images/custody_groups-618ec72cb12608d73010645f4d82ca38.png\")
The beacon_custody_groups metric is only exposed by Lodestar, Grandine, and Teku. Lodestar and Grandine both report 128 custody groups, as expected for nodes with validators attached. Teku reports only 4 custody groups across all 6 EC pairings, despite having a validator client connected. This is unexpected and may indicate that Teku 26.4.0 does not auto-upgrade custody group count when validator duties are assigned, or that a separate CLI flag is required.
Lighthouse does not expose beacon_custody_groups, but a different metric tells the story: peers_per_custody_group_count shows Lighthouse tracking 104 to 106 custody groups (out of 128) across all EC pairings. That is consistent with supernode behavior. The missing 22-24 groups likely just have no peers at the moment rather than being uncustodied.
Nimbus and Prysm do not expose any custody-related metrics at all. We cannot confirm their actual custody group count from Prometheus. However, Nimbus's low data column processing rate (0.23 sidecars/s received, comparable to Teku's 0.29 req/s at 4 custody groups) suggests it may also not be running at full supernode level. Prysm's higher processing rate (3.7 req/s) is more consistent with supernode-level traffic, though still lower than Grandine (11.1 req/s) and Lighthouse (7.5 req/s).
\nThe custody group count has a direct impact on everything that follows in this post. A node custodying 128 groups downloads, verifies, and stores 32x more data column sidecars than a node custodying 4 groups. This has to be considered when comparing absolute processing rates and latency tails across clients.
\nTeku operators with validators attached should verify that their node actually custodies all 128 groups. With only 4 groups, the node may not have full data availability when it needs to propose a block. Check the beacon_custody_groups metric in your Grafana.
When a data column sidecar arrives via gossip, the consensus client must verify it before accepting it. This includes KZG proof and inclusion proof checks. Verification speed determines how quickly a node can process incoming data columns and meet attestation deadlines.
\n![\"Gossip](\"https://docs.stereumlabs.com/assets/images/gossip_verification_latency-d6616d04f9662f7699938918e4152de3.png\")
| CC | p50 | p95 | Notes |
|---|---|---|---|
| Teku | 3.9 ms | 9.5 ms | Fastest, tightest distribution |
| Nimbus | 3.2 ms | 8.7 ms | Own metric name* |
| Prysm | 4.2 ms | 22.7 ms | Fast p50, wider tail |
| Grandine | 4.0 ms | 19.0 ms | Despite 128 custody groups |
| Lighthouse | 10.6 ms | 24.2 ms | Middle of the pack |
| Lodestar | 35.7 ms | 469 ms | ~50x slower than Teku at p95 |
Nimbus uses data_column_sidecar_validation_duration rather than the beacon_data_column_sidecar_gossip_verification_seconds metric that other clients expose. The values are not perfectly 1:1 comparable but represent the same operation.
Operator perspective: At p95, Lodestar takes nearly half a second to verify a single data column. In a blob-heavy slot with 6 blobs (which expand to many data columns), this verification backlog can delay attestation timing. Attestations need to be published within the first 4 seconds of a slot, and each 469ms verification eats into that window.
\nResearcher perspective: Lodestar is the only consensus client written in TypeScript/Node.js. The cryptographic operations (KZG proofs, hash computations) are heavily optimized in compiled languages: Rust for Lighthouse/Grandine, Go for Prysm, Java with JNI for Teku, Nim for Nimbus. A JIT-compiled runtime faces a hard ceiling here.
\nKZG (Kate-Zaverucha-Goldberg) commitments are the mathematical proofs that guarantee data availability without requiring every node to download every blob. CCs batch-verify multiple KZG proofs at once for efficiency.
\n![\"KZG](\"https://docs.stereumlabs.com/assets/images/kzg_batch_verification-0ce19dcf77d8fdb2342c2b2875498636.png\")
| CC | p50 | p95 |
|---|---|---|
| Teku | 3.4 ms | 9.3 ms |
| Prysm | 3.5 ms | 31.6 ms |
| Grandine | 5.1 ms | 13.7 ms |
| Lighthouse | 11.0 ms | 32.0 ms |
| Lodestar | 112 ms | 286 ms |
Same pattern: Teku leads, Lodestar trails by an order of magnitude. We looked at Teku's performance evolution across three versions in a separate post. Worth noting here: Grandine performs better here than in gossip verification, which indicates its KZG implementation (likely Rust-native) holds up well even under the supernode load of 128 custody groups.
\nPrysm's verification speed is competitive (4.2ms p50). Its problem sits elsewhere: it fails to process the majority of incoming data column sidecars.
\n![\"Processing](\"https://docs.stereumlabs.com/assets/images/processing_success_rate-1a43af29d620a9d5367daa8d07bfd57d.png\")
| CC | Requests/s | Successes/s | Success Rate |
|---|---|---|---|
| Grandine | 11.1 | 11.1 | 100% |
| Lighthouse | 7.5 | 7.2 | 96.4% |
| Teku | 0.29 | 0.25 | 84.9% |
| Lodestar | 5.4 | 4.4 | 81.4% |
| Prysm | 3.7 | 1.6 | 42.9% |
Prysm processes only 43% of data column sidecar requests successfully. More than half fail, forcing the client into a costly fallback: data column reconstruction.
\nTeku's low absolute rate (0.29 req/s) reflects its standard 4-custody-group configuration. It receives fewer data columns. Its 85% success rate may be partially inflated by sampling effects at low volume.
\nWhen a node does not collect all required data columns via gossip, it can attempt to reconstruct missing columns from the ones it has, provided it has at least 50% of the 128 columns. Reconstruction uses erasure coding and is computationally expensive.
\n![\"Reconstruction\"](\"https://docs.stereumlabs.com/assets/images/reconstruction-97c463481a3cc5cebca6ee23acdedd47.png\")
| CC | Reconstructions/s | p50 Latency | p95 Latency |
|---|---|---|---|
| Prysm | 0.33/s | 379 ms | 689 ms |
| Lodestar | 0.007/s | 321 ms | 610 ms |
| Lighthouse | ~0/s | 289 ms | 834 ms |
| Teku | 0 | n/a | n/a |
| Grandine | 0 | n/a | n/a |
Prysm triggers reconstruction roughly once every 3 seconds, orders of magnitude more than any other client. With a 379ms median reconstruction latency, this adds real computational overhead.
\nTeku and Grandine never need reconstruction. They collect their custody columns via gossip every time, consistent with their high processing success rates.
\nFor Prysm operators: The 43% success rate and constant reconstruction point to an issue in the gossip subscription or processing pipeline for data column sidecars. If reconstruction starts failing too, the node will miss data availability deadlines. Worth monitoring across future versions.
Since Fusaka, consensus clients can fetch blobs directly from their local execution client via the engine_getBlobsV2 API. This is faster than waiting for gossip and helps the CC complete data availability checks sooner after a new block arrives.
We measured the getBlobsV2 response latency from Teku's perspective (the only CC that exposes per-EC latency histograms for this call):
![\"getBlobsV2](\"https://docs.stereumlabs.com/assets/images/getblobs_ec_latency-736f9ec1bad96ae9d81c8bb86e34106e.png\")
| EC | p50 | p95 |
|---|---|---|
| Reth | 6.3 ms | 40 ms |
| Ethrex | 8.2 ms | 49 ms |
| Besu | 9.4 ms | 47 ms |
| Nethermind | 12.1 ms | 55 ms |
| Geth | 34.0 ms | 143 ms |
| Erigon | 41.4 ms | 201 ms |
Reth serves blobs 6.5x faster than Erigon at p50. Reth and Ethrex are both Rust-based, newer execution clients with storage designs built after blobs existed. Geth and Erigon have blob pool implementations that predate the PeerDAS access pattern, which likely explains the gap. EC implementation differences also show up in sync speed, where we measured a range from 1.5 hours to 8+ days on the same hardware.
\nFor operators optimizing attestation timing: The EC's blob serving speed determines how quickly your CC can complete data availability checks after a new block arrives. With Reth or Ethrex, that takes ~6-8ms. With Erigon, it can take 200ms+ at the tail.
Total getBlobsV2 request counts from the Geth EC perspective (Geth is paired with every CC, so this gives a consistent comparison):
![\"getBlobsV2](\"https://docs.stereumlabs.com/assets/images/getblobs_usage-6efeca022a7d141081c21d6b405db760.png\")
| CC | Requested Blobs | Misses | Miss Rate |
|---|---|---|---|
| Lighthouse | 174,057 | 787 | 0.5% |
| Teku | 174,138 | 843 | 0.5% |
| Prysm | 168,721 | 844 | 0.5% |
| Grandine | 173,870 | 1,085 | 0.6% |
| Lodestar | 284,042 | 16,632 | 5.9% |
| Nimbus | 10,838 | 55 | 0.5% |
Two outliers:
\nNimbus makes 16x fewer getBlobsV2 requests than other clients. It relies primarily on P2P gossip to obtain blob data and uses the getBlobsV2 API as a fallback only. Whether this is intentional (gossip-first strategy) or because the feature is still in early adoption is not clear from the metrics alone. We observed similar architectural differences when investigating Nimbus block building behavior.
\nLodestar makes 63% more requests than the average and has a 5.9% miss rate, 10x higher than other clients. With 128 custody groups (supernode behavior), Lodestar aggressively tries to fetch blobs from the EL. The high miss rate suggests blobs have already expired from the EC's blob pool by the time Lodestar requests them, pointing to a timing issue in Lodestar's fetch pipeline.
\nFor context, we also measured when data column sidecars arrive relative to slot start. This is a pure network metric, independent of client implementation:
\n| Metric source | p50 | p95 |
|---|---|---|
| Lighthouse (all EC pairings) | ~1.57s after slot start | ~3.2s |
| Nimbus (all EC pairings) | ~1.39s after slot start | ~3.6s |
Data columns typically arrive 1.4 to 1.6 seconds after the slot begins, with 95% arriving within 3.2 to 3.6 seconds. This is consistent across all EC pairings. Gossip arrival is purely a P2P network phenomenon, not influenced by the local EC.
\nCCs therefore have roughly 0.4 to 2.6 seconds between data column arrival and the attestation deadline (4 seconds into the slot) to complete verification. At Teku's 3.9ms verification speed, that is plenty. At Lodestar's 469ms p95, it gets tight.
\n| Metric | Grandine | Lighthouse | Teku | Nimbus | Prysm | Lodestar |
|---|---|---|---|---|---|---|
| Custody groups | 128 | ~105† | 4 🔴 | ? | ? | 128 |
| Gossip verify p50 | 4.0 ms | 10.6 ms | 3.9 ms 🟢 | 3.2 ms 🟢 | 4.2 ms | 35.7 ms 🔴 |
| Gossip verify p95 | 19 ms | 24 ms | 10 ms 🟢 | 9 ms 🟢 | 23 ms | 469 ms 🔴 |
| KZG batch p50 | 5.1 ms | 11.0 ms | 3.4 ms 🟢 | n/a | 3.5 ms 🟢 | 112 ms 🔴 |
| Processing success | 100% 🟢 | 96.4% | 84.9% | n/a | 42.9% 🔴 | 81.4% |
| Reconstruction/s | 0 🟢 | ~0 | 0 🟢 | n/a | 0.33 🔴 | 0.007 |
| getBlobsV2 usage | 174k | 174k | 174k | 11k 🟠 | 169k | 284k 🟠 |
| getBlobsV2 miss% | 0.6% | 0.5% | 0.5% | 0.5% | 0.5% | 5.9% 🔴 |
🟢 = best in class · 🟠 = notable outlier · 🔴 = potential concern · †indirect via peers_per_custody_group_count
The 50x spread in gossip verification latency between Teku (3.9ms p50) and Lodestar (469ms p95) is the headline number from this analysis. It feeds directly into data availability timing models: how long does a CC need between receiving a data column and publishing its attestation? At 3.9ms the answer is \"no measurable impact\". At 469ms, each verification burns a chunk of the ~2.6 seconds between typical gossip arrival and the attestation deadline. PeerDAS security models that assume uniform verification times across clients need to account for this spread, especially since Lodestar is the only TypeScript-based CC and carries a runtime disadvantage for the cryptographic operations involved (KZG proofs, hashing).
\nThe custody group situation turned out more nuanced than expected. All nodes on our fleet have validator clients attached and should therefore custody all 128 groups. Lodestar, Grandine, and Lighthouse do this correctly. Teku 26.4.0 reports only 4 custody groups despite having validators assigned, which raises a question about its auto-upgrade behavior or whether a specific CLI flag is needed. Nimbus and Prysm do not expose custody metrics at all, making it impossible to verify from Prometheus alone. For operators, this is worth checking manually. For anyone modeling PeerDAS network-level data availability, the default custody behavior is not consistent across implementations.
\nPrysm's 43% processing success rate with constant reconstruction (0.33/s at 379ms p50 latency) points to a problem upstream of verification. The KZG verification itself is fast (3.5ms p50), so the bottleneck is in the gossip collection or processing pipeline. If reconstruction were to fail on top of that, the node would miss data availability deadlines.
\nOn the EC side, the choice of execution client has a measurable effect on blob serving latency via getBlobsV2. Reth and Ethrex respond in 6 to 8ms at p50. Geth and Erigon take 34 to 41ms, with Erigon reaching 201ms at p95. Over thousands of attestations, this compounds. It is a new performance dimension that did not exist before Fusaka and is worth considering when choosing an EC pairing, particularly for validators where attestation timing matters.
Nimbus's minimal use of getBlobsV2 (16x fewer requests than other clients) remains an open question. It may be a deliberate gossip-first architecture, or it may reflect early-stage getBlobsV2 adoption. The operational impact depends on whether gossip alone is sufficient to meet data availability deadlines reliably. Our earlier analysis of Nimbus block building behavior showed a similar pattern of Nimbus taking its own path on feature adoption.
Metric naming: Prysm uses millisecond-unit variants of several histogram metrics (e.g., beacon_data_column_sidecar_gossip_verification_milliseconds_bucket). Values were converted to seconds for comparison. Nimbus uses distinct metric names (data_column_sidecar_validation_duration, data_column_sidecars_received_total) that are not 1:1 comparable but represent the same operations. All data was queried from our prometheus-cold datasource. For details on our metric collection, see the StereumLabs client metrics documentation.
![\"Execution](\"https://docs.stereumlabs.com/assets/images/thumbnail-d3b86f364f7a3a2355efb9d61813bcf3.png\")
We deployed all 6 Ethereum execution clients from scratch on bare metal and tracked exactly how long each one took to fully sync. The results range from impressive to alarming.
\nOn April 8, 2026 at 19:41 UTC, we deployed the entire NDC2 bare metal fleet from scratch. All 36 execution client instances (6 ECs × 6 consensus client pairings) started within a 12 second window. The chain tip was at block ~24,837,000 (slot ~14,071,100).
\n| Execution Client | Version |
|---|---|
| Besu | 26.2.0 |
| Erigon | v3.3.10 |
| Ethrex | 9.0.0 |
| Geth | v1.17.2 |
| Nethermind | 1.36.2 |
| Reth | v1.11.3 |
Each EC is paired with all 6 consensus clients (Grandine 2.0.4, Lighthouse v8.1.3, Lodestar v1.41.1, Nimbus multiarch-v26.3.1, Prysm v7.1.3, Teku 26.4.0). All nodes run on identical bare metal hardware with NVMe storage at our NDC2 datacenter in Vienna. Every EC runs on its own dedicated host, separate from the consensus client.
\nFour of six execution clients (Reth, Geth, Ethrex, Besu) allowed their paired consensus client to report \"in sync\" while the EC itself was still hours or days from completing its own sync. The CC's beacon_head_slot metric is not a reliable indicator of EC sync status. All findings in this post are based on EC container logs and metrics, not CC reported head slots.
This analysis focuses on our NDC2 bare metal fleet, where each node has dedicated CPU, RAM, and NVMe storage with no resource contention. We also run the same client combinations on Google Cloud Platform (GCP) instances, which are available to StereumLabs users for side by side comparison. The two environments serve different purposes: bare metal gives us a controlled, reproducible baseline where hardware is never the variable, while GCP reflects the reality many operators face with shared infrastructure, variable I/O performance, and cloud specific constraints like burst credits and network throttling. Sync behavior can differ significantly between these environments, so we encourage users to explore both datasets on our platform.
\n![\"Execution](\"https://docs.stereumlabs.com/assets/images/sync-timeline-f7b2d000ade83fba3cab3731eb1386ac.png\")
| Rank | EC | Total Sync Time | Sync Method |
|---|---|---|---|
| 1 | Nethermind | ~1h 22min | Snap sync to pivot, then StateNodes |
| 2 | Erigon | ~13 hours | Staged sync (5K block batches) |
| 3 | Besu | ~13h snap + ~24h backfill = ~37h total | Pivot based snap sync + backward sync |
| 4 | Reth | ~5 days 18 hours | 14 stage pipeline (full replay) |
| 5 | Geth | ~7 days 15 hours | Snap sync (state healing bottleneck) |
| 6 | Ethrex | Not completed (8+ days) | P2P snap sync (stalled) |
Nethermind was the fastest client by a wide margin. Its container logs show a clean three phase sync:
\nPhase 1, FastHeaders (19:42 to 20:59, ~1h 17min): Nethermind set a pivot at block 24,837,019 and downloaded all headers. The sync mode transitioned through UpdatingPivot, FastHeaders, SnapSync, and finally StateNodes.
Phase 2, StateNodes (20:59 to 21:04, ~5min): World state download at the pivot point.
\nPhase 3, Block processing (21:04 onward): First \"Block throughput\" log appeared at 21:04 UTC, showing live block processing at 107 MGas/s and 850 tps. From this point, Nethermind was fully operational.
\n08 Apr 21:04:39 | Block throughput 107.72 MGas/s | 850.4 tps | blobs 12
Total: ~1 hour 22 minutes from cold start to processing live blocks.
\nErigon uses its unique staged sync pipeline. The sync Prometheus metric with per stage labels provides precise tracking:
Two Erigon instances (Lodestar and Teku pairings) required unscheduled process restarts 11 to 12 hours after deployment, indicating memory pressure during the execution stage.
\nTotal: ~13 hours.
\nBesu performed a pivot based snap sync from block 0:
\nPhase 1, Snap sync (19:42 to Apr 9 08:57, ~13h 15min): Besu downloaded headers backward from pivot 24,837,019 to block 0 while simultaneously fetching world state. The ethereum_blockchain_height metric starts at block 15,537,394, which is the first post Merge block (The Merge occurred at block 15,537,393 on September 15, 2022). Besu only needs to forward execute post Merge blocks since pre Merge PoW data is handled differently.
Sync completed successfully with pivot block 24839885
Phase 2, Backward sync (Apr 9 08:57 to Apr 10 08:54, ~24h): After pivot sync completed, Besu backfilled pre Merge block bodies and receipts in a separate backward pass.
\nStarting a new backward sync session
...
Current backward sync session is done
Total: ~13h 15min to live operations, ~37h for complete historical data.
\nBlock 15,537,394 is the first Proof of Stake block on Ethereum mainnet. Besu's snap sync downloads headers all the way back to genesis, but it only needs to execute blocks from The Merge forward since pre Merge PoW blocks use a different execution model. The ethereum_blockchain_height metric reflects this forward execution boundary, not an incomplete database.
Reth runs a sequential 14 stage pipeline from scratch. Container logs with Finished stage messages provide exact timing for every stage:
![\"Reth](\"https://docs.stereumlabs.com/assets/images/reth-pipeline-d72faca30bae8b52dd8ba33ad1fb2181.png\")
| Stage | Name | Started | Finished | Duration |
|---|---|---|---|---|
| 1/14 | Headers | Apr 8, 19:45 | Apr 8, 19:53 | 8 min |
| 2/14 | Bodies | Apr 8, 19:53 | Apr 8, 23:35 | 3h 42min |
| 3/14 | SenderRecovery | Apr 8, 23:35 | Apr 8, 23:35 | instant |
| 4/14 | Execution | Apr 8, 23:35 | Apr 14, 10:25 | 5d 10h 50min |
| 5/14 | PruneSenderRecovery | Apr 14, 10:25 | Apr 14, 10:25 | instant |
| 6/14 | MerkleUnwind | Apr 14, 10:25 | Apr 14, 10:25 | instant |
| 7/14 | AccountHashing | Apr 14, 10:25 | Apr 14, 10:34 | 9 min |
| 8/14 | StorageHashing | Apr 14, 10:34 | Apr 14, 11:40 | 1h 6min |
| 9/14 | MerkleExecute | Apr 14, 11:40 | Apr 14, 12:48 | 1h 7min |
| 10/14 | TransactionLookup | Apr 14, 12:48 | Apr 14, 13:47 | 59 min |
| 11 to 14 | IndexHistory, Prune, Finish | Apr 14, 13:47 | Apr 14, 13:47 | seconds |
The Execution stage consumed 93% of the total sync time, replaying all 24.8 million blocks sequentially. Headers downloaded in 8 minutes and bodies in under 4 hours, but the actual block execution took over 5 days.
\nTotal: 5 days 18 hours. Pipeline finished April 14, 13:47 UTC.
\nGeth performed a snap sync from genesis with parallel header and state downloads:
\nHeaders (19:42 to ~01:00 Apr 9, ~5h): Geth's header download progressed at ~1.25M headers/hour. Early logs showed \"Syncing beacon headers downloaded=512 left=24,836,569 eta=4h20m\". Geth explicitly rejected live CC blocks during this phase: \"Ignoring payload while snap syncing\".
\nState download (parallel with headers, ~1.5h): State download started simultaneously and progressed rapidly, showing \"state download synced=0.15% eta=1h24m\" at minute 10.
\nState healing (dominant bottleneck): After headers and initial state were downloaded within hours, state healing ran for days. This is the phase where Geth patches gaps and inconsistencies in the downloaded state trie.
\nSnap sync complete: April 16, 10:32 UTC.
\nSwitching from snap-sync to full-sync reason=\"snap-sync complete\"
Total: ~7 days 15 hours. State healing was the multi day bottleneck.
\nEthrex operates in dual mode: it immediately processes live blocks via the engine API while running P2P snap sync in the background. This means the paired CC tracks the chain head within seconds of startup, but the EC lacks complete historical state.
\nLive blocks (immediate): At 19:42:32 UTC, Ethrex received its first engine payload for block 24,837,020, just 26 seconds after startup.
\nP2P snap sync (ongoing): Snap sync started at 19:42:35. After ~3 hours it reached the \"Requesting Bytecodes\" phase. However, a restart at some point reset all progress. As of April 16, the snap sync status showed:
\nP2P Snap Sync | elapsed 00h 00m 07s | peers 7 | step Downloading Headers
The sync had effectively restarted from scratch with minimal peer connectivity.
\nTotal: Not completed after 8+ days. The P2P snap sync is effectively stalled.
\nOnce Ethrex completes its initial sync, our experience with it has been excellent. Block building performance and attestation behavior are on par with established clients, as shown in our Nimbus block building analysis. Getting to that point, however, requires significant patience and may involve manual restarts. The P2P snap sync implementation is still maturing.
We have developed internal workflows and tooling to work around the current sync challenges, and these are available to StereumLabs users who want to run Ethrex in their setup. Additionally, the LambdaClass team is actively working on newer Ethrex versions that are expected to address many of these sync related issues. A deeper analysis of Ethrex sync strategies and workarounds is outside the scope of this post, but we may cover it in a dedicated article in the future.
These are maximum values observed on each EC host during the first 6 hours after deployment (the peak sync burst period), measured via node exporter. Since each EC runs on its own dedicated host with no other significant services, these numbers reflect EC resource consumption directly.
\n![\"Peak](\"https://docs.stereumlabs.com/assets/images/peak-resources-629c705b2fbc2aa0435252110a2a0efd.png\")
| Metric | Nethermind | Besu | Geth | Erigon | Reth | Ethrex |
|---|---|---|---|---|---|---|
| Peak CPU (cores) | 10.8 | 5.2 | 9.7 | 4.5 | 2.9 | 7.9 |
| Peak RAM (GB) | 10.8 | 12.0 | 5.6 | 14.1 | 9.4 | 19.7 |
| Peak Disk Read (MB/s) | 561 | 27 | 44 | 176 | 61 | 219 |
| Peak Disk Write (MB/s) | 533 | 176 | 281 | 105 | 92 | 484 |
| Peak Net RX (MB/s) | 111 | 33 | 98 | 75 | 96 | 86 |
![\"Peak](\"https://docs.stereumlabs.com/assets/images/disk-io-4c4d04936df818997b3cac29f9771d7d.png\")
Nethermind was the heaviest on disk I/O: 561 MB/s reads and 533 MB/s writes during its snap sync burst. It traded high resource intensity for the fastest sync by far.
\nGeth was CPU intensive (9.7 cores) but had the lowest memory footprint (5.6 GB) of any EC during sync.
\nErigon consumed the most RAM among established clients (14.1 GB) despite being mid pack on sync speed, consistent with its known memory pressure pattern during staged execution.
\nEthrex had the highest RAM usage overall (19.7 GB) and very high disk writes (484 MB/s), driven by simultaneous live block processing and background snap sync.
\nReth was the most resource efficient during the initial 6 hour window: lowest CPU (2.9 cores), moderate RAM (9.4 GB), minimal I/O. However, this is misleading because Reth's pipeline was still in the Bodies download stage during this window, with the heavy Execution stage starting later and running for 5+ days.
\nThe gap between the fastest (Nethermind, 1.5h) and the slowest completed sync (Geth, 7.5 days) is a factor of 120×. For operators, this means the difference between being back online during a lunch break versus being offline for more than a week.
\nThis is the most operationally dangerous finding. Reth, Geth, Ethrex, and Besu all reported themselves as ready to serve the consensus client while their historical sync was far from complete. Operators monitoring only CC metrics (head slot, attestation inclusion) would see a healthy node while the EC was still days from full sync.
\nNethermind's 561 MB/s peak disk reads demonstrate why: its snap sync strategy saturates fast storage to finish quickly. On slower drives (SATA SSD or HDD), the sync time would scale proportionally, potentially turning Nethermind's 1.5 hours into many hours.
\nErigon's 5K block batch processing through the execution stage is the primary bottleneck. The staged approach provides clear progress visibility and checkpointing, but at the cost of being 10× slower than Nethermind. Two instances also required restarts due to memory pressure during this phase.
\nBoth clients must replay the entire chain history. Reth does this sequentially through its Execution stage (5+ days), while Geth's bottleneck is state trie reconciliation after its initial snap download. These are fundamentally different bottlenecks but produce similar multi day timelines.
\nEthrex delivers solid performance once fully synced, as our previous analyses have shown. The challenge is getting there. Its P2P snap sync is still maturing, and a restart during our test reset all progress. Peer connectivity (7 peers vs. 90+ for other clients) suggests the sync P2P discovery needs further work. The LambdaClass team is actively iterating, and upcoming Ethrex versions are expected to improve the sync experience considerably. For operators willing to invest patience (or use alternative bootstrapping methods), Ethrex is absolutely viable in production once it reaches the chain tip. We offer tooling and guidance for Ethrex sync on StereumLabs for those who want to get started today.
\nIn our previous analysis of Nimbus block building across 5 ECs, we found that Erigon's slow block execution (423 to 567ms per imported block) causes it to deliver empty block payloads. This sync investigation confirms the pattern: Erigon's execution pipeline is the consistent bottleneck, whether processing historical blocks during sync or building new blocks for proposals.
\nReth was excluded from that analysis because it had not completed its initial sync at the time. This post explains why: its full pipeline takes nearly 6 days.
\nprometheus-cold datasource) and Elasticsearch (Filebeat container log shipping) on the StereumLabs Grafana instance.Finished stage (Reth), Snap sync complete (Geth), Sync completed successfully / Backward sync session is done (Besu), Block throughput first appearance (Nethermind), P2P Snap Sync status messages (Ethrex), sync metric per stage progression (Erigon).node_exporter metrics with max_over_time(...[6h]) queries at each EC's dedicated host.How does each execution client behave when Nimbus asks it to build a block? We monitored 1,000 validator pubkeys across 5 EC pairings for 48 hours and found that block building performance varies dramatically, with one client producing near-empty blocks while the other four packed in millions of gas.
\n![\"Nimbus](\"https://docs.stereumlabs.com/assets/images/thumbnail-2723b2fb05afb7059583d0961cf8ae67.png\")
We run 5 Nimbus multiarch-v26.3.1 beacon nodes on our NDC2 bare-metal fleet in Vienna, each paired with a different execution client:
| Consensus Client | Execution Client | Location |
|---|---|---|
| Nimbus v26.3.1 | Besu 26.2.0 | NDC2, Vienna |
| Nimbus v26.3.1 | Erigon v3.3.10 | NDC2, Vienna |
| Nimbus v26.3.1 | Ethrex 9.0.0 | NDC2, Vienna |
| Nimbus v26.3.1 | Geth v1.17.2 | NDC2, Vienna |
| Nimbus v26.3.1 | Nethermind 1.36.2 | NDC2, Vienna |
A sixth node (Nimbus + Reth v1.11.3) is also deployed, but Reth has not yet completed its initial sync and is therefore excluded from this analysis.
\nAll 5 nodes use Nimbus' built-in validator_monitor feature to passively observe the same set of 1,000 validator pubkeys on-chain. The validator monitor tracks attestation inclusion, vote correctness, block proposals, and timing data for these validators without requiring the signing keys to be locally attached.
This is a shadow configuration: a reverse proxy mirrors the validator client's requests to these beacon nodes, but the beacon nodes' responses do not reach the validator client. This means the data reflects how each CC+EC pairing observes and reacts to validator duties, but is not fully representative of a production validator setup where the EC's block building output would actually be submitted to the network.
The full analysis covers a 48-hour window ending April 13, 2026. Data sources: Prometheus (prometheus-cold) for metrics and Elasticsearch (Filebeat) for both EC container logs and Nimbus CC container logs.
When one of the 1,000 monitored validators is selected as block proposer, Nimbus triggers engine_forkchoiceUpdatedV3 with payload attributes on its paired EC, asking it to build a block. The EC then constructs the execution payload iteratively, improving it over several seconds until Nimbus calls engine_getPayloadV4 to retrieve the result.
Over 48 hours, block building was triggered for approximately 13 unique blocks. The results differ dramatically between execution clients.
\nNimbus logs the exact timestamps for Requesting engine payload and Received engine payload, giving us the end-to-end build latency and resulting gas_used for every payload. This CC-side perspective is consistent across all ECs and fills in data even when the EC's own logs lack detail.
![\"Build](\"https://docs.stereumlabs.com/assets/images/latency_vs_gas-69fff28ab0a4030ad6fed61d39477262.png\")
| EC | Avg latency | Min | Max | Avg gas (Mgas) | Blocks |
|---|---|---|---|---|---|
| Besu | 546ms | 75ms | 848ms | 23.5 | 11 |
| Ethrex | 535ms | 27ms | 728ms | 21.8 | 13 |
| Nethermind | 519ms | 28ms | 696ms | 21.2 | 13 |
| Geth | 524ms | 24ms | 756ms | 15.0 | 13 |
| Erigon | 479ms (fastest) | 18ms | 523ms | 0.0 | 12 |
The most counterintuitive finding: Erigon is the fastest responder (479ms avg) yet delivers 0 gas. Its transaction pool is apparently unable to supply transactions within the build window, so it returns an empty block quickly rather than spending time filling it. Besu takes 67ms longer on average but uses that time to pack 23.5 Mgas into the payload.
\nFor high-gas blocks (slots with blob transactions), latency increases across all ECs: Besu peaks at 848ms for a 60M gas block, Ethrex at 728ms for the same block. This confirms that build latency scales with payload complexity.
\nOne slot shows anomalously low latency for all ECs (Erigon 18ms, Geth 24ms, Ethrex 27ms, Nethermind 28ms, Besu 75ms). This is likely a cached or pre-built payload that Nimbus retrieved immediately.
The averages above compress a wide range of behavior into single numbers. Plotting every individual block build reveals the full distribution:
\n![\"All](\"https://docs.stereumlabs.com/assets/images/all_blocks_scatter-9c77f4b92a19b6b2b2539a2041d9a758.png\")
Three distinct patterns emerge:
\nStandard blocks (500-650ms, 7-18 Mgas): The main cluster where most builds land. Besu, Ethrex, and Nethermind consistently occupy the upper band (14-18 Mgas), while Geth sits lower (7-15 Mgas). All four ECs overlap in latency, confirming that response time differences between them are marginal for regular blocks.
\nBlob-heavy blocks (700-850ms, 38-60 Mgas): A few outlier slots where blob transactions push gas usage to 38-60M. These blocks take noticeably longer to build across all ECs, with Besu and Ethrex reaching the gas limit (60M) while Geth tops out around 40-48M. The latency increase is proportional to payload complexity.
\nErigon (480-520ms, 0 Mgas): Every single Erigon build sits flat on the x-axis at 0 gas, forming a tight horizontal cluster. Regardless of whether the same slot produced a 17M gas block on Besu or a 60M gas block on Ethrex, Erigon delivered an empty payload. This pattern is consistent across all 12 blocks with no exceptions.
\nThe scatter also reveals a handful of cached payloads (sub-100ms) where all ECs returned near-instantly, likely from a previously prepared payload that Nimbus retrieved before the build window elapsed.
\nBy examining the Received engine payload log from all 5 Nimbus CC instances, we can compare the actual gas_used delivered by each EC for the same blocks:
![\"Engine](\"https://docs.stereumlabs.com/assets/images/payload_gas_comparison-ec8d1578b8c8c71b985d0a9322110f8a.png\")
| EC | Block #869114 | Block #869617 | Block #867424 | Block #866897 | Avg gas |
|---|---|---|---|---|---|
| Besu | 17.6M | 15.1M | 13.0M | 17.5M | ~23.5M |
| Ethrex | 16.9M | 17.8M | 18.0M | 18.0M | ~21.8M |
| Nethermind | 17.3M | 10.1M | 15.4M | 16.6M | ~21.2M |
| Geth | 14.0M | 7.1M | 11.0M | 8.3M | ~15.0M |
| Erigon | 0 | 41K | — | 0 | ~0 |
Besu, Ethrex, and Nethermind are the strongest block builders, routinely filling blocks to 15-30% gas utilization. Ethrex is particularly consistent, delivering 14-18M gas for standard blocks. Geth produces lighter payloads. Erigon's payloads are effectively empty.
\nBesu produced 661 block improvement iterations across its built blocks. Its logs show the full iterative building process with reward tracking:
\nNew proposal for payloadId 0x36f36d block 24869114
gas used 17,624,773 transactions 366 reward 2.22 finney
is better than the previous one by 103.24 szabo
For block #24,869,114, Besu's best build reached 366 transactions, 17.6M gas, and a 2.22 finney block reward. It logged every improvement step, including the marginal reward increase between iterations.
\nGeth logged 214 payload updates and provides the clearest view of the block building lifecycle:
\n![\"Geth](\"https://docs.stereumlabs.com/assets/images/geth_build_timeline-12b69b415ebfa7471f8c852a05fae94d.png\")
The build started with 6 transactions and grew to 349 transactions over ~10.5 seconds, with each update taking 25-67ms. Geth logs Starting work on payload, then multiple Updated payload entries (with txs, gas, fees, elapsed time), and finally Stopping work on payload reason=delivery when Nimbus retrieves the result.
Ethrex does not log block building progress in its own container logs at the default log level. However, Nimbus' CC logs reveal it is one of the strongest builders in the fleet, delivering 21.8 Mgas on average with one block hitting 100% gas utilization (60M gas). Each payload carries extra_data: ethrex 9.0.0.
What Ethrex does log is an excellent per-block execution breakdown when processing incoming blocks:
\nBLOCK EXECUTION THROUGHPUT (24869233): 0.366 Ggas/s
TIME SPENT: 55 ms. Gas Used: 0.020 (33%), #Txs: 134
block validation: 1% | exec(w/merkle): 91% | merkle-only: 2% | store: 7%
This level of transparency into where execution time is spent (91% in execution+merkle, 7% in storage, 1% in validation) is unique among the tested ECs and valuable for performance profiling.
\nNethermind's own logs only show production requests (Production Request 24869114 PayloadId: 0x2321052e5846b8a2) without per-iteration detail at the default log level. However, Nimbus' CC logs reveal the full picture:
INF Received engine payload slot=14103195 payload=\"(block_number: 24869114,
gas_used: 17257515, gas_limit: 60000000, extra_data: Nethermind v1.36.2, ...)\"
For block #24,869,114, Nethermind delivered 17.3M gas, competitive with Besu's 17.6M and Ethrex's 16.9M. Across all 13 payloads in 48h, Nethermind consistently produced blocks in the 9-17M gas range.
\nErigon's own logs show it actively attempts to build blocks but consistently fails to include transactions:
\n![\"Erigon](\"https://docs.stereumlabs.com/assets/images/erigon_build_outcomes-838549fad876cbf1739ea702cadba87f.png\")
76% of Erigon's 54 block build iterations produced completely empty blocks with 0 transactions and 0 gas. When it did include transactions, the count was dramatically lower than other clients: a maximum of 36 transactions versus 300+ for Besu and Geth.
\nBuilt block height=24869114 txs=0 executionRequests=0
gas used %=0.000 time=782.686ms
The Nimbus CC logs confirm this: across 12 blocks, every single payload from Erigon contained 0 or near-0 gas. This is not a latency issue. At 479ms average, Erigon is the fastest EC to return a payload. The problem is upstream: Erigon's block execution latency (423-567ms per imported block) delays transaction pool maintenance, so when Nimbus asks for a payload, Erigon has nothing to put in it.
\nIn a production setup, if Erigon were the EC responsible for building the block that gets submitted to the network, the validator would propose a near-empty block, forfeiting transaction fees and MEV revenue. For the ~13 blocks built in our 48h window, the difference between Besu's output (2.22 finney reward) and Erigon's (0 reward) is a direct loss per proposal.
While all 5 nodes observe the same 1,000 validators and see identical on-chain results (99.996% attestation inclusion rate), how quickly each node observes attestation events varies by EC pairing. Three pipeline stages were measured:
\n![\"Attestation](\"https://docs.stereumlabs.com/assets/images/attestation_delay-b252db05f19f50beb3c94573ea74ed22.png\")
| Pipeline stage | Besu | Erigon | Ethrex | Geth | Nethermind | Spread |
|---|---|---|---|---|---|---|
| Unaggregated attestation (p50) | 49ms | 50ms | 53ms | 52ms | 52ms | 4ms |
| Aggregated attestation (p50) | 35ms | 35ms | 38ms | 38ms | 38ms | 3ms |
| Attestation in aggregate (p50) | 142ms | 153ms | 141ms | 135ms | 145ms | 18ms |
Besu is the fastest for raw attestation observation. However, Geth leads in the third stage (attestation appearing inside aggregates), which is the most relevant for actual inclusion. Erigon is consistently the slowest in this final stage, adding 18ms of latency versus Geth. Ethrex performs well across all three stages.
\nAll 5 nodes should see roughly the same number of attestations for the monitored validators. Over 48 hours, they do not:
\n![\"Unaggregated](\"https://docs.stereumlabs.com/assets/images/attestation_volume-65c17df9eb75b2ff90bb44f489d823b2.png\")
| EC pairing | Attestations received (API) | Delta vs. best |
|---|---|---|
| Geth | 404,876 | baseline |
| Nethermind | 403,726 | -0.3% |
| Besu | 402,563 | -0.6% |
| Ethrex | 400,752 | -1.0% |
| Erigon | 359,104 | -11.3% |
Erigon misses ~45,000 attestations that the other 4 nodes see within the same 48-hour window. This is not a network issue; all nodes are on the same bare-metal fleet in Vienna. Erigon's slower block processing causes attestations to expire before the node can process them.
\nThe 1,000 monitored validators' three Casper FFG vote types show dramatically different difficulty levels:
\n![\"Attestation](\"https://docs.stereumlabs.com/assets/images/vote_type_hierarchy-f30e1137467b13ca7e57685aa8b5c144.png\")
| Vote type | Hits (48h) | Misses (48h) | Hit rate | What it measures |
|---|---|---|---|---|
| Source | 450,137 | 19 | 99.996% | Correct justified checkpoint |
| Target | 449,857 | 299 | 99.934% | Correct finalized checkpoint |
| Head | ~446,200 | ~3,987 | 99.12% | Correct chain head at slot boundary |
Head vote accuracy is the hardest duty, with ~210x more misses than source. It requires the node to see the latest block before the slot boundary, making it the most sensitive to processing speed.
\nOver 48 hours, head vote accuracy shows a clear cyclic pattern:
\n![\"Head](\"https://docs.stereumlabs.com/assets/images/head_vote_48h-d6de3647341b7f10476221c7db2be58d.png\")
The rate fluctuates between 98.64% and 99.54%, likely driven by network congestion patterns. All 5 nodes report identical values at each time point, confirming this reflects on-chain truth, not individual node performance.
\nOver the monitoring period, the nodes observed 27 block proposals from the 1,000 monitored validators via gossip. In the last 48 hours, 12 block proposals were detected via the validator_monitor_block_hit_total counter. All proposals arrived via the gossip network, confirming the validators are actively proposing on-chain.
One of the practical takeaways from this analysis: Nimbus' consensus client logs are a valuable observability layer for block building, regardless of how detailed the execution client's own logs are.
\nNimbus logs three key events per block building cycle:
\nRequesting engine payload — timestamp, slot, beacon head, fee recipientReceived engine payload — full payload contents: gas_used, gas_limit, block_number, extra_dataBlock proposal included — slot, validator IDFor Ethrex and Nethermind, whose default log levels don't expose per-iteration block building details, the Nimbus CC logs were the only way to determine that both clients are strong builders (17-22M gas avg). This CC-side perspective also revealed the build latency data that showed Erigon's problem is not response time but empty transaction pools.
\nAn interesting infrastructure finding: the default log levels produce vastly different volumes across execution clients.
\n![\"EC](\"https://docs.stereumlabs.com/assets/images/log_verbosity-70585912782b01de617fcc4171b9f731.png\")
| Execution Client | Log entries per hour | Factor vs. least |
|---|---|---|
| Nethermind 1.36.2 | 5,661 | 9.4x |
| Ethrex 9.0.0 | 1,569 | 2.6x |
| Geth v1.17.2 | 1,205 | 2.0x |
| Reth v1.11.3 | 838 | 1.4x |
| Erigon v3.3.10 | 781 | 1.3x |
| Besu 26.2.0 | 604 | 1.0x |
Nethermind produces ~9x more log output than Besu at default log levels. All 6 ECs in our fleet (including Reth, which is syncing) are included here since this is a general infrastructure observation.
\nIf you're running Nethermind with centralized logging (Filebeat/Elasticsearch or similar), account for its higher log volume when sizing your log storage and ingestion capacity. Consider adjusting Nethermind's log level if storage is constrained.
| Dimension | Besu | Erigon | Ethrex | Geth | Nethermind |
|---|---|---|---|---|---|
| Block building (avg gas) | 🟢 23.5M (best) | 🔴 0M (empty) | 🟢 21.8M | ⚪ 15.0M | 🟢 21.2M |
| Build latency | 546ms | 479ms (fastest) | 535ms | 524ms | 519ms |
| Attestation delay (agg.) | ⚪ 142ms | 🔴 153ms | ⚪ 141ms | 🟢 135ms | ⚪ 145ms |
| Attestation volume | ⚪ -0.6% | 🔴 -11.3% | ⚪ -1.0% | 🟢 Baseline | ⚪ -0.3% |
| EC build log detail | 🟢 Full | ⚪ txs + gas | 🟠 None (CC fills gap) | 🟢 Full | 🟠 Minimal (CC fills gap) |
| Log verbosity (default) | 🟢 604/hr | ⚪ 781/hr | ⚪ 1,569/hr | ⚪ 1,205/hr | 🟠 5,661/hr |
Erigon's block building is severely impaired. 76% of built blocks were empty, and even when it did include transactions, the count was a fraction of what other clients achieved. Paradoxically, Erigon responds fastest (479ms) but with nothing in the payload. The root cause is its slow block execution (423-567ms) cascading into transaction pool readiness.
\nBesu, Ethrex, and Nethermind are all strong block builders. Besu leads on raw gas output (23.5M avg) and iteration count (661 per block). Ethrex is the most consistent (14-18M gas for standard blocks, 60M for full blocks). Nethermind is competitive at 21.2M avg despite minimal logging.
\nNimbus' CC logs are a valuable observability source. The Received engine payload log provides payload gas_used, build latency, and block details for every EC, making it the most reliable cross-client comparison tool, especially for ECs like Ethrex and Nethermind whose own logs lack block building detail at default levels.
Erigon misses ~11% of attestation observations due to slower block processing. This volume gap is unique to Erigon; the other 4 ECs are within 1% of each other.
\nHead vote accuracy (~99.1%) shows a diurnal pattern visible across all nodes identically, driven by network congestion rather than client behavior.
\nLog volume varies 9x between the most and least verbose EC at default log levels. Nethermind's high verbosity (5,661/hr) is an infrastructure sizing consideration.
\nAll Prometheus data sourced from the prometheus-cold datasource in the StereumLabs Grafana instance. Validator monitor metrics use the validator_monitor_* family with cc_client=\"nimbus\" and validator=\"total\" label selectors. 48-hour data uses increase(...[48h]) instant queries and histogram_quantile over rate(...[48h]) for delay distributions.
EC container logs are stored in Elasticsearch (Filebeat 9.3.0 → Elasticsearch 9.3.0) and queried via the container.image.name field to isolate each execution client's output. Block building logs were identified by client-specific patterns: Besu's New proposal for payloadId, Geth's Updated payload/Starting work on payload/Stopping work on payload, and Erigon's Built block/Building block.
Additionally, Nimbus CC container logs (statusim/nimbus-eth2:multiarch-v26.3.1) were analyzed for engine API interactions: Requesting engine payload (build request timestamps), Received engine payload (payload contents including gas_used), and Block proposal included (on-chain confirmation). This CC-side perspective provides build latency measurements and fills in payload details for ECs whose own logs lack that information at default log levels (notably Ethrex and Nethermind).
All nodes run on NDC2 bare-metal hardware (Vienna), eliminating cloud-induced variance.
\nFor details on our label conventions and how to build your own dashboards against our data, see Build your own dashboards.
", "url": "https://docs.stereumlabs.com/blog/nimbus-v26-3-1-validator-monitoring-block-building", "title": "Nimbus v26.3.1: Validator monitoring and block building across 5 execution clients", "summary": "A deep dive into how Nimbus observes 1,000 validators and how Besu, Erigon, Ethrex, Geth, and Nethermind behave when building blocks, processing attestations, and handling the Engine API, measured on our NDC2 bare-metal fleet over 48 hours.", "date_modified": "2026-04-15T00:00:00.000Z", "author": { "name": "Stefan Kobrc" }, "tags": [ "Nimbus", "consensus client", "block building", "validator monitoring", "Erigon", "Geth", "Besu", "Nethermind", "Ethrex", "execution client", "PeerDAS", "performance" ] }, { "id": "https://docs.stereumlabs.com/blog/teku-version-25-12-0-26-2-0-26-3-0-comparison-resources", "content_html": "A deep dive into how Teku evolved across three releases: the RocksDB migration in 26.2.0, jemalloc in 26.3.0, and what both changes mean for CPU, memory, GC overhead, disk I/O, and block import latency on real hardware.
\n![\"Teku](\"https://docs.stereumlabs.com/assets/images/teku-version-comparison-thumbnail-bf60fe808551255aa87f575a8ee8d59e.png\")
We compared three Teku consensus client versions on our NDC2 bare-metal fleet in Vienna, each paired with all 6 execution clients (Besu, Erigon, Ethrex, Geth, Nethermind, Reth). Each version was measured over a 14-day window during its active deployment period using avg_over_time(...[14d:1h]) instant queries against our Prometheus-cold datasource.
| Version | Release Date | Measurement Window | Key Changes |
|---|---|---|---|
| 25.12.0 | Dec 16, 2025 | Jan 15 – Jan 29, 2026 | Late block reorg, block building prep, sidecar recovery |
| 26.2.0 | Feb 11, 2026 | Feb 15 – Mar 1, 2026 | RocksDB as default DB, DAS backfiller, getBlobs API |
| 26.3.0 | Mar 5, 2026 | Mar 15 – Mar 29, 2026 | jemalloc allocator, SSZ serialization fix, partial sidecar import |
The full report is available as a PDF download at the bottom of this post.
\nThe two architectural changes — LevelDB → RocksDB in 26.2.0 and the jemalloc memory allocator in 26.3.0 — drove the majority of the performance shifts:
\n![\"Fleet-wide](\"https://docs.stereumlabs.com/assets/images/headline_kpis-9393ae7df83cfe2a3df56133aea45e52.png\")
| Metric | 25.12.0 → 26.3.0 | What happened |
|---|---|---|
| CPU usage | −44% | RocksDB caching + jemalloc reducing GC-driven CPU spikes |
| GC overhead | −36% | jemalloc reduced heap fragmentation → fewer full GC cycles |
| Disk reads (host) | −79% | RocksDB block cache eliminates most disk reads |
| Disk writes (host) | −61% | LSM-tree architecture writes more efficiently than LevelDB |
| Block import delay | −24% | Fastest average: 297ms in 26.3.0 |
| RSS memory | +2.8% | RocksDB uses more in-memory structures (peaked at +5.5% in 26.2.0, recovered) |
| Open file descriptors | +101% | Expected: RocksDB holds many SST files open |
| Peer count (libp2p) | +9% | Steady improvement across versions |
Metric: rate(process_cpu_seconds_total{job=\"teku\"}[5m]) — Teku JVM process CPU in CPU-seconds per second.
![\"Teku](\"https://docs.stereumlabs.com/assets/images/cpu_usage-78a75dc4aff4ca7fc3172ca116acf05d.png\")
| EC Pairing | 25.12.0 | 26.2.0 | 26.3.0 | Δ 25.12→26.2 | Δ 26.2→26.3 |
|---|---|---|---|---|---|
| Besu | 0.761 | 0.441 | 0.431 | 🟢 −42% | 🟢 −2% |
| Erigon | 0.704 | 0.414 | 0.409 | 🟢 −41% | 🟢 −1% |
| Ethrex | 0.594 | 0.552 | 0.419 | 🟢 −7% | 🟢 −24% |
| Geth | 0.771 | 0.553 | 0.429 | 🟢 −28% | 🟢 −22% |
| Nethermind | 0.720 | 0.330 | 0.305 | 🟢 −54% | 🟢 −8% |
| Reth | 0.722 | 0.364 | 0.377 | 🟢 −50% | 🟠 +4% |
| Fleet Average | 0.712 | 0.442 | 0.395 | 🟢 −38% | 🟢 −11% |
CPU dropped 38% from 25.12.0 to 26.2.0. RocksDB's block cache and bloom filters reduce per-lookup processing compared to LevelDB. Version 26.3.0 added another 11% reduction through jemalloc's more efficient allocation patterns reducing GC-driven CPU spikes. The Nethermind pairing consistently shows the lowest Teku CPU usage across all three versions.
\n![\"RSS](\"https://docs.stereumlabs.com/assets/images/memory-1b180e20a83db4b2dca15b8c53d99d6b.png\")
Metric: process_resident_memory_bytes{job=\"teku\"} — total physical memory consumed by the Teku JVM process.
| EC Pairing | 25.12.0 (GB) | 26.2.0 (GB) | 26.3.0 (GB) | Δ 25.12→26.3 |
|---|---|---|---|---|
| Besu | 8.67 | 9.43 | 9.20 | 🟠 +6.2% |
| Erigon | 9.10 | 9.31 | 9.16 | ⚪ +0.6% |
| Ethrex | 8.74 | 9.48 | 9.20 | 🟠 +5.3% |
| Geth | 9.03 | 9.50 | 9.10 | ⚪ +0.7% |
| Nethermind | 9.07 | 9.34 | 9.13 | ⚪ +0.6% |
| Reth | 8.90 | 9.39 | 9.20 | 🟠 +3.4% |
| Fleet Average | 8.92 | 9.41 | 9.16 | 🟠 +2.8% |
RSS peaked in 26.2.0 (+5.5% vs 25.12.0), consistent with RocksDB's larger in-memory structures (block cache, memtables, bloom filters). Version 26.3.0 clawed back roughly half through jemalloc's reduced memory fragmentation, leaving a net +2.8%.
\nMetric: jvm_memory_used_bytes{area=\"heap\"} — Java heap utilization.
| EC Pairing | 25.12.0 (GB) | 26.2.0 (GB) | 26.3.0 (GB) | Δ 25.12→26.3 |
|---|---|---|---|---|
| Besu | 5.33 | 5.73 | 5.99 | 🟠 +12.3% |
| Erigon | 5.76 | 5.70 | 5.82 | ⚪ +1.0% |
| Ethrex | 5.89 | 5.77 | 6.05 | 🟠 +2.7% |
| Geth | 5.38 | 6.14 | 6.05 | 🟠 +12.3% |
| Nethermind | 5.36 | 6.01 | 6.16 | 🟠 +15.0% |
| Reth | 5.32 | 5.78 | 6.15 | 🟠 +15.5% |
| Fleet Average | 5.51 | 5.85 | 6.04 | 🟠 +9.6% |
Heap grew steadily, reflecting increased in-heap state caching from the RocksDB JNI bridge and the natural growth of Ethereum's state tree. All values remain well within Teku's default 8 GB max heap.
\n| Version | Fleet Avg (MB) | Delta |
|---|---|---|
| 25.12.0 | 705 | — |
| 26.2.0 | 739 | 🟠 +4.8% |
| 26.3.0 | 736 | ⚪ −0.4% |
Native (off-heap) memory rose modestly with RocksDB and stabilized with jemalloc.
\nMetric: rate(jvm_gc_collection_seconds_sum[5m]) — fraction of time spent in GC per second. This is a critical Teku-specific metric because GC pauses directly impact block processing latency.
![\"GC](\"https://docs.stereumlabs.com/assets/images/gc_overhead-bcbfc290a973ec34413d7880d0ed45be.png\")
| EC Pairing | 25.12.0 (ms/s) | 26.2.0 (ms/s) | 26.3.0 (ms/s) | Δ 25.12→26.3 |
|---|---|---|---|---|
| Besu | 2.77 | 2.82 | 1.90 | 🟢 −31% |
| Erigon | 1.59 | 3.01 | 2.29 | 🔴 +44% |
| Ethrex | 2.85 | 3.87 | 1.58 | 🟢 −45% |
| Geth | 3.29 | 3.01 | 1.60 | 🟢 −51% |
| Nethermind | 3.15 | 2.14 | 1.73 | 🟢 −45% |
| Reth | 2.83 | 2.34 | 1.47 | 🟢 −48% |
| Fleet Average | 2.75 | 2.87 | 1.76 | 🟢 −36% |
GC overhead was flat from 25.12.0 to 26.2.0. The introduction of jemalloc in 26.3.0 is the standout: fleet-wide GC time dropped 36%. jemalloc reduces heap fragmentation, meaning fewer large-object promotions to old gen and fewer full GC cycles. For a Java client like Teku, this matters directly — GC pauses are a primary contributor to block import latency.
\nThe Erigon pairing shows an anomalous GC increase across all versions. This may reflect interaction effects with Erigon's execution response patterns rather than a Teku-internal issue.
Metrics: rate(node_disk_read_bytes_total[5m]) and rate(node_disk_written_bytes_total[5m]) — host-level metrics that include both Teku CC and its paired EC. Since EC versions didn't change across measurement windows, deltas are attributable to the Teku version change.
![\"Disk](\"https://docs.stereumlabs.com/assets/images/disk_io-40fbd5ce837b8511fb1e0e53aee49c3f.png\")
| EC Pairing | Read 25.12 | Read 26.2 | Read 26.3 | Write 25.12 | Write 26.2 | Write 26.3 |
|---|---|---|---|---|---|---|
| Besu | 2,625 | 483 | 390 | 6,040 | 1,305 | 1,272 |
| Erigon | 424 | 470 | 350 | 1,141 | 1,282 | 1,294 |
| Ethrex | 1,127 | 603 | 392 | 2,143 | 1,888 | 1,303 |
| Geth | 1,528 | 710 | 259 | 3,172 | 2,003 | 1,227 |
| Nethermind | 1,489 | 220 | 186 | 2,583 | 1,145 | 1,304 |
| Reth | 1,916 | 326 | 283 | 4,468 | 1,243 | 1,220 |
| Fleet Avg | 1,518 | 469 | 310 | 3,258 | 1,478 | 1,270 |
All values in KB/s.
\nRead throughput dropped 79% (1,518 → 310 KB/s) and write throughput fell 61% (3,258 → 1,270 KB/s). RocksDB's block cache and SST-based design is far more read-efficient than LevelDB. The Besu pairing saw the most dramatic read improvement (−85%).
\nThese metrics are only available for versions using RocksDB. Version 25.12.0 ran LevelDB which does not expose equivalent counters.
\n| Metric | 26.2.0 Fleet Avg | 26.3.0 Fleet Avg | Delta |
|---|---|---|---|
storage_bytes_read rate (KB/s) | 84.3 | 103.6 | 🟠 +23% |
storage_bytes_written rate (MB/s) | 1.53 | 1.35 | 🟢 −12% |
storage_compact_write_bytes rate (KB/s) | 543.5 | 484.0 | 🟢 −11% |
Write amplification improved in 26.3.0: both raw write rate and compaction writes decreased ~11%. The slight read increase likely reflects the partial sidecar import feature doing more background reads.
\nMetric: beacon_block_import_delay_latest — most recent block import delay in milliseconds. This captures end-to-end latency from receiving a block to completing its import into the beacon state.
![\"Block](\"https://docs.stereumlabs.com/assets/images/block_import_delay-69329f9ae50f3237c014138e14594f80.png\")
| EC Pairing | 25.12.0 (ms) | 26.2.0 (ms) | 26.3.0 (ms) | Δ 25.12→26.3 |
|---|---|---|---|---|
| Besu | 268 | 482 | 221 | 🟢 −18% |
| Erigon | 519 | 490 | 347 | 🟢 −33% |
| Ethrex | 772 | 303 | 209 | 🟢 −73% |
| Geth | 251 | 319 | 209 | 🟢 −17% |
| Nethermind | 239 | 513 | 444 | 🔴 +86% |
| Reth | 284 | 426 | 352 | 🔴 +24% |
| Fleet Average | 389 | 422 | 297 | 🟢 −24% |
Version 26.2.0 was slightly slower fleet-wide — likely early-stage RocksDB tuning and the concurrent DAS backfiller adding load. Version 26.3.0 brought a strong recovery, achieving the lowest fleet-wide latency at 297 ms. The Ethrex pairing improved most dramatically (−73%), while the Nethermind pairing shows persistently elevated import times that warrant EC-side investigation.
\nLower block import delay means attestations can be created sooner after a block arrives, directly improving validator effectiveness and reducing inclusion delay.
![\"Peer](\"https://docs.stereumlabs.com/assets/images/peers-1e4c26552d85845413d4355f1a142091.png\")
| Metric | 25.12.0 | 26.2.0 | 26.3.0 | Trend |
|---|---|---|---|---|
beacon_peer_count (fleet avg) | 44.9 | 45.9 | 49.5 | 🟢 +10% |
libp2p_peers (fleet avg) | 90.8 | 91.9 | 98.9 | 🟢 +9% |
discovery_live_nodes (fleet avg) | 160.7 | 168.4 | 176.4 | 🟢 +10% |
| Gossip rate (msg/s, fleet avg) | 6.23 | 5.99 | 7.85 | 🟢 +26% |
All peer metrics improved steadily. Discovery live nodes grew from ~161 to ~176, suggesting the Discv5 layer is maintaining more active ENR records. Gossip throughput peaked in 26.3.0 — consistent with faster block processing enabling quicker re-gossip.
\nMetric: rate(beacon_data_column_sidecar_processing_requests_total[5m]) — DAS workload rate.
![\"DAS](\"https://docs.stereumlabs.com/assets/images/das_processing-2b6facd135b0599c4d44633538eeed03.png\")
DAS processing rates show significant per-pairing variance. The zero values in 25.12.0 for Erigon/Ethrex reflect sync issues that resolved in later versions. Version 26.3.0 allows nodes with >50% custody requirements to begin importing blocks after downloading only 50% of sidecars — an architectural improvement that doesn't compromise data availability guarantees.
\n![\"Open](\"https://docs.stereumlabs.com/assets/images/file_descriptors-9a01c845ae303b6bf2da759e464720f2.png\")
| EC Pairing | 25.12.0 | 26.2.0 | 26.3.0 | Δ 25.12→26.3 |
|---|---|---|---|---|
| Besu | 480 | 957 | 1,329 | 🔴 +177% |
| Erigon | 780 | 931 | 1,375 | 🔴 +76% |
| Ethrex | 801 | 1,031 | 1,299 | 🔴 +62% |
| Geth | 643 | 939 | 1,319 | 🔴 +105% |
| Nethermind | 682 | 751 | 1,245 | 🔴 +83% |
| Reth | 527 | 787 | 1,279 | 🔴 +143% |
| Fleet Average | 652 | 899 | 1,308 | 🔴 +101% |
File descriptors doubled — the most visible trade-off of RocksDB. Its LSM-tree architecture holds many SST files open simultaneously for efficient reads. Growth continued from 26.2.0 to 26.3.0 as databases accumulated more SST files through compaction.
\nEnsure ulimit -n is set to at least 65536 on hosts running Teku 26.x. The default 1024 on many Linux distributions will cause failures. Docker containers should pass --ulimit nofile=65536:65536.
| Dimension | 25.12.0 | 26.2.0 | 26.3.0 |
|---|---|---|---|
| CPU Efficiency | Baseline | 🟢 Major improvement | 🟢 Best |
| Memory Footprint | 🟢 Lowest | 🟠 +5.5% | 🟠 +2.8% (recovering) |
| GC Overhead | Baseline | ⚪ Similar | 🟢 Best (−36%) |
| Disk I/O | 🔴 Highest | 🟢 Major improvement | 🟢 Best |
| Block Import Speed | Moderate | 🟠 Slight regression | 🟢 Best (297ms) |
| Peer Connectivity | Good | Good | 🟢 Best |
| File Descriptors | 🟢 Lowest | 🟠 +38% | 🔴 +101% (monitor) |
| Storage Backend | LevelDB | RocksDB | RocksDB + jemalloc |
| Stability | 🟢 Stable | 🟢 Stable | 🟢 Mandatory (SSZ fix) |
Upgrade to 26.3.0 is mandatory. Beyond the SSZ serialization bug fix, 26.3.0 delivers the best performance profile across nearly every dimension.
\nVerify file descriptor limits. With 26.3.0 averaging 1,308 open FDs (and likely higher under peak load), ensure systems allow at least 65,536 file descriptors.
\nMonitor RocksDB compaction. Add storage_compact_write_bytes and storage_bytes_written to your dashboards. Abnormal compaction spikes can indicate database health issues.
Consider DAS backfiller tuning. If 26.x nodes show elevated CPU during backfill, --Xp2p-reworked-sidecar-custody-sync-batch-size=1 can throttle the backfiller.
Watch the Nethermind pairing. Block import delays are persistently higher with Nethermind across 26.x versions — this warrants investigation on the EL side.
\nThe complete analysis with additional detail on JVM thread pools, executor queue depths, and RocksDB internal counters is available as a styled PDF:
\n\nAll data sourced from the prometheus-cold datasource (Org 6) in the StereumLabs Grafana instance. Query pattern: avg by (ec_client) (avg_over_time(metric{cc_client=\"teku\", cc_version=\"...\", role=\"cc\", job=\"teku\"}[14d:1h])) evaluated as instant queries at the end of each 14-day window. Rate metrics use rate(...[5m]) inside the subquery. Host-level metrics filtered with device!~\"lo|veth.*|docker.*|br.*\" to exclude virtual interfaces. All nodes run on NDC2 bare-metal (Vienna), eliminating cloud noise.
For details on our label conventions and how to build your own dashboards against our data, see Build your own dashboards.
", "url": "https://docs.stereumlabs.com/blog/teku-version-25-12-0-26-2-0-26-3-0-comparison-resources", "title": "Teku 25.12.0 vs 26.2.0 vs 26.3.0: Cross-version resource & performance analysis", "summary": "A comprehensive comparison of three Teku consensus client releases across CPU, memory, JVM garbage collection, disk I/O, block import latency, P2P networking, and PeerDAS metrics — measured on our NDC2 bare-metal fleet across all 6 execution client pairings.", "date_modified": "2026-04-08T00:00:00.000Z", "author": { "name": "Stefan Kobrc" }, "tags": [ "Teku", "consensus client", "version comparison", "RocksDB", "jemalloc", "PeerDAS", "Fulu", "performance" ] }, { "id": "https://docs.stereumlabs.com/blog/ethcc9-talk-recap-ai-observability-ethereum-staking", "content_html": "A recap of our EthCC[9] presentation in Cannes: what StereumLabs is, how we use AI on top of our monitoring data, and what Fusaka actually did to hardware across 36 client pairings.
\n![\"EthCC[9]](\"https://docs.stereumlabs.com/assets/images/ethcc9-talk-thumbnail-bc69eec266ddac6e9360f6f71e57071f.jpg\")
On April 2, 2026 we presented StereumLabs at EthCC[9] in Cannes. The talk covered what we've built, why AI on raw metrics alone produces useless output, and concrete Fusaka/PeerDAS runtime data from our bare-metal fleet.
\nThis post is a written companion to that talk. If you prefer watching, the recording is embedded below. The slide deck is available as a PDF download.
\nStereumLabs is our observability and analytics platform for Ethereum staking infrastructure. We run every relevant client combination on dedicated bare-metal hardware: 6 execution layer clients (Geth, Nethermind, Besu, Erigon, Reth, Ethrex), 6 consensus layer clients (Lighthouse, Prysm, Teku, Nimbus, Lodestar, Grandine), plus the standalone Erigon + Caplin pairing. That's 37 combinations, monitored 24/7 with 90-day rolling metrics.
\nAll nodes run on isolated bare metal. No shared cloud instances, no noisy-neighbor effects. When we measure performance differences between clients, the data is reproducible and directly comparable.
\nThe platform provides 20+ dashboards covering resource consumption (CPU, RAM, disk, network), client-specific metrics (attestation rates, block processing times, peer counts, GC behavior), and system logs. Client development teams already have free access to the dashboards. The project is supported by an Ethereum Foundation grant.
\nBut dashboards have limits. And that's where AI comes in.
\nWe built an AI chatbot connected directly to our full monitoring dataset. Instead of navigating dashboards and writing queries, users ask questions in plain English:
\nThe result is a structured analysis with actual numbers, across all EL pairings, in seconds.
\nHere's what most people get wrong about AI in infrastructure monitoring: the model alone doesn't produce useful results. If you point a language model at raw Prometheus metrics, it doesn't know which queries to run, what normal ranges look like, or how to interpret differences between client architectures.
\nThat's why we've built a continuously evolving Instruction Set. It encodes which metrics matter for which client combination, what normal ranges look like per pairing, how to interpret architectural differences (Go vs. Java GC behavior, Rust memory models), and which queries to run in which order to build a meaningful analysis.
\nWithout the Instruction Set, the AI produces generic answers. With it, it produces the kind of analysis that would take an experienced engineer hours to assemble manually. We expand it continuously as we encounter new patterns, new client versions, and new edge cases. It's built from years of running these clients professionally.
\nWhen the Prysm team shipped v7.1.2, we asked the chatbot one question and got a full resource impact analysis across all 6 EL pairings:
\n| Metric | Result |
|---|---|
| Memory (RSS) | Dropped 5.1% on average. Biggest improvement with Geth pairing (-8.8%) |
| Block processing | Improved 25% overall. Erigon pairing went from 403ms to 90ms (-78%) |
| Peer count | Stable at ~71 across both versions. No regression |
| CPU | Mixed results, EL-dependent. Reth dropped 21%, Besu increased 28% |
This analysis would normally take hours of manual work. The chatbot produced it from a single question. The full report is published on our blog: Prysm v7.1.1 & 7.1.2 resources.
\nThe chatbot is great for proactive analysis. But what about when things go wrong at 3am? That's where AI Alerting comes in.
\nStage 1: Near-real-time threshold alerts. Monitors hard thresholds like attestation rate, disk usage, peer count, and missed blocks. Fires within seconds. No AI inference delay, no additional cost. If the AI layer is slow or unavailable, the basic alert still arrives.
\nStage 2: AI root-cause analysis. When a threshold alert fires, a webhook triggers the AI. It pulls relevant metrics and logs, correlates the data against our neutral baseline from all 37 client combinations, and delivers a root-cause analysis with actionable next steps. Delivered in 5 to 15 seconds.
\nThe result: operators don't just get \"attestation rate dropped below 95%.\" They get: \"Your attestation rate dropped because Geth's peer count fell to 3, likely due to a network partition. Your Prysm instance is healthy. Recommended action: check firewall rules and restart the EL client.\"
\nWhat makes our alerting especially useful is the neutral baseline dataset from our own fleet. When an alert fires, the AI automatically compares against data from all 37 client combinations.
\nEvery alert answers three questions: Is this happening across the Ethereum network right now? Is it specific to this client version? Or is it unique to your local environment?
\nThat distinction between \"the whole network is seeing elevated block processing times after a fork\" and \"your Geth instance is the only one with this problem\" is the difference between waiting it out and taking immediate action.
\nThe same two-stage architecture applies to security events:
\nTraditional monitoring tells you \"Geth restarted.\" Our AI layer tells you \"Geth restarted, fee recipient address changed from 0xABC to 0xDEF, this was not initiated through the operator's usual deployment pipeline, severity: critical.\"
\nFor operators staking millions in ETH, the difference between detecting a compromised reward address in minutes versus days is the difference between a security incident and a financial disaster.
\nA significant portion of the talk covered our Fusaka hardfork measurements. We compared two 14-day windows (before and after the December 3, 2025 activation) across all 36 non-supernode client pairings.
\nThe fleet-level headline numbers:
\n| Metric | Change | What happened |
|---|---|---|
| Network RX | -60% | PeerDAS in action: nodes sample slices instead of downloading full blobs |
| CPU | +30% | Expected trade-off: sampling routines cost compute |
| Memory | -8% | Less blob data held in RAM |
| Disk reads | -53% | Fewer full-blob fetches from disk |
Notable client outliers: Nimbus CPU jumped +257% (most compute-intensive PeerDAS implementation), Lighthouse was the only consensus client to reduce CPU (-13%), and Besu saw the largest memory drop (-35%).
\nThe worst pairing post-fork: Nimbus + Reth at 16.78% CPU.
\nThe full analysis with per-client breakdowns, heatmaps, daily trends, and PromQL queries is available as a dedicated blog post: Fusaka hardfork: hardware impact on non-supernodes.
\nOne thing we hear constantly from professional operators: \"I'm interested, but I can't send my metrics to your cloud.\" That's why StereumLabs supports multiple deployment options:
\nThe key message: your infrastructure data doesn't become someone else's competitive intelligence.
\n| Component | Status |
|---|---|
| Dashboards (20+) | Live, all 37 client combinations |
| AI Chatbot | Working proof of concept against live data |
| AI Alerting | Q2 2026 |
| Security Monitoring | Q2 2026 |
| On-premise deployment | Ready |
We're looking for node operators who want to try the AI chatbot, client development teams interested in automated cross-EL impact analysis, and staking protocols looking for monitoring and security standards across their operator ecosystem.
\nReach out at contact@stereumlabs.com or visit stereumlabs.com.
\nThe full slide deck from the talk is available for download:
\n", "url": "https://docs.stereumlabs.com/blog/ethcc9-talk-recap-ai-observability-ethereum-staking", "title": "EthCC[9] talk recap: AI-powered observability for Ethereum staking", "summary": "A recap of our EthCC[9] presentation in Cannes covering StereumLabs, AI-powered infrastructure monitoring, Fusaka/PeerDAS runtime metrics, and what we're building next.", "date_modified": "2026-04-02T00:00:00.000Z", "author": { "name": "Stefan Kobrc" }, "tags": [ "AI", "EthCC", "talk", "observability", "Fusaka", "PeerDAS", "alerting", "security" ] }, { "id": "https://docs.stereumlabs.com/blog/fusaka-hardfork-hardware-impact-non-supernodes", "content_html": "We measured CPU, memory, disk I/O, and network across all 36 CC×EC pairings on our non-supernode fleet — here's what the Fusaka hardfork actually did to hardware consumption.
\nEthereum's Fusaka hardfork activated on December 3, 2025 at 21:49 UTC (slot 13,164,544). It was the second hard fork of 2025 after Pectra (May 2025) and arguably the most consequential upgrade since the Merge. The name combines \"Fulu\" (consensus layer, named after a star) and \"Osaka\" (execution layer, named after the host city of Devcon 2025).
\nThe headline feature is PeerDAS (Peer Data Availability Sampling, EIP-7594) — a fundamental change in how Ethereum verifies blob data. Instead of every node downloading and verifying every blob, nodes now only need to sample small slices, verifying that the full data exists without actually possessing all of it. Vitalik Buterin called it \"literally sharding\" — Ethereum reaching consensus on blocks without requiring any single node to see more than a tiny fraction of the data.
\nBeyond PeerDAS, Fusaka shipped approximately 12 additional EIPs:
\nTwo scheduled BPO forks followed:
\nWe wanted to know: what did all this actually do to the hardware running our nodes?
\nStereumLabs runs approximately 90 hosts split across GCP cloud instances and NDC2 bare-metal nodes in Vienna, Austria. Each non-supernode host runs one consensus client (CC) paired with one execution client (EC), covering all 36 possible combinations of:
\nAll metrics come from our prometheus-cold datasource (UID aez9ck4wz05q8e, Org 6), which covers all available metrics without retention or delay restrictions. Node-level system metrics are collected via prometheus-node-exporter at a 15-second scrape interval.
We compared two 14-day windows:
\n| Period | Range | Label |
|---|---|---|
| Before Fusaka | November 19 – December 3, 2025 | Pre-fork baseline |
| After Fusaka | December 4 – December 18, 2025 | Post-fork + BPO-1 |
Both windows use avg_over_time(...[14d:1h]) — a 14-day average at 1-hour subquery resolution, computed as an instant query at the boundary timestamp. This smooths out transient spikes while preserving meaningful shifts.
Supernodes are excluded via the label filter cc_client!~\".*-super\". Only instances with role=~\"cc|ec\" are included. Network metrics exclude loopback and virtual interfaces (device!~\"lo|veth.*|docker.*|br.*\").
Here's what happened across the entire non-supernode fleet:
\n| Metric | Before | After | Change |
|---|---|---|---|
| CPU utilization (avg) | 4.66% | 6.08% | +30% |
| Memory used (avg) | 6.06 GiB | 5.59 GiB | −8% |
| Network RX (avg) | 2.20 MiB/s | 0.87 MiB/s | −60% |
| Network TX (avg) | 0.20 MiB/s | 0.18 MiB/s | −13% |
| Disk read rate (avg) | 2.58 MiB/s | 1.21 MiB/s | −53% |
| Disk write rate (avg) | 1.73 MiB/s | 1.92 MiB/s | +11% |
![\"Fleet-level](\"https://docs.stereumlabs.com/assets/images/fleet_summary-ce362e657d05051e0087cfade23c76f3.png\")
PeerDAS delivered exactly what it promised: dramatically less network bandwidth and lower memory at the cost of modestly higher CPU. Disk reads halved. Operators paying per-GB egress on cloud providers should see meaningful cost savings.
Let's dig into each metric.
\nThe most striking result is the 60% drop in network receive bandwidth — from 2.20 MiB/s to 0.87 MiB/s on average. This is PeerDAS in action: nodes now verify only sampled slices of blob data rather than downloading entire blobs.
\nLooking at the daily time series, the decline actually began before the fork itself — around November 28–29 — suggesting some nodes started running fork-compatible client versions with PeerDAS-like optimizations ahead of the December 3 activation. On the fork day itself, the fleet-average RX was already down to 0.26 MiB/s.
\nThe post-fork trend shows a brief recovery as nodes settled into the new protocol, stabilizing around 1.4 MiB/s by mid-December — still a 36% reduction from the pre-decline baseline of ~2.2 MiB/s.
\n![\"Fleet](\"https://docs.stereumlabs.com/assets/images/network_rx_trend-24db51516409121ff6c6be59af4f38ff.png\")
| Date | RX (MiB/s) | Event |
|---|---|---|
| Nov 19 | 3.04 | |
| Nov 28 | 2.38 | Pre-fork decline begins |
| Dec 3 | 0.26 | Fusaka fork |
| Dec 4 | 1.09 | Post-fork stabilization |
| Dec 10 | 0.59 | BPO-1 |
| Dec 18 | 1.39 | Settled baseline |
Network transmit bandwidth also decreased, though more modestly — from 0.20 to 0.18 MiB/s (−13%).
\nFor GCP instances, network egress costs $0.085–$0.12/GB depending on region and volume. A node running at 2.2 MiB/s averages ~5.7 TB/month of inbound traffic. At the new 0.87 MiB/s rate, that drops to ~2.2 TB/month — a potential saving of $300–$400/month per node on egress alone, depending on provider and plan.
Fleet-average CPU rose from 4.66% to 6.08% — a 30% relative increase, but in absolute terms still very modest. This is the expected trade-off: PeerDAS introduces data availability sampling routines (compute) in exchange for reduced data transfer (bandwidth).
\nThe daily time series reveals two notable spikes:
\nAfter settling, the new baseline sits around 5.5–5.7% — roughly 1 percentage point above pre-fork levels.
\n![\"Fleet](\"https://docs.stereumlabs.com/assets/images/cpu_mem_trend-c292ba84c771acfd2e501aae1c3e59ff.png\")
Not all consensus clients handled Fusaka equally:
\n| CC client | Before | After | Change |
|---|---|---|---|
| Grandine | 5.84% | 7.06% | +21% |
| Lighthouse | 3.81% | 3.30% | −13% |
| Lodestar | 5.80% | 5.85% | +1% |
| Nimbus | 2.63% | 9.40% | +257% |
| Prysm | 4.38% | 4.86% | +11% |
| Teku | 5.66% | 5.96% | +5% |
![\"CPU](\"https://docs.stereumlabs.com/assets/images/cc_cpu-368f9f6d056fb8f610c6aedcc6e3defc.png\")
Nimbus stands out dramatically with a +257% CPU increase (2.63% → 9.40%). This suggests its PeerDAS implementation is currently more compute-intensive than competitors. The nimbus + reth pairing hit 16.78% post-fork — the highest of any combination in the fleet. Nimbus operators should monitor their CPU headroom closely, especially on resource-constrained setups.
On the positive side, Lighthouse was the only consensus client to decrease CPU usage (−13%), suggesting either an efficient PeerDAS implementation or concurrent optimizations shipped in its fork-compatible release.
\nLodestar remained essentially flat (+1%), and Prysm (+11%), Teku (+5%), and Grandine (+21%) showed moderate increases — all within comfortable bounds for typical node hardware.
\nExecution clients saw a more uniform increase, consistent with the raised gas limit (EIP-7935) and new EVM opcodes (EOF):
\n| EC client | Before | After | Change |
|---|---|---|---|
| Besu | 2.58% | 5.19% | +101% |
| Erigon | 3.62% | 5.83% | +61% |
| Ethrex | 4.07% | 6.04% | +48% |
| Geth | 3.99% | 6.53% | +64% |
| Nethermind | 6.83% | 5.08% | −26% |
| Reth | 5.56% | 7.89% | +42% |
![\"CPU](\"https://docs.stereumlabs.com/assets/images/ec_cpu-7d2dfdd6d657c0fe0d5ed88e1ab25b53.png\")
Besu (+101%) and Geth (+64%) saw the largest relative increases. For Besu, the doubling (from a very low 2.58% baseline) likely reflects the new gas limit activating code paths that were previously idle. Geth's increase is consistent with heavier block processing under the raised 60M gas ceiling.
\nNethermind bucked the trend entirely with a −26% decrease — a surprising result that may indicate a coincidental version update with CPU optimizations during the measurement window.
\nThis heatmap shows the percentage change in CPU utilization for every CC×EC combination. Red means higher CPU after Fusaka, green means lower:
\n![\"CPU](\"https://docs.stereumlabs.com/assets/images/cpu_heatmap-74964bdbd01959392a57e2849ded1f7c.png\")
The Nimbus row is unmistakable — deep red across all EC pairings, with nimbus + ethrex showing an extreme +1060% (from 0.83% to 9.63%, though the low baseline suggests the pairing may have been partially offline pre-fork). The Nethermind column is notably green across most CC pairings, reinforcing that Nethermind itself saw a concurrent optimization.
Average memory usage dropped from 6.06 GiB to 5.59 GiB (−8%) across the fleet. This aligns perfectly with PeerDAS's design: nodes no longer hold entire blobs in memory, only the sampled slices they need for verification.
\nThe CPU + memory daily trend chart above also shows the memory trajectory (orange line). There's a sharp drop on December 4 (from 6.22 to 5.26 GiB) — the first full day after the fork — followed by a gradual recovery over the next two weeks as caches and new protocol state accumulated. By December 18, memory was back to 6.27 GiB, suggesting the initial drop was partly transient (e.g., cleared blob caches from the pre-fork protocol).
\nThe 14-day average still shows a net decrease because the early post-fork days pull the average down significantly.
\n| CC client | Before (GiB) | After (GiB) | Change |
|---|---|---|---|
| Grandine | 5.79 | 4.80 | −17% |
| Lighthouse | 5.41 | 5.42 | 0% |
| Lodestar | 7.97 | 6.82 | −14% |
| Nimbus | 4.89 | 3.91 | −20% |
| Prysm | 6.14 | 6.51 | +6% |
| Teku | 6.39 | 6.24 | −2% |
![\"Memory](\"https://docs.stereumlabs.com/assets/images/cc_mem-7499fe4250f171c71386bfd77d51b1c9.png\")
Nimbus (−20%) and Grandine (−17%) saw the largest memory reductions. Interestingly, Nimbus traded memory savings for CPU — a classic compute-vs-memory trade-off in its PeerDAS implementation.
\nPrysm was the only consensus client to increase memory (+6%), suggesting it keeps more sampling state in memory than peers. Given Prysm's CPU increase was moderate (+11%), this is a reasonable engineering choice.
\n| EC client | Before (GiB) | After (GiB) | Change |
|---|---|---|---|
| Besu | 6.59 | 4.31 | −35% |
| Erigon | 5.82 | 5.96 | +2% |
| Ethrex | 6.50 | 6.13 | −6% |
| Geth | 7.09 | 6.51 | −8% |
| Nethermind | 6.15 | 5.37 | −13% |
| Reth | 4.70 | 5.33 | +13% |
![\"Memory](\"https://docs.stereumlabs.com/assets/images/ec_mem-55a0874c0fb4d37f68a7f284b82e9d2b.png\")
Besu saw a dramatic −35% memory drop — from 6.59 to 4.31 GiB. This is the largest single-client change in the dataset and likely reflects aggressive cache invalidation during the fork transition. Reth moved in the opposite direction (+13%), possibly due to its database engine accumulating more state under the new protocol.
\nDisk reads dropped 53% — from 2.58 to 1.21 MiB/s. This is directly attributable to PeerDAS: nodes no longer need to retrieve full blobs from disk for verification. The reduction was visible across nearly all pairings.
\nSome notable per-pairing changes:
\nDisk writes increased 11% — from 1.73 to 1.92 MiB/s. The increase is modest and likely comes from:
\n![\"Disk](\"https://docs.stereumlabs.com/assets/images/disk_io-7ad5e3f1e9177cc2791083cb1a709622.png\")
This is a small overhead relative to the substantial read savings.
\nFor the detail-oriented, here are the complete CPU and memory matrices across all CC×EC combinations.
\n| CC \\ EC | Besu | Erigon | Ethrex | Geth | Nethermind | Reth |
|---|---|---|---|---|---|---|
| Grandine | 2.02 | 4.04 | 4.92 | 5.32 | 11.25 | 7.48 |
| Lighthouse | 2.39 | 3.20 | 2.74 | 3.65 | 5.20 | 4.43 |
| Lodestar | 2.32 | 3.75 | 9.62 | 3.66 | 7.68 | 7.73 |
| Nimbus | 1.41 | 2.42 | 0.83 | 2.40 | 5.32 | 3.40 |
| Prysm | 2.58 | 3.56 | 3.42 | 3.35 | 4.95 | 7.02 |
| Teku | 4.74 | 4.63 | 4.23 | 5.55 | 8.81 | 4.11 |
| CC \\ EC | Besu | Erigon | Ethrex | Geth | Nethermind | Reth |
|---|---|---|---|---|---|---|
| Grandine | 5.54 | 7.28 | 6.58 | 7.71 | 6.62 | 9.05 |
| Lighthouse | 1.28 | 1.89 | 3.87 | 4.64 | 3.98 | 4.13 |
| Lodestar | 8.88 | 5.69 | 5.42 | 5.37 | 3.46 | 6.27 |
| Nimbus | 6.57 | 8.21 | 9.63 | 8.46 | 6.77 | 16.78 |
| Prysm | 4.00 | 5.17 | 4.42 | 5.05 | 4.11 | 6.40 |
| Teku | 5.37 | 5.97 | 7.36 | 7.08 | 4.98 | 5.03 |
| CC \\ EC | Besu | Erigon | Ethrex | Geth | Nethermind | Reth |
|---|---|---|---|---|---|---|
| Grandine | 6.12 | 5.70 | 5.16 | 6.18 | 7.15 | 4.46 |
| Lighthouse | 6.15 | 5.01 | 6.52 | 6.73 | 4.92 | 4.40 |
| Lodestar | 7.27 | 6.64 | 11.33 | 7.76 | 7.35 | 7.46 |
| Nimbus | 5.66 | 5.21 | 2.94 | 6.42 | 5.94 | 3.17 |
| Prysm | 6.31 | 6.08 | 8.42 | 6.70 | 5.66 | 4.28 |
| Teku | 8.01 | 6.51 | 4.59 | 8.74 | 7.20 | 4.93 |
| CC \\ EC | Besu | Erigon | Ethrex | Geth | Nethermind | Reth |
|---|---|---|---|---|---|---|
| Grandine | 3.68 | 5.59 | 5.48 | 5.07 | 4.43 | 4.96 |
| Lighthouse | 2.48 | 3.57 | 7.36 | 7.10 | 5.58 | 6.41 |
| Lodestar | 5.61 | 7.30 | 7.05 | 8.21 | 5.45 | 7.32 |
| Nimbus | 2.24 | 5.74 | 3.29 | 4.73 | 4.97 | 2.47 |
| Prysm | 6.35 | 6.95 | 7.05 | 7.33 | 5.22 | 6.18 |
| Teku | 4.74 | 5.96 | 6.26 | 8.67 | 7.69 | 4.14 |
The +257% CPU increase on Nimbus deserves monitoring over subsequent weeks and versions. If this is a temporary bootstrapping cost (PeerDAS column sync, initial sampling table construction), it should decline. If it persists, Nimbus operators on lower-spec hardware may need to re-evaluate their headroom.
\nBPO-2 was scheduled for late December 2025 to early January 2026, raising blob parameters from 10/15 to 14/21. Our post-fork window (Dec 4–18) captures BPO-1 but not BPO-2. A follow-up analysis with a wider window would reveal whether the second parameter increase added further CPU or bandwidth pressure.
\nThe sharp initial memory drop post-fork appears to be partly transient — memory was trending back upward by December 18. We'll continue monitoring whether this recovery plateaus below the pre-fork baseline or converges back to original levels.
\nThe nimbus + ethrex pairing showed anomalously low CPU pre-fork (0.83%), which may indicate the pairing was partially offline or in a degraded state during the pre-fork window. The post-fork value of 9.63% is more consistent with fleet norms, suggesting the pairing recovered during or after the fork transition.
Client versions were not pinned across the fork boundary — some pairings may have undergone version upgrades alongside Fusaka. This means we cannot cleanly attribute all changes to the fork itself. In particular, Nethermind's −26% CPU decrease may reflect a version update rather than a Fusaka effect.
\nFusaka delivered on its core promise for non-supernode operators: dramatically lower bandwidth requirements (−60% receive, −13% transmit) and modestly lower memory (−8%), at the cost of a moderate CPU increase (+30%) that remains well within typical hardware headroom. Disk reads dropped by half.
\nFor operators managing hosting costs, the network savings alone are significant — especially on cloud providers where egress is metered. The CPU overhead is real but manageable: even the worst-case fleet average post-fork (6.08%) leaves substantial headroom on modern hardware.
\nThe main action item is monitoring Nimbus deployments for elevated CPU, and tracking BPO-2 effects as blob capacity continues to expand.
\nFor reproducibility, here are the exact queries run against prometheus-cold:
avg by (cc_client, ec_client) (
avg_over_time(
(1 - rate(node_cpu_seconds_total{mode=\"idle\", role=~\"cc|ec\", cc_client!~\".*-super\"}[1h]))
[14d:1h]
)
)
avg by (cc_client, ec_client) (
avg_over_time(
(node_memory_MemTotal_bytes{role=~\"cc|ec\", cc_client!~\".*-super\"}
- node_memory_MemAvailable_bytes{role=~\"cc|ec\", cc_client!~\".*-super\"})
[14d:1h]
)
) / 1024 / 1024 / 1024
avg by (cc_client, ec_client) (
avg_over_time(
rate(node_disk_read_bytes_total{role=~\"cc|ec\", cc_client!~\".*-super\"}[1h])
[14d:1h]
)
) / 1024 / 1024
avg by (cc_client, ec_client) (
avg_over_time(
rate(node_network_receive_bytes_total{role=~\"cc|ec\", cc_client!~\".*-super\",
device!~\"lo|veth.*|docker.*|br.*\"}[1h])
[14d:1h]
)
) / 1024 / 1024
Datasource: prometheus-cold (UID: aez9ck4wz05q8e, Org 6)
Before snapshot: instant query at 2025-12-03T00:00:00Z
After snapshot: instant query at 2025-12-18T00:00:00Z
Daily time series: range query from 2025-11-19T00:00:00Z to 2025-12-18T00:00:00Z, step = 86400s
Let's have a look at both versions of Prysm 7.1.1 and 7.1.2 and it's resource consumption
\nWe at StereumLabs run Prysm continuously across all six supported execution-layer clients — Besu, Erigon, Ethrex, Geth, Nethermind, and Reth — on isolated bare-metal nodes. When v7.1.2 landed, we pulled 90-day averages from our Prometheus-cold datasource to see exactly what changed. Here's the short version.
\nMemory is the clearest win. Process RSS dropped by 5.1% on average (3.36 → 3.19 GB), with the biggest gains when paired with Geth (−8.8%) and Besu (−7.8%). Heap in-use stayed flat, which points to the savings coming from outside the Go heap — likely reduced stack allocations or more efficient mmap regions.
\nBlock processing time also improved significantly — down 25% on average (142 → 107 ms). The headline number is dominated by the Erigon pairing, which went from 403 ms to 90 ms (−78%). For the production-grade trio of Geth, Nethermind, and Reth, results were stable to slightly better.
\nNetwork connectivity was unaffected. Average libp2p peer count held at ~71 across both versions.
\nCPU utilization showed a mixed picture that varies by EL pairing — Reth dropped 21%, while Besu increased 28%. Because we measure system-wide CPU across the full node, EL-side changes and PeerDAS load from the Fusaka hard fork both contribute noise here. No clear regression in Prysm itself.
\nGC pause duration ticked up marginally (+5.5%, from 0.735 to 0.775 ms) — well within acceptable bounds and unlikely to affect attestation or block proposal timelines.
\nv7.1.2 is a straightforward upgrade for production operators: lower memory footprint, faster block processing on the pairings that matter most, and no regressions in network behavior. If you're running Ethrex, treat its +138% block processing regression as an outlier specific to that experimental client, not a Prysm issue.
\n📄 Full report with per-client breakdowns and methodology: Download PDF
", "url": "https://docs.stereumlabs.com/blog/prysm-version-7-1-1-and-7-1-2-comparison-resources", "title": "Prysm v7.1.1 & 7.1.2 resources", "summary": "Let's have a look at both versions of Prysm 7.1.1 and 7.1.2 and it's resource consumption", "date_modified": "2026-03-25T00:00:00.000Z", "author": { "name": "Stefan Kobrc" }, "tags": [ "AI", "analysis", "prysm" ] } ] }