Navigating the Nethermind Execution Client Issue

January 22, 2024

TLDR

  • Figment does not run Nethermind as an execution client
  • Figment runs Geth as its primary execution client
  • Importantly Figment has the ability to switch execution clients in a matter of hours, should Geth experience a similar bug
  • Inactivity leak was NOT engaged in this event
  • It appears the participation rate dropped to about 91%, but then recovered
  • Nethermind team released a fix within a few hours (participation rate has recovered to around 99%)

What Happened?

On January 21st, the execution client Nethermind began experiencing some problems at block 19056922, or around 15:00 EST. It appears that Nethermind versions 1.23 to 1.25 began rejecting blocks as invalid. This means that anyone running a validator connected to a Nethermind node with those versions (1.23 to 1.25) would have had some missed attestations.

Reviewing participation figures on Beaconcha.in, it looks as if participation dropped to around 90%:

(Source)

Participation was below 95% between epochs 257,908 and 257,923, inclusive – a period of 16 epochs (about 2 hours).

Nethermind rolled out a fix within a few hours of the issue:

(Source)

 

The cause of the issue is still unclear, but it is likely the Nethermind team will release details soon.

Concerns over Geth

The bug in Nethermind has led some to question what would happen had the same problem occurred on a majority client like Geth.

In this case it is likely that any validators connected to Geth would fail to attest, and Ethereum would enter inactivity leak – a period of increasing penalties for failing to attest.

These penalties are small, but grow over time.

On a per validator basis, the penalties during inactivity leak are*

*Assumes an effective balance of 32 ETH.

What should Figment customers know?

Figment does run Geth as its primary execution client. That said, Figment has the capability to switch both its execution and consensus clients. All Figment validators could be switched from Geth to a different, minority client in between 30 minutes and a couple of hours.

Even during a period of inactivity leak, six hours would lead to a loss of 0.0026 ETH/validator (or $6.16 at a price of $2,369.80/ETH).

Figment takes the philosophy of Safety Over Liveness seriously; that is, the safety of our customers’ stake is our primary concern. We have built out infrastructure and redundancy strategy around this principle.

SHARE POST

Meet with us

Bring the Complete Staking Solution to Your Organization

Figment respects your privacy. By submitting this form, you are acknowledging that you have read and agree to our Privacy Policy, which details how we collect and use your information.