To ensure clarity and prevent any misinterpretation to the post below, linked here is an updated blog post around Figment’s execution layer strategies.

TLDR

• Figment does not run Nethermind as an execution client
• Figment runs Geth as its primary execution client
• Inactivity leak was NOT engaged in this event
• It appears the participation rate dropped to about 91%, but then recovered
• Nethermind team released a fix within a few hours (participation rate has recovered to around 99%)

What Happened?

On January 21st, the execution client Nethermind began experiencing some problems at block 19056922, or around 15:00 EST. It appears that Nethermind versions 1.23 to 1.25 began rejecting blocks as invalid. This means that anyone running a validator connected to a Nethermind node with those versions (1.23 to 1.25) would have had some missed attestations.

Reviewing participation figures on Beaconcha.in, it looks as if participation dropped to around 90%:

(Source)

Participation was below 95% between epochs 257,908 and 257,923, inclusive – a period of 16 epochs (about 2 hours).

Nethermind rolled out a fix within a few hours of the issue:

(Source)

The cause of the issue is still unclear, but it is likely the Nethermind team will release details soon.

Concerns over Geth

The bug in Nethermind has led some to question what would happen had the same problem occurred on a majority client like Geth.

Much depends on the nature of the bug. If the bug causes a failure to attest, Ethereum would enter inactivity leak – a period of increasing penalties for those validators failing to attest.

These penalties are small, but grow over time.

On a per validator basis, the penalties during inactivity leak are*

*Assumes an effective balance of 32 ETH.

In the event that the bug does not cause a failure to attest, but instead causes validators running the buggy EL client to fail to attest to new blocks on the chain, inactivity leak is not triggered, but the network’s liveness suffers, i.e., no new blocks are being added to the chain.

What should Figment customers know?

Figment is currently running Geth in some of its operations but is moving away from this execution layer client.

Figment takes the philosophy of Safety Over Liveness seriously; that is, the safety of our customers’ stake is our primary concern. We have built out infrastructure and redundancy strategy around this principle.

One implication of this philosophy is that should an event like the one described above occur, the response is not a panicked, knee-jerk reaction. It is an all-hands effort to first figure out what is happening. Both downtime penalties and even inactivity leak are acceptable over some period of time to properly assess the nature of the issue and determine a course of action.

Even during a period of inactivity leak, six hours would lead to a loss of 0.0026. This loss is acceptable to the alternative, which is to move quickly and potentially incur a much worse outcome.

It is this philosophy along with operating procedures and architecture that Figment has never suffered a slashable offense on Ethereum. 

*This article had previously stated facts about Figment’s Ethereum validator capabilities, especially the time it would take to change clients. The intent was to describe capabilities rather than to imply what Figment would do in a hypothetical scenario. In the end, the decision was made to remove this language to avoid confusion and misinterpretation.

Additionally, it was previously stated that it is likely inactivity leak would be triggered if a similar bug occurred on geth. That is not necessarily the case and the article has been changed to reflect that.

The information herein is being provided to you for general informational purposes only. It is not intended to be, nor should it be relied upon as, legal, business, tax or investment advice. Figment undertakes no obligation to update the information herein.