Solana Developers Highlight Mitigation Measures to Achieve Network Resilience

Solana suffered a gloomy weekend as the network went down for up to seven hours, leading to Sunday at 3:30 AM UTC when the validators were finally able to boot it back up. Markedly, this hitch wasn't an entirely unfamiliar one. It was the seventh reported incident of the Solana network suffering an outage. One would be justified to term it a recurrent issue rather than occasional.

Solana developers have since published a post-mortem report for the weekend events, in which it confirmed that the network was downed by massive bot activity that sent the transaction activity through the roof. The bots targeted the popular Solana NFT minting platform Candy Machine and were able to channel up to 100 Gbps of traffic to network nodes. Running out of memory to accommodate the data, the validators lost consensus, and the network halted.

Three steps towards stability and robustness

In response to the weekend troubles, Solana developers have set forth mechanisms they are implementing to make the network much more resilient. The first of the three major mitigations that Solana is executing is migration from the current User Datagram Protocol (UDP) to Google's quick UDP internet connection (QUIC). The Solana team faulted UDP's lack of flow control and acknowledgments as an inhibition to tackling errant behaviors.

The new protocol, QUIC, was developed by now Amazon executive Jim Roskind in 2012 as a general-purpose transport layer network protocol. It serves as an integration of the session and flow control characteristics of the transmission control protocol (TCP) and asynchronous communication akin to UDP's. Roskind's engineering result was a protocol able to handle latency-sensitive applications, on which Solana is now rebuilding its core protocols on, with the move expected to breed more options to optimize data intake.

Chasing robustness, the second measure Solana developers penned in the report is that the network envisions stake-weighted transaction processing. In addition to the fact that stake-weighted transaction processing would sync well with QUIC, the developers also believe replacing the current first-come-first-served basis would improve network bandwidth utilization.

Third, they spoke of the introduction of fee prioritization with the imminent v1.11 release. With fee-based execution priority, Solana users would be required to submit an additional fee atop the base fee. The final fee would be collected once a transaction has been successfully completed and settled in a block.

Censorship concerns

The Solana developers acknowledged that bot activity targeting NFT minting has been frequent this year and is guilty of congestion problems that have plagued the network since January.

Following the most recent network outage, head of communication Austin Federa had to quell censorship concerns over the revelation that validators had been instructed to block Candy Machine transactions at an attempted network restart to boost lock out the bots.

“Solana Foundation (later corrected to ‘multiple validator operator entities') published beta cluster restart instructions […] to block NFT minting bots manually at the L1 layer. That means multiple validator operators left the censorship portion in. Even if optional, Solana founders, Solana Foundation Twitter accounts, etc. retweeted these instructions, which is an endorsement of optional compiled instructions to censor,” the user, Hudson Jameson, argued.

In what turned out to be an exchange, Federa pointed out a similar incident in June 2016 where Ethereum stakeholders voted in favor of rolling history back using the hard fork approach. The move, which resulted in the birth of two chains – Ethereum and Ethereum classic, was seen as the viable recourse to address an attack in which the perpetrator stole more than 3.6 million Ether (worth $50 million at the time) by exploiting vulnerabilities in the chain.

To learn more about Solana, visit our Investing in Solana guide.