What Caused The Recent Solana Outage and What Is Being Done to Prevent Further Network Congestion?
What caused the recent Solana outage?
Bots related to a new NFT project based on Solana led to a seven-hour network outage, according to project developers. The outage, which began at about 20:30 UTC on Saturday and ended at 03:30 UTC on Sunday, was caused by a massive increase of inbound transactions (6 million per second) that overloaded the network, exceeding the network’s capacity of 100Gbps at individual nodes.
“There is no evidence of a denial of service attack, but instead evidence indicates bots tried to programatically win a new NFT being minted using the popular Candy Machine program,” the project developers stated in the blog post, “the root cause of the high memory usage was insufficient votes landing to finalize earlier blocks, preventing abandoned fork cleanup. The number of forks validators had to evaluate exceeded their capacity to do so, even after a reboot, necessitating manual intervention.”
What is being done?
According to the dev team, since early January, Solana has suffered intermittent congestion issues resulting from bot activity targeted at NFT mints. The previous outage of Mainnet Beta occurred in September 2021 and lasted for 17 hours. The outage of April 30 shares characteristics with the September outage, but the network this time continued to function even as transaction request volumes reached 10,000% of the level from September, reflecting subsequent updates made by the validator community.
The beta release branch, v1.10, which is currently stabilizing on Testnet, includes memory use improvements to prolong the time nodes can endure slow or stalled consensus. Test nodes running v1.10 deployed on Mainnet Beta continued for 2000 additional slots beyond their similarly spec’d v1.9 peers.
Three mitigations are in the works to address the stability and resilience of the network.
- QUIC – Today, Solana uses a custom raw UDP-based protocol to pass transactions between RPC nodes and the current leader. Since UDP is connectionless and lacks both flow control and receipt acknowledgements, there is no meaningful way to discourage or mitigate abusive behaviour. In order to affect control over network traffic, Solana core protocols are being reimplemented atop QUIC, a protocol built by Google, designed for fast asynchronous communication like UDP, but with sessions and flow control like TCP. Once adopted, there will be many more options available to adapt and optimize data ingestion.
- Stake-weighted transaction QoS – Leader network bandwidth has a fixed capacity and to effectively use it, prioritization is a must in order to end the current practice of indiscriminately accepting transactions on a first-come-first-served basis, without regard for source. Given that Solana is a PoS network, extending the utility of stake-weighting to transaction quality of service is a natural choice. Under this model, a node with a 0.5% stake will have the right to transmit at least 0.5% of the packets to the leader, and the rest of the network and no combination of the remaining stake will be able to fully wash them out. Stake-weighted QoS is in parallel development with QUIC today. Stake-weighted QoS will be more robust in conjunction with QUIC.
- Fee-based execution priority – Once ingested, transactions can still contend for modifying shared account data. This contention has been dealt with by a simple first-come-first-served similarly to network data ingestions, leaving users no means to express the urgency of their transactions’ execution. Given that anyone can submit transactions to the network, stake-weighting is not suitable for this prioritization. Instead, a new instruction is being introduced into the Compute Budget program, offering users the ability to specify an arbitrary “additional fee” to be collected upon execution of the transaction and its inclusion in a block. The ratio of this fee to the requested compute units will serve as a transaction’s execution priority weight. Additional fees will be treated identically to the base fee today.
Fee prioritization is in process and is targeted for the v1.11 release.
DISCLAIMER: The Information on this website is provided as general market commentary and does not constitute investment advice. We encourage you to do your own research before investing.
Join CoinCu Telegram to keep track of news: https://t.me/coincunews
Follow CoinCu Youtube Channel | Follow CoinCu Facebook page
Hazel
CoinCu News
solana solana solana
Author
Knowledge, Blockchain
What Caused The Recent Solana Outage and What Is Being Done to Prevent Further Network Congestion?
What caused the recent Solana outage?
Bots related to a new NFT project based on Solana led to a seven-hour network outage, according to project developers. The outage, which began at about 20:30 UTC on Saturday and ended at 03:30 UTC on Sunday, was caused by a massive increase of inbound transactions (6 million per second) that overloaded the network, exceeding the network’s capacity of 100Gbps at individual nodes.
“There is no evidence of a denial of service attack, but instead evidence indicates bots tried to programatically win a new NFT being minted using the popular Candy Machine program,” the project developers stated in the blog post, “the root cause of the high memory usage was insufficient votes landing to finalize earlier blocks, preventing abandoned fork cleanup. The number of forks validators had to evaluate exceeded their capacity to do so, even after a reboot, necessitating manual intervention.”
What is being done?
According to the dev team, since early January, Solana has suffered intermittent congestion issues resulting from bot activity targeted at NFT mints. The previous outage of Mainnet Beta occurred in September 2021 and lasted for 17 hours. The outage of April 30 shares characteristics with the September outage, but the network this time continued to function even as transaction request volumes reached 10,000% of the level from September, reflecting subsequent updates made by the validator community.
The beta release branch, v1.10, which is currently stabilizing on Testnet, includes memory use improvements to prolong the time nodes can endure slow or stalled consensus. Test nodes running v1.10 deployed on Mainnet Beta continued for 2000 additional slots beyond their similarly spec’d v1.9 peers.
Three mitigations are in the works to address the stability and resilience of the network.
- QUIC – Today, Solana uses a custom raw UDP-based protocol to pass transactions between RPC nodes and the current leader. Since UDP is connectionless and lacks both flow control and receipt acknowledgements, there is no meaningful way to discourage or mitigate abusive behaviour. In order to affect control over network traffic, Solana core protocols are being reimplemented atop QUIC, a protocol built by Google, designed for fast asynchronous communication like UDP, but with sessions and flow control like TCP. Once adopted, there will be many more options available to adapt and optimize data ingestion.
- Stake-weighted transaction QoS – Leader network bandwidth has a fixed capacity and to effectively use it, prioritization is a must in order to end the current practice of indiscriminately accepting transactions on a first-come-first-served basis, without regard for source. Given that Solana is a PoS network, extending the utility of stake-weighting to transaction quality of service is a natural choice. Under this model, a node with a 0.5% stake will have the right to transmit at least 0.5% of the packets to the leader, and the rest of the network and no combination of the remaining stake will be able to fully wash them out. Stake-weighted QoS is in parallel development with QUIC today. Stake-weighted QoS will be more robust in conjunction with QUIC.
- Fee-based execution priority – Once ingested, transactions can still contend for modifying shared account data. This contention has been dealt with by a simple first-come-first-served similarly to network data ingestions, leaving users no means to express the urgency of their transactions’ execution. Given that anyone can submit transactions to the network, stake-weighting is not suitable for this prioritization. Instead, a new instruction is being introduced into the Compute Budget program, offering users the ability to specify an arbitrary “additional fee” to be collected upon execution of the transaction and its inclusion in a block. The ratio of this fee to the requested compute units will serve as a transaction’s execution priority weight. Additional fees will be treated identically to the base fee today.
Fee prioritization is in process and is targeted for the v1.11 release.
DISCLAIMER: The Information on this website is provided as general market commentary and does not constitute investment advice. We encourage you to do your own research before investing.
Join CoinCu Telegram to keep track of news: https://t.me/coincunews
Follow CoinCu Youtube Channel | Follow CoinCu Facebook page
Hazel
CoinCu News
solana solana solana
Other Posts
Related Posts
