RouteProcessor2 Post Mortem

💡 Update 26 April 2023: the claim portal for users to claim their lost tokens has just gone live, please read the instructions here.

What Happened?

On April 8, 2023, Sushi core contributors soft launched V3 upgrades for the protocol that included a new router called RouteProcessor2 to facilitate swaps from the frontend. The router was planned to be used across each iteration of Sushi’s AMM pools (v2, Trident, v3), and to also be used for future aggregating across other protocols (Uni, Pancake, etc.). The RouteProcessor2 contract was deployed to 14 networks including: Arbitrum, Arbitrum Nova, Avalanche, Boba, BSC, Ethereum, Fantom, Fuse, Gnosis, Moonbeam, Moonriver, Optimism, Polygon, and Polygon ZkEVM.

Later that night, the HYDN security team identified a critical vulnerability in the RouteProcessor2 contract that affected all user tokens that had been approved on the contract and raised the issue to Sushi’s core contributors. A subsequent war room was formed with HYDN’s security team, and Sushi promptly rolled back upgrades to the UI that included the new router to prevent any further token approvals from happening on the RouteProcessor2 contract. It became immediately clear that creating a fix, even a temporary one, was going to be challenging. This was due to the fact that the contract was non-upgradeable, could not be paused, and it was not possible to revoke access for users. As we’ve seen with similar approval bugs in the past, the only real options in this situation is to go public and recommend all affected users remove pending approvals and / or perform a whitehat rescue.

As the war room was progressing with identifying all affected wallets and working on contracts to perform the white hat rescue, an ImmuneFi report came in under the Sushi bounty program to report the same issue. Initially, the report was auto-closed due to the contract falling out of scope for not being included in the list of contract addresses within the bounty program. Sushi core contributors later re-opened the report 20 minutes later, and notified the bounty hunter that we were aware of the issue and were currently looking into it.

Early on in the data mining process to identify all affected addresses, it became clear early on that one specific address made up a majority of the balances at risk. Sushi contributors recognized the address at risk, and moved quickly to establish communication lines with the at risk user to inform them of the approval bug with the user responding shortly after that they would un-approve the contract.

Unfortunately, the user was not quick enough to un-approve the contract. Within minutes of being made aware of the situation, the bounty hunter that reported the bug through Immunefi attempted a whitehat hack to save 100 WETH from the affected user. Ultimately, their transaction was detected by MEV bots in the public mempool, who then quickly replayed the attack and drained ~1800 WETH within seconds from the at risk user. Shortly after, other bad actors were made aware of the exploit through several sources that posted results of the wallet drain via Twitter.

In response, Sushi contributors gave the greenlight to the HYDN security team to start the white hat rescue to try and save the rest of the funds that were at risk across each chain. HYDN then proceeded with the rescue, drained as much vulnerable funds as possible, and deployed a cross-chain watcher contract to front-run bad actors from performing the same exploit. HYDN was successfully able to save over $750k worth of user funds across multiple networks, and as of writing is still saving tokens as unaware users send funds to at-risk wallets.

For more information into the rescue and technical descriptions of the exploit, check out HYDN’s blog on what happened.

---

Where Did The 1,800 WETH Go?

The initial 1800 WETH that was drained from the first user’s wallet ended up in multiple wallets as a result of the 18 replayed transactions. From that amount, a total of 885 ETH has been returned so far:

• ~685 ETH sent to Sushi core contributors operations multisig
• 190 ETH sent to the affected user
• 10 ETH sent to the Sushi rescue contract

Due to the nature of the attack some of these transactions were built by independent block builders, where in one case a substantial amount of ETH had been transferred as a MEV reward to the block builder that then redirected it to the Lido: Execution Layer Rewards Vault. This resulted in ~795 ETH of the total being dispersed accordingly as "rewards"; more information on this portion of ETH can be found in the Lido Governance Forum and we are working with the Lido community on a solution to recuperate the funds.

Also as of writing, another wallet that was a part of the exploit was able to steal 94.9 ETH that is still sitting in this wallet address: 0x8AC0B9656b7c39be0d3D73828D2041E8C0e27712. In the event that the address owner is reading this, we urge them to return the funds. Failure to do so may necessitate reporting the wallet address to the relevant authorities.

---

Results Of The Whitehat Rescue

We’ve made significant progress in recovering the remaining lost funds. At the time of writing, HYDN has helped Sushi rescue over $750k of user assets and the rescued funds were deposited to HYDN’s labeled whitehat wallet: 0x74ebb8e8d0b0cc65f06040eb0f77b5da0e33ffee. Furthermore, we have retrieved a total of 885 ETH, and an additional amount of 795 ETH is presently in the execution layer rewards vault as stated in the paragraph above.

Sushi is committed to making all users whole, and the very last remaining part of the stolen funds lost to black hat hackers will be covered and refunded by Sushi.

Sushi understands the gravity of this incident and the impact it may have had on their users and is therefore taking every possible step to resolve this issue. To this end, Sushi has released a claim process for users affected by both the white and black hat funds. The user claim forms are currently undergoing validation, and the Merkle claim contract is being audited.

To summarize, there are 2 types of affected users and therefore 2 different ways for these users to claim back their funds:

---

User Group 1: Funds Rescued

For users with tokens that ended up in the 0x74ebb8e8d0b0cc65f06040eb0f77b5da0e33ffee rescue contract, their funds are safe and will be claimable through the Sushi UI by a Merkle tree contract. As of writing, all work on the claim process has been completed and we are awaiting the results of an audit before rolling out the claim portal.

This group will have the opportunity to reclaim their lost tokens on a 1:1 basis. The Sushi claim portal will also verify that each user has unapproved the vulnerable router before allowing the token reclamation process to proceed.

For all future users who have their tokens recovered after the opening of the claim portal, we will update the Merkle tree periodically to include these tokens as well. Given the probability of further token recoveries, the claim process will continue to run on an ongoing basis.

User Group 2: Funds Not Rescued

For users with tokens that did not end up in the 0x74ebb8e8d0b0cc65f06040eb0f77b5da0e33ffee rescue contract, it is likely that these funds cannot be recovered. Sushi will set-up a separate claim process for this group, which the user has to opt-in to, and where the claims have to be reviewed on a case-by-case basis. This process will therefore not be as quick as the first group of users, as each of the claims will need to be verified for legitimacy against on-chain data to validate the claim.

To initiate this process, we have opened a Google Form, which affected users in this group should complete with all the necessary information.

Our goal is to return all user tokens to legitimate claimants, and we appreciate everyone’s patience as we work through this.

---

Bounty Reward

Sushi deeply appreciates the outstanding efforts of the HYDN security team in identifying the critical vulnerability, assisting with the whitehat rescue, and helping to protect users' assets. To recognize their invaluable contribution, we have awarded HYDN $200k. This gesture underscores our commitment to the security of our platform and the importance of collaborating with skilled individuals and teams in the blockchain ecosystem.

We believe that fostering strong relationships with security experts and whitehats is essential for building a safer, more resilient DeFi ecosystem. The proactive measures taken by the HYDN security team have undoubtedly saved users from significant losses and reinforced the importance of collaboration in addressing security challenges.

As we continue to move forward, we remain dedicated to working closely with security researchers and experts to ensure the highest level of security for our users and the entire DeFi community.

---

Moving Forward

Firstly, please know that Sushi’s UI is completely safe to use now. However, we do want to reiterate that users check their wallet for any pending approvals that they may have missed on the RouteProcessor2 contract. You can check your wallet approvals through revoke.cash and sushi.com.

In regards to the router, the at risk contract has been fixed and sent off for a second audit before we plan to roll it out again.

Lessons Learned

1. The Importance of Pausability in High Activity Contracts: Approval bugs can have catastrophic consequences, and mitigating them can be a messy affair. For high activity contracts that don't custody funds, incorporating pausability features is likely the answer. This allows contracts to be temporarily halted in case of any issues, reducing the risk of devastating outcomes.
2. Moving Away from Unlimited Approvals: As blockchains become cheaper and Layer 2 solutions gain prominence, unlimited approvals are both dangerous and unnecessary. We will follow the user experience of 1inch and DeFiLlama, promoting one-time approvals per swap rather than infinite approval, enhancing security for all users.
3. Respecting Auditors' Timelines: Attempting to fast-track contracts through the auditing process can lead to overlooked vulnerabilities. It's crucial to allow auditors ample time to thoroughly examine contracts, even when they appear to be low-risk. Every detail matters in ensuring contract security.
4. Emphasizing Internal Review and Gradual Rollouts: To bolster security, we will implement a higher level of internal review and testing. Gradual rollouts will be preferred over mass deployments, enabling us to identify and address issues more effectively.
5. Encouraging Responsible Whitehat Practices: While we appreciate the efforts of whitehats, it's vital that they are certain of their actions before intervening. Providing teams with the opportunity to respond to reports and waiting for approval before making decisions can prevent unintended consequences.
6. Enhancing the Rollout Process for New Contracts: To further strengthen security, we will include new contracts in the Immunefi scope list before rolling them out on the user interface. This proactive approach ensures that potential vulnerabilities are addressed before contracts become widely accessible. While it also will give whitehats some more time to report bugs in a responsible way.

---

Closing Thoughts

As Sushi core contributors, we extend our sincerest apologies to all users and members of our community. We understand the gravity of this incident and the impact it may have had on users and are therefore taking every possible step to resolve this issue. We accept complete accountability for the incident and are fully committed to making all users whole. Our focus remains on executing the claims process we have established to recover the tokens lost by affected users. We are committed to this objective and will remain dedicated to its fulfillment.

We would like to express our gratitude to all those who have assisted us in the recovery process. We extend a big thank you to the HYDN security team for their exceptional effort in aiding us to salvage as much of the funds as possible. We will continue to provide regular updates on the claim process and urge affected users to remain vigilant for future updates on our official channels: Discord & Twitter.

---

⚠️ Last but not least, please take note of the following security tips:

• All official communication occurs through Sushi Twitter and Sushi Discord ONLY.

• Do not click on any links that people DM you. The only links you should follow are from Sushi Twitter and Sushi Discord.

• The Sushi claim portal is not live yet, any websites that look like a claim portal are NOT from Sushi.

• Sushi team members will never DM you about refunds. If someone does, it's likely a scam.

• Sushi is not conducting any airdrops at the moment. If someone DMs you about it, it is NOT from Sushi.