On The Limitations of Audit Competitions
The Ethereum DAO hack was a defining event in blockchain history, spotlighting smart contract security. In response, the industry evolved, emphasizing the need for security measures. One effective strategy that emerged was hosting audit competitions, offering cash prizes to identify and fix security flaws.
Audit competitions have surged in popularity recently, with numerous platforms vying to connect projects with independent security researchers. The premise is straightforward: organizations offer a cash prize to attract auditors eager to scrutinize their code for vulnerabilities. While cost-effective, the evolving ecosystem necessitates a closer examination of the drawbacks inherent in relying solely on audit contests.
2024 is for record setting bounties
Earlier this year, zkSync made a significant impact in the security community by allocating 5 million security budget to a Code4rena contest prize. An auditor known as xuwinnie secured over USD500,000 from that prize pool, setting a new record for audit competitions. Meanwhile, Blast has announced a $1.2 million prize for an upcoming contest, with various DAO governance discussions underway aiming to surpass that amount in the future.
The concept of the contests is straightforward: the codebase is made public, and researchers are given a few weeks to look for vulnerabilities. By the deadline, they report their findings, which wardens categorize as critical, high, medium, or low severity, distributing the prize money accordingly. This system is particularly beneficial for emerging security researchers, offering them a virtually permissionless opportunity to get paid for their contributions, regardless of experience level. It also provides projects with a cost-efficient strategy for having their codebase checked by a large number of researchers from the community.
But security contests are not perfect
As we move into the 2024 bull market, the number of projects engaging in security contests is on the rise. With the five main platforms—Code4rena, Sherlock, Immunefi, Codehawks and Cantina—hosting a combined total of 10 ongoing contests, the community has voiced concerns. It’s impossible for a leading auditor such as xuwinnie to investigate all 10 projects in a meaningful way. It’s worrisome that some of these contests may only get submissions from junior researchers, working with only a few weeks to audit the code. It’s easy to see how this arrangement may not be enough to stand the test of time against experienced black hat hackers post-launch.
For context, the last 4 contests averaged 5342 lines of solidity code each, and ran concurrently on one platform for either 2 or 3 weeks. Projects offering the highest bounties are likely to attract more freelance auditors. However, does this ensure a thorough audit? The bounty system incentivizes the discovery of high severity bugs, which command the largest rewards. When multiple auditors identify the same vulnerability, the prize for that finding is divided among them. Consequently, a common low severity bug reported by many can lead to extremely low individual payouts, at times less than $1.
In practice, auditors are incentivized to search for high severity bugs they specialize in, then move on to the next project. It’s conceivable that parts of the code deemed unlikely to contain a unique, high-severity bug may not receive a thorough review during a competition, or worse, may be overlooked entirely.
Time is money
Evaluating the actual value of experienced auditor hours per dollar spent in an audit competition, along with their effectiveness, is challenging. The self-reported nature of these hours often leads to unreliable metrics. A common issue developers face during audits is the time-consuming nature of interacting with multiple auditors. Effective communication is crucial for successful security engagement. An audit firm usually provides a single point of contact to streamline communications, but audit contests can lead to repetitive inquiries about the same issues. A lead dev for a well known project going through an audit contest recently shared with me that responding to questions from Code4rena auditors consumed the majority of their work time during the initial two weeks of the contest.
Other important factors to consider include potential intellectual property risks by opening the codebase to the public in the pre-launch phase. Some auditors are anonymous, and generally the contest websites will make the codebase available to the public. Lastly, auditors earn by finding vulnerabilities - but not necessarily by finding the best possible solution. Remediation and re-audit processes will typically be more thorough with a dedicated team.
Introducing a hybrid contest-audit approach to blockchain security
All of that is to say, spending the entire security budget on an audit competition is not a good idea. Even for the high-dollar contests for blue chip projects such as zkSync Era or Eigenlayer, only a small percentage of their overall security budget was spent on the security contest.
Our recommendation is to allocate at most 50% of the initial security budget on the contest prizes. For smaller projects, 50% feels right - the other 50% should go towards engaging with a dedicated audit company. For medium and large projects, the proportion of contest prizes to overall budget should decrease further. We recommend first engaging with an audit firm - they can do a design review, prepare the codebase, and provide concise scope and specifications for the audit contest. Then run the security contest, and finally, go back to the firm to remediate the findings and run a final end-to-end audit. This will ensure a balanced approach towards low and high severity vulnerabilities, ensure complete coverage, get expert help with bug fixes - in short - it will yield the most comprehensive security strategy.
For larger projects, dedicating a portion of the budget to an ongoing white hat bounty and exploring options such as formal verification are steps that can provide further security assurances. Blockchains are unique in the way raw financial value relies on the code. In web2 exploits, the hackers often require funds to be transferred to them using cryptocurrency. In web3, they usually get the crypto directly in their account after executing the code. If your code is responsible for millions of dollars, consider the hybrid approach to auditing/contests, and you will sleep better at night knowing you’ve taken a comprehensive security approach on behalf of your users.