March 27, 2024
ChatGPT can write smart contracts, just don’t use it as a security auditor

ChatGPT: Write Smart Contracts, Not Security Auditor

Researchers from Salus Security, a blockchain security company with a global presence, have recently conducted research highlighting GPT-4’s capabilities in parsing and auditing smart contracts. Despite its proficiency in generating and parsing code, GPT-4 is not deemed suitable for the role of a security auditor.

“GPT-4 can be a useful tool in assisting with smart contract auditing, especially in code parsing and providing vulnerability hints. However, given its limitations in vulnerability detection, it cannot fully replace professional auditing tools and experienced auditors at this time.”

The researchers utilized a dataset comprising 35 smart contracts, known as the SolidiFI-benchmark vulnerability library, containing a total of 732 vulnerabilities. This dataset was used to assess GPT-4’s ability to identify security weaknesses across seven common vulnerability types. The findings indicate that ChatGPT performs well in detecting true positives, reaching over 80% precision in testing.

True positives refer to actual vulnerabilities that would merit investigation outside a testing environment. However, GPT-4 exhibits a notable challenge in generating false negatives, as reflected in its “recall rate,” which, in the Salus team’s experiments, was as low as 11% (where a higher rate is desirable). The researchers concluded that GPT-4’s vulnerability detection capabilities are lacking, with the highest accuracy reaching only 33%.

Consequently, they recommend the use of dedicated auditing tools and traditional human expertise for auditing smart contracts until AI systems like GPT-4 can be enhanced to meet the necessary standards. The study underscores the importance of a cautious approach to relying solely on AI for security audits in complex systems like smart contracts, emphasizing the continued need for human oversight and specialized tools to ensure thorough and accurate evaluations.

Photo by Andrew Neel

Related posts

Teaching Machines to Teach: Google’s AI Mastery Leads to 40% Boost in Coding Skills

Chloe Taylor

LinkedIn Empowers Recruiters with Innovative AI Features

Chloe Taylor

US Senator Elizabeth Warren Calls for Fairness in Crypto and Big Tech AI Landscape

Cheryl  Lee

This website uses cookies to improve your experience. We'll assume you're ok with this, but you can opt-out if you wish. Accept Read More