With cybercrime on the rise, the code quality and security is becoming more important than ever before. Since it is not uncommon for modern applications to embody hundreds of thousands of lines of code (LoCs), it is crucial to apply an effective and precise method in order to detect potential vulnerabilities and bugs.
Static code analysis (SAST) is the most prominent measure that can easily be integrated into the software development lifecycle to actively identify such vulnerabilities. The biggest advantage of static code analysis is that it can give the developer instant feedback on his code, even if the code is not yet executable. However, most tool providers and companies are increasingly integrating SAST into their CI/CD pipelines, which turns out to be far from sufficient.
The amount of false positives, the resulting manual effort and the undetected bugs are limiting this technology. This blog post will explain the disadvantages of static code analysis when integrated into a CI/CD, and the alternative security testing approaches available.
Static Code Analysis Doesn’t Belong Into Your CI
SAST, or Static Application Security Testing, has been around for many decades. In SAST, the source code is scanned without actually executing it. Here, the focus is to search for suspicious patterns in control/data flow by using heuristics.
Code matching specific patterns, which could indicate potential vulnerabilities, is then presented to the user. Since SAST tools do not execute the code, they can be used at any stage of the software development process, in the best case already in the IDE. Thereby it leads to a good coding style and prevents for example code smells.
The fundamental disadvantage of static analysis is that it produces numerous false positives (warnings that do not actually involve vulnerabilities). Some of the industry’s best SAST tools are designed to have false positives rates around 5%. If we assume a common metric of 20 bugs per 1k lines of code (LoC), the number of potential bugs identified by SAST in an application with 1 million LoC is approximately 20,000.
Of these bugs, we can typically expect 1,000 to be false positive (if our SAST tool is good). Through intelligent rules, developers try to avoid these false positives. In turn, these rules often lead to false negatives (vulnerabilities that do not get detected).
In practice, large projects can easily have hundreds of thousands of warnings and even in toy examples can produce thousands of warnings. This leads to tremendous usability issues and most developers and testers dislike these tools strongly.
A common idea is to outsource the examination of the warnings, thus overcoming the purpose of running the tools in-house. A number of SAST companies now offer heuristics to reduce false positives, however, they do not solve the key issues of SAST.
SAST leads to the following problems:
Given these issues, the question comes up if SAST alone is sufficient to increase the security of a product and the efficiency of the reviewing development teams? Many organizations choose SAST due to its simplicity and the feeling to take security/quality measures.
While there are definitely bugs and vulnerabilities that are avoided with static analysis, the main limitations are the effectiveness and cultural acceptance of it. In order to secure large codebases on a scale, another testing approach is required.
In contrast to SAST, fuzzing always tests the application during runtime. Fuzzing feds the application with a series of inputs, which are purposefully mutated in the testing process. The fuzzer then gets feedback about the code covered during the execution of inputs. Thereby fuzzing explores the state efficiently and discovers bugs hidden deep in the code.
Technology leaders such as Google and Microsoft already use fuzzing to automatically test their code for vulnerabilities. For example, over 27,000 bugs have been found in Chrome and several open-source projects and Google stated that it finds around 80% of its bugs with modern fuzzing techniques. This clearly illustrates the effectiveness of fuzzing to uncover bugs and vulnerabilities.
Additionally, developers benefit from fuzzing in numerous aspects:
All in all, modern fuzzing provides developers with everything needed to detect and fix all kinds of vulnerabilities and bugs while cutting away the usability issues of SAST. Furthermore, testing can happen at scale with fuzzing. Similar to the way unit tests were introduced back then, fuzzing is now revolutionizing the way the world tests software.
Developers do not have to worry about manually writing unit tests, which have to cover all kinds of possible edge cases, anymore. Instead, developers can easily set up fuzzing tests that automatically generate endless amounts of unbiased test cases. Fuzzing can drastically reduce development costs through the improvement of developer’s efficiency.
Finally, the aggregation of dynamically generated inputs can be used to apply regression testing of applications prior to the deployment. In combination, SAST can be helpful in identifying targets for fuzzing and maximizing code coverage.
In summary, it can be said that fuzzing should no longer be missing in any development process. Otherwise, a huge potential for efficiency and security is given away.
Have you become curious about the potential fuzzing has to offer for your development process?