What Is Fuzz Testing?
Fuzzing is a dynamic application security testing method used for finding functional bugs
and security issues in software.
What to Expect on this Page
What Are the Benefits of Fuzz Testing?
Fuzz testing enables developers to ship secure software fast, by detecting security and stability issues in the early stages of software development.
1. Run Security Tests On the Source Code
During a fuzz test, a program gets executed with invalid, unexpected, or random inputs, with the aim to crash the application. Modern fuzzing solutions can analyze the structure of the code they are supposed to test. They can generate thousands of automated test cases per second, and mark each path the inputs take through the program. This way, a fuzzer gets detailed feedback about the code coverage, the inputs are reaching during the execution of the source code.
2. Each Finding Leads to More Findings
Once a fuzzer found an input that has caused a crash, it uses mutation algorithms to generate even more inputs that can reproduce the finding with a high probability.
3. Fuzzing Protects Against Unexpected Edge Cases
Modern software fuzzers execute a program with invalid, unexpected, or random inputs. This way you can also cover unlikely or unexpected edge cases, that you would not cover with other testing approaches.
4. Fuzzing Maximizes Code Coverage Without False Positives
Since fuzzers execute the software under test, they always provide inputs that you can use to reproduce the bug. They can also pinpoint the exact line of the code in the repository where the issue is:
Fuzzing enables you to reach up to 100% code coverage without any false positives. Every flagged issue represents an actual bug in the running code. Most are highly critical, e.g., buffer overflows, memory corruption, and leaks.
More and More Industry Standards Require Fuzz Testing
Due to increasing security regulations, more and more software companies have to run automated security tests before shipping their software. That's why many industries and ISO standards recommend integrating automated fuzz testing into the development process. Especially in industries, that already have advanced quality and security regulations. A good example is ISO/SAE 21434 and UNECE WP.29, which deal with the security of automotive software.
What Standards and ISO Norms Recommend Fuzzing?
ISO 26262 Road vehicles – Functional Safety
UNECE WP.29 United Nations World Forum for Harmonization of Vehicle Regulations
ISA/IEC 62443-4-1 Secure Product Development Lifecycle Requirements
ISO/SAE 21434 Road Vehicles — Cybersecurity Engineering
Automotive SPICE for Cybersecurity Guidelines
Cybersecurity in Medical Devices: Quality System Considerations and Content of Premarket Submissions by the U.S. Food and Drug Administration (FDA)
AAMI TIR 57:2016 Principles For Medical Device Security - Risk Management
MDCG 2019-16 Guidance on Cybersecurity for medical devices
UL2900-1 and UL2900-2-1 Healthcare and Wellness Systems - Software Cybersecurity for Network-Connectable Products
ISO/IEC/IEEE 29119 Software and Systems Engineering - Software Testing
ISO/IEC 12207 Systems and Software Engineering – Software Life Cycle Processes
ISO 27001 Information Technology – Security Techniques – Information Security Management Systems
ISO 22301 Security and Resilience — Business Continuity Management Systems
IT-Grundschutz (Germany) Based on ISO 27001
NIST SP 800-95 Web Services — standard for software testing (USA) and others
Microsoft and Google Are Driving Industry Research
Tech corporations like Microsoft and Google were early adopters of fuzzing technologies to test their own systems with it. Since launching in 2016, Google's OSS-Fuzz, a free fuzzing platform for critical open-source projects, has helped fix over 8,800 vulnerabilities and 28,000 bugs across 850 projects.
Numerous open source tools (such as AFL, libFuzzer, HonggFuzz, and OSS-Fuzz) have also emerged, enabling more and more people to use fuzzing technologies. At this point, it was already becoming apparent that fuzzing was going to be a best practice in security testing. However, one major problem that was still looming at the time was the usability of fuzz testing. Open source tools such as AFL and libFuzzer were primarily developed by security experts for security experts and were intended for very specific fields of application. At first, normal developers could do little with them.
Best Practice: How Is Fuzzing Used in Practice?
Fuzzing is most effective when testing is done continuously. Therefore, it is a good idea to integrate fuzzing into CI/CD pipelines. This enables short feedback cycles and makes it possible for developers to quickly fix security vulnerabilities before they become a problem. But when implementing CI/CD fuzzing, developers should be careful not to block the pipeline for too long. This is because fuzzing can take a very long time in some circumstances. This problem can be solved by defining fixed time periods in which fuzzing should take place. For example, at night, between 10 pm and 6 am. Or, by performing regression tests on a regular basis. The fuzzer can also take over the corpus and crash inputs from past test runs, so that you don't have to start from scratch with every test.
When working together in larger development teams, it can also be useful to integrate fuzzing tools into code hosting systems like Gitlab or GitHub. This facilitates team communication and alerts developers if a problem is found during a pull request. Some commercial fuzzing solutions offer such integrations as well as helpful interfaces to issue-tracking systems like Jira and Jenkins.
When vulnerabilities in dependencies of modern Java applications are exploited, they can be used to maliciously spread data throughout multiple microservices rapidly. Since many microservices fully trust inputs from internal sources, attackers can often access large parts of the entire web backend through a small entry point. This was most recently shown by the RCE found in the popular Java library log4j, which affected thousands of Java applications around the world.
Being easier to maintain and more fail-safe, the complexity of modern microservice architectures leaves little room for error. Correspondingly, Java applications have to be tested thoroughly and with testing tools that can deal with their complexity.
Fuzzing 101 - Fuzzing FAQs
New to fuzzing? As fuzzing is seeing increasing adoption in the dev community, we want to answer your questions about this topic as good as we can. In the below FAQ, we want to get you acquainted with fuzzing so you can create your own fuzz tests soon. Let's start with the basics. Feel free to skip ahead if you are already familiar with this.
1. What Is a Fuzzer?
Fuzzing is a dynamic testing method used to identify bugs and vulnerabilities in software. It is mainly used for security and stability testing of the codebase. A fuzzer tests the software under test by feeding it with a series of inputs. In their most basic form, fuzzers generate these test inputs at random or based on a predefined set of values.
This bare-bones form of fuzzing is a black-box approach, that is often used by attackers, as it does not require access to the source code. Black-box fuzzers are easy to set up and inexpensive. However, they are not overly sophisticated, which has earned them the name dumb fuzzers in some circles. The more sophisticated counterpart to so-called dumb fuzzers are feedback-based fuzzers, also called smart fuzzers.
2. What Is Feedback-Based Fuzzing?
As the name implies, feedback-based fuzzers collect feedback about the code covered during the execution of inputs. Unlike testing tools that generate test inputs randomly, this information allows feedback-based fuzzers to constantly refine and improve test inputs and to monitor which parts of the program they reach. By leveraging code coverage, feedback-based fuzzers can traverse larger parts of the source code and trigger deeply hidden bugs and vulnerabilities. Since code coverage is one of the most important metrics in software security testing, feedback-based fuzzing tools can provide useful reports that aid in the interpretation of test findings.
3. Why Is Fuzzing (Especially) Useful for Security Testing?
There are some features that make software fuzzing enormously useful for security testing. Here is why:
- Fuzzing can be almost completely automated
- Fuzzing tools do not only find vulnerabilities; they also provide the inputs that trigger them
- Fuzzing detects bugs without false positives
4. What Is a Fuzz Target?
Fuzz targets are small programs that test predefined API functions, similar to unit tests. However, the inputs are not provided by the developer, but produced by fuzz generators. The generators are responsible for creating random mutations of inputs that are sent to the software under test (SUT). The output of a fuzz generator (i.e., random inputs) is then sent to the SUT. The delivery mechanism processes inputs from the fuzz generator and feeds them to the SUT for execution.
Finally, the monitoring system keeps track of how inputs are executed within the SUT and detects triggered bugs. This plays a critical part in the software fuzzing process, as it also influences what types of vulnerabilities can be discovered during fuzzing.
Example of a Fuzz Target, for Java Applications
Click here to learn how to set up fuzz targets yourself.
public class ExampleValueProfileFuzzer {
// consume data from fuzzer and give it to function under test
public static void fuzzerTestOneInput(FuzzedDataProvider data) {
String str = data.consumeString(100); // get a string with max_len 100
long input1 = data.consumeLong();
long input2 = data.consumeLong();
// call function under test with fuzz data
ApplicationUnderTest.consume(str, input1, input2);
}
5. How Do I Set Up Continuous Fuzzing?
Continuous fuzzing enables dev team to automatically test their code within their favorite dev tools. The effort required to set up a continuous fuzzing cycle with open-source fuzzers is often underestimated. It requires a huge amount of manual configuration and maintenance, while usually not delivering the same results as an enterprise solution. With an enterprise solution, the set-up can be done much faster by simply following an installation manual. You can find out what such an enterprise solution looks like in our free demo app.
6. How Can I Debug My Code After a Fuzz Test?
Modern fuzzing tools enable debugging with a few simple clicks. Since the code is executed during fuzz testing, the crashing input can easily be identified. As presented below, modern fuzzing platforms can immediately take users to the section of the code and set break-points in the stack trace, which enables effortless debugging.
7. We use Static Analysis. Why Do We Need Fuzzing?
While static analysis (SAST) has undeniable advantages, such as immediate feedback loops, the moment developers write code in the IDE, it also has a few disadvantages:
-
The fatigue among developers due to false positives
-
Some bugs and vulnerabilities cannot be identified through static analysis. Some examples include Runtime or Dynamic Behavior Issues, Input-Dependent Flaws, Integration and Environment-Related Issues, Security Misconfigurations, and Behavioral and Logic Flaws.
The best security practice involves leveraging both static and dynamic testing, such as fuzz testing. Integrating fuzz testing and SAST helps cover a broader range of potential issues early in development, reducing false positives and negatives and ensuring compliance with requirements.
Static Analysis and Code Fuzzing in the V-model
An automotive company using static analysis detects 32% of bugs solely through fuzz testing. Learn more
8. What Are the Benefits of Fuzzing C/C++?
Fuzzing is tremendously effective at securing Memory Corruptions and other C/C++-typical vulnerabilities, which makes it highly popular among C/C++ developers. Google's open-source fuzzer OSS-Fuzz, was able to single-handedly uncover over 50 000 bugs in 300+ open-source applications, in just three years. The majority of these applications were written in C/C++. However, the great track record of C/C++-fuzzing, caused some people to falsely assume that finding memory corruptions in C/C++ is the only thing that fuzzing is good at.
9. What Are the Benefits of Fuzzing Java Kotlin & Go?
Next to the memory-unsafe C/C++, fuzzing is nowadays used in a variety of memory-safe languages, such as Java, Kotlin, and Go. In these languages, fuzzing is highly effective at testing APIs and uncovering critical web vulnerabilities such as the OWASP top 10. At Code Intelligence, we open-sourced our very own Java fuzzer Jazzer, which has since been integrated into Google's OSS-Fuzz.
A common misconception surrounding memory-safe languages is that they are inherently secure. Yes, memory-safe languages use runtime error prevention mechanisms, which basically make them immune to memory corruptions. However, there is a variety of other vulnerabilities that can occur in memory-safe languages such as Cross-Site-Scripting (XXS), Denial-of-Services (DoS) and Wrong Handled Exceptions, just to name a few.
How Fuzzing Prevented a Total Ethereum Shutdown
Fuzz testing finds bugs that other testing methods cannot detect. In November 2020 a serious DoS vulnerability was fixed in the source code of the Ethereum network using advanced fuzz tests. In the wrong hands, this vulnerability (CVE-2020-28362) could have caused a shutdown of the entire Ethereum network. Although the memory-safe Golang module has already undergone extensive security testing, this vulnerability could only be found through fuzz testing.
10. What Bugs Can I Find With Fuzzing?
Since 2018, Code intelligence provides a platform for automated fuzz testing. Working closely together with industry and academia, engineers at Code Intelligence were able to find thousands of bugs and extended the reach of modern fuzz testing to a variety of different use cases. Here, you will find an overview of some of the bug classes that the CI team found over the past years. Click here to see the full list.
-
Memory Leaks
Incorrect management of memory allocation, which is exhausting available memory -
Injections
Untrusted input supplied to a program -
Sensitive Data Exposure
When an application inadvertently exposes personal data -
Insecure Deserialization
Manipulated object injected into the context of a web application -
Buffer Overflow
Exceeding the buffer when writing data, which can lead to overwriting adjacent memory locations -
Use After Free
No pointing to a valid object that can lead to data corruption, segmentation faults, or general protection faults
-
Data Races
Accessing the same memory location concurrently in multi-thread process, which can lead to security vulnerabilities -
Software Crashes
Failing to function properly and exiting, possible fatal system errors -
Functional Bugs
Ceasing to respond to inputs -
Uncaught Exceptions
Not properly handling exceptions, which can lead to disrupting program executing -
Undefined Behavior
Executing code not prescribed by language specification (C/C++), which can lead to security vulnerabilities - And many more
11. What Is the Difference Between Black-Box Fuzzing vs. White-Box Fuzzing?
Much has happened since the early days of fuzzing in the 80s. Among modern solutions, one has to differentiate between white-box and black-box fuzzers.
What Is Black-Box, aka Protocol fuzzing?
Black-box fuzzing generates inputs for a target program without knowing its internal behavior or implementation. A black-box fuzzer may generate inputs from scratch, or rely on a static corpus of valid input files to base mutations. Unlike coverage-guided approaches and white-box fuzzing, the corpus does not grow here.
What Is White-Box aka Source Code Fuzzing?
White-box fuzzers analyze the internal structure of a program, and with each new fuzzing run, they learn to track and maximize code coverage. White-box fuzzers usually use adaptable algorithms and intelligent instrumentation during fuzz testing, which makes them more effective and accurate in detecting vulnerabilities.
12. What Are Common Fuzz Testing Tools?
Developers can benefit from a whole range of open-source software fuzz testing tools. There are often specialized for specific use cases (e.g. Kernel fuzzing) or programming languages. However, there are some enterprise fuzz tools that are highly effective if you're working in larger development teams or DevOps environments. Usually, they come with more integrations and features, such as automated bug reporting, CI/CD and dev tool integration, REST API fuzzing, or OWASP vulnerability detection.
Open Source Fuzzers
There are plenty of enterprise solutions, that come with features like reporting, CI/CD/dev tool integration, and API fuzzing. If you want to give such an enterprise solution a go, you can book a free product demo with us.