In the domain of software testing and security analysis, fuzzing has emerged as a powerful technique for uncovering vulnerabilities and enhancing the resilience of software systems. Microsoft and Google have been using fuzzing for ages. They were early adopters of fuzzing technologies to test their own systems. Since launching in 2016, Google's OSS-Fuzz, a free fuzzing platform for critical open-source projects, has helped fix over 8,800 vulnerabilities and 28,000 bugs across 850 projects.
Two primary methodologies within the fuzzing landscape are Protocol fuzzing aka Black-box fuzzing and Code fuzzing. Code fuzzing is often a synonym for white-box fuzzing. Each fuzzing methodology has its own unique approach and set of advantages.
This guide explores the nuances of Protocol and Code fuzzing, offering valuable insights into their methodologies.
Fuzz testing, also known as fuzzing, is an automated software testing technique used to uncover vulnerabilities in software systems by subjecting them to invalid, unexpected, or random data inputs, known as "fuzz."
The main idea behind fuzz testing is to find bugs or security vulnerabilities that may not be easily identifiable through other testing methods or can’t be imagined by those who write manual tests. Let’s be honest, even the best developers forget some cases about what could go wrong, what an input may look like in production. Fuzzing allows companies to test their software early in the development process. Check out the blog What is Fuzz testing? to learn more about how fuzz testing works.
The evolution of fuzzing since its inception in the 1980s has been profound. In today's landscape of advanced testing methodologies, it's essential to distinguish between Code and Protocol fuzzers.
What is Protocol Fuzzing?
Protocol fuzzing involves generating inputs for a target program without prior knowledge of its internal behavior or implementation. In this approach, a Protocol fuzzer may generate inputs from scratch or utilize a static corpus of valid input files to initiate mutations. Unlike coverage-guided techniques and Code fuzzing, the corpus remains static and does not expand over time.
Code fuzzers analyze the internal structure of a program, progressively refining their ability to trace and optimize code coverage with each iteration of fuzzing. Typically employing adaptive algorithms and sophisticated instrumentation, Code fuzzers demonstrate heightened effectiveness and precision in vulnerability detection during fuzz testing.
| Protocol Fuzzing | Source Code Fuzzing | |
| Typical Users | Security Teams | Developers and Testers |
| Application Knowledge | Little Knowledge Analyzes the functionality of an application without a thorough knowledge of the internal design. |
Full Knowledge Analyzes the internal structure of a program knowing the code of the software system. |
| Access to source code | Not needed | Required |
| Code coverage | Unknown and random Usually on interface level. |
High, up to 100% Code coverage metrics. |
| Ability to uncover complex issues | Low Needs human experts with deep knowledge and experience. |
High The process can be automated. |
| Issues reproducibility | No It’s hard to understand issues due to missing source code connection. |
Yes Showing the exact location in source code. |
| Integration into development process | Difficult Misses developers acceptance. |
Full Automatically testing every single code change with high developer acceptance. |
| Complexity of setup | Interface and protocol support and monitoring | Code customization and wrappers |
| Test cases | From random to feedback-based generation | Self-learning input corpus |
| Findings | Often superficial (depends on monitoring) | In-depth |
|
|
| Protocol fuzzing | Code fuzzing | |
| Advantages | Protocol fuzzing can uncover vulnerabilities that are not apparent from examining the source code alone, making it useful for testing closed-source or proprietary software. | Code fuzzing’s main benefit is bug reproducibility. All uncovered vulnerabilities are pinpointed to the exact line of code in the repository and accompanied by inputs that triggered an issue and clear actions to remediate those. So developers can quickly reproduce and start fixing them. |
| Limitations | Bugs and vulnerabilities are hard to reproduce. That’s why protocol fuzzers lack developers' adoption. | Code fuzzing requires access to the source code, which may not always be feasible. |
Overall, both Protocol and Code fuzzing have their strengths and weaknesses, and the choice between them often depends on factors such as the nature of the software being tested, the availability of source code, and the specific goals of the testing process.
CI Fuzz helps developers identify bugs very early in the testing process. It runs from the command line or in IDE and automatically generates thousands of test cases. Learn more about CI Fuzz here.