How Can Fuzzing Help Find Bugs in Hardware?

The growing complexity of embedded systems, coupled with the advent of increasingly sophisticated security attacks, highlights a dire need for advanced automated vulnerability analysis tools.

Fuzzing is a proven-effective technique to find security-critical issues in systems, that often do not require fully understanding the system's internals. While fuzzing has been widely adopted in software testing, fuzzing tools and techniques to find security bugs in hardware are gaining traction in both academia and industry.

Fuzzing Soft IPs for Fun and Profit

In electronic design, a semiconductor intellectual property (IP) core is a reusable unit of logic, cell, or integrated circuit layout design. Soft Intellectual Properties (SIPs) are IP cores generally offered as synthesizable register-transfer level (RTL) modules. These are developed in Hardware description languages like Verilog. An SIP implements a specific logic that is concise enough to be tested using fuzzing, while not running into scalability issues often encountered while testing a system on a chip (SoC).

Compiling Verilog into C++ executables or SystemC models is seeing increasing adoption, with tools such as Verilator in the forefront. This opens exciting new opportunities to harness state-of-the-art coverage-driven scalable fuzzers such as AFL++ to test Verilog modules by converting them to C++ binaries. Although the technique of applying software fuzzers to verilated hardware modules has been proposed earlier, this article focusses on identifying specific Common Weakness Enumeration (CWEs) via SIP fuzzing.

This blog references the research paper "Fuzzing Soft IPs for Fun and Profit", which I presented at FuzzCon Europe 2021.

A New (Fun) Approach to Fuzz Sip Modules With AFL++

In our research paper, we present a newer approach to fuzzing SIP modules, i.e., using AFL++ to fuzz interfaces of verilated SIPs. It is based on the following methodology.

1. Identify/Isolate the Logic & Interface to Fuzz

SIP interface at the top-level module is an excellent fuzzing target. Sometimes, it may require modeling behavior of other logic that interacts with the SIP. These interactions may already be defined to support SIP verification, e.g., Bus Functional Models, which can be reused for fuzzing purposes. Additionally, we can drill further down into the SIP logic to isolate sections of code that implement a security critical functionality and focus on fuzzing the interfaces of such logic in isolation.

2. Write a C++ Test Harness

Next, we write a C++ test harness, which defines the standard main() that instantiates the cycle-accurate behavioral model as a C++ object. The C++ object in this case is the verilated SIP DUT (device under test). This test harness must read stimulus from a file and input it to the DUT. The goal is to malform this input file using fuzzers such as AFL++ and use it to drive device under test (DUT) stimuli.

3. Compile Verilog / C++ Test Harness Into C++ Executable (Verilation)

The synthesizable Verilog code is then converted to a cycle-accurate C++ behavioral model using Verilator which then, along with the C++ test harness, is compiled into a simulation executable by Verilator. To instrument the executables for AFL++, we can redirect Verilator to use afl-g++ instead of g++ to enable coverage-driven unique test case generation (avoiding redundant tests).

4. Fuzz the Interfaces (Using AFL++)

Generally, fuzzers log test cases that generate a crash or a hang. One can write assertions or define rules to monitor and log malformed inputs that drive the DUT to such unexpected states. Techniques to automatically generate assertions or rules including the test harness by parsing design specification documents and Verilog modules along with tainting inputs to understand how inputs flow during simulation can be explored to improve the capabilities of this fuzzer.

5. Review/Replay/Reuse Test Cases That Cause Failures

The goal here is to retrieve the test cases that were logged by the fuzzer and replay those as directed tests in a simulation or emulation environment to ascertain the correctness of the DUT. Test cases can be reused in a regression environment to verify fixes are in place.

Challenges of Hardware Fuzzing and Proposed Solutions

During our research, we encountered a few challenges in fuzzing hardware components, which we also discussed in our research paper.

Fuzzers Are Not Built to Handle Hardware State Machines

Finite State Machine (FSM) outputs depend on current inputs, past inputs, current states, and past states (are sequential vs combinational). An out of box fuzzer is not built to remember past states and past inputs between fuzz iterations, so state transitions need to be accurately handled between each fuzz iteration. The intermediate state of a Verilated model may be saved on disk, so that it may later be restored.

This save/restore operation can be called from the C++ test harness. Leveraging this save/restore ability between each fuzz iteration enables the fuzzer to drive the DUT to the next state based on the previous state, previous input, and current input to generate the output signal. It also allows the fuzzer to not only log test inputs causing failures but also save a snapshot of the state during failure for later introspection.

Generating Clock and Reset Signals Is Not a Native Fuzzer Capability

Out of the box, software fuzzers cannot generate a clock signal to drive state transitions in the DUT. Similarly, instructing a software fuzzer to assert or deassert a reset signal after “n” clock cycles or when specific conditions are met is challenging. Both these issues can be solved using the state save/restore ability provided by Verilator.

On the first fuzz iteration, the current state is saved to disk. On subsequent fuzz iterations, the previous state is restored from the disk, the previous state of the clock is read, toggled, and input to the DUT in the current state. This new state is now saved to the disk including the clock and the process repeats until a failure case is encountered and logged.

A reset signal can similarly be asserted or deasserted via the test harness at random, after ‘n’ clock cycles or by monitoring state transitions and checking to see if a specific condition is met (e.g., stuck states).

Hangs and Crashes Are Unlike Software in Hardware

Buffer overflows causing crashes or application resource exhaustion leading to hangs are software-centric. Hardware hangs can be due to but not limited to transitions to invalid states, states stuck without transitions, memory or I/O reads that do not return, waiting indefinitely for completions during non-posted transactions etc.

On the other hand, a crash occurs when the fuzzer encounters runtime assertions that include $fatal and $error in the verilated DUT. Similarly, a runtime warning or runtime information message is generated when the fuzzer encounters a $warning or $info assertion respectively. This ability of the fuzzer to comprehend Verilog assertions allows us to monitor for interesting behaviors that may have a security impact.

Non-Asserted Failures Are Risky

In security-critical flows, there can be undetectable scenarios where assertions are missed, but it can still result in unexpected hardware behavior, invalid (potentially dangerous) output, or undefined state.

For example, a privilege escalation bug leading to asset access by an untrusted agent can go undetected during fuzzing because an assertion is missing. Consequently, the fuzzer is made to believe this is indeed a valid scenario and no failure test case is logged.

Hence, we must ensure that assertions are correctly written to capture all potential failure scenarios or the test harness itself can incorporate these checks as a library of issues to evaluate for during SIP fuzzing.

Fuzzing Multiple Stimuli Can Get Complex

To drive multiple stimuli accurately, we need grammar-based test case generation combined with coverage-driven fuzzing. There is a growing interest in custom mutators for AFL++ to handle highly structured inputs. We are exploring these custom mutators for AFL++ for SIP interface fuzzing.

Fuzzing Hierarchical Designs Requires Domain Knowledge

Hierarchical designs facilitate the design of complex architectures and promote design reuse. Fuzzing such designs can get complex and requires a good understanding of the design hierarchy.

Verilators support the inclusion of /*verilator hier_block*/ metacomment in the Verilog code of instantiated submodules and nested hierarchical blocks to enable the compilation of the entire design hierarchy into a single executable for testing purposes.

Not All Verilog is Synthesizable

Synthesizable Verilog translates into gates, registers, RAMs, etc. But not all Verilog should be synthesized, e.g., delays are implemented using clocks and flipflops or loops end up as multiple hardware instances, which may not be the ideally expected outcome, especially for someone who is accustomed to writing and compiling C/C++ code. Although this is not a fuzzer issue, it might still be worth noting here.

Case Study Results

We evaluate the proposed methodology and tooling to identify certain hardware weaknesses in Verilog code snippets provided as examples to demonstrate the CWE. The following table summarizes the results of the CWE case study. Additionally, all failure scenarios were detected within the first minute of fuzzing.

For more access the full research paper "Fuzzing Soft IPs for Fun an Profit"

Conclusion & Future Plans

We believe the proposed Soft Intellectual Property (SIP) fuzzing approach is promising and can achieve acceptance in the SIP verification community in due course.

We propose exploring how the SIP fuzzing framework can utilize existing verification collateral via automation to scale adoption. Another key area to explore is how to implement a reusable library of assertions for a broader set of hardware weaknesses, i.e., attack patterns, that the SIP fuzzing framework can leverage.

The tools themselves are free, open source, community-supported. The barrier to entry is considerably low.