In this post, we will show how you can write fuzz tests for your JavaScript projects in Jest as easily as regular unit tests. To make this possible, we have added integration for Jazzer.js into Jest, which enables you to write fuzz tests using the familiar Jest API. Additionally, you get great IDE support with features such as debugging and test coverage reporting out-of-the-box. This integration enables a smooth user experience with the advanced fuzzing technology provided by Jazzer.js. Usability is essential to making fuzzing an integral part of the standard development workflow and thus enabling developers to build more secure and stable JavaScript projects.
%20(3)14af.jpg)
Writing a Jest Fuzz Test
Let us start from the big picture and look at how Jazzer.js fuzz tests look in Jest. To this end, we will use an example project from the Jazzer.js GitHub repository. In this example, we have a simple function, fuzzMe, that takes a Buffer as a parameter and throws an error on a specific input (“Awesome Fuzzing!”). A standard Jest unit test looks like this:
const target = require("./target");
describe("My test suite", () => {
it("My normal Jest test", () => {
expect(() => target.fuzzMe(Buffer.from("abc"))).not.toThrow();
expect(() => target.fuzzMe(Buffer.from("Awesome Fuzzing!"))).toThrow(
"Welcome to Awesome Fuzzing!"
);
});
})
To create a fuzz test, you can use the new fuzz function on Jest's test and it.
const target = require("./target");
describe("My describe", () => {
it.fuzz("My fuzz test", (data) => {
target.fuzzMe(data);
});
});
This example also shows that if you already have unit tests for your code, it is straightforward to create corresponding fuzz tests. As you can see in the example above, the fuzz test uses a very similar syntax to the corresponding unit test. The key difference here is that in the unit test, you have to specify the concrete inputs with which you test your code. In the fuzz test, Jazzer.js automatically generates inputs to maximize code coverage. Here, Jazzer.js calls the fuzz function in a loop and provides a new input (data) in each iteration.
The fuzz Addition to Jest
The function fuzz expects similar parameters as standard Jest tests but provides fuzzer-generated input as a first parameter to the test function. This parameter is of the type Buffer, a subclass of Uint8Array, and can be used to create needed parameters for the actual code under test. Jazzer.js then uses code instrumentation to get coverage and execution feedback and uses those to generate inputs that trigger as many program states as possible. This is the data argument in the example above.
Now that we have a high-level overview of the Jest integration of Jazzer.js, let’s dive into the details of how you can set it up for your project.
Setting Up the Jazzer.js Jest Integration
Jest enables the execution of tests by third-party tools through custom test runners. Jazzer.js provides such a runner in the @jazzer.js/jest-runner package.
To use the integration, add a dev-dependency to @jazzer.js/jest-runner to your project. To do so, execute the following command in your project root directory:
'npm install --save-dev @jazzer.js/jest-runner'For Jest to pick up the custom fuzz test runner, it has to be added to the Jest configuration in the package.json or jest.config.js. The following snippet shows how to add the Jazzer.js fuzz test runner to the Jest configuration in package.json, so that all files with the suffix .fuzz.js will be executed by it. The default test runner still executes all other tests, which enables running fuzz tests and unit tests simultaneously:
{
"name": "jest-integration-example",
"scripts": {
"test": "jest",
"fuzz": "JAZZER_FUZZ=1 jest",
"coverage": "jest --coverage"
},
"devDependencies": {
"@jazzer.js/jest-runner": "1.3.0",
"jest": "29.3.1"
},
"jest": {
"projects": [
{
"displayName": "test"
},
{
"runner": "@jazzer.js/jest-runner",
"displayName": {
"name": "Jazzer.js",
"color": "cyan"
},
"testMatch": ["/**/*.fuzz.js"]
}
]
}
} Executing Jest Fuzz Tests
The Jazzer.js fuzz test runner provides two modes of execution: fuzzing mode and regression mode.
Fuzzing Mode
As the name suggests, the fuzzing mode uses Jazzer.js to fuzz a test function. This mode fuzzes a function in an exploratory way to uncover bugs and security vulnerabilities not previously known.
To execute tests in the fuzzing mode, set the environment variable JAZZER_FUZZ to a non-null value and execute Jest as normal:
'JAZZER_FUZZ=1 npm test'Please note that currently, only one fuzz test can be executed in the fuzzing mode. The runner skips all others. To start specific fuzz tests, you can use the Jest parameter --testNamePattern, e.g. npx jest --testNamePattern="My describe".
Inputs triggering issues , like uncaught exceptions, timeouts, etc., are stored in a directory structure named according to the test file and internal test structure. For example, an issue found in the test "My fuzz test" in the describe block "My describe" in the test file fuzztests.fuzz.js would end up in the directory ./fuzztest.fuzz/My_describe/My_fuzz_test. Files in these directories will be used in the regression mode to verify that the underlying issues have been resolved.
Furthermore, as fuzzing is a time-consuming task, interesting inputs triggering uniquely new code coverage are stored in a similar directory structure located in .cifuzz-corpus at the project root. Jazzer.js uses these inputs to speed up the next run by reusing the already discovered inputs. This means that for every run, the fuzzer starts from the aggregated knowledge about your code that it gathered in all previous runs.
Executing fuzz tests in fuzzing mode on the command line could look like the following snippet. This output may differ depending on the used Jest reporter:
» JAZZER_FUZZ=1 npm test
RUNS Jazzer.js ./integration.fuzz.js
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 3712434369
INFO: Loaded 1 modules (512 inline 8-bit counters): 512 [0x7fb804068010, 0x7fb804068210),
INFO: Loaded 1 PC tables (512 PCs): 512 [0x7fb7fc7fa010,0x7fb7fc7fc010),
INFO: 0 files found in /jazzer.js/examples/jest_integration/.cifuzz-corpus/integration.fuzz/My_describe/My_fuzz_test/
INFO: 1 files found in /jazzer.js/examples/jest_integration/integration.fuzz/My_describe/My_fuzz_test/
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: seed corpus: files: 1 min: 4b max: 4b total: 4b rss: 173Mb
#2 INITED cov: 3 ft: 3 corp: 1/4b exec/s: 0 rss: 173Mb
#1377 REDUCE cov: 6 ft: 6 corp: 2/17b lim: 17 exec/s: 0 rss: 173Mb L: 16/16 MS: 4 CopyPart-CMP-CopyPart-InsertRepeatedBytes- DE: "\377\377\377\37
FAIL Jazzer.js ./integration.fuzz.js (19.887 s)
● My describe › My fuzz test
Error: Welcome to Awesome Fuzzing!
25 | s[15] === "!"
26 | ) {
> 27 | throw Error("Welcome to Awesome Fuzzing!");
| ^
28 | }
29 | };
30 |
at Object.fuzzMe (target.js:27:11)
at integration.fuzz.js:24:12
● My describe › My callback fuzz test
[FuzzerStartError: Fuzzer already started. Please provide single fuzz test using --testNamePattern. Skipping test "My callback fuzz test"]
● My describe › My async fuzz test
[FuzzerStartError: Fuzzer already started. Please provide single fuzz test using --testNamePattern. Skipping test "My async fuzz test"]
Test Suites: 1 failed, 0 passed, 1 total
Tests: 1 failed, 2 skipped, 0 passed, 3 total
Snapshots: 0 total
Time: 21.137 s
Ran all test suites
Regression Mode
The regression mode is the default execution mode and verifies that once found issues are resolved they stay that way.
In this mode, the previously found problematic inputs are used to invoke the fuzz tests and verify that no error is generated anymore. If the described directory structure does not contain inputs for a given test, it will be skipped by the runner.
Jazzer.js generates a dedicated test entry in the overall Jest report for each input it executes in the regression mode.
Executing fuzz tests in the regression mode on the command line could look like the following snippet. This output may differ depending on the used Jest reporter:
» npm test
PASS Jazzer.js ./integration.fuzz.js
My describe
○ skipped Sync timeout
○ skipped Async timeout
○ skipped Done callback timeout
My fuzz test
✓ one
My callback fuzz test
✓ two
My async fuzz test
✓ three
Test Suites: 1 passed, 1 total
Tests: 3 skipped, 3 passed, 6 total
Snapshots: 0 total
Time: 0.335 s, estimated 1 s
Ran all test suites.
IDE Integration
As the Jest test framework foundations are used by the Jazzer.js fuzz test runner, all major IDEs supporting Jest should pick it up as well. This means that IDE-specific Jest configuration can be used for fuzz tests.
Especially handy is the debug functionality of individual tests provided by most IDEs. Simply set a breakpoint inside your IDE and re-run a failed fuzz test to enter the debugger for that particular input.
VS Code Jest Supportdb8e.png)
IntelliJ Jest Support
Conclusion
In this post, we showed you can write fuzz tests with Jazzer.js using standard Jest syntax, which gives you a smooth user experience and great IDE support while still providing state-of-the-art fuzzing technology. We also discussed a few advanced topics to make your fuzz tests more effective and easier to write.
Do you use or maintain a different testing framework and would like to see Jazzer.js integrated there? If so, you can get in touch with us. We also welcome contributions if you have ideas to make Jazzer.js better and more useful.
We still have plenty planned for Jazzer.js including novel bug detectors for JavaScript to find critical vulnerabilities such as injections and remote code execution. We are also adding support for Jazzer.js in our CLI tool CI Fuzz and our SaaS platform.
