Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

improve CI build times by limiting runs #337

Merged
merged 1 commit into from
Feb 21, 2023
Merged

Conversation

0xricksanchez
Copy link
Contributor

@0xricksanchez 0xricksanchez commented Feb 21, 2023

It seems like the randomness of a fuzzing run and the generally slower CI makes some runs really unpredictable. The string_compare example seems to be one of them. Bumping up the runs in the hope to fix the flakiness. Offline 100-200k runs were sufficient in 95% of the time, so the initial upper limit of 1 million was deemed enough. However, #335 failed to succeed

@0xricksanchez 0xricksanchez requested a review from a team February 21, 2023 07:46
@0xricksanchez 0xricksanchez added this pull request to the merge queue Feb 21, 2023
Copy link
Contributor

@bertschneider bertschneider left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Strange, any idea why it differs so much?

@0xricksanchez
Copy link
Contributor Author

@bertschneider From what I can observe locally is that setting a seed x on Linux produces somewhat different behavior compared to setting the same seed on Windows..

In the mentioned example and the current seed, on Linux the fuzzing behavior in over 20 test runs roughly follows the pattern of:

## Consistent
#2	INITED cov: 2 ft: 2 corp: 1/1b exec/s: 0 rss: 120Mb
## Consistent: The used mutator is always the same
#1326	NEW    cov: 5 ft: 5 corp: 2/17b lim: 17 exec/s: 0 rss: 120Mb L: 16/16 MS: 4 InsertRepeatedBytes-CrossOver-EraseBytes-InsertRepeatedBytes-
## Always either a CMP- or CopyPart-CMP mutator 
#19838	NEW    cov: 6 ft: 6 corp: 3/33b lim: 198 exec/s: 0 rss: 126Mb L: 16/16 MS: 2 CopyPart-CMP- DE: "Awesome "-
## This has a tad more variance of ChangeByte-CMP, CMP-, ChangeBit-ShuffleBytes-PersAutoDict-CMP, or ChangeBit-CMP-
#186390	NEW    cov: 7 ft: 7 corp: 4/49b lim: 1850 exec/s: 0 rss: 153Mb L: 16/16 MS: 2 ChangeBit-CMP- DE: "Fuzzing"-

On Windows, the behavior is a lot more "random" as in it can produce a similar output but also tends to completely end up with other mutators, or in other cases fails to find the necessary coverage from 67 for whatever reason.

Bottom line, setting a seed (in libfuzzer) is not a guarantee for 100% deterministic behavior (across different systems and/or operating systems)?


I tried checking if there's a difference in PRNG that is being used, but at first glance it doesn't look like it:

Ripping out the used PRNG as a standalone test produced the same random numbers over 100 iterations:

// clang++ test.cpp -O3 -o test
// ./test 111994470
#include <cstdlib>
#include <iostream>
#include <random>
#include <stdlib.h>
#include <string>

class Random : public std::minstd_rand {
public:
  Random(unsigned int seed) : std::minstd_rand(seed) {}
  result_type operator()() { return this->std::minstd_rand::operator()(); }
  template <typename T>
  typename std::enable_if<std::is_integral<T>::value, T>::type Rand() {
    return static_cast<T>(this->operator()());
  }
  size_t RandBool() { return this->operator()() % 2; }
  size_t SkewTowardsLast(size_t n) {
    size_t T = this->operator()(n * n);
    size_t Res = static_cast<size_t>(sqrt(T));
    return Res;
  }
  template <typename T>
  typename std::enable_if<std::is_integral<T>::value, T>::type operator()(T n) {
    return n ? Rand<T>() % n : 0;
  }
  template <typename T>
  typename std::enable_if<std::is_integral<T>::value, T>::type
  operator()(T From, T To) {
    assert(From < To);
    auto RangeSize = static_cast<unsigned long long>(To) -
                     static_cast<unsigned long long>(From) + 1;
    return static_cast<T>(this->operator()(RangeSize) + From);
  }
};

int main(int argc, char **argv) {
  unsigned int seed = atoi(argv[1]);
  std::cout << "Seed: " << seed << std::endl;

  Random Rand(seed);
  for (int i = 0; i <= 100; i++) {
    std::cout << "Rolled value: " << Rand() << std::endl;
  }
  return 0;
}

So, there might be something else at play here, which would need some more investigating

@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Feb 21, 2023
@0xricksanchez 0xricksanchez added this pull request to the merge queue Feb 21, 2023
Merged via the queue into main with commit 2d20f42 Feb 21, 2023
@0xricksanchez 0xricksanchez deleted the limit_runs_fix branch February 21, 2023 09:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants