FuzzBench: {{ experiment.name }} report

{% if in_progress %}
(experiment incomplete/still running...)
{% endif %}
warning
Please consider this as a preliminary report to demonstrate the capabilities of FuzzBench. While we have tried our best, we have not confirmed that we configured everything correctly. We are hoping to work together with the community to validate results and improve the set of fuzzers, benchmarks, and their configurations in the future. See FAQ for more details.

experiment summary

Aggregate critical difference diagram showing average ranks when ranking fuzzers on each benchmark according to their median reached coverages. Critical difference is based on Friedman/Nemenyi post-hoc test. See more in the documentation.
Note: If a fuzzer does not support all benchmarks, its ranking as shown in this diagram can be lower than it should be. So please check the list of supported benchmarks for the fuzzer(s) of your interest. The list could be specified in the fuzzer's README.md like this.
  • Median coverages on each benchmark
    {{ experiment.summary_table.to_html() }}
{% for benchmark in experiment.benchmarks %}

{{ benchmark.name }} summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
{% if benchmark.fuzzers_with_not_enough_samples and not in_progress %}
error
The following fuzzers do not have enough samples: {{ ', '.join(benchmark.fuzzers_with_not_enough_samples) }}.
{% endif %}
  • Sample statistics and statistical significance
    Coverage sample statistics
    {{ benchmark.summary_table.to_html() }}
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
{% endfor %}

experiment data

You can download the raw data for this report here. {% if experiment.git_hash %}

The experiment was conducted using this FuzzBench commit: {{ experiment.git_hash }} {% endif %}