Our work on the evaluation of fuzzers has been accepted at 26th European Symposium on Research in Computer Security (ESORICS) 2021. In our paper My Fuzzer Beats Them All! Developing a Framework for Fair Evaluation and Comparison of Fuzzers (David Paaßen, Sebastian Surminski, Michael Rodler, Lucas Davi) we present our novel setup to evaluate and compare different fuzzers.

Fuzzers are used very successfully to automatically find bugs and security vulnerabilities in computer programs. Scientists and experts from the industry present new and improved fuzzers. However, due to the complexity of fuzzing and the wide variety of different programs and bugs it is not trivial to compare different fuzzers with each other. Currently published fuzzing papers follow a wide variety of different approaches and not always follow all existing recommendations.

Our work systematically studies the influence of various parameters on the evaluation of fuzzers. To do so, we conducted extensive experiments which took over 280 thousand CPU hours to complete. We could empirically show, for example, that the choice of tested programs has a significant influence on the evaluation results. To assist future fuzzing research and work in the same area, we publish our evaluation framework SENF (Statistical EvaluatioN of Fuzzers) and all the experiment data on Github. Our paper is currently available as a pre-print on arXiv.