This weekend I spent a lot of time writing a fuzzer for UglifyJS. A fuzzer is used in fuzzy testing and is a piece of code that generates pseudo-random things to throw at whatever it is you want to test. It's fuzzy in the sense that the results are absolutely not guaranteed but a decently built fuzzer should be able to catch a lot of edge cases that normal tests may not think to attempt.
It all started when UglifyJS tripped over minifying a compound operator (see
this post) and introduced a very subtle bug into the code. One thing led to another and so
I built a fuzzer to help dig up some more edge cases and
it got merged into the main project quickly. And the fuzzer certainly found some bugs.
Over the weekend it uncovered about 13 bugs in various levels of obviousness. Note that the project is already pretty mature and already has
a plethora of tests. Those new bugs were swiftly squashed and it seems the fuzzer is currently not finding any new bugs. To be honest, I was expecting way more bugs. Even considering the maturity of the project. And there probably are plenty super obscure edge cases left to find. The problem with fuzzing uglify is the explosion of possibilities. I mean, that's exactly why you're fuzzing in the first place but in this case these options come from two origins; one is the different ways source code can be generated and the other are the different options with which uglify can minify your code. Certain settings have different code paths and they should all be sound.
I think the fuzzer is currently doing great. It's fairly easy to turn certain features on or off and it runs about as quick as the minifier can work, since that's by far the biggest bottleneck in the fuzzing stack.
The output from the fuzzer is not complete. Although it can build most edge cases by now there will always be certain patterns that are hard to fuzz with without causing a huge amount of incorrect output. Particularly patterns that reuse variables in certain ways are troublesome, both to generate generically as well as to not cause crashes at runtime. Testing (arbitrary) functions calls, property access, all that dynamic stuff. It's all fairly complicated to do with just a static generator.
Therein lies a difference between fuzzing a parser and a minifier; the minifier needs to confirm the runtime output is still the same. For a parser you only need to check whether or not the code is valid and whether your parser reflects this properly. But because the minifier will want to know confirm the output conforms the output of the original code, some care has to be taken in building proper code. And of course ways of checking for discrepancies at all. For this reason three global vars are added which act as "canaries". The generator will randomly add some manipulations of those vars and their output is compared between running the original and minified code. If they don't match the fuzzer stops.
The fuzzing suite can definitely be worked out to something proper and option rich and all that. But for something I threw together over the weekend I think it's a pretty cool thing. On the one hand it uncovered some weird bugs while at the same time proving that the minification process is pretty resilient for surviving all the crap that the fuzzer is throwing at it ;)
Fuzzing is fun and a magical process to watch. This whole thing is making me want to work on an es6 parser again.