The Absolute state of Fuzzing
You don’t know
You don’t know what you don’t know.
Web Developers will never take testing and correctness seriously. I can tell by the tools they focus on.
Unit tests fall into a trap: the people who write them don’t know the systems they’re testing.
How can you ever hope to combat things you don’t know? You have to discover them.
How do you find things you don’t know exist? Well the answer is fuzzy
Literally; one way to find the elusive answer to the things you don’t know is to make a “fuzzy” process to find those things. I’m currently writing about how I managed to find a lead on something similar; what systems and tools do technical writers use to manage the ideas and text that goes along with specific parts of the systems they’re writing in? That rabbit hole is its own thing.
But software is not too different: how do programmers keep the disparate bits of information tied together long enough to tell the machine what to do with it? How do you verify that your software will work? How does it all tie back to this article?
Test What You Know
If you would be a real seeker after truth, it is necessary that at least once in your life you doubt, as far as possible, all things.
Rene Descartes: Principles of Philosophy
We programmers are tasked with establishing a semblance of truth in the systems we create. Usually this means that we need to know how the parts of our system operate in well-defined situations. I’ll call this unit-testing for the sake of this article, but know that there’s a lot of discourse surrounding this.
Unit tests are not the end-all-be-all. Unit tests don’t protect you from most bugs and errors. Code coverage is also a near-useless metric. The thing is: if you don’t want to find the truth it can’t find you.
Does this mean that Unit testing is useless?
No! (Write Tests), in fact most of the critiques of code coverage, unit testing and testing boils down to quality. Bad tests don’t help, but no testing isn’t the answer.
Unit testing is a helpful canary. Unit testing is a seatbelt, and a guide rail. But seat belts don’t get you to where you want to be, they just keep you safe from dumb mistakes. And you should always wear your seatbelt.
Unit tests should give you and your team an early warning to any
Unit Testing frameworks are a deep group. Jest, Mocha, Jasmine, QUnit etc. etc. All of these tools keep you safe from mistakes you know about. They check your code and what is really a huge list of assumptions.
Test What You Don’t Know You Don’t Know
even when we find not what we seek, we find something as well worth seeking as what we missed.
Robert Boyle
Ok, so you’re testing for what you know and you have a framework for testing the things you know about and building knowledge about the unknown parts and adding them to your library. What about the thinks that hide from you?
Sometimes bugs and errors come up from unexpected places. Sometimes systems work in unexpected ways away from edge cases. You can’t find all of the bugs that could possibly come up in a system.
For this I suggest “Fuzzing”
Take your inputs, apply random variability to them, and make sure they still work.
This methodology is great for other reasons: a lot of the time our test cases in regular unit testing is constrained by our imagination. I’m sure that a lot of us have written unit tests that go for the edge-cases, a few of the middle cases, and called it there. After about 5 test cases, I usually can’t force myself to think of any more,
Why is fuzzing so bad?
ECMAscript (or Ja**script) ecosystem is anything but coherent ecosystem is anything but coherent ecosystem is anything but coherent ecosystem is anything but coherent ecosystem is anything but coherent ecosystem is anything but coherent. That’s largely because no one (living) company (not even the O company) actually controls all parts of the language: Node.js and ECMA are different, npm is owned by Microsoft, and for the longest time the language itself was fractured among different paradigms, preferences and competing libraries, not to mention a fractured ecosystem (see left-pad).
On top of this, unit testing is an unattractive non-functional tool: developers pick which framework (if any) and use the kit that comes with it. Most developers on their own projects do not use it.
Search for Unit Testing, you’ll find Jest, Mocha, JSunit, and empty articles talking about the different flavors and which one is best. Fuzzing is a tool on top of that tool (typically). Digging and scraping, I found this which gave me a few tools (most of them are not NPM tools)
Maybe it’s because the Node and JS ecosystem revolves around a language that’s not particularly fast, nor particularly suited for testing. While writing this I found 3 other projects that aren’t tuned to JS.
Will Fuzzing Ever be Mainstream? No.
In JS, I think the answer is no. At least not for a long time. React is busy re-inventing the web, Selenium and Cypress are vying for UX testing, and JS in general doesn’t care about testing.
Go seems to care: most of the fuzzing libraries I found seem are written in Go (Python too). Until making UI’s in the Web is so normal it’s boring, we won’t see any real progress in testing.
Is this detrimental?
Not critically so, but it is telling that the ECMAScript community is invested in “making things” rather than making things work. The lack of fuzz testing in JS is telling. Maybe it’s because JS developers are concerned with things that are trivial and solved problems. Maybe it’s because JS developers can’t fix a fundamentally broken language and ecosystem.
I’m not sure, but the future isn’t entirely dark. I’ve found good options, including Jazzer. The JS community tends to be focused on front ends, but with enough time tools like Jazzer might be part of core testing capabilities, like Jest or Mocha are today.