There is no doubt writing tests brings value. I believe every professional should write tests for their code, and no software should ever be released without tests. We may discuss whether metrics such as code coverage are useful or not. We may argue that integration tests bring more value than unit tests. But I think it's undebatable that with tests, we have at least some degree of certainty that our code is correct.
Classic tests
Typically, regardless of framework, language, or kind of testing, a test consists of three main parts:
Conditions / Assumptions (Given)
Changes / Actions (When)
Observations / Assertions (Then)
The exact names depend on the testing paradigm and community, but the notion is the same: We make assumptions, then perform operations, and finally verify whether the code behaves expectedly.
The problem with tests
Having said all that, I would argue that there's a problem with how tests are usually created: all three steps need to be handled by humans. And humans make mistakes.
Another issue is that this approach to testing requires us to think of every possible scenario. Every step that can be taken in the app, every possible value of variables, and their combination… It's a lot to handle for our minds. And as the number of features in the codebase increases, the complexity of testing grows exponentially. Testing everything is infeasible.
And while there are techniques that help us deal with this complexity, how many times have you released to prod without any bugs? Let me take a guess: 0 times.
Property-based testing
Now, here's a silly idea: what if inputs to tests and actions taken in tests were created automatically by a machine? That's exactly where property-based testing comes into play!
This idea, like a lot of good ideas, comes from functional programming (Haskell, Quickcheck). Later, it was refined in Elixir. But the same concept exists in multiple different languages: C++, Rust, Python, TypeScript… I'll focus on the latter.
Property-based testing is a concept where:
The simplest example
Let's take a quick look at the simplest example. It's repeated in probably thousands of articles about property-based testing. We have a function sort that takes a list of numbers, sorts it, and returns. Ignore the fact that most languages have such a function in their standard library.
We could easily write unit tests for our sort. For example:
Given a list (3, 5, 1)
When it's sorted
The result is (1, 3, 5)
Or in code:
it("sorts numbers", () => {
const input = [3, 5, 1];
const result = sort(input);
expect(result).toEqual([1, 3, 5]);
});
We could add a few more tests for edge cases, too: sorted list, empty list, list where all elements are equal… However, unit testing is often called example-based testing. We only test cases we think of, so our test is only as good as our imagination.
QA Engineer walks into a bar, and they order a beer. Order 0 beers. Order 999999 beers. Order -1 beers. Order a lizard. Order a ueicbksjdhd.
✅ Unit tests pass
A real customer walks into the bar and asks where the bathroom is.
The bar goes up in flames.
Now, property-based testing takes a different approach. We think of properties and tell the framework: "Our sorting function behaves such that given any list of numbers…". Let's list a few properties:
When a list is sorted, each element is greater or equal to the previous one.
The lengths of the original list and the sorted list are equal.
Each element from the original list is also in the sorted list.
We do not think of specific cases. Instead, we think of properties that can be applied to any list of numbers. It's way more difficult to think about tests this way, but it's also more beneficial. When we're done designing the properties, the framework will generate hundreds or thousands of tests for us.
Enter fast-check
The framework for property-based testing in TypeScript is called fast-check. What does the above example look like in code?
import assert from "node:assert";
import { it } from "node:test";
import fc from "fast-check";
it("When a list is sorted, each element is greater or equal to the previous one.", () =>
fc.assert(
fc.property(fc.array(fc.integer()), (arr) => {
const sorted = sort(arr);
assert(sorted.every((v, i) => i === 0 || v >= sorted[i - 1]));
})
));
What this code does is tell fast-check that we're testing a property that requires a list of integers (fc.array(fc.integer())). Fast-check will generate hundreds of randomized inputs on which our assertion will be run.
Shrinking and counterexamples
But the truly interesting thing happens when one of our tests fails. I've introduced a bug to my sorting function to demonstrate this. We get an error message saying Error: Property failed after 6 tests, then an undecipherable string (seed and path), and finally Counterexample: [[23,22]] / Shrunk 526 time(s). What does all that mean?
Fast-check generated a bunch of random arrays and found a case where the test failed. The arrays generated could've been huge. Wouldn't it be nice if Fast-check could somehow produce a minimal reproducible example instead? This is exactly what's going on here! The framework started looking for a smaller input for which the test fails. This process is called shrinking.
Fast-check started with a large array, shrunk it 526 times, and produced a counterexample: an array of just two elements [23, 22]. Isn't that just wonderful?
Fixing bugs
You might be tempted to fix the bug, run the tests again, and move on with your life. But how do you actually know that the bug is fixed? Inputs generated by fast-check are randomized, so there's the off chance of not generating a failing case anymore!
Therefore, the first thing you must do whenever you discover a bug with fast-check is to write a unit test for that specific scenario. That's where another feature of fast-check comes in handy: examples. After a slight refactoring:
it("When a list is sorted, each element is greater or equal to the previous one.", () => {
const eachGreaterThanPreviousProperty = fc.property(
fc.array(fc.integer()),
(arr) => {
const sorted = sort(arr);
assert(sorted.every((v, i) => i === 0 || v >= sorted[i - 1]));
}
);
fc.assert(eachGreaterThanPreviousProperty, { examples: [[[23, 22]]] });
fc.assert(eachGreaterThanPreviousProperty);
});
Summary
I highly recommend playing with fast-check and trying to write a few tests with it. However, in practice, property-based unit tests often become more complex than the underlying tested implementation itself. I treat it as a fun exercise to change the way we reason about the problem.
The next step? Property-based end-to-end testing. This is where the framework really shines, and the cost vs. benefit ratio is much more profitable. I'll focus on that topic in my next article, so follow my blog if you don't want to miss it!
