JSCheck

Douglas Crockford
2012-04-22

JSCheck is a testing tool for JavaScript. It was inspired by QuickCheck, a testing tool for Haskell developed by Koen Claessen and John Hughes of Chalmers University of Technology.

JSCheck is a specification-driven testing tool. From a description of the properties of a system, function, or object, it will generate random test cases attempting to prove those properties, and then report its findings. That can be especially effective in managing the evolution of a program because it can show the conformance of new code to old code. It also provides an interesting level of self-documentation, because the executable specifications it relies on can provide a good view of the workings of a program.

The JSCheck program is loaded from the jscheck.js file. It produces a single global variable, JSC, which contains the JSCheck object.

JSCheck is concerned with the specification and checking of claims. (We use the term claim instead of property to avoid confusion with JavaScript's use of property to mean a member of an object.) To create a claim, call JSC.claim, passing in

JSC.claim returns a claim function, which may be passed as an argument to the JSC.check function, which will randomly generate the cases that will attempt to reject the claim. You can set the number of cases generated per claim with the JSC.reps function.

Making a Claim

To make a claim, you pass three or four components to JSC.claim, which will then return a function.

name

The name is descriptive text that will be used in making the report.

predicate

The predicate is a function that will return true of the claim holds. The predicate will do something with the system in question, perhaps examining its result examining the consistency of its data structures. If you are testing functions that do encoding and decoding, the predicate can assert things like

return value === decode(encode(value))

You won't need to select the value. JSCheck will generate random values for you.

specifiers

An array of specifiers describe the types of the predicate's arguments. (From a procedural perspective, specifiers are generators, but JavaScript may get a new generator feature which is very different, so avoid confusion, we will take a declarative view.)

JSCheck provides a small library of specifiers which you can use in your claim. For example, JSC.integer(10) declares that a parameter should an integer between 1 and 10. JSC.one_of(['Curly, 'Larry', 'Moe']) declares that a parameter can be one of three string. Some of the specifiers can be combined, so JSC.array(JSC.integer(10), JSC.character('a', 'z')) declares that a parameter can be an array of 1 to 10 lowercase letters.

An array of specifiers can also contain constants (such as string, numbers, or objects), so you can pass anything you need to into the predicate. If you need to pass in a function, then you must to wrap the function value with the JSC.literal specifier.

You can also create your own specifiers.

classifier

You can optionally pass a classifier function as part of the claim. The classifier will receive the same arguments as the predicate. A classifier can do two things:

  1. It can examine the arguments, and return a string which classifies the case. The string is descriptive. The report can include how many of the cases belong to each classification. This can be used to identify the classes that are trivial or problematic, or to help analyze the results.
  2. Since the cases are being generated randomly, some cases might not be meaningful or useful. The classifier can have a case rejected by returning false. JSCheck will attempt to generate another case to replace it. It is recommended that the classifier reject fewer than 90% of the cases. If you accepting less than 10% of the potential cases, then you should probably reformulate your claim.

The JSCheck functions

The JSCheck object contains several functions.

Configuration:

The configuration functions set up the JSCheck object.

JSC.clear()

The clear function reinitializes the JSCheck object, discarding its collection of claims and groups.

It returns the JSCheck object.

JSC.detail(level)

By setting the level of detail to a particular number, you can determine how much information is included in the report. The report will be delivered to the function designated with JSC.on_report. The default is 3.

  1. none: There will be no report
  2. terse: There will be a minimal report, showing the pass score of each claim
  3. failures: The cases that fail will be reported
  4. qualification: The qualification summaries will be reported (default)
  5. verbose: All cases will be reported

It returns the JSCheck object.

JSC.on_fail(function(object))

The on_fail function allows the registration of a callback function that will be given an object for each failed case. This can be used to trigger a throw to cancel further tests or to begin deeper processing. The callback will be passed an object containing these properties:

It returns the JSCheck object.

JSC.on_pass(function(object))

The on_fail function allows the registration of a callback function that will be given an object for each passing case. This can be used to trigger further tests or to begin deeper processing or reporting. The callback will be passed an object containing these properties:

It returns the JSCheck object.

JSC.on_report(function(string))

The on_report function allows the registration of a callback function that will be given a string containing the results for each claim. The callback could route the report to a console or a log or alert. The level of detail is set with JSC.detail.

It returns the JSCheck object.

JSC.reps(repetitions)

The reps function allows setting the number of repetitions per claim. The default is 100.

It returns the JSCheck object.

Specifiers:

A specifier is a function that returns a function that can generate values of a particular type. The specifiers are used in building the array of specifiers that is used to construct a claim.

JSC.array(array)

The array specifier takes an array as a template. It will go through the array, expanding the specifiers it contains. So, for example,

JSC.array([
    JSC.integer(),
    JSC.number(100),
    JSC.string(8, JSC.character('A', 'Z'))
])

can generate arrays like

[3,21.228644298389554,"TJFJPLQA"]
[5,57.05485427752137,"CWQDVXWY"]
[7,91.98980208020657,"QVMGNVXK"]
[11,87.07735128700733,"GXBSVLKJ"]
...

JSC.array(dimension, value)

The array specifier takes a number and a value, and produces an array using the number as the length of the array, populating the array with the value. So, for example,

JSC.array(3, JSC.integer(640))

can generate arrays like

[305,603,371]
[561,623,477]
[263,534,530]
[163,148,17]
...

JSC.boolean()

The boolean specifier will produce true and false with equal probability.

JSC.boolean(bias)

The boolean specifier will produce true and false. If the bias is 0.50, it will produce them with equal probability. A lower bias will produce more falses, and a higher bias will produce more trues.

JSC.character(code)

The character specifier treats its argument as a char code, generating a character.

JSC.character(min_character, max_character)

The character specifier generates characters within a given range.

JSC.integer()

The integer specifier generates prime numbers. Sometimes when testing formulas, it is useful to plug prime numbers into the variables.

JSC.integer(i)

The integer specifier generates an integer between 1 and i.

JSC.integer(i, j)

The integer specifier generates an integer between i and j.

JSC.literal(value)

The literal specifier generates the value without interpreting it. For most values (strings, numbers, boolean, objects, arrays), the literal specifier is not needed. It is needed if you want to pass a function value to a predicate, because function values are assumed to be the products of generators.

JSC.number(x)

The number specifier produces random numbers between 0 and x.

JSC.number(x, y)

The number specifier produces random numbers between x and y.

JSC.object(object)

The object specifier takes an object as a template. It will go through the enumerable own properties of object, expanding the specifiers it contains. So, for example,

JSC.object({
	 left: JSC.integer(640),
    top: JSC.integer(480),
    color: one_of(['black', 'white', 'red', 'blue', 'green', 'gray'])
})

can generate objects like

{"left":104,"top":139,"color":"gray"}
{"left":62,"top":96,"color":"white"}
{"left":501,"top":164,"color":"white"}
{"left":584,"top":85,"color":"white"}
...

JSC.object(keys, values)

The object specifier takes an array of keys and produces an object using those keys. The values are taken from an array of values or a specifier. So for example,

JSC.object(
    JSC.array(JSC.integer(3, 8), JSC.string(4, JSC.character('a', 'z'))),
    JSC.boolean()
)

can generate objects like

{"jodo":true,"zhzm":false,"rcqz":true}
{"odcr":true,"azax":true,"bnfx":true,"hmmc":false}
{"wjew":true,"kgqj":true,"abid":true,"cjva":false,"qsgj":true,"wtsu":true}
{"qtbo":false,"vqzc":false,"zpij":true,"ogss":false,"lxnp":false,"psso":true,"irha":true,"ghnj":true}
...

and

JSC.object(
    ['x', 'y', 'z'],
    [JSC.integer(320), JSC.integer(240), JSC.integer(100)]
)

can generate objects like

{"x":99,"y":51,"z":51}
{"x":114,"y":166,"z":82}
{"x":35,"y":124,"z":60}
{"x":13,"y":41,"z":63}
...

JSC.one_of(array)

The one_of specifier takes an array of specifiers, and selects values from the array with equal probability. So, for example

JSC.one_of([
    JSC.number(),
    JSC.boolean(),
    null
])

produces values like

0.09817210142500699
0.3351482313591987
null
false
...

JSC.one_of(array, weights)

The one_of specifier takes an array of specifiers and an array of weights. The weights are used to adjust the probabilities. So for example,

JSC.one_of(
   [1,2,3,4,5,6,7,8,9,10],
   [1,2,3,4,5,6,7,8,9,10]
)
produces values like
8
10
1
10
...

JSC.sequence(array)

The sequence specifier takes an array of values, and produces them in sequence, repeating the sequence as needed.

JSC.string(value)

The string specifier generates the stringification of the value. So for example,

JSC.string(JSC.integer(1000, 9999))

produces values like

"4791"
"9523"
"2774"
"4288"
...

JSC.string(number, value)

The string specifier generates strings by joining some number of values. So for example,

JSC.string(JSC.integer(1, 8), JSC.one_of("aeiou")))

produces values like

"ieauae"
"io"
"iuieio"
"iuu"
...

Claim processing:

The JSC.claim function creates claims, and the JSC.check function generates the cases and produces the reports.

JSC.check()

Process all of the claims that have been construct since the beginning or the most recent call to JSC.clear.

It returns true if every case passes.

JSC.check(group)

Process all of the claims of a specific group.

It returns true if every case passes.

JSC.check(claim)

Process the specific claim function.

It returns true if every case passes.

JSC.claim(name, predicate, specifiers)

The claim function takes a name, a predicate function, and an array of specifiers.

The predicate function should return true if a case passes. It will take a list of arguments that is generated by the array of specifiers. The array of specifiers looks like a type declaration for the predicate function.

It returns a function that can be processed by JSC.check.

JSC.claim(name, predicate, specifiers, classifier)

The claim function takes a name, a predicate function, and an array of specifiers.

The predicate function should return true if a case passes. It will take a list of arguments that is generated by the array of specifiers. The array of specifiers looks like a type declaration for the predicate function.

The classifier function is called before the predicate function. It gets a chance to determine if the random values in its arguments will be a reasonable case. It can return false if the case should be thrown out and a new case generated to replace it. The classifier function can also return a descriptive string that describes the case. These can be counted and displayed in the report.

It returns a function that can be processed by JSC.check.

JSC.group(name)

The group function is used to specify a group name that will be attached to all new claims. This makes it easy to process a group of claims together.

JSC.test(name, predicate, specifiers, classifier)

The test function calls the check function, passing the result of the claim function.

Using JSCheck

Since JSCheck performs a useful specification and description function as well as a testing function, it is recommended that claims be inserted into the relevant source code, and not in separate source files. JSDev can make this easier to manage, so that claims can easily be removed from production code. All of the calls to JSC can be hidden in special comments, which are activated during development, and removed by minification in production.

Writing specifiers

JSCheck provides a small set of specifier that can be combined in many ways. But for some purposes, you may need to create your own specifiers.

It is easy to do. A specifier is a function that returns a function. The function that is returned will be stored in the specifiers array, and will be called for each value that needs to be generated. Its arguments might be other specifiers, so if an argument is a function, use the result of calling the function.

Demonstration

One difficulty in demonstrating testing systems is that the exposition of the system to be tested is usually significantly more complex than the testing tool being demonstrated. So in this case, we will be testing a trivial function. We will make an incorrect claim. JSCheck will help us to find the error in the claim. It might seem counter productive to demonstrate bad claim making, but it turns out that it is as important to get the claims right as it is to get the program right.

We are going to test the le function.

function le(a, b) {
    return a <= b;
}

We will construct a claim. Our predicate simply returns the result of le. It takes two integers, one with a max of 10 and another with a max of 20. We will classify the cases by the relationship between the arguments.

/*check
    JSC.test("Less than", function (a, b) {
        return le(a, b);
    }, [
        JSC.integer(10),
        JSC.integer(20)
    ], function (a, b) {
        if (a < b) {
            return 'lt';
        } else if (a === b) {
            return 'eq';
        } else {
            return 'gt';
        }
    });
*/

But when we check the claim, many cases fail. The summary of the specifiers tells the story:

eq pass 7
gt pass 0 fail 22
lt pass 71

The predicate failed because 22 of the generated cases had an a that was larger than b. This is because JSC.integer(10) produces from the range 1 to 10, and JSC.integer(20) produces from the range 1 to 20. Sometimes the first value will be larger. This tells us that we should make the predicate more sophisticated, or we could have the specifier return false instead of 'gt' to reject those cases.