Print Story LetroTest lives!
By tmoertel (Thu Sep 09, 2004 at 05:09:38 PM EST) (all tags)
You may know that I love the Haskell programming language and regularly extol the many virtues of the Haskell-based QuickCheck automatic testing system. But did you know that I so missed QuickCheck when writing Perl code that I spent a few days last week writing a Perl version called LectroTest? And did you know that the LectroTest mascot is an adorable yet angry robot with scary mechanical claws? (Inside).

Perl already has great unit testing support with modules like Test::Simple and Test::More. But even so, these modules follow the traditional test-case model of testing that requires you to do a lot of work that probably could be delegated to the computer.

Think of it this way. You're looking for needles (errors) in a haystack (your program's behavior). You want to find the needles and get rid of them. Most unit testing systems make you point out to the computer each and every piece of hay you want it to inspect and what to expect at each. Consider this case. OK, now this one, here. And this one, too. Now this one. And, yes, this one.... To build confidence in whether any particular desired property of your software holds, you must create enough test cases to expose the needles in the haystack.

Here's a typical Perl unit test using Test::More:

# AngularDifference.t

use AngularDifference;
use Test::More tests => 6;

# identical angles should be zero
    is( angdiff(  0,  0),   0, "zero at zero" );
    is( angdiff( 90, 90),   0, "zero at 90" );
# order of angles shouldn't matter
    is( angdiff(  0, 45),  45, "0,45 -> 45" );
    is( angdiff( 45,  0),  45, "45,0 -> 45" );
# should return the smallest angle between
    is( angdiff(  0,270),  90, "0,270 -> 90" );
# multiples of 360-degrees shouldn't matter
    my ($a,$b) = (360 * 2, 360 * 4);
    is( angdiff($a,$b+23),23, "$a,$b+23 -> 23" );
Each one of the is(...) statements is a particular case to test. The first argument is the output of the thing we're testing. The second is the expected output. The third is the name of the test case, to be used in testing reports. The comments above each block of test cases give you an idea of the needles we're trying to expose with the following cases.

In this example, we're testing a function angdiff that computes the difference between two angles a and b (in degrees). The correct result is the smallest angle between a and b. The test cases all attempt to see whether the function faithfully holds to this one simple definition. It seems doubtful that even this many cases will provide enough coverage of our implementation's behavior to give us much confidence. So we may be forced to write even more cases. What a pain.

With QuickCheck and LectroTest, on the other hand, things are different. You make the computer write the test cases for you. All you do is describe the shape of the haystack and tell the computer what a needle looks like, in general. The computer then runs over to the haystack and starts testing pieces of hay randomly and at a blinding pace. After a while it may find a needle. In that case, it emits a counterexample and stops. You can plug the counterexample into your code to figure out what went wrong. (You can also add the counterexample to a list of regression tests to run later.)

Here's the LectroTest property to check whether our implementation of angdiff faithfully holds to the definition:

# AngularDifference.l.t

use AngularDifference;
use Test::LectroTest;

Property {
    ##[ a <- Int, n <- Int, diff <- Int(range=>[-180,180]) ]##
    my $b = $a + 360 * $n + $diff;
    angdiff($a, $b) == abs($diff);
}, name => "definition via inverse";
The first part of the Property gives the shape of the haystack we're searching. It's the part in between the ##[ and ]## delimiters. The second part is the needle-recognizer. It's a bit of code that results in either true (no needle) or false (needle present). Finally, the property has a name, again for human consumption in reports and such.

The above is actually a just Perl program. To check whether our property holds, we simply run it:

$ perl AngularDifference.l.t
not ok 1 - 'definition via inverse' falsified in 9 attempts
# Counterexample:
# $a = -1;
# $diff = 1;
# $n = -1;
Oops! Looks like there's a problem in our implementation of angdiff.

If you are interested in this kind of thing, check out the slides from the talk I recently gave for the Pittsburgh Perl Mongers: "Free Unit Tests In Perl with LectroTest". In the talk I elaborate on the example above and explain how to build LectroTest properties. I also show how to check more complicated properties, such as whether an RFC-2822-compliant email-address parser meets the spec.

P.S. If you read this far, thank you! I had to write quickly, so this probably isn't easy to read. Since I haven't posted to HuSi in a while, I figured I ought to let you know what I've been up to.

Your pal, Tom.

P.P.S. Did the robot's claws frighten you? (See poll.)

< I'm giving up. | BBC White season: 'Rivers of Blood' >
LetroTest lives! | 8 comments (8 topical, 0 hidden)
Bitchin' font, dude. by ti dave (6.00 / 1) #1 Thu Sep 09, 2004 at 09:07:54 PM EST
[gnarly text]

I don't care if people hate my guts; I assume most of them do.
The important question is whether they are in a position to do anything about it. --W.S. Burroughs

You're right by Rogerborg (6.00 / 1) #2 Thu Sep 09, 2004 at 09:52:50 PM EST
Perl is way too easy to read and use.  Only Haskell is truly incomprehensible by design.

Metus amatores matrum compescit, non clementia.
Very cool by jacob (6.00 / 1) #3 Fri Sep 10, 2004 at 02:13:09 AM EST
I'll have to start working on a Scheme version when I can find some spare time.

What software did you use to make that slideshow, by the way? It's very nice-looking.


On the slideshow by tmoertel (6.00 / 1) #4 Fri Sep 10, 2004 at 04:20:40 AM EST
My presentation "system" – and I'm using the term loosely – is a collection of scripts held together by a Makefile that grew in ad-hoc fashion to support the needs of my talks. The spine of each presentation presentation is a .slides file, which is text file in Emacs' outline format that I run through a custom preprocessor (Haskell-based) to produce a LaTeX file. The LaTeX file uses one of the standard LaTeX presentation packages to generate the actual PDF version of the slides.

Recently I switched the back-side engine to the LaTeX Beamer package, and it's very good. (It's what I used for the LectroTest talk.) Compared to TeXPower, which I used previously, it has much more convenient means of creating hierarchies of slides and intra-slide effects like hidden bullet points and stepped highlights. The PDF output is also clean and professional looking.

The crappy robot cartoons were drawn (crappily) by me in Adobe Illustrator on Macintosh, saved in EPS format, and converted to PDF format for inclusion in the slideshow via "perl -015 /usr/bin/epstopdf blah.eps". (The perl -015 part is necessary because the EPS files have Mac-style line endings.) The professional-looking diagrams were created by tiny Haskell programs that use Functional Metapost, which rocks. (Unfortunately, Functional Metapost seems to be unmaintained, but the old sources work just fine.)

Do give LaTeX Beamer and FMP a looksee.

Write Perl code? Check out LectroTest. Write markup-dense XML? Check out PXSL.

[ Parent ]
Queuestion: by ti dave (3.00 / 0) #5 Fri Sep 10, 2004 at 07:45:55 AM EST
What's that font called? I need it.

I don't care if people hate my guts; I assume most of them do.
The important question is whether they are in a position to do anything about it. --W.S. Burroughs

[ Parent ]
Which font? by tmoertel (6.00 / 1) #6 Fri Sep 10, 2004 at 09:17:35 AM EST
I used at least two. The body text is none other than the inimitable Gill Sans. The LectroTest logo is set in Republik Serif.

Write Perl code? Check out LectroTest. Write markup-dense XML? Check out PXSL.

[ Parent ]
The latter. by ti dave (3.00 / 0) #7 Fri Sep 10, 2004 at 09:01:16 PM EST

I don't care if people hate my guts; I assume most of them do.
The important question is whether they are in a position to do anything about it. --W.S. Burroughs

[ Parent ]
Write-in poll option: by ajf (3.00 / 0) #8 Sat Sep 11, 2004 at 01:41:50 AM EST
Since I first saw the Schematron mascot, nothing else scares me any more.

"I am not buying this jam, it's full of conservatives"

LetroTest lives! | 8 comments (8 topical, 0 hidden)