Monday, November 10, 2014

Testing with Perl - TAP (the Test Anything Protocol)

While there are many programming languages out there that folks seem to prefer when implementing automation for web and PC based applications (e.g. Python, Ruby), I've been inclined to choose Perl just about every time I have been asked to solve some testing problem. There are several reasons, I believe, why I have always chosen Perl over other candidates, below are my current top three:
  1. Cross platform support for your tests (write once run everywhere)
  2. Powerful text parsing
  3. No need to re-invent the wheel (there are thousands of modules available on CPAN)
TAP (or Test Anything Protocol) is a "simple text-based interface between testing modules in a test harness. TAP started life as part of the test harness for Perl"; it should be noted that, even though it was originally designed for and used in the development of Perl itself, TAP now has implementations in C, C++, Python, PHP, Perl, Java, JavaScript, and others as well. So this should tell you that at least programmers have noticed and have found the protocol helpful enough to port it to their language.

Our purpose for TAP is very simple: to say 'ok' or 'not ok' in a standard way with the aim of facilitating communication between the tests and the services used to run / support the tests. That is we use the Test Anything Protocol to report their successes or failures. We can then use other tools to aggregate the information produced by our tests and present it in a human readable form. Below is a sample TAP stream so you can get an idea of what it looks like.

  ok 1 - Input file opened  
  not ok 2 - First line of the input valid  
  ok 3 - Read the rest of the file  
  not ok 4 - Summarized correctly # TODO Not written yet  

The above stream output states that we ran 4 tests (1..4). Two of them passed (1,3) while two of them failed (2,4). Simple, isn't it?

So what do you need to test with Perl? You need three things:
  1. The AUT
  2. TAP producer (e.g. Test::More module)
  3. TAP consumer (e.g. TAP::Harness module)
I'll go into details on each one of these on future posts but for now just know that a TAP producer is just a module that does "automation magic" (such as Test::More) and communicates its successes and failures to a TAP consumer (such as TAP::Harness) using the test anything protocol.

There are many testing modules on CPAN that use TAP to report their successes or failures. In this blog series we will focus on Test::More. We'll also cover (briefly) the other modules in the Test::* family of modules mainly just to be aware that they are there for you if you need them. To write robust tests in Perl to verify a web application or web site, however, you'll find that Test::More is more than adequate (no pun intended).


  1. Personally, I found TAP output less than ideal to distinguish between a failed TODO test vs a real failure (both "not ok"). Otherwise it is a very nice format allowing subtests and other features...

    However, the choice of anyone's pet language is a matter of taste and I believe the reasons given here are not valid - which modern language does not support cross-platform powerful text parsing with a ton of good libraries???

    I prefer Python for new projects, since I never understood this "more than 1 way to do it" paradigm of Perl and I am still confused about sigils, refs, prototypes, classes and the like in Perl:
    e.g. compare:
    if scalar(@{$a->{b}})>1 { &{${$a->{b}}[1]}($_[0], $_[1], $_[2]); }
    if len(a["b"])>1: a["b"][1](*args)

  2. Not sure if I'd leave it up to the protocol to distinguish TODO or any other type of test. I think that's the job of the Test Harness.

    Perl, while I do sort of love it, is not my pet language. C is actually my pet language.

    Are you saying that you cannot do something more than one way in python? I find that hard to believe. In fact I know you can do something in more than one way in Python, Perl or any other programming language for that matter.

    Regarding the comparison you gave:
    1. Some context?
    2. Can you apply either one of those to a testing problem?

    Thanks for commenting.


Creative Commons License
VGP-Miami Web and Mobile Automation Blog by Alfred Vega is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License.