Wednesday, November 12, 2014

Testing with Perl - Test::More

The More module available via CPAN is part of the Test::* series and provides a wide range of testing functions. In the previous post we learned that TAP is just a simple text-based interface between testing modules that are part of  a test harness; Test::More is one of these modules that utilizes TAP and expands the "simpleness" of the protocol by providing various ways to say "ok". It includes, among other things:
  • Better diagnostics (not just OK / NOT OK, but also for example WHY)
  • Capability to skip tests given a range of criteria
  • Capability to test futures features (using TODO)
  • Compare complicated data structures
Lets jump right in and use the above referenced module to test a web application available at (this is a fictitious web site that does not exist). As part of our test we should:
  1. Navigate the login page
  2. Make sure we are in the correct starting point
  3. Enter our login credentials
  4. Login to the site
  5. Verify our landing page after the login operation.
Implemented as a Perl test script that uses Selenium::Remote::Driver and Test::More we might write the above as:
1:  $driver->get($page);  
2:  my $loc1 = $driver->get_current_url();  
3:  is ( $loc1, LOGINHOME, 'Verify landing page' );  # THIS IS TEST 1
4:  login_as( $user_id, $password );  
5:  my $loc2 = $driver->get_current_url();  
6:  is ( $loc2, APPHOME, 'Verify landing page after login' );  # THIS IS TEST 2

In line 1 we navigate to our starting page (the login page) using Selenium's get method. In line 2 we declare a variable that will hold the URL we just navigated to and that will be returned by Selenium's get_current_url method. Then, in line 3, we utilize Test::More's is() is function to assert that the page we landed on, '$loc1', is in fact the expected one 'LOGINHOME'. Line 4 executes a page object who's sole purpose is to login to the web application under test. After the login operation we once again get the url of the page we landed on, '$loc2', and compare that to the page we expect to be after the login which is 'APPHOME'.

NOTE: I used the term Selenium above for readability to refer to Selenium::Remote::Driver - the Perl binding to WebDriver. 

Below is the output that would be sent to the harness in a PASS case:
 ok 1 - Verify landing page  
 ok 2 - Verify landing page after login  

And the output for a FAIL case:
 not ok 1 - Verify landing page  
 not ok 2 - Verify landing page after login  

Whenever a test fails, one of the features of Test::More is that it gives you meaningful information (the whys) you can use when evaluating test failures. For example, in the above failure the following would be sent to the error output file:
 #  Failed test 'Verify landing page'  
 #  at C:\Users\Freddy Vega\SsApp\ line 107.  
 #        got: ''  
 #   expected: ''  
 #  Failed test 'Verify landing page after login'  
 #  at C:\Users\Freddy Vega\SsApp\ line 113.  
 #        got: ''  
 #   expected: ''  
 # Looks like you failed 2 tests of 2.  

As you can probably see by now, testing with Perl means not to re-invent the wheel every time a testing problem arises. In our solution we were able to use Selenium::Remote::Driver to drive our application under test (a web app). We used Test::More to make our assertions while we are testing and TAP::Harness to tie it all together and produce results that can later be mined, interpreted and presented to different audience types (management, users, developers, etc).

In the next post in this series I tell you about test harnesses and how you can combine these with other tools to help you design a robust automation framework.

Monday, November 10, 2014

Testing with Perl - TAP (the Test Anything Protocol)

While there are many programming languages out there that folks seem to prefer when implementing automation for web and PC based applications (e.g. Python, Ruby), I've been inclined to choose Perl just about every time I have been asked to solve some testing problem. There are several reasons, I believe, why I have always chosen Perl over other candidates, below are my current top three:
  1. Cross platform support for your tests (write once run everywhere)
  2. Powerful text parsing
  3. No need to re-invent the wheel (there are thousands of modules available on CPAN)
TAP (or Test Anything Protocol) is a "simple text-based interface between testing modules in a test harness. TAP started life as part of the test harness for Perl"; it should be noted that, even though it was originally designed for and used in the development of Perl itself, TAP now has implementations in C, C++, Python, PHP, Perl, Java, JavaScript, and others as well. So this should tell you that at least programmers have noticed and have found the protocol helpful enough to port it to their language.

Our purpose for TAP is very simple: to say 'ok' or 'not ok' in a standard way with the aim of facilitating communication between the tests and the services used to run / support the tests. That is we use the Test Anything Protocol to report their successes or failures. We can then use other tools to aggregate the information produced by our tests and present it in a human readable form. Below is a sample TAP stream so you can get an idea of what it looks like.

  ok 1 - Input file opened  
  not ok 2 - First line of the input valid  
  ok 3 - Read the rest of the file  
  not ok 4 - Summarized correctly # TODO Not written yet  

The above stream output states that we ran 4 tests (1..4). Two of them passed (1,3) while two of them failed (2,4). Simple, isn't it?

So what do you need to test with Perl? You need three things:
  1. The AUT
  2. TAP producer (e.g. Test::More module)
  3. TAP consumer (e.g. TAP::Harness module)
I'll go into details on each one of these on future posts but for now just know that a TAP producer is just a module that does "automation magic" (such as Test::More) and communicates its successes and failures to a TAP consumer (such as TAP::Harness) using the test anything protocol.

There are many testing modules on CPAN that use TAP to report their successes or failures. In this blog series we will focus on Test::More. We'll also cover (briefly) the other modules in the Test::* family of modules mainly just to be aware that they are there for you if you need them. To write robust tests in Perl to verify a web application or web site, however, you'll find that Test::More is more than adequate (no pun intended).

Creative Commons License
VGP-Miami Web and Mobile Automation Blog by Alfred Vega is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License.