Sunday, July 24, 2011

So you wanna play hardball huh?

Developing automated tests to verify web site functionality is already a major task in any web application testing effort. But what if you also had to validate the integrity of the html document itself? Or what if you've been tasked with designing a system that monitors websites html and alerts when changes are encountered?
Consider this approach to data driven tests when the integrity of the html code is also being verified against known (or at least last known) good values. Last known because the html element or its attributes can change at any time, however, using this approach, you'll be alerted of changes by your test scripts on the first test run after the change. This method is especially useful in scenarios such as the one depicted here: test page objects have been defined and implemented and now must be maintained. Meaning that when things change on the page the object will have to be updated. See diagram at the end of the post.
  1. Assign each web page a unique id
  2. Since a page can have multiple objects, page objects also have unique id
  3. During test setup, load all the test data into variables from your scripts
  4. When the page objects are instantiated it loads the locators it needs for the test execution from the database
  1. Ease of maintenance
    • If your locators change, your page objects dont have to change
    • When locators change update the db with new locators
  2. Programming skills not needed to maintain the locator store
    • A person can be trained to use browser developer tools to determine new xpath
    • New xpath can then be easily entered into db by non technical person using db web forms access
  3. Notification whenever html changes are made to documents
  4. But the number one benefit, IMHO, by far is the resuction in test software maintenance costs
I will be the first to admit that that there is no cookie cutter solution to atomating web application tests (although Selenium comes close!), This is just one example I'm throwing out there to, not only add to your arsenal of solutions, but to get discussions going across the globe on this topic as well. It is my opinion that one of the main objectives of any automated software testing initiative be easy to maintain. This solution helps in that regard.

diagram 1

We welcome comments, feedback and questions!

Thursday, July 21, 2011

Automation System, Explained - Part 2 "Components"

This is Part 2 of, well, Part 1 :). I will outline the system components that make up this functional and web test automation tool implemented using all open source software (Linux, Selenium RC, MySQL, Perl). The challenges our client faced and we aimed to address were mainly:
  • multiple dynamic sites, structure changed often
Which in turn led to:
  • too much maintenance time spent on test scripts.

Since there were so many sites we decided to split the testing by site, although there would be a group of commonly used functions housed in a Perl package for reference by any of the classes or scripts. This conclusion was drawn after numerous discussions where the following was identified:
  • Some areas / functions of the sites where similar
  • Most sites implemented unique html layouts
  • Conditional access functional testing was required
  • Gender specific functional testing was required
  • Back-end validation was required
  • A central point of reference for automation system status was necessary
For this we created a structure that required a class per web site, or more. I say "or more" because, although this client has not had the need to yet, it is possible that your class can get so extensive that it becomes harder to manage and update. At this point, it may make sense to split the class into smaller, more manageable modules. I should mention that, although we implemented this using Perl, you can apply the same principles to any language and come up with a similar system.

So now have a class called WebSiteOne and another called WebSiteTwo. Both of these correspond to property one website and property two website. This website objects include methods that encapsulate users actions. Lets use a login page example. Both websiteone and websitetwo have a login page that they chose to implement differently, but even if they hadn't, I still would have two login methods (one per property). Just because in time, most likely these two websites will end up with slightly different  login pages. Login into the website is a function that we can test with Selenium. In order to achieve this all we have to do, and it can be this simple, is use IDE to record the function. Then replace the hard coded field data with your own test data from a database or wherever, add any other logic that is required to test the function such as loops, conditions, etc., and you're done.

When you think about it; all the method is doing is performing a function that a human normally would on the site. These functions vary depending on the context of the web site. Here are some examples:
  • Signup to a newsletter
  • Upgrade your account
  • Check your transactions
  • Log off
  • Log on
As a test developer you may also have access to methods that things like:
  • Verify images are displayed on the screen
  • Verify certain text and links are displayed to the user
  • Capture a screen shot
  • Writes specific test debug info to a specific location, etc
This modular approach made sense for this client mainly because there were so many properties to develop and later maintain tests for.

The next component is made up of test scripts that make use of the exposed methods of WebSiteOne and WebSiteTwo. These scripts are nothing more than a list of functions that have been grouped together based on what made sense. For example using the same login example; you may have a test script that tests the login function of the web site under positive conditions (correct user name, password, etc) and another test script that tests the login function of the web site under negative conditions (wrong user name, password, etc). The idea is that for every function that  a human can do on a site, there is a corresponding method that you can call in your scripts to perform the action.

Most functional tests need some type of data input so that they can perform their functions. We chose a data driven test approach because of the massive amount of test data that would be consumed by these tests. Here are some of the factors that went into our decision to implement a database into our design:
  • As mentioned, lots of permutations, meant lots of data
  • Ease of maintenance
  • Faster execution times (vs using flat text files or csv's)
  • Test data integrity
The next logical step was to implement a dashboard that will present the test results to all interested parties via the web and without the need to install any other plugins. For this, Perl came to the rescue again and we implemented a web based dashboard that is constantly mining the database's test results tables and making that information available as soon as the tests are done. The dashboard is extendable and can be customized to add any data set we choose.

As a matter of fact, the entire system is extendable in that if you add functional areas, you add more scripts to your test arsenal. If a new property (web site) is added then we add a new class. Lets say for example you want to be able to track how many times a specific credit card biller is used in your signup form. All you would have to do add this functionality to the test framework is add a table to track the biller, a method to the class that will do all the work, and then call this method from your scripts and display this info on the dashboard.

So far, this system is running without any major hiccups and have even survived a partial migration from Selenium RC to WebDriver that was done by the customers staff with almost no support from us.

We welcome any feedback, questions or comments and look forward to interacting with you.

Tuesday, July 12, 2011

Data driven functional testing using Perl and MySQL

So I stumbled upon this conversation, I don't remember how, but the question was asked about a "quick sample", if there is such a thing, of using a database to store test data for functional tests. Specifically this question was asked in a forum of Selenium users.

I find that Perl has a good set of modules for, well for EVERYTHING! literally! Including interfacing with databases. It is very well documented by the way. Check out for complete details as well as online docs with loads of reference material.

But basically in order to get data to be used in the test scripts you can do the following (again, this is in Perl and I'm using DBI with the msql driver):

1. Define the test data by writing the sql queries needed to get the data from the db
2. Create a connection to a database
3. Execute the query and stuff the data into an array
4. I then iterate through the data in the array until there is no more data.
5. Cleanup

In Perl here is an example:

use DBI;

my $query = "SELECT email, pword, open_url, provider_name
             FROM messaging_test_data_tbl 
             WHERE is_active = true 
             LIMIT 4";

my $dbh = DBI->connect("DBI:$dbtype:$dbname",$dbun,$dbpw,
            { RaiseError => 1, PrintError => 1, AutoCommit => 1 });

my $sth = $dbh->prepare($query);

          or die "Cannot execute the query: $sth->errstr\n";

### AND THEN ITERATE USING fetchrow_array ###
while ($email,$pword,$open_url,$provider_name) = $sth->fetchrow_array())

 do all of your test stuff with the first set of data, second,
 within this while loop


Basically, no matter what language you are using you can use a similar flow. And don't forget to check out the linked and referenced documentation. Its a good reference; no matter what language you are programming your tests in.

Tuesday, July 5, 2011

Automation System, Explained - Part 1 "The Classes"

A reader and fellow engineer suggested we start the "break down" of the automation system we introduced here last week by explaining the Selenium commands abstraction process. I will also explain the rationale behind our choosing a paged object design pattern for the implementation. The website used in the example for this exercise, is fictitious and did not exist when I published this.


  • Dynamic sites where element id's change often, especially the signup and login pages.
  • Automated tests failures usually alert the QA team of element changes
  • Element name updates have to be made to all of the scripts in the suite
  • Test engineers, at this company, were directed to concentrate only on test logic and were spending much time learning the Selenium-RC API.

Our Solution:

It is worth noting that we did spend a good amount of time understanding how our client's customer's used the site. Had we not done this, the resulting system would have most likely failed. Our proposed system has the following advantages:
  • Abstraction of Selenium commands
  • Test engineers will have access to and interface with exposed methods
  • Test scripts will be unaffected by element name changes. Other than failing tests, the logic of the test script will not need to be changed.
  • Whenever element names change, only the specific methods in the class that are affected need to be updated.
  • Learning curve for new test engineers is decreased because there is no Selenium to learn; well not right away. has a login page where users access the members only area.

Login form:
<form method="POST" action="main.php" id="login_form">
    <td><input style="type="text" name="email" value="EmailOrUsername"></td>
         <td align="left"><input type="password" name="password" value=""> </td>
         <td><input name="remember_me" type="checkbox"></td>
          <td> remember me</td>
      <input type="submit" value="sign in" name="login_submit" id="login_submit">

Login method:
sub login {
    my $self = shift;
    my $print_heading = shift;

    if ( $print_heading == 1 )  {

      write_log(4, "\nusing browser: $self->{browser}\n", $self->{log_file});
      write_log(4, "using email: $self->{email}\n", $self->{log_file});
      write_log(4, "using pword: $self->{password}\n", $self->{log_file});
      write_log(4, "using gender: $self->{gender}\n\n", $self->{log_file});
    # Open the home page
    $self->{sel}->open_ok("/", $self->{browser});

    # Inject 1000 ms in between selenium calls

    # Enter email address and Password then Click Sign In button
    $self->{sel}->type_ok("email", $self->{email});

    $self->{sel}->type_ok("login_password", $self->{password});

    # Set time between selenium calls back to 1000 ms

New object method:
sub new {

    my $class = shift;
    my $self = bless { @_ }, $class;

    return $self;

As you can see from the login method above there is a lot going on there for a simple login function. This is a very simple login page, by the way.

In order to use the login method above we first have to instantiate an object from the class. For this our class includes a new method that returns to us an object instantiated with all of the test data that will be used in the test execution.

In the test script:
Only test logic will be included in the test script. So for the example above:

use SomeWebSite;

my $swb_obj = SomeWebSite->new(
                               gender => $gender,
                               browser => $browser,
                               test_site => $test_site,
                               email => $email,
                               password => $pword 

undef $swb_obj;


In the above example the test engineer has effectively interfaced with a web site using Selenium without using a single selenium command; directly :). Writing all of those selenium commands every time you write a test that includes login into the page is cumbersome as well. Granted you have to at least tell the test how to use selenium or even to start it! We have a method for that as well.

Setup Selenium method:

    sub setup_selenium {

    my $self = shift;
    my $sel = Test::WWW::Selenium->new( host => "localhost",
                                        port => 4444,
                                        browser => $_[0],
                                        browser_url => $_[1] );
    return $sel;

In order to be able to use the same selenium instance throughout the test, we create a selenium object and include it as a value when creating the page object.

Lets update the example above to include the selenium initialization, changes highlighted in yellow:

use SomeWebSite;

my $sel =SomeWebSite::setup_selenium(undef, $browser, $test_site);

my $swb_obj = SomeWebSite->new(
                               gender => $gender,
                               browser => $browser,
                               test_site => $test_site,
                               email => $email,
                               password => $pword,
                               sel => $sel 


undef $swb_obj;


Now once you log in, you might want to check some text on the page. The check_text method will do just that.

Check text method:

sub check_text {

    my $self = shift;
    my $text = shift;
    my $mode = shift;

    given ($mode) {
        when "regexpi" { $self->{sel}->is_text_present_ok("regexpi:$text"); }
        when "glob" { $self->{sel}->is_text_present_ok("glob:$text"); }
        when "exact" { $self->{sel}->is_text_present_ok("exact:$text"); }
        default { Custom::GPSubs::write_log(4, "Search MODE was not supplied\n", $self->{logfile}) }

Lets continue modifying the script above and check for the text "i am here", "you are there", "we are nowhere", changes highlighted in blue:

use SomeWebSite;

my $sel =SomeWebSite::setup_selenium(undef, $browser, $test_site);

my $swb_obj = SomeWebSite->new(
                               gender => $gender,
                               browser => $browser,
                               test_site => $test_site,
                               email => $email,
                               password => $pword,
                               sel => $sel 


my @check_text = "i am here", "you are there", "we are nowhere";

foreach my $text ( @check_text ) {

         $swb_obj->check_text($text, "regexpi");

undef $swb_obj;


In the example above, we first create an object with the new method. We then call the different methods to "do work" for us. In the example above selenium knows what to use for email, password and browser because when we created the object we defined all of these parameters (email, password, etc), and it uses it to login to the page. At the end of this sub execution the page under test will be left at the screen immediately after the login screen (i.e. the users Home Screen).

We can now do things like check what page we are on, using another method, get_current_page_url and once verified we are on the correct page, do some text verification using the check_text method.

As you can probably decipher by now, pretty much any "function" that needs to be done by a user, which will normally include a few selenium commands, can be abstracted in a class and expose human readable methods to test engineers in charge of writing automated regression tests. This is, by the way, where we recommend you use the most automation, in your regression test strategy.

In the next post in this series I will outline the system components and talk about the challenges faced and how we over came them.

We encourage feedback and dialogue so feel free to ask any questions or leave any comments. The lines are open... :)

Sunday, July 3, 2011

Page Object anyone?

I've heard a lot of discussions lately regarding the implementation of the Page Object design pattern for test teams that are struggling with web applications constantly changing element names and id's. Granted that most of the times developers do a great job at maintaining their element id's uniform throughout the life cycle of the page, however, sometimes such dynamic id's are required by the web site for a range of other reasons. Care to post any? :)

I remember one instance when we implemented page objects for one of our clients, whose goal was not only to alleviate them having to rewrite their failing tests whenever an element id changed on the page, but also to abstract all of the complicated selenium commands from the test developers, freeing them up to concentrate on coding test logic. They had a team of sharp Test Engineers capable of writing Perl code, and named one of them to become the Selenium SME. This subject matter expert would be the one charged with creating the different page objects and exposing methods to interface with the page.

In order to achieve the above goals, we decided on a class for each web site (their company owns numerous websites) that exposes methods to do things like:

  • Signup to the site
  • Login to the site
  • Buy a pay subscription to the site, etc

In addition we also added methods that facilitate numerous tasks that QA engineers perform in their scripts while testing a web site:

  • Check for text on the screen
  • Verify an element is present
  • Check the current page location (url), etc

Below is sample code from one of the methods (check_text) from the first class we created for them. It takes two arguments: the text being sought and the search "mode".

$obj->check_text("find a person", "regexpi");

searches for the text "find a person" anywhere on the page using regular expression mode
Supported modes are:


to add more modes just add more cases to the switch statement:

sub check_text {

    my $self = shift;
    my $text = shift;
    my $mode = shift;

    given ($mode) {
        when "regexpi" { $self->{sel}->is_text_present_ok("regexpi:$text");  }
        when "glob" { $self->{sel}->is_text_present_ok("glob:$text"); }
        when "exact" { $self->{sel}->is_text_present_ok("exact:$text"); }
        default { write_log(4, "Search MODE was not supplied\n", $self->{logfile}) }

Then, in my script I just instantiate an object, and then use it:

my $text = "some text";

my $obj = PageObj->new();

     if ($text =~ /\*/) {
           $mode = "glob" 
     } else { 
           $mode = "regexpi"; 
     $obj->check_text($text, $mode);

As you can see calling check_text passing in a string is a lot simpler to write than having to know all of the selenium commands required for this. Again, this was important in this organization because they wanted their  test developers to focus on writing code to test functionality and not necessarily have knowledge of the web page under test element id's, names, etc.

Web automation system using Selenium-RC, MySQL and Perl

A short while ago we were contracted by a client to design an automation system for some of their web site's functions. Off the shelve solutions, such as QTP, were out of the question because our customer was constraint by a small budget.

Our team understands that not all web sites are created equal, and, in order for any automation initiative to be successful a thorough analysis must be performed of not only the web application but the business as well. Since our client's QA team, while technically savvy, were not programmers we decided to implement a page object design pattern where all of the "magical selenium commands" where abstracted in a class and exposed to the test developers as methods they would call from their test scripts.

The resulting implementation, as seen in the diagram below, is currently deployed and keeping management and interested parties up to date on the status of key functional areas of the web site.

In the coming weeks, we are going to be covering each implementation step in more detail. We look forward to any feedback!

Saturday, July 2, 2011

Requirements? What requirements?

Recently we came across an organization that was following a development cycle that was so agile that things went live before they compiled!
All kidding aside, have you ever found yourself or your organization in a similar situation? Below linked diagram depicts a possible solution to this ever so common scenario:

In this sample scenario the company uses a central DOC where all features and requests for work to be done by the software team are listed and prioritized. The dev team then goes to work on the features, after they are assigned and report status in a scrum style weekly meeting. The management team meets once weekly where status is reported on the PRIORITY items.

As an output, this simple process will produce the following artifacts:
1. A defined set of business rules for future versions
2. A manual test procedure
3. An automated regression test

Can I capture a screenshot, in Selenium, when an error occurs?

I would think that you can probably just do something like this (in Perl):

     $sel->open_ok("$homepage") or
               and BAIL_OUT("Unable to open URL");

The above will first try to open the page defined by $homepage and if it fails
will capture a screenshot AND bail out printing "Unable to open

This was an interesting question asked to me by a colleague and, as well, we have answered it in user groups a couple of times so I figured it would be worth noting here.

Is there a way to save Selenium captured screen shots with timestamp in its name?

I use Perl to interface with RC and in all of my test scripts I always
generate a timestamp.
This timestamp is then used to identify all test artifacts including
screen captures as well as log and test files produced during the test
The example below is how I implemented it. Basically I wrote a perl
sub that does this work for me and I call it from any of my scripts.

sub get_timestamp
  my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime(time);

  $year += 1900;

  return $year . sprintf('%02u%02u_%02u%02u%02u', $mon, $mday, $hour, $min, $sec);

Handy timestamp subroutine. Returns formatted timestamp:

   my $unique_filename = 'filename_' . get_timestamp() . '.out';

will create a file named filename_20100915_144955.out

Please NOTE that formatted means formatted for my needs; to change
displayed timestamp format, manipulate sprintf as needed.

Run loop forever

Strangely enough, this question was asked to us. And there can be instances where the use of a "forever loop" would be appropriate. If you ever need to implement this, it is very simple.

In Perl I'd do it like this:

while (1) {
test stuff here


for (;;) {
test stuff here

That's it! Its really that simple ;)

How to identify frame name in a web page

This is especially an issue for folks trying to test their scripts out on Gmail for example.
Well, luckily for our audience we have crossed that bridge before and here is how we did it:
NOTE: In our test I am testing multiple email providers (gmail is just one of them)

if ($provider_name eq "Gmail"){
  $sel->type_ok("Email", $email, 'Enter Email');
  $sel->type_ok("Passwd", $pword, 'Enter Password');
  $sel->click_ok("PersistentCookie", 'Click stay signed in');
  $sel->click_ok("signIn", 'Click Login button');
<<<<<<<<<<< this is the key.
<<<<<<< then select the "canvas_frame"
  # more test code here

Your test will always fail if you do not select the frame by using a locator such as name.

How to create a selenium test suite using Perl

Selenium is becoming increasingly popular now a days, mainly because, well it's FREE! :)

Recently, while checking out one of the Selenium automation forums we monitor I came across the question of how to run multiple test script written in Perl as part of a suite of tests. While I am sure there are many ways of implementing this, here is one of my methods of choice in Perl. Another option is to use a MySQL database. Ask me how!

creating a driver script that calls
all of your Perl scripts in order. This driver script could be a
windows .bat file, Linux shell script or even a Perl script that
simply contains n system calls to your individual scripts.
For example:
Assuming that your test scripts reside in the directory C:\Automation\Scripts\, this is how i would do it in perl.

  my $result_dir = "C:\\Automation\\Scripts\\"
  opendir (my $dh, $result_dir) or die "can't open dir: $!";
  while (readdir $dh){
   if ($_ =~ /pl$/){
     system ( $_ );

The above:
  1. Opens the directory C:\Automation\Scripts
  2. Reads the directory
  3. If any of the files contained in it end in PL it gets executed by the system call
Files that end in .pl are assumed to be Perl scripts.
Simple isn't it? :)

Introduction to VGP-Miami

The mobile and web application markets have become increasingly popular in the recent past, not only in the US but abroad as well. The total number of web page visits is now greater on mobile devices than it is on a stand alone PC.

We are a software and engineering services power house located in Lima, Peru [VGP::Peru] with offices in Miami, FL [VGP::Miami]. Our core competencies are Web Application development, Automation and Quality Assurance. We develop and test applications for the web as well as mobile devices (iPhone, Android).

Via this blog, we will attempt to share our experiences and challenges while developing and testing advanced applications for the web as well as mobile devices.

Our staff of talented Software and Quality Assurance engineering professionals span two countries (US and Peru) and collectively possess over 50 years of verifiable experience. We look forward to the opportunity to serve you as well as providing guidance and answers to any questions you may have.
Creative Commons License
VGP-Miami Web and Mobile Automation Blog by Alfred Vega is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License.