Friday, December 4, 2015

Verifying PDF file contents

Sometimes the applications we are testing require some level of verification of reports that are output by the application under test under a variety  of formats. One of these formats is the ever popular PDF.

An option for verifying PDFs might be using PDFLib Text Extraction Toolkit from PDFlib GmbHT.  This toolkit appears to be very powerful and supports the extraction of text, images as well as all objects that make up a PDF file. 

The example below are using Squish's API to drive an application developed using Qt framework. As well, while the script examples here are in Perl, you can replace it with the language of  your choice. As long as it offers similar services to the programmer.

In the following example: 

The function get_pdf_objs conceptually uses the external application tet.exe (Text Extraction Toolkit) in combination with Perl’s system function to extract all text and images from a PDF file and store it in a txt file or image file respectively.

parse_pdf is not really a parser, in this example its only role is to produce an array of text strings that it will extract from the text file produced by get_pdf_objs. 

Using Perl or any other of the supported scripting languages, we can easily script the handling of PDF files using an external tool like PDFLib TET and a custom utility function developed for this purpose. The above example illustrates this concept; the process is outlined below: 
  1. Start the application 
  2. Navigate the UI until a report is generated
  3. Print that report 
  4. Pass the file to a PDFLib function to extract text, images, etc.
  5. Parse the extracted text, images, etc. and compare to known good values 

I'd like to know your thoughts via comments.

Saturday, October 24, 2015

Test Cases - Some Benefits Derived

On numerous occasions I've been involved in discussions regarding test cases with other folks involved in a software development life cycle where the overall accepted sentiment is that when all test cases are "executed" we are done testing; or testing has not been done unless test cases have been executed.

Unfortunately for our field, I've often found myself alone in trying to show that just because you executed 100 test cases does not necessarily mean that you did any testing at all. You could have, but this depends on the person executing the test cases and whether they understand test cases and how to use them in testing.

The below presentation I gave at a former employer that had a huge catalog of test cases with detailed steps and was still struggling with product quality. They were hesitant, however, to accept that they weren't really testing when purely executing the test cases. Mainly because they feared that all the effort that went into the creation and maintenance of the test cases would be wasted.

In the session I explained to them how they can actually use these test cases to get the maximum return on your investment in the creation of these artifacts. Test cases, since they are designed by subject matter experts, contain all the information that most real world users of your product will need to perform the very same functions; and since they contain detailed steps, are suitable for many applications.

Your thoughts, comments are welcomed.

Saturday, February 28, 2015

Virtual (Simulated) Users Rock!

I've been absent from blogosphere for the last two months because I've been busy working on a new assignment. This assignment has me knee deep into designing an automated web application framework for a leading SaaS company.

One of the first things I identified as being needed for this implementation of my framework (other than the basics: reporting, logging, etc.) was the need to have a group of virtual users (little robots if you will) that will mimic not only real users of the system, but their behaviors as well.

User State System
Enter the User State System. While the name may sound fancy this is nothing more than a series of database tables who's sole responsibility is to keep stored the state of each user as well as their interactions with the SaaS system and any artifacts they create along the way. When running tests in parallel (or even sequentially) keeping track of user state has the following advantages:
  •  Restore the entire user list to a known default state
  •  Restore a particular user to a known or default state
  •  Test scripts can share users and their current states (e.g. logged in, not logged in)
A user state system holds the application's user information (e.g. user name, password, email), as well as current user state as it relates to the application under test (e.g. user is busy, paid user, free user, security answers).

An Example
As an example, lets say you have a system that supports a SaaS for an accounting firm. You have SLA's in place that guarantee certain response times and load times as well as unscheduled down time. In a user state table for this type of system all users will share a common starting state, however this state is only guaranteed when the user record is first added or reset to a default state; from then on each user’s state will be represented in this table. In other words this is a dynamic table constantly being updated / changed by the running test scripts or it may be updated by resetting it to a default state either globally (for all users) or for a particular group of users (i.e. 1 or more users).

With a user state system implemented when test scripts execute they request, as part of the test setup, a specific user with a specific set of characteristics from the user state system. The specific set of characteristics would be determined by the features / functions being tested. Test scripts then execute using the requested user. For example if you wanted to test invoices being generated properly you would request a user that has the appropriate rights (i.e. is able to add invoices) and that has an account configured with a budget with enough money to satisfy the invoice. Or conversely a user that has an account configured with a budget that does not have enough money to satisfy the invoice.

As you can see in the above example you can effectively use a state system to pinpoint specific functionality as required by the test you are designing without the need of building this test data into the test. This level of abstraction gives the test designer, and the automated test system, the flexibility and speed that is gained from using shared virtual users with a known current state.

As well, although there is no one size fit all solution when it comes to saving user state, there are some common features that can be shared in any user state system that will be utilized for functional verification in QA or even for TiP (testing in production). I will share some of these in a future post.
Creative Commons License
VGP-Miami Web and Mobile Automation Blog by Alfred Vega is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License.