Saturday, March 2, 2013

Qt application Keyword driven verification

Recently I had the pleasure of designing a test framework for a Qt based application that incorporated Squishserver, Squishrunner and the Squish IDE initially as well as a system of keywords developed in house with the sole purpose of executing some actions that verified some requirement.

The below diagram illustrates the major components of the automated system design. Below it is a brief explanation of each part:
Figure 1 - System Layers (abstraction)

Test Execution Layer

This is the only layer that faces the user (i.e. the test script programmer). From here we make calls to the methods / services exposed by the functional layer. For example: create_report, load_data. Pass / Fail status is determined at this layer (based on 1 or 0 returned from functional layer).

Functional Layer

This is the ONLY layer that speaks AUT language and, as such, drives the AUT. It makes calls to the System Layer as needed to access services like the DB, capture screenshots, write to log files, etc.
All actions and functions that we need to perform on the AUT (application under test) are defined here and made available to the test execution layer. All functions should return 1 when successful or 0 when unsuccessful.

System I/F Layer

The System Layer is responsible for interfacing with any other component that is not the AUT. It should provide services for db access, reading and writing. Log file creation and parsing. Test results processing and QC integration (post test run).

Keyword Testing Framework

Using the keywords approach the separation / abstraction of layers will be supported by having two sets of keywords. Both sets of keywords are exposed as methods in a Perl library file and are:

Functional keywords: These keywords interact with the AUT and define the actions to be followed when the keyword is invoked. 

System keywords:  Interface with the system and provide services to db access, log files, copying and moving files, launching the application, etc.

Keywords System Diagram

As shown above, the system includes 4 major parts: 

  1. keyword / actions definitions (
  2. data files (keywords+arguments), can be .tsv, .csv, .xls (keywords.csv)
  3. test case file (
  4. driver (

Each part has its own responsibilities and functions to carry out (as explained below). In conjunction they drive the Application Under Test and capture test artifacts along the way. Including execution PASS / FAIL determination and reporting. 

Data files in the form of tsv, csv or xls contain “keyword” and “argument” data separated by a tab, comma, etc depending on file format used.

The keywords are the method names used in the function definition. For example, start_application keyword is mapped to start_application Perl sub. The “arguments” in the data file are the functions parameters. They are like passed in parameters, being provided by the keyword data file.

Before the keywords with arguments included in the keywords data file can be executed (called upon) they must be turned into the scripting language specific format. This is the job of the file. It parses the data file constructing the method name as it goes and then eventually executing the function call using, in our case, Perl eval function.

The function call defined in the action class is now called upon with the arguments in the data file passed in. The actions are executed on the AUT and success or failure is reported back as 1 or 0 respectively.

Contact us if you'd like to hear more or if you'd like to hire us to provide a similar service.

1 comment:

Creative Commons License
VGP-Miami Web and Mobile Automation Blog by Alfred Vega is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License.