Generic Test Automation Framework Design

Test Automation Framework Design

When test automation is taken to highest level tests can be started with a push of a button, they can be left running over-night unattended and next morning test results are published. This kind of automation clearly requires some kind of a system which makes creating, executing and maintaining test cases easy. The system should provide some core functionality (e.g. monitoring and reporting) and allow extending itself to make it possible to create new kinds of tests.

Characteristics of a good Test Automation Framework

1)      The framework MUST execute test cases automatically. That includes also for example verifying results, handling errors and reporting results.

2)      The framework MUST be easy to use without programming skills.

3)      The framework MUST be easily maintainable.

4)      The framework MUST be able to execute tests unattended.

5)      It MUST be possible to start and stop test execution manually.

6)      It SHOULD be possible to start test execution automatically at predefined time.

7)      Non-fatal errors caused by the SUT or the test environment MUST be handled gracefully without stopping test execution.

8)      Every executed test case MUST be assigned either Pass or Fail status and failed test cases SHOULD have a short error message explaining the failure.

9)      Test execution MUST be logged.

10)   Test report MUST be created automatically. Test report SHOULD be published automatically.

11)   The framework MUST have coding and naming conventions.

12)   The framework MUST be adequately documented.

High level Framework design

 

Components of a Automation Framework Architecture

Test Design System

Test design system is used for creating new test cases and editing existing ones. It is test engineers’ and domain experts’ playground and must be easy to use with minimal training and without programming skills.

The created test data can be stored into flat files or databases. A simple solution is using an existing tool like spreadsheet program or HTML editor as a test design system. A special test design tool made just for this purpose would of course have its benefits but building something like that is out of the reach for most automation projects.

 Test Monitoring System

Test monitoring system is used for controlling test execution and checking test results.

It should have at least the capabilities listed below.

• Starting test execution manually.

• Stopping test execution.

• Monitoring test execution while tests are running.

• Viewing test logs while tests are running and afterwards.

• Viewing test report.

Test Execution System

Test execution system is the core of the framework. Its main components are

a)      driver scripts,

b)      the test library,

c)        the test data parser and

d)      other utilities like logger and report generator

a)      Driver Scripts

Test execution is controlled by driver scripts which are started using the test monitoring system. Driver scripts are pretty short and simple because they mainly use services provided by test libraries and other reusable modules. 

b)      Test Libraries

Test libraries contain all those reusable functions and modules which are used in testing or in activities supporting it. Testing is mainly interacting with the tested system and checking that it behaves correctly. Supporting activities are often related to setting up the test environment and include tasks like copying files, initializing databases and installing software. Functions in the test library can do their tasks independently but they can also use other functions or even external tools.

c)       Test data parser

The test data parser  gets test data easily to driver scripts. Its task is processing the test data and returning it to the driver script in easy to use test data containers. The role of the parser may seem small but in reality it is the heart of the whole test execution system. 

d)      Logger

An easy and modular way to fulfill them is having a common logger component with the required features. Both the driver script and test functions use the same logger which in turn writes messages to the common test log.

e)      Report Generator

The reporting requirements of the framework can be summarized saying that a concise report must be created and published.

Working of a Data Driven Framework

Simple test scripts have test data embedded into them. This leads to a problem that when test data needs to be updated actual script code must be changed .This kind of reuse is also problematic because one particular in the tested system may require updating all scripts. Because of these problems embedding test data into scripts is clearly not a viable solution when building larger test automation frameworks. A better approach is reading the test data from external data sources and executing test based on it. This approach is called data-driven testing.

Step 1: Editing and Storing Test Data

Because data-driven test data is tabular it is natural to use spreadsheet programs to edit it. Another benefit is that spreadsheet programs are often used for simple test management tasks and in that case there is less need for storing same data in different places or copying it back-and-forth.

Step 2: Processing Test Data

Implementing a script for parsing data-driven test data can be surprisingly easy with modern scripting languages. Test data is first read into data variable and then split to lines so that header row is excluded. Data lines are then processed one by one. Lines are split to cells from tabulator character and cells’ data assigned to variables making test data available in the script for actual testing.

Step 3: Test Execution

The test scripts written are invoked from the data sheet and then executed. The test scripts will in short act on the application under test and execute the test cases.

Step 4: Logging and Reporting

The logger will log the data in a file after each step of execution and after the test is complete, the test reporting mechanism, will generate reports with status.