BluePrint: Functional Test Suite

Introduction

Creating an automated testing suite using ​Robot Framework which is configurable, easy to write, easy to maintain and provides in depth logs and reports.

Functional testing needs to be an integrated part of the development process. A maintained suite of functional tests:

• Captures user requirements in a useful way
• Gives the team (users and developers) confidence that the system meets those requirements

The current Functional Testsuite (using just selenium) has blurred lines between script action and code. The tests break often because they can not adapt to changes in requirements or system's implementation. The tests tend to become out of sync with either of them, people stop running the tests or stop trusting the results.

This will add value to Sahanna as it will provide a handy and usable tool to script and run tests, to diagnose failures and to empower people to use and maintain the test suite.

Stakeholders

• Developers - They will be the users as well as maintainers of the suite.
• People using Sahana Eden - They can see which all functionalities are working as expected.

User Stories

• A developer can use the suite to confirm that the system does what users are expecting it to and get quality in-depth report and logs to help fix the functionalities that are not behaving as expected.

Requirements

Functional

• Detailed reports and logs. It should provide:
  • Tickets
  • Screenshots
  • Tracebacks
  • Summary reports
  • Logs of actions step by step
• The tests should give immediate feedback.
• Complete system testing?.
• Ability to add a single test as well as group of tests.
• Ability to group tests by template and functionality.

Non-functional

• Low maintenance cost
• Easy to understand
• Easy to enhance
• Fast
• Report should be:
  • Structured
  • Easy to understand
  • Precise

Design

Testing Architecture

Directory Structure

• Everything is bundled under one directory which can be called FunctionalTests.
  • TestRunner.py can be used to provide to a wrapper to easily call the testsuite or a group of tests or tests on a particular template through command line arguments.
  • robot contains the implementation and the execution aspects of the functional tests using robot framework.
    • In the implementation side, all the testsuites are grouped according to the modules inside the testsuite folder.
    • All the resources go to the resources folder which will have various keywords grouped according to the functionality they serve.
    • Inside execution folder, settings holds the specific variable files. It should be noted that especially the settings are only written once in the beginning of the project and are very rarely changed afterwards.
    • Inside the lib-folder we check-in our own Test Libraries, but it might also make sense to store ready-made libraries here to ensure they are kept stable.
      • Eden Test Library can be used to implement Eden specific functionalities for eg: a keyword 'Run on templates IFRC default' or 'Not run on templates default' etc.

Use Case Diagram

Technologies

• ​Robot Framework

Implementation

This is being implemented as a GSoC project

• Proposal : ​GSoC 14 - Functional test suite using Robot Framework by ​Arnav Sharma
• GSoC page : GSoC/Testing

Current Implementation

​edentest by nerdis

Automated tests Selenium (without using robot framework)

Future Extensions

• Robot Framework has a plugin for Jenkins which can then be merged into the CI server.

Outstanding Questions

• Formalization of the deeper structure of resources, tests inside testsuites and eden test library.
• Best way to involve the community in the process of writing tests.

References

​Blog post by Andreas Ebbert-Karroum on How to structure a scalable and maintainable accpetance test suite

​Writing Maintainable Automated Acceptance Tests, Dale H. Emery

​Robot Framework User Guide

---

BluePrint