|Version 11 (modified by 9 years ago) ( diff ),|
BluePrint: Test Suite
Table of Contents
This blueprint outlines the development of the automatic testing framework. This automatic testing framework will provide robust testing of the code-base and ensure proper maintenance of the code.
Whenever some changes are added in the current code, it has to be validated and tested upon with the integration with the other components of the code. So, this framework will provide this support.
With tests running on a scheduler, continuous testing can be done. This is important to Sahana, seeing the rapid movement of the code-base. Currently, Sahana has automatic test framework, whose details can be found here
- Developers - With an automatic testing framework setup, it will be relatively easier for the developers to test their changes in the code.
- People who want to deploy Sahana - They would like to run the tests to ensure that the system is integrated well and ready for deployment.
- Sahana as a service - Automated testing will provide Quality Assurance to it’s clients.
- Bug Marshalls
- Developers will run the test suite on making changes to the code to test if their code works with the integration of the system. If their changes do not break the tests, then their code is viable for merging.
- Developers may also see the test results mailed to the list to see the possible bugs introduced into the system.
- On a regular basis, the "bug marshals" review the test results sent by the CI Server and see if the reported negatives are false negatives or true negatives. If they are false negatives, they fix the tests or log a ticket on the trac for a bug in the testing code. If they are true negatives, they log a ticket on the trac for a bug in the code on which testing is done or fix the bug themselves.
- People who want to deploy will run the test suite to check the functionality of the system.
Use Case Diagram
- Maintain the CI Server to run the tests periodically and send an aggregated report for all the templates via email.
- Adapt tests to meet needs of evolving CI Server(SysAdmin/ContinuousIntegration). The tests should successfully run on the CI Server both locally and remotely.
- Run Selenium Tests in multiple templates with multiple user accounts.
- Create an abstraction for the Selenium Tests so that they are easier to read, write tests and the tests are more robust.
- Extend Role Tests to run on different templates(Currently is limited to the IFRC roles for RMS)
- Run Smoke tests in multiple templates.
- Load Tests : Will measure performance of Eden given the load under both normal and anticipated peak conditions.
General To All Tests
- Clearer Error Messages in a way that anyone can reproduce them.
- The CI Server should fetch the latest code from Github Repository and run the tests on the latest code fetched.
- The CI Server should run all the tests across templates on a daily basis both locally and remotely. It should save the aggregate report of the tests on the server and as well as mail it to the concerned people. People may also subscribe to receive the test results via email. The previous implementation of the CI Server saved the results at http://18.104.22.168/RESULTS/. The same location can be used for storing the test results.
- For each test, the report should include the template it is run on, test name and location of the test script in eden. In case of failure of the test, it should include the traceback and the record details it was intended to work on. In case the test passes, it should include the test name and the pass status of the test.
- These tests should run against each and every template where the target functionality is available. For templates where the functionality is not available, the test should auto-deactivate.
- Given that Sahana is a web-based tool, Selenium tests should be made robust, easy to use. The main idea is to have a set of functions(create, search, edit) which will take the tablename, labels of the field and the record data as input.
- The Test suite should automatically check the type of field(option, autocomplete, text, date, datetime, etc) being used in the current template and fill the form accordingly. With this, we can create a code-template which just needs to be copied, pasted and fed in the record when creating a new Selenium test. This will ensure that writing Selenium tests are as easy as giving a test case.
- A sample code-template can be of the form -
# Create Method def test_<testname>_create_<module_name>(self): self.login(account, nexturl) # already implemented # The data which is to be added to the table data_list = [ ( column_name1, data1 ), ( column_name2, data2 ), .. so on ] self.create(tablename , data_list) # Search Method def test_<testname>_search_type_<module_name>(self): self.login(account, nexturl) # already implemented search_fields = [ ( column_name1 , [label1, label2, ..] ), # the list of labels are those which # are to be searched for under column_name1 ( column_name2 , [label1, label2, ..] ), .. so on ] self.search(type, tablename, search_fields) # type can be simple or advanced # Edit Method def test_<testname>_edit_<module_name>(self): self.login(account, nexturl) # already implemented # The new data which is to be edited in the table edit_list = [ ( column_name1, data1 ), ( column_name2, data2 ), .. so on ] self.create(tablename , edit_list)
- So, the workflow of the developer will include running the unit tests to see if the new changes break anything and providing a Selenium test for the newly written module, which will be run in Continuous Integration.
- Command line arguments need to be passed to specify which browser the tests should run (Firefox and Chrome are already implemented)
- The role tests are currently run only on the IFRC template. The test suite should automatically generate data for multiple templates so that the role tests can be run on them as well.
- The load tests will test the amount of load Eden can take under normal and peak conditions. So, the load tests should be divided into phases, with each phase incrementally increasing the number of users and requests. Features of Tsung can be used to do this effectively.
Module Enabled or Disabled
- To ensure that tests(role, selenium, smoke) work across templates and do not show false negatives when run on a different template, a check has to be made if the module it is to work on is enabled/disabled. If the module is disabled, then the test should automatically deactivate and an appropriate message should be included in the test report.
- For setting up the CI Server, some of the technologies which can be used are given here - ContinuousIntegration
- For Continuous Integration, an instance of Eden can also be used which will enable Scheduling, enable subscription of notifications for test results, can also provide formatted results.
- For functional tests, currently, Selenium and Smoke Tests are used. The current Selenium tests should be made more robust and they should work across browsers. Currently, there is support for Firefox Webdriver(upto version 16) and Chrome Webdriver. We need to provide support for Safari Webdriver, Opera Webdriver and Internet Explorer Webdriver.
- The Role tests(which currently run only on the IFRC template and uses Selenium Unit Test) are to extend to run on multiple templates.
The Seleium tests can be run on mainly IFRC template. With some changes, they can be run on the default template as well. However, they don’t work across templates.
The unit tests expect that some particular modules are enabled in the template. If they are not enabled and unit tests are run in that template, then false negatives are reported.
Current implementation of the selenium tests, smoke tests, role tests can be found here -
Unit tests and benchmark tests can be found here -
Blueprint on General Ideas - BluePrintTesting
Current Implementation - QA