wiki:BluePrint/Testing

Table of Contents

  1. Acceptance or 'Customer' tests
  2. Unit Tests (must do)
  3. Continuous Integration
  4. Regression Testing
  5. Documentation
  6. Boundary Testing (should do)
    1. Selenium
      1. Selenium Remote Tests
    2. Alternatives
  7. Roles Testing
  8. Integration Testing (good thing)
  9. Usability Tests
    1. Accessibility
  10. Performance Tests
    1. Load Tests
    2. Stress Tests
  11. Security Tests
  12. Smoke Tests
  13. Test Coverage

Blueprint: QA (General Ideas)

We have a Testing process in current use.
This page is for looking at potential improvements.

Behaviour-Driven Development takes Test-Driven Development further to focus on the Specification rather than the Verification using something like ​pyspec or ​PyFITto provide testable specs:

There are a huge number of Testing Tools available to cover the various parts of the Testing process:

A community available for assistance:

Testing procedure regarding checking in code and merging:

  • After a merge, run all tests.
  • If the tests don't pass, don't commit the merge.

In other words, all tests must pass before pushing to a stable branch. This does not stop buggy code without tests getting into a branch, so a possible future enhancement might be to ensure all new code gets tested, e.g. by ensuring 100% coverage.

There may be a script added to automate this.

Testing that Users/ customers should be doing:

Acceptance or 'Customer' tests

These are the highest level tests that check that the right thing has been built. These can often be manual tests, but automating them early on helps avoid wasted effort and disappointment by highlighting things the customer does not need. It may be enough to let the customer test the system for a few days to ensure satisfaction.

Testing that Developers should be doing:

Unit Tests (must do)

"Building the Code Right"
The current implementation of Unit Tests use python's unittest.
It's details can be found here - Unit Tests

Continuous Integration

The current implementation can be found here - SysAdmin/ContinuousIntegration

Whenever a commit is made it should be checked to see that it doesn't break anything

Note: As of January 2012, BZR/Launchpad info for eden is deprecated. Please visit the GitHub page. Thanks.

Alternate options which could be investigated:

  • Jenkins
    • Advantages
      • Easy to build Eden
      • Easy integration with selenium grid, so that the selenium tests can be run remotely and parallely
      • Git plugin -
        • Can trigger tests on a commit easily.
        • Can automatically publish build status on the git commit.
        • Can create tag/push to master automatically on successful tests after the commit.
      • Run tests on new pull request (GitHub pull request builder plugin)
      • Email notifications about the test results.
      • IRC notifications about the test results. (IRC plugin)
      • Add more nodes on which jenkins is run, so that the load is distributed
    • Setting Up
      • Jenkins is easy to install.
      • Installation of Selenium Grid needed.
      • Setting up some slaves for Selenium grid for cross browser, parallel selenium tests (Can be done on a single machine too)
      • Slight modifications in the Selenium test suite to incorporate Selenium Grid.
      • Configuring Jenkins to run tests on a git commit, etc, as required.
      • Configuration is done by an authenticated web interface.
    • Reference - ​https://wiki.jenkins-ci.org/display/JENKINS/Home
  • Selenium Grid
    • Advantages
      • Can run selenium tests across -
        • Different browsers
        • Different operating systems
        • Different machines in Parallel.
      • Works on the concept of hub and nodes.
      • The tests are run on a single machine - hub.
      • Execution will be done on different machines - nodes.
      • Will increase the speed of the Selenium tests since they will be distributed across machines.
        • Eg - Some nodes will run create, some will run the search tests.
      • We can then trigger the tests on a commit/pull request.
      • The time taken will roughly decrease depending on the number of nodes.
        • Eg - If there are 4 nodes, the tests will run roughly 4 times faster.
    • Setting Up
      • To install, need to download the Selenium Server jar file - on the hub as well as on all nodes.
      • Will have to change some initialization code for the Selenium tests where the Webdriver is initialised.
    • Reference - ​http://code.google.com/p/selenium/wiki/Grid2
  • Solano
    • Comparison with Jenkins -
      • Jenkins has a lot more plugins, which are written by a much larger community. So, it becomes more extensible
        • Eg - git plugin, IRC plugin, github pull request plugin.
      • Jenkins is easier to integrate with Selenium Grid, which enables us to run the tests in parallel.
      • Jenkins has a better documentation, easier to use.
    • Disadvantages -
      • Our tests run using python’s unittest. We will have to change it to run using nosetests or py.test. Also, on changing too, the tests will not be able to run in parallel.
      • Reference - ​http://docs.tddium.com/python/

Alternate options which could be investigated:

Regression Testing

Fired by dev after certain number of changes or whenever they like.

Documentation

As well as writing DocStrings in all functions, we can generate an overall API using:

If writing a separate manual then we can use:

Testing that Testers should be doing as part of ​Acceptance:

Boundary Testing (should do)

Building the Right Code

Checks functionality of modules against specs

This sees the application as a black box & so the same tests could be run here against both the Python & PHP versions, for instance.

Selenium

Sahana is a Web-based application, so testing should be from browser perspective:

Functional tests can be written using ​Selenium:

Selenium Remote Tests

The design for Selenium Remote Tests -

  • Prerequisite 1: There are some more instances of Eden with varying configurations (template/python version/database type) running on some servers(preferably in local network with the CI Server, as this will increase the speed of handling requests by the CI Server).
  • Prerequisite 2: There is an 'testing' instance of Eden running on the CI Server which contains the test suite.
  • The 'testing' instance of Eden will shoot the test suite, while changing the base URL in the test suite. The base URL currently is 127.0.0.1. If we change it to the IP address of some of the currently running instances of Eden on other servers, the tests will start running for those servers.
  • The above task can be done as a background process on the CI Server or running the tests using ​SeleniumGrid to run them in parallel.
  • The selenium tests should work across browsers. Currently, there is support for Firefox Webdriver(upto version 16) and Chrome Webdriver. We need to provide support for Safari Webdriver, Opera Webdriver and Internet Explorer Webdriver.

Alternatives

A new alternative that we should look at is ​Windmill.

Alternate opions which could be investigated:

Roles Testing

  • The current implementation and documentation of role tests can be found here - Testing/Roles.
  • The role tests are currently run only on the IFRC template and implementation for DRRPP is under review ​here.
  • The test suite should automatically generate data for multiple templates so that the role tests can be run on them as well.

Integration Testing (good thing)

We depend on various 3rd-party components so we need to ensure that as these components are upgraded this doesn't break any of our functionality:

  • Web2Py
    • CherryPy
    • SimpleJSON
  • T2
  • OpenLayers
  • jQuery
  • Ext

Usability Tests

Accessibility

  • Are we XHTML 1.0 compliant?
  • Are we usable without JavaScript?

Performance Tests

Whilst the Web2Py framework is fast, we should check that we're not doing anything stupid to slow it down:

Load Tests

How many simultaneous users can the system support?

  • The blueprint for the load tests can be found here - Load
  • Furthur documentation can be found here - Testing/Load

Stress Tests

If extreme load is applied to the application, does it recover gracefully?

  • Tools above but using more extreme parameters

Security Tests

Whilst the Web2Py framework is secure by design, we should validate this:

Things developers can do to reduce risks:

Smoke Tests

The current implementation can be found here - SmokeTests
"Smoke tests" provide a way to highlight the worse failures, and cover a large part of the system. Basically these are like a generic 'can I view this page?' acceptance test. The idea is to write a small script that discovers all the views in the system, and make requests to them, highlighting exceptions. These tests can run quickly as they do not require a round-trip HTTP request. When exceptions occur, these can be turned into regression tests.

Test Coverage

coverage is a python command/module that allows easy measurement of test coverage over a python program. You can generate summary reports or pretty HTML reports of the code coverage. ​http://nedbatchelder.com/code/coverage/

BluePrints

Last modified 11 years ago Last modified on 09/23/13 17:40:09

Attachments (1)

Download all attachments as: .zip

Note: See TracWiki for help on using the wiki.