= Testing =
[[TOC]]
''"A bug is a test case you haven't written yet"''

''"Unit Tests allow merciless [http://diveintopython.org/refactoring/refactoring.html refactoring]"''

This page defines what our current approach versus our [wiki:BluePrintTesting BluePrint for future options]

Test-Driven Development is a programming style which says that you 1st write your test cases (from the [BluePrint specs]) & then proceed to make them pass.

== Introduction ==
Selenium provides the ability to test Sahana Eden as users see it - namely through a web browser. This therefore does end-to-end Functional Testing, however it can also be used as Unit Testing

We are building our framework around the new !WebDriver, despite having some legacy code in the older format:
* http://seleniumhq.org/docs/appendix_migrating_from_rc_to_webdriver.html#why-migrate-to-webdriver

The tests are stored in {{{eden/modules/tests}}}

== Installation of the testing environment in your machine ==
In order to execute the automated Selenium powered test scripts, you must install the Selenium Web Drivers into Python.

Download and install the latest Selenium package, which can be found at [http://pypi.python.org/pypi/selenium#downloads]:
{{{
wget http://pypi.python.org/packages/source/s/selenium/selenium-2.25.0.tar.gz
tar zxvf selenium-2.25.0.tar.gz
cd selenium-2.25.0
python setup.py install
}}}
== Running / Executing Automated test scripts: ==
Before running the Selenium scripts, you should put your database into a known state:
{{{
clean
}}}

For the whole test suite, it is assumed that you are using:
{{{
deployment_settings.base.prepopulate = ["IFRC_Train"]
}}}

Run the whole test suite for the Eden application:
{{{
cd web2py
python web2py.py -S eden -M -R applications/eden/modules/tests/suite.py
}}}

Run a class and all tests in that class:
{{{
cd web2py
python web2py.py -S eden -M -R applications/eden/modules/tests/suite.py -A -C mytestclass
}}}

Run just one test within a class:
{{{
cd web2py
python web2py.py -S eden -M -R applications/eden/modules/tests/suite.py -A -C mytestclass -M mytestmethod
}}}

=== Command Line Arguments ===

A number of command line arguments have been added and more are being added so to get the latest list of available options use the --help switch, which you can quickly do as follows:
{{{
python web2py.py -S eden -M -R applications/eden/modules/tests/suite.py -A --help
}}}

Important options include:

-C for class to run

-M for test method within a class to run, when you use this option either use the -C option or provide the method in the format class.method

If you have HTMLTestRunner installed then a nicely formatted html report will be generated, should you want to disable this then use the --nohtml option. The HTML report will be written to the path given by the switch --html_path which by default will the web2py folder since that is where the tests scripts are run from. The file name will have a timestamp appended to it, if you want you can have just a date by using the option html_name_date.

The option --suite will run certain predefined tests. At the moment it supports '''smoke''', which runs a test to look for broken urls otherwise it will run all the tests. If a class or method are selected then this option is ignored. For the smoke test to work you need to install two packages, [http://twill.idyll.org/ twill] and [http://pypi.python.org/pypi/mechanize/ mechanize], since twill uses mechanize if you install via a package manager then you probably only need to install twill.

== Writing / Creating your own test scripts: ==
We aim to make it as easy as possible to write additional tests, which can easily be plugged into the Sahana Eden Testing System Framework. 

The canonical example is: {{{eden/modules/tests/org/create_organisation.py}}}

New tests should be stored in a subfolder per module, adding the foldername to {{{eden/modules/tests/__init__.py}}} & creating an {{{__init__.py}}} in the subfolder.

'''A walk-through example:'''[[BR]]
We want to make an automated test script to test the Create Organization module. 
Create Organization falls under Human Resources Management (HRM) feature, therefore we must add this test script module file in the subfolder /hrm (eden/modules/tests/hrm/)

Steps of automated the test for Create Organization:[[BR]][[BR]]
 
1) Let's call this test file: create_organization.py and save it in the subfolder /hrm. [[BR]][[BR]]

2) We must now make sure we add the file name in the __init__.py
Open file __init__.py in the subfolder /hrm and add:
{{{
from create_organisation import *
}}}
3) create_organisation.py
{{{
import os
import time
from selenium.common.exceptions import NoSuchElementException
from gluon import current
from s3 import s3_debug
from tests.web2unittest import SeleniumUnitTest

class CreateOrganization(SeleniumUnitTest):

    # -------------------------------------------------------------------------
    def test_org001_create_organisation(self, items=[0]):
        """
            Create an Organisation
            
            @param items: Organisation(s) to create from the data

            @ToDo: currently optimised for a single record
        """
        print "\n"
        # Configuration
        self.login(account="normal", nexturl="org/organisation/create")
        tablename = "org_organisation"
        #data = current.data["organisation"]
        # Data for the Organisation resource
        self.create("org_organisation", 
                    [( "name", "Romanian Food Assistance Association (Test)"), ( "acronym", "RFAAT"), ("website", "http://www.rfaat.com"), ("comments", "This is a Test Organization")]
                     )
        for item in items:
            _data = data[item]
            # Check whether the data already exists
            s3db = current.s3db
            db = current.db
            table = s3db[tablename]
            fieldname = _data[0][0]
            value = _data[0][1]
            query = (table[fieldname] == value) & (table.deleted == "F")
            record = db(query).select(table.id,
                                      limitby=(0, 1)).first()
            if record:
                print "org_create_organisation skipped as %s already exists in the db\n" % value
                return False

            # Login, if not-already done so
            self.login(account=account, nexturl=url)
}}}

4) To run this automated test script.
Open suite.py (modules/tests/suite.py) and add at the bottom of suite.py script so it can be executed. (NOTE: We add the class name, not the function name).
{{{
addTests(loadTests(CreateOrganization))
}}}

5) Run your test script. (Refer above on how to run/execute test scripts from the Test Suite).[[BR]][[BR]][[BR]]

'''Splitting Test Functions:''' [[BR]]
We aim to keep the test script as clean and understandable as possible. Therefore, we must separate the test functions which drives the a particular test script to another file.
As you can see with the Inventory feature tests, there is a file in the INV subdirectory called helper.py. This file contains all functions which helps drive the Inventory module tests such as send_item.py, receive_item.py and so forth.[[BR]][[BR]]

'''Other information on the Test System framework:'''[[BR]]

The key is to make tests which are as least fragile as possible through:
* State (we should be able to run individual tests easilty, which check their current state as-required)
* Deployment_Settings
* Localisation
* Theme

This suggests refactoring tests to centralise common elements into a library to mean fixes should only happen in 1 place.

There are a number of possible selectors to use to find your elements...the 'ID' may be the most stable, so don't be afraid of patching the code to add IDs where you'd like to be able to reach them reliably:
* http://selenium.googlecode.com/svn/trunk/docs/api/py/selenium/selenium.selenium.html

We separate data out into a separate file, so that this is easy to change to allow reruns of the tsts with different data sets.

== Unit Tests ==
There are additional unit tests available, which also require running with the IFRC_Train preopulate, 'settings.base.prepopulate = 27':

To run all unit tests:
{{{
python web2py.py -S eden -M -R applications/eden/modules/unit_tests/suite.py
}}}

These unit tests are meant to detect problems early during development, which means you should run them quite often while you're still working on the code (read: before and after every little change you make) rather than expecting them to be run during QA cycles. That again means you would more often need to run tests for particular modules than the whole suite.

For every module in {{{modules/s3}}} and {{{modules/eden}}}, you can find the corresponding unit test module under {{{modules/unit_tests/eden}}} resp. {{{modules/unit_tests/s3}}} (if one exists).

To run tests for particular modules:
{{{
# e.g. for modules in modules/s3
python web2py.py -S eden -M -R applications/eden/modules/unit_tests/s3/s3resource.py
# e.g. for modules in modules/eden
python web2py.py -S eden -M -R applications/eden/modules/unit_tests/eden/pr.py
}}}

It can be a very powerful development strategy - especially for back-end APIs - to first implement unit test cases for the functionality you intend to implement before actually implementing it. Apart from preventing bugs, this helps you to validate your design against requirements, and to keep the implementation simple and focussed. Additionally, the test cases can be a rich source of code samples how to apply your API methods.
== Load Testing ==
We recommend using [Testing/Load Tsung]

== See Also ==
 * TestCases - List of things to test
 * http://code.google.com/p/selenium/wiki/PageObjects - Style suggestion

Systers' approach:
 * http://systers.org/systers-dev/doku.php/automated_functional_testing
 * List of Tests: http://systers.org/systers-dev/doku.php/master_checklist_template
 * GSoC project: http://systers.org/systers-dev/doku.php/svaksha:patches_release_testing_automation

Alternative Options:
 * http://zesty.ca/scrape/
  * [http://pycon.blip.tv/file/3261277 Lightning Talk] (2.30)