[[TOC]]
= Testing =
"A bug is a test case you haven't written yet" [[BR]]
"Unit Tests allow merciless [http://diveintopython.org/refactoring/refactoring.html refactoring]"

This page defines what our current approach versus our [wiki:BluePrintTesting BluePrint for future options]

Test-Driven Development is a programming styles which says that you 1st write your test cases (from the [BluePrints specs]) & then proceed to make them pass.

Can pre-populate the database with random data using [http://groups.google.com/group/web2py/browse_thread/thread/4b2e5ed166220ba5# gluon.contrib.populate]

 * [wiki:InstallationGuidelinesDeveloperTesting]
 * TestCases - User Testing - List of things to test

== Functional Tests ==
Building the Right Code

We have integrated [http://seleniumhq.org/projects/remote-control/] into {{{/static/selenium}}} so that Functional Tests can be run easily.

Tests can be developed using [http://seleniumhq.org/projects/ide Selenium IDE], if desired.

See the [http://eden.sahanafoundation.org/attachment/wiki/DeveloperGuidelinesTesting/Regression%20tests.pdf Documentation] for details on how to use this test suite.

!ToDo: create a {{{HandleResults.py}}} for storing results (to make visible to CI):
 * e.g. Convert this one from PHP: http://wiki.openqa.org/display/SEL/Integrating+Selenium+And+CruiseControl.Net
 * e.g. Extract from [http://jrandolph.com/selenium-plone/selenium-0.3rc2-plone.zip PloneTool]'s {{{FunctionalTestTool.py}}}

Hints:
 * Improving the stability of Selenium tests: http://googletesting.blogspot.com/2009/06/my-selenium-tests-arent-stable.html
 * Lots of useful tips: http://www.eviltester.com
 * Autocomplete is fiddly to test as need to trigger specific events:
{{{
# Enter the search String
sel.type("gis_location_autocomplete", "SearchString")
# Trigger the event to get the AJAX to send
sel.fire_event("gis_location_autocomplete", "keydown")
# Wait for the popup menu
for i in range(60):
    try:
        if "SearchString" == sel.get_text("css=ul.ui-autocomplete li:first-child a"):
            break
    except:
        pass
    time.sleep(1)
else:
    self.fail("time out")
# Select the Result
sel.fire_event("css=ul.ui-autocomplete li:first-child a", "mouseover")
sel.click("css=ul.ui-autocomplete li:first-child a")
time.sleep(4)
}}}

Systers' approach:
 * http://systers.org/systers-dev/doku.php/automated_functional_testing
 * List of Tests: http://systers.org/systers-dev/doku.php/master_checklist_template
 * GSoC project: http://systers.org/systers-dev/doku.php/svaksha:patches_release_testing_automation

Alternative Options:
 * http://zesty.ca/scrape/
  * [http://pycon.blip.tv/file/3261277 Lightning Talk] (2.30)
== Unit Tests ==
Building the Code Right

[http://seleniumhq.org/projects/remote-control Selenium RC] is great due to ability to handle !JavaScript & also due to having an [http://seleniumhq.org/projects/ide IDE] for generating them (export as Python).[[BR]]
The IDE output needs to be [http://groups.google.com/group/web2py/msg/d8c9fd6008029f6b modified] to work with Web2Py.[[BR]]
NB Custom functions (e.g. for Random) cannot be shared with the Functional Tests (custom=JS) but the rest of the tests can be.[[BR]]
These tests are stored in {{{/tests}}}.[[BR]]
!ToDo:  Port the storeRandom function from JS to Python:
{{{
import random
print "test_%i@xxxxxxxx" % random.randint(1, 10000)
}}}
!ToDo: Investigate how we can test multiple browsers.

 * [http://web2py.com/AlterEgo/default/show/260 AltereEgo entry] on testing in Web2Py
  * [http://stackoverflow.com/questions/2762294/unit-testing-in-web2py StackOverflow discussion]
 * [http://web2pyslices.com/main/slices/take_slice/67 Web2Py Slice] on Unit Testing

[http://cherrypy.org CherryPy]'s [http://cherrypy.org/browser/trunk/cherrypy/test/webtest.py WebTest] is good for in-process testing & can be run from a browser-less server(easier for CI-integration).[[BR]]
These tests are stored in {{{/tests/webtest}}}.[[BR]]
NB These are a work-in-progress...need to enable access to Web2Py environment (db, etc) using:
{{{
from gluon.shell import exec_environment
env=exec_environment('the_model_file.py') 
}}}
Or could write as a 'Controller' & call from CLI:
{{{
python web2py.py -S appname -M -R yourscript.py 
}}}

Another similar option could be [http://pylonshq.com/docs/en/0.9.7/thirdparty/webtest/ Pylon's WebTest]
== Doc Tests ==
Agile documentation which can be run using Web2Py's Admin UI.
 * http://www.python.org/doc/2.6/library/doctest.html
 * e.g. http://127.0.0.1:8000/admin/default/test/sahana/default.py
To extend these to web applications, we have a module which uses [http://code.google.com/p/wsgi-intercept wsgi_intercept] & [http://cherrypy.org CherryPy]'s [http://cherrypy.org/browser/trunk/cherrypy/test/webtest.py WebTest]: {{{modules/s3/s3test.py}}} [[BR]]
This can be used from Controllers like:
{{{
def login():
    """ Login
    >>> from applications.sahana.modules.s3.s3test import WSGI_Test
    >>> test=WSGI_Test(db)
    >>> '200 OK' in test.getPage('/sahana/%s/login' % module)
    True
    >>> test.assertHeader("Content-Type", "text/html")
    >>> test.assertInBody('Login')
    """
}}}
This works fine,although if an assert fails then the UI gets stuck :/ [[BR]]
The 'db' access part isn't yet working.

Note that Web2Py uses a big doctest at the end of each file: {{{def test_all()}}}

== Continuous Integration ==
We are starting to use the Trac-integrated [http://bitten.edgewall.org Bitten] to monitor code quality.

 * Other options: http://stackoverflow.com/questions/225598/pretty-continuous-integration-for-python
  * [http://buildbot.net/trac Buildbot]
  * [http://hudson-ci.org/ Hudson]

!ToDo: Write a step to parse/store the results of Selenium's {{{HandleResults.py}}}
{{{
<step id="lint" description="Run PyLint">
  <python:exec module="pylint.lint" output="pylint-report.txt" args="/var/www/trac/sahana3/bzr"/>
  <python:pylint file="pylint-report.txt" />
</step>
}}}
!ToDo: Amend so that it can find the {{{gluon}}} module (or configure to not follow this dependency):[[BR]]
{{{ImportError: Unable to find module for modules/s3/s3validators.py in /tmp/bittenA787wC/build_Trunk_1}}}[[BR]]
!ToDo: Fix Windows path (NB plain 'bzr' fails too, even if we're in the folder up from that):
{{{F:  1: No module named /var/www/trac/sahana3/bzr}}}
We need to write a {{{build/test-results.xml}}} for this one (& it also wants a {{{setup.py}}}?):
{{{
<step id="test" description="Run unit tests">
  <python:distutils command="unittest"/>
  <python:unittest file="build/test-results.xml"/>
  <python:trace summary="build/test-coverage.txt" coverdir="build/coverage" include="trac*" exclude="*.tests.*"/>
</step>
}}}
This one is pointless for us as we don't build anything using a {{{setup.py}}}:
{{{
<step id="build" description="Compile to byte code">
  <python:distutils command="build"/>
</step>
}}}
----
BluePrintTesting

DeveloperGuidelines