Testing

"A bug is a test case you haven't written yet"
"Unit Tests allow merciless ​refactoring"

This page defines what our current approach versus our BluePrint for future options

Test-Driven Development is a programming styles which says that you 1st write your test cases (from the specs) & then proceed to make them pass.

Can pre-populate the database with random data using ​gluon.contrib.populate

• InstallationGuidelinesDeveloperTesting
• TestCases - User Testing - List of things to test

Functional Tests

Building the Right Code

We have integrated ​http://seleniumhq.org/projects/remote-control/ into /static/selenium so that Functional Tests can be run (via 'Test' menu option visible to Admins).

Tests can be developed using ​Selenium IDE, if desired.

See the Documentation for details on how to use this test suite.

ToDo: create a HandleResults.py for storing results (to make visible to CI):

• e.g. Convert this one from PHP: ​http://wiki.openqa.org/display/SEL/Integrating+Selenium+And+CruiseControl.Net
• e.g. Extract from ​PloneTool's FunctionalTestTool.py

Hints on improving the stability of Selenium tests:

• ​http://googletesting.blogspot.com/2009/06/my-selenium-tests-arent-stable.html

Systers' approach:

• ​http://systers.org/systers-dev/doku.php/automated_functional_testing
• List of Tests: ​http://systers.org/systers-dev/doku.php/master_checklist_template
• GSoC project: ​http://systers.org/systers-dev/doku.php/svaksha:patches_release_testing_automation

Alternative Options:

• ​http://zesty.ca/scrape/
  • ​Lightning Talk (2.30)

Unit Tests

Building the Code Right

​Selenium RC is great due to ability to handle JavaScript & also due to having an ​IDE for generating them (export as Python).
The IDE output needs to be ​modified to work with Web2Py.
NB Custom functions (e.g. for Random) cannot be shared with the Functional Tests (custom=JS) but the rest of the tests can be.
These tests are stored in /tests.
ToDo: Port the storeRandom function from JS to Python:

import random
print "test_%i@xxxxxxxx" % random.randint(1, 10000)

ToDo: Investigate how we can test multiple browsers.

• ​AltereEgo entry on testing in Web2Py
  • ​StackOverflow discussion
• ​Web2Py Slice on Unit Testing

​CherryPy's ​WebTest is good for in-process testing & can be run from a browser-less server(easier for CI-integration).
These tests are stored in /tests/webtest.
NB These are a work-in-progress...need to enable access to Web2Py environment (db, etc) using:

from gluon.shell import exec_environment
env=exec_environment('the_model_file.py') 

Or could write as a 'Controller' & call from CLI:

python web2py.py -S appname -M -R yourscript.py 

Another similar option could be ​Pylon's WebTest

Doc Tests

Agile documentation which can be run using Web2Py's Admin UI.

• ​http://www.python.org/doc/2.6/library/doctest.html
• e.g. ​http://127.0.0.1:8000/admin/default/test/sahana/default.py

To extend these to web applications, we have a module which uses ​wsgi_intercept & ​CherryPy's ​WebTest: modules/s3_test.py
This can be used from Controllers like:

def login():
    """ Login
    >>> from applications.sahana.modules.s3_test import WSGI_Test
    >>> test=WSGI_Test(db)
    >>> '200 OK' in test.getPage('/sahana/%s/login' % module)
    True
    >>> test.assertHeader("Content-Type", "text/html")
    >>> test.assertInBody('Login')
    """

This works fine,although if an assert fails then the UI gets stuck :/
The 'db' access part isn't yet working.

Note that Web2Py uses a big doctest at the end of each file: def test_all()

Continuous Integration

We are starting to use the Trac-integrated ​Bitten to monitor code quality.

• Other options: ​http://stackoverflow.com/questions/225598/pretty-continuous-integration-for-python

ToDo: Write a step to parse/store the results of Selenium's HandleResults.py

<step id="lint" description="Run PyLint">
  <python:exec module="pylint.lint" output="pylint-report.txt" args="/var/www/trac/sahana3/bzr"/>
  <python:pylint file="pylint-report.txt" />
</step>

ToDo: Amend so that it can find the gluon module (or configure to not follow this dependency):
ImportError: Unable to find module for modules/validators.py in /tmp/bittenA787wC/build_Trunk_1
ToDo: Fix Windows path (NB plain 'bzr' fails too, even if we're in the folder up from that): F: 1: No module named /var/www/trac/sahana3/bzr We need to write a build/test-results.xml for this one (& it also wants a setup.py?):

<step id="test" description="Run unit tests">
  <python:distutils command="unittest"/>
  <python:unittest file="build/test-results.xml"/>
  <python:trace summary="build/test-coverage.txt" coverdir="build/coverage" include="trac*" exclude="*.tests.*"/>
</step>

This one is pointless for us as we don't build anything using a setup.py:

<step id="build" description="Compile to byte code">
  <python:distutils command="build"/>
</step>

---

BluePrintTesting

DeveloperGuidelines