wiki:DeveloperGuidelines/Testing

Version 20 (modified by Dominic König, 4 years ago) ( diff )

--

Testing

"A bug is a test case you haven't written yet"

Automated Tests

EdenTest

EdenTest is a Robot Framework based test framework used for automated testing in Sahana Eden. It is located in the eden codebase under directory eden/tests.

Before running these tests, see setup instructions for installation and setup. Then to run the whole test suite:

python web2py.py --no-banner -M -S eden  -R applications/eden/tests/edentest.py -A testsuites

Smoke Tests

Smoke Tests click through on every link within Sahana Eden and can be used to check for errors on pages and broken links. They are a light-weight approach to detecting basic errors, however they do not test form submission or any interaction. They are run using EdenTest

To run the Smoke Tests

python web2py.py --no-banner -M -S eden  -R applications/eden/tests/edentest_runner.py -A  smoke_tests  -o NONE -l NONE

Before running smoke tests, check Smoke Tests for additional options.

Unit Tests

"Unit Tests allow merciless refactoring"

Unit Tests can be used to test whether specific "Units" of code are working. They are used extensively to test the Sahana Eden "S3" Framework. Unit Tests require running with the following prepopulate:

settings.base.prepopulate = ("default", "default/users")

To run all Unit Tests:

python web2py.py -S eden -M -R applications/eden/modules/unit_tests/suite.py

These unit tests are meant to detect problems early during development, which means you should run them quite often while you're still working on the code (read: before and after every little change you make) rather than expecting them to be run during QA cycles. That again means you would more often need to run tests for particular modules than the whole suite.

For every module in modules/s3 and modules/s3db, you can find the corresponding unit test module under modules/unit_tests/s3db resp. modules/unit_tests/s3 (if one exists).

To run tests for particular modules:

# e.g. for modules in modules/s3
python web2py.py -S eden -M -R applications/eden/modules/unit_tests/s3/s3resource.py
# e.g. for modules in modules/s3db
python web2py.py -S eden -M -R applications/eden/modules/unit_tests/s3db/pr.py

It can be a very powerful development strategy - especially for back-end APIs - to first implement unit test cases for the functionality you intend to implement before actually implementing it. Apart from preventing bugs, this helps you to validate your design against requirements, and to keep the implementation simple and focussed. Additionally, the test cases can be a rich source of code samples how to apply your API methods.

Cautionary notes on Unit Testing:

Role Tests

Role tests are used to check the permissions of user roles. Currently is limited to the IFRC (Red Cross) roles for RMS, but could be extended: To run the Role Tests:

python web2py.py -S eden -M -R applications/eden/modules/tests/suite.py -A --suite roles

See Testing/Roles for more details

Benchmark Tests

The Benchmark Tests are a simple way of measuring the performance of the Sahana Eden "S3" Framework. The result of these tests will give you a measure of the performance relative to the system running them. To run the Benchmark Tests:

python web2py.py -S eden -M -R applications/eden/modules/unit_tests/s3/benchmark.py

Load Tests

Load Testing will measure the performance of the entire Sahana Eden stack. We recommend using Tsung. See: Testing/Load for more details

Continuous Integration (CI)Server

A CI server using Travis has been set up which sets up a new installation of whole Eden stack and then runs the unit tests on every pull request that has been sent. Based upon the status of the tests, the pull request's status is updated on GitHub. For more information, please see CI Server.

Manual Tests

Testing/TestCases

VMs of various Internet Explorer versions can be downloaded from:

Note: See TracWiki for help on using the wiki.