Continuous Integration

Current Setup

CI Server's cron runs the tests every 24 hours

• Using Web2Py from Tue Jun 11 08:00:48 - Version 2.5.1-stable+timestamp.2013.06.11.08.00.05
• Using latest Trunk Eden
• Using IFRC, default, SandyRelief, DRMP, CRMT templates
• Using SQLite
• Using Chrome

Results are stored as HTML files in the filesystem which can be accessed via the Eden instance under test:

• ​http://82.71.213.53/RESULTS

Also, the results are being mailed to a ​mailing list.

• You can simply follow the above link and click on "Join group to post" to join the mailing list.
• The CI Server sends out a daily test report email to this list.
• You can manage your subscription by clicking on "My Settings" tab on the top right corner.

/usr/local/bin/testsuite

• Updates Web2Py to current Trunk (from github.com/web2py/web2py)
  • Currently disabled
• Updates Eden to current Trunk (from github.com/flavour/eden)
• Cleans database & prepopulates
• Runs Functional Tests (Selenium)
• Runs Smoke Tests
• Runs Roles Tests
  • Currently disabled
• Mails out test results (using /usr/local/bin/mailTestResults.sh)
• The code for /usr/local/bin/testsuite and /usr/local/bin/mailTestResults.sh can be found in the attachments.

The workflow is depicted here -

NB Tests require Python 2.7, so Pythonbrew is used to allow this to coexist with the system Python 2.6

To run the test suite manually, run:

/usr/local/bin/testsuite

To just run smoke tests:

/usr/local/bin/web2pystarter
cd /home/web2py
xvfb-run -a python web2py.py -S eden -M -R applications/eden/modules/tests/suite.py -A --html-path /home/web2py/applications/eden/static/test_smoke/ --suite smoke --force-debug --link-depth 16 -V 4

Start Window based applications (x11):

• If the GUI window of the application is of no interest to you:

  xvfb-run -a *test commands here*
  

  For example :

  xvfb-run -a python web2py.py -S eden -M -R applications/eden/modules/tests/suite.py -A -V 2
  

  Note: Xvfb basically makes a virtual display which isn't displayed anywhere. Xserver itself actually requires a real display
• If you want to see the GUI interface of the application
  • If the machine you use to ssh into the server, has X Server installed, then you can access the GUI window by using the -X flag when loggin into the server

    ssh -X youraccount@82.71.213.53
    

    And now, if you simply execute any application which has a Xserver attached to it, it will be displayed on your screen.

Plans for Future

• Move mailing script out of a personal user directory
• Move cron commands to system /etc/crontab rather than root's crontab
• Fix Reporting of Smoke Tests
• Enable Roles Tests & fix any issues which caused it to be disabled
• Run Unit Tests
• More templates (default, etc)
• More databases
• More browsers
• More web2py versions
  • e.g. 'Current Stable' (as-defined by Eden) & 'Trunk'
  • having errors in Trunk Web2pY which aren't seen in 'Current Stable' homes us in on where problems lie
• Install an Eden instance to act as a CI server
  • Move the cron scripts to being run by Eden Scheduler
  • Have Eden manage the mailing of results & manage subscriptions
  • Have Eden store the results in the database for Graphical analysis
  • Same Eden instance can be Sunflower which replaces Trac for tickets
  • Have Eden analyse results intelligently - e.g. logging tickets for Web2Py issues when web2py version is seen as the cause
  • AUT should be possible on remote servers
    • e.g. Spawn AWS instances using ​https://github.com/flavour/spawn-eden
      • AMIs built for Cherokee/PostgreSQL & Apache/MySQL to speed-up bootstrapping of instances
    • Can do Performance Testing
    • Can do Tests on servers with User Data?
  • This replaces the old install of Jenkins

Setup Notes & Discussions (to be ported to Wiki):

• ​https://docs.google.com/document/d/1T29cF3kkK4lIAVdiATfE8a-tLm_Tyh3_dIh6FJWY7sU/edit