Changes between Version 22 and Version 23 of BluePrint/Testing


Ignore:
Timestamp:
09/23/13 16:53:48 (8 years ago)
Author:
somayjain
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • BluePrint/Testing

    v22 v23  
    5656
    5757== Continuous Integration ==
     58The current implementation can be found here - [wiki:SysAdmin/ContinuousIntegration SysAdmin/ContinuousIntegration]
     59
    5860Whenever a commit is made it should be checked to see that it doesn't break anything
    5961 * [http://bitten.edgewall.org Bitten] - integrates with Trac
     
    6163 * [https://launchpad.net/pqm Patch Queue Manager] - integrates with Bzr (allows branch merging)
    6264'''Note: As of January 2012, BZR/Launchpad info for eden is deprecated. Please visit the GitHub page. Thanks.'''[[BR]]
     65
     66Alternate options which could be investigated:
     67 * http://buildbot.net/trac
     68  * http://redsymbol.net/talks/auto-qa-python/
     69 * http://confluence.public.thoughtworks.org/display/CC/CI+Feature+Matrix
     70 * An instance of Eden can also be used which will enable Scheduling, enable subscription of notifications for test results, can also provide formatted results.
     71* Jenkins
     72 * Advantages
     73  * Easy to build Eden
     74  * Easy integration with selenium grid, so that the selenium tests can be run remotely and parallely
     75  * Git plugin -
     76   * Can trigger tests on a commit easily.
     77   * Can automatically publish build status on the git commit.
     78   * Can create tag/push to master automatically on successful tests after the commit.
     79  * Run tests on new pull request (GitHub pull request builder plugin)
     80  * Email notifications about the test results.
     81  * IRC notifications about the test results. (IRC plugin)
     82  * Add more nodes on which jenkins is run, so that the load is distributed
     83 * Setting Up
     84  * Jenkins is easy to install.
     85  * Installation of Selenium Grid needed.
     86  * Setting up some slaves for Selenium grid for cross browser, parallel selenium tests (Can be done on a single machine too)
     87  * Slight modifications in the Selenium test suite to incorporate Selenium Grid.
     88  * Configuring Jenkins to run tests on a git commit, etc, as required.
     89  * Configuration is done by an authenticated web interface.
     90 * Reference - https://wiki.jenkins-ci.org/display/JENKINS/Home
     91* Selenium Grid
     92 * Advantages
     93  * Can run selenium tests across -
     94   * Different browsers
     95   * Different operating systems
     96   * Different machines in Parallel.
     97  * Works on the concept of hub and nodes.
     98  * The tests are run on a single machine - hub.
     99  * Execution will be done on different machines - nodes.
     100  * Will increase the speed of the Selenium tests since they will be distributed across machines.
     101   * Eg - Some nodes will run create, some will run the search tests.
     102  * We can then trigger the tests on a commit/pull request.
     103  * The time taken will roughly decrease depending on the number of nodes.
     104   * Eg - If there are 4 nodes, the tests will run roughly 4 times faster.
     105 * Setting Up
     106  * To install, need to download the Selenium Server jar file - on the hub as well as on all nodes.
     107  * Will have to change some initialization code for the Selenium tests where the Webdriver is initialised.
     108 * Reference - http://code.google.com/p/selenium/wiki/Grid2
     109* Solano
     110 * Comparison with Jenkins -
     111  * Jenkins has a lot more plugins, which are written by a much larger community. So, it becomes more extensible
     112   * Eg - git plugin, IRC plugin, github pull request plugin.
     113  * Jenkins is easier to integrate with Selenium Grid, which enables us to run the tests in parallel.
     114  * Jenkins has a better documentation, easier to use.
     115 * Disadvantages -
     116  * Our tests run using python’s unittest. We will have to change it to run using nosetests or py.test. Also, on changing too, the tests will not be able to run in parallel.
     117  * Reference - http://docs.tddium.com/python/
     118
    63119
    64120Alternate options which could be investigated:
     
    90146This sees the application as a black box & so the same tests could be run here against both the Python & PHP versions, for instance.
    91147
     148=== Selenium ===
    92149Sahana is a Web-based application, so testing should be from browser perspective:
    93150
     
    96153  * A lot of Selenium-related articles: http://vallista.idyll.org/~grig/articles/
    97154  * Nice slides on Selenium: http://www.slideshare.net/alexchaffee/fullstack-webapp-testing-with-selenium-and-rails
     155
     156==== Selenium Remote Tests ====
     157
     158The design for Selenium Remote Tests -
     159
     160*    Prerequisite 1: There are some more instances of Eden with varying configurations (template/python version/database type) running on some servers(preferably in local network with the CI Server, as this will increase the speed of handling requests by the CI Server).
     161*    Prerequisite 2: There is an 'testing' instance of Eden running on the CI Server which contains the test suite.
     162*    The 'testing' instance of Eden will shoot the test suite, while changing the base URL in the test suite. The base URL currently is 127.0.0.1. If we change it to the IP address of some of the currently running instances of Eden on other servers, the tests will start running for those servers.
     163*    The above task can be done as a background process on the CI Server or running the tests using [http://selenium-grid.seleniumhq.org/ SeleniumGrid] to run them in parallel.
     164* The selenium tests should work across browsers. Currently, there is support for Firefox Webdriver(upto version 16) and Chrome Webdriver. We need to provide support for Safari Webdriver, Opera Webdriver and Internet Explorer Webdriver.
     165
     166=== Alternatives ===
    98167
    99168A new alternative that we should look at is [http://www.getwindmill.com/features Windmill].