wiki:DeveloperGuidelines/Testing/Load

Version 7 (modified by Fran Boon, 13 years ago) ( diff )

--

Table of Contents

  1. Server Setup
  2. Client Setup
    1. Installation
    2. Static Tests
    3. Dynamic Tests
    4. Run Tests
    5. Analyse Results

Load Testing

We recommend to use ​Tsung as a feature-rich & high-performing tool.

Testing should normally be done from a separate machine. Amazon is great as a platform for accessing a lof of resources for a short period of time.

  • A 'Micro' instance is sufficient for testing 100 concurrent users (the dynamic parts of the site)
  • A 'High CPU' instance is required for testing 8,000 concurrent users (the static parts of the site)

Server Setup

Tsung can include results on Server Load if SNMP is enabled: 

apt-get install -y snmpd
vim /etc/snmp/snmpd.conf
#agentAddress  udp:127.0.0.1:161
agentAddress udp:161,udp6:[::1]:161
#rocommunity public  default    -V systemonly       
rocommunity public 0.0.0.0/0

/etc/init.d/snmpd restart

Notes:

  • Log files can fill-up quickly, especially when testing a large number of users for the static parts of the site
  • Database will need resetting to clear any transactions made
  • Reboot between tests to reset the RAM/Swap

Client Setup

Installation

apt-get install -y erlang gnuplot libtemplate-perl make lrzsz vim
wget http://tsung.erlang-projects.org/dist/tsung-1.4.1.tar.gz
tar zxvf tsung-1.4.1.tar.gz
cd tsung-1.4.1
./configure
make
make install
ssh localhost
yes to accept hostkey
exit

Static Tests

It is recommended to make the homepage of the site static. The performance of this page can be tested using an 8-way CPU ('High CPU' on Amazon).

  • This requires the system to be able to SSH to itself

~/.tsung/tsung.xml: 

<?xml version="1.0"?>
<!DOCTYPE tsung SYSTEM "/usr/share/tsung/tsung-1.0.dtd">
<tsung loglevel="notice" version="1.0">

  <!-- Client side setup -->
  <clients>
    <client host="localhost" cpu="8"/>
  </clients>
  
  <!-- Server side setup -->
<servers>
  <server host="10.171.46.7" port="80" type="tcp"></server>
</servers>

  <!-- to start OS monitoring (cpu, network, memory) --> 
  <monitoring>
    <monitor host="10.171.46.7" type="snmp"></monitor>
  </monitoring>
  
  <load duration="30" unit="minute">
   <arrivalphase phase="1" duration="10" unit="minute">
     <users maxnumber="3000" arrivalrate="10" unit="second"></users>
   </arrivalphase>
   <arrivalphase phase="2" duration="10" unit="minute">
     <users maxnumber="6000" arrivalrate="10" unit="second"></users>
   </arrivalphase>
   <arrivalphase phase="3" duration="10" unit="minute">
     <users maxnumber="10000" arrivalrate="10" unit="second"></users>
   </arrivalphase>
   <!-- Will run fully-loaded for 3 minutes -->
  </load>

  <options>
   <option type="ts_http" name="user_agent">
    <user_agent probability="80">Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.8) Gecko/20050513 Galeon/1.3.21</user_agent>
    <user_agent probability="20">Mozilla/5.0 (Windows; U; Windows NT 5.2; fr-FR; rv:1.7.8) Gecko/20050511 Firefox/1.0.4</user_agent>
   </option>
  </options>

  <!-- start a session for a http user. the probability is the
  frequency of this type of session. The sum of all session's
  probabilities must be 100 -->

 <sessions>
  <session name="static-homepage" probability="100" type="ts_http">

    <!-- Since we use maxnumber, we need each client to loop indefinitely to sustain the concurrency -->
    <for from="1" to="1000" var="i">
        <!-- full url with server name, this overrides the "server" config value -->

        <!-- Visit Home page -->
        <request> <http url="/" method="GET" version="1.1"></http> </request>
        <!-- Doesn't follow the 301 -->
        <request> <http url="/eden/static/index.html" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/styles/S3/sahana.min.css" method="GET" version="1.1"></http> </request>
        <!-- Moved to CDN
        <request> <http url="/eden/static/scripts/web2py/jquery-1.6.2.min.js" method="GET" version="1.1"></http> </request>
        -->
        <request> <http url="/eden/static/img/la/cityofla_logo.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/logo.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/mayor_antonio_vilaraigosa.jpg" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/la_seal.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/sahanasmall_05.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/bg.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/footer_end.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/header_bg.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/nav_divider.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/open_quote.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/close_quote.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/donate_home.jpg" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/paper_corner.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/bottom_paper_shadow.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/arrow_bullet_large.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/volunteer_home.jpg" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/paper_corner_right.png" method="GET" version="1.1"></http> </request>

        <thinktime value="10" random="true"></thinktime>

    </for>

  </session>
 </sessions>
</tsung>

Dynamic Tests

A set of transactions can be performed with an increasing level of load.

Content can be pulled from CSV if it needs to vary per run (e.g. usernames to register). Can create a large CSV using a simple Python script: 

#!/usr/bin/env python
FILENAME = "users.csv"
f = open(FILENAME, "wb")
for i in range(1, 100000):
    f.write("Test%i%40example.com\n" % i)
f.close()

Note how we read the _formkey from the form in order to be able to submit the data.

~/.tsung/tsung.xml: 

<?xml version="1.0"?>
<!DOCTYPE tsung SYSTEM "/usr/share/tsung/tsung-1.0.dtd">
<tsung loglevel="notice" version="1.0">

  <!-- Client side setup -->
  <clients>
    <client host="localhost" use_controller_vm="true"/>
  </clients>
  
  <!-- Server side setup -->
<servers>
  <server host="10.171.46.7" port="80" type="tcp"></server>
</servers>

  <!-- to start OS monitoring (cpu, network, memory) -->
  <monitoring>
    <monitor host="10.171.46.7" type="snmp"></monitor>
  </monitoring>
  
  <load duration="30" unit="minute">
   <arrivalphase phase="1" duration="5" unit="minute">
     <users maxnumber="5" arrivalrate="1" unit="second"></users>
   </arrivalphase>
   <arrivalphase phase="2" duration="5" unit="minute">
      <!-- Add another 5 to bring the total to 10 -->
      <users maxnumber="5" arrivalrate="1" unit="second"></users>
   </arrivalphase>
   <arrivalphase phase="3" duration="5" unit="minute">
      <!-- Add another 15 to bring the total to 25 -->
      <users maxnumber="15" arrivalrate="1" unit="second"></users>
   </arrivalphase>
   <arrivalphase phase="4" duration="5" unit="minute">
      <!-- Add another 25 to bring the total to 50 -->
      <users maxnumber="25" arrivalrate="1" unit="second"></users>
   </arrivalphase>
   <arrivalphase phase="5" duration="5" unit="minute">
      <!-- Add another 50 to bring the total to 100 -->
      <users maxnumber="50" arrivalrate="1" unit="second"></users>
   </arrivalphase>
   <!-- Will run fully-loaded for 9 minutes -->
  </load>

  <options>
   <option name="file_server" id='userdb' value="/root/.tsung/users.csv"/>
   <option type="ts_http" name="user_agent">
    <user_agent probability="80">Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.8) Gecko/20050513 Galeon/1.3.21</user_agent>
    <user_agent probability="20">Mozilla/5.0 (Windows; U; Windows NT 5.2; fr-FR; rv:1.7.8) Gecko/20050511 Firefox/1.0.4</user_agent>
   </option>
  </options>

  <!-- start a session for a http user. the probability is the
  frequency of this type of session. The sum of all session's
  probabilities must be 100 -->

 <sessions>
  <session name="dynamic-with-progressive-load" probability="100" type="ts_http">

    <!-- Since we use maxnumber, we need each client to loop indefinitely to sustain the concurrency -->
    <for from="1" to="1000" var="i">
        <!-- full url with server name, this overrides the "server" config value -->

        <!-- Visit Home page -->
        <request> <http url="/" method="GET" version="1.1"></http> </request>
        <!-- Doesn't follow the 301 -->
        <request> <http url="/eden/static/index.html" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/styles/S3/sahana.min.css" method="GET" version="1.1"></http> </request>
        <!-- Moved to CDN
        <request> <http url="/eden/static/scripts/web2py/jquery-1.6.2.min.js" method="GET" version="1.1"></http> </request>
        -->
        <request> <http url="/eden/static/img/la/cityofla_logo.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/logo.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/mayor_antonio_vilaraigosa.jpg" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/la_seal.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/sahanasmall_05.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/bg.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/footer_end.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/header_bg.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/nav_divider.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/open_quote.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/close_quote.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/donate_home.jpg" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/paper_corner.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/bottom_paper_shadow.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/arrow_bullet_large.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/volunteer_home.jpg" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/paper_corner_right.png" method="GET" version="1.1"></http> </request>

        <thinktime value="10" random="true"></thinktime>

        <!-- Visit Donate page -->
        <request> <http url="/eden/don/index" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/scripts/ext/resources/css/ext-theme.min.css" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/scripts/S3/S3.min.js" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/scripts/ext/src/locale/ext-lang-en.js" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/scripts/S3/jquery.hoverIntent.minified.js" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/logo-laepf.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/logo_arc.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/logo_wvi.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/logo_sa.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/corner_box.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/arrow_bullet_medium.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/donate_donate.jpg" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/volunteer_donate.jpg" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/colorbox/controls.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/colorbox/border.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/colorbox/loading_background.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/jquery-ui/ui-anim_basic_16x16.gif" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/media/closebox.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/popup_donate.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/popup_donate_bg.png" method="GET" version="1.1"></http> </request>

        <thinktime value="10" random="true"></thinktime>

        <!-- Visit Volunteer page -->
        <request> <http url="/eden/vol/req_skill" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/logo_cert.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/logo_dhv.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/laworks.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/logo_phev.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/vcla.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/icon-xls.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/RSS_16.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/kml_icon.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/scripts/S3/s3.dataTables.min.js" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/sub_nav_shadow.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/light_blue_button_end.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/light_blue_button.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/popup_footer.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/popup_header.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/external_link.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/heading_shadow.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/jquery.dataTables/sort_asc.jpg" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/jquery.dataTables/sort_both.jpg" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/pagination_sprite.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/apply_arrow_head.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/apply_arrow_tail.png" method="GET" version="1.1"></http> </request>

        <thinktime value="10" random="true"></thinktime>

        <!-- Visit Registration page -->
        <request>
            <!-- Collect the formkey from the registration form (not the sign-in form!) -->
            <dyn_variable name="_formkey" xpath="//form[@id='regform']//input[@name='_formkey']/@value" />
            <http url="/eden/vol/register" method="GET" version="1.1"></http>
        </request>
        <request> <http url="/eden/static/scripts/S3/jquery.validate.min.js" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/scripts/S3/jquery.pstrength.1.3.min.js" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/help_off.gif" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/la/long_red_button.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/jquery.cluetip/wait.gif" method="GET" version="1.1"></http> </request>

        <thinktime value="10" random="true"></thinktime>

        <!-- Register -->
        <setdynvars sourcetype="file" fileid="userdb" delimiter="," order="iter">
            <var name="username" />
            <!--<var name="password" />-->
        </setdynvars>
        <request subst="true">
            <http url="/eden/vol/register" method="POST" version="1.1" content_type='application/x-www-form-urlencoded'
                  contents="_formname=register&amp;_formkey=%%__formkey%%&amp;first_name=Test&amp;middle_name=&amp;last_name=Test&amp;email=%%_username%%&amp;password=eden&amp;password_two=eden&amp;language=en&amp;phone=123456789&amp;phone_type=SMS&amp;address1=1%20Main%20Street&amp;city=Los%20Angeles&amp;location_id=251&amp;zip=91024&amp;eighteen=on&amp;citizen=on">
            </http>
        </request>
        <!-- Next Page -->
        <request>
            <http url="/eden/vol/skill/create" method="GET" version="1.1">
                <www_authenticate userid="%%_username%%" passwd="eden"/>
            </http>
        </request>
        <request> <http url="/eden/static/img/jquery-ui/ui-bg_highlight-soft_100_eeeeee_1x100.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/jquery-ui/ui-bg_glass_100_f6f6f6_1x400.png" method="GET" version="1.1"></http> </request>
        <request> <http url="/eden/static/img/jquery-ui/ui-icons_ef8c08_256x240.png" method="GET" version="1.1"></http> </request>

        <thinktime value="10" random="true"></thinktime>

        <!-- Apply for a task -->
        <request>
            <http url="/eden/vol/req/application/create?skill_id=1" method="GET" version="1.1">
                <www_authenticate userid="%%_username%%" passwd="eden"/>
            </http>
        </request>
        <request>
            <http url="/eden/vol/req/7/application/create" method="GET" version="1.1">
                <www_authenticate userid="%%_username%%" passwd="eden"/>
            </http>
        </request>
        <request> <http url="/eden/static/img/dialog-warning.png" method="GET" version="1.1"></http> </request>

        <thinktime value="10" random="true"></thinktime>

        <!-- Logout -->
        <request> <http url="/eden/default/user/logout" method="GET" version="1.1"></http> </request>

        <thinktime value="10" random="true"></thinktime>

    </for>

  </session>
 </sessions>
</tsung>

Run Tests

(replace 20xxx with the name of the folder created) 

tsung start
cd ~/.tsung/log/20xxx
/usr/lib/tsung/bin/tsung_stats.pl
cd ..
tar cvf 20xxx.tar 20xxx/
gzip -9 20xxx.tar
sz 20xxx.tar.gz
rm -rf 20xxx*

Analyse Results

Open report.html in a web browser:

  • Check that there are minimal errors
  • Check the Response time (Request is per-request, Transaction is for the whole script)

DeveloperGuidelinesTesting

Note: See TracWiki for help on using the wiki.