Advanced Test Design

Continuing from the tutorial on How to write a testcase?, this article focuses on the advanced aspects of EdenTest.

If you have implemented Login with invalid email and passwd should fail, your testsuite login_funtionality.robot should look like this.

*** Settings ***
Resource  ../resources/main.robot

*** Test Cases ***
Login with valid email and valid passwd should be successful
   #opens the browser to the argument url
    Go To  http://${SERVER}/eden/default/user/login
    #puts arg2 in textbox(arg1)
    Input Text  auth_user_email  admin@example.com
    Input Text  auth_user_password  testing
    #Clicks the submit button
    Click Button  xpath=//input[@class='btn' and @value='Login']
    #Checks if the text 'Logged in' is there or not
    Page Should Contain  Logged in

Login with invalid email and valid passwd should fail
    Logout From Eden 
    Go To  http://${SERVER}/eden/default/user/login
    Input Text  auth_user_email  nottheadmin@example.com
    Input Text  auth_user_password  testing
    Click Button  xpath=//input[@class='btn' and @value='Login']
    Page Should Contain  Invalid login

Variables

What are Variables?

Variables are an integral feature of Robot Framework, and they can be used in most places in test data. Most commonly, they are used in arguments for keywords and settings. Eg: config.py contains variables like SERVER, APPNAME etc.

Types of Variables

There are three types of variables in Robot Framework:

• Scalars - ${SCALAR}
• Lists - @{LIST}
• Environment Variables - %{Environment Variable}

When to use Variables?

The use of variables is recommended in the following cases:

• When strings change often in the test data. With variables you only need to make these changes in one place.
• When different keywords, even in different test libraries, need to communicate. You can assign a return value from one keyword to a variable and give that as an argument to another.
• When values in the test data are long or otherwise complicated.

Using Variables

*** Variables ***
${LOGIN URL}  http://${SERVER}/eden/default/user/login
${SUBMIT}  xpath=//input[@class='btn' and @value='Login']
${EMAIL ID}  auth_user_email
${PASSWORD ID}  auth_user_password


*** Test Cases ***
Login with valid email and valid passwd should be successful
    Open Browser  ${LOGIN URL}
    Input Text   ${EMAIL}  admin@example.com
    Input Text  ${PASSWORD}  testing
    Click Button  ${SUBMIT}
    Page Should Contain  Logged in

Login with invalid email and valid passwd should be successful
    Open Browser  ${LOGIN URL}
    Input Text  ${EMAIL}  nottheadmin@example.com
    Input Text  ${PASSWORD}  testing
    Click Button  ${SUBMIT}
    Page Should Contain  Invalid login

Note: The *** Settings *** would remain as it is.

User Keywords

What are User Keywords?

Keywords are the functions of Robot Framework.

User Keywords are new higher-level keywords by combining existing keywords together. These keywords are called user keywords to differentiate them from lowest level library keywords that are implemented in test libraries. The syntax for creating user keywords is very close to the syntax for creating test cases, which makes it easy to learn.

Keywords must not anticipate! or to say it simply never shall a keyword make any assertions of its own! All the assertions (pass fail situations) like Page Should Contain [text] etc should be in the testcase not in the keyword.

Implementing User Keywords

In the example we have been following, we had to repeat few steps in both the testcases. We can take out those common steps into a higher level user keyword. Lets call it Login with email and passwd. It takes the email and passwd as the arguments.

*** Keywords ***
Login with email and passwd
    [Documentation]  Opens a browser to login url, inputs username and password
    [Arguments]  ${email}  ${passwd}
    Open Browser  ${LOGIN URL}
    Input Text  ${EMAIL ID}  ${email}
    Input Text  ${PASSWORD ID}  ${passwd}
    Click Button  ${SUBMIT}

*** Test Cases ***
Login with valid email and valid passwd should be successful
    Login with email and passwd  admin@example.com  testing
    Page Should Contain  Logged in

Login with invalid email and valid passwd should be successful
    Login with email and passwd  nottheadmin@example.com  testing
    Page Should Contain  Invalid login

Note: *** Settings ***, *** Variables *** sections stay as it is.

Smoke Tests

Smoke testing is preliminary testing to reveal simple failures severe enough to reject a prospective software. In Eden, smoke tests crawl through whole of Eden upto a configurable depth and checks to see if any URL it visits is broken. If broken, it catches the traceback/error (if present) and reports it on the console. It also generates a log file in the folder it is run by the name <timestamp>_smoke_tests_log.txt which contain every URL it visited and its status.

How to run smoke tests?

Smoke tests are present in the testsuite file eden/tests/implementation/testsuites/smoke_tests.txt. There are various configuration options present at the top of the suite in the *** Variables *** section like MAXDEPTH, EXT LINKS, START URL, ROOT URL etc.

Assuming you are in the directory where web2py is located and your eden app is named eden, run this command to run the smoke tests:

python web2py.py --no-banner -M -S eden  -R applications/eden/tests/edentest.py -A  smoke_tests  -o NONE -l NONE

It is advised not to create the log and output file as they are too detailed to be informative. If any traceback is generated, it will be shown on the command line. The log file that will be generated will contain something like

ttp://localhost:8000/eden/default/index - PASSED
http://localhost:8000/eden/admin/index - PASSED
http://localhost:8000/eden/default/person - PASSED
http://localhost:8000/eden/default/user/change_password - FAILED
...

As of right now, smoke tests are a little slow. So, the next step will be to optimize the speed of the smoke tests.

Data driven testing

What is Data-Driven testing?

Data-driven approach to testing is where test cases use only one higher-level keyword, normally created as a user keyword (as shown above), that hides the actual test workflow.

These tests are very useful when there is a need to test the same scenario with different input and/or output data. It would be possible to repeat the same keyword with every test, but the test template functionality allows specifying the keyword to use only once. Thus, a testsuite file will run the same Test template with a lot of test data, each of which will be a separate testcase. Example follows.

Using Data-Driven Testing

Now, testing the login functionality with various permutations of email and password can be done using the Data-driven approach. Here is how:

*** Settings ***
Documentation  Test case to check the login functionality of Eden
Library  ../resources/main.txt

*** Variables ***
${LOGIN URL}  http://${SERVER}/eden/default/user/login
${SUBMIT}  xpath=//input[@class='btn' and @value='Login']
${EMAIL ID}  auth_user_email
${PASSWORD ID}  auth_user_password


*** Keywords ***
Login with email and passwd
    [Documentation]  Opens a browser to login url, inputs username and password
    [Arguments]  ${email}  ${passwd}
    Open Browser  ${LOGIN URL}
    Input Text  ${EMAIL ID}  ${email}
    Input Text  ${PASSWORD ID}  ${passwd}
    Click Button  ${SUBMIT}

Login should fail
    [Documentation]  Opens a browser to login url, inputs invalid username or password and checks for failure
    [Arguments]  ${email}  ${passwd}
    Login with email and passwd  ${email}  ${passwd}
    Page Should Contain  Invalid login


*** Test Cases ***
The email is invalid  nottheadmin@example.com  testing
The password is Invalid  admin@example.com  incorrect
Both are Invalid  nottheadmin@example.com  incorrect

There are three test cases above, all of which implement the keyword Login should fail given against the setting Test Template. All the testcases implement the keyword with the arguments given against it. The keyword Login should fail takes two arguments and asserts that the login should fail for that case.

Further exercise

As an exercise, implement these two tests in a separate file

• Login form validation
• Login should pass [email] [password]

Also, try and improve the Teardown and Setup of the above tests. Refer to this ​documentation for help. Also, try and improve the Teardown and Setup of the above tests. Refer to this ​documentation for help.