Continuous automated web tests using Jenkins and JMeter

Objective

The Objective is to 

• Run regression functional web tests at periodic interval
• Run tests using using open source continuous integration server and testing tool
• Create common test scripts which can be used to test different server base URLs. We should be able to test an application which is deployed in different application servers by providing base URL at the run time as a parameter
• Make assertions against web requests
• Create meaningful report from the test results, ignore unnecessary information
• Generate HTML report to publish as a webpage
• Pass or fail jenkins test job depending on the result

Tool Selection

• Jenkins as continuous integration server as our development team also uses it to build artifacts and run unit tests.
• JMeter is free and can run in windows/linux
• JMeter has a GUI. Easier for a new person to create and edit the test scripts. No programming knowledge needed
• JMeter can do assertions what other test tools can also do
• JMeter functional tests can be converted to load tests if needed
• JMeter supports all request methods i.e. GET/POST/PUT/PATCH/DELETE etc
• JMeter can be run as command line and parameters can be passed as command line arguments. No need to hard code them in the test script.

Prerequisite

• JMeter is installed
• Jenkins is installed
• Saxon xslt and query processor is installed. It can be downloaded from http://sourceforge.net/projects/saxon/files/Saxon-HE/9.6/
• Basic knowledge of creating JMeter scripts
• Basic knowledge of creating a jenkins job

Test Setup

• Let us first create a simple JMeter test script. Let's say I want to test the URL http://localhost:8080/test
  a) Open JMeter. Right click on Test plan and click Add>Threads>Thread Group. Keep the number of threads as 1 as we we want do a functional test with 1 user
  b) Right click on Thread Group. Click Add>Config Element>HTTP Request Defaults. Enter localhost as server name , 8080 as port. This is going to be default server name and port for all HTTP requests. While creating the HTTP requests you only need to specify page path. If you want to pass these values at run time enter ${__P(Server)} as server name and ${__P(Port,)} as port number. Note that in ${__P(Port,)}, there is there is no value after comma as I want no port number to be used if the Port parameter is not passed at command line. This is for handling a URL like www.google.com
  c) Right click on Thread Group. Click Add>Sampler>HTTP Request. Give a meaningful name to the http request. As path, enter /test. If you want to parameterize this too, enter ${__P(ContextPath)} in Path.
  d)  Right click on HTTP Request. Click Add>Assertions>Response Assertion. Select Response code, click on ignore status. Click on Add and Type 200 in the box. This means that we are expecting HTTP 200 as HTTP status and we want to continue the test even if it fails.
  e) Right click on Thread Group. Click Add>Listeners>Assertion Results. Enter file name as C:\jmeter-reports\result.xml. Click on Configure and select following
    

  
  

  

  
  

  
  
  f) Let us save this test plan as C:\jmeter-scripts\test.jmx
  
  

• Now we will run this test from jenkins and generate HTML report.
  a) In Jenkins, create a new item,give a name and choose freestyle project
  b) In the configuration page, Add a build step as "Execute windows batch command". Enter following code

  del /f /q report.html
  del /f /q "C:\jmeter-reports\result.xml"
  
  cd C:\apache-jmeter-2.13\bin
  jmeter -n -t C:\jmeter-scripts\test.jmx -JServer=localhost -JPort=8080 -JContextPath=test

In the above code block, report.html is the html report that we will generate in the following step. We want to delete the old result.xml and report.html generated from previous run. Note that we are passing Server name, Port and ContextPath at run time while running jmeter test from command line. This code block will run the test and generate result.xml in C:\jmeter-reports path.

c) We now need to generate a meaningful HTML report using result.xml. For this we will use a xslt (source given below). Save it as C:\jmeter-scripts\transform.xsl. 

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

<xsl:template match="/">

<html>

<body>
<h2>Test Results</h2>

<table class="sortable" cellpadding="2" cellspacing="2" border="1">
<thread>
<tr bgcolor="#9acd32">
<th style="text-align:left">Test Name</th>
<th style="text-align:left">Request URL</th>
<th style="text-align:left">Pass</th>
<th style="text-align:left">Elapsed Time(ms)</th>
<th style="text-align:left">Response Code</th>
<th style="text-align:left">Failure Message</th>
</tr>
</thread>
<xsl:for-each select="testResults/httpSample">
<tr>
<td><xsl:value-of select="current()/@lb"/></td>
<td><xsl:value-of select="java.net.URL"/></td>
<td><xsl:value-of select="current()/@s"/></td>
<td><xsl:value-of select="current()/@t"/></td>
<td><xsl:value-of select="current()/@rc"/></td>
<td>
<xsl:if test="assertionResult/failureMessage">
<ul>
<xsl:for-each select="assertionResult">
<li> <xsl:value-of select="failureMessage"/></li>
</xsl:for-each>
</ul>
</xsl:if>
</td>
</tr>
</xsl:for-each>
</table>

</body>

</html>

</xsl:template>
</xsl:stylesheet>

d) Add a build step as "Execute windows batch command". Enter following code.

cd C:\Program Files\Saxonica\SaxonHE9.6N\bin
Transform -t -s:C:\jmeter-reports\result.xml -xsl:C:\jmeter-scripts\transform.xsl -o:C:\Jenkins\workspace\<jobname>\report.html

 We use Saxon xslt and query processor (http://sourceforge.net/projects/saxon/files/Saxon-HE/9.6/). In the above example, replace <<jobname> with the actual jenkins job name. The html report will be generated in the workspace folder of your jenkins job

• Now we need to pass or fail the jenkins job depending on test result. If the result.xml contains the text "<failure>true</failure>", then some test has failed.
  So we Add a build step as "Execute windows batch command". Enter following code 

  FIND /C "<failure>true</failure>" C:\jmeter-reports\result.xml
  if %errorlevel% EQU 0 Exit 1
  Exit 0

  When FIND is a success, errorlevel is returned as 0. So we are checking if there is any failure. If so, exit jenkins job as 1 (failure), else exit as 0 (success)

• We need to archive the html report. Add a post-build action, Select "Archive the artifacts" and type "*.html" as files to archive

 When you run the jenkins job, you will get a html artifact (report.html) which contains output similar to following