Automated Test Regression Reporting with TestNG and JIRA

Published: 6:10 PM GMT+12, Saturday, 7 October 2006 under: technology
java  testing  testng  agile  extremeprogramming  regression  atlassian  jira 

Back in 2003 I wrote about my habit of annotating code with @jira javadoc tags, for a long time I've wanted to move these from javadoc annotations to JDK5 annotations and do something "smart" with them - well now I have...

The other day on the TestNG mailing list I'd mentioned a crazy idea of having TestNG automatically send comments to bugs when tests related to them start failing - something an automated regression failure report.

This afternoon I spent a few hours hacking around and now the TestNG Issue Reporter project lives. The project consists of two modules - annotations and reporter. The annotations package simply defines two annotions: @RelatedIssue and @RelatedIssues with the reporter module providing the TestNG integration.

Once you've added the projects jars to your classpath you can update your tests to refer to your issues:

package com.theoryinpractice.testng;

import org.testng.annotations.Test;

public class TestIssue {

    @Test
    @RelatedIssue("jira://localhost/TST-1")
    public void testSomething() {
        assert false : "Test failed - boo hoo go cry to mother...";
    }
}

This is just a standard TestNG test with the addition of the @RelatedIssue annotation. The annotation takes a single parameter identifying the issue we say the test relates to. In the above example we're saying its a) jira, b) on the server at localhost, c) and has a key/id of TST-1. Initially only the jira "protocol" is supported but I could envisage added support for bugzilla, traq, dev.java.net etc.

Including the listener in your test process is a simple matter of modifying your TestNG launch script:

java -ea -DtestngIssueUsername=talios -DtestngIssuePassword=XXXX \
  -cp ...... \
  org.testng.TestNG ./suite.xml -listener com.theoryinpractice.testng.IssueReporter

Authentication details such as username and password are currently supplied by system parameters and are global, but will probably switch to some form of configuration file to allow for separate authentication details for different issue servers (in the case you might wish to comment against multiple bug trackers).

After successfully failing a test, you should see something like the following commented against your related JIRA issue:

The aim of the project is to increase the visibility of any potential regression problems, the hope is that you'd never seen any of these comments, but now if a test fails - you'll likely get an email from JIRA alerting you to the problem with the advantage of also giving basic contextual information about the original problem.

Comments (1)

It's a great idea to post comments in JIRA for related issue, but I have problem with @RelatedIssues and @RelatedIssueSource annotation using. 1. I have two jars in my test project(see pom.xml fragment): <dependency> <groupId>com.theoryinpractice.testng</groupId> <artifactId>testng-issues-annotations</artifactId> <version>1.0-SNAPSHOT</version> </dependency> <dependency> <groupId>com.theoryinpractice.testng</groupId> <artifactId>testng-issues-reporter</artifactId> <version>1.0-SNAPSHOT</version> </dependency> 2. TestNG eclipse plug-in version - 5.8.0.2 3. My project is a seam ear project, that imported in Eclipse(seam-version - 2.1.1.GA);jboss server 4.2.2; 4. Selenium test code listed below: package org.richfaces.testng.rf4507;

import org.richfaces.SeleniumTestBase; import org.testng.Assert;

import com.theoryinpractice.testng.RelatedIssueSource; import com.theoryinpractice.testng.RelatedIssues;

@RelatedIssueSource(value="https://jira.jboss.org") public class Test extends SeleniumTestBase {

private String getIdSelector(String id) { return "//*"; } private void testCalendarSwitch(int buttonPosition, String cellText) { renderPage();

AssertTextEquals(getIdSelector("form:calendarDayCell6") + "/div", "4-10");

selenium.assignId(getIdSelector("form:calendarHeader") + "//td][" + buttonPosition + "]/div", "testButton"); fireMouseEvent("testButton", "click", 0, 0, false); waitForAjaxCompletion();

AssertTextEquals(getIdSelector( "form:calendarDayCell6") + "/div", cellText); } private void testLink(String linkId, String data) throws Exception { renderPage();

Assert.assertEquals("null", selenium.getEval("window.receivedData"));

clickAjaxCommandAndWait(getIdSelector(linkId));

Assert.assertEquals(data, selenium.getEval("window.receivedData")); } @org.testng.annotations.Test @RelatedIssues("RF-6454") public void testNextMonth() throws Exception { testCalendarSwitch(4, "12-13"); } @org.testng.annotations.Test public void testPreviousMonth() throws Exception { testCalendarSwitch(2, "6-9"); }

@org.testng.annotations.Test public void testFirstLink() throws Exception { testLink("form:link1", "firstLink"); } @org.testng.annotations.Test public void testSecondLink() throws Exception { testLink("form:link2", "secondLink"); } public String getTestUrl() { return "pages/rf4507.xhtml"; }

} 5. "Run dialog settings"->"Arguments"->"Program Arguments" text area contains: -listener com.theoryinpractice.testng.IssueReporter 6. I run suite of tests, but failing of methods with @RelatedIssues annotation doesn't lead to post comment in JIRA, just following log in console: ... Processing test failure Found an associated issue Looking for @RelatedIssueSource on org.richfaces.testng.rf4507.Test...

There is no new comment for RF-6454 issue. Am I doing something wrong?

left by mihey.vit . Tuesday, 10 March 2009 10:23 PM
Add Comment