Combining Test Regression Failure Reports
After hacking away on some code/test changes yesterday I clicked the magic "commit" button and wandered off for a drink, a few minutes later I returned to find 60 or so "regression failure" comments against a bunch of JIRA tickets - turns out the tests passed from within IDEA but failed when ran from Ant and the build server.
I was happy to have had the immediate feedback of failing tests (although based on the anti-spam hate mail I received - not every one shared my happiness)., The problem stemmed from the fact the tests in question used TestNG's data provider support so JIRA was commented with failures for each iteration of the tests (plus the tests were related to multiple tickets).
Not wanting to receive any more hate mail from coworkers I hacked around a bit and altered the reporter to generate a nicer combined failure report:
Issues are now commented once per issue per stack trace, with the comment now also including test parameter information. I'm still not entirely happy with this thou as it still has the potential to generate a large number of comments/emails, an alternative idea I'm now considering is to create a new issue which includes all of test failures, parameters and stack traces in a single combined report, with JIRA issue links to the related issues. I'm thinking this will give the best of both worlds: automated regression feedback, centrally recorded feedback with context, low spam.
Now I just have to code it...