Strategies for separating unit and integration tests, using Maven, Eclipse / IDEA, cobertura

What I have

  • Maven2
  • Eclipse/IntelliJ IDEA
  • Fast tests (unit tests)
  • Slow tests (integration tests)
  • Test coverage maven plugin (Cobertura)

I wish to

  • run my fast tests (unit tests) from my IDE
  • run my slow test (integration tests) from my IDE and maven
  • run them all before committing code.
  • create test coverage reports combining unit and integration tests
  • create test coverage reports from unit tests only


If you’re like me, you wish to run all your fast unit tests often! And the process shouldn’t take more than 15 sec in total. In Eclipse or IDEA, you simply right click the project or the src/test/java and run tests. But how do you exclude your integration tests? Move them?
In maven2, you may create include and exclude statements, like **IntegrationTest.java. This is all fine, but in my frustrated this doesn’t help you in Eclipse or IntelliJ.

I’ve googled a lot, stating that there’s gotta exist a simple solution to this!? I found some unsatisfactory suggestions here
http://docs.codehaus.org/display/MAVENUSER/Maven+and+Integration+Testing
They might be good, but they don’t meet my needs of combining IDE test run and combined code coverage! Below is a list of the various suggestion, with the one I ended up with at the end.

  1. Separate Integration Tests Module. Works for both maven and Eclipse. But what about code coverage? You will not get the combined code coverage of fast and slow test. You may say this is a cobertura (clover as well I guess) issue, that it doesn’t instrument packaged jars.
  2. Test Suites. Create a test suite for integration tests and one for unit tests. With hundreds of tests it can get quite tedious. I guess you may code it more intelligently to find the tests, but I have not seen a good solution yet.
  3. src/test/java and src/it/java. Would solve everything elegantly, but we’ll have to wait until maven supports it. (Update 29. sep 09: Bertrand commented that the mojo build-helper-maven-plugin might help. Report back to me if it does!)
  4. my.package.tests.a.b.c and my.package.it.a.b.c. Looks lik 3 but is very different! The tests should be in the same package as the class it’s testing. Else you would suddenly find yourself having to open up the domain object, as it were a simple bean! Protect your domain objects, make package protected constructors or method if needed for the tests.
  5. Run the tests in maven only. Yes, many chooses this… But it’s plain ineffective if it takes 10 sec to run them in Eclipse and 1 minute (often 2) with “maven test”. Besides you often need to include “clean”, maven clean test.
  6. TestNG. I got this to work for all, maven, IntelliJ IDEA and cobertura. But there are things about TestNG I don’t like. Solution and elaboration follows below.

TestNG supports
@Test (groups ={“unit-tests”}) and
@Test (groups = {“integration-tests”})
which are easy to use in IDEA. The Maven Surefire Plugin also understands it and the Cobertura instrumentation works fine when you select to run them all together. Yeah!

But what I hate about TestNG is that it doesn’t provide the execution time per test nor the test results for each test class, only a summary at the end “Tests run: 272 failed:1 … duration:5 minutes” or something like that. Goggling this all I find is arrogant answers to the this question, stating that “That’s the way testNG is supposed to work” or “Why are you using it if you don’t utilize it’s parallel test execution potential”. Bullocks ;)

Wishlist:

  • maven supporting the directory src/it
  • Junit supporting grouping as testNG does

Oh, you know a better way? Great news! Tell me:)

EDIT 23.jul-09 : I added some stuff from my pom file. Will not work out of the box, but should be helpful anyhow.

<profiles>
 <!-- Profile for running both TestNG integration tests and Junit/TestNG unit
      tests. Example of usage: mvn clean install -Pintegration-tests -->
 <profile>
  <id>integration-tests</id>
   <build>
    <plugins>
     <plugin>
      <artifactId>maven-surefire-plugin</artifactId>
      <configuration>
       <forkMode>once</forkMode>
       <argLine>-XX:MaxPermSize=256m -Xmx512m</argLine>
       <includes>
        <include>**/*Tests.java</include>
        <include>**/*Test.java</include>
       </includes>
      <excludes>
       <exclude>**/Abstract*.java</exclude>
      </excludes>
      <groups>unit-tests,integration-tests</groups>
     </configuration>
    </plugin>
   </plugins>
  </build>
 </profile>

If you’re using Hudson, include this to display it in the build results and look at test coverage history.

<!-- If we are running in hudson use cobertura.   -->
 <profile>
  <id>hudson</id>
  <activation>
   <property>
    <name>BUILD_NUMBER</name>
   </property>
  </activation>
  <build>
   <plugins>
    <plugin>
     <groupId>org.codehaus.mojo</groupId>
     <artifactId>cobertura-maven-plugin</artifactId>
    </plugin>
   </plugins>
  </build>
 </profile>
About these ads

About Ole Morten Amundsen

Developer, programmer, entrepreneur. Java, .Net, ruby, rails, agile, lean. Opinionated enthusiast!
This entry was posted in agile, integration, Java, maven, Testing and tagged , , , , , , , . Bookmark the permalink.

10 Responses to Strategies for separating unit and integration tests, using Maven, Eclipse / IDEA, cobertura

  1. Cedric says:

    It’s certainly trivial to write a TestNG reporter that will calculate whatever times you need, feel free to email the testng-users mailing-list if you need some guidance on how to do that.

    As for the arrogant comments, I certainly hope they didn’t come from me :-)

    • Ole Morten says:

      I should watch my step when saying such a thing, as there are always just a few people ruining it for the rest. I have a feeling that you’re not one of them :)

      TestNG reporter? That’s great, I don’t know how I could have missed it. Thanks for the tip! A link is always appreciated.
      My colleague wrote a time-taker using Advice to start counting upon @BeforeMethod and @AfterMethod. It helped us find the slower tests so we could focus on them. It was custom, but sufficient, maybe that’s why I never came to discover the reporter. Cheers!

  2. Cedric says:

    You can use either Listeners or Reporters, both of which discussed here:

    http://testng.org/doc/documentation-main.html#logging

  3. I managed to put together some simple test-categories for JUnit4 by using the Spring TestContext framework’s @IfProfileValue annotation. I blogged about it some time ago and there are some code snippets here.

    • Ole Morten says:

      Thanks for the input, Thomas! I’ve used both JUnit4 and Spring extensively, but never used @IfProfileValue. Too bad, obviously. Does this solution works fine with the IDE as well? Would you mind providing an example for, take Eclipse, that enables you to run all unit (fast) tests in a project, and similarily, run all integration tests?

      Hope you’re doing well in Germany! Let me know when you’re back in Norway so that I may buy you a beer :)

      • It will take me some time to put together an example, but I’ll try. Anyhow, it’s really realy easy to start off with cause the concept is simple.

        It requires Spring 2.5+, and you need to have @RunWith(SpringJUnit4Runner.class) or something on top of the test-class. Then write @IfProfileValue(name=”integration”,value=”true”) on top of a test, and it will only execute if you run JUnit/Java with a -Dintegration=true argument (forgive me if there are typos in there). You can configure this in all IDE’s quite easily (run configurations..).

        So, a pretty easy technique for excluding a category of tests from running, but far from as powerful as the TestNG bells and whistles, of course.

  4. Bertrand Renuart says:

    The “build-helper” maven plugin may give you a solution to your third point (tests located in two separate directories). See http://mojo.codehaus.org/build-helper-maven-plugin/index.html

    Another solution might be to point the (from the secton) to a different path depending on the active profile. But this won’t probably integrate very well with your IDE (still a single source directory at once).

    Hope this helps ;-)

  5. Ole Morten says:

    Thanks Bertrand! I’ve always wanted alternative 3, separating unit tests and integration tests in source folders. I can’t verify it, but it looks like it could do the trick! A friend of mine uses the mojo build-helper for adding a source for autogenerated crappy code. As it is an option to a maven helper, I guess you could configure maven test to run what you want.

    Great addition Bertrand, I’ll update nr 3 with your comment :)

    • Bruno Rezende says:

      I’ve installed mojo build-helper-maven-plugin to help in a situation like yours, I’ve added a new test source folder and it worked pretty well in eclipse and jenkins. However, I don’t have code coverage reports setup.

  6. ClasspathSuite extends JUnit to provide the filtering capability, e.g.:

    @RunWith(ClasspathSuite.class)
    @ClassnameFilters({“!.*IT” })

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s