TestPackage

Set your tests free


$ java -jar AboutTestPackage.jar
✔ Packages up tests to be executed anywhere, any time (1 ms)
✔ Just command line; no dev tool dependencies (1 ms)
✔ Uses coverage to select smoke tests for you (1 ms)
✔ Clear, easy to scan output (1 ms)
✔ Runs recently-failed tests first, with optional fail-fast support (1 ms)
✔ Compatible with regular JUnit tests and Jenkins (1 ms)
✔ Test sharding for easy parallel running (1 ms)


*** TESTS COMPLETE
*** 7 passed, 0 failed, 0 ignored

TestPackage is a simple Java library designed to make it easy to bundle JUnit tests in standalone executable JAR form, rather than the traditional model of running them from an Ant/Maven/Gradle build script.

It is not intended for running unit tests, which clearly need to be built around the units of code under test (on the same classpath). However, for situations like integration tests, functional tests and smoke tests, inclusion in a build script adds unnecessary complexity and makes some opportunities harder to grasp.

Additionally, TestPackage aims to provide more useful and intuitive output than other JUnit runners give, for example making it easier to spot assertion errors and root causes of exceptions at a glance.

Features



Clean, clear output

TestPackage is designed to display test output cleanly and clearly, enabling developers and ops people to quickly get to the core of any issues.

Test subsets based on coverage

TestPackage can use JaCoCo to work out which tests achieve the most coverage, and to select which tests to run to achieve a desired coverage in the quickest time. 50% coverage good enough for deployment smoke tests or a first pass of functional tests? Specify --optimize-coverage=50% and let the tool figure out which tests to run. Or want to the best tests that fit into five minutes? --optimize-time=5m will do this for you.

Makes tests part of your deployment pipeline

TestPackage helps you to version and pipeline your tests just like your deployable binaries - build-once-run-many, with no dev tool requirements in the environment you want to run your tests.

Fits in with existing tooling

Executes standard JUnit tests and produces JUnit XML reports, ready for use with Continuous Integration tools.

Externalised configuration

Configurable properties (such as per-environment paths, URLs and other settings) can easily be loaded in from a properties file at run time.

Standalone test execution

Supports running regular JUnit tests standalone from the command line. This is ideal for tests that don't belong as part of the build process, such as functional, infrastructure or smoke tests.

Test prioritisation

Runs recently failed tests first, on the basis that these are most likely to identify persistent problems. This can be particularly powerful when used with fail-fast mode - checking whether you've fixed a failing test no longer involves waiting for dozens of other stable tests to be run first.

Test sharding

Allows tests to be divided across multiple executors for parallel running. Executors do not need to have any network connectivity to distribute tests, and configuration is simple.

Visible steps

Provides `VisibleAssertion` replacement for `org.junit.Assert` which makes it easier to identify causes of step errors.

How it works

  1. You build a JAR file containing your test classes with TestPackage as a dependency. The JAR must point to org.testpackage.TestPackage as the main class which is executed when the JAR is run. The TestPackage maven plugin helps make assembling this JAR file easier (see below).
  2. When the JAR is run, TestPackage orchestrates JUnit to run your test classes. If tests have been run before, it uses historical data to help prioritize/select which tests to run.
  3. Throughout the test execution TestPackage gives nice colorized output on the console, and summarizes causes for test failures at the end. Also, it generates JUnit XML report files which can be parsed by your CI tool.

Simple object diagram

Usage

examples/maven is a simple Maven project which uses TestPackage. The following elements are core prerequisites for using TestPackage:

TestPackage library dependency

<dependency>
    <groupId>org.testpackage</group>
    <artifactId>testpackage</artifactId>
</dependency>

Maven plugin to compile test sources, dependencies and TestPackage into a single executable JAR

Note that package-name must be set to the package where test classes reside. flags may be set to configure JVM properties for the fat jar if invoked as a plain unix executable.

<plugin>
    <groupId>org.testpackage</groupId>
    <artifactId>testpackage-maven-plugin</artifactId>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>fatjar</goal>
            </goals>
            <configuration>
                <package-name>org.testpackage.example.maven</package-name>
                <flags>-Xmx2G</flags>
            </configuration>
        </execution>
    </executions>
</plugin>

Configuration of integration test source location

N.B. This is not needed if you don't mind keeping integration test sources under src/main/java)

<plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>build-helper-maven-plugin</artifactId>
    <version>1.4</version>
    <executions>
        <execution>
            <id>add-test-source</id>
            <phase>generate-sources</phase>
            <goals>
                <goal>add-source</goal>
            </goals>
            <configuration>
                <sources>
                    <source>${basedir}/src/int-test/java</source>
                </sources>
            </configuration>
        </execution>
    </executions>
</plugin>

Building standalone test JAR with Maven

With the configuration outlined above, simply run mvn clean package to produce an executable JAR.

Running the tests

Once built, execute the build JAR file (e.g. named maven-example-1.0-SNAPSHOT.jar) by running:

java -jar target/maven-example-1.0-SNAPSHOT.jar

Of course, this JAR file can be moved anywhere on the filesystem, deployed to an artifact repository, etc (e.g. with mvn deploy).

After tests have run, a directory named .testpackage will be created in the working directory. This directory contains some metadata used internally by TestPackage to prioritise recently-failed tests over tests that have never failed or not failed recently. To get the most value out of the test prioritisation support, you should not remove this folder. In fact, you should check this folder into version control or configure your CI system to retain it between test runs. Doing so will ensure that CI always runs recently-failed tests first.

While the above runs with sensible defaults, the following arguments may be passed at the command line to customise behaviour:

Usage

If you have built a fat jar file which includes TestPackage as a dependency, it can be run as follows:

java -jar JARFILE [OPTIONS] [ARGUMENTS]

If you have used the TestPackage maven plugin to build your test fat jar, it will be directly executable as well as being executable via java -jar:

java -jar JARFILE [OPTIONS] [ARGUMENTS]
or
JARFILE [OPTIONS] [ARGUMENTS]

Options

Fail fast

--failfast or -ff: Abort testing when the first failure occurs (good for validation before pushing changes to the team repository)

Quiet mode

--quiet or -q: Enables quiet mode (hides all test output)

Verbose mode

--verbose or -v: Enables verbose mode (shows all test output, including stdout/stderr)

Sharding

--shard=n: Set the index of the current test shard (default: 0)

--numShards=n: Set this to the total number of shards (default: 1)

Loading environment variables from a file

--propertiesfile=name or -P name: Load a properties file (either plain Java .properties or XML) and set system properties accordingly during test run.

Coverage optimization

Recording coverage data

N.B. To record coverage data:

  • org.testpackage:coverage-jacoco must be on the test package's classpath
  • The JacoCo coverage agent must be injected into the system under test in TCP server mode (e.g. adding a Java agent option like -javaagent:path/to/jacocoagent.jar=output=tcpserver)

See examples/coverage for an example.

--jacoco-host: Specify which host is running the system under test (default: localhost)

--jacoco-port: Specify which port to find the JaCoCo agent on (if running. default: 6300)

Optimizing tests once coverage data has been collected

--optimize-coverage=n%: Tell TestPackage to aim for n% test coverage. If attainable, it will try to select the quickest subset of tests that achieve this coverage.

--optimize-time=x: Tell TestPackage to aim for x test execution time (e.g. 1m, 1min, 1h). It will try and select a subset of tests which achieve the best coverage without exceeding this execution time.

Arguments

The full package names which should be searched (non-recursively) for test classes

Usage examples

Run all test classes found in org.example.tests:

target/functests.jar org.example.tests

Run tests spread across three different test executors (note: afterwards, JUnit test report files output to target need to be merged separately):

target/functests.jar --shard=1 --numShards=3 org.example.tests          # On executor 1
target/functests.jar --shard=2 --numShards=3 org.example.tests          # On executor 2
target/functests.jar --shard=3 --numShards=3 org.example.tests          # On executor 3

Record test coverage (e.g. with the app being tested running in Tomcat):

# Start Tomcat with a JaCoCo coverage recording agent
export CATALINA_OPTS="$CATALINA_OPTS -javaagent:path/to/jacocoagent.jar=output=tcpserver"
/path/to/tomcat/bin/startup.sh

# Run all tests
target/functests.jar org.example.tests

Run an optimized subset of tests using previously captured coverage data. e.g. we might want to run a smoke test that takes no longer than 5 minutes, but achieves the best coverage possible in that time:

target/functests.jar --optimize-time=5min org.example.tests
Created by Richard North

Page template based on Pratt created by BLACKTIE.CO. This site uses glyphs from Typicons.