September 25, 2012

Testing Custom Exceptions w/ JUnit's ExpectedException and @Rule


Exception Testing

Why test exception flows? Just like with all of your code, test coverage writes a contract between your code and the business functionality that the code is supposed to produce leaving you with a living documentation of the code along with the added ability to stress the functionality early and often. I won't go into the many benefits of testing instead I will focus on just Exception Testing.

There are many ways to test an exception flow thrown from a piece of code. Lets say that you have a guarded method that requires an argument to be not null. How would you test that condition? How do you keep JUnit from reporting a failure when the exception is thrown? This blog covers a few different methods culminating with JUnit's ExpectedException implemented with JUnit's @Rule functionality.


The "old" way

In a not so distant past the process to test an exception required a dense amount of boilerplate code in which you would start a try/catch block, report a failure if your code did not produce the expected behavior and then catch the exception looking for the specific type. Here is an example:

public class MyObjTest {

    @Test
    public void getNameWithNullValue() {

        try {
            MyObj obj = new MyObj();
            myObj.setName(null);
            
            fail("This should have thrown an exception");

        } catch (IllegalArgumentException e) {
            assertThat(e.getMessage().equals("Name must not be null"));
        }
    }
}

As you can see from this old example, many of the lines in the test case are just to support the lack of functionality present to specifically test exception handling. One good point to make for the try/catch method is the ability to test the specific message and any custom fields on the expected exception. We will explore this a bit further down with JUnit's ExpectedException and @Rule annotation.


JUnit adds expected exceptions

JUnit responded back to the users need for exception handling by adding a @Test annotation field "expected". The intention is that the entire test case will pass if the type of exception thrown matched the exception class present in the annotation.

public class MyObjTest {

    @Test(expected = IllegalArgumentException.class)
    public void getNameWithNullValue() {
        MyObj obj = new MyObj();
        myObj.setName(null);
    }
}

As you can see from the newer example, there is quite a bit less boiler plate code and the test is very concise, however, there are a few flaws. The main flaw is that the test condition is too broad. Suppose you have two variables in a signature and both cannot be null, then how do you know which variable the IllegalArgumentException was thrown for? What happens when you have extended a Throwable and need to check for the presence of a field? Keep these in mind as you read further, solutions will follow.


JUnit @Rule and ExpectedException

If you look at the previous example you might see that you are expecting an IllegalArgumentException to be thrown, but what if you have a custom exception? What if you want to make sure that the message contains a specific error code or message? This is where JUnit really excelled by providing a JUnit @Rule object specifically tailored to exception testing. If you are unfamiliar with JUnit @Rule, read the docs here.


ExpectedException

JUnit provides a JUnit class ExpectedException intended to be used as a @Rule. The ExpectedException allows for your test to declare that an exception is expected and gives you some basic built in functionality to clearly express the expected behavior. Unlike the @Test(expected) annotation feature, ExpectedException class allows you to test for specific error messages and custom fields via the Hamcrest matchers library.

An example of JUnit's ExpectedException

import org.junit.rules.ExpectedException;

public class MyObjTest {

    @Rule
    public ExpectedException thrown = ExpectedException.none();

    @Test
    public void getNameWithNullValue() {
        thrown.expect(IllegalArgumentException.class);
        thrown.expectMessage("Name must not be null");

        MyObj obj = new MyObj();
        obj.setName(null);
    }
}

As I eluded to above, the framework allows you to test for specific messages ensuring that the exception being thrown is the case that the test is specifically looking for. This is very helpful when the nullability of multiple arguments is in question.


Custom Fields

Arguably the most useful feature of the ExpectedException framework is the ability to use Hamcrest matchers to test your custom/extended exceptions. For example, you have a custom/extended exception that is to be thrown in a method and inside the exception has an "errorCode". How do you test that functionality without introducing the boiler plate code from the try/catch block listed above? How about a custom Matcher!

This code is available at: https://github.com/mike-ensor/custom-exception-testing


Solution: First the test case

import org.junit.rules.ExpectedException;

public class MyObjTest {

    @Rule
    public ExpectedException thrown = ExpectedException.none();

    @Test
    public void someMethodThatThrowsCustomException() {
        thrown.expect(CustomException.class);
        thrown.expect(CustomMatcher.hasCode("110501"));

        MyObj obj = new MyObj();
        obj.methodThatThrowsCustomException();
    }
}

Solution: Custom matcher

import com.thepixlounge.exceptions.CustomException;
import org.hamcrest.Description;
import org.hamcrest.TypeSafeMatcher;

public class CustomMatcher extends TypeSafeMatcher<CustomException> {

    public static BusinessMatcher hasCode(String item) {
        return new BusinessMatcher(item);
    }

    private String foundErrorCode;
    private final String expectedErrorCode;

    private CustomMatcher(String expectedErrorCode) {
        this.expectedErrorCode = expectedErrorCode;
    }

    @Override
    protected boolean matchesSafely(final CustomException exception) {
        foundErrorCode = exception.getErrorCode();
        return foundErrorCode.equalsIgnoreCase(expectedErrorCode);
    }

    @Override
    public void describeTo(Description description) {
        description.appendValue(foundErrorCode)
                .appendText(" was not found instead of ")
                .appendValue(expectedErrorCode);
    }
}

NOTE: Please visit https://github.com/mike-ensor/custom-exception-testing to get a copy of a working Hamcrest Matcher, JUnit @Rule and ExpectedException.

And there you have it, a quick overview of different ways to test Exceptions thrown by your code along with the ability to test for specific messages and fields from within custom exception classes. Please be specific with your test cases and try to target the exact case you have setup for your test, remember, tests can save you from introducing side-effect bugs!

September 18, 2012

Brief Overview of Java Assertions

What are asserts?

An assertion is a predicate (a true–false statement) placed in a program to indicate that the developer thinks that the predicate is always true at that place. [wikipedia]

Traditional asserts

Traditional testing frameworks started with the built in keyword assert.

An example of the keyword assert:

assert <condition> value

Assert has a few drawbacks including stopping the test execution and lengthy and hard to describe assert statements.

Second generation assertions

Along comes JUnit's assert framework. Built on top of the assert keyword, JUnit provided developers the ability to be more descriptive about the testing statements.

An example of JUnit's asserts:
// asserts that the condition must be true
assertTrue("This should be true", "abc".equalsIgnoreCase("ABC"));
// asserts that the object must not be null
assertNotNull(new MyObject());
//...
assertFalse(false == true);
//
assertNull(null);
// etc...

While there are some improvements on readability and usability to the basic assert keyword provided by JUnit, they share some of the same drawbacks in that many developers just use the "assertTrue()", "assertEquals()" and "assertFalse()" methods still providing a very cryptic assertion statement.

Third generation assertions

In an effort to guide developers into writing test assertions that are more readable and usable the Hamcrest library was created which switched the philosophy from many assert functions to just one basic function. The fundamental thought is that the assert is the same but the conditions will change. Hamcrest was built using the concepts of BDD (behavior driven design) where the test assertion is closer to that of a sentence.

An example of hamcrest assertions

@Test
public void showoffSomeHamcrestAssertsAndMatchers() {
    // asserts that string "abc" is "ABC" ignoring case
    assertThat("abc", is(equalToIgnoringCase("ABC")));
    assertThat(myObject.getFirst(), is("Mike!"));
    assertThat(myObject.getAddress(), is(notNullValue));
}

Hamcrest is a great improvement on top of the JUnit framework providing a flexible and readable testing platform. Hamcrest + JUnit is a comprehensive testing framework and when combined with Mockito (or other mocking framework) can provide a very descriptive and thorough unit testing solution. One of the drawbacks to using Hamcrest is that while descriptive, multiple assertions must be made to ensure that a test case has been covered. Many TDD purest agree that a test case should contain one and only one assertion, but how is this possible with a complex object? (purest will say refactoring, but oftentimes this is not feasible)

Fourth generation frameworks

And finally we come to the present with the latest assertion frameworks. Fest Assertion framework takes off where Hamcrest stopped providing a fluent style assertion framework giving the developer the ability to test objects with a single assertion.

An example of Fest assertions

@Test
public void getAddressOnStreet() {
    List<Address> addresses = addressDAO.liveOnStreet("main");
    assertThat(addresses).hasSize(10).contains(address1, address2);
    assertThat(stringObj).hasSize(7).isEqualToIgnoringCase("abcdefg");
}

As you can see, FEST exceptions provide a cleaner approach to assertions. FEST is very extensible and easy to use.

Conclusion

Assertions are a key tool in a professional developer's toolbox in which to stress test their code. This post is a part in a series with the culmination being a fluent style ExpectedException mechanism, backed by Fest, in which to add better clarity to your test cases when exceptions are being introduced. Feel free to read the lead up to this post; Allowing known failing JUnit Tests to pass test cases.

NOTE: This blog is providing an overview on assertion frameworks it should be noted that there are various other second and third generation testing frameworks that strive to provide better clarity and usability for testing including notables such as TestNG and JTest

September 9, 2012

Allowing JUnit Tests to Pass Test Case on Failures


Why create a mechanism to expect a test failure?

There comes a time when one would want and expect a JUnit @Test case fail. Though this is pretty rare, it happens. I had the need to detect when a JUnit Test fails and then, if expected, to pass instead of fail. The specific case was that I was testing a piece of code that could throw an Assert error inside of a call of the object. The code was written to be an enhancement to the popular new Fest Assertions framework, so in order to test the functionality, one would expect test cases to fail on purpose.


A Solution

Jump to the Code
One possible solution is to utilize the functionality provided by a JUnit @Rule in conjunction with a custom marker in the form of an annotation.


Why use a @Rule?

@Rule objects provide an AOP-like interface to a test class and each test cases. Rules are reset prior to each test case being run and they expose the workings of the test case in the style of an @Around AspectJ advice would.

Required code elements

  • @Rule object to check the status of each @Test case
  • @ExpectedFailure custom marker annotation
  • Test cases proving code works!
  • Optional specific exception to be thrown if annotated test case does not fail

NOTE: working code is available on my github page and has been added to Maven Central. Feel free to Fork the project and submit a pull request

Maven Usage

<dependency>
    <groupId>com.clickconcepts.junit</groupId>
    <artifactId>expected-failure</artifactId>
    <version>0.0.9</version>
</dependency>

Example Usage

In this example, the "exception" object is a Fest assertion enhanced ExpectedException (look for my next post to expose this functionality). The expected exception will make assertions and in order to test those, the test case must be marked as @ExpectedFailure

public class ExceptionAssertTest {

    @Rule
    public ExpectedException exception = ExpectedException.none();

    @Rule
    public ExpectedTestFailureWatcher watcher = ExpectedTestFailureWatcher.instance();

    @Test
    @ExpectedFailure("The matcher should fail becasue exception is not a SimpleException")
    public void assertSimpleExceptionAssert_exceptionIsOfType() {
        // expected exception will be of type "SimpleException"
        exception.instanceOf(SimpleException.class);
        // throw something other than SimpleException...expect failure
        throw new RuntimeException("this is an exception");
    }
}

Implementation of Solution

Reminder, the latest code is available on my github page.

@Rule code (ExpectedTestFailureWatcher.java)

import org.junit.rules.TestRule;
import org.junit.runner.Description;
import org.junit.runners.model.Statement;
// YEAH Guava!!
import static com.google.common.base.Strings.isNullOrEmpty;

public class ExpectedTestFailureWatcher implements TestRule {

    /**
     * Static factory to an instance of this watcher
     *
     * @return New instance of this watcher
     */
    public static ExpectedTestFailureWatcher instance() {
        return new ExpectedTestFailureWatcher();
    }

    @Override
    public Statement apply(final Statement base, final Description description) {
        return new Statement() {
            @Override
            public void evaluate() throws Throwable {
                boolean expectedToFail = description.getAnnotation(ExpectedFailure.class) != null;
                boolean failed = false;
                try {
                    // allow test case to execute
                    base.evaluate();
                } catch (Throwable exception) {
                    failed = true;
                    if (!expectedToFail) {
                        throw exception; // did not expect to fail and failed...fail
                    }
                }
                // placed outside of catch
                if (expectedToFail && !failed) {
                    throw new ExpectedTestFailureException(getUnFulfilledFailedMessage(description));
                }
            }

            /**
             * Extracts detailed message about why test failed
             * @param description
             * @return
             */
            private String getUnFulfilledFailedMessage(Description description) {
                String reason = null;
                if (description.getAnnotation(ExpectedFailure.class) != null) {
                    reason = description.getAnnotation(ExpectedFailure.class).reason();
                }
                if (isNullOrEmpty(reason)) {
                    reason = "Should have failed but didn't";
                }
                return reason;
            }
        };
    }
}

@ExpectedFailure custom annotation (ExpectedFailure.java)

import java.lang.annotation.*;

/**
 * Initially this is just a marker annotation to be used by a JUnit4 Test case in conjunction
 * with ExpectedTestFailure @Rule to indicate that a test is supposed to be failing
 */
@Documented
@Retention(RetentionPolicy.RUNTIME)
@Target(value = ElementType.METHOD)
public @interface ExpectedFailure {
    // TODO: enhance by adding specific information about what type of failure expected
    //Class assertType() default Throwable.class;

    /**
     * Text based reason for marking test as ExpectedFailure
     * @return String
     */
    String reason() default "";
}

Custom Exception (Optional, you can easily just throw RuntimeException or existing custom exception)

public class ExpectedTestFailureException extends Throwable {
    public ExpectedTestFailureException(String message) {
        super(message);
    }
}

Can't one exploit the ability to mark a failure as expected?

With great power comes great responsibility, it is advised that you do not mark a test as being @ExpectedFailure if you do not understand exactly why the test if failing. It is recommended that this testing method be implemented with care. DO NOT use the @ExpectedFailure annotation as an alternative to @Ignore

Possible future enhancements could include ways to specify the specific assertion or the specific message asserted during the test case execution.

Known issues

In this current state, the @ExpectedFailure annotation can cover up additional assertions and until the future enhancements have been put into place, it is advised to use this methodology wisely.