September 25, 2012

Testing Custom Exceptions w/ JUnit's ExpectedException and @Rule


Exception Testing

Why test exception flows? Just like with all of your code, test coverage writes a contract between your code and the business functionality that the code is supposed to produce leaving you with a living documentation of the code along with the added ability to stress the functionality early and often. I won't go into the many benefits of testing instead I will focus on just Exception Testing.

There are many ways to test an exception flow thrown from a piece of code. Lets say that you have a guarded method that requires an argument to be not null. How would you test that condition? How do you keep JUnit from reporting a failure when the exception is thrown? This blog covers a few different methods culminating with JUnit's ExpectedException implemented with JUnit's @Rule functionality.


The "old" way

In a not so distant past the process to test an exception required a dense amount of boilerplate code in which you would start a try/catch block, report a failure if your code did not produce the expected behavior and then catch the exception looking for the specific type. Here is an example:

public class MyObjTest {

    @Test
    public void getNameWithNullValue() {

        try {
            MyObj obj = new MyObj();
            myObj.setName(null);
            
            fail("This should have thrown an exception");

        } catch (IllegalArgumentException e) {
            assertThat(e.getMessage().equals("Name must not be null"));
        }
    }
}

As you can see from this old example, many of the lines in the test case are just to support the lack of functionality present to specifically test exception handling. One good point to make for the try/catch method is the ability to test the specific message and any custom fields on the expected exception. We will explore this a bit further down with JUnit's ExpectedException and @Rule annotation.


JUnit adds expected exceptions

JUnit responded back to the users need for exception handling by adding a @Test annotation field "expected". The intention is that the entire test case will pass if the type of exception thrown matched the exception class present in the annotation.

public class MyObjTest {

    @Test(expected = IllegalArgumentException.class)
    public void getNameWithNullValue() {
        MyObj obj = new MyObj();
        myObj.setName(null);
    }
}

As you can see from the newer example, there is quite a bit less boiler plate code and the test is very concise, however, there are a few flaws. The main flaw is that the test condition is too broad. Suppose you have two variables in a signature and both cannot be null, then how do you know which variable the IllegalArgumentException was thrown for? What happens when you have extended a Throwable and need to check for the presence of a field? Keep these in mind as you read further, solutions will follow.


JUnit @Rule and ExpectedException

If you look at the previous example you might see that you are expecting an IllegalArgumentException to be thrown, but what if you have a custom exception? What if you want to make sure that the message contains a specific error code or message? This is where JUnit really excelled by providing a JUnit @Rule object specifically tailored to exception testing. If you are unfamiliar with JUnit @Rule, read the docs here.


ExpectedException

JUnit provides a JUnit class ExpectedException intended to be used as a @Rule. The ExpectedException allows for your test to declare that an exception is expected and gives you some basic built in functionality to clearly express the expected behavior. Unlike the @Test(expected) annotation feature, ExpectedException class allows you to test for specific error messages and custom fields via the Hamcrest matchers library.

An example of JUnit's ExpectedException

import org.junit.rules.ExpectedException;

public class MyObjTest {

    @Rule
    public ExpectedException thrown = ExpectedException.none();

    @Test
    public void getNameWithNullValue() {
        thrown.expect(IllegalArgumentException.class);
        thrown.expectMessage("Name must not be null");

        MyObj obj = new MyObj();
        obj.setName(null);
    }
}

As I eluded to above, the framework allows you to test for specific messages ensuring that the exception being thrown is the case that the test is specifically looking for. This is very helpful when the nullability of multiple arguments is in question.


Custom Fields

Arguably the most useful feature of the ExpectedException framework is the ability to use Hamcrest matchers to test your custom/extended exceptions. For example, you have a custom/extended exception that is to be thrown in a method and inside the exception has an "errorCode". How do you test that functionality without introducing the boiler plate code from the try/catch block listed above? How about a custom Matcher!

This code is available at: https://github.com/mike-ensor/custom-exception-testing


Solution: First the test case

import org.junit.rules.ExpectedException;

public class MyObjTest {

    @Rule
    public ExpectedException thrown = ExpectedException.none();

    @Test
    public void someMethodThatThrowsCustomException() {
        thrown.expect(CustomException.class);
        thrown.expect(CustomMatcher.hasCode("110501"));

        MyObj obj = new MyObj();
        obj.methodThatThrowsCustomException();
    }
}

Solution: Custom matcher

import com.thepixlounge.exceptions.CustomException;
import org.hamcrest.Description;
import org.hamcrest.TypeSafeMatcher;

public class CustomMatcher extends TypeSafeMatcher<CustomException> {

    public static BusinessMatcher hasCode(String item) {
        return new BusinessMatcher(item);
    }

    private String foundErrorCode;
    private final String expectedErrorCode;

    private CustomMatcher(String expectedErrorCode) {
        this.expectedErrorCode = expectedErrorCode;
    }

    @Override
    protected boolean matchesSafely(final CustomException exception) {
        foundErrorCode = exception.getErrorCode();
        return foundErrorCode.equalsIgnoreCase(expectedErrorCode);
    }

    @Override
    public void describeTo(Description description) {
        description.appendValue(foundErrorCode)
                .appendText(" was not found instead of ")
                .appendValue(expectedErrorCode);
    }
}

NOTE: Please visit https://github.com/mike-ensor/custom-exception-testing to get a copy of a working Hamcrest Matcher, JUnit @Rule and ExpectedException.

And there you have it, a quick overview of different ways to test Exceptions thrown by your code along with the ability to test for specific messages and fields from within custom exception classes. Please be specific with your test cases and try to target the exact case you have setup for your test, remember, tests can save you from introducing side-effect bugs!

5 comments:

Anders Skarby said...

Hey, I really like the API (it's a convernient way of typing something that's very repetative) - and I'm ready to "give it a spin" in a couple of my projects.

The only remaining question is, how do I include it in a maven project? Sure, I could easily inject it into my local repository, but I'd prefer to grab it from the central Maven repo - any plans on pushing it to there?

It might also be that it's already in the central Maven repo, but I simply can't seem to locate it?

Any help would be appreciated,
Thanks!

Mike! said...

Hey Anders,

Thanks for the feedback, I appreciate it! This blog posting is using the @Rule and ExpectedException class available in the JUnit 4.7+ libraries. I would recommend using the "junit-dep" with version 4.10 (*future proof* or the latest).

You can look at a working example here: https://github.com/mike-ensor/custom-exception-testing.git

In my next blog I will be publishing a solution for using Fest assertions and ExpectedException and that will be available in Maven and github.

Mike!

Tomek said...

Hi Mike,

why implement your own solution when you have catch-exception library ready to be used? See http://code.google.com/p/catch-exception/

Mike! said...

Hey Tomek,

Most likely there are hundreds of possible solutions out there by many different developers and teams. This method is one that JUnit (Kent Beck) decided was simple and clear. I agree with Kent on the simplicity and clarity of the solution. In terms of use, the syntax is typically only one or two lines (depending on custom Hamcrest matchers) and I believe that the code is very easy to read.

The framework you are referring to is a mid-sized framework that just duplicates the functionality provided by JUnit and I generally do not like to duplicate code/functionality. In addition, custom functionality must be written using Hamcrest matchers, so the same amount of work must be done.

One more important flaw I can point out is that the DSL for the TDD is similar (or the same-as) Mockito which can make the code hard to read when the static methods need to be fully qualified.

Let me know if you'd like me to clarify my points above.

Mike!

Sandeep Bhandari said...

I am new to @Rule annotation which seems to be introduced in JUnit 4.7. But when researching on the topic of testing exceptions, I also came across this guy who is saying that his magic test library can allow for testing exceptions in pre 4.7 versions too. See the blog post: magic test, specially the table shown by him on that page. Would like to know your comments on the same. Also since I am beginner on JUnit, I also see that @Rule annotation can be applied for timeout of unit tests. So does this mean that @Rule is a generic annotation and we can use it for multiple purposes. right!