Maven's WAR Overlay: What are WAR Overlays?

What is a WAR overlay?

def: Overlays are used to share common resources across multiple web applications. The dependencies of a WAR project are collected in WEB-INF/lib, except for WAR artifacts which are overlayed on to the WAR project itself. http://maven.apache.org/plugins/maven-war-plugin/overlays.html

Why and where would you use a WAR overlay?

WAR overlays make up one more tool in your arsenal to combat repetitive code and/or resources. Overlays allow you to combine many similar resources, for example, images, CSS and JavaScript files all into one project. At build time, the overlay will be predictably merged into a master WAR project. A great example of a WAR overlay is a branded website, combined with Sitemesh decorators. Imagine you have a branded company website which consists of multiple web applications. An industry standard approach would be to utilize Sitemesh decorators to decorate the application with brand specific styles and allow each of your applications to focus on the specific purpose of that application. At this time, this blog post will not go into depth on Sitemesh decorators, however, more details are available at http://www.sitemesh.org/.

Leading on with our example, each of the decorators will need a set of CSS, images and possibly JavaScript files to define the application's branded style. If your applications are to be deployed to production to look and act as one cohesive branded site, you will need to make sure that the images, CSS, JavaScript, decorator files and JSP files are all the same in each of your applications in order to keep a consistant brand across your applications. Well, this is duplicated code (currently ranked in my top 5 biggest "code avoid-at-all-cost" list).

How do you solve code duplication using WAR Overlays?

One approach to reducing the amount of duplicated code is to create a separate maven project with the packaging target type of WAR. Then in each of your application's pom.xml file, include the overlay project as a dependency and construct it as an overlay using the maven-war-plugin. This is known as a Fat WAR Overlay. (NOTE: In the next post I will cover other types of WAR overlays to help mitigate transient dependency problems)

More specifics please!

First, let's create a new maven WAR project and name it "branded-templates". (NOTE: This post is not intended to be a comprehensive guide on creating a maven project, only adding to an already working maven project. If you need more assistance, please refer to some of my other post or maven's website for more guidance.)

In our new branded-templates application, we will need to fill this with our resources. Here is my recommended locations to place web-related resources within your project:

• /src/main/webapp/styles – used to hold your CSS files, (example: brand.css, reset.css, etc)
• /src/main/webapp/scripts/ – used to hold brand-wide JavaScript files, (example: jquery.1.x.x-min.js, jquery-my-company-validation.js, etc
• /src/main/webapp/images/ – used to hold brand-wide images, (example: company-logo.png, facebook-icon.png, company-background.jpg, etc)

Naming for now will not be important, but you will have to make decisions here because you have two naming options here that will have development implications. You can name your resources and/or folders the same in both of your applications or you can name them different to get different desired results. How is this going to impact my design and why do you need to think about this? (Answer coming after we add to the containing webapp project, keep following!)

Let's continue. We can now install your branded-templates WAR overlay to your local maven repository:

mvn clean install

If all is good, you should see a "Successful build" along with a 1.0-SNAPSHOT for your WAR's version.

Next, let's crate a new main application called "main-webapp". If you have an existing webapp, feel free to use that and work along with this example. Inside of the main-webapp's pom.xml file, we need to add a plugin to build the two WARs into one WAR and add the overlay as a dependency to the project.

Add to your dependencies section:

<dependency>
   <groupId>com.yourcompany</groupId>
   <artifactId>branded-templates</artifactId>
   <type>war</type>
</dependency>

Note that we are adding this as a WAR type, do not leave the type blank.

Add to your plugins section:

<plugin>
 <groupId>org.apache.maven.plugins</groupId>
 <artifactId>maven-war-plugin</artifactId>
 <version>2.1.1</version>
 <configuration>
   <overlays>
     <overlay>
       <groupId>com.yourcompany</groupId>
       <artifactId>branded-templates</artifactId>
     </overlay>
   </overlays>
 </configuration>
</plugin>

Next, we've added the war-plugin which will take our application and build a WAR. If you're familiar with maven's artifact types, you'll already know that your project builds a WAR already, so why the plugin? The plugin is where the "magic" of the overlay happens. This is where the overlay is placed into the main-webapp's WAR file.

How many WAR files will I have?

When using a WAR overlay, you will start with two or more WAR files (presumably one or more overlays and a main-webapp), but will ultimately end up with one WAR artifact (your main-webapp) in which to deploy to your environment. Though it is completely possible to run your overlay as a standalone application, I recommended you read my next post (coming soon) on dependency management within overlay projects to avoid dependency problems in your main-webapp project.

So what happens when there is a conflicting resource?

As you can imagine, there can be conflicts between overlay's resource names and the main project's resources. This is what I asked you to think about from the question posed above. If you have the same location and named file in your overlay application as you do in your main application, the overlay version will be ignored for the main application's version. This can work for or against you, so think about how you might use both. How would you want to leverage the customization abilities in the main webapp.

Note, if you plan on utilizing Sitemesh decorators, it's not a bad idea to put your decorators.xml and sitemesh.xml file in your WEB-INF folder and the corresponding JSPs in the WEB-INF folder (suggest placing them in /WEB-INF/decorators folder). Remember to add the sitemesh web.xml information into your containing main-webapp's web.xml!

One more gotcha that might arise out of this would be dealing with the web.xml. The overlay's web.xml file is NOT copied over nor is it merged in any way, so if your overlay has specific web.xml file information you can either manually add it to the web.xml or use a <include file="/WEB-INF/web-overlay.xml"> tag where you've added the contents to a file named web-overlay.xml to the overlay war project.

Tips for rapid development

At this point you should have an overlay and a working master webapp leveraging that WAR overlay. Since you have your resources in the overlay and you are probably working in your main-webapp, and it is probably becoming a pain to keep deploying a new version of your overlay project, you can simply refer to the SNAPSHOT version of your overlay inside your main-webapp pom.xml file and do a local install of your overlay.

Leaving notes

Follow my blog and get the next blog post dealing with dependency management with overlays.

Feel free to leave comments or questions and I'll do my best to get back to you in a reasonable time frame. I would like to thank Dynacron Group (my consulting firm) for allowing me some time to work on this blog post!

Unit Testing Named Queues: Spring 3+maven2+Google App Engine

Intro
Problem, you have a task that you know can take more than 30 seconds to complete, what do you do?  What if this task needs to be triggered every day at a specific time?  Google provides several mechanisms to to solve just this problem, queues and scheduled task, respectively.

Queues
First, let's explore the "queue" system implemented in GAE.  Note, at this time, the "queue" is still experimental which means that the API can change around some, so make sure you have strong unit testing, but don't worry, we'll cover that a bit further down the page.  To implement queues, you need to let the web app know that your expecting queues to be used, and to do this, you need to create a file named queue.xml in your WEB-INF folder (example: /src/main/webapp/WEB-INF/queue.xml).  The queue.xml file has a specific layout and that will not be covered in depth in this post.  Please read up on the queue.xml structure on Google's documentation.  We will expand on the knowledge since the documentation is not complete.

Scheduled Task
Scheduled task are task that will trigger by GAE at repeatable times, emulating linux cron, or Quartz application.  To implement a cron task, simply create a file called cron.xml and place it in your WEB-INF folder (example:/src/main/webapp/WEB-INF/cron.xml).  This post will not go into detail on the implementation of scheduled task so please check Google's documentation to get a firm understanding before reading on. In order to not pack too much into one post, Scheduled Task will be covered in the next post.

Our Problem
The scenario we're going to explain is: You have an birthday tracking software and you want a user's friends to be notified, via email, that their friend is having a birthday.  Assume that the code for the services named are tested and fully working.

To solve this we will use a queue and the scheduled task system provided by Google App Engine.

First, let's tackle the queue portion, then we will create the scheduled task.  Google's documentation on unit testing queues is very limited and only shows how to retrieve the "default" queue.  Since we could have multiple task in our system that need to be run at different rates, we will want to create a new Queue.  Let's open (or create) the queue.xml file and insert the following:

<?xml version="1.0" encoding="UTF-8"?>
<queue-entries>
    <queue>
        <name>birthdayemail</name>
        <rate>10/s</rate>
        <bucket-size>10</bucket-size>
    </queue>
</queue-entries>

Next, let's create a simple service (Note: assume that the Autowired resources are working and imports are included at the top of the file, this has been reduced to emphasize the solution)

@Service("birthdayService")
public class BirthdayServiceImpl implements BirthdayService {

    // this is the same name as in your queue.xml file
    private static final String EVENT_REMINDER_TASK_NAME = "birthdayemail";
    
    @Autowired
    private BirthdayDAO birthdayDAO;

    public List handleGuestEmailsForBirthday(Date date) {
        if (date == null) {
            throw new IllegalArgumentException("Date can not be null");
        }
        List<Guest> guestBirthdays = birthdayDAO.getByDate(date);
        for (Guest guest : guestBirthdays) {
            String TASK_URL = "/queue/birthday-emails/" + guest.getId()
            final TaskOptions taskOptions = TaskOptions.Builder.url(TASK_URL);
            // create a unique task name, note, must conform to [a-z] regex
            taskOptions.taskName(guest.getName() + "Task");
            taskOptions.method(TaskOptions.Method.POST);

            Queue queue = QueueFactory.getQueue(EVENT_REMINDER_TASK_NAME);
            // commented out to show the default queue shown in the docs
            //Queue queue = QueueFactory.getDefaultQueue();
            queue.add(taskOptions);
        }
        return events;
    }
}

Let's look at what's here:
First, we have to create a URL that will be run when the task is implemented. This URL will handle the unit of work specified, and should run in under 30 seconds per GAE's restrictions. We will not cover what this servlet does, just assume it uses the ID given in the URL path, pulls up the Guest, then uses that to get the Guest's friends' emails; then uses that to construct an email and pass to an EmailService. (HINT: the EmailService could also implement a queue so each email is separated out to run individually)

Next, notice that we are implementing a "Task Name". The this is optional, but does help when debugging to figure out what task is failing or what is running.

Now we're on to the TaskOptions. TaskOptions are a helper function used to combine URL and model data to pass on to the servlet handling the queue. Just a suggestion, but to follow the REST ideals, setting the RequestMethod to POST or PUT is advisable, depending on what you are trying to do. All Task should be idempotent, meaning that if the task fails or is interrupted, the task can be run again and not harm the data.

Lastly, we have the QueueFactory retrieving the Named Queue (the same name in the queue.xml file). Simply add your taskOptions object to the queue.add and move on.

Unit Testing Named Queues
Next, let's setup a unit test to test our named queue.  The Google documentation does not explicitly explain how to properly setup a test case for one, datastore aware context and named queries.  Below is an example of a test case written to test our Named Queries.  (Note, as before, imports have been removed and the services used have been used before.  This is not intended to be the end-all-be-all for unit testing GAE, please refer to previous post for more help, and yes, you might be able to use JMock or Mockito instead of persisting data, but I have not explored the option).

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations = {"/testApplicationContext.xml"})
public class QueueBirthdayServiceTest
        extends AbstractJUnit4SpringContextTests {

    @Autowired
    private BirthdayService birthdayService;
    private final LocalServiceTestConfig[] configs = {
        new LocalDatastoreServiceTestConfig(),
    /* NOTE: THE QUEUE XML PATH RELATIVE TO WEB APP ROOT, More info below */
        new LocalTaskQueueTestConfig()
                .setQueueXmlPath("src/test/resources/test-queue.xml")};
    private final LocalServiceTestHelper helper
            = new LocalServiceTestHelper(configs);

    @Before
    public void setUp() {
        helper.setUp();
    }

    @After
    public void tearDown() {
        helper.tearDown();
    }

    @Test
    public void testHandleBirthdayQueueEmails() throws InterruptedException {
        // setup data
        Date birtdayDate = DateUtils.parse("10/16/1900");
        // assume this builds the data for guest/guest's friends
        int guestWithBD = 5; // # of birthdays on date
        int gPerBirthday = 10; // # of emails being sent per birthday
        setupDataForBirthdays(birthdayDate, guestWithBD, gPerBirthday);

        List<Guest> birthdayOnDay = birthdayService.
                handleGuestEmailsForBirthday(birthdayDate);
        assertEquals(guestWithBD, birthdayOnDay.size());
        // pause for a moment to allow queue to fill from previous statment
        Thread.sleep(1000);
        // verify # of birthdays with that day's expire date
        LocalTaskQueue ltq = LocalTaskQueueTestConfig.getLocalTaskQueue();
        final Queue queue = QueueFactory
                .getQueue(BirthdayService.EVENT_REMINDER_TASK_NAME);
        //final Queue queue = QueueFactory.getDefaultQueue();
        QueueStateInfo qsi = ltq.getQueueStateInfo()
                .get(queue.getQueueName());
        assertNotNull(qsi);
        int expectedTaskCount = guestWithBD*gPerBirthday;
        assertEquals(expectedTaskCount, qsi.getTaskInfo().size());
        assertEquals(birthdayOnDay.get(0).getID() + "Task",
                qsi.getTaskInfo().get(0).getTaskName());

    }
}

NOTE: If the syntax or unit testing is very foreign to you, please visit my previous post on Unit Testing on Google App Engine.

Let's explore more
The first half is not too exiting, just setting up data and calling the service that creates the queue.  We are creating 5 guest with birthdays on the given date, and each of those 5 guest have 10 friends who we intend to email.  This service is already "written" out (above)

What you need to pay attention to is at the top of the file, during the setup of the "helper", we have constructed it with a LocalDatastoreServiceTestConfig and LocalTaskQueueTestConfig objects. The second, the "LocalTaskQueueTestConfig" is the important one to add when using a queue. If you are not using the DefaultQueue, you will need to explicitly state where the queue.xml file is. I suggest that you create a test-queue.xml file and place it in your /src/test/resources folder, so as to not mix production data and testing. NOTE: This file is loaded based relative to the ROOT application folder. The rest of the test should be pretty self explanatory.

In the next post, we will uncover how to wire the service up to a Scheduled Task so you can automate the emails being sent. Stay tuned!