Testing NServiceBus Applications

Testing these days is thankfully more common place but there is still some confusion in terminology so i’ll start by clarifying what I mean by the terms I use later.

Unit Test – A unit test can be run on any computer that can compile your code, it does not need relational databases or other infrastructure like the file system to run or pass.

Integration Test – An integration test does rely on some infrastructure being present. It tests that more than one piece of technology works with another. For example that a repository can persist data to a database.

Acceptance Test – A test that requires the entire application be deployed and tests journey’s designed to simulate user activity in the system.

I’m not going to talk about Unit test’s as far as i’m concerned they are a known quantity with NServiceBus. Just select your mock object library of choice and start writing your tests. That said the amount of repetitive code you have to write as you mock out handler after handler does seem tiresome.

Our team is trying to automate as much of the testing as is reasonably possible. We want developers to be able to write tests that they can run locally and that will run against our WIP, UAT & RC environments to indicate the state of those environments. Our application is divided up into SOA serivces. So when a developer is working on a feature ticket they will likely only be writing code for a single service.

We started out with the idea that we would write unit tests and acceptance tests and that would give us enough coverage and to be honest that is still true to this day. What has changed is that our codebase has grown and correspondingly the build / deploy / test times have increased as well. So what could we do about it? Well the first thing we did was start rewriting our build scripts to perform tasks in parallel, this process is trickier than we first imagined as dependencies complicated matters. If not done properly it could destabilize our development environment so whilst this will help in the long term it will not solve our problems today. Another thing that happened as our application grew, the amount of resources it took to deploy the entire system onto a single machine also grew. We started to have to upgrade developer workstations to have 16Gb of Ram just to be able to run the entire application locally. After a while it became clear that this approach of deploying the entire application locally was not going to work forever but after some thought it seemed that it did not have to. We had already split our app into SOA services why not just build / deploy / and test the service you were creating a feature for? It turns out that this is not only straight forward but that it has other benefits when it comes to testing.

So now we just have the service that we are working on to worry about instead of taking 15 minutes to build the entire codebase we only spend 90 seconds building one service. When it comes time to deploy the changes we only need to update the endpoints we have modified, locally we deploy a SOA service to just one endpoint this takes only 1 minute. Then we write “integration tests” that simply inject commands and events into our SOA service’s input queue that simulate user input and events published by the other services. The benefit of this approach is that simple tests take < 2 seconds to complete and complicated ones take less than 10 seconds. These test times might seem a little bit longer than you would expect the next paragraph explains why. Testing Asynchronous Messaging Applications Martin Fowler talks about the highs and lows of testing asynchronous applications and I can confirm from first hand experience that he is correct. If you inject a command into your input queue and then wish to make an assertion against a query that expects that data to be written to a database then you have to poll that database for a reasonable amount of time to check the data. Your test then either passes because the data is returned as expected or the timeout period is reached. This sounds simple and it is but only when you have figured out exactly what you need to poll for. If your system requires that messages be processed in a particular order then your tests must poll to ensure that the preceding message has be processed before you inject a subsequent one. In and ideal world you’ll have everything setup so that your system can handle messages in any order but in the real world you may not. What gets even more interesting is when you need to make data modifications to simulate time passing. As soon as your tests start modifying data then they are likely to encounter deadlocks, so you must include retry logic inside your tests if you need to update data like this. We have created a framework that helps make developers a lot more productive it has functionality to poll for a specified interval say every second or it can use a dynamically increasing interval that will check after 250ms, 500ms,1sec, every 5secs this helps reduce test times and shorten feedback. We are using the MbUnit framework which has some nice features for running tests in parallel which can dramatically reduce test run times.

This post is a work in progress but I wanted to put it out there to see what others are doing so if you have anything you like to share or disagree with the above let me know via twitter.

Groking Greg Young’s Super Simple CQRS Example

Greg Young has coded up a sample CQRS app with the aim of creating the simplest possible implementation.

I have been looking through the code to try and better understand CQRS.
Greg Young’s Blog
Download the Sample App Here

The diagram above is based on stepping through the code after the user submits a new Inventory Item.

I’m still not sure I fully understand things but it is starting to become clearer.

 

Monitoring NServiceBus Message Queues

I have seen a few posts on the NServiceBus yahoo groups mailing list and a few tweets asking about how best to monitor NServiceBus. This is a problem I have been researching over the last few weeks, here is ths solution I came up with for triggering alerts when too many messages are in a queue.

Performance Counters

MSMQ Exposes several performance counters out of the box, these can be accessed via WMI. There are a few caveats, only active queues or queues with messages will have performance counters see here for details.

Triggering Alerts

You could write your own WMI scripts to monitor these performance counters and trigger alerts, another option is to use a monitoring tool that has the ability to setup WMI monitors. OpManager is such a tool, in a couple of minutes you can configure a new monitor that will trigger emails or SMS’s to be sent if a certain threshold is reached.

Improving NServiceBus

I don’t know enough about the innner workings of NServiceBus to make an educates assement of how we could create an NServiceBus monitoring tool but I’ll try anyway.

Based on these facts:

  • Every NServceBus Service knows about the Queues it is configured to use for sending / recieving messages.
  • It should be simple to make an NServiceBus service discoverable.

Would it not be possible to create a generic NServiceBus.Monitor.exe with a provider model that would allow plugins to be written for any monitoring tool / protocol?

Edit:

Found this
http://monitorwang.codeplex.com/

Understanding NServiceBus Subscription Storage and Profiles

Official NserviceBus Documentation

The NServiceBus Host
Profiles for the Generic
Host

Our Deployment Details

Version: NServiceBus v2.0 RTM
OS: Windows 2003 Server
App Type: Windows Service
Host: NServiceBus.Host.exe
Subscription Storage: MSMQ

Our Goal

We have just created a new publisher service and it has taken a bit of time to understand exactly how to deploy it so that subscriptions were persisted after the service is restarted.

Out of the box NServiceBus has three options for storing subscription data, Memory, MSMQ or Database.

To use MSMQ you must either create your own custom profile that configures subscriptions to be stored in MSMQ or you must select Integration from  the default profiles.

Regardless of the choice you make you must specify the profile you wish to use when you install the windows service, here is how you do that:

Basic Integration Example:
NServiceBus.Host.exe /install NServiceBus.Integration

Advanced Production Example:

NServiceBus.Host.exe /install NServiceBus.Production /serviceName:MyService /displayName:MyDisplayName/description:MyDescription /username:DOMAINusername /password:MyPassword

To check the service has been setup correctly you can go and look in the properties of the service and check the text at the end of the “Path to executable” field. The example in the screenshot below shows the service is configured to use the Lite profile.

We had one more gotcha our implementation was copy/pasted from one of 
our existing v1.9 services so we had to remove the .MsmqSubscriptionStorage()
option from our config. Now our endpoint config looks like this

public void Init()

{
            NServiceBus.Configure.With()
                        .DefaultBuilder()
                        .XmlSerializer()
                        .MsmqTransport()
                        .IsTransactional(true)
                        .PurgeOnStartup(false)
                        .UnicastBus()
                        .ImpersonateSender(false)
                        .CreateBus()
                        .Start();
}

The Default Profile’s Subscription Storage

Note: – If you select Production profile you may need to enable some settings inside MSDTC (in Component Services) if your database is on another server.
         – If you are installing on to a machine that has UAC enabled you must run the install from a cmd prompt with elevated priviledges (Run As Administrator)

nServiceBus Case Study – Inbound SMS Processing

We have been using nServiceBus in production for over 6 months now, we have had a very positive experience, so I wanted to show how we have been using it with some real world examples. One implementation we have just released handles inbound SMS messages and checks the sender’s number to see if they are an existing or new customer then reacts accordingly.

1) Our SMS provider calls our http handler (hosted in IIS) each time an inbound SMS is recieved.
2) The http handler calls the SmsGateway via a WCF service, which in turn publishes an InboundSmsMessage to the bus.
3) The Subscriber receives the InboundSmsMessage it interrogates the database to see if this is an exising customer or not.
4) Finally is sends a command to the worker service depending on whether this is an existing customer or not.
5) If the SMS is not from an existing customer then the worker initiates a call between the number and a CS agent.

Benefits of this approach

If their is a sudden surge in inbound Sms messages due to a successful advertising campaign we can control processing to avoid degrading overall system performance. We could further isolate our message processing from the core system by creating a cache on the Subscriber of all existing customer’s mobile numbers. This cache could be kept up to date by the publishing new customer messages.

If other services need to know about inbound SMS message say for invoice reconcilliation or just marketing reports they can subscribe to the messages and aggregate them. This can be done in realtime to allow the marketing team access to a dashboard that provides feedback on the effectiveness of their campaigns.

The most obvious benefit is that the Sms Gateway is autonomous, and the services that subscribe to the inbound messages need no understanding of what an SMS message is or it’s semantics.

UPDATE:
We have since realised that our implementation is not optimal, we are going to host NServiceBus in IIS and send and Inbound message notification to the SMS Gateway service and remove the WCF piece.

Grok’ing nServiceBus

First read this nServiceBus overview.

I’m recording my learning so I can refer to it later.

Messaging Concepts

Transport

For messaging to work you need to transport messages from one endpoint to another, once an application has created the message addressed it and put it into a queue it is up to the transport to attempt delivery. MSMQ is the default choice of nServiceBus.

Command Query Separation (CQS)

Wikipedia describes CQS as a pattern where “… every method should either be a command that performs an action, or a query
that returns data to the caller, but not both. More formally, methods
should return a value only if they are referentially transparent and
hence possess no side effects.”
Udi Dahan’s Blog Post on CQS

The Cost of Messaging

Using a messaging approach to building systems offers lots benefits that other approaches don’t but you rarely get something for nothing, Ayende’s article discusses the cost of messaging.

nServiceBus Concepts

Publish vs Send – Wiki Article

The Distributor

The distributor as it’s name suggests distributes work to be done to worker services. This avoids the need for the service that decides what work is to be done from having to contact all workers or have logic that handles balancing the work load.
Worker services inform the distributor how many idle threads they have available and it dispatches the same number of jobs. When a thread completes the distributor is notified so that if any jobs are in the work queue one will be sent.
This means an administrator can monitor the work queue to see if there are enough workers to handle the load.

nService Bus Code Samples Articles

Synchronous Web Service Bridge – Udi Dahan

Pub / Sub Walkthrough – Erik Westermann

Useful info for beginners – Matt Salmon

Installation and Config

Wiki Article on configuration

Getting started article on Art of BabelErik Westermann

Reviews of nServiceBus

Ayende Rahien’s Review and his review of the distributor

nService Bus Info on the Web

Mailing List on Yahoo Groups

Udi’s FAQ on the nServiceBus Website – Full of programming tips

Case Studies and Scenarios for Usage

Asynchronous High Performance Login Example – Udi Dahan

Update:

New packaging for NServiceBus 2.0

The link above is to Andreas Ohlund’s blog post that explains the differences between the packaging for version 1.9 and 2.0
The main difference in the introduction of a nServicebus.core.dll and nServiceBus.Host.exe

Hosting NServiceBus in IIS

Change this:

NServiceBus.Configure.With()...

to:

NServiceBus.Configure.WithWeb()...

from Mike Dellanoce’s Blog Post