NServiceBus – Internals message headers in MSMQ

NServiceBus Version: 3.3.8

Recently I ended up investigating how NServiceBus attaches headers to MSMQ messages. I’m recording my learning here as I found it difficult to track this information down on the web.

NServiceBus uses the Message.Extension property to attach a serialised Dictionary(<string>, <string) it uses the built in .net Xml Serialiser to serialise the object.

An example of the header XML :



  
		NServiceBus.Version
		3.3.8
	
	
		NServiceBus.TimeSent
		2013-07-04 15:40:14:019635 Z
	
	
		NServiceBus.RelatedTo
		7dc42ed0-7f95-4372-ba23-eea903c405816798665
	
	
		NServiceBus.EnclosedMessageTypes
		MyLib.Ns.MessageType, My.Assembly, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
	
	
		NServiceBus.OriginatingSagaId
		e96308e0-e2f1-48ae-916a-a1f10112b8ef
	
	
		NServiceBus.OriginatingSagaType
		MyLib.ns.Foo, My.Assembly, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
	
	
		WinIdName
		Domjon.smith
	
	
		CorrId
	

This function shows how NSB serialises the dictionary to XML using the frameworks Xml serialiser.

 private static byte[] SerialiseMessageHeaders(Dictionary headers)
        {
            using (var stream = new MemoryStream())
            {
                headerSerializer.Serialize(stream, headers.Select(pair => new HeaderInfo { Key = pair.Key, Value = pair.Value }).ToList());
                return stream.ToArray();
            }
        }

NSBCon 2014

nsbcon14-300-px-x-300-px

On June 26th I will be speaking at the first NServiceBus conference, in London. The conference hosted by Skillsmatter will run for two days where some great speakers like Greg Young and Udi Dahan will be giving talks on the technologies and techniques you can use to build distributed systems.

I will focus on how our engineering team have dealt with the problem of scaling our platform to meet rapid growth in customers whilst at the same time increasing our technical headcount from fifteen to two hundred people spread across five countries. I’ll also look at how we transitioned from an n-tier to a service oriented architecture, whilst dealing with pressure from the business to deliver new features and keep the platform stable.

Our customer growth was rapid we saw 10+% increase month on month for several years. Our n-tier architecture was brittle, when a 3rd party service went offline or ran slowly we would gradually see our web tier become starved of threads as the synchronous calls backed up. We tried to mitigate this by adding more servers and using async threads but we could not guarantee that our website would stay up if there was a prolonged 3rd party outage during a busy period. Our load is spikey, typically we see 2-3 times more activity in the last couple of days of the month than normal. We needed a system that would keep serving our customers when under heavy load or when one of our 3rd party services went offline or started to run in a degraded state. NserviceBus helped us acheive this because it is inherently asynchronous due to being built on top of a messaging transport layer. Now if a 3rd party service goes down or starts running slowly command messages either queue up or are transferred to the error queue for retrying later.

Our move from synchronous web service calls to asynchronous messaging did present challenges, we had to think differently about how we constructed our acceptance tests, often introducing polling into our tests to wait for messages to be processed. We developed a framework that assisted developers in writing these sort of tests. We also had to create testing tools that could inject test messages to simulate customer activity so that we could test specific parts of the system.

When integrating with with third party API’s that were mostly HTTP endpoints we used a pattern that ensured the non-transactional nature of HTTP would not detrimentally affect our customers. For example when processing card payments we did not want to risk collecting the same payment twice so we set out retries to zero and created a saga to gracefully handle exceptions.

Need_for_Speed_film_logo

As we moved into new regions and they started to gain traction the number of messages increased, in order to continue to meet our message processing SLA’s we had to split our handlers into separate endpoints. We found our monitoring solution proved invaluable as it allowed us to see the volume of each message type and the length of time it took to process them. Armed with this information it was a relatively straightforward task of identifying which message handlers needed to be moved into their own endpoint.

If these are the sort of problems you are working on and would like to know more you can find details about the conference here, if you have any questions feel free to contact me on Twitter my nickname is @porkstone.

Reprint: “Gaming Nostalgia” a post from Dualbotic.com on 23rd October 2003

Welcome to the misty water coloured memories of the experience that is networked gaming

In the beginning..

Before the light there was the Amiga A500, a quality gaming machine, then came the PC a Viglen 386 Sx 16mhz with a turbo button! (20mhz). One of the first games I played on this was good old elite. My bro then bought a modem and found out about a local BBS that you could dial up to for free during off peak hours called GamesNet BBS. You had to have a cable phone because BT don’t give much away for free. At first we just used the IRC to talk to Americans this was kind of novel at the time. Soon we learned that you could play games against other BBS’ers using something called Kali, this gave birth to Command & Conquer battles over the BBS.

Whilst chatting to some of the techies on the BBS we learned that it was possible to connect several PC’s together at home and form a home network. Not long after this my bro was invited to a 4pc LAN game of command & conquer owned by a dude who ran another BBS his nick was IronHammer. Anyway when he got back from the 24hr gaming marathon he was raving about how mad 2 vs 2 CnC was so as we had a couple of pc’s and also knew a few more peeps who also had them it was time to take a step into the unknown and setup our own network.

After chatting some more with the techies on the BBS we learned that every Saturday morning in the Tottenham Ct road area there were a number of computer fairs where goodies like network cards could be purchased at knock down prices. I think £80 secured enough cards, cable and terminators to setup the Lan. After some swearing and fiddling the first Token ring Lan in our house was up and running, there was no going back.

What followed can only be described as multi player madness, once the word got round that we had four player C&C up and running all our friends came over and a mammoth 48 hr non stop weekend gaming session ensued, the first of many such weekends. 4 player command and conquer provided some amazing spectacles, people falling asleep on the keyboard only to be replaced by someone else keen to get in on the action. I don’t think any game since has created that level of amazement and addiction. The only thing that would stop us from playing on was sheer exhaustion.

Well some months later the rumors were doing the rounds that Westwood were going to release a new game ‘Red Alert’ the rumors also said that this baby was going to support up to 8 players at once. This lead to a little head scratching, our current computer room (the upstairs lounge) didn’t have the capacity to take another four machines and we weren’t sure where we were going to find another four pc’s anyway, but in the immortal words of Bull Durham… ‘Build it and they will come.’ we set about extending the network upstairs. Sure enough the game came out and eventually we sampled the 8 player insanity that was CnC Red Alert.

With the advent of internet online gaming the need to construct a huge lan at home wained, nowadays we still have the techno shed but it usually only has four machines and sometimes a couple of guest pooters that friends bring round. The favourite games are Counter Strike, UO and Mankind.

This reprint was made possible due to The Wayback Machine

Rapoo S500 First Impressions

Rapoo S500 bluetooth headset

I recently purchased this headset on ebay for £35, it took forever to arrive. I wanted to see how it compared with the Rapoo H6020. When you pay this little for a rechargeable bluetooth headset with a built in mic, you can’t set your expectations too high. The best use case for this headset is with a mobile phone or tablet on your daily commute or when travelling. I have tried using bluetooth headsets with windows laptops but never with much success.

Ergonomics

I have only had the headset for a short while, it is very comfortable it feels lighter than the H6020. Rapoo have improved the headband just by making it flat. The H6020 headband could become uncomfortable when worn for long periods of time.

 

Bluetooth

Pairing with an iPhone or iPad is simple, when the headset is off simply press and hold the power button for five seconds this puts it into pairing mode. Rapoo say this headset uses Bluetooth 4.0 which should consume less power. Range is up to 30 meters in ideal conditions. I found you have to be in the same room as the source.

 

Sound

The sound quality when listening to music is fair but nowhere near as good as my Sony DR-BTN200 that cost nearly double. If you are looking for audiophile quality this is not the headset for you and you should be spending a lot more money. That said I think there is some improvement over the H6020. Volume is good, when I listen to podcasts on the underground I have to turn it up as loud as it goes but this is just enough to make the voices audible. If you worry about upsetting your fellow passengers when listening to your favourite death metal track then you’ll be pleased to know this headset leaks a minimal amount  of sound. There is no active noise reduction and a small amount of passive reduction, but this headset is not really suitable for listening in noisy conditions unless you don’t mind cranking up the volume to the maximum.

 

Charging and Battery Life

The headset can be charged using a standard micro USB cable, one is supplied in the box. Claimed battery life is 16 hours but I have not verified this yet. Charging time is one hour again, I have not verified this myself.

 

Microphone

The quality of the mic is okay but you can only hold a telephone conversation if there is minimal background noise as the mic is not directional.

 

Phone Integration

When listening to music you can pause/play the current track by briefly pressing the power button. If you recieve an incoming call whilst listening to music, the music will be interrupted by your ring tone and you can answer the call by briefly pressing the power button. The volume is adjusted by two up/down buttons. Unfortunately there is no way to skip tracks, on the H6020 the was a slider button that permitted this and I found it useful.

 

Reliability

It is too early to tell but I will post regular updates here until the headset fails. The H6020 lasted for a year with daily use so hopefully the S500 can improve on that.

 

A tale of three wireless headsets

I have three wireless headsets, two pairs made by Rapoo and a pair made by Logitech. This post is about the pros and cons of each.

Rapoo H6020

h6020blue

 

I bought this headset a year ago and it only cost £30 including delivery from HK, I can’t say enough good things about it.

  • Battery life is ~9 hrs, recharge takes ~1hr via mini usb cable.
  • Bluetooth pairs easily and reliably with iPhone & iPad
  • Audio quality when playing music is decent with a crisp clear sound
  • Comfort is good I wear the headset on my daily commute which takes 1 1/2 hrs without a problem.
  • Controls are easy to use and pausing and skip fwd/back is possible via the headset.
  • Taking phone calls is fine as long as you are in quiet surroundings as the mic is not directional so picks up background noise.
  • Range is about 30ft with a clear line of sight but you really need to be in the same room as walls block the signal.

Update: After just over a year of daily use and faultless service this headset appears to have died, I have just order these to replace them

Rapoo H8030

H8030

 

I recently bought this headset it also cost £30, early days but I am a little bit dissapointed.

  • Battery life ~10hrs claimed (not tested by me), recharge via micro usb ~1hr (not verified)
  • 2.4ghz wireless via supplied dongle (not bluetooth) Windows 7 recognized and installed drivers automatically.
  • Audio quality when playing music is okay, not as crisp as the H6020
  • Comfort is good, they are light and a good fit.
  • Controls are fiddly but less important as they are mostly controled via Pc
  • When used with skype other callers said they could hear me clearly.
  • Range is about 30ft with a clear line of sight but you really need to be in the same room as walls block the signal

 

Logitech F540

logitech-wireless-headset-f540

I have had this headset for over a year, it cost £95 and they currently retail on amazon for near that price. It is a big headset the ear cups completely cover your ears and passively block out background noise. I use these at work in a sometimes noisey office.

  • Battery life ~10hrs verified by moi, recharge takes ~2hrs via micro Usb
  • 2.4 ghz wireless via supplied base unit
  • Audio quality is excellent noticeably better than the Rapoo H6020
  • Range is good you can listen in the next room or up to 50 ft away with clear line of sight
  • Comfort is excellent I can wear them for several hours without discomfort.
  • Controls are good you can adjust volume of music and switch inputs on the headset
  •  When used for Skype calls other callers say they can hear me clearly and do not complain about background noise despite my noisy office.
  • The power supply for the base unit died on me so I purchase a generic replacement for £10.

Sony DR-BTN200M

WirelessHeadsetDR-BTN200M_1

Okay so those of you who’ve been paying attention will have noticed that this is the fourth pair of headphones in this post. As my Rapoo H6020 died I found these and decided to give them a go after watching a review on youtube. As I have only had them for 24 hrs it’s early days so these are just my first impressions. It seems like this headset works much better for phone calls that the H6020 did mainly down to the mic not picking up so much background noise. Their is also slightly better passive noise reduction and and the sound quality when listening to music also seem to be better with improved bass response. I’m not sure yet if these will be as comfortable  but so far so good.

 

 

 

 

Upgrading RavenDB from build #992 => #2700 for NServiceBus

Disclaimer: It is likely there is an easier way of doing this but I failed to find it.

NServiceBus Version 3.3.8

In Raven Studio Export each endpoints database.

open CMD prompt as Administrator and run:

 sc stop RavenDB
 sc delete RavenDB

 Rename C:Program FilesNServiceBus.Persistence  NServiceBus.Persistence.old
 md C:Program FilesNServiceBus.Persistence

Unzip contents of RavenDB zip into folder C:Program FilesNServiceBus.Persistence  (download from here )

In Raven Studio create database(s) for each database file you exported.

Navigate into each database and import the file you exported, take care to select the correct file Raven will let you import documents from a different database if you select the wrong file.

 

Sugru Hack – Timex Ironman Watch Strap Repair

I have had my Timex Ironman watch for just over a year, I never take it off it is very comfortable and useful for timing my runs.

TimexIronman

 

Unfortunately the resin straps on these watches tend to perish after about a year and due to the integrated design of the watch cannot be replaced. The watch was not expensive so I thought I would buy a new one, unfortunately I think this model is now discontinued.

Sugru to the rescue… Fortunately the strap had not completely failed, this made the repair easier to effect.

2013-09-10 10.34.40

I have a starter pack that has 2x blue 2x yellow 2x red and 1 black pack, this was my first attempt to fix something with Sugru so I decided to use up one of the blue packs.

SugruBlue

 

I took about a third of the sachet and moulded it around the damaged part of the strap.

2013-09-10 11.08.56

 

I left the repair to cure for about six hours then put the watch back on. It’s early days as to how long the repair will last, I will update this post periodically until the repair fails.

I had some sugru left over that was going to go to wasted so I fashioned a charger cable tidy withe remainder. I had previously been using blu tack but it was messy and a bit unsightly.

2013-09-11 09.46.33

 

If this proves to be a lasting repair then £2.50 worth of sugru will have saved me £40 to replace my watch!

Update:

Unfortunately yesterday this repair failed, I was compressing some boxes into our recycling bin and must have caught the strap.

The sugru part remained intact but the strap came free. I put the failure down to lack of adhesion between the resin strap material and the sugru.

If you read the sugru documentation it doesn’t claim great adhesion with plastics.

The repair *might* have lasted longer had I used a bit more sugru but it is hard to know for sure.

NServiceBus Unique attribute – A painful lesson

In NSB v 3.3.5 you can add a [Unique] attribute to a property of a saga’s state class. This ensures NServiceBus will only create one saga instance for each property value. This also increases performance when the saga is found using the property. In NSB v3.3.5 when you do this NServiceBus will create a SagaUniqueIdentity document when the saga is created that will ensure uniqueness across all instances of the saga in RavenDB.

Our issue was that we added the [Unique] attribute after having already deployed the saga code to production. We did not realise that in addition we needed to create a tool that would generate SagaUniqueIdentity docs for all existing saga instances. If you fail to generate the SagaUniqueIdentity docs, the next time a saga message is handled NServiceBus will not find the existing saga and will create a duplicate saga doc with blank state. Needless to say this is a bad thing.

In NSB v4.0+ if you introduce custom saga finding logic like the example below but you forget to add the [Unique] attribute to the saga state class you will get a warning (I think at compile time).

 

public override void ConfigureHowToFindSaga()
    {
        ConfigureMapping<Message2>(s => s.SomeID, m => m.SomeID);
    }

 

 

RavenDB Feature Wish List

Note that I’m using Build #992 so maybe these features exist in more recent versions.

  • Backup Compression. (Ability to compress a backup as it is being created.)
  • Gui Tool for Backup/Restoring Databases (Ideally inside Raven Studio).
    • Found this under the Tasks menu,  import/export/backup a db
  • Ability to Restore a Database without stopping the Raven Service.
    • you can’t restore online but you can import which may achieve the same.