Greg Young has coded up a sample CQRS app with the aim of creating the simplest possible implementation.

I have been looking through the code to try and better understand CQRS.
Greg Young's Blog   
Download the Sample App Here

The diagram above is based on stepping through the code after the user submits a new Inventory Item.

I'm still not sure I fully understand things but it is starting to become clearer.

This article on StorageSearch.com shows an interesting trend and prediction in the pricing change between DRAM and NAND Flash, as you can see from the chart both are dropping but NAND is dropping faster.


What acutally happened was that NAND prices rose, likely due to demand from companies like Apple, below the first chart shows the price of 8GB MLC NAND Flash from 2007 to today. These modules are commonly used in smaller capacity 2.5" solid state disk drives. Dram prices also rose as depicted in the second chart below.

MLC Flash 8Gb Module Pricing 2007 - Today

Dram Pricing DDR2 1Gb Module 2007 - Today

There is currently $$$ being spent in building new manufacturing facilities to produce NAND flash so unless new sources of demand appear prices will start to fall again soon. So what does this mean if you are currently designing an enterprise solution that will be deployed to production in the next 12 months and be used for up to the the next 10 years? Well it depends on your application but if it makes significant use of a relational database management system or RDBMS then you can expect a huge decrease in the cost of delivering IOPS.

Calculating IOPS cost

Today you can purchase a FusionIO card for ~$7500 that offers 320GB of storage and a R/W 100k/140k IOPS, a single 15k 450GB SAS drive costs around $300 and will deliver ~175 IOPS to match the mixed  read/write performance of the card you would need to purchase 300 drives, enclosures and controllers, a rough calculation:
15k Disks - 300 x $300 = $90,000
Enclosures - 10 x $5000 = $50,000
Total = $140,000

RAID IOPS Calculator
The problem with the FusionIO solution is that it does not scale easily each card must be installed into a Pci-E slot in a server, most 1U servers will only have one slot. It is also directly attached to the server, when using a SAN storage can be shared amongst servers. If on top of this you are taking advantage of virtualisation it is possible to move VM's from one physical host to another in realtime and your server will remain online and suffer a small reduction in performance for a few minutes. This makes it possible to take physical machines offline for maintenance without affective the availablity of your application. FusionIO does not offer fault tolerance if the physical machine that the card is installed in fails then you lose access to that storage until the machine becomes available again. In this situation you must have a mirror machine available with a recent copy of your data. But this is still a good value solution at $15,000 vs $140,000.

The water is a little bit murky

It is still not clear exactly how vendors are going to deliver the level of IOPS performance offerd by FusionIO in a SAN based format. The problem seems to center around the speed of interfaces like iSCSI and the bandwidth and latency of gigabit ethernet. Although the storage devices already exist there is a lot of work needed to improve the pipeline between hosts and SAN. That said this is a solvable problem, it's not "if" but "when".

The impact on database performance

RDBMS's like Sql Server, MySql and Oracle are going to get a huge performance boost as NAND Flash based storage systems come into use, typically IO has been the bottleneck in any database based application in future this will likely change to CPU. Oracle and Sql Server are both licensed per CPU/Core so when building applications this should be taken into account. Any operation that requires a significant amount of CPU time should not be perfromed by the database unless absolutely necessary. Databases should just be used for what they were originaly built and that is to persist data and allow it to be retrieved. The need for inline caches should be reduced due to this performance boost.

I have seen a few posts on the NServiceBus yahoo groups mailing list and a few tweets asking about how best to monitor NServiceBus. This is a problem I have been researching over the last few weeks, here is ths solution I came up with for triggering alerts when too many messages are in a queue.

Performance Counters

MSMQ Exposes several performance counters out of the box, these can be accessed via WMI. There are a few caveats, only active queues or queues with messages will have performance counters see here for details.

Triggering Alerts

You could write your own WMI scripts to monitor these performance counters and trigger alerts, another option is to use a monitoring tool that has the ability to setup WMI monitors. OpManager is such a tool, in a couple of minutes you can configure a new monitor that will trigger emails or SMS's to be sent if a certain threshold is reached.

Improving NServiceBus

I don't know enough about the innner workings of NServiceBus to make an educates assement of how we could create an NServiceBus monitoring tool but I'll try anyway.

Based on these facts:

• Every NServceBus Service knows about the Queues it is configured to use for sending / recieving messages.
• It should be simple to make an NServiceBus service discoverable.

Would it not be possible to create a generic NServiceBus.Monitor.exe with a provider model that would allow plugins to be written for any monitoring tool / protocol?

I'm digging into SNMP (Simple Network Management Protocol) to see what it offers in terms of application / server monitoring.

The specific scenario I have in mind is the monitoring of queues in MSMQ for our NServiceBus windows services. We need email alerts to be triggered if a message is moved to the error queue or if the number of items in a queue exceeds a certain threshold.

I'm working on a Windows 7 x64 laptop so the first thing I had to do was install the SNMP service (Under CP > Programs and Features).

After that was installed I had to restart the services applet and right click > Properties > Security on the SNMP Service and create a "public" community.

After this I had to download a command line tool Net-SNMP for issuing SNMP queries to test all was well.
I opted for the basic installation and uncheck all the other options.
Open a new command prompt so that it picks up the PATH variable added during installation then run the following command

snmptranslate -IR -Td IF-MIB::linkDown

See the README.txt for the correct response but if you see no errors you are in good shape.

Next download SnmpTools Follow the instructions. I opted to install the 32bit version using the regagentWow6432.reg
Restart the Microsoft SNMP Service then run the command:
snmpget -v1 -c public localhost 1.3.6.1.4.1.15
If all is well you should see:
SNMPv2-SMI::enterprises.15 = STRING: "snmptools by erwan.l@free.fr"

That's all for now but I will update this post again.

As Promised Part 2

I have made a little more progress accessing the MSMQ Performance counters

I have added the following two lines to the Counters.ini file and restarted the SNMP Service:

[1.3.6.1.4.1.15.5]

counter=MSMQ Queue\Messages in Queue\acerbotic\private$\distributordatabus

Now when I issue the following snmpget query from the command line:

snmpget -v1 -c public localhost 1.3.6.1.4.1.15.5

I see the response:

SNMPv2-SMI::enterprises.15.5 = INTEGER: 2

Thus indicating there are two messages in the private$\distributordatabus queue on my laptop.

The End Game

At time of writing it seems that in order to allow our SNMP monitoring software to access this OID I need to supply it with a MIB file.

The story continues...

I have found this page that talks about using a MIB editor to create such a file.

Conclusion

SNMP is a great way to monitor your application, but it involves a good deal of upfront investment if you don't already have a developer on your team who is proficient. We don't have such a developer so have decided to use the .Net framework to expose peformance counters and use WMI to monitor those counters. If the thresholds we set are exceeded we will get alerts.

Yesterday I attended my first UK SQL Server User Group meeting.
Event web page

Topics

- SQL Server I/O - Tony Rogerson
- Understanding how Sql performs Joins with NestedLoops, Merge & Hash - Ami Levin
- DbSophic QURE - Workload Tuning - Ami Levin
- SQL Streaminsight - Allan Mitchell - twitter: @allanSQLIS

I was most interested in the first topic but missed the talk as I did not arrive till 18:30 and the evening started @ 17:30. I don't finish work till 18:00 and had to get across town. Of the three talks that I caught the content and delivery was at a very high standard.

Understanding how Sql Server performs Joins with NestedLoops, Merge & Hash - Ami Levin

If you write a lot of SQL queries or are often called upon to tune SQL Server then you should learn how/why the query optimizer selects from three physical operators to perform joins and how sometime it can be caught out. This talk was given by Ami Levin who is a SQL MVP and works for DBSophic. It was really interesting and the first time I have had any understanding of what these terms that appear in the execution plan mean. One really interesting thing that Ami pointed out is that sometimes the method Sql uses to estimate the number of rows and actual number of rows will differ by a large amount, this can result in a sub optimal operator being selected to perform the join. If when tuning you spot this sort of discrepancy further investigation is warranted. The following is a scenario that Ami gave when this can occur:

SELECT *
FROM SalesLT.SalesOrderHeader SOH
INNER JOIN SalesLT.SalesOrderDetail SOD ON SOH.SalesOrderID = SOD.SalesOrderID
INNER JOIN SalesLT.Product P ON P.ProductID = SOD.ProductID
WHERE SOH.TotalDue > 1500
AND SOD.UnitPrice > 300
AND P.StandardCost > 300

So in the SQL above the optimizer will use statistics to estimate roughly how many rows it will return by looking at each table in turn and estimating how many rows each where clause will return. In this case say that each Where clause will return ~1% of the rows in a table containing 100k rows so the optimizer estimates the number of rows to be 1% of 100,000 = 1000 rows then, 1% of 1000 = 10, then 1% of 10 < 1. So the optimizer thinks 1 row or less will be in the results set. Unfortunately what the optimizer cannot know is that the where clauses are releated to the data, in this case ORDERS that cost more and have more expensive unit cost and a higher so the actual number of rows returned will be nearer to 1000, hence why sometimes the optimizer can choose a sub optimal physical operator to perform a join.

If you think the wrong operator is being chosen you can use the OPTION keyword this article discusses in more detail:

First let me say I love Spotify and use it everyday at work, because I love it I feel compelled to discuss the company's failure to captialize on their initial brilliant idea to use P2P networks as a way for people to choose the music they want to listen with either the advert subsidised or premium service.

What they got right was the business model and the user interface design. The software is easy to use, the obvious feature that is missing is improved playlist management. What I cannot get my head around is how long it has taken for new functionality to be added to the client to illustrate my point I trawled the Spotify blog for the details of releases during 2008 the list is below. Bear in mind development started in September 06' and the first beta was released in May 07' so by 2008 the software should have been pretty stable.

New features added in 2008 for Windows Client

• User settings can be changed.
• We mark all collaborative playlist (including your own) with a padlock icon.
• Context menus have been added for the artist title on artist and album pages. (0.2.12)
• a new start page with top lists has been added (0.2.12)
• Spotify now pauses when waking up from hibernation (0.2.13)
• A new view has been added to visualize the play queue. (0.2.14)
• Growl support has been added. (0.2.14)
• Dropping a link to a playlist, artist or an album on the play queue will cause Spotify to play future tracks from that source. (0.2.15)
• History now shows the currently playing track as well. (0.2.15)
• New streaming protocol (0.3.0)
• Proxy Server Support (0.3.3)
• Improved Buffering Algorithm (0.3.3)
• Undo for playlist operations has been added (0.3.8)

So in the course of a year there are no notable features added, there was a ton of bug fixes and load of work done to polish the UI. The big question is why, were the team too busy firefighting or doing lots of work to the infrastrcuture or were they hampered by their choice of language and tooling whatever the reason it is shocking.

Spotify's main failures

• Not Entered the US Market / gone global
  
• Not solved the scalability issues
• Still missing some important labels / artists (ACDC / Oasis / Beatles etc.)
• Not delivered any meaningful features to the Windows client for 3 years (with the exception of collaborative playlists)

As things stand they have left a clear path for competitors to come into the marktet and take the lions share. The barriers to entry are deals with the record lables and the p2p technology.
If you want to send feedback to spotify email feedback@spotify.com