A down side of durable messaging

Overview

Durable messaging can be very fast, as fast as non-durable messaging up to a point.

Limitations of durable messaging

Durable messaging is dependant on the size of your main memory and the speed of your hard drive.  If you have a HDD, this can be as low as 20 MB/s and as high as 60 MB/s.  A RAID set of HDD can support between 100 and 300 MB/s. An SATA SSD can support between 100 and 500 MB/s and a PCI SSD can support up to 1.5 GB/s.

Case study

Say you have 8 GB of memory, writing two million 100 bytes messages per second and a HDD which support 25 MB/s.

This works fine at the speed in bursts but you reach a point where your disk cache is full.  Depending on your OS this can be between 20% and 80% of your main memory size.  In my experience, Windows tends to be closer to 20% even if you have plenty of free memory whereas Linux tends to allow in the region of 30% of your memory in uncommitted writes.

Say you are writing two million 100 byte message per second or 200 MB/s and you have 1600 MB of disk cache. The difference in speed is 175 MB/s between the rate you are writing and the rate you are generating it so in just 9 seconds you have filled the cache.  At this point your performance plummets to the write speed of your disk which is 25 MB/second. With each messaging being 100 bytes, you are now writing 250,000 messages per second or 8x slower.

What is the solution?

  • Keep your micro-bursts to less than you can fit in disk cache e.g. in the above case this would be about 18 million messages.
  • Increase the amount of memory you have.  While memory is cheap and you can buy 32 GB for about £150, all this does in include the duration of the micro-burst you can support.
  • Increase the speed of your drive.  With SSD you can support much higher bandwidths.  SATA SSD drives support up to 500 MB/sec which is higher than Chronicle can typically serialize messages, i.e. more than enough.  The downside of this is it reduces the total number of messages you can write. A 500 MB SSD can store 5 billion 100 byte messages. A 6x4 TB RAID-5 set can support a transfer rate of over 200 MB/s which would be enough for the above case study, and can store 200 billion messages.

Conclusion

If you see any durable messaging solution suddenly slow down under load, you need to look at the size of your buffers and the throughput of your disk sub-system.

Comments

  1. Yeroc6 February 2013 at 19:42

    I'm confused. How can messages be durable if they're only in I/O cache in main memory?!? I'm certain any product advertising durable messaging queues will be flushing all writes to disk so any main memory disk cache should be irrelevant. The only disk cache size that may be relevant would be battery-backed cache on the disk controller itself.

    Corey

    ReplyDelete
  2. Peter Lawrey6 February 2013 at 20:49

    You are right that the buffer is question depends on the implementation. In the case of chronicle, if you only need reliability in case of your program dying you can use it as is. If you need reliability in case of a machine dying or a power failure, it can be replicated to a second machine. You could commit every write to battery backed up disk, which is also supported if you really need it but the impact to performance is dramatic for very little gain in most cases. In fact I should do a page on how much difference flushing every message to disk makes.

    As I have noted before most messaging solutions avoid benchmarking durable messaging, but those that do clearly don't commit writes given the numbers they report. ;)

    ReplyDelete
    Replies
    1. Yeroc7 February 2013 at 19:20

      Ok, you seem to use the term durable in a rather loose sense. Perhaps you could edit your blog post to define what you mean by durable at the top of your post. I believe most people think of durable as equating to data being safely written to non-volatile storage. To me, calling something durable when it exists in I/O cache in main memory is not durable. So what you're benchmarking here is not really durable messaging either. As JP noted, durable messaging is pretty similar to transaction logging for a database and comes with the same performance penalties.

      Delete
  3. JP7 February 2013 at 03:55

    Peter you are correct to doubt those sorts of numbers. The durable write path for a messaging system isn't terribly different from the transaction log for a database. A single disk can only support 1 commit/flush per revolution. The first few generations of SATA SSDs weren't much better, though they are improving. By the way, RAID-5 is fairly terrible at that particular kind of write load, due to the parity calculation overhead. I suspect you'd be better off with either RAID 1 or RAID 0+1. As always, best to test it out, and durability tests may well deserve checking how the system does in a hard power fault.

    ReplyDelete

Post a Comment

Popular posts from this blog

Java is Very Fast, If You Don’t Create Many Objects

Image
  You still have to watch how many objects you create. This article looks at a benchmark passing events over TCP/IP at 4 billion events per minute using the net.openhft.chronicle.wire.channel package in Chronicle Wire and why we still avoid object allocations..  One of the key optimisations is creating almost no garbage. Allocation is a very cheap operation and collection of very short-lived objects is also very cheap. Does this really make a difference? What difference does one small object per event (44 bytes) make to the performance in a throughput test where GC pauses are amortised? While allocation is as efficient as possible, it doesn’t avoid the memory pressure on the L1/L2 caches of your CPUs and when many cores are busy, they are contending for memory in the shared L3 cache.  Results Benchmark on a Ryzen 5950X with Ubuntu 22.10. JVM Vendor, Version No objects Throughput, Average Latency* One object per event Throughput, Average Latency* Azul Zulu 1.8.0_322 60.6 ...
Read more

System wide unique nanosecond timestamps

A Unique Identifier can be very useful for tracing. Those ids are even more useful when they contain a high-resolution timestamp.  Not only do they record the time of an event, but if unique can help trace events as they pass through the system. Such unique timestamps however can be expensive depending on how they are implemented.   This post explores a lightweight means of producing a unique, monotonically increasing system-wide nano-second resolution timestamp available in our open-source library. Uses for Unique Identifiers Unique identifiers can be useful to associate with a piece of information so that information can be referred to later unambiguously. This could be an event, a request, an order id, or a customer id. They can naturally be used as a primary key in a database or key/value store to retrieve that information later. One of the challenges of generating these identifiers is avoiding creating duplicates while not having an increasing cost.  You could ...
Read more

What does Chronicle Software do?

Image
Overview Chronicle Software is about simplifying fast data.  It is a suite of libraries to make it easier to write, monitor and tune data processing systems where performance and scalability are concerned. But its free, how do you make money, through support? We offer premium support . However, we often hear from users for the first time about a year after they have it in production.  Most users find they can support the software themselves.  It is only after a year or so, they have questions or concerns about where to take the product next.  In many cases, it is to a problem we have already solved and they just need to upgrade their software or use a new module we have added. As we don't know who is using most of our software, we can't advise them on how best to use our software and what updates or enhancements they would benefit from. We need users to contact us and ask questions.  We have a free forum , and we respond to 50% of questions in 2 hours...
Read more