Low Latency Slides

Last weekend was LJC Open Conference #4, and like many people I got a lot out of it.

My talk was up first which meant I could relax for the rest of the day.

Here are the slides

Note: the message size was 16 longs or 128 bytes. This makes a difference at higher throughputs.

In answer to @aix's questions

On slide 12; what are we measuring here, elapsed time from the message hitting the buffer to the "echo" reply showing up on the client socket?
In each message I add a timestamp as it is written. When each message is read, I compare the timestamp with the current time. I sort the results and take the middle / 50%tile as typical timings and the worst 0.01% as the 99.99%tile.
What's causing the large difference between "nanoTime(), Normal" and "RDTSC, Normal" in the bottom half of the slide (2M/s)?
The reason for taking the RDTSC directly (9 ns) is that System.nanoTime() is quite slow on Centos 5.7 (180 ns) and the later is a system calls which may disturb the cache. At modest message rate 200K/s (one message every 5000 ns) the difference is minor however as the message rates increase 2M/s (one message every 500 ns) the addition to latency is significant. Its not possible to send messages over 5 M/s if I was using System.nanoTime() where as with RDTSC I got up to 12 M/s. Without timing each message, I got a throughput of 17 M/s. In answer to @Matt's questions
Is that a safe way to call tsc on a multicore machine? Is it consistent across cores?
Calling RDTSC on a multi-core system is safe as there is one counter per socket. However, calling TSC on a multi-socket system is not safe and there will be difference and possibly a drift between sockets.

This is not a problem if you have only one socket, or you do all you timing on one socket. e.g. You have used thread affinity and know the timings will all be on the same socket.

Don't you need some sort of serialisation in the instruction stream to avoid out of order behaviour?
This is a potential problem. However, the total time taken with JNI is around 9 ns which is more than 40 instructions. This is longer than the CPU pipeline can re-order. (typically around 32 instructions) If this instruction were embedded the way the some Unsafe methods are, it could be re-ordered with Java instruction. However, provided this re-ordering is not random, the difference is likely to be a bias of << 10 ns. If the re-ordering was random this could add up to 10 ns jitter.

Given I am timing latencies to an accuracy of 100 ns, I decided it wasn't a problem for me. It could be a problem if you want 10 ns accuracy.

On the "reproducible results" theme, to be completely anal you need to repeat the test n times (20-30?) and examine the distribution of results.
What I do is repeatedly run the test for 5 seconds and print the results. This will consist of a minimum of one million individual messages. This is repeated 30 times and I print an aggregate distribution. I compare the individual distributions with the aggregate to see if they are "close".

I will try to document how the test are done in more detail when I am happy with the reproducibility of the 99.99%tile values. :|

Comments

  1. Anonymous28 November 2011 at 19:13

    Peter,

    Thanks for sharing. I have a couple of questions about slide 12:

    1. What are we measuring here, elapsed time from the message hitting the buffer to the "echo" reply showing up on the client socket?

    2. What's causing the large difference between "nanoTime(), Normal" and "RDTSC, Normal" in the bottom half of the slide (2M/s)?

    P.S. I wish I knew about the conference so I could attend.

    -aix

    ReplyDelete
  2. Peter Lawrey28 November 2011 at 19:23

    @aix, I have added your excellent questions with answers to the main entry.

    I missed devoxx for the same reason and its one of the biggest Java conferences in Europe. :|

    ReplyDelete
  3. Matt28 November 2011 at 21:19

    Is that a safe way to call tsc on a multicore machine? is it consistent across cores? don't you need some sort of serialisation in the instruction stream to avoid out of order behaviour?

    On the "reproducible results" theme, to be completely anal you need to repeat the test n times (20-30?) and examine the distribution of results.

    ReplyDelete
  4. Anonymous29 November 2011 at 10:59

    For comparison, I've timed System.nanoTime() on a couple of systems I had to hand, and the results are quite interesting:

    1. 64-bit Ubuntu 10.04 running on Xeon E5410: 50ns per call.

    2. 64-bit RHEL 5.6 running on Xeon L5520: 350ns per call.

    As to RDTSC, I presume it can't be used to measure intervals across CPU chips? What about different cores on the same chip?

    ReplyDelete
  5. Peter Lawrey29 November 2011 at 11:03

    @aix, My understanding is there is only one TSC per socket/chip and I haven't seen a problem comparing times between cores on the same socket.

    For testing purposes, when having more than one socket, I would dedicate a whole socket (not the socket with cpu0 as it gets the OS by default) to the core application and only use the TSC in threads on that socket.

    ReplyDelete
  6. Anonymous29 November 2011 at 11:23

    Do you know of a good open-source Java wrapper for RDTSC/RDTSCP? Something that would have low overhead yet take care of the subtleties (out-of-order execution etc)?

    ReplyDelete
  7. Peter Lawrey29 November 2011 at 13:07

    @aix, In that case I would use System.nanoTime(). If you are making a few 100K/s calls it shouldn't make much difference. Also not using Centos/RHEL 5 appears to give a significant improvement.

    ReplyDelete
  8. --not this one--2 December 2011 at 09:37

    Hello,

    I am no specialist on the question so I have a few questions:
    - RDTSC calls are made using JNI, right ? If so, I thought the overhead was much more than the 9 ns (just for the JNI call) - do you have any recent value on the latency overhead when calling JNI ?
    - are there any decent benchmark on the performance of System.nanoTime over various OSes / versions ?

    ReplyDelete
  9. Peter Lawrey2 December 2011 at 09:48

    @HareRama, I tend to use JNI as little as possible, so I am no expert on what makes JNI slow. However my understanding is that the number and type of parameters makes a difference.

    In the case of RDTSC, it takes no arguments and returns a primitive which may explain its speed.

    Another reason JNI calls can be slow is that they often use a system call. The cost of a simple system call can be quite high (System.nanoTime() is basic a wrapper for a system call) RDTSC is a single machine code instruction and there are no many useful things you can do purely in machine code that you can't do in Java already (without a system call)

    It would be interesting to get a comparison of how long System.nanoTime() takes on different OSes (with the same hardware) and its accuracy and resolution. Unfortunately the time it takes to install just about every mainstream OS on one box and test it is more than most people would consider. ;)

    ReplyDelete
  10. Michael Barker8 December 2011 at 06:49

    Hi Peter,

    The library that you use for setting the thread affinity, is that home grown or is there an open source version floating around?

    Mike

    ReplyDelete
  11. Peter Lawrey8 December 2011 at 08:39

    @Michael Its home grown. The tricky (non Java) code is in the slides. I could write open source version if people are interested. ;)

    ReplyDelete
  12. nmcy14 December 2012 at 01:53

    Hi, I can't see the slides.
    How can I get them?

    ReplyDelete

Post a Comment

Popular posts from this blog

Java is Very Fast, If You Don’t Create Many Objects

Image
  You still have to watch how many objects you create. This article looks at a benchmark passing events over TCP/IP at 4 billion events per minute using the net.openhft.chronicle.wire.channel package in Chronicle Wire and why we still avoid object allocations..  One of the key optimisations is creating almost no garbage. Allocation is a very cheap operation and collection of very short-lived objects is also very cheap. Does this really make a difference? What difference does one small object per event (44 bytes) make to the performance in a throughput test where GC pauses are amortised? While allocation is as efficient as possible, it doesn’t avoid the memory pressure on the L1/L2 caches of your CPUs and when many cores are busy, they are contending for memory in the shared L3 cache.  Results Benchmark on a Ryzen 5950X with Ubuntu 22.10. JVM Vendor, Version No objects Throughput, Average Latency* One object per event Throughput, Average Latency* Azul Zulu 1.8.0_322 60.6 ...
Read more

System wide unique nanosecond timestamps

A Unique Identifier can be very useful for tracing. Those ids are even more useful when they contain a high-resolution timestamp.  Not only do they record the time of an event, but if unique can help trace events as they pass through the system. Such unique timestamps however can be expensive depending on how they are implemented.   This post explores a lightweight means of producing a unique, monotonically increasing system-wide nano-second resolution timestamp available in our open-source library. Uses for Unique Identifiers Unique identifiers can be useful to associate with a piece of information so that information can be referred to later unambiguously. This could be an event, a request, an order id, or a customer id. They can naturally be used as a primary key in a database or key/value store to retrieve that information later. One of the challenges of generating these identifiers is avoiding creating duplicates while not having an increasing cost.  You could ...
Read more

What does Chronicle Software do?

Image
Overview Chronicle Software is about simplifying fast data.  It is a suite of libraries to make it easier to write, monitor and tune data processing systems where performance and scalability are concerned. But its free, how do you make money, through support? We offer premium support . However, we often hear from users for the first time about a year after they have it in production.  Most users find they can support the software themselves.  It is only after a year or so, they have questions or concerns about where to take the product next.  In many cases, it is to a problem we have already solved and they just need to upgrade their software or use a new module we have added. As we don't know who is using most of our software, we can't advise them on how best to use our software and what updates or enhancements they would benefit from. We need users to contact us and ask questions.  We have a free forum , and we respond to 50% of questions in 2 hours...
Read more