Low latency services

Overview

Low latency services are designed to be as simple as possible.  All the same it is good to a have a picture of the the iteration between a low latency processing engine and the rest of the world.

Why are we doing all this?

Using the following model you can create a processing engine which is deterministic both in behaviour and performance, reproducible for testing, replication and restart of the application, and keeps a record of all actions for issue analysis. including micro-second timings.


High level

From a high level, a processing engine need inputs from gateway processes or thread or components which normalise incoming data or requests.  These requests are consumed by the processing engine and a log is produced.  From this log the gateway processes can respond to request or send outbound messages to key systems.

Also reading the processing engine's log are database persister needed to support reporting and ad hoc queries.  It is also used to send updates via a gateway to GUI clients.

Replication and restart

All the state of the system and everything it does is recorded in the logs.  This mean the engine can be restarted to the same state by re-reading the logs.  If the processing engine can read the logs fast enough, this restart time can be seconds.  For fail over, you need a copy of these logs.

Performance

The biggest challenge is in feeding the processing engine enough request/events to keep it busy.  The gateway processes have a typical delay of 20 to 200 micro-seconds and a throughput of 5,000 to 50,000 messages per second.  If you compare this to a typical processing engine this takes 1 to 10 micro-seconds and process 100,000 to 1,000,000 events per second, the gateways are pretty slow.  You can scale the number of processing engines, but it is likely you won't need to as you won't have enough events being passed through the gateways.

The main advantage of such a fast processing engine is stable performance even into the 99.99% of the time, and fast restarting (as the restart isn't slowed down by the gateways)

Java Chronicle

Java Chronicle is designed with these assumptions in mind.  It supports messaging between processes with a latency around 100 nano-seconds. It persists every message synchronously, meaning the message cannot be read until it has been written to the OS and if all processes die, the data will still be saved to disk.

Java Chronicle is lock free, supports ultra low GC foot print and busy waiting/thread affinity so you don't give up the CPU or context switch (except non-maskable interrupts)

In my next blog entry I will discuss a demo application written this way and how to measure it's performance.

Comments

Post a Comment

Popular posts from this blog

Java is Very Fast, If You Don’t Create Many Objects

Image
  You still have to watch how many objects you create. This article looks at a benchmark passing events over TCP/IP at 4 billion events per minute using the net.openhft.chronicle.wire.channel package in Chronicle Wire and why we still avoid object allocations..  One of the key optimisations is creating almost no garbage. Allocation is a very cheap operation and collection of very short-lived objects is also very cheap. Does this really make a difference? What difference does one small object per event (44 bytes) make to the performance in a throughput test where GC pauses are amortised? While allocation is as efficient as possible, it doesn’t avoid the memory pressure on the L1/L2 caches of your CPUs and when many cores are busy, they are contending for memory in the shared L3 cache.  Results Benchmark on a Ryzen 5950X with Ubuntu 22.10. JVM Vendor, Version No objects Throughput, Average Latency* One object per event Throughput, Average Latency* Azul Zulu 1.8.0_322 60.6 ...
Read more

System wide unique nanosecond timestamps

A Unique Identifier can be very useful for tracing. Those ids are even more useful when they contain a high-resolution timestamp.  Not only do they record the time of an event, but if unique can help trace events as they pass through the system. Such unique timestamps however can be expensive depending on how they are implemented.   This post explores a lightweight means of producing a unique, monotonically increasing system-wide nano-second resolution timestamp available in our open-source library. Uses for Unique Identifiers Unique identifiers can be useful to associate with a piece of information so that information can be referred to later unambiguously. This could be an event, a request, an order id, or a customer id. They can naturally be used as a primary key in a database or key/value store to retrieve that information later. One of the challenges of generating these identifiers is avoiding creating duplicates while not having an increasing cost.  You could ...
Read more

What does Chronicle Software do?

Image
Overview Chronicle Software is about simplifying fast data.  It is a suite of libraries to make it easier to write, monitor and tune data processing systems where performance and scalability are concerned. But its free, how do you make money, through support? We offer premium support . However, we often hear from users for the first time about a year after they have it in production.  Most users find they can support the software themselves.  It is only after a year or so, they have questions or concerns about where to take the product next.  In many cases, it is to a problem we have already solved and they just need to upgrade their software or use a new module we have added. As we don't know who is using most of our software, we can't advise them on how best to use our software and what updates or enhancements they would benefit from. We need users to contact us and ask questions.  We have a free forum , and we respond to 50% of questions in 2 hours...
Read more