Showing posts from October, 2014

Chronicle Map and Yahoo Cloud Service Benchmark

Overview Yahoo Cloud Service Benchmark is a reasonably widely used benchmarking tool for testing key value stores for a significant number of key e.g 100 million, and a modest number of clients i.e. served from one machine. In this article I look at how a test of 100 million * 1 KB key/values performed using Chronicle Map on a single machine with 128 GB memory, dual Intel E5-2650 v2 @ 2.60GHz, and six Samsung 840 EVO SSDs. The 1 KB value consists of ten fields of 100 byte Strings.  For a more optimal solution, primitive numbers would be a better choice. While the SSDs helped, the peak transfer rate was 700 MB/s which could be supported by two SATA SSD drives. These benchmarks were performed using the latest version at the time of the report, Chronicle Map 2.0.6a-SNAPSHOT. Micro-second world. Something which confounds me when reading benchmarks about key-value stores is that they start with the premise that performance is really important.  IMHO, about 90% of the time, pe

Essentialism and Technology

Overview Essentialism isn't just flawed, it is flawed at many levels, some of which should be familiar to technologists.  I have read a number of arguments against essentialism, however for me they miss a number of points which relate to technology, an area I claim to know more about.  In technology you can measurably demonstrate the performance difference between two solutions, and yet it is difficult to convince people to let go of their biases. What is Essentialism (from Google) "a belief that things have a set of characteristics that make them what they are, and that the task of science and philosophy is their discovery and expression; the doctrine that essence is prior to existence. the view that all children should be taught on traditional lines the ideas and methods regarded as essential to the prevalent culture. the view that categories of people, such as women and men, or heterosexuals and homosexuals, or members of ethnic groups, have intrinsically different

A Java conversion puzzler, not suitable for work (or interviews)

A really hard interview question would be something like this int i = Integer.MAX_VALUE; i += 0.0f; int j = i; System.out.println(j == Integer.MAX_VALUE); // true Why does this print true? At first glace, the answer seem obvious, until you realise that if you change  int i for long i things get weird long i = Integer.MAX_VALUE; i += 0.0f; int j = (int) i; System.out.println(j == Integer.MAX_VALUE); // false System.out.println(j == Integer.MIN_VALUE); // true What is going on you might wonder? When did Java become JavaScript? Let me start by explaining why long gives a such a strange result. An important detail about += is that it does an implicit cast.  You might think that a += b; is the same as a = a + b; and basically it is except with a subtle difference which most of the time doesn't matter; a = (typeOf(a)) (a + b); Another subtle feature of addition is the the result is the "wider" of the two types.  This means that i += 0.0f;

Kafka Benchmark on Chronicle Queue

Overview I was recently asked to compare the performance of Kafka with Chronicle Queue.  No two products are exactly alike, and performing a fair comparison is not easy.  We can try to run similar tests and see what results we get. This test is based on Apache Kafka Performance Results What was the test used? One area Kafka tests is multi-threaded performance.  In tests we have done, it is neither better or worse to use more more threads (up to the number CPUs you have).  We didn't benchmark this here. All tests use one producer. Another difference, is that we flush to disk periodically by time rather than by count.  Being able to say you are never behind by more than X milli-seconds is often more useful than say 600 messages, as you don't know how long those messages could have been waiting there.  For our tests, we look at flush periods of between 1 ms and 10 ms.  In Kafka's tests, they appears to be every 3 ms approximately. The message size used was 200