Wiring YAML to a file or network with Chronicle Wire.

August 16, 2015

Overview

Chronicle Wire is designed to get the benefits of a text protocol in terms of ease of development and faster debugging, but be able to switch to a more performant and smaller binary protocol without having to change your code.

In this post, I looked at the performance you might achieve using a YAML based text format for serialization, deserialization. One of the nice features is that you can use YAML for testing, debugging but switch to Binary YAML where it makes sense.

It is even possible to mix and match in a single stream. e.g. use YAML for the handshake and switch to Binary YAML or even some other format when you determine both ends are compatible.

What does the code look like?

First you need to have a buffer to write to. This can be a byte[], a ByteBuffer, off heap memory, or even an address and length you have obtained from some other library.

// Bytes which wraps a ByteBuffer which is resized as needed.
Bytes<ByteBuffer> bytes = Bytes.elasticByteBuffer();

Now you can choice which format you are using. As the wire formats are themselves unbuffered, you can use them with the same buffer, but in general using one wire format is easier.

Wire wire = new TextWire(bytes);
// or
Bytes<ByteBuffer> bytes2 = Bytes.elasticByteBuffer();
Wire wire2 = new BinaryWire(bytes2);
// or
Bytes<ByteBuffer> bytes3 = Bytes.elasticByteBuffer();
Wire wire3 = new RawWire(bytes3);

There is lots of options such as whether you want numeric fields like protobuf or SBE, or you want to use variable (smallest) or fixed length (fastest) data values.

To write out an object you can make it Marshallable and define a readmarshallable and writeMarshallable, but for now let just write some data.

wire.write(() -> "message").text("Hello World")
      .write(() -> "number").int64(1234567890L)
       .write(() -> "code").asEnum(TimeUnit.SECONDS)
      .write(() -> "price").float64(10.50);
System.out.println(bytes);

// to obtain the underlying ByteBuffer to write to a Channel
ByteBuffer byteBuffer = bytes2.underlyingObject();
byteBuffer.position(0);
byteBuffer.limit(bytes2.length());

With TextWire this prints

message: Hello World
number: 1234567890
code: SECONDS
price: 10.5

However use BinaryWire instead and it writes (when printed in hex)

00000000 C7 6D 65 73 73 61 67 65  EB 48 65 6C 6C 6F 20 57 ·message ·Hello W
00000010 6F 72 6C 64 C6 6E 75 6D  62 65 72 A3 D2 02 96 49 orld·num ber····I
00000020 C4 63 6F 64 65 E7 53 45  43 4F 4E 44 53 C5 70 72 ·code·SE CONDS·pr
00000030 69 63 65 90 00 00 28 41                          ice···(A

Using BinaryWire can be half the size and twice the speed, however we have the option of using RawWire which drops all the meta data and can be 8x faster than plan text.

For more Examples and Documentation see the main README for Chronicle Wire.
Additional Examples of using Chronicle Wire.

Conclusion

While binary protocols can be faster, they can be much harder to work with. If you can use a text format you can develop and debug much faster, but if you have the option to switch dynamically you can choose performance when you need it.

Even though TextWire takes less than 5 micro-seconds 99.99% of the time to write are read a 6 field object, this might be fast enough for many applications.

What is useful, is having the option to go faster if you need to. RawWire was under 600 nano-seconds 99.99% of the time without the need for a code change.

If maximum speed is required, we have a BytesMarshallable interface with a stripped down API which less than 220 nano-seconds 99.99% of the time.

Vanilla Java