Java chronicle library tutorial #1: Basic examples

Introduction

I dont remember how, but occasionally I ended up reading amazing blog of Peter Lawrey about different aspects of Java programming, focused, particularly on performance. Peter also author of Java Chronicle library which he describes as:

This library is an ultra low latency, high throughput, persisted, messaging and event driven in memory database.

Basically, it allows an application to write and read high amount of messages which will be persisted to disc by operating system. Chronicle library supposed to be much faster that simple write to file, since it is using memory mapped files with direct access, which gets persisted to disc by OS.

Since this library looks so cool, I decided to play with it. Unfortunately, there is no tutorials on internet about how to start, so I will share some pieces of code now and then, while I am exploring features of Chronicle

Step 1. Download Chronicle and connect it to your project

You can build chronicle library from source, which you can grub from github. If you are using maven, you can add it as maven dependency:

<dependency>
    <groupId>com.higherfrequencytrading<groupId>
    <artifactId>chronicle</artifactId>
    <version>1.6</version>
</dependency>

Step 2. Create a chronicle

There are two basic interfaces in library: Chronicle and Excerpt. Chronicle is “container” for Excerpts. To start using library first thing you need to do is create Chronicle. There are different types of Chronicles, I will describe them later. In this tutorial we will be using IndexedChronicle:

try {
    Chronicle chr = new IndexedChronicle("/tmp/chronicle");
} catch (IOException e) {
    e.printStackTrace();
}

If you run this code, you will see that library have created two files:

ayanami:tmp kenota$ ls -lah |grep chronicle
-rw-r--r--   1 kenota      wheel   128M 13 Apr 16:21 chronicle.data
-rw-r--r--   1 kenota      wheel    16M 13 Apr 16:21 chronicle.index</pre>

As you can see, files are quite big. That is because library preallocates a lot of data to provide good performance.

Step 3. Create Excerpt in Chronicle

Chronicle is only a “storage” which “holds” excerpts (under the hood it maintains index of each Excerpt in a memory-mapped file). To be able to actually write/read something, you need to create an Excerpt in particular Chronicle:

Excerpt is used both for reading and writing data. But before you can read or write to it, you need to either position it at some index in Chronicle ( to read) or, start new excerpt inside linked chronicle with some capacity. Lets go with writing first:

Step 4. Start new excerpt for writing

final Excerpt excerpt = chr.createExcerpt();
excerpt.startExcerpt(200);
excerpt.writeBytes("testString");
excerpt.finish();

After you call startExcerpt() you can use writeInt/writeLong/write methods from RandomDataOutput (defined in com.higherfrequencytrading.chronicle) interface. You need to remember about two points here:

You should know how much data you are going to write, since excerpt has capacity. You can have different capacity for each Excerpt.
You need to call Excerpt.finish(); method for data to be actually saved to Chronicle (and to disc later). This is kind of “commit” for transaction. If you write() to Excerpt but forgot to call finish(), data will be lost.

Step 5. Read data from Chronicle using Excerpt

Next step is to read data from Chronicle. Each excerpt has some index inside Chronicle. When you are reading from Chronicle, you can think of Excerpt as Cursor in terms of java collection (although not exactly classic cursor). Reading our test string form Step 4 will look like this:

final Excerpt excerpt = chr.createExcerpt();
excerpt.index(0);
String data = excerpt.readByteString();

Step 6. Iterating through all data in Chronicle

Ok, we were able to get one record from Chronicle, but how to we get all records from it? Assuming that all records in Chronicle has one string (as in step 4), you can iterate over all data like this:

final Excerpt excerpt = chr.createExcerpt();

while (excerpt.nextIndex()) {
    System.out.println("Read string from chronicle: " + excerpt.readByteString());
}

Now I think it is more obvious why you can think of Excerpt as some kind of cursor :)

Step 7. Putting it all together

Ok, time to put all steps together. Here is sample code which opens chronicle in temp directory, writes 3 strings and dumps it. You can run it several times and see that each times you run it, more strings are printed, because data from previous runs is also stored in chronicle.

package ChronicleTest;

import com.higherfrequencytrading.chronicle.Chronicle;
import com.higherfrequencytrading.chronicle.Excerpt;
import com.higherfrequencytrading.chronicle.impl.IndexedChronicle;

import java.io.File;
import java.io.IOException;

/**
 * Hello world!
 */
public class App {
    private static int STRING_SIZE_OVERHEAD = 4;

    public static void writeToChronicle(Chronicle chr, String someString) {
        final Excerpt excerpt = chr.createExcerpt();
        excerpt.startExcerpt(someString.length() + STRING_SIZE_OVERHEAD);

        excerpt.writeBytes(someString);
        excerpt.finish();
    }

    public static void dumpChronicle(Chronicle chr) {
        final Excerpt excerpt = chr.createExcerpt();

        while (excerpt.nextIndex()) {
            System.out.println("Read string from chronicle: " + excerpt.readByteString());
        }
    }

    public static void main(String[] args) {
        try {
            String tempPath = System.getProperty("java.io.tmpdir");
            String basePrefix = tempPath + File.separator + "chronicle";
            System.out.println("base prefix: " + basePrefix);
            Chronicle chr = new IndexedChronicle(basePrefix);

            writeToChronicle(chr, "Some text");
            writeToChronicle(chr, "more text");
            writeToChronicle(chr, "and a little bit more");

            dumpChronicle(chr);
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

Summary

Chronicle is a big piece of data where Excerpts are stored. Each Excerpt has index inside Chronicle. Excerpt is a pointer to some memory where you can write data to. Data should be read in the same order it was written, so if you write data to excerpt with two writes: writeLong() and then writeBytes(), it is yours responsibility to read it in the same order (once you positioned your Excerpt in Chronicle). And each Excerpt has a predefined capacity, but you can have different capacity for different Excerpts

I hope this post helped you and now you have basic example of how to read and write data from chronicle and what is looks like. I will continue to play with library and hopefully will write more tutorials. Stay tuned

←

Home

Binary Buffer