Posted by & filed under java.

Introduction

I dont remember how, but occasionally I ended up reading amazing blog of Peter Lawrey about different aspects of Java programming, focused, particularly on performance. Peter also author of Java Chronicle library which he describes as:

This library is an ultra low latency, high throughput, persisted, messaging and event driven in memory database.

Basically, it allows an application to write and read high amount of messages which will be persisted to disc by operating system. Chronicle library supposed to be much faster that simple write to file, since it is using memory mapped files with direct access, which gets persisted to disc by OS.

Since this library looks so cool, I decided to play with it. Unfortunately, there is no tutorials on internet about how to start, so I will share some pieces of code now and then, while I am exploring features of Chronicle

Step 1. Download Chronicle and connect it to your project

You can build chronicle library from source, which you can grub from github. If you are using maven, you can add it as maven dependency:

Step 2. Create a chronicle

There are two basic interfaces in library: Chronicle and Excerpt. Chronicle is “container” for Excerpts. To start using library first thing you need to do is create  Chronicle. There are different types of Chronicles, I will describe them later. In this tutorial we will be using IndexedChronicle:

If you run this code, you will see that library have created two files:

As you can see, files are quite big. That is because library preallocates a lot of data to provide good performance.

Step 3. Create Excerpt in Chronicle

Chronicle is only a “storage” which “holds” excerpts (under the hood it maintains index of each Excerpt in a memory-mapped file). To be able to actually write/read something, you need to create an Excerpt in particular Chronicle:

Excerpt is used both for reading and writing data. But before you can read or write to it, you need to either position it at some index in Chronicle (to read) or, start new excerpt inside linked chronicle with some capacity. Lets go with writing first:

Step 4. Start new excerpt for writing

After you call startExcerpt() you can use writeInt/writeLong/write* methods from RandomDataOutput (defined in com.higherfrequencytrading.chronicle) interface. You need to remember about two points here:

  1. You should know how much data you are going to write, since excerpt has capacity. You can have different capacity for each Excerpt.
  2. You need to call Excerpt.finish(); method for data to be actually saved to Chronicle (and to disc later). This is kind of “commit” for transaction. If you write*() to Excerpt but forgot to call finish(), data will be lost.

Step 5. Read data from Chronicle using Excerpt

Next step is to read data from Chronicle. Each excerpt has some index inside Chronicle. When you are reading from Chronicle, you can sit of Excerpt as Cursor in terms of java collection (although not exactly classic cursor). Reading our test string form Step 4 will look like this:

Step 6. Iterating through all data in Chronicle

Ok, we were able to get one record from Chronicle, but how to we get all records from it? Assuming that all records in Chronicle has one string (as in step 4), you  can iterate over all data like this:

Now I think it is more obvious why you can think of Excerpt as some kind of cursor :)

Step 7. Putting it all together

Ok, time to put all steps together. Here is sample code which opens chronicle in temp directory, writes 3 strings and dumps it. You can run it several times and see that each times you run it, more strings are printed, because data from previous runs is also stored in chronicle.

Summary

Chronicle is a big piece of data where Excerpt are stored. Each Excerpt has index inside Chronicle. Excerpt is a pointer to some memory where you can write data to. Data should be read in the same order it was written, so if you write data to excerpt with two writes: writeLong() and then writeBytes(), it is yours responsibility to read it in the same order (once you positioned your Excerpt in Chronicle). And each Excerpt has a predefined capacity, but you can have different capacity for different Excerpts

I hope this post helped you and now you have basic example of how to read and write data from chronicle and what is looks like. I will continue to play with library and hopefully will write more tutorials. Stay tuned

17 Responses to “Java chronicle library tutorial #1: Basic examples”

    • kenota

      Thank you. You can post a link from project if you like :)

    • kenota

      Thank you Abraham. No, you can’t start excerpt without knowing the size upfront. But why do you need to do that? Are you trying to write stream or something? I assume that at the moment when you are ready to write, you should be able to calculate the size of stuff you want to write.

  1. Szan

    Hello, nice example.
    But it looks like writeBytes changes excerpt started capacity !!!
    I’ve got working a litte bit changed example added updateToChronicle and hardcoded maximum capacity:
    private static int STRING_SIZE_OVERHEAD = 128;

    public static void updateToChronicle(Chronicle chr, String someString, int idx ) {
    final Excerpt excerpt = chr.createExcerpt();

    excerpt.startExcerpt( STRING_SIZE_OVERHEAD);

    excerpt.index(idx );

    System.out.println(“Update string in chronicle: ” + someString +” at idx: ” +idx
    +” length: ” +someString.length()
    +” capacity: ” +excerpt.capacity()
    );

    excerpt.position(0); // Set start of string

    excerpt.writeBytes(someString);
    excerpt.finish();
    }

    public static void writeToChronicle(Chronicle chr, String someString) {
    final Excerpt excerpt = chr.createExcerpt();

    excerpt.startExcerpt( STRING_SIZE_OVERHEAD);

    System.out.println(“Write string to chronicle: ” + someString
    +” length: ” +someString.length()
    +” capacity: ” +excerpt.capacity()
    );

    excerpt.writeBytes(someString);
    excerpt.finish();
    }
    Then in main:
    writeToChronicle(chr, “some text”); // is OK
    but
    updateToChronicle(chr, “Updated some text”, 1-1 ); // crashes and shows capacity 10 (not hardcoded 128 when excerpt was started) !!!

    So, when try to update excerpt with longer bytes(however <128 ) which was written lets say 10 bytes long using writeBytes with 128 started capacity, it crashes and I have such trace log:
    java -cp .:chronicle.jar HelloWorldChronicle
    base prefix: /tmp/helloworldchronicle
    Jul 16, 2013 4:15:05 PM com.higherfrequencytrading.chronicle.impl.IndexedChronicle
    INFO: /tmp/helloworldchronicle, size=33
    Write string to chronicle: some text length: 9 capacity: 128
    Write string to chronicle: more text length: 9 capacity: 128
    Write string to chronicle: and a little bit more length: 21 capacity: 128
    Update string in chronicle: Updated some text at idx: 0 length: 17 capacity: 10
    Exception in thread “main” java.lang.IllegalStateException: Capacity allowed: 10 data read/written: 18
    at com.higherfrequencytrading.chronicle.impl.AbstractExcerpt.checkEndOfBuffer(AbstractExcerpt.java:173)
    at com.higherfrequencytrading.chronicle.impl.AbstractExcerpt.finish(AbstractExcerpt.java:154)
    at HelloWorldChronicle.updateToChronicle(HelloWorldChronicle.java:31)
    at HelloWorldChronicle.main(HelloWorldChronicle.java:67)
    How to update data with longer text, which is shorter than declared 128 during excerpt creation and writing?
    When added extra spaces it worked, and no crashes….

    • kenota

      Szan, you missed a little bit of my code. When I am starting an excerpt, I am doing this “excerpt.startExcerpt(someString.length() + STRING_SIZE_OVERHEAD);” Basically, I am asking library to start an excerpt with capacity of string I am about to write + bytes for library to write size of the string. If you do the same, it should be fine. Let me know if it helped.

  2. Rajiv

    Thanks for the tutorial. I wish there were more Chronicle tutorials online. Hoping to see #2 and more :)

    • kenota

      Thank you for the comment! I am planning to write new tutorial soon, but Peter also planing on releasing Chronicle 2.0 :)

    • kenota

      Thank you. Unfortunately, I was busy and Chronicle moved to version 2 now, so I am not sure wether I should make tutorial based on version 1 or 2.

  3. Ashley

    Kenota, now since 2.3 is out, it would be great if you are able to write #2,3….this is great stuff.

    Thanks in advance.

    • kenota

      1. If you want to write object to Chronicle you need to serialize it yourself. You will need to read fields in the same order you write them.
      2. You will need to calculate size of object yourself as well.

  4. kart

    Thank you very much.This helps a lot………..Since I am starting to use this it helps

  5. Daniela

    Very nice tutorial. I was wondering if you know any client that actually uses the chronicle-map library. It would be interesting to know who else is using this library.

    • kenota

      Daniela,

      Personally I don’t have clients using Chronicle Map library, but I think Peter Lawrey (author of the library) is using it while providing consulting services for his clients at HFT.

Leave a Reply

  • (will not be published)