Preon Encoding Started

I started work on supporting Preon the other day. One of the first things I will have to do is to define the interface that will be used to write data. Picking that interface is not trivial. One of the ideas that I suggested last year is not to try to keep Preon symmetrical; that is, it probably makes sense not to preserve the random access type of access to the data when writing data, but write to a streaming type of interface instead.

There are a couple of reasons why that would make sense:

• In-place editing is just going to be way to painful: if you would a couple of elements to a List, you would need to move blocks of data around. I can hardly imagine what that would look like, and it's probably not going to be very efficient.
• Now, regardless on what the repercussions would be in terms of moving blocks of data around in memory and on disk, it would also imply changing all pointers that still have a reference to other places in that buffer of bytes. So, suppose that you would have two lazy loading lists after each other. Then in order to be able to lookup elements of the second list on the fly, that list will keep a reference to its starting point. That reference will be based on the offset of the start of its content relatively to the beginning of the file. Now, inserting elements in the first list also needs to result in updating those references.
• Preon aims to be threadsafe. That complicates matters further. As a result of adding an element to a list, thousands of pointers might potentially require an update, but from the perspective of the client consistency would need to be preserved during that transaction. This calls for an STM type-of solution.

I'm still not sure if it would be totally ridiculous to even support in-place editing, but it's just going to be to hard at this stage. Having the ability to 1) process changes, and 2) to persist all of those changes to disk in a single snapshot would already be quite a challenge, but it would be an awesome feature.

Tomorrow, I will try to add some thoughts on the shape of the interface that I am currently considering.