Sunday, August 30, 2009
Preon Presentation at OOPSLA 2009
Thursday, August 13, 2009
Getting the Ethernet Frame
A couple of days ago, I blogged about using Preon to take a Snoop file apart. It's interesting that it actually worked, but then again, it hardly retrieved any valuable data. Given that most of the data floating around on our networks is actually ethernet based, it would be nice if Preon would actually be able to take apart the data sections as well.
Obviously, I wouldn't be telling you all of this, if Preon would not be able to do that. So let me show you how. The full code is listed below.
The important bits are in line 60.
The code that we used to have there was this:
In the new version of SnoopFile, that changed into:
Basically, instead of decoding the packet data into a byte array, it's now immediately turned into an object. An object that actually has the relevant data already decoded. Now, don't be alarmed by the type of packetData. In the new code it's declared to be of type java.lang.Object. That's just to make sure that it will be able to cover for all of the types of packet data that you could possibly encounter. In the common case, it will be ethernet frames. But Snoop is able to capture data for other data link types as well, and it's fairly unlikely that these different types of packet data would have commonalities to be captured in a base class.
So, even thought he type of packetData is java.lang.Object, Preon will actually decode packet data in objects of a particular type, based on the data link type. The selectFrom attribute on @BoundObject contains the rules for picking the appropriate type of object. Currently, it only covers for Ethernet packages. It basically reads: if the data link type read as part of the header is Ethernet, then decode the data into an instance of EthernetFrame.
The EthernetFrame itself is defined like this:
Not much to explain here. It's pretty much straightforward Preon code. Note that the number of bytes is calculated based on an attribute of the packet record. Since the EthernetFrame is read as part of the PacketRecord, you *can* actually refer to the 'outer' context, and reference attributes of that outer context.
Obviously, I wouldn't be telling you all of this, if Preon would not be able to do that. So let me show you how. The full code is listed below.
The important bits are in line 60.
The code that we used to have there was this:
@Slice(size="(packetRecordLength - 24) * 8")
@BoundList(size="includedLength")
private byte[] packetData;
In the new version of SnoopFile, that changed into:
@Slice(size = "(packetRecordLength - 24) * 8")
@BoundObject(selectFrom = @Choices(alternatives =
@Choice(condition = "outer.header.datalinkType==DatalinkType.ETHERNET", type = EthernetFrame.class))
)
private Object packetData;
Basically, instead of decoding the packet data into a byte array, it's now immediately turned into an object. An object that actually has the relevant data already decoded. Now, don't be alarmed by the type of packetData. In the new code it's declared to be of type java.lang.Object. That's just to make sure that it will be able to cover for all of the types of packet data that you could possibly encounter. In the common case, it will be ethernet frames. But Snoop is able to capture data for other data link types as well, and it's fairly unlikely that these different types of packet data would have commonalities to be captured in a base class.
So, even thought he type of packetData is java.lang.Object, Preon will actually decode packet data in objects of a particular type, based on the data link type. The selectFrom attribute on @BoundObject contains the rules for picking the appropriate type of object. Currently, it only covers for Ethernet packages. It basically reads: if the data link type read as part of the header is Ethernet, then decode the data into an instance of EthernetFrame.
The EthernetFrame itself is defined like this:
public static class EthernetFrame {
@BoundList(size = "6")
private byte[] destinationAddress;
@BoundList(size = "6")
private byte[] sourceAddress;
@BoundNumber(size = "16")
private int type;
@BoundList(size="outer.includedLength - (6 + 6 + 2)")
private byte[] data;
}
Not much to explain here. It's pretty much straightforward Preon code. Note that the number of bytes is calculated based on an attribute of the packet record. Since the EthernetFrame is read as part of the PacketRecord, you *can* actually refer to the 'outer' context, and reference attributes of that outer context.
public class SnoopFile {
@Bound
private FileHeader header;
@BoundList(type = PacketRecord.class)
private List<PacketRecord> records;
public FileHeader getHeader() {
return header;
}
public List<PacketRecord> getRecords() {
return records;
}
public static class FileHeader {
@BoundBuffer(match = { 0x73, 0x6e, 0x6f, 0x6f, 0x70, 0x00, 0x00, 0x00 })
private byte[] identificationPattern;
@BoundNumber(byteOrder = ByteOrder.BigEndian)
private int versionNumber;
@BoundNumber(size = "32", byteOrder = ByteOrder.BigEndian)
private DatalinkType datalinkType;
public int getVersionNumber() {
return versionNumber;
}
public DatalinkType getDatalinkType() {
return datalinkType;
}
}
@ImportStatic(DatalinkType.class)
public static class PacketRecord {
@BoundNumber(byteOrder = ByteOrder.BigEndian, size = "32")
private long originalLength;
@BoundNumber(byteOrder = ByteOrder.BigEndian, size = "32")
private long includedLength;
@BoundNumber(byteOrder = ByteOrder.BigEndian, size = "32")
private long packetRecordLength;
@BoundNumber(byteOrder = ByteOrder.BigEndian, size = "32")
private long cumulativeDrops;
@BoundNumber(byteOrder = ByteOrder.BigEndian, size = "32")
private long timestampSeconds;
@BoundNumber(byteOrder = ByteOrder.BigEndian, size = "32")
private long timestampMicroseconds;
@Slice(size = "(packetRecordLength - 24) * 8")
@BoundObject(selectFrom = @Choices(alternatives = @Choice(condition = "outer.header.datalinkType==DatalinkType.ETHERNET", type = EthernetFrame.class)))
private Object packetData;
public long getOriginalLength() {
return originalLength;
}
public long getIncludedLength() {
return includedLength;
}
public long getPacketRecordLength() {
return packetRecordLength;
}
public long getCumulativeDrops() {
return cumulativeDrops;
}
public long getTimestampSeconds() {
return timestampSeconds;
}
public long getTimestampMicroseconds() {
return timestampMicroseconds;
}
public Object getPacketData() {
return packetData;
}
public static class EthernetFrame {
@BoundList(size = "6")
private byte[] destinationAddress;
@BoundList(size = "6")
private byte[] sourceAddress;
@BoundNumber(size = "16")
private int type;
@BoundList(size="outer.includedLength - (6 + 6 + 2)")
private byte[] data;
public String getDestinationAddress() {
return render(destinationAddress);
}
public String getSourceAddress() {
return render(sourceAddress);
}
private String render(byte[] address) {
StringBuilder builder = new StringBuilder();
for (int i = 0; i < address.length; i++) {
if (i != 0) {
builder.append(':');
}
builder.append(Integer.toHexString(0xff&address[i]));
}
return builder.toString();
}
}
}
public static enum DatalinkType {
@BoundEnumOption(0)
IEEE_802_3,
@BoundEnumOption(1)
IEEE_802_4_TOKEN_BUS,
@BoundEnumOption(2)
IEEE_802_5_TOKEN_RING,
@BoundEnumOption(3)
IEEE_802_6_METRO_NET,
@BoundEnumOption(4)
ETHERNET,
@BoundEnumOption(5)
HLDC,
@BoundEnumOption(6)
CHARACTER_SYNCHRONOUS,
@BoundEnumOption(7)
IBM_CHANNEL_TO_CHANNEL,
@BoundEnumOption(8)
FDDI,
@BoundEnumOption(9)
OTHER,
UNASSIGNED
}
}
Wednesday, August 12, 2009
Enums in Preon
You might have noticed in my previous entry that I introduced a new annotation to map enum values to their encoded representation. I already thought about that a couple of times, but never found time to do it. Now I introduced it, I notice that there is actually something missing here. Question is: will it be possible to introduce it without overhauling the entire framework? I think it is possible.
The Problem
So what is actually missing here? Well, with the latest addition of @BoundEnumOption, you can easily map an enum value to its encoded representation. That's nice; now you can have a typed collection of enum values instead of arbitrary integer values.
However, there is a gotcha. What will you will see a lot is that the enum value read influences the way you are supposed to read the remainder of the file. In the snoop example, this is no different. Currently, the example doesn't do anything with the actual data packets, but if it did, then it would have to decode it differently depending on the value of datalinkType.
In the past, you could easily use @If or @Switch and refer to the value of datalinkType, as long as it was an integer, String or boolean. However, now you need to compare that value to another enum value, and that's something that wasn't supported yet.
The Requirements
We want to be able to do something like this:
This means that Preon needs to be able to compare enums, and that it also needs to be able to resolve references to enum values.
The Challenge
Having the ability to compare enums doesn't look like a huge problem. In fact, I just wrote an additional test in LimboTest, and it seems to be working fine:
The difficulty is somewhere else: it's in resolving the enum values. Problem is Direction.LEFT is not something Limbo is capable of dereferencing; it's just not in lexical scope. So, the question is, do we need another version of Limbo now? One that is capable of resolving enum value references? The good new is: no, you don't.
The reason for that is that Limbo is extensible. It defines something called a ReferenceContext that needs to be implemented to allow Limbo to execute in a context.
Let's look at an example to clear things up a little. Let's assume that you want to evaluate "foo.bar + 3". When you feed this expression to Limbo, it will first try to see if this expression makes sense at all. So it will check if it *will* actually be able to resolve something named 'foo', and if that thing has an attribute called 'bar'. If it doesn't then it will just fail directly. Now, in order to check if there is something called 'foo' in its context, it will simply call selectAttribute("foo") on the current ReferenceContext. If that ReferenceContext decides that something like that is referenceable, it will return a Reference object representing the "foo" reference. Next, Limbo will check if there is actually an attribute "bar" on the type of object referenced by the Reference just created. If so, then it will return a Reference to that property.
The important thing to note here is that - whatever Limbo is trying to do - all requests always hit the ReferenceContext first. Once the References are 'linked', it will call resolve(…) on the references to resolve the values on an instance of that context.
So, if we want to put the enums into lexical scope, all we need to do is rewrite the ReferenceContext. And our ReferenceContext is defined by Preon, so we don't have to rewrite Limbo; we just need to write a small adjustment in Preon, in order to integrate Limbo slightly differently.
The Solution
The actual code required to make this work is a little bit more than I'm willing to list here. (It turns out you need a decorator for RefenceContext, and then two additional Reference implementations. One to refer to Classes, and one to reference static fields on a class. The solution works in the same way as outlined above. The only thing that you need to add is an @ImportStatic annotation, at the top of the class from which you are referencing the enum values:
The Problem
So what is actually missing here? Well, with the latest addition of @BoundEnumOption, you can easily map an enum value to its encoded representation. That's nice; now you can have a typed collection of enum values instead of arbitrary integer values.
However, there is a gotcha. What will you will see a lot is that the enum value read influences the way you are supposed to read the remainder of the file. In the snoop example, this is no different. Currently, the example doesn't do anything with the actual data packets, but if it did, then it would have to decode it differently depending on the value of datalinkType.
In the past, you could easily use @If or @Switch and refer to the value of datalinkType, as long as it was an integer, String or boolean. However, now you need to compare that value to another enum value, and that's something that wasn't supported yet.
The Requirements
We want to be able to do something like this:
@BoundNumber(size="8")
private DatalinkType datalinkType;
…
@If("datalinkType == DatalinkType.ETHERNET")
This means that Preon needs to be able to compare enums, and that it also needs to be able to resolve references to enum values.
The Challenge
Having the ability to compare enums doesn't look like a huge problem. In fact, I just wrote an additional test in LimboTest, and it seems to be working fine:
public void testComparingEnums() {
EasyMock.expect(defs.getType("a")).andReturn(Direction.class).anyTimes();
EasyMock.expect(defs.getType("b")).andReturn(Direction.class).anyTimes();
EasyMock.expect(resolver.get("a")).andReturn(Direction.LEFT).anyTimes();
EasyMock.expect(resolver.get("b")).andReturn(Direction.RIGHT).anyTimes();
EasyMock.replay(defs, resolver);
assertTrue(condition(context, resolver, "a == a"));
assertFalse(condition(context, resolver, "a == b"));
assertTrue(condition(context, resolver, "b == b"));
}
public static enum Direction {
LEFT,
RIGHT;
}
The difficulty is somewhere else: it's in resolving the enum values. Problem is Direction.LEFT is not something Limbo is capable of dereferencing; it's just not in lexical scope. So, the question is, do we need another version of Limbo now? One that is capable of resolving enum value references? The good new is: no, you don't.
The reason for that is that Limbo is extensible. It defines something called a ReferenceContext that needs to be implemented to allow Limbo to execute in a context.
Let's look at an example to clear things up a little. Let's assume that you want to evaluate "foo.bar + 3". When you feed this expression to Limbo, it will first try to see if this expression makes sense at all. So it will check if it *will* actually be able to resolve something named 'foo', and if that thing has an attribute called 'bar'. If it doesn't then it will just fail directly. Now, in order to check if there is something called 'foo' in its context, it will simply call selectAttribute("foo") on the current ReferenceContext. If that ReferenceContext decides that something like that is referenceable, it will return a Reference object representing the "foo" reference. Next, Limbo will check if there is actually an attribute "bar" on the type of object referenced by the Reference just created. If so, then it will return a Reference to that property.
The important thing to note here is that - whatever Limbo is trying to do - all requests always hit the ReferenceContext first. Once the References are 'linked', it will call resolve(…) on the references to resolve the values on an instance of that context.
So, if we want to put the enums into lexical scope, all we need to do is rewrite the ReferenceContext. And our ReferenceContext is defined by Preon, so we don't have to rewrite Limbo; we just need to write a small adjustment in Preon, in order to integrate Limbo slightly differently.
The Solution
The actual code required to make this work is a little bit more than I'm willing to list here. (It turns out you need a decorator for RefenceContext, and then two additional Reference implementations. One to refer to Classes, and one to reference static fields on a class. The solution works in the same way as outlined above. The only thing that you need to add is an @ImportStatic annotation, at the top of the class from which you are referencing the enum values:
@ImportStatic(DatalinkType.class)
public class SnoopFile {
@BoundNumber(size="8")
private DatalinkType datalinkType;
…
@If("datalinkType == DatalinkType.ETHERNET")
Monday, August 10, 2009
Snoop Using Preon
Just before my holiday started, Adriaan Thomas sent me an email, asking me about using Preon for decoding Solaris snoop files. That sounded like an interesting challenge. So, this weekend, in my spare hours, I wrote some code to see what that would look like. The code required is given below, and based on RFC 1761.
Now, if you want to decode a snoop file, that's pretty easy: you just create the Codec, and then pass that Codec to one of Codecs decode operations:
Codec<SnoopFile> codec = Codecs.createCodec(SnoopFile.class);
SnoopFile capture = Codecs.decode(codec, ...);
This is the definition of the SnoopFile class.
public class SnoopFile {
@Bound
private FileHeader header;
@BoundList(type=PacketRecord.class)
private List<PacketRecord> records;
public FileHeader getHeader() {
return header;
}
public List<PacketRecord> getRecords() {
return records;
}
public static class FileHeader {
@BoundBuffer(match = { 0x73, 0x6e, 0x6f, 0x6f, 0x70, 0x00, 0x00, 0x00 })
private byte[] identificationPattern;
@BoundNumber(byteOrder=ByteOrder.BigEndian)
private int versionNumber;
@BoundNumber(size="32", byteOrder=ByteOrder.BigEndian)
private DatalinkType datalinkType;
public int getVersionNumber() {
return versionNumber;
}
public DatalinkType getDatalinkType() {
return datalinkType;
}
}
public static class PacketRecord {
@BoundNumber(byteOrder = ByteOrder.BigEndian, size="32")
private long originalLength;
@BoundNumber(byteOrder = ByteOrder.BigEndian, size="32")
private long includedLength;
@BoundNumber(byteOrder = ByteOrder.BigEndian, size="32")
private long packetRecordLength;
@BoundNumber(byteOrder = ByteOrder.BigEndian, size="32")
private long cumulativeDrops;
@BoundNumber(byteOrder = ByteOrder.BigEndian, size="32")
private long timestampSeconds;
@BoundNumber(byteOrder = ByteOrder.BigEndian, size="32")
private long timestampMicroseconds;
@Slice(size="(packetRecordLength - 24) * 8")
@BoundList(size="includedLength")
private byte[] packetData;
public long getOriginalLength() {
return originalLength;
}
public long getIncludedLength() {
return includedLength;
}
public long getPacketRecordLength() {
return packetRecordLength;
}
public long getCumulativeDrops() {
return cumulativeDrops;
}
public long getTimestampSeconds() {
return timestampSeconds;
}
public long getTimestampMicroseconds() {
return timestampMicroseconds;
}
public byte[] getPacketData() {
return packetData;
}
}
public static enum DatalinkType {
@BoundEnumOption(0)
IEEE_802_3,
@BoundEnumOption(1)
IEEE_802_4_TOKEN_BUS,
@BoundEnumOption(2)
IEEE_802_5_TOKEN_RING,
@BoundEnumOption(3)
IEEE_802_6_METRO_NET,
@BoundEnumOption(4)
ETHERNET,
@BoundEnumOption(5)
HLDC,
@BoundEnumOption(6)
CHARACTER_SYNCHRONOUS,
@BoundEnumOption(7)
IBM_CHANNEL_TO_CHANNEL,
@BoundEnumOption(8)
FDDI,
@BoundEnumOption(9)
OTHER,
UNASSIGNED
}
}
Sunday, August 2, 2009
Preon 1.0
Preon 1.0 is out. It took a while, but it has arrived. It may have lost some features on the way, but I still consider it to be quite exceptional in its ambitions.
Currently, Preon and its dependent libraries are not in the central Maven repository yet. That's a little inconvenient, but it will be fairly easy to work around it, by copying this snippet into your pom. Note that - if you intend to use the binding framework - you only need a reference on preon-binding, version 1.0.
Currently, Preon and its dependent libraries are not in the central Maven repository yet. That's a little inconvenient, but it will be fairly easy to work around it, by copying this snippet into your pom. Note that - if you intend to use the binding framework - you only need a reference on preon-binding, version 1.0.
<repositories>
<repository>
<id>limbo-repository</id>
<url>http://limbo.sourceforge.net/repository</url>
<snapshots>
<enabled>false</enabled>
</snapshots>
<releases>
<enabled>true</enabled>
</releases>
</repository>
<repository>
<id>pecia-repository</id>
<url>http://pecia.sourceforge.net/repository</url>
<snapshots>
<enabled>false</enabled>
</snapshots>
<releases>
<enabled>true</enabled>
</releases>
</repository>
<repository>
<id>preon-repository</id>
<url>http://preon.sourceforge.net/repository</url>
<snapshots>
<enabled>false</enabled>
</snapshots>
<releases>
<enabled>true</enabled>
</releases>
</repository>
</repositories>
Subscribe to:
Posts (Atom)