The Protocol Problem

In the interactive-diagrams project we have a bunch of components (processes) running independently and even under different UIDs. They however need to communicate with each other, and we’ve decided to go full UNIX-way (since already most of our code depend on POSIX-compatibility) and use IPC/UNIX-sockets for communication.

Originally, I’ve implemented the following protocol for sending the
data around:

When sending data:

  1. Encode the data using the functions from the cereal package.
  2. Take the ‘length’ of the resulting bytestring, send it over the
    socket.
  3. Send the encoded data on the next line.

Upon receiving data:

  1. Read the first line, deserialize it to an x :: Int
  2. Read the the next x bytes, deserialize the data.

A programmer experienced in the area would have probably already
spotted an error in this approach, but for me it took some time to
find a bug, once I’ve realised that I was getting deserialization
errors from time to time.

The problem of course is that I was relying on reading lines from the
socket, not bytes. For example, the number 2623 on my 64bit system has
serialized to something with the newline character in it

Q6I3.png

The solution to this problem is to read the first ‘8’ (or ‘4’) bytes
to get the length of the upcoming data. To make sure that the number
of bytes representing the length of the data is constant on all the
platforms I’ve switched to using Word32.

This approach too, of course, is not prone to errors. If you are
sending and receiving a lot of data consider using a streaming
library.

PS: We’ve planned originally to release the first public alpha-version yesterday, but it turned out that it is actually taking more time configuring SELinux, chroots and other security measure than expected. I’ve also changed the general structure of the project, I am using a different design model than I had a week ago. The new changes would make the application more salable and the resulting library can (and most likely will) be reused for other stuff. Stay tuned!

About these ads

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s