Tag Archives: networking

The Protocol Problem

In the interactive-diagrams project we have a bunch of components (processes) running independently and even under different UIDs. They however need to communicate with each other, and we’ve decided to go full UNIX-way (since already most of our code depend on POSIX-compatibility) and use IPC/UNIX-sockets for communication.

Originally, I’ve implemented the following protocol for sending the
data around:

When sending data:

  1. Encode the data using the functions from the cereal package.
  2. Take the ‘length’ of the resulting bytestring, send it over the
  3. Send the encoded data on the next line.

Upon receiving data:

  1. Read the first line, deserialize it to an x :: Int
  2. Read the the next x bytes, deserialize the data.

A programmer experienced in the area would have probably already
spotted an error in this approach, but for me it took some time to
find a bug, once I’ve realised that I was getting deserialization
errors from time to time.

The problem of course is that I was relying on reading lines from the
socket, not bytes. For example, the number 2623 on my 64bit system has
serialized to something with the newline character in it


The solution to this problem is to read the first ‘8’ (or ‘4’) bytes
to get the length of the upcoming data. To make sure that the number
of bytes representing the length of the data is constant on all the
platforms I’ve switched to using Word32.

This approach too, of course, is not prone to errors. If you are
sending and receiving a lot of data consider using a streaming

PS: We’ve planned originally to release the first public alpha-version yesterday, but it turned out that it is actually taking more time configuring SELinux, chroots and other security measure than expected. I’ve also changed the general structure of the project, I am using a different design model than I had a week ago. The new changes would make the application more salable and the resulting library can (and most likely will) be reused for other stuff. Stay tuned!