Message Framing Protocol Failing Under Strenuous Conditions

Message Framing Protocol Failing Under Strenuous Conditions

I am using the open-source networking library called Griffin.Networking to
develop a program that allows me to monitor remote computers.
The author includes a sample binary protocol called
Griffin.Networking.Protocols.Basic. The purpose of this protocol is to
effectively serialize any outgoing messages and build any incoming
messages.
I am experiencing problems with the Message Builder when the network is
under a lot of stress. The client application attempts to send a 300kb
message every 250 milliseconds or so.



I appreciate that many of you will not be familiar with the library but
that does not mean you cannot answer my question. The author has done an
excellent job of making everything very loosely coupled which allows me to
quickly explain the specific part of the library I am having trouble with.
When incoming data is read from the internal TCP buffer into the
application's managed buffer, the managed buffer and the number of bytes
read are passed to a method called HandleRead
protected override void HandleRead(IBufferSlice slice, int bytesRead)
{
if (_messageBuilder.Append(new SliceStream(slice, bytesRead)))
{
object message;
while (_messageBuilder.TryDequeue(out message))
{
TriggerClientReceive(message);
}
}
}
This HandleRead method effectively feeds the message builder the incoming
data. If the incoming data forms an entire message (maybe more) then the
message builder will successfully proceed to dequeue the messages and
raise an appropriate event. If the incoming data does not form an entire
message then the message builder will maintain it's state (i.e. the data
we have already read and how much data is left to read) until the
application makes another successful read.



The MessageBuilder class itself is as followed. Please excuse the long
segment of code. The code is available on GitHub here.
Note that I modified the code presented here slightly for formatting
purposes. Also note that I replaced the JsonSerializer with the
BinaryFormatter and that is reflected in my code sample
public class BasicMessageBuilder : IMessageBuilder
{
private readonly byte[] _header = new byte[Packet.HeaderLength];
private readonly Queue<Packet> _messages = new Queue<Packet>();
private int _bytesLeft = Packet.HeaderLength;
private Packet _packet;
private Func<IBufferReader, bool> _parserMethod;
public BasicMessageBuilder()
{
_parserMethod = ReadHeaderBytes;
}
public bool Append(IBufferReader reader)
{
while (_parserMethod(reader))
{
}
return _messages.Count > 0;
}
public bool TryDequeue(out object message)
{
if (_messages.Count == 0)
{
message = null;
return false;
}
var packet = _messages.Dequeue();
return true;
}
public void Reset()
{
_messages.Clear();
_packet = null;
_bytesLeft = Packet.HeaderLength;
_parserMethod = ReadHeaderBytes;
}
protected virtual bool ReadHeaderBytes(IBufferReader stream)
{
var bytesLeftInStream = stream.Count - stream.Position;
var bytesToCopy = bytesLeftInStream < _bytesLeft
? bytesLeftInStream
: _bytesLeft;
stream.Read(_header, 0, bytesToCopy);
_bytesLeft -= bytesToCopy;
if (_bytesLeft > 0)
return false;
_packet = CreatePacket(_header);
_bytesLeft = _packet.ContentLength;
_parserMethod = ReadBodyBytes;
return true;
}
protected virtual bool ReadBodyBytes(IBufferReader reader)
{
var bytesLeftInStream = reader.Count - reader.Position;
var bytesToCopy = bytesLeftInStream < _bytesLeft
? bytesLeftInStream
: _bytesLeft;
reader.CopyTo(_packet.Message, bytesToCopy);
_bytesLeft -= bytesToCopy;
if (_bytesLeft > 0)
return false;
_packet.Message.Position = 0;
_messages.Enqueue(_packet);
_packet = null;
_bytesLeft = Packet.HeaderLength;
_parserMethod = ReadHeaderBytes;
return true;
}
protected virtual Packet CreatePacket(byte[] header)
{
var message = new Packet
{
Version = _header[0],
ContentLength = BitConverter.ToInt32(header, 1)
};
if (message.Version <= 0)
throw new InvalidDataException();
if (message.ContentLength <= 0)
throw new InvalidDataException();
message.Message = new MemoryStream(message.ContentLength);
return message;
}
}



When the client application sends relatively large messages in quick
succession the MessageBuilder.CreatePacket() method throws an
InvalidDataException.
I really don't know what is wrong but I do have a suspicion.
I believe that the MessageBuilder is reading header bytes when it should
be reading body bytes and because of this, the incoming data it perceives
to be a header will be invalid and an exception will be thrown. I cannot
explain this behaviour though
I do not believe lock will help here. This doesn't appear to have anything
to do with multiple threads accessing the same resource. I tested this by
writing the current thread's ID to a trace log.
I thought that perhaps the asynchronous call back was being called in
quick succession and that was going to cause the MessageBuilder to confuse
its' state but it is my understanding that every call will be executed in
sequential order because the message builder code is synchronous.
What I find to be interesting (and confusing) is that the client
application can send a few hundred 300kb messages (often more, sometimes
less) before the server application crashes.
This is not an urgent problem. I wrote my own MessageBuilder that operates
completely differently and everything works fine. I would like to solve
this conundrum as to why the aforementioned MessageBuilder fails though.
Could anybody help me understand the problem here?