Teaching Indigo to do REST/POX: Part 6

I threw a lot of unexplained code at you in Part 5 and that wasn’t really fair.

The PoxEncoder class is a replacement for Indigo’s default TextMessageEncoder class that’s used by the HTTP transport unless you explicitly configure something different. Indigo comes with three built-in encoders, namely:

· The TextMessageEncoder serializes Indigo’s internal Message into SOAP 1.1 or SOAP 1.2 envelopes using (applies only to the latter) the desired character encodings (UTF-8, UTF-16, etc.) and of course it also deserializes incoming SOAP envelopes into the Indigo representation.

· The MtomMessageEncoder serializes messages into SOAP 1.2 messages as specified by the MTOM specification, which allows for a much more compact transmission of binary-heavy SOAP envelopes than if you were simply using base64Binary encoded element data. MTOM is a good choice whenever the size of binary content in a SOAP envelope far exceeds the size of the rest of the data. Your mileage may vary, so that’s a thing to measure carefully unless it’s blatantly obvious such as in the case of writing a service for a digital imaging library.

· The BinaryMessageEncoder serializes messages into SOAP 1.2 envelopes, but does so in a very compact binary format that preserves the XML information set, but is not at all like XML text. The gist of the binary encoding is the assumption that if both communicating parties are implemented with Indigo and share the same contract, the metadata existing at both ends reduces the hints that need to go on the wire. In other words: The binary encoding doesn’t need to throw all these lengthy XML tag names and namespace names explicitly onto the wire, but can refer to them by pointing to a dictionary that’s identically constructed on both ends. The binary encoding in Indigo is a bit like the modern-day, loosely coupled grand-child of NDR and “midl.exe /Oicf” if you like. What’s important to note about this encoding is that its primary design goal is performance and interoperability is in fact a non-goal. The BinaryMessageEncoder assumes Indigo endpoints. If you don’t like that, you can always use the text encoding, which is designed for interoperability.

Our PoxEncoder here differs from all three Indigo encoders in that it does specifically not serialize SOAP messages, but rather just the body contents of a Message.

In order for you to understand what’s happening here, I’ll pick the most relevant methods and explain them in detail. We will start with the Initialize() method that is invoked by all three constructor overloads:

/// <summary>
/// Initializes common properties of the encoder.
/// </summary>
private void Initialize()
{
if (this.MessageVersion.Envelope == EnvelopeVersion.Soap12)
{
    // set the aprorpiate media type for SOAP 1.2
   this. mediaType = "application/soap+xml";
}
else if (this.MessageVersion.Envelope == EnvelopeVersion.Soap11)
{
    // set the appropriate media type for SOAP 1.1
     this. mediaType = "text/xml";
}
// compose the content type from charset and media type
  this. contentType = string.Format(CultureInfo.InvariantCulture, "{0}; charset={1}", mediaType, textEncoding.WebName);
}

It is required for each MessageEncoder-derived class to implement the abstract properties MediaType, ContentType, and MessageVersion, and therefore we have to initialize the backing fields for these properties properly and return meaningful values even though the PoxEncoder is exactly the “anti-SOAP” encoder. The message version specified in the encoder is relevant for Indigo higher up on the stack, because it needs to know what rules and constraints apply to Message instances as they are constructed and processed. The content type and media types are required by the transports so that they know what content and/or media type to specify as metadata in their transport frame (eg. the Content-Type header in HTTP). If we initialize the encoder with the Soap12 message version, it will consequently report the application/soap+xml media type, even though the encoder doesn’t ever write such envelopes to the wire. You might consider that a bug in the PoxEncoder and you might be right, but it doesn’t really matter. Because any methods can return all sorts of payloads, we will override the content-type on the message-level so that this information has really no effect. I do need to clean this up a little. Later.

Now let’s look at the parts that actually do the work. I will start with the two WriteMessage overloads.

The first overload’s signature is
public override ArraySegment<byte> WriteMessage(Message msg, int maxMessageSize, BufferManager bufferManager, int messageOffset)
and is invoked by the transport whenever a message must be wire-encoded and the output transfer mode is set to TransferMode.Buffered or TransferMode.StreamedRequest (which implies a buffered response). The second overload’s signature is
public override void WriteMessage(Message msg, System.IO.Stream stream)
and is invoked by the transport whenever a message must be wire-encoded and the output transfer mode is set of TransferMode.Streamed or TransferMode.StreamedResponse (which implies a streamed response).

The transfer-mode property is configurable on all of the pre-built HTTP bindings and on the <httpTransport> binding element. “Buffered” encoding means that the entire message is encoded at once and written into a buffer, which is then given to the the transport for sending. “Streamed” encoding means that the message is pushed into to a stream, whereby the stream is typically immediately layered directly over the transport. That means that whenever our encoder writes data to that stream, it is immediately pushed to the remote communication partner. The “streamed” mode is the optimal choice for sending very large messages that are, for instance, too big to be reasonably handled as a single memory block. The buffered mode is better (and faster) for compact messages. I’ll dissect the buffered variant first:

public override ArraySegment<byte> WriteMessage(Message msg, int maxMessageSize, BufferManager bufferManager, int messageOffset)
{
   if (msg.IsEmpty)
   {
      // if the message is empty (no body defined) the result is an empty
      // byte array.
      byte[] buffer = bufferManager.TakeBuffer(maxMessageSize);
      return new ArraySegment<byte>(buffer, 0, 0);
   }

If the message is empty (that means: the body is empty), we request a buffer from the buffer manager and return an empty slice of that buffer, because this encoder’s output is “nothing” if the body is empty.The BufferManager is an Indigo helper class that manages a pool of pre-allocated buffers and serves to optimize memory management by avoiding the allocation and the discarding of buffers for every message. And encoder should therefore use the buffer manager argument and use it to obtain the buffers backing the array segment that is to be returned. Once the message has been handled by the transport, the transport will return the buffer into the pool.

   else
   {
      // check RawBinary bit in the message property
          bool rawBinary = false;
          if (msg.Properties.ContainsKey(PoxEncoderMessageProperty.Name))
      {
         rawBinary = ((PoxEncoderMessageProperty)msg.Properties[PoxEncoderMessageProperty.Name]).RawBinary;
      }

If the message is not empty (we have a body), we check whether there is a PoxEncoderMessageProperty present in the message. This property is a plain CLR class that is part of my extensions and has two significant properties: Name is a static, constant string value used as the key for the message properties collection and RawBinary is a Boolean instance value that contains and indicator for whether the encoder shall encode the data as XML or as raw binary data. The Message property collection is a simple dictionary of objects keyed by strings. The properties allow application-level code to interact with infrastructure-level code in the way illustrated by this property. Whenever I want the encoder to use its “raw binary” mode, I add this property to the message and the encoder can pick up the information.

      ArraySegment<byte> retval = new ArraySegment<byte>();
      byte[] buffer = bufferManager.TakeBuffer(maxMessageSize);
      if (!rawBinary)
      {
         // If we're rendering XML data, we construct a memory stream
         // over the output buffer, layer an XMLDictionaryWriter on top of it
         // and have the message write the body content into the buffer as XML.
         // The buffer is then wrapped into an array segment and returned.
         MemoryStream stream = new MemoryStream(buffer);
         XmlWriterSettings settings = new XmlWriterSettings();
         settings.OmitXmlDeclaration = true;
         settings.Indent = true;
         settings.Encoding = this. textEncoding;
         XmlWriter innerWriter = XmlWriter.Create(stream, settings);
         XmlDictionaryWriter writer = XmlDictionaryWriter.CreateDictionaryWriter(innerWriter, false);
         msg.WriteBodyContents(writer);
         writer.Flush();
         retval = new ArraySegment<byte>(buffer, 0, (int)stream.Position);
      }

Next we take a buffer from the buffer manager and if we’re not in “raw binary” mode, we’ll construct a memory stream over the buffer, construct an XmlDictionaryWriter over that stream and ask the message to render its “body contents” into the writer and therefore into the memory stream and into the buffer. The “body contents” of a message is what would be the child nodes of the <soap:Body> element, if we were using that (but we don’t). Once the body contents have been written, we flush the writer to make sure that all buffered data is committed into the underlying stream and then construct the return value as an array segment over the buffer with the length of the bytes written to the stream.

      else
      {
         // If we're rendering raw binary data, we grab at most 'buffer.Length'
         // bytes from the binary content of the base64Binary element (if that
         // exists) and return the result wrapped into an array segment.
         XmlDictionaryReader dictReader = msg.GetReaderAtBodyContents();
         if (dictReader.NodeType == XmlNodeType.Element &&
            dictReader.LocalName == "base64Binary")
         {
            if (dictReader.Read() && dictReader.NodeType == XmlNodeType.Text)
            {
               int size = dictReader.ReadContentAsBase64(buffer, 0, buffer.Length);
               retval = new ArraySegment<byte>(buffer, 0, size);
            }
         }
      }
      return retval;
   }
}

If the “raw binary” mode is to be used, we are making a bit of an assumption inside the encoder. The assumption is that the body content consists of a single element named “base64Binary” and that its content is just that: base64 binary encoded content. That is of course the other side of the PoxBase64XmlStreamReader trick I explained in Part 5. For binary data we simply assume here that the body reader is our wrapper class and this is how arbitrary binary data is smuggled through the Indigo infrastructure. The array segment to be returned is constructed by reading the binary data into the buffer and setting the array segment length to the number of bytes we could get from the element content.

The streamed version of WriteMessage is quite different:

public override void WriteMessage(Message msg, System.IO.Stream stream)
{
    try
    {
        if (!msg.IsEmpty)
        {
            // check RawBinary bit in the message property
            bool rawBinary = false;
            if (msg.Properties.ContainsKey(PoxEncoderMessageProperty.Name))
            {
                rawBinary = ((PoxEncoderMessageProperty)msg.Properties[PoxEncoderMessageProperty.Name]).RawBinary;
            }
            if (!rawBinary)
            {
                // If we're rendering XML, we layer an XMLDictionaryWriter over the
                // output stream and have the message render its body content into
                // that writer and therefore onto the stream.
                XmlWriterSettings settings = new XmlWriterSettings();
                settings.OmitXmlDeclaration = true;
                settings.Indent = true;
                settings.Encoding = this. textEncoding;
                XmlWriter innerWriter = XmlWriter.Create(stream, settings);
                XmlDictionaryWriter writer = XmlDictionaryWriter.CreateDictionaryWriter(innerWriter, false);
                msg.WriteBodyContents(writer);
                writer.Flush();
            }

The first significant difference is that if we’re using streams, we will simply ignore empty messages and do nothing with them. In streaming mode, the transport will do any setup work required for sending a message before invoking the encoder and ready the output network stream so that the encoder can write to it. When the encoder returns, the transport considers the write action done. So if we don’t write to the output stream, there’s no payload data hitting the wire and that happens to be what we want.

If we have data and we’re not in “raw binary” mode, the encoder will construct an XmlDictionaryWriter over the supplied stream and have the message write its body contents to it. That’s all.

            Else
            {
                // If we're rendering raw binary data, we grab chunks of at most 1MByte
                // from the 'base64Binary' content element (if that exists) and write them
                // out as binary data to the output stream. Chunking is done, because we
                // have to assume that the body content is arbitrarily large. To optimize the
                // behavior for large streams, we read and write concurrently and swap buffers.
                XmlDictionaryReader dictReader = msg.GetReaderAtBodyContents();
                if (dictReader.NodeType == XmlNodeType.Element && dictReader.LocalName == "base64Binary")
                {
                    if (dictReader.Read() && dictReader.NodeType == XmlNodeType.Text)
                    {
                        byte[] buffer1 = new byte[1024*1024], buffer2 = new byte[1024*1024];
                       byte[] readBuffer = buffer1, writeBuffer = buffer2;

                        int bytesRead = 0;
                        // read the first chunk into the read buffer
                        bytesRead = dictReader.ReadContentAsBase64(readBuffer, 0, readBuffer.Length);
                        do
                        {
                            // the abort condition for the loop is that we can't read
                            // any more bytes from the input because the base64Binary element is
                            // exhausted.
                            if (bytesRead > 0 )
                            {
                                // make the last read buffer the write buffer
                                writeBuffer = readBuffer;
                                // write the write buffer to the output stream asynchronously
                                IAsyncResult result = stream.BeginWrite(writeBuffer, 0, bytesRead,null,null);
                                // swap the read buffer
                                readBuffer = (readBuffer == buffer1) ? buffer2 : buffer1;
                                // read a new chunk into the 'other' buffer synchronously
                                bytesRead = dictReader.ReadContentAsBase64(readBuffer, 0, readBuffer.Length);
                                // wait for the write operation to complete
                                result.AsyncWaitHandle.WaitOne();
                                stream.EndWrite(result);
                            }
                        }
                        while (bytesRead > 0);
                    }
                }
            }
        }
    }
    catch
    {
        // the client may disconnect at any time, so that's an expected exception and absorbed.
    }
}

In streamed “raw binary” mode things get a bit more complicated. Under these circumstances we assume that the output we are sending is HUGE. The use-case I had in mind when I wrote this is the download of multi-GByte video recordings. Therefore I construct two 1MByte buffers that are used in turns to read a chunk of data from the source body reader (for which we make the same content assumption as for the buffered case: This is believed to be a PoxBase64XmlStreamReader compatible infoset) and asynchronously push the read data into the output stream.

Because it may take a while to get a huge data stream to the other side, a lot of things can happen to the network connection during that time. Therefore the encoder fully expects that the network connection terminates unexpectedly. If that happens, we’ll catch and absorb the network exception and happily return to the caller as if we’re done.

Compared to all the complexity of the WriteMessage overloads, the respective ReadMessage methods look fairly innocent, simple, and similar:

/// <summary>
/// Reads an incoming array segment containing a message and
/// wraps it with a buffered message. The assumption is that the incoming
/// data stream is <i>not</i> a SOAP envelope, but rather an unencapsulated
/// data item, may it be some raw binary, an XML document or HTML form
/// postback data. This method is called if the inbound transfer mode of the
/// transport is "buffered".
/// </summary>
/// <param name="buffer">Buffer to wrap</param>
/// <param name="bufferManager">Buffer manager to help with allocating a copy</param>
/// <returns>Buffered message</returns>
public override Message ReadMessage(ArraySegment<byte> buffer, BufferManager bufferManager)
{
return new PoxBufferedMessage(buffer, bufferManager);
}

/// <summary>
/// Reads an incoming stream containing a message and
/// wraps it with a streamed message. The assumption is that the incoming
/// data stream is <i>not</i> a SOAP envelope, but rather an unencapsulated
/// data item, may it be some raw binary, an XML document or HTML form
/// postback data. This method is called if the inbound transfer mode of the
/// transport is "streamed".
/// </summary>
/// <param name="stream">Input stream</param>
/// <param name="maxSizeOfHeaders">Maximum size of headers in bytes</param>
/// <returns>Stream message</returns>
public override Message ReadMessage(System.IO.Stream stream, int maxSizeOfHeaders)
{
return new PoxStreamedMessage(stream, maxSizeOfHeaders);
}

Both variants take the raw incoming data (whatever it is) and hand it to the PoxStreamMessage class or PoxBufferedMessage class that adopt the buffer or stream as their body content, respectively. I’ll explain those in Part 7.

Happy New Year!

Clemens Vasters