Clemens Vasters

Football. Cloud. Tech. Politics. Pictures. Principal Architect for Messaging/Eventing at Microsoft.

Part 1, Part 2, Part 3, Part 4, Part 5, Part 6, Part 7, Part 8, Part 9

Where are we?

·         In Parts 1 and 2, I explained contracts in the REST/POX context and the dispatch mechanisms that we need to enable Indigo to accept and handle REST/POX requests. With that I introduced a metadata extension, the [HttpMethod] attribute that can be used to mark up operations on an Indigo contract with HTTP methods and a URI suffixes that we can dispatch on. I also showed how we can employ a parameter inspector to extract URI-embedded arguments and flow them to the operation in a message property.

·         In Parts 3 and 4, I showed how we use the [HttpMethodOperationSelector] attribute to replace Indigo’s default address filtering and operation selection mechanisms, basically the entire message dispatch mechanism, with our own variants. The SuffixFilter is used to find the appropriate endpoint for an incoming request and the HttpMethodOperationSelectorBehavior  find the operation (method) on that endpoint which shall receive the incoming request message.

·         In Parts 5 and 6, you saw how the PoxEncoder puts outbound envelope-less POX documents onto the wire in its WriteMessage methods and accepts incoming non-SOAP XML requests through its ReadMessage methods and wraps them with an in-memory envelope (“message”) for further processing. I also showed the PoxBase64XmlStreamReader, which is an XML Infoset wrapper for arbitrary binary streams that interacts with the PoxEncoder to allow smuggling any sort of raw binary content through the Indigo infrastructure and onto the wire.

We’re pretty far along already. We’ve got the dispatch mechanisms, we know how to hook the dispatch metadata into the services, we’ve got the wire-encoding – we have most of the core pieces together. In fact, the last two key classes we’re missing (configuration hooks aside) are two specialized message classes that we need to handle incoming requests. In Part 6, you could see that the two ReadMessage overloads of the PoxEncoder delegate all work to the PoxBufferedMessage for the “buffered” transfer-mode overload and to PoxStreamedMessage for the “streamed” transfer mode overload.

ReadMessage is called on an encoder whenever a transport has received a complete message buffer (buffered mode) or has accepted and opened a network stream (streamed mode).

Using streamed mode means very concretely that Indigo will start handling the message even though the message might not have completely arrived. A transport in streaming mode will only do as much as it needs to do in order to deal with the transport-level framing protocol. I use “framing protocol” as a general term for what is done at the transport level to know what the nature of the payload is and where the payload starts and ends. For HTTP, the HTTP transport figures out whether an incoming request is indeed an HTTP request, will read/parse the HTTP headers, and will then layer a stream over the request’s content, irrespective of whether the transfer of that byte sequence has already been completed. This stream is immediately handed off to the rest of the Indigo infrastructure and the transport has done its work by doing so.

Pulling the remaining bytes from that stream is someone else’s responsibility in streamed mode. Whenever a piece of the infrastructure pulls data directly or indirectly from the stream and the data chunk requested is still in transfer, the stream will block and wait until the data is there. The transport’s handling of the framing protocol will typically also take care of chunking and thus make a chunked stream appear to be continuous. When I say “indirect pull” I mean that it may very well be an XmlDictionaryReader layered over an XmlReader layered over the incoming network stream.

The streaming mode is of particular interest for very large messages that may, in an extreme case, be virtually limitless in size. There’s no specification that says that you cannot stick 500 Terabyte or 500 Exabyte worth of data (think 365x24 live 1080i video streams) into a single message. As long as you have some reason to believe that the sender will eventually, in 20 years from now, give you “</soap:Body></soap:Envelope>” to terminate the message, the message can be assumed to be well-formed and complete.

No matter whether you use buffered or streamed mode, the configured encoder’s ReadMessage method is the first place where the read data chunk or the stream goes and that delegates, as shown to our two message classes. So let’s look at them.

We’ll primarily look at the PoxBufferedMessage, which is constructed over the read message buffer in the PoxEncoder like this:

public override Message ReadMessage(ArraySegment<byte> buffer, BufferManager bufferManager)
{
   return new PoxBufferedMessage(buffer, bufferManager);
}

The class PoxBufferedMessage is derived from the abstract System.ServiceModel.Message class and implements the base-class’s abstract properties Headers, Properties, and Version and overrides the OnClose(), OnGetReaderAtBodyContents(), and OnWriteBodyContents() virtual methods. Internally, Indigo has several such Message implementations that are each customized for certain scenarios. Implementing own variants of Message is simply another extensibility mechanism that Indigo gives us.

Using System;
using System.Collections.Generic;
using System.Text;
using System.ServiceModel;
using System.IO;
using System.Xml;
using System.Runtime.CompilerServices;
using System.ServiceModel.Channels;

namespace newtelligence.ServiceModelExtensions
{
    /// <summary>
    /// This class is one of the message classes used by the <see cref="T:PoxEncoder"/>
    /// It serves to wrap an unencapsulated data buffer with a message structure.
    /// The data buffer becomes the body content of the message.
    /// </summary>
   public class PoxBufferedMessage : Message, IPoxRawBodyMessage
   {
      MessageHeaders headers = new MessageHeaders(MessageVersion.Soap11Addressing1);
      MessageProperties properties = new MessageProperties();
      byte[] buffer;
      int bufferSize;
      BufferManager bufferManager;
      Stream body;
       
        /// <summary>
        /// Initializes a new instance of the <see cref="T:PoxBufferedMessage"/> class.
        /// </summary>
        /// <param name="buffer">The buffer.</param>
      public PoxBufferedMessage(byte[] buffer)
      {
            bufferManager = null;
            buffer = buffer;
            bufferSize = buffer.Length;
      }

        /// <summary>
        /// Initializes a new instance of the <see cref="T:PoxBufferedMessage"/> class.
        /// </summary>
        /// <param name="buffer">The buffer.</param>
        /// <param name="bufferManager">The buffer manager.</param>
        public PoxBufferedMessage(ArraySegment<byte> buffer, BufferManager bufferManager)
        {
            bufferManager = bufferManager;
            bufferSize = buffer.Count;
            buffer = bufferManager.TakeBuffer( bufferSize);
            Array.Copy(buffer.Array, buffer.Offset, buffer, 0, bufferSize);
        }     

We can construct instances of the class over a raw byte array or an “array segment” layered over such an array. Array segments are preferred over raw arrays, because their use eases memory management. You can keep a pool of buffers with a common size, even though the actual content is shorter than the buffer size and probably even offset from the lower buffer boundary. If we get a raw byte array we simply adopt it, but if we get an array segment alongside a reference to a buffer manager we take a new buffer from the buffer manager and copy the array segment to that acquired buffer.

        /// <summary>
        /// Called when the message is being closed.
        /// </summary>
        protected override void OnClose()
        {
            base.OnClose();
            if ( bufferManager != null)
            {
                bufferManager.ReturnBuffer( buffer);
            }           
        }

When we close the message and we have acquired it using the buffer manager (which is signaled by the presence of the reference) we duly return it once the message is being closed (or disposed or finalized).

The next two methods are an implementation of the IPoxRawBodyMessage interface that is, you guessed it, defined in my extensions. If the handler method wants to get straight at the raw body content knowing that it doesn’t expect XML, it can shortcut by the whole XmlReader and XML serialization story by asking for the BodyContentType and pull out the raw body data as a stream layered over the buffer:

        /// <summary>
        /// Gets the raw body stream.
        /// </summary>
        /// <returns></returns>
      [MethodImpl(MethodImplOptions.Synchronized)]
      public Stream GetRawBodyStream()
      {
         if ( body == null)
         {
             body = new MemoryStream( buffer,0, bufferSize,false,true);
         }
         return body;
      }

        /// <summary>
        /// Gets the content type of the raw message body based on the Content-Type HTTP header
        /// contained in the HttpRequestMessageProperty or HttpResponseMessageProperty of this
        /// message. The value is null if the type is unknown.
        /// </summary>
        public string BodyContentType
        {
            get
            {
                if (Properties.ContainsKey(HttpRequestMessageProperty.Name))
                {
                    return ((HttpRequestMessageProperty)Properties[HttpRequestMessageProperty.Name]).Headers["Content-Type"];
                }
                if (Properties.ContainsKey(HttpResponseMessageProperty.Name))
                {
                    return ((HttpResponseMessageProperty)Properties[HttpResponseMessageProperty.Name]).Headers["Content-Type"];
                }
                return null;
            }
        }

There is a bit of caution required using this mechanism, though. Because the message State (Created, Written, Read, Copied, Closed) is controlled by the base-class and cannot be set by derived classes, the message should be considered to be in the State==MessageState.Read after calling the GetRawBodyStream() method. That doesn’t seem to be necessary because we have a buffer here, but for the streamed variant that’s a must. And for the sake of consistency we introduce this constraint here.

The BodyContentType property implementation seems, admittedly, a bit strange at first sight. Even though you won’t see the message properties being populated anywhere inside this class, we’re asking for them and base the content-type detection on their values. That only makes sense when we consider the way messages are being populated by Indigo. As I explained, the first thing that gets called once the transport has a raw data chunk or stream in its hands that it believes to be a message, it invokes the encoder. For incoming requests/messages, the encoder is really serving as the message factory constructing Message-derived instances over raw data. Once the encoder has constructed the message in one of the ReadMessage overloads, the message is returned to the transport. If the transport wants, it can then (and the HTTP transport does) stick properties into that newly created message and then hand it off to the rest of the channel infrastructure for processing and dispatching. Because these extensions are built for REST/POX and therefore have HTTP affinity, that’s precisely what we assume to be happening for the BodyContentType property and the CreateBodyReader() method below. As I already explained in Part 1, the HTTP transport will always add a HttpRequestMessageProperty  to the message and that’s consequently from which we can grab the content-type of the incoming request data.

        private XmlDictionaryReader CreateBodyReader()
        {
            XmlDictionaryReader reader = null;

            /*
             * Check whether the message properties indicate that this is a raw binary message.
             * In that case, we'll wrap the body with a PoxBase64XmlStreamReader
             */
            bool hasPoxEncoderProperty = Properties.ContainsKey(PoxEncoderMessageProperty.Name);
            if (!(hasPoxEncoderProperty && ((PoxEncoderMessageProperty)Properties[PoxEncoderMessageProperty.Name]).RawBinary))
            {
                string contentType = null;

                /*
                 * Check for whether either the HttpRequestMessageProperty or the HttpResponseMessageProperty
                 * are present. If so, extract the HTTP Content-Type header. Otherwise the content-type is
                 * assumed to be text/xml ("POX")
                 */
                bool hasRequestProperty = Properties.ContainsKey(HttpRequestMessageProperty.Name);
                bool hasResponseProperty = Properties.ContainsKey(HttpResponseMessageProperty.Name);
                if (hasResponseProperty)
                {
                    HttpResponseMessageProperty responseProperty =
                      Properties[HttpResponseMessageProperty.Name] as HttpResponseMessageProperty;
                    contentType = responseProperty.Headers["Content-Type"];
                }
                else if (hasRequestProperty)
                {
                    HttpRequestMessageProperty requestProperty =
                       Properties[HttpRequestMessageProperty.Name] as HttpRequestMessageProperty;
                    contentType = requestProperty.Headers["Content-Type"];
                }

                if (contentType == null)
                {
                    contentType = "text/xml";
                }

                /*
                 * If the content type is text/xml (POX) we will create a plain XmlTextReader for the body.
                 */
                if (contentType.StartsWith("text/xml", StringComparison.OrdinalIgnoreCase))
                {
                   // do we only have a UTF byte-order mark?
                   if (_bufferSize <= 4)
                   {
                       // create a new reader over a fake infoset and place it on the EndElement
                       reader = XmlDictionaryReader.CreateDictionaryReader(
                          new XmlTextReader(new StringReader("<no-data></no-data>")));
                       reader.Read(); reader.Read();
                   }
                   else
                   {
                       reader = XmlDictionaryReader.CreateDictionaryReader(new XmlTextReader(GetRawBodyStream()));
                   }

                }
            }
            /*
             * If the content wasn't identified to be POX, we'll wrap it as binary. 
             */
            if (reader == null)
            {
                reader = XmlDictionaryReader.CreateDictionaryReader(new PoxBase64XmlStreamReader(GetRawBodyStream()));
            }
            return reader;
        }

The private CreateBodyReader() method that constructs XML readers for the both, the OnGetBodyReaderAtBodyContents() and the OnWriteBodyContents() overrides shown below, uses the same strategy to figure out the content-type of the message and therefore to guess what’s hidden inside the byte-array (or array segment) the message was constructed over. To make the message class useful for the request and response direction, we’ll distinguish there two separate cases here:

·         If the message is a response, the handling method in the user code might have indicated that it wants the encoder to serialize the message onto the wire in “raw binary” mode. The indicator for that is the presence of the PoxEncoderMessageProperty having the RawBinary property set to true. If that is the case, the reader we return is always our PoxBase64XmlStreamReader. The property cannot occur in request messages because the Indigo transports simply don’t know about it.

·         If the message is a request or a response with the mentioned property missing, we will try figuring out the message’s content-type using the described strategy of using the HTTP transport’s message properties. If we can’t figure out a content-type for a response (it’s optional for the responding handler code to supply it), we will assume that the content-type is “text/xml”. If the message is a request we can rely of getting a content-type as long as the underlying transport is Indigo’s HTTP transport implementation. If the content-type is indeed “text/xml” we construct an XmlTextReader over the raw data and return it. If the content-type is anything else, we use our PoxBase64XmlStreamReader wrapper, because we have to assume that the encapsulated data we’re dealing with is not XML.

The OnGetBodyReaderAtBodyContents() and the OnWriteBodyContents() overrides are consequently very simple:

        /// <summary>
        /// Called when the client requests a reader for the body contents.
        /// </summary>
        /// <returns></returns>
      protected override XmlDictionaryReader OnGetReaderAtBodyContents()
      {
         XmlDictionaryReader reader = CreateBodyReader();
         reader.MoveToContent();
         return reader;
      }

        /// <summary>
        /// Called when the client requests to write the body contents.
        /// </summary>
        /// <param name="writer">The writer.</param>
      protected override void OnWriteBodyContents(XmlDictionaryWriter writer)
      {
         XmlDictionaryReader reader = CreateBodyReader();
         writer.WriteNode(reader, false);
      }

What’s left to complete the message implementation are the compulsory overrides of the abstract properties of Message, for which we have backing fields declared at the top of the class:

        /// <summary>
        /// Gets the message version.
        /// </summary>
        /// <value>The message version.</value>
      public override MessageVersion Version
      {
         get
         {
            return MessageVersion.Soap11Addressing1;
         }
      }

        /// <summary>
        /// Gets the SOAP headers.
        /// </summary>
        /// <value>The headers.</value>
      public override MessageHeaders Headers
      {
         get
         {
            return headers;
         }
      }

        /// <summary>
        /// Gets the message properties.
        /// </summary>
        /// <value>The properties.</value>
      public override MessageProperties Properties
      {
         get
         {
            return properties;
         }
      }
    }
}

The PoxStreamedMessage is only different from this class insofar as that it doesn’t have the buffer management. The GetRawBodyStream() method immediately returns the encapsulated stream and the remaining implementation is largely equivalent, if not identical (yes, I should consolidate that into a base class). Therefore I am not pasting that class here as code but rather just append as a downloadable file, alongside the declaration of IPoxRawBodyMessage and the twice mentioned and not yet shown PoxEncoderMessageProperty class.

With this, we’ve got all the moving pieces we need to build what’s essentially becoming an Indigo-based, message-oriented web-server infrastructure with a REST-oriented programming model. What’s missing is how we get our encoder configured into a binding so that we can put it all together and run it.

Configuration is next; wait for part 8.

Download: PoxEncoderMessageProperty.zip
Download: PoxStreamedMessage.zip
Download: IPoxRawBodyMessage.zip

[2006-01-13: Updated PoxBufferedMessage code to deal with entity bodies that only consist of a UTF BOM]

Updated: