Part 1, Part 2, Part 3, Part 4, Part 5, Part 6, Part 7

We’ve got all the moving pieces and what’s left is a way to configure the PoxEncoder into bindings and use those to hook a service up to the network and run it.

Bindings? Well, all Indigo (WCF) services need to know their ABC to function. ABC? I’ll quote from my WCF Introduction on MSDN:

A binding is a layered combination of a transport, an encoder and of any additional protocol channels (reliable session, transaction flow, etc.) that you’d like to assemble into a transport stack for exposing a service implementation on a specific endpoint address.

For exposing a HTTP-based RESTish service we need:

A.    The HTTP address to host the service at.

B.     Some binding configuration that tells Indigo’s HTTP transport to use our PoxEncoder.

C.     An implementation of a service-contract that’s configured (using the HttpMethodOperationSelectorSection config extension) or marked-up (with the HttpMethodOperationSelectorAttribute) to use our HttpMethodOperationSelectorBehavior for endpoint address filtering and selecting methods.

I’ve shown you quite a few contract variants in the first parts of this series and therefore I don’t really have to explain the C in too much detail anymore; except: While the A is just a plain HTTP address such as http://www.example.com/service, it’s interesting insofar as this address is, unlike as with SOAP services, really just the common address prefix for the dispatch URIs of the particular service and that there is a split between what is A and what is C.

As I’ve explained, the philosophy behind the contract design of my extensions is around the namespaces that are the basis for forming URIs. Because REST services aren’t simply using HTTP as a transport tunnel as SOAP services do, but rather leverage HTTP as the application protocol it is, the URI is a lot more than just a drop-off point for messages. With REST services, the URI is an expression that has both, addressing (transport) and contract (dispatch) aspects to it and we need to separate those out. A clear distinction between global and local namespaces allows us to do that (And I am going into a bit more detail than I usually would, to further address an objection of Mark Baker, 1^st comment, on my choice of the programming model):

There is a global namespace that’s managed by the global DNS system of which anyone can reserve a chunk for themselves by registering a domain-name. The domain-name provides a self-manageable namespace root of which sub-namespaces (subdomains) can be derived and allocated to specific hosts/services or groups of hosts/services by the domain owners. On the particular host, you can put an application behind a specific port, which might either the default port of your particular application protocol or – diverging from the protocol standard – some other port of your choosing. Each Internet application deployment has therefore at least one unique mapping into this global namespace system.

Any further segmentation of the namespace except the host-name and the port-number are private matters of the application listing to that endpoint. With Indigo self-hosted HTTP services, the listening application is Windows (!) – more precisely it’s the HTTP.SYS kernel listener. For IIS/WAS hosted services, the listening application is IIS (for IIS 5.1 and below) or, again, Windows – through that very listener. At the HTTP.SYS listener, handler processes can register their interest in requests sent to certain sub-namespaces of the global namespace mapping (host/port), which are identified by relative URIs. To be clear: The HTTP.SYS API indeed requires the caller to provide an absolute URI like http://hostname:port/service, but the two main parts (scheme/host/port and path) of that URI are used for different purposes:

·         Global mapping: The hostname and port are used to establish a new listener on that particular port (if there is already a listener it is shared) and to populate the hostname-filter table that’s used to disambiguate requests by the Host header in case the IP address is mapped to multiple DNS host entries.

·         Local mapping: The path information of the URI (the remaining relative URI with scheme, hostname and port stripped) is used as a prefix-matching expression to figure out which handler process shall receive the request and, inside that handler process, to further identify and invoke the appropriate endpoint and handler that deals with the resource that the complete URI path represents.

As indicated, mapping URIs to an endpoint needs to distinguish between how we segment and map a local namespace and how we hook that into the global namespace. Hence, the root for any absolute URIs that’s establishing the mapping into the global namespace shall always be separate from the code and reside in configuration for reasons of flexibility as Mark was rightfully pointing out in his objection, while the shape and mapping of the local namespace is typically very application and use-case specific and might well be partially or entirely hardwired.

·         The Indigo A(ddress) of a REST service implemented with my particular programming model is used to hook a given service (or resource representation manager, if you like) into the global namespace: http://myservice.example.com/. Only to be pragmatic and to allow multiple such services to locally share a particular hostname and port and indeed only as an alternative and workaround to creating a separate DNS entry for each service, that mapping might include a path prefix allowing the local low-level infrastructure to demultiplex requests sent to the same global namespace mapping: http://myservices.example.com/serviceA and http://myservices.example.com/serviceB.

·         The Indigo C(ontract) of a REST service implemented with my particular programming model is used to define the shape of the local namespace that the service owns and which is used to provide access to the representations of the resource-types the service is responsible for.

The following configuration snippet for a simple web-server based on my extensions is illustrating that split:

The address http://localhost:8020/ is how I map the service into the global addressing namespace. The local namespace shape for that particular service is a defined by the layout of the file-system directory from which the service grabs files and returns them. What? You can’t see the directory structure and the resulting URLs from the above mapping? Of course not. It’s a private matter of the service implementation what that the local namespace structure is and it’s up to me what parts I am exposing. If I am nice enough I will give you something on a GET/HEAD request on the root of my local namespace (= global address without any suffix), and if I am not nice you get a 404 and will just have to know what to ask for. The “will have to know” part is contract. It’s an assurance that if you come looking at a particular place in my namespace you will have access to a particular thing. My [HttpMethod] attributes manifest that assurance on Indigo contracts.

That leaves B. Before I got carried away by A and C, I wrote [now a bit annotated] “A binding is a layered combination of a transport, an encoder and of any additional protocol channels (reliable session, transaction flow, etc.) that you’d like to assemble into a transport stack for exposing a service implementation [C] on a specific endpoint address [A].”

Putting together such a binding is not much more work than putting a little text between angle brackets and quotation marks in config as shown in the following snippet:

I am building a custom binding that’s combining the HTTP transport with a custom binding element config extension I built for the PoxEncoder. It’s that simple. And adding the binding element extension does not require black magic, either. It’s just another XML snippet that maps the extension class to an element name (“poxEncoder”) as you can see in the extensions section of the complete config file:

The PoxEncoderBindingExtension is a class that is based on System.ServiceModel.Configuration.BindingElementExtensionSection. Whenever the configuration is processed by Indigo, the presence of the “poxEncoder” element in a binding triggers the creation of an instance of the class and if we’d require any configuration attributes (which we don’t), those would be stuffed into the Properties collection.    

Once the configuration information has been read, the extension is asked to create a BindingElement from the acquired information. So these extensions are really just factories for binding elements. The binding element, which can also be used to compose such a binding in code by explicitly adding it to a System.ServiceModel.CustomBinding is shown below:

Binding elements are typically used to put together client-side (channel factory) or service-side (channel listener) transport stacks. At the bottom is the transport and layered on top of it are security, reliable sessions, transaction flow and all other protocol features you need. Each protocol or feature on the channel/listener level has its own binding element and using those you configure yourself a binding combining the features you need and in the order that they should be applied.

The binding elements for message encoders are a bit different, because they are not contributing their own channel factories or channel listeners into the stack, but rather “only” supply the message encoder for the configured transport.

Whenever a binding is instantiated, Indigo creates a ChannelBuildContext which contains the sequence of the binding elements that shall be stacked onto each other into a channel or listener stack and starts stacking them from top to bottom by invoking the topmost binding element’s BuildChannelListener or BuildChannelFactory method. Once the binding element is done creating its channel factory or channel listener, it invokes BuildInnerChannel[Listener/Factory] on the context to have the binding element underneath do its work. (The context is also used to validate whether combination of the elements yields a functional binding stack, but I won’t go into that here).

Our binding element, however, won’t create a channel factory or listener, but rather put itself into the UnhandledBindingElements collection on the build context and will then just have the context complete the construction work. With putting itself into that collection, the binding element makes itself and its most irresistible feature (you’d also think that if you were an Indigo transport) – the IMessageEncodingBindingElement implementation – visible to the transport and waves its hand that it wants to be used. The transport’s binding element, which is at the bottom of the stack and therefore asked to build its channel factory/listener after our binding element has been invoked, will go look in the UnhandledBindingElements  collection whether a message encoding binding element is advertising itself for use. And if that’s so it will forget all of its defaults and happily embrace and use an encoder created by the factory returned by IMessageEncodingBindingElement.CreateMessageEncoderFactory, which is, in our case, this rather simple class:

Soooooooo….!

If you had actually copied all those classes from Parts 1-8 down into local files and compiled them into an assembly, you’d have all my REST/POX plumbing code by now (except, admittedly, an application-level utility class that helps putting messages together).

But wait … don’t do that. In the next part(s) I’ll give you the code all packed up and ready to compile along with the little web server that we’ve configured here and will also share some code snippets from my TV app … maybe the RSS and ASX pieces?

Part 1, Part 2, Part 3, Part 4, Part 5, Part 6, Part 7, Part 8, Part 9

Where are we?

·         In Parts 1 and 2, I explained contracts in the REST/POX context and the dispatch mechanisms that we need to enable Indigo to accept and handle REST/POX requests. With that I introduced a metadata extension, the [HttpMethod] attribute that can be used to mark up operations on an Indigo contract with HTTP methods and a URI suffixes that we can dispatch on. I also showed how we can employ a parameter inspector to extract URI-embedded arguments and flow them to the operation in a message property.

·         In Parts 3 and 4, I showed how we use the [HttpMethodOperationSelector] attribute to replace Indigo’s default address filtering and operation selection mechanisms, basically the entire message dispatch mechanism, with our own variants. The SuffixFilter is used to find the appropriate endpoint for an incoming request and the HttpMethodOperationSelectorBehavior  find the operation (method) on that endpoint which shall receive the incoming request message.

·         In Parts 5 and 6, you saw how the PoxEncoder puts outbound envelope-less POX documents onto the wire in its WriteMessage methods and accepts incoming non-SOAP XML requests through its ReadMessage methods and wraps them with an in-memory envelope (“message”) for further processing. I also showed the PoxBase64XmlStreamReader, which is an XML Infoset wrapper for arbitrary binary streams that interacts with the PoxEncoder to allow smuggling any sort of raw binary content through the Indigo infrastructure and onto the wire.

We’re pretty far along already. We’ve got the dispatch mechanisms, we know how to hook the dispatch metadata into the services, we’ve got the wire-encoding – we have most of the core pieces together. In fact, the last two key classes we’re missing (configuration hooks aside) are two specialized message classes that we need to handle incoming requests. In Part 6, you could see that the two ReadMessage overloads of the PoxEncoder delegate all work to the PoxBufferedMessage for the “buffered” transfer-mode overload and to PoxStreamedMessage for the “streamed” transfer mode overload.

ReadMessage is called on an encoder whenever a transport has received a complete message buffer (buffered mode) or has accepted and opened a network stream (streamed mode).

Using streamed mode means very concretely that Indigo will start handling the message even though the message might not have completely arrived. A transport in streaming mode will only do as much as it needs to do in order to deal with the transport-level framing protocol. I use “framing protocol” as a general term for what is done at the transport level to know what the nature of the payload is and where the payload starts and ends. For HTTP, the HTTP transport figures out whether an incoming request is indeed an HTTP request, will read/parse the HTTP headers, and will then layer a stream over the request’s content, irrespective of whether the transfer of that byte sequence has already been completed. This stream is immediately handed off to the rest of the Indigo infrastructure and the transport has done its work by doing so.

Pulling the remaining bytes from that stream is someone else’s responsibility in streamed mode. Whenever a piece of the infrastructure pulls data directly or indirectly from the stream and the data chunk requested is still in transfer, the stream will block and wait until the data is there. The transport’s handling of the framing protocol will typically also take care of chunking and thus make a chunked stream appear to be continuous. When I say “indirect pull” I mean that it may very well be an XmlDictionaryReader layered over an XmlReader layered over the incoming network stream.

The streaming mode is of particular interest for very large messages that may, in an extreme case, be virtually limitless in size. There’s no specification that says that you cannot stick 500 Terabyte or 500 Exabyte worth of data (think 365x24 live 1080i video streams) into a single message. As long as you have some reason to believe that the sender will eventually, in 20 years from now, give you “</soap:Body></soap:Envelope>” to terminate the message, the message can be assumed to be well-formed and complete.

No matter whether you use buffered or streamed mode, the configured encoder’s ReadMessage method is the first place where the read data chunk or the stream goes and that delegates, as shown to our two message classes. So let’s look at them.

We’ll primarily look at the PoxBufferedMessage, which is constructed over the read message buffer in the PoxEncoder like this:

The class PoxBufferedMessage is derived from the abstract System.ServiceModel.Message class and implements the base-class’s abstract properties Headers, Properties, and Version and overrides the OnClose(), OnGetReaderAtBodyContents(), and OnWriteBodyContents() virtual methods. Internally, Indigo has several such Message implementations that are each customized for certain scenarios. Implementing own variants of Message is simply another extensibility mechanism that Indigo gives us.

We can construct instances of the class over a raw byte array or an “array segment” layered over such an array. Array segments are preferred over raw arrays, because their use eases memory management. You can keep a pool of buffers with a common size, even though the actual content is shorter than the buffer size and probably even offset from the lower buffer boundary. If we get a raw byte array we simply adopt it, but if we get an array segment alongside a reference to a buffer manager we take a new buffer from the buffer manager and copy the array segment to that acquired buffer.

When we close the message and we have acquired it using the buffer manager (which is signaled by the presence of the reference) we duly return it once the message is being closed (or disposed or finalized).

The next two methods are an implementation of the IPoxRawBodyMessage interface that is, you guessed it, defined in my extensions. If the handler method wants to get straight at the raw body content knowing that it doesn’t expect XML, it can shortcut by the whole XmlReader and XML serialization story by asking for the BodyContentType and pull out the raw body data as a stream layered over the buffer:

There is a bit of caution required using this mechanism, though. Because the message State (Created, Written, Read, Copied, Closed) is controlled by the base-class and cannot be set by derived classes, the message should be considered to be in the State==MessageState.Read after calling the GetRawBodyStream() method. That doesn’t seem to be necessary because we have a buffer here, but for the streamed variant that’s a must. And for the sake of consistency we introduce this constraint here.

The BodyContentType property implementation seems, admittedly, a bit strange at first sight. Even though you won’t see the message properties being populated anywhere inside this class, we’re asking for them and base the content-type detection on their values. That only makes sense when we consider the way messages are being populated by Indigo. As I explained, the first thing that gets called once the transport has a raw data chunk or stream in its hands that it believes to be a message, it invokes the encoder. For incoming requests/messages, the encoder is really serving as the message factory constructing Message-derived instances over raw data. Once the encoder has constructed the message in one of the ReadMessage overloads, the message is returned to the transport. If the transport wants, it can then (and the HTTP transport does) stick properties into that newly created message and then hand it off to the rest of the channel infrastructure for processing and dispatching. Because these extensions are built for REST/POX and therefore have HTTP affinity, that’s precisely what we assume to be happening for the BodyContentType property and the CreateBodyReader() method below. As I already explained in Part 1, the HTTP transport will always add a HttpRequestMessageProperty  to the message and that’s consequently from which we can grab the content-type of the incoming request data.

The private CreateBodyReader() method that constructs XML readers for the both, the OnGetBodyReaderAtBodyContents() and the OnWriteBodyContents() overrides shown below, uses the same strategy to figure out the content-type of the message and therefore to guess what’s hidden inside the byte-array (or array segment) the message was constructed over. To make the message class useful for the request and response direction, we’ll distinguish there two separate cases here:

·         If the message is a response, the handling method in the user code might have indicated that it wants the encoder to serialize the message onto the wire in “raw binary” mode. The indicator for that is the presence of the PoxEncoderMessageProperty having the RawBinary property set to true. If that is the case, the reader we return is always our PoxBase64XmlStreamReader. The property cannot occur in request messages because the Indigo transports simply don’t know about it.

·         If the message is a request or a response with the mentioned property missing, we will try figuring out the message’s content-type using the described strategy of using the HTTP transport’s message properties. If we can’t figure out a content-type for a response (it’s optional for the responding handler code to supply it), we will assume that the content-type is “text/xml”. If the message is a request we can rely of getting a content-type as long as the underlying transport is Indigo’s HTTP transport implementation. If the content-type is indeed “text/xml” we construct an XmlTextReader over the raw data and return it. If the content-type is anything else, we use our PoxBase64XmlStreamReader wrapper, because we have to assume that the encapsulated data we’re dealing with is not XML.

The OnGetBodyReaderAtBodyContents() and the OnWriteBodyContents() overrides are consequently very simple:

What’s left to complete the message implementation are the compulsory overrides of the abstract properties of Message, for which we have backing fields declared at the top of the class:

The PoxStreamedMessage is only different from this class insofar as that it doesn’t have the buffer management. The GetRawBodyStream() method immediately returns the encapsulated stream and the remaining implementation is largely equivalent, if not identical (yes, I should consolidate that into a base class). Therefore I am not pasting that class here as code but rather just append as a downloadable file, alongside the declaration of IPoxRawBodyMessage and the twice mentioned and not yet shown PoxEncoderMessageProperty class.

With this, we’ve got all the moving pieces we need to build what’s essentially becoming an Indigo-based, message-oriented web-server infrastructure with a REST-oriented programming model. What’s missing is how we get our encoder configured into a binding so that we can put it all together and run it.

Configuration is next; wait for part 8.

I recently needed a TCP port-forwarder that sits on a socket connection and monitors it. My concrete use-case is that I need to front the backend-server of my TV application with such a port forwarder in order to create live-TV streaming sessions as soon as a client requests them and also tears them down shortly after the client disconnects so that the session doesn’t need to time out and blocks the tuner until then. The backend also requires that I do a periodical “keep-alive” ping every 30-40 seconds, which isn’t a very practical requirement for some of my client scenarios. Therefore, I needed, generally speaking, something that would sit between the client and the backend server, monitors the data stream and would let me run some code (set up the live session and start the keep-alive timer) when I get a new client connection and just before I connect through to the target and which would let me run some code (shut down the session and stop the keep-alive) as soon as the connection is torn down.

Since, I didn’t find one (or was too blind or too lazy, you know how that goes), I wrote one. It’s a fully asynchronous TcpListener/TcpClient based implementation, it’s fast and stable enough for my purposes and it might or might not be for yours, it has a bit of tolerance for targets that don’t accept a connection on the first try, and you can hook up events to “before target connect” and “after target disconnect”. Since all the bytes fly by, you can instrument the thing further or monitor the stream as you like.  

The code is pretty straightforward, even though the asynchronous calls/callbacks admittedly make the execution paths in the implementation a bit challenging to follow, and should not require much further explanation. You construct an instance of TcpPortForwader passing the local port and the target port and host to forward to, call Start() and the listener starts listening. Stop() stops the listener. You can call Start() from any thread; the listener will implicitly use thread-pool threads to run on its own. Hook up the events and they are being raised. Simple enough. Download below.