In the upcoming week I will be cleaning up the categories on my blog. So if you've subscribed to any of the category feeds directly, some of them may stop working.

As of tomorrow, the "official" blog address will be http://friends.newtelligence.net/clemensv as I won't be part of the newtelligence "staff" any longer. However, all existing subscriptions to the main RSS and Atom feeds will continue to work, because this blog isn't really going anywhere. I will see what it takes to get a mirror blog on blogs.msdn.comand whether I can use the dasBlog cross-blogging feature to push select content there.

Eventually, and once I am done reshuffling, I will explain the new category structure and actually encourage you to subscribe to select categories, because my blogging habits will change quite a bit in my new role at Microsoft. For instance, you'll see a lot more German here (which you might want to tune out of if you are not interested), I'll start a link-blog category and I will introduce some other categories that are not at all about technology.

Categories: Blog

January 31, 2006
@ 12:23 AM

Get the Goods     (Yes, that page will get a cleanup once I've got the docs done)

Categories: newtellivision

There are roughly 24 hours left before I will switch jobs and become a "Microsoftie". My parting gift to the company I co-founded is this:

newtellivision is a framework and application for accessing live streaming television remotely. The use-case I had in mind building this is quite simple and is closely related to my current (and the new) job and the perspective of moving to the U.S. some time later this year: I want access to my local, German TV channels whenever I am traveling and I also want access to those channels when I've moved to Seattle. That's most important for two types of programs: sports and news. I care a lot more about Bundesliga football than for Major League baseball.

Consequently, newtellivision is focused on one thing: streaming video. And it comes with two user experiences that are a good choice for the above use-cases:

Windows Media Player

The Windows Media Player (WMP) experience (for WMP 9 and above), is the "television to go" version. It works from any Windows PC that has Windows Media Player 9/10 installed and that can connect via any network to a newtellivision server.  There is no additional software that needs to be installed on the client. All interaction with the newtellivision server is hosted inside Media Player.

Features of the Windows Media Player experience in a nutshell:

  • Access to live streaming television, your video library, live web streams and video blogs from single user interface.
  • "Speed bar" across the top of the side panel allows manual adjustment of the backend streaming encoder settings to the available bandwidth with a single click.
  • Channels panel provides hover-over tooltips showing guide data for all channels for which guide data or content lists are available.
  • Guide data for current channel allow single-click scheduling of recordings if the newtelligence plug-in provider of the selected channel supports recording.
  • Direct playback access to shows that were recorded on the current channel.

     

  •  

     

     

     

    Windows XP Media Center Edition

    The Windows XP Media Center Edition (MCE) experience is built for the "home TV at home" experience. If you move to another country and your parents or your friends host the newtellivision server for you, you can access live streamed TV programs from far-far away and your recordings right within the usual MCE experience with your remote control. And from the very same experience you have access to video blogs and web streams such as DW TV, NASA TV and thousands of other live streaming TV sources, complete with support for XMLTV guide data.

    Features of the Windows XP Media Center experience in a nutshell:

  • Remote control navigable, dynamic schedule grid for scheduling data modeled after the familiar Windows Media Center "Live TV" experience. Of course, the grid adjusts automatically as time progresses and shows you whether shows are scheduled for recording.
  • Detail information about individual upcoming shows including one-click scheduling of recordings.
  • Channel detail view with one click access to video blog content (the example on the right shows Microsoft's Channel 9) and recorded shows.

    For the Windows Media Center experience you need a Windows Media Center PC as client.

     

  •  

    The newtellivision Server and Framework

    The newtellivision Server is built on the Windows Communication Foundation (WCF). The server can run either as an interactive application or as a Windows Service and the configuration wizard makes it very easy to switch between these modes. The server is extensible and comes with two plug-in providers that utilize the newtellivision Framework to provide access to live streaming video.

  • The "Beyond TV" provider encapsulates the SnapStream Media Beyond TV server engine and provides access to the recorder features, live TV streaming, the Beyond TV library and the electronic program guide (EPG).
  • The "Web Streams" provider provides access to public web streams and video blogs and has direct support for integrating EPG information in XMLTV format.

    The newtellivision Framework and the Server application are available in source code for experimentation, improvement and, of course, for customizing the existing providers or for writing entirely new providers.

    The server renders the list of available channels in OPML format and the electronic program guide data for each channel in RSS 2.0 format, each with the minimal set of extensions required to add vital information that is otherwise not representable in the core standard. Video playlists are rendered in Microsoft ASX 3.0 media metadata format.

  •  

     

    Requirements

    For running newtellivision you need the following minimum server configuration:

  • Windows XP or Windows Server 2003 with the .NET Framework 2.0 and the WinFX 3.0 Runtime Components (Beta 2 CTP)

    Furthermore, for the web stream provider:

  • An internet connection with sufficient bandwidth for viewing live video streams
  • Optionally, XMLTV for downloading guide data.
  • Optionally, a source for picking web stream links.

    Furthermore, for the Beyond TV provider:

  • Beyond TV 4 by SnapStream Media (The demo version should work -- until expiration of course)
  • A Beyond TV compatible software encoder card for Windows Media streaming.
  • Beyond TV may run on a separate machine from the newtellivision Server machine. For this setup you MUST have a valid Beyond TV Link license (demo is ok while it lasts).
  • For accessing streams remotely and even though this is not enforced, you need a valid Beyond TV Link license as a CAL for each machine concurrently accessing the web streams.
  • Bandwidth, bandwidth, bandwidth -- upstream. I have 3000/512 DSL connection and I would still like it a bit quicker. It turns out that a sustained bitrate of ~400KBps yields a video quality that is good enough to watch sports remotely, but don't expect brilliant detail resolution. ~128KBps is plenty to watch news.

    The Catch I

    newtellivision is released in binary and source code form for any non-commercial purpose. In short, this is a "extend it for fun" application, but you can't derive any direct commercial gain from the application or any of its parts or from the operation of the application. If you'd like to use the application commercially, you must contact newtelligence to learn about your options. Commercial use expressly does not include teaching, speaking at conferences, writing books, etc.

    The Catch II

    This application is big, runs on beta software and is tested in only one or two server configurations to speak of. I have no idea whether anybody but me will get it to run. I've built it for myself only. It might burn your PC into a small pile of ash. I don't know what will happen. The app looks pretty polished, but evil lurks within. Consider it "alpha quality".

    The Catch III

    The server is, at this point, designed(!) not to scale to more than a single concurrent user. You can theoretically consume the streams from two or three clients concurrently, but there is only support for a single tuner and hence everyone has to watch the same live show. If someone switches the channel, all others view the channel too. The WMP experience is, in this case, nice enough to adjust the schedule panel to reflect the currently watched channel.

    Future Direction

    This is a demo app. I've written it just as that and for my personal use purposes. The reason for this public drop at this time is to avoid any troubles between me, newtelligence and Microsoft as I am making the transition between the firms. The license allows me to continue working on it privately, while all the commercialization rights up to this point rest with newtelligence. That said, I am not out of ideas. If you love writing (WCF) applications or are interested in the whole streaming TV story and you can live with the non-commercial constraint, I'd love to get you on board as an active contributor to the cause. Concretely, I'd be interested in investigating whether a provider for PPLive is feasible, whether the newtellivision server can tap the MCE streaming capabilities and whether a standalone provider can be built. See it as my next thing after DasBlog. It'll be a killer community project.

    Availability

    The code and the installable binaries sit here. Get them.

    Meanwhile you might want to get (or go dig out) an old, used WinTV PCI card (those go on eBay for 10-15 US$)  or equivalent (see SnapStream's site analog/without hardware encoding). One of my test configurations used a 1996 (!) WinTV PCI and it worked. Now my streaming source is a slightly newer Hauppauge WinTV card from 1999 that I bought on eBay.de for a whopping €10.70 a few months ago. Once you have that, it's useful to get Beyond TV configured and up and running for web streaming with the Beyond TV Web Admin interface (accessible with the "Live TV" link on the "Tasks" pane on the left of the web admin tool; http://localhost:8129 once you have installed Beyond TV). Once you get video there, you are good to go.

  • Categories: newtellivision

    January 23, 2006
    @ 09:24 PM

    In the 5 years I've been "newtelligent" I have never been *really* late to an event. Not a single time. Admittedly,

    • I've been an hour late to a talk in Kuala Lumpur because the schedule I received was wrong,
    • I arrived in Vienna "just in time" and still intoxicated after missing the planned 05:20 (AM!) connection from Istanbul after a serious party night and being able to rebook to a later flight after rushing to the airport and still surrounded by a complete haze and only got to the venue about 5 minutes too late because the taxi driver didn't know where to go (I can't remember any content of the first session that day, but the attendees loved it),
    • I got stuck in Heathrow one night due to bad weather on a connection to Dublin, but with an early enough connection to make it to the event in the morning.

    And now ... on the very last event I am doing for newtelligence I got stranded in a hotel room at the Munich Airport and can't get to Istanbul for a workshop that's supposed to start at 9:30AM tomorrow. Istanbul is snowed in. Next opportunity to get out there is at 11AM tomorrow. Hope that works.

    [Update: It did work, even though with an hour delay. In Istanbul, the chaos continued. 15-25 cm of snow per day in a city where "winter tires" are a rather unknown phenomenon are a bit problematic. I love Istanbul, but I am very happy to be back home this time.]

    Categories: Travel

    January 18, 2006
    @ 07:28 PM
    Categories: newtellivision

    January 14, 2006
    @ 04:11 PM

    private void exitMenuItem Click(object sender, EventArgs e)
    {
        if (runtime != null && runtime.Running)
        {
            ThreadPool.QueueUserWorkItem(delegate(object obj)
            {

                runtime.Stop();
                Invoke(new ThreadStart(delegate()
                {

                    this.Close();
                }));
            });
        }
        else
        {
            this.Close();
        }
               
    }

    Just caught myself coding this up. The shown method is a Windows Forms event handler for a menu item. The task at hand is to check a local variable and if that’s set to a specific value, to switch to a different thread and perform an action (that particular job can’t be done on the Windows Forms STA thread), and to close the form back on the main STA thread once that’s done. I colored the executing threads; yellow is the Windows Forms STA thread, blue is an arbitrary pool thread. Works brilliantly. Sick, eh?

     

    [Mind that I am using ThreadStart just because it’s a convenient void target(void) delegate]

    Categories: Technology

    January 10, 2006
    @ 08:14 PM
    Categories: Other Stuff

    January 10, 2006
    @ 12:26 PM

    Everyone seems to be giving some tech predictions for 2006 on their blog. Here's mine:

    The DRM opposition will start turning DRM against the music and film industry and embrace DRM to create a distributed file-sharing protocol and applications that implements "lending between friends" and ensures that within a group of users, only as many copies of digital works can be played concurrently as the number of original, legally acquired media that have been contributed into a pool. The protocol will ensure that the media integrity is preserved insofar that no two people can, say, play different songs from the same album concurrently unless there are enough copies of the same album in the pool. Media players supporting this protocol will eventually be clever enough to prioritize and shuffle playlists in a way that the fewest possible media are required in the pool. 

    Since "lending" will later be found to be still too problematic from a legal standpoint, the physical media constituting the media pool will be put into a network of escrow services and acquiring a temporary DRM license to play a particular music album or video will automatically result in a "one-cent" sale transaction and transfer of ownership rights of the physical media for the period while the license is valid and therefore result in a legal digital copy of an legally owned media to be played.

    In short: It'll remain interesting how the whole DRM situation develops. To me it seems consequential that the content consumer side will take a page out of the content provider side's book and use the same technology arsenal to try to achieve exactly opposite goals. I think there's a resonable chance that DRM will at least backfire for all digital media that's already published and out there. And that's quite a lot.

    My stance on what's appropriate and inappropriate with respect to DRM is "undecided", even though I deplore Sony's rootkit DRM trickery. At the same time there must be a reasonable business model for entertainment media. Someone just has to find one that is sustainable and works. Luckily not my job.

    Categories: Other Stuff

    Part 1, Part 2, Part 3, Part 4, Part 5, Part 6, Part 7, Part 8
    If you are in a hurry and all you want is the code, scroll down to the bottom. ;)
    And for the few who are still reading: This is the 9th and final part of this series, which turned out to be a little bigger than planned. And if you read all parts up to here, you have meanwhile figured out that the extensions that I have presented here are not only about REST and POX but primarily a demonstration of how customizable Indigo is for your own needs.  Indigo – Indigo was really a cool code-name. Just like many folks on the Indigo WCF team it’s a bit difficult for me to trade it for a clunky moniker like WCF, and it seems that this somewhat reflects public opinion. Much less am I inclined to really spell out “Windows Communication Foundation” in presentations, because that doesn’t exactly roll off the tongue like a poem, does it? But, hey, there’s always the namespace name and that doesn’t suck. “Service Model” is what everyone will be using in code. We don’t need three-letter acronyms, or do we?
    But I digress. I’ve explained a complete set of extensions that outfit the service model with the ability to receive and respond to HTTP requests with arbitrary payloads and without SOAP envelopes and do so by dispatching to request handlers (method) by matching the request URI and the HTTP method against metadata that we stick on the methods using attributes.
    And now I’ll show you the “Hello World!” for the extensions and about the simplest thing I can think of is a web server ;-)  In fact, you already have the complete configuration file for this sample; I showed you that in Part 8.
    Since things like “Hello World!” are supposed to be simple and it’s ok not to make that a full coverage test case, we start with the following, very plain contract:

    [ServiceContract, HttpMethodOperationSelector]
    interface IMyWebServer
    {
        [OperationContract, HttpMethod("GET", UriSuffix = "/*")]
        Message Get(Message msg);       
        [OperationContract(Action = "*")]
        Message UnknownMessage(Message msg);
    }

    By now that should not need much explanation. The Get() method receives all HTTP GET requests on the URI suffix “/*” whereby “*” is a wildcard. In other words, all GET requests go to that method.
    The implementation of the service is pretty simple.  I am implementing it as a singleton service that is constructed passing a directory name (path).
    The Get() implementation gets the URI of the incoming request from the To header of the message’s Headers collection. (Note that the HTTP transport maps the incoming request’s absolute URI to that header once the encoder has constructed the message.)  

    [ServiceBehavior(InstanceContextMode=InstanceContextMode.Single)]
    public class MyWebServer : IMyWebServer
    {
        string directory;

        public MyWebServer(string directory)
        {
            this.directory = directory;
        }

        public Message Get(Message msg)
        {
            string requestPath = msg.Headers.To.AbsolutePath;
            // get the path
            if (requestPath.Length > 0 && requestPath[0] == '/')
            {
                // if the path is just the "/", append "default.htm" as the
                // default page.
                if (requestPath.Substring(1).Length == 0)
                {
                    requestPath = "/default.htm";
                }
                // otherwise check whether a file by the requested name exists
                string filePath = Path.Combine(directory, requestPath.Substring(1).Replace('/', '\\'));
                if (File.Exists(filePath))
                {
                    // and return a file message
                    return PoxMessages.CreateFileReplyMessage(filePath, PoxMessages.ReplyOptions.None);
                }
            }
            // if all fails, send a 404
            return PoxMessages.CreateErrorMessage(HttpStatusCode.NotFound);
        }

       
        public Message UnknownMessage(Message msg)
        {
            return PoxMessages.CreateErrorMessage(HttpStatusCode.NotImplemented);
        }
    }

    Once we’ve done a few checks on the URIs path portion, we construct a file path from the base directory and the URI path, check whether such a file exists, and if it does we create a message for that file and return it. If we can’t find the resulting file name, we construct a 404 “not found” error message. The UnknownMessage() method receives all requests with HTTP methods other than GET and appropriately returns a 501 “not implemented” message. Web server done.
    Well, ok. The actual messages are constructed in a helper class PoxMessages that aids in constructing the most common reply messages. The class is part of the extension assembly code you can download and therefore I just quote the relevant methods that are used above. We’ll start with PoxMessages.CreateErrorMessage(), because that its very simple:

    public static Message CreateErrorMessage(System.Net.HttpStatusCode code)
    {
       Message reply = Message.CreateMessage("urn:reply");
       HttpResponseMessageProperty responseProperty = new HttpResponseMessageProperty();
       responseProperty.StatusCode = code;
       reply.Properties.Add(HttpResponseMessageProperty.Name, responseProperty);
       return reply;
    }

    We create a plain, empty service model Message, create an HttpResponseMessageProperty instance, set the StatusCode to the status code we want and add the property to the message. Return, done.
    The PoxMessages.CreateFileReplyMessage() method is a bit more complex, because it, well, involves opening files. I am not showing you the exact overload that’s used in the above example but the one that’s being delegated to:  

    public static Message CreateFileReplyMessage(string fileName, long rangeOffset, long rangeLength, ReplyOptions options)
    {
       string contentType = GetContentTypeFromFileName(fileName);
       try
       {
          FileStream fileStream = new FileStream(fileName, FileMode.Open, FileAccess.Read, FileShare.Read);
          if (rangeOffset != -1 && rangeLength != -1)
          {
             SegmentStream segmentStream = new SegmentStream(fileStream, rangeOffset, rangeLength, true);
             return PoxMessages.CreateRawReplyMessage(segmentStream, contentType, rangeOffset, rangeLength, fileStream.Length, Path.GetFileName(fileName), options);
          }
          else
          {
             return PoxMessages.CreateRawReplyMessage(fileStream, contentType, Path.GetFileName(fileName), options);
          }
       }
       catch
       {
          return PoxMessages.CreateNotFoundMessage();
       }
    }

    The implementation will first make a guess for the file’s content-type based on the file-name, which is a simple registry lookup with a fallback to application/octet-stream.  Then it’ll try to open the file using a FileStream object. If that works –  ignoring the special case with rangeOffset/rangeLength being set – we delegate to the CreateRawReplyMessage() method:

    public static Message CreateRawReplyMessage(Stream stm, string contentType, long rangeOffset, long rangeLength, long totalLength, string streamName, ReplyOptions options)
    {
       PoxStreamedMessage reply = new PoxStreamedMessage(stm, 16384);
       HttpResponseMessageProperty responseProperty = new HttpResponseMessageProperty();
       if ((options & ReplyOptions.ContentDisposition) == ReplyOptions.ContentDisposition)
       {
          responseProperty.Headers.Add("Content-Disposition", String.Format("Content-Disposition: attachment; filename=\"{0}\"", streamName));
       }
        responseProperty.Headers.Add("Content-Type", contentType);
       if (rangeOffset != -1 && rangeLength != -1)
       {
          responseProperty.StatusCode = System.Net.HttpStatusCode.PartialContent;
          responseProperty.Headers.Add("Content-Range", String.Format("bytes {0}-{1}/{2}",rangeOffset,rangeOffset+rangeLength,totalLength));
          responseProperty.Headers.Add("Content-Length", rangeLength.ToString());
       }
       else
       {
            if ((options & ReplyOptions.AcceptRange) == ReplyOptions.AcceptRange)
            {
                responseProperty.Headers.Add("Content-Range", String.Format("bytes {0}-{1}/{2}", 0, totalLength-1, totalLength));
            }
            responseProperty.Headers.Add("Content-Length", totalLength.ToString());
       }
       if ((options & ReplyOptions.NoCache) == ReplyOptions.NoCache)
       {
          responseProperty.Headers.Add("Cache-Control", "no-cache");
          responseProperty.Headers.Add("Expires", "-1");
       }
       if ((options & ReplyOptions.AcceptRange) == ReplyOptions.AcceptRange)
       {
          responseProperty.Headers.Add("Accept-Ranges", "bytes");
       }
        reply.Properties.Add(HttpResponseMessageProperty.Name, responseProperty);
       reply.Properties.Add(PoxEncoderMessageProperty.Name, new PoxEncoderMessageProperty(true));
       return reply;
    }

    That method takes the stream, wraps it in our PoxStreamedMessage, sets all desired HTTP headers on the HttpResponseMessageProperty, adds the property to the message and lastly adds the PoxEncoderMessageProperty indicating that we want the encoder to operate in raw binary mode. However, all these helper methods are already part of the library and therefore the application code doesn’t really have to deal with all of that anymore. You just construct the fitting message, stick the content into it and return it.
    So now we have a service class and what’s left to do is to host it. For that we need a simple service host with a tiny little twist. Since the ServiceMetadataBehavior that typically gives you the WSDL file and the service information page would conflict with our direct interaction with HTTP, we need to switch it off. We do that by removing it from the list of behaviors before the service is initialized.

    public class MyWebServerHost : ServiceHost
    {
        public MyWebServerHost(object instance)
            : base(instance)
        {
        }

        protected override void OnInitialize()
        {
            Description.Behaviors.Remove<ServiceMetadataBehavior>();
            base.OnInitialize();
        }
    }


    class Program
    {
        static void Main(string[] args)
        {
            string directoryName = args[0];

            Console.WriteLine("LittleIndigoWebServer");
            if (args.Length == 0)
            {
                Console.WriteLine("Usage: LittleIndigoWebServer.exe [root path]");
                return;
            }
           
            if (!Directory.Exists(directoryName))
            {
                Console.WriteLine("Directory '{0}' does not exist");
                return;
            }

            Console.WriteLine("Web server starting.");

            using (MyWebServerHost host = new MyWebServerHost(new MyWebServer(directoryName)))
            {
                host.Open();
                Console.WriteLine("Web Server running. Press ENTER to quit.");
                Console.ReadLine();
                host.Close();
            }
        }
    }

    The rest is just normal business for hosting and setting up a service model service in a console application. We read the first argument from the command-line and assume that’s a directory name. We verify that that is indeed so, construct the service host passing the singleton new’ed up with the directory name. We open the service host and we have the web server listening. The details for how the service is exposed on the network is the job of configuration and binding and, again, exhaustively explained in Part 8.

    Below is the downloadable archive that contains two C# projects. Newtelligence.ServiceModelExtensions contains the extension set and LittleIndigoWebServer is the above demo app. The code complies and works with the WinFX November and December CTPs.

    If you have installed Visual Studio on drive C: you should be able to run the sample immediately with F5, since the LittleIndigoWebServer project’s debugging settings pass the .NET SDK directory to the application on startup. So if you have that, start the server, and then browse http://localhost:8020/StartHere.htm you get this:

    Otherwise, you can just start the server using any directory of your choosing, preferably one with HTML content.

    And that’s it. I am happy that I’ve got all of the stuff out. This is probably the most documentation I’ve ever written in one stretch for some public giveaway infrastructure, but I am sure it’s worth it. I will follow up with more examples using these extensions. For instance I will show to use this for actual POX apps (the web server is just spitting out raw data, after all) using RSS, OPML and ASX. Stay tuned.

    Oh, and … if you like this stuff I’d be happy about comments, questions, blog mentions and, first and foremost, public examples of other people using this stuff. License is BSD: Use as you like, risk is all yours, mention the creators. Enjoy.

    Download: newtelligence-WCFExtensions-20060901.zip

    [Note: I am preparing an update with client-side support and a few bugfixes right now. Should be available before or on 2006-01-16]

    Categories: Indigo

    Just in:

    Important Information for Thursday 5 January 2006

    Microsoft announced that it would release a security update to help protect customers from exploitations of a vulnerability in the Windows Meta File (WMF) area of code in the Windows operating system on Tuesday, January 2, 2006, in response to malicious and criminal attacks on computer users that were discovered last week.

    Microsoft will release the update today on Thursday, January 5, 2006, earlier than planned.

    http://windowsupdate.microsoft.com

    Categories:

    Part 1, Part 2, Part 3, Part 4, Part 5, Part 6, Part 7

    We’ve got all the moving pieces and what’s left is a way to configure the PoxEncoder into bindings and use those to hook a service up to the network and run it.

    Bindings? Well, all Indigo (WCF) services need to know their ABC to function. ABC? I’ll quote from my WCF Introduction on MSDN:

    “ABC” is the WCF mantra. “ABC” is the key to understanding how a WCF service endpoint is composed. Think Ernie, Bert, Cookie Monster or Big Bird. Remember "ABC".

    ·         "A" stands for Address: Where is the service?

    ·         "B" stands for Binding: How do I talk to the service?

    ·         "C" stands for Contract: What can the service do for me?

    Web services zealots who read Web Service Description Language (WSDL) descriptions at the breakfast table will easily recognize these three concepts as the three levels of abstraction expressed in WSDL. So if you live in a world full of angle brackets, you can look at it this way:

    ·         "A" stands for Address—as expressed in the wsdl:service section and links wsdl:binding to a concrete service endpoint address.

    ·         "B" stands for Binding—as expressed in the wsdl:binding section and binds a wsdl:portType contract description to a concrete transport, an envelope format and associated policies.

    ·         "C" stands for Contract—as expressed in the wsdl:portType, wsdl:message and wsdl:type sections and describes types, messages, message exchange patterns and operations.

    "ABC" means that writing (and configuring) a WCF service is always a three-step process:

    ·         You define a contract and implement it on a service

    ·         You choose or define a service binding that selects a transport along with quality of service, security and other options

    ·         You deploy an endpoint for the contract by binding it (using the binding definition, hence the name) to a network address.

    A binding is a layered combination of a transport, an encoder and of any additional protocol channels (reliable session, transaction flow, etc.) that you’d like to assemble into a transport stack for exposing a service implementation on a specific endpoint address.

    For exposing a HTTP-based RESTish service we need:

    A.    The HTTP address to host the service at.

    B.     Some binding configuration that tells Indigo’s HTTP transport to use our PoxEncoder.

    C.     An implementation of a service-contract that’s configured (using the HttpMethodOperationSelectorSection config extension) or marked-up (with the HttpMethodOperationSelectorAttribute) to use our HttpMethodOperationSelectorBehavior for endpoint address filtering and selecting methods.

    I’ve shown you quite a few contract variants in the first parts of this series and therefore I don’t really have to explain the C in too much detail anymore; except: While the A is just a plain HTTP address such as http://www.example.com/service, it’s interesting insofar as this address is, unlike as with SOAP services, really just the common address prefix for the dispatch URIs of the particular service and that there is a split between what is A and what is C.

    As I’ve explained, the philosophy behind the contract design of my extensions is around the namespaces that are the basis for forming URIs. Because REST services aren’t simply using HTTP as a transport tunnel as SOAP services do, but rather leverage HTTP as the application protocol it is, the URI is a lot more than just a drop-off point for messages. With REST services, the URI is an expression that has both, addressing (transport) and contract (dispatch) aspects to it and we need to separate those out. A clear distinction between global and local namespaces allows us to do that (And I am going into a bit more detail than I usually would, to further address an objection of Mark Baker, 1st comment, on my choice of the programming model):

    There is a global namespace that’s managed by the global DNS system of which anyone can reserve a chunk for themselves by registering a domain-name. The domain-name provides a self-manageable namespace root of which sub-namespaces (subdomains) can be derived and allocated to specific hosts/services or groups of hosts/services by the domain owners. On the particular host, you can put an application behind a specific port, which might either the default port of your particular application protocol or – diverging from the protocol standard – some other port of your choosing. Each Internet application deployment has therefore at least one unique mapping into this global namespace system.

    Any further segmentation of the namespace except the host-name and the port-number are private matters of the application listing to that endpoint. With Indigo self-hosted HTTP services, the listening application is Windows (!) – more precisely it’s the HTTP.SYS kernel listener. For IIS/WAS hosted services, the listening application is IIS (for IIS 5.1 and below) or, again, Windows – through that very listener. At the HTTP.SYS listener, handler processes can register their interest in requests sent to certain sub-namespaces of the global namespace mapping (host/port), which are identified by relative URIs. To be clear: The HTTP.SYS API indeed requires the caller to provide an absolute URI like http://hostname:port/service, but the two main parts (scheme/host/port and path) of that URI are used for different purposes:

    ·         Global mapping: The hostname and port are used to establish a new listener on that particular port (if there is already a listener it is shared) and to populate the hostname-filter table that’s used to disambiguate requests by the Host header in case the IP address is mapped to multiple DNS host entries.

    ·         Local mapping: The path information of the URI (the remaining relative URI with scheme, hostname and port stripped) is used as a prefix-matching expression to figure out which handler process shall receive the request and, inside that handler process, to further identify and invoke the appropriate endpoint and handler that deals with the resource that the complete URI path represents.

    As indicated, mapping URIs to an endpoint needs to distinguish between how we segment and map a local namespace and how we hook that into the global namespace. Hence, the root for any absolute URIs that’s establishing the mapping into the global namespace shall always be separate from the code and reside in configuration for reasons of flexibility as Mark was rightfully pointing out in his objection, while the shape and mapping of the local namespace is typically very application and use-case specific and might well be partially or entirely hardwired.

    ·         The Indigo A(ddress) of a REST service implemented with my particular programming model is used to hook a given service (or resource representation manager, if you like) into the global namespace: http://myservice.example.com/. Only to be pragmatic and to allow multiple such services to locally share a particular hostname and port and indeed only as an alternative and workaround to creating a separate DNS entry for each service, that mapping might include a path prefix allowing the local low-level infrastructure to demultiplex requests sent to the same global namespace mapping: http://myservices.example.com/serviceA and http://myservices.example.com/serviceB.

    ·         The Indigo C(ontract) of a REST service implemented with my particular programming model is used to define the shape of the local namespace that the service owns and which is used to provide access to the representations of the resource-types the service is responsible for.

    The following configuration snippet for a simple web-server based on my extensions is illustrating that split:

    <services>
       <
    service type="LittleIndigoWebServer.MyWebServer">
          <
    endpoint contract="LittleIndigoWebServer.IMyWebServer"
                    address="http://localhost:8020/"
                    
    binding="customBinding"
                    
    bindingConfiguration="poxBinding"/>
       </
    service>
    </
    services>

    The address http://localhost:8020/ is how I map the service into the global addressing namespace. The local namespace shape for that particular service is a defined by the layout of the file-system directory from which the service grabs files and returns them. What? You can’t see the directory structure and the resulting URLs from the above mapping? Of course not. It’s a private matter of the service implementation what that the local namespace structure is and it’s up to me what parts I am exposing. If I am nice enough I will give you something on a GET/HEAD request on the root of my local namespace (= global address without any suffix), and if I am not nice you get a 404 and will just have to know what to ask for. The “will have to know” part is contract. It’s an assurance that if you come looking at a particular place in my namespace you will have access to a particular thing. My [HttpMethod] attributes manifest that assurance on Indigo contracts.

    That leaves B. Before I got carried away by A and C, I wrote [now a bit annotated] “A binding is a layered combination of a transport, an encoder and of any additional protocol channels (reliable session, transaction flow, etc.) that you’d like to assemble into a transport stack for exposing a service implementation [C] on a specific endpoint address [A].”

    Putting together such a binding is not much more work than putting a little text between angle brackets and quotation marks in config as shown in the following snippet:

    <customBinding>
       <
    binding name="poxBinding">
          <
    poxEncoder/>
          <
    httpTransport mapAddressingHeadersToHttpHeaders="true"
               
    maxMessageSize="2048000" maxBufferSize="2048000" manualAddressing="true"
              
    authenticationScheme="Anonymous" transferMode="StreamedResponse"  />
       </
    binding>
    </
    customBinding>

    I am building a custom binding that’s combining the HTTP transport with a custom binding element config extension I built for the PoxEncoder. It’s that simple. And adding the binding element extension does not require black magic, either. It’s just another XML snippet that maps the extension class to an element name (“poxEncoder”) as you can see in the extensions section of the complete config file:

     <?xml version=1.0 encoding=utf-8 ?>
    <
    configuration>
       <
    system.serviceModel>
          <
    extensions>
             <
    bindingElementExtensions>
                <
    add name="poxEncoder" type="newtelligence.ServiceModelExtensions.PoxEncoderBindingExtension, newtelligence.ServiceModelExtensions"/>
             </
    bindingElementExtensions>
          </
    extensions>
          <
    bindings>
             <
    customBinding>
            <
    binding name="poxBinding">
                   <
    poxEncoder/>
                   <
    httpTransport mapAddressingHeadersToHttpHeaders="true"
                               
    maxMessageSize="2048000" maxBufferSize="2048000" manualAddressing="true"
                               
    authenticationScheme="Anonymous" transferMode="Streamed"  />
                </
    binding>
             </
    customBinding>
          </
    bindings>
          <
    services>
             <
    service type="LittleIndigoWebServer.MyWebServer">
                <
    endpoint contract="LittleIndigoWebServer.IMyWebServer"
                        address="http://localhost:8020/"
                        
    binding="customBinding"
                        
    bindingConfiguration="poxBinding"/>
             </
    service>
          </
    services>
      </
    system.serviceModel>
    </
    configuration>

    The PoxEncoderBindingExtension is a class that is based on System.ServiceModel.Configuration.BindingElementExtensionSection. Whenever the configuration is processed by Indigo, the presence of the “poxEncoder” element in a binding triggers the creation of an instance of the class and if we’d require any configuration attributes (which we don’t), those would be stuffed into the Properties collection.    

    using System;
    using System.ServiceModel.Configuration;
    using System.ServiceModel;
    using System.Configuration;

    namespace newtelligence.ServiceModelExtensions
    {
       public class PoxEncoderBindingExtension : BindingElementExtensionSection
       {
            /// <summary>
            /// Initializes a new instance of the <see cref="T:PoxEncoderBindingExtension"/> class.
            /// </summary>
          public PoxEncoderBindingExtension()
          {
          }

            /// <summary>
            /// Creates the binding element.
            /// </summary>
            /// <returns></returns>
          protected override BindingElement CreateBindingElement()
          {
             PoxEncoderBindingElement pcc = new PoxEncoderBindingElement();
             return pcc;
          }

            /// <summary>
            /// Gets the type of the binding element.
            /// </summary>
            /// <value>The type of the binding element.</value>
          public override Type BindingElementType
          {
             get
             {
                return typeof(PoxEncoderBindingElement);
             }
          }

            /// <summary>
            /// Gets the name of the configured section.
            /// </summary>
            /// <value>The name of the configured section.</value>
          public override string ConfiguredSectionName
          {
             get
             {
                return "poxEncoder";
             }
          }

          private ConfigurationPropertyCollection properties;
            /// <summary>
            /// Gets the collection of properties.
            /// </summary>
            /// <value></value>
            /// <returns>The <see cref="T:System.Configuration.ConfigurationPropertyCollection"></see> collection of properties for the element.</returns>
          protected override ConfigurationPropertyCollection Properties
          {
             get
             {
                if (this.properties == null)
                {
                   ConfigurationPropertyCollection configProperties = new ConfigurationPropertyCollection();
                   this.properties = configProperties;
                }
                return this.properties;
             }
          }
       }
    }

    Once the configuration information has been read, the extension is asked to create a BindingElement from the acquired information. So these extensions are really just factories for binding elements. The binding element, which can also be used to compose such a binding in code by explicitly adding it to a System.ServiceModel.CustomBinding is shown below:

     using System;
    using System.Collections.Generic;
    using System.Text;
    using System.ServiceModel;
    using System.ServiceModel.Design;
    using System.ServiceModel.Channels;

    namespace newtelligence.ServiceModelExtensions
    {
        public class PoxEncoderBindingElement : BindingElement, IMessageEncodingBindingElement
       {
            /// <summary>
            /// Clones this instance.
            /// </summary>
            /// <returns></returns>
          public override BindingElement Clone()
          {
             return new PoxEncoderBindingElement();
          }

            /// <summary>
            /// Creates the message encoder factory.
            /// </summary>
            /// <returns></returns>
          public MessageEncoderFactory CreateMessageEncoderFactory()
          {
             return new PoxEncoderFactory();
          }

            /// <summary>
            /// Gets the addressing version.
            /// </summary>
            /// <value>The addressing version.</value>
          public AddressingVersion AddressingVersion
          {
             get
             {
                return AddressingVersion.Addressing1;
             }
          }

            /// <summary>
            /// Gets the protection requirements.
            /// </summary>
            /// <returns></returns>
          public override System.ServiceModel.Security.Protocols.ChannelProtectionRequirements GetProtectionRequirements()
          {
             return null;
          }

            /// <summary>
            /// Builds the channel factory.
            /// </summary>
            /// <param name="context">The context.</param>
            /// <returns></returns>
          public override IChannelFactory BuildChannelFactory(ChannelBuildContext context)
          {
             if (context == null)
                throw new ArgumentNullException("context");

             context.UnhandledBindingElements.Add(this);
             return context.BuildInnerChannelFactory();
          }

            /// <summary>
            /// Builds the channel listener.
            /// </summary>
            /// <param name="context">The context.</param>
            /// <returns></returns>
            public override IChannelListener<TChannel> BuildChannelListener<TChannel>(ChannelBuildContext context)
            {
                if (context == null)
                throw new ArgumentNullException("context");

             context.UnhandledBindingElements.Add(this);
             return context.BuildInnerChannelListener<TChannel>();
          }
       }
    }

    Binding elements are typically used to put together client-side (channel factory) or service-side (channel listener) transport stacks. At the bottom is the transport and layered on top of it are security, reliable sessions, transaction flow and all other protocol features you need. Each protocol or feature on the channel/listener level has its own binding element and using those you configure yourself a binding combining the features you need and in the order that they should be applied.

    The binding elements for message encoders are a bit different, because they are not contributing their own channel factories or channel listeners into the stack, but rather “only” supply the message encoder for the configured transport.

    Whenever a binding is instantiated, Indigo creates a ChannelBuildContext which contains the sequence of the binding elements that shall be stacked onto each other into a channel or listener stack and starts stacking them from top to bottom by invoking the topmost binding element’s BuildChannelListener or BuildChannelFactory method. Once the binding element is done creating its channel factory or channel listener, it invokes BuildInnerChannel[Listener/Factory] on the context to have the binding element underneath do its work. (The context is also used to validate whether combination of the elements yields a functional binding stack, but I won’t go into that here).

    Our binding element, however, won’t create a channel factory or listener, but rather put itself into the UnhandledBindingElements collection on the build context and will then just have the context complete the construction work. With putting itself into that collection, the binding element makes itself and its most irresistible feature (you’d also think that if you were an Indigo transport) – the IMessageEncodingBindingElement implementation – visible to the transport and waves its hand that it wants to be used. The transport’s binding element, which is at the bottom of the stack and therefore asked to build its channel factory/listener after our binding element has been invoked, will go look in the UnhandledBindingElements  collection whether a message encoding binding element is advertising itself for use. And if that’s so it will forget all of its defaults and happily embrace and use an encoder created by the factory returned by IMessageEncodingBindingElement.CreateMessageEncoderFactory, which is, in our case, this rather simple class:

     using System;
    using System.Collections.Generic;
    using System.Text;
    using System.ServiceModel.Channels;
    using System.ServiceModel;

    namespace newtelligence.ServiceModelExtensions
    {
        /// <summary>
        ///
        /// </summary>
       public class PoxEncoderFactory : MessageEncoderFactory
       {
          MessageEncoder encoder;

            /// <summary>
            /// Initializes a new instance of the <see cref="T:PoxEncoderFactory"/> class.
            /// </summary>
          public PoxEncoderFactory()
          {
              encoder = new PoxEncoder();

          }

             /// <summary>
            /// Gets the encoder.
            /// </summary>
            /// <value>The encoder.</value>
            public override MessageEncoder Encoder
          {
             get
             {
                return encoder;
             }
          }

            /// <summary>
            /// Gets the message version.
            /// </summary>
            /// <value>The message version.</value>
          public override MessageVersion MessageVersion
          {
             get
             {
                return encoder.MessageVersion;
             }
          }
       }
    }

    Soooooooo….!

    If you had actually copied all those classes from Parts 1-8 down into local files and compiled them into an assembly, you’d have all my REST/POX plumbing code by now (except, admittedly, an application-level utility class that helps putting messages together).

    But wait … don’t do that. In the next part(s) I’ll give you the code all packed up and ready to compile along with the little web server that we’ve configured here and will also share some code snippets from my TV app … maybe the RSS and ASX pieces?

    Categories: Indigo

    January 4, 2006
    @ 09:21 PM

    In case you are not following my Indigo REST/POX series, I quote one paragraph from today's Part 7 that is well worth to be quoted out of context. It talks about (SOAP-) messages and the misconception that a message is a small thing:

    There’s no specification that says that you cannot stick 500 Terabyte or 500 Exabyte worth of data (think 365x24 live 1080i video streams) into a single message. As long as you have some reason to believe that the sender will eventually, in 20 years from now, give you “</soap:Body></soap:Envelope>” to terminate the message, the message can be assumed to be well-formed and complete.

    The WCF transports that support "streamed" transfer-mode (all except MSMQ) all consider messages to be monsters like that when streaming is enabled. I have a bit more on the streaming mode in today's part of the series.

    Categories: Indigo

    Part 1, Part 2, Part 3, Part 4, Part 5, Part 6, Part 7, Part 8, Part 9

    Where are we?

    ·         In Parts 1 and 2, I explained contracts in the REST/POX context and the dispatch mechanisms that we need to enable Indigo to accept and handle REST/POX requests. With that I introduced a metadata extension, the [HttpMethod] attribute that can be used to mark up operations on an Indigo contract with HTTP methods and a URI suffixes that we can dispatch on. I also showed how we can employ a parameter inspector to extract URI-embedded arguments and flow them to the operation in a message property.

    ·         In Parts 3 and 4, I showed how we use the [HttpMethodOperationSelector] attribute to replace Indigo’s default address filtering and operation selection mechanisms, basically the entire message dispatch mechanism, with our own variants. The SuffixFilter is used to find the appropriate endpoint for an incoming request and the HttpMethodOperationSelectorBehavior  find the operation (method) on that endpoint which shall receive the incoming request message.

    ·         In Parts 5 and 6, you saw how the PoxEncoder puts outbound envelope-less POX documents onto the wire in its WriteMessage methods and accepts incoming non-SOAP XML requests through its ReadMessage methods and wraps them with an in-memory envelope (“message”) for further processing. I also showed the PoxBase64XmlStreamReader, which is an XML Infoset wrapper for arbitrary binary streams that interacts with the PoxEncoder to allow smuggling any sort of raw binary content through the Indigo infrastructure and onto the wire.

    We’re pretty far along already. We’ve got the dispatch mechanisms, we know how to hook the dispatch metadata into the services, we’ve got the wire-encoding – we have most of the core pieces together. In fact, the last two key classes we’re missing (configuration hooks aside) are two specialized message classes that we need to handle incoming requests. In Part 6, you could see that the two ReadMessage overloads of the PoxEncoder delegate all work to the PoxBufferedMessage for the “buffered” transfer-mode overload and to PoxStreamedMessage for the “streamed” transfer mode overload.

    ReadMessage is called on an encoder whenever a transport has received a complete message buffer (buffered mode) or has accepted and opened a network stream (streamed mode).

    Using streamed mode means very concretely that Indigo will start handling the message even though the message might not have completely arrived. A transport in streaming mode will only do as much as it needs to do in order to deal with the transport-level framing protocol. I use “framing protocol” as a general term for what is done at the transport level to know what the nature of the payload is and where the payload starts and ends. For HTTP, the HTTP transport figures out whether an incoming request is indeed an HTTP request, will read/parse the HTTP headers, and will then layer a stream over the request’s content, irrespective of whether the transfer of that byte sequence has already been completed. This stream is immediately handed off to the rest of the Indigo infrastructure and the transport has done its work by doing so.

    Pulling the remaining bytes from that stream is someone else’s responsibility in streamed mode. Whenever a piece of the infrastructure pulls data directly or indirectly from the stream and the data chunk requested is still in transfer, the stream will block and wait until the data is there. The transport’s handling of the framing protocol will typically also take care of chunking and thus make a chunked stream appear to be continuous. When I say “indirect pull” I mean that it may very well be an XmlDictionaryReader layered over an XmlReader layered over the incoming network stream.

    The streaming mode is of particular interest for very large messages that may, in an extreme case, be virtually limitless in size. There’s no specification that says that you cannot stick 500 Terabyte or 500 Exabyte worth of data (think 365x24 live 1080i video streams) into a single message. As long as you have some reason to believe that the sender will eventually, in 20 years from now, give you “</soap:Body></soap:Envelope>” to terminate the message, the message can be assumed to be well-formed and complete.

    No matter whether you use buffered or streamed mode, the configured encoder’s ReadMessage method is the first place where the read data chunk or the stream goes and that delegates, as shown to our two message classes. So let’s look at them.

    We’ll primarily look at the PoxBufferedMessage, which is constructed over the read message buffer in the PoxEncoder like this:

    public override Message ReadMessage(ArraySegment<byte> buffer, BufferManager bufferManager)
    {
       return new PoxBufferedMessage(buffer, bufferManager);
    }

    The class PoxBufferedMessage is derived from the abstract System.ServiceModel.Message class and implements the base-class’s abstract properties Headers, Properties, and Version and overrides the OnClose(), OnGetReaderAtBodyContents(), and OnWriteBodyContents() virtual methods. Internally, Indigo has several such Message implementations that are each customized for certain scenarios. Implementing own variants of Message is simply another extensibility mechanism that Indigo gives us.

    Using System;
    using System.Collections.Generic;
    using System.Text;
    using System.ServiceModel;
    using System.IO;
    using System.Xml;
    using System.Runtime.CompilerServices;
    using System.ServiceModel.Channels;

    namespace newtelligence.ServiceModelExtensions
    {
        /// <summary>
        /// This class is one of the message classes used by the <see cref="T:PoxEncoder"/>
        /// It serves to wrap an unencapsulated data buffer with a message structure.
        /// The data buffer becomes the body content of the message.
        /// </summary>
       public class PoxBufferedMessage : Message, IPoxRawBodyMessage
       {
          MessageHeaders headers = new MessageHeaders(MessageVersion.Soap11Addressing1);
          MessageProperties properties = new MessageProperties();
          byte[] buffer;
          int bufferSize;
          BufferManager bufferManager;
          Stream body;
           
            /// <summary>
            /// Initializes a new instance of the <see cref="T:PoxBufferedMessage"/> class.
            /// </summary>
            /// <param name="buffer">The buffer.</param>
          public PoxBufferedMessage(byte[] buffer)
          {
                bufferManager = null;
                buffer = buffer;
                bufferSize = buffer.Length;
          }

            /// <summary>
            /// Initializes a new instance of the <see cref="T:PoxBufferedMessage"/> class.
            /// </summary>
            /// <param name="buffer">The buffer.</param>
            /// <param name="bufferManager">The buffer manager.</param>
            public PoxBufferedMessage(ArraySegment<byte> buffer, BufferManager bufferManager)
            {
                bufferManager = bufferManager;
                bufferSize = buffer.Count;
                buffer = bufferManager.TakeBuffer( bufferSize);
                Array.Copy(buffer.Array, buffer.Offset, buffer, 0, bufferSize);
            }     

    We can construct instances of the class over a raw byte array or an “array segment” layered over such an array. Array segments are preferred over raw arrays, because their use eases memory management. You can keep a pool of buffers with a common size, even though the actual content is shorter than the buffer size and probably even offset from the lower buffer boundary. If we get a raw byte array we simply adopt it, but if we get an array segment alongside a reference to a buffer manager we take a new buffer from the buffer manager and copy the array segment to that acquired buffer.

            /// <summary>
            /// Called when the message is being closed.
            /// </summary>
            protected override void OnClose()
            {
                base.OnClose();
                if ( bufferManager != null)
                {
                    bufferManager.ReturnBuffer( buffer);
                }           
            }

    When we close the message and we have acquired it using the buffer manager (which is signaled by the presence of the reference) we duly return it once the message is being closed (or disposed or finalized).

    The next two methods are an implementation of the IPoxRawBodyMessage interface that is, you guessed it, defined in my extensions. If the handler method wants to get straight at the raw body content knowing that it doesn’t expect XML, it can shortcut by the whole XmlReader and XML serialization story by asking for the BodyContentType and pull out the raw body data as a stream layered over the buffer:

            /// <summary>
            /// Gets the raw body stream.
            /// </summary>
            /// <returns></returns>
          [MethodImpl(MethodImplOptions.Synchronized)]
          public Stream GetRawBodyStream()
          {
             if ( body == null)
             {
                 body = new MemoryStream( buffer,0, bufferSize,false,true);
             }
             return body;
          }

            /// <summary>
            /// Gets the content type of the raw message body based on the Content-Type HTTP header
            /// contained in the HttpRequestMessageProperty or HttpResponseMessageProperty of this
            /// message. The value is null if the type is unknown.
            /// </summary>
            public string BodyContentType
            {
                get
                {
                    if (Properties.ContainsKey(HttpRequestMessageProperty.Name))
                    {
                        return ((HttpRequestMessageProperty)Properties[HttpRequestMessageProperty.Name]).Headers["Content-Type"];
                    }
                    if (Properties.ContainsKey(HttpResponseMessageProperty.Name))
                    {
                        return ((HttpResponseMessageProperty)Properties[HttpResponseMessageProperty.Name]).Headers["Content-Type"];
                    }
                    return null;
                }
            }

    There is a bit of caution required using this mechanism, though. Because the message State (Created, Written, Read, Copied, Closed) is controlled by the base-class and cannot be set by derived classes, the message should be considered to be in the State==MessageState.Read after calling the GetRawBodyStream() method. That doesn’t seem to be necessary because we have a buffer here, but for the streamed variant that’s a must. And for the sake of consistency we introduce this constraint here.

    The BodyContentType property implementation seems, admittedly, a bit strange at first sight. Even though you won’t see the message properties being populated anywhere inside this class, we’re asking for them and base the content-type detection on their values. That only makes sense when we consider the way messages are being populated by Indigo. As I explained, the first thing that gets called once the transport has a raw data chunk or stream in its hands that it believes to be a message, it invokes the encoder. For incoming requests/messages, the encoder is really serving as the message factory constructing Message-derived instances over raw data. Once the encoder has constructed the message in one of the ReadMessage overloads, the message is returned to the transport. If the transport wants, it can then (and the HTTP transport does) stick properties into that newly created message and then hand it off to the rest of the channel infrastructure for processing and dispatching. Because these extensions are built for REST/POX and therefore have HTTP affinity, that’s precisely what we assume to be happening for the BodyContentType property and the CreateBodyReader() method below. As I already explained in Part 1, the HTTP transport will always add a HttpRequestMessageProperty  to the message and that’s consequently from which we can grab the content-type of the incoming request data.

            private XmlDictionaryReader CreateBodyReader()
            {
                XmlDictionaryReader reader = null;

                /*
                 * Check whether the message properties indicate that this is a raw binary message.
                 * In that case, we'll wrap the body with a PoxBase64XmlStreamReader
                 */
                bool hasPoxEncoderProperty = Properties.ContainsKey(PoxEncoderMessageProperty.Name);
                if (!(hasPoxEncoderProperty && ((PoxEncoderMessageProperty)Properties[PoxEncoderMessageProperty.Name]).RawBinary))
                {
                    string contentType = null;

                    /*
                     * Check for whether either the HttpRequestMessageProperty or the HttpResponseMessageProperty
                     * are present. If so, extract the HTTP Content-Type header. Otherwise the content-type is
                     * assumed to be text/xml ("POX")
                     */
                    bool hasRequestProperty = Properties.ContainsKey(HttpRequestMessageProperty.Name);
                    bool hasResponseProperty = Properties.ContainsKey(HttpResponseMessageProperty.Name);
                    if (hasResponseProperty)
                    {
                        HttpResponseMessageProperty responseProperty =
                          Properties[HttpResponseMessageProperty.Name] as HttpResponseMessageProperty;
                        contentType = responseProperty.Headers["Content-Type"];
                    }
                    else if (hasRequestProperty)
                    {
                        HttpRequestMessageProperty requestProperty =
                           Properties[HttpRequestMessageProperty.Name] as HttpRequestMessageProperty;
                        contentType = requestProperty.Headers["Content-Type"];
                    }

                    if (contentType == null)
                    {
                        contentType = "text/xml";
                    }

                    /*
                     * If the content type is text/xml (POX) we will create a plain XmlTextReader for the body.
                     */
                    if (contentType.StartsWith("text/xml", StringComparison.OrdinalIgnoreCase))
                    {
                       // do we only have a UTF byte-order mark?
                       if (_bufferSize <= 4)
                       {
                           // create a new reader over a fake infoset and place it on the EndElement
                          
    reader = XmlDictionaryReader.CreateDictionaryReader(
                              new XmlTextReader(new StringReader("<no-data></no-data>")));
                           reader.Read(); reader.Read();
                       }
                       else
                      
    {
                           reader = XmlDictionaryReader.CreateDictionaryReader(new XmlTextReader(GetRawBodyStream()));
                       }

                    }
                }
                /*
                 * If the content wasn't identified to be POX, we'll wrap it as binary. 
                 */
                if (reader == null)
                {
                    reader = XmlDictionaryReader.CreateDictionaryReader(new PoxBase64XmlStreamReader(GetRawBodyStream()));
                }
                return reader;
            }

    The private CreateBodyReader() method that constructs XML readers for the both, the OnGetBodyReaderAtBodyContents() and the OnWriteBodyContents() overrides shown below, uses the same strategy to figure out the content-type of the message and therefore to guess what’s hidden inside the byte-array (or array segment) the message was constructed over. To make the message class useful for the request and response direction, we’ll distinguish there two separate cases here:

    ·         If the message is a response, the handling method in the user code might have indicated that it wants the encoder to serialize the message onto the wire in “raw binary” mode. The indicator for that is the presence of the PoxEncoderMessageProperty having the RawBinary property set to true. If that is the case, the reader we return is always our PoxBase64XmlStreamReader. The property cannot occur in request messages because the Indigo transports simply don’t know about it.

    ·         If the message is a request or a response with the mentioned property missing, we will try figuring out the message’s content-type using the described strategy of using the HTTP transport’s message properties. If we can’t figure out a content-type for a response (it’s optional for the responding handler code to supply it), we will assume that the content-type is “text/xml”. If the message is a request we can rely of getting a content-type as long as the underlying transport is Indigo’s HTTP transport implementation. If the content-type is indeed “text/xml” we construct an XmlTextReader over the raw data and return it. If the content-type is anything else, we use our PoxBase64XmlStreamReader wrapper, because we have to assume that the encapsulated data we’re dealing with is not XML.

    The OnGetBodyReaderAtBodyContents() and the OnWriteBodyContents() overrides are consequently very simple:

            /// <summary>
            /// Called when the client requests a reader for the body contents.
            /// </summary>
            /// <returns></returns>
          protected override XmlDictionaryReader OnGetReaderAtBodyContents()
          {
             XmlDictionaryReader reader = CreateBodyReader();
             reader.MoveToContent();
             return reader;
          }

            /// <summary>
            /// Called when the client requests to write the body contents.
            /// </summary>
            /// <param name="writer">The writer.</param>
          protected override void OnWriteBodyContents(XmlDictionaryWriter writer)
          {
             XmlDictionaryReader reader = CreateBodyReader();
             writer.WriteNode(reader, false);
          }

    What’s left to complete the message implementation are the compulsory overrides of the abstract properties of Message, for which we have backing fields declared at the top of the class:

            /// <summary>
            /// Gets the message version.
            /// </summary>
            /// <value>The message version.</value>
          public override MessageVersion Version
          {
             get
             {
                return MessageVersion.Soap11Addressing1;
             }
          }

            /// <summary>
            /// Gets the SOAP headers.
            /// </summary>
            /// <value>The headers.</value>
          public override MessageHeaders Headers
          {
             get
             {
                return headers;
             }
          }

            /// <summary>
            /// Gets the message properties.
            /// </summary>
            /// <value>The properties.</value>
          public override MessageProperties Properties
          {
             get
             {
                return properties;
             }
          }
        }
    }

    The PoxStreamedMessage is only different from this class insofar as that it doesn’t have the buffer management. The GetRawBodyStream() method immediately returns the encapsulated stream and the remaining implementation is largely equivalent, if not identical (yes, I should consolidate that into a base class). Therefore I am not pasting that class here as code but rather just append as a downloadable file, alongside the declaration of IPoxRawBodyMessage and the twice mentioned and not yet shown PoxEncoderMessageProperty class.

    With this, we’ve got all the moving pieces we need to build what’s essentially becoming an Indigo-based, message-oriented web-server infrastructure with a REST-oriented programming model. What’s missing is how we get our encoder configured into a binding so that we can put it all together and run it.

    Configuration is next; wait for part 8.

    Download: PoxEncoderMessageProperty.zip
    Download: PoxStreamedMessage.zip
    Download: IPoxRawBodyMessage.zip

    [2006-01-13: Updated PoxBufferedMessage code to deal with entity bodies that only consist of a UTF BOM]

    Categories: Indigo

    January 3, 2006
    @ 12:00 AM

    Sabine and I were just browsing Channel 9 using TVTonic on our Media Center PC that's been recently connected to this christmas gift. We watched a few snippets of Microsoft PMs and other folks presenting their latest stuff and then that. Sabine (she's a nurse) said "...oh, that's like Hospital TV".

    I can't help but admit that she does indeed have more than just one point in saying that.

    Categories: Other Stuff

    January 2, 2006
    @ 11:08 AM

    I recently needed a TCP port-forwarder that sits on a socket connection and monitors it. My concrete use-case is that I need to front the backend-server of my TV application with such a port forwarder in order to create live-TV streaming sessions as soon as a client requests them and also tears them down shortly after the client disconnects so that the session doesn’t need to time out and blocks the tuner until then. The backend also requires that I do a periodical “keep-alive” ping every 30-40 seconds, which isn’t a very practical requirement for some of my client scenarios. Therefore, I needed, generally speaking, something that would sit between the client and the backend server, monitors the data stream and would let me run some code (set up the live session and start the keep-alive timer) when I get a new client connection and just before I connect through to the target and which would let me run some code (shut down the session and stop the keep-alive) as soon as the connection is torn down.

    Since, I didn’t find one (or was too blind or too lazy, you know how that goes), I wrote one. It’s a fully asynchronous TcpListener/TcpClient based implementation, it’s fast and stable enough for my purposes and it might or might not be for yours, it has a bit of tolerance for targets that don’t accept a connection on the first try, and you can hook up events to “before target connect” and “after target disconnect”. Since all the bytes fly by, you can instrument the thing further or monitor the stream as you like.  

    The code is pretty straightforward, even though the asynchronous calls/callbacks admittedly make the execution paths in the implementation a bit challenging to follow, and should not require much further explanation. You construct an instance of TcpPortForwader passing the local port and the target port and host to forward to, call Start() and the listener starts listening. Stop() stops the listener. You can call Start() from any thread; the listener will implicitly use thread-pool threads to run on its own. Hook up the events and they are being raised. Simple enough. Download below.

    Download: TcpPortForwarder.zip

    Categories: CLR

    Part 1, Part 2, Part 3, Part 4, Part 5

    I threw a lot of unexplained code at you in Part 5 and that wasn’t really fair.

    The PoxEncoder class is a replacement for Indigo’s default TextMessageEncoder class that’s used by the HTTP transport unless you explicitly configure something different. Indigo comes with three built-in encoders, namely:

    ·         The TextMessageEncoder serializes Indigo’s internal Message into SOAP 1.1 or SOAP 1.2 envelopes using (applies only to the latter) the desired character encodings (UTF-8, UTF-16, etc.) and of course it also deserializes incoming SOAP envelopes into the Indigo representation.

    ·         The MtomMessageEncoder serializes messages into SOAP 1.2 messages as specified by the MTOM specification, which allows for a much more compact transmission of binary-heavy SOAP envelopes than if you were simply using base64Binary encoded element data. MTOM is a good choice whenever the size of binary content in a SOAP envelope far exceeds the size of the rest of the data. Your mileage may vary, so that’s a thing to measure carefully unless it’s blatantly obvious such as in the case of writing a service for a digital imaging library.

    ·         The BinaryMessageEncoder serializes messages into SOAP 1.2 envelopes, but does so in a very compact binary format that preserves the XML information set, but is not at all like XML text. The gist of the binary encoding is the assumption that if both communicating parties are implemented with Indigo and share the same contract, the metadata existing at both ends reduces the hints that need to go on the wire. In other words: The binary encoding doesn’t need to throw all  these lengthy XML tag names and namespace names explicitly onto the wire, but can refer to them by pointing to a dictionary that’s identically constructed on both ends. The binary encoding in Indigo is a bit like the modern-day, loosely coupled grand-child of NDR and “midl.exe /Oicf” if you like. What’s important to note about this encoding is that its primary design goal is performance and interoperability is in fact a non-goal. The BinaryMessageEncoder assumes Indigo endpoints. If you don’t like that, you can always use the text encoding, which is designed for interoperability.

    Our PoxEncoder here differs from all three Indigo encoders in that it does specifically not serialize SOAP messages, but rather just the body contents of a Message.

    In order for you to understand what’s happening here, I’ll pick the most relevant methods and explain them in detail. We will start with the Initialize() method that is invoked by all three constructor overloads:

    /// <summary>
    ///
    Initializes common properties of the encoder.
    /// </summary>
    private void Initialize()
    {
      if (this.MessageVersion.Envelope == EnvelopeVersion.Soap12)
      {
        // set the aprorpiate media type for SOAP 1.2
         this. mediaType = "application/soap+xml";
      }
      else if (this.MessageVersion.Envelope == EnvelopeVersion.Soap11)
      {
        // set the appropriate media type for SOAP 1.1
         this. mediaType = "text/xml";
      }
      // compose the content type from charset and media type
      this. contentType = string.Format(CultureInfo.InvariantCulture, "{0}; charset={1}", mediaType, textEncoding.WebName);
    }

    It is required for each MessageEncoder-derived class to implement the abstract properties MediaType, ContentType, and MessageVersion, and therefore we have to initialize the backing fields for these properties properly and return meaningful values even though the PoxEncoder is exactly the “anti-SOAP” encoder. The message version specified in the encoder is relevant for Indigo higher up on the stack, because it needs to know what rules and constraints apply to Message instances as they are constructed and processed. The content type and media types are required by the transports so that they know what content and/or media type to specify as metadata in their transport frame (eg. the Content-Type header in HTTP). If we initialize the encoder with the Soap12 message version, it will consequently report the application/soap+xml media type, even though the encoder doesn’t ever write such envelopes to the wire. You might consider that a bug in the PoxEncoder and you might be right, but it doesn’t really matter. Because any methods can return all sorts of payloads, we will override the content-type on the message-level so that this information has really no effect. I do need to clean this up a little. Later.

    Now let’s look at the parts that actually do the work. I will start with the two WriteMessage overloads.

    The first overload’s signature is
         public override ArraySegment<byte> WriteMessage(Message msg, int maxMessageSize, BufferManager bufferManager, int messageOffset)
    and is invoked by the transport whenever a message must be wire-encoded and the output transfer mode is set to TransferMode.Buffered or TransferMode.StreamedRequest (which implies a buffered response). The second overload’s signature is
        public override void WriteMessage(Message msg, System.IO.Stream stream)
    and is invoked by the transport whenever a message must be wire-encoded and the output transfer mode is set of TransferMode.Streamed or TransferMode.StreamedResponse (which implies a streamed response).

    The transfer-mode property is configurable on all of the pre-built HTTP bindings and on the <httpTransport> binding element. “Buffered” encoding means that the entire message is encoded at once and written into a buffer, which is then given to the the transport for sending. “Streamed” encoding means that the message is pushed into to a stream, whereby the stream is typically immediately layered directly over the transport. That means that whenever our encoder writes data to that stream, it is immediately pushed to the remote communication partner. The “streamed” mode is the optimal choice for sending very large messages that are, for instance, too big to be reasonably handled as a single memory block. The buffered mode is better (and faster) for compact messages. I’ll dissect the buffered variant first:

    public override ArraySegment<byte> WriteMessage(Message msg, int maxMessageSize, BufferManager bufferManager, int messageOffset)
    {
       if (msg.IsEmpty)
       {
          // if the message is empty (no body defined) the result is an empty
          // byte array.
          byte[] buffer = bufferManager.TakeBuffer(maxMessageSize);
          return new ArraySegment<byte>(buffer, 0, 0);
       } 

    If the message is empty (that means: the body is empty), we request a buffer from the buffer manager and return an empty slice of that buffer, because this encoder’s output is “nothing” if the body is empty.The BufferManager is an Indigo helper class that manages a pool of pre-allocated buffers and serves to optimize memory management by avoiding the allocation and the discarding of buffers for every message. And encoder should therefore use the buffer manager argument and use it to obtain the buffers backing the array segment that is to be returned. Once the message has been handled by the transport, the transport will return the buffer into the pool.

       else
       {
          // check RawBinary bit in the message property
              bool rawBinary = false;
              if (msg.Properties.ContainsKey(PoxEncoderMessageProperty.Name))
          {
             rawBinary = ((PoxEncoderMessageProperty)msg.Properties[PoxEncoderMessageProperty.Name]).RawBinary;
          }

    If the message is not empty (we have a body), we check whether there is a PoxEncoderMessageProperty present in the message. This property is a plain CLR class that is part of my extensions and has two significant properties: Name is a static, constant string value used as the key for the message properties collection and RawBinary is a Boolean instance value that contains and indicator for whether the encoder shall encode the data as XML or as raw binary data. The Message property collection is a simple dictionary of objects keyed by strings. The properties allow application-level code to interact with infrastructure-level code in the way illustrated by this property. Whenever I want the encoder to use its “raw binary” mode, I add this property to the message and the encoder can pick up the information.

          ArraySegment<byte> retval = new ArraySegment<byte>();
          byte[] buffer = bufferManager.TakeBuffer(maxMessageSize);
          if (!rawBinary)
          {
             // If we're rendering XML data, we construct a memory stream
             // over the output buffer, layer an XMLDictionaryWriter on top of it
             // and have the message write the body content into the buffer as XML.
             // The buffer is then wrapped into an array segment and returned.
             MemoryStream stream = new MemoryStream(buffer);
             XmlWriterSettings settings = new XmlWriterSettings();
             settings.OmitXmlDeclaration = true;
             settings.Indent = true;
             settings.Encoding = this. textEncoding;
             XmlWriter innerWriter = XmlWriter.Create(stream, settings);
             XmlDictionaryWriter writer = XmlDictionaryWriter.CreateDictionaryWriter(innerWriter, false);
             msg.WriteBodyContents(writer);
             writer.Flush();
             retval = new ArraySegment<byte>(buffer, 0, (int)stream.Position);
          }

    Next we take a buffer from the buffer manager and if we’re not in “raw binary” mode, we’ll construct a memory stream over the buffer, construct an XmlDictionaryWriter over that stream and ask the message to render its “body contents” into the writer and therefore into the memory stream and into the buffer. The “body contents” of a message is what would be the child nodes of the <soap:Body> element, if we were using that (but we don’t). Once the body contents have been written, we flush the writer to make sure that all buffered data is committed into the underlying stream and then construct the return value as an array segment over the buffer with the length of the bytes written to the stream.

          else
          {
             // If we're rendering raw binary data, we grab at most 'buffer.Length'
             // bytes from the binary content of the base64Binary element (if that
             // exists) and return the result wrapped into an array segment.
             XmlDictionaryReader dictReader = msg.GetReaderAtBodyContents();
             if (dictReader.NodeType == XmlNodeType.Element &&
                dictReader.LocalName == "base64Binary")
             {
                if (dictReader.Read() && dictReader.NodeType == XmlNodeType.Text)
                {
                   int size = dictReader.ReadContentAsBase64(buffer, 0, buffer.Length);
                   retval = new ArraySegment<byte>(buffer, 0, size);
                }
             }
          }
          return retval;
       }
    }
     

    If the “raw binary” mode is to be used, we are making a bit of an assumption inside the encoder. The assumption is that the body content consists of a single element named “base64Binary” and that its content is just that: base64 binary encoded content. That is of course the other side of the PoxBase64XmlStreamReader trick I explained in Part 5. For binary data we simply assume here that the body reader is our wrapper class and this is how arbitrary binary data is smuggled through the Indigo infrastructure. The array segment to be returned is constructed by reading the binary data into the buffer and setting the array segment length to the number of bytes we could get from the element content.

    The streamed version of WriteMessage is quite different:

    public override void WriteMessage(Message msg, System.IO.Stream stream)
    {
        try
        {
            if (!msg.IsEmpty)
            {
                // check RawBinary bit in the message property
                bool rawBinary = false;
                if (msg.Properties.ContainsKey(PoxEncoderMessageProperty.Name))
                {
                    rawBinary = ((PoxEncoderMessageProperty)msg.Properties[PoxEncoderMessageProperty.Name]).RawBinary;
                }
                if (!rawBinary)
                {
                    // If we're rendering XML, we layer an XMLDictionaryWriter over the
                    // output stream and have the message render its body content into
                    // that writer and therefore onto the stream.
                    XmlWriterSettings settings = new XmlWriterSettings();
                    settings.OmitXmlDeclaration = true;
                    settings.Indent = true;
                    settings.Encoding = this. textEncoding;
                    XmlWriter innerWriter = XmlWriter.Create(stream, settings);
                    XmlDictionaryWriter writer = XmlDictionaryWriter.CreateDictionaryWriter(innerWriter, false);
                    msg.WriteBodyContents(writer);
                    writer.Flush();
                }

    The first significant difference is that if we’re using streams, we will simply ignore empty messages and do nothing with them. In streaming mode, the transport will do any setup work required for  sending a message before invoking the encoder and ready the output network stream so that the encoder can write to it. When the encoder returns, the transport considers the write action done. So if we don’t write to the output stream, there’s no payload data hitting the wire and that happens to be what we want.

    If we have data and we’re not in “raw binary” mode, the encoder will construct an XmlDictionaryWriter over the supplied stream and have the message write its body contents to it. That’s all.

                Else
                {
                    // If we're rendering raw binary data, we grab chunks of at most 1MByte
                    // from the 'base64Binary' content element (if that exists) and write them
                    // out as binary data to the output stream. Chunking is done, because we
                    // have to assume that the body content is arbitrarily large. To optimize the
                    // behavior for large streams, we read and write concurrently and swap buffers.
                    XmlDictionaryReader dictReader = msg.GetReaderAtBodyContents();
                    if (dictReader.NodeType == XmlNodeType.Element && dictReader.LocalName == "base64Binary")
                    {
                        if (dictReader.Read() && dictReader.NodeType == XmlNodeType.Text)
                        {
                            byte[] buffer1 = new byte[1024*1024], buffer2 = new byte[1024*1024];
                            byte[] readBuffer = buffer1, writeBuffer = buffer2;
                           
                            int bytesRead = 0;
                            // read the first chunk into the read buffer
                            bytesRead = dictReader.ReadContentAsBase64(readBuffer, 0, readBuffer.Length);
                            do
                            {
                                // the abort condition for the loop is that we can't read
                                // any more bytes from the input because the base64Binary element is
                                // exhausted.
                                if (bytesRead > 0 )
                                {
                                    // make the last read buffer the write buffer
                                    writeBuffer = readBuffer;
                                    // write the write buffer to the output stream asynchronously
                                    IAsyncResult result = stream.BeginWrite(writeBuffer, 0, bytesRead,null,null);
                                    // swap the read buffer
                                    readBuffer = (readBuffer == buffer1) ? buffer2 : buffer1;
                                    // read a new chunk into the 'other' buffer synchronously
                                    bytesRead = dictReader.ReadContentAsBase64(readBuffer, 0, readBuffer.Length);
                                    // wait for the write operation to complete
                                    result.AsyncWaitHandle.WaitOne();
                                    stream.EndWrite(result);
                                }
                            }
                            while (bytesRead > 0);
                        }
                    }
                }
            }
        }
        catch
        {
            // the client may disconnect at any time, so that's an expected exception and absorbed.
        }
    }

    In streamed “raw binary” mode things get a bit more complicated. Under these circumstances we assume that the output we are sending is HUGE. The use-case I had in mind when I wrote this is the download of multi-GByte video recordings. Therefore I construct two 1MByte buffers that are used in turns to read a chunk of data from the source body reader (for which we make the same content assumption as for the buffered case: This is believed to be a PoxBase64XmlStreamReader compatible infoset) and asynchronously push the read data into the output stream.

    Because it may take a while to get a huge data stream to the other side, a lot of things can happen to the network connection during that time. Therefore the encoder fully expects that the network connection terminates unexpectedly. If that happens, we’ll catch and absorb the network exception and happily return to the caller as if we’re done.

    Compared to all the complexity of the WriteMessage  overloads, the respective ReadMessage methods look fairly innocent, simple, and similar:

    /// <summary>
    ///
    Reads an incoming array segment containing a message and
    /// wraps it with a buffered message. The assumption is that the incoming
    /// data stream is <i>not</i> a SOAP envelope, but rather an unencapsulated
    /// data item, may it be some raw binary, an XML document or HTML form
    /// postback data. This method is called if the inbound transfer mode of the
    /// transport is "buffered".
    /// </summary>
    ///
    <param name="buffer">Buffer to wrap</param>
    ///
    <param name="bufferManager">Buffer manager to help with allocating a copy</param>
    ///
    <returns>Buffered message</returns>
    public override Message ReadMessage(ArraySegment<byte> buffer, BufferManager bufferManager)
    {
       return new PoxBufferedMessage(buffer, bufferManager);
    }

    /// <summary>
    ///
    Reads an incoming stream containing a message and
    /// wraps it with a streamed message. The assumption is that the incoming
    /// data stream is <i>not</i> a SOAP envelope, but rather an unencapsulated
    /// data item, may it be some raw binary, an XML document or HTML form
    /// postback data. This method is called if the inbound transfer mode of the
    /// transport is "streamed".
    /// </summary>
    ///
    <param name="stream">Input stream</param>
    ///
    <param name="maxSizeOfHeaders">Maximum size of headers in bytes</param>
    ///
    <returns>Stream message</returns>
    public override Message ReadMessage(System.IO.Stream stream, int maxSizeOfHeaders)
    {
       return new PoxStreamedMessage(stream, maxSizeOfHeaders);
    }

    Both variants take the raw incoming data (whatever it is) and hand it to the PoxStreamMessage class or PoxBufferedMessage class that adopt the buffer or stream as their body content, respectively. I’ll explain those in Part 7.

    Happy New Year!

    Categories: Indigo