A flock of pigs has been doing aerobatics high up over Microsoft Campus in Redmond in the past three weeks. Neither City of Redmond nor Microsoft spokespeople returned calls requesting comments in time for this article. An Microsoft worker who requested anonymity and has seen the pigs flying overhead commented that "they are as good as the Blue Angels at Seafair, just funnier" and "they seem to circle over building 42 a lot, but I wouldn't know why".

In related news ...

We wrapped up the BizTalk Services "R11" CTP this last Thursday and put the latest SDK release up on http://labs.biztalk.net/. As you may or may not know, "BizTalk Services" is the codename for Microsoft's cloud-based Identity and Connectivity services - with a significant set of further services in the pipeline. The R11 release is a major milestone for the data center side of BizTalk Services, but we've also added several new client-facing features, especially on the Identity services. You can now authenticate using a certificate in addition to username and CardSpace authentication, we have enabled support for 3rd party managed CardSpace cards, and there is extended support for claims based authorization.

Now the surprising bit:

Only about an hour before we locked down the SDK on Thursday, we checked a sample into the samples tree that has a rather unusual set of prerequisites for something coming out of Microsoft:

Runtime: Java EE 5 on Sun Glassfish v2 + Sun WSIT/Metro (JAX-WS extensions), Tool: Netbeans 6.0 IDE.

The sample shows how to use the BizTalk Services Identity Security Token Service (STS) to secure the communication between a Java client and a Java service providing federated authentication and claims-based authorization.

The sample, which you can find in ./Samples/OtherPlatforms/StandaloneAccessControl/JavaEE5 once you installed the SDK, is a pure Java sample not requiring any of our bits on either the service or client side. The interaction with our services is purely happening on the wire.

If you are a "Javahead", it might seem odd that we're shipping this sample inside a Windows-only MSI installer and I will agree that that's odd. It's simply a function of timing and the point in time when we knew that we could get it done (some more on that below). For the next BizTalk Services SDK release I expect there to be an additional .jar file for the Java samples.

It's important to note that this isn't just a thing we did as a one-time thing and because we could. We have done a significant amount of work on the backend protocol implementations to start opening up a very broad set of scenarios on the BizTalk Services Connectivity services for platforms other than .NET. We already have a set of additional Java EE samples lined up for when we enable that functionality on the backend. However, since getting security and identity working is a prerequisite for making all other services work, that's where we started. There'll be more and there'll be more platform and language choice than Java down the road.

Just to be perfectly clear: Around here we strongly believe that .NET and the Windows Communication Foundation in particular is the most advanced platform to build services, irrespective of whether they are of the WS-* or REST variety. If you care about my personal opinion, I'll say that several months of research into the capabilities of other platforms has only reaffirmed that belief for me and I don't even need to put a Microsoft hat on to say that.

But we recognize and respect that there are a great variety of individual reasons why people might not be using .NET and WCF. The obvious one is "platform". If you run on Linux or Unix and/or if your deployment target is a Java Application Server, then your platform is very likely not .NET. It's something else. If that's your world, we still think that our services are something that's useful for your applications and we want to show you why. And it is absolutely not enough for us to say "here is the wire protocol documentation; go party!". Only Code is Truth.

I'm also writing "Only Code is Truth" also because we've found - perhaps not too surprisingly - that there is a significant difference between reading and implementing the WS-* specs and having things actually work. And here I get to the point where a round of public "Thank You" is due:

The Metro team over at Sun Microsystems has made a very significant contribution to making this all work. Before we started making changes to accommodate Java, there would have been very little hope for anyone to get this seemingly simple scenario to work. We had to make quite a few changes even though our service did follow the specs.

While we were adjusting our backend STS accordingly, the Sun Metro team worked on a set of issues that we identified on their end (with fantastic turnaround times) and worked those into their public nightly builds. The Sun team also 'promoted' a nightly build of Metro 1.2 to a semi-permanent download location (the first 1.2 build that got that treatment), because it is the build tested to successfully interop with our SDK release, even though that build is known to have some regressions for some of their other test scenarios. As they work towards wrapping up their 1.2 release and fix those other bugs, we’ll continue to test and talk to help that the interop scenarios keep working.

As a result of this collaboration, Metro 1.2 is going to be a better and more interoperable release for the Sun's customers and the greater Java community and BizTalk Services as well as our future identity products will be better and more interoperable, too. Win-Win. Thank you, Sun.

As a goodie, I put some code into the Java sample that might be useful even if you don't even care about our services. Since configuring the Java certificate stores for standalone applications can be really painful, I added some simple code that's using a week-old feature of the latest Metro 1.2 bits that allows configuring the Truststores/Keystores dynamically and pull the stores from the client's .jar at runtime. The code also has an authorization utility class that shows how to get and evaluate claims on the service side by pulling the SAML token out of the context and pulling the correct attributes from the token.

Have fun.

[By the way, this is not an April Fool's joke, in case you were wondering]

Check this out.

Having an Internet Service Bus up in the cloud is not very entertaining unless there are services in the bus. Therefore, I built one (and already showed some of the code basics) that’s hopefully fun to play with and will soon share the first version with you after some scrubbing and pending a few updates to the ISB that will optimize the authentication process. It’s a 0.1 version and an experiment. The code download should be ready in the next two weeks, including those adjustments. But you can actually play with parts of it today without compiling or installing anything. The info is at the bottom of this post.

To make matters really interesting, this sample not only shows how to plug a service into the cloud and call it from some Console app, but is a combo of two rather unusual hosts for WCF services: A Windows Live Messenger Add-In that acts as the server, and a Windows Vista Sidebar gadget that acts as the client.

Since the Silicon Valley scene is currently all over Twitter and clones of Twitter are apparently popping up somewhere every day, I thought I could easily provide fodder to the proponents of the alleged Microsoft tradition of purely relying on copying other’s ideas and clone them as well   Well, no, maybe not. This is a bit different.

TweetieBot is an example of a simple personal service. If you choose to host it, you own it, you run it, you control it. The data is held nowhere but on your personal machine and it’s using the BizTalk Services ISB to stick its head up into the cloud and at a stable endpoint so that its easily reachable for a circle of friends, bridging the common obstacles of dynamic IPs, firewalls and NAT. No need to use UPnP or open up ports on your router. If you choose to do so, you can encrypt traffic so that there’s no chance that anyone looking at our ISB nor anyone else can see the what’s actually going across the wire.

Right now, lots of the Web 2.0 world lives on the assumption that everything needs to live at central places and that community forms around ad-driven hubs. The mainframe folks had a similar stance in the 70s and 80s and then Personal Computers came along. The pendulum is always swinging and I have little doubt that it will swing back to “personal” once more and that the federation of personal services will seriously challenge the hub model once more.

So what does the sample do? As indicated, TweetieBot is a bot that plugs into a Windows Live Messenger using a simple Add-In. Bart De Smet has a brilliant summary for how to build such Add-Ins. When the Add-In is active and someone chats the bot, it answers politely and remembers the chat line, time and sender. The bird has a leaky long term memory, though. It forgets everything past the last 40 lines.

Where it gets interesting is that the Add-In can stick three endpoints into the BizTalk Services ISB:

• A Request/Response Web Service that allows retrieving the list of the last 40 (or less) “tweets” and also allows client to submit tweets programmatically.
• An RSS service that allows (right now) anyone to peek in to the chat log of the last 40 tweets.
• An Event service that allows subscribers to get real-time notifications whenever a new tweet is recorded.

The accompanying Sidebar Gadget, which is implemented using WPF, is a client for two of these services.

 When you drop the Gadget on the Sidebar, it will prompt for the IM address of the TweetieBot service you’d like to subscribe to. Once you’ve authenticated at the relay using your registered Information Card, the gadget will pull and show the current list of Tweets and subscribe to the Events service for real-time updates. And whenever someone chats the bot, the Sidebar gadget will immediately show the new entry. So even though the Gadget lives on some client machine that’s hidden between several layers of firewalls and behind NAT, it can actually get push-style event notifications through the cloud!

“How do I send events to clients?” must be one of the most frequent questions that I’ve been asked about Web Services in the past several years. Well, this is your answer right here.

While I’m still toying around with the code and the guys on the 1st floor in my building are doing some tweaks on the ISB infrastructure to make multi-endpoint authentication simpler, you can already play with the bot and help me a bit:

Using Windows Live Messenger you can chat (click here) tweetiebot@hotmail.com now. Drop a few lines. If the bot is online (which means that I’m not tinkering with it) it will reply. Then look at this RSS feed [1] and you can see what you and everyone else have been telling the bot recently. Enjoy.

[1] http://connect.biztalk.net/services/tweetiebot/tweetiebot%40hotmail.com/rss

I wrote a slightly Twitter-inspired, fun app over the weekend that's using the BizTalk Services Connectivity service and relay. In the spirit of Software+Services I'm going to give you half of it [for now]    You must have the BizTalk Services SDK installed to run the sample.

The server app, which I'm keeping to myself for the next few days as part of the experiment, is an extension (add-in) to Windows Live Messenger. The Messenger add-in monitors all chats with tweetiebot@hotmail.com and keeps circular buffer with the last 40 incoming messages. Using the client (which is in the attached archive), you can get a list of "Tweets" and add a new one (same as chatting)

[ServiceContract(Name = "TweetieBot", Namespace = http://samples.vasters.com/2007/05/tweetiebot)]
public interface ITweetieBot
{
  [OperationContract]
  IList<Tweet> GetTweets(DateTime? since);
  [OperationContract]
  void Tweet(string nickname, string text);
}

or you can subscribe to new tweets and get them as they arrive

[ServiceContract(Name = "TweetieEvents", Namespace = http://samples.vasters.com/2007/05/tweetiebot)]
public interface ITweetieEvents
{
  [OperationContract(IsOneWay=true)]
  void OnTweet(Tweet tweet);
}

The client application hooks up to the client (that lives right on my desktop machine) through the BizTalk Services ISB and the server fires events back through the ISB relay into the client as new tweets arrive. So when you run the attached client app, you'll find that it starts with a dump of the current log of the bot and then keeps spitting out events as they arrive.

The client is actually pretty simple. The EventsClient is the subscriber for the pub/sub service (ConnectionMode.RelayMulticast) that writes out the received events to the console. The rest all happens in Main (parsing an validating the command line argument) and in Run.

    class Program
    {
        class EventsClient : ITweetieEvents
        {
            public void OnTweet(Tweet tweet)
            {
                Console.WriteLine("[{0}] {1}:{2}", tweet.Time, tweet.User, tweet.Text);
            }
        }

        static void Main(string[] args)
        {
            string usageMessage = "Usage: IMBotClient <messenger-email-address>";
            if (args.Length == 0)
            {
                Console.WriteLine(usageMessage);
            }
            else
            {
                if (!Regex.IsMatch(args[0], @"^([\w\-\.]+)@((\[([0-9]{1,3}\.){3}[0-9]{1,3}\])|(([\w\-]+\.)+)([a-zA-Z]{2,4}))$"))
                {
                    Console.WriteLine(usageMessage);
                    Console.WriteLine("'{0}' is not a valid email address");
                }
                else
                {
                    Run(args[0]);
                }
            }
        }

        private static void Run(string emailAddress)
        {
            EndpointAddress serviceAddress =
                new EndpointAddress(String.Format(String.Format("sb://{0}/services/tweetiebot/{1}/service", 
                                    RelayBinding.DefaultRelayHostName, Uri.EscapeDataString(emailAddress))));
            EndpointAddress eventsAddress =
                new EndpointAddress(String.Format(String.Format("sb://{0}/services/tweetiebot/{1}/events",
                                    RelayBinding.DefaultRelayHostName, Uri.EscapeDataString(emailAddress))));

The URI scheme for services that hook into the ISB is "sb:" and the default address of the relay is encoded in the SDK assemblies. We set up two endpoints here. One for the client channel to fetch the initial list and one for the event subscriber. 

            RelayBinding relayBinding = new RelayBinding();
     
            ServiceHost eventsHost = new ServiceHost(typeof(EventsClient));
            RelayBinding eventBinding = new RelayBinding(RelayConnectionMode.RelayedMulticast);
            eventsHost.AddServiceEndpoint(typeof(ITweetieEvents), eventBinding, eventsAddress.ToString());
            eventsHost.Open();

            ChannelFactory<TweetieBotChannel> channelFactory = new ChannelFactory<TweetieBotChannel>(relayBinding, serviceAddress);
            TweetieBotChannel channel = channelFactory.CreateChannel();
            channel.Open();

The two *.Open() calls will each prompt for a CardSpace authentication, so you will have to be registered to run the sample. Once you have opened the channels (and my service is running), you'll be able to pull the list of current tweets. Meanwhile, whenever a new event pops up, the EventsClient above will write out a new line.

            IList<Tweet> tweets = channel.GetTweets(lastTime);
            foreach (Tweet tweet in tweets)
            {
                Console.WriteLine("[{0}] {1}:{2}", tweet.Time, tweet.User, tweet.Text);
            }

            Console.WriteLine("Press ENTER to quit at any time");
            Console.ReadLine();

            eventsHost.Close();
            channel.Close();
            channelFactory.Close();
        }        

For those of you who couldn't make it to MIX, here are the (Silverlight-) videos of the talks from the Connected Systems Division deep-linked to sessions.visitmix.com

• Don Box, Steve Maine: Navigating the Programmable Web
• Garrett Serack, Mike Jones, Pat Felsted: Enable Windows CardSpace and Information Cards on Your Web Site
• Kim Cameron and Panel: Digital Identity and the Psychology of Security

We love WS-* as much as we do love Web-Style services. I say "Web-style", full knowing that the buzzterm is REST. Since REST is an architectural style and not an implementation technology, it makes sense to make a distinction and, also, claiming complete RESTfulness for a system is actually a pretty high bar to aspire to. So in order to avoid monikers like POX or Lo-REST/Hi-REST, I just call it what it what this is all about to mere mortals whose don't have an advanced degree in HTTP Philosophy: Services that work like the Web - or Web-Style. That's not to say that a Web-Style service cannot be fully RESTful. It surely can be. But if all you want to do is GET to serve up data into mashups and manipulate your backend resources in some other way, that's up to you. Anyways....

Tomorrow at 10:00am (Session DEV03, Room Delfino 4101A), our resident Lo-REST/Hi-REST/POX/Web-Style Program Manager Steve Maine and our Architect Don Box will explain to you how to use the new Web-Style "Programmable Web" features that we're adding to the .NET Framework 3.5 to implement the server magic and the service-client magic to power all the user experience goodness you've seen here at MIX.

Navigating the Programmable Web

Speaker(s): Don Box - Microsoft, Steve Maine

Audience(s): Developer

RSS. ATOM. JSON. POX. REST. WS-*. What are all these terms, and how do they impact the daily life of a developer trying to navigate today’s programmable Web? Join us as we explore how to consume and create Web services using a variety of different formats and protocols. Using popular services (Flickr, GData, and Amazon S3) as case studies, we look at what it takes to program against these services using the Microsoft platform today and how that will change in the future.

Steve has a great analysis of what BizTalk Services means for Corzen and how he views it in the broader industry context.

It's been slashdotted and also otherwise widely discussed that Google has deprecated their SOAP API. A deadly blow for SOAP as people are speculating? Guess not.

What I find striking are the differences in the licenses between the AJAX API and the SOAP API. That's where the beef is. While the results obtained through the SOAP API can be used (for non-commercial purposes) practically in any way except that "you may not use the search results provided by the Google SOAP Search API service with an existing product or service that competes with products or services offered by Google.", the AJAX API is constrained to use with web sites with the terms of use stating that "The API is limited to allowing You to host and display Google Search Results on your site, and does not provide You with the ability to access other underlying Google Services or data."

The AJAX API is a Web service that works for Google because its terms of use are very prescriptive for how to build a service that ensures Google's advertising machine gets exposure and clicks. That's certainly a reasonable business decision, but has nothing to do with SOAP vs. REST or anything else technical. There's just no money in application-to-application messaging for Google (unless they'd actually set up an infrastructure to charge for software as a service and provide support and proper SLAs for it that is saying more than "we don't make any guarantees whatsoever") while there's a lot of money for them in being able to get lots and lots of people to give them a free spot on their own site onto which they can place their advertising. That's what their business is about, not software.

I was sad when "Indigo" and "Avalon" went away. It'd be great if we'd have a pool of cool legal-approved code-names for which we own the trademark rights and which we could stick to. Think Delphi or Safari. "Indigo" was cool insofar as it was very handy to refer to the technology set, but was removed far enough from the specifics that it doesn't create a sharply defined, product-like island within the larger managed-code landscape or has legacy connotations like "ADO.NET".  Also, my talks these days could be 10 minutes shorter if I could refer to Indigo instead of "Windows Communications Foundation". Likewise, my job title wouldn't have to have a line wrap on the business card of I ever spelled it out in full.

However, when I learned about the WinFX name going away (several weeks before the public announcement) and the new "Vista Wave" technologies (WPF/WF/WCF/WCS) being rolled up under the .NET Framework brand, I was quite happy. Ever since it became clear in 2004 that the grand plan to put a complete, covers-all-and-everything managed API on top (and on quite a bit of the bottom) of everything Windows would have to wait until siginificantly after Vista and that therefore the Win16>Win32>WinFX continuity would not tell the true story, that name made only limited sense to stick to. The .NET Framework is the #1 choice for business applications and a well established brand. People refer to themselves as being "dotnet" developers. But even though the .NET Framework covers a lot of ground and "Indigo", "Avalon", "InfoCard", and "Workflow" are overwhelmingly (or exclusively) managed-code based, there are still quite a few things in Windows Vista that still require using P/Invoke or COM/Interop from managed code or unmanaged code outright. That's not a problem. Something has to manage the managed code and there's no urgent need to rewrite entire subsystems to managed code if you only want to add or revise features. 

So now all the new stuff is now part of the .NET Framework. That is a good, good, good change. This says what it all is.

Admittedly confusing is the "3.0" bit. What we'll ship is a Framework 3.0 that rides on top of the 2.0 CLR and includes the 2.0 versions of the Base-Class Library, Windows Forms, and ASP.NET. It doesn't include the formerly-announced-as-to-be-part-of-3.0 technologies like VB9 (there you have the version number consistency flying out the window outright), C# 3.0, and LINQ. Personally, I think that it might be a tiny bit less confusing if the Framework had a version-number neutral name such as ".NET Framework 2006" which would allow doing what we do now with less potential for confusion, but only a tiny bit. Certainly not enough to stage a war over "2006" vs. "3.0".

It's a matter of project management reality and also one of platform predictability that the ASP.NET, or Windows Forms teams do not and should not ship a full major-version revision of their bits every year. They shipped Whidbey (2.0) in late 2005 and hence it's healthy for them to have boarded the scheduled-to-arrive-in-2007 boat heading to Orcas. We (the "WinFX" teams) subscribed to the Vista ship docking later this year and we bring great innovation which will be preinstalled on every copy of it. LINQ as well as VB9 and C# incorporating it on a language-level are very obviously Visual Studio bound and hence they are on the Orcas ferry as well. The .NET Framework is a steadily growing development platform that spans technologies from the Developer Division, Connected Systems, Windows Server, Windows Client, SQL Server, and other groups, and my gut feeling is that it will become the norm that it will be extended off-cycle from the Developer Division's Visual Studio and CLR releases. Whenever a big ship docks in the port, may it be Office, SQL, BizTalk, Windows Server, or Windows Client, and as more and more of the still-unmanaged Win32/Win64 surface area gets wrapped, augmented or replaced by managed-code APIs over time and entirely new things are added, there might be bits that fit into and update the Framework.  

So one sane way to think about the .NET Framework version number is that it merely labels the overall package and not the individual assemblies and components included within it. Up to 2.0 everything was pretty synchronized, but given the ever-increasing scale of the thing, it's good to think of that being a lucky (even if intended) coindicence of scheduling. This surely isn't the first time that packages were versioned independently of their components. There was and is no reason for the ASP.NET team to gratuitously recompile their existing bits with a new version number just to have the GAC look pretty and to create the illusion that everything is new - and to break Visual Studio compatibility in the process.

Of course, once we cover 100% of the Win32 surface area, we can rename it all into WinFX again   (just kidding)

[All the usual "personal opinion" disclaimers apply to this post]

Update: Removed reference to "Win64".

Just so that you know: In addition to the regular breakout sessions, we have a number of interactive chalk talks scheduled here at the Connected Systems Technical Learning Center in the Expo Hall. Come by.

This is my first TechEd! - as a Microsoft employee. It's of course not my first tech event in my new job (Egypt, Jordan, UK, France, Switzerland, Holland, Belgium, Denmark, Las Vegas/USA, Slovenia, and Israel are on the year-to-date list - on top of three long-distance commutes to Redmond), but the big TechEds are always special. It'll be fun. Come by the Connected Systems area in the exhibition hall and find me to chat if you are here in Boston.

Frankly, I didn't expect a Sunday night keynote to be nearly as well attended as it was, but it looks that experiment mostly worked. The theme of the keynote were Microsoft's 4 Core Promises for IT Pros and Developers nicely wrapped into a video story based on the TV show "24" and with that show's IT superwoman Chloe O'Brian (actress Mary Lynn Rajskub) up on stage with Bob Muglia (our team's VP far up above in my chain of command), who acted as the MC for the show. Finally we got an apology from a Hollywood character for all the IT idiocy the put up on screen. Thanks, Chloe.

Our team has a lot of very cool stuff to talk about at this show. The first highlight is John Justice's WCF Intro talk (Session CON208, Room 157ABC) today at 5:00pm with a "meet the team" panel Q&A session at the end. Block the time.

I may work for the firm, but ... As a good corporate citizen I just installed the Windows Update item that got pushed out to me. The "Windows Genuine Advantage Notification" tool that's supposed to notify me -- I am paraphrasing the decription that I clicked away already -- whether my copy of Windows is genuine and to help me acquire a legal copy if it finds out that its not (whatever that might do). I think I have a bit of an understanding why there's such a tool and why that Windows Genuine Advantage program exists. Like it or not, we make software that we ask people to buy.

But why, why, why does that particular update want me to reboot my machine after the install?  

Inside the big house....

Back in December of last year and about two weeks before I publicly announced that I will be working from Microsoft, I started a nine-part series on REST/POX* programming with Indigo WCF. (1, 2, 3, 4, 5, 6, 7, 8, 9). Since then, the WCF object model has seen quite a few feature and usability improvements across the board and those are significant enough to justify that I rewrite the entire series to get it up to the February CTP level and I will keep updating it through Vista/WinFX Beta2 and as we are marching towards our RTM. We've got a few changes/extensions in our production pipeline to make the REST/POX story for WCF v1 stronger and I will track those changes with yet another re-release of this series.

Except in one or two occasions, I haven't re-posted a reworked story on my blog. This here is quite a bit different, because of it sheer size and the things I learned in the process of writing it and developing the code along the way. So even though it is relatively new, it's already due for an end-to-end overhaul to represent my current thinking. It's also different, because I am starting to cross-post content to http://blogs.msdn.com/clemensv with this post; however http://friends.newtelligence.net/clemensv remains my primary blog since that runs my engine

Listening

The "current thinking" is of course very much influenced by now working for the team that builds WCF instead of being a customer looking at things from the outside. That changes the perspective quite a bit. One great insight I gained is how non-dogmatic and customer-oriented our team is. When I started the concrete REST/POX work with WCF back in last September (on the customer side still working with newtelligence), the extensions to the HTTP transport that enabled this work were just showing up in the public builds and they were sometimes referred to as the "Tim/Aaaron feature". Tim Ewald and Aaron Skonnard had beat the drums for having simple XML (non-SOAP) support in WCF so loudly that the team investigated the options and figured that some minimal changes to the HTTP transport would enable most of these scenarios**. Based on that feature, I wrote the set of dispatcher extensions that I've been presenting in the V1 of this series and newtellivision as the applied example did not only turn out to be a big hit as a demo, it also was one of many motivations to give the REST/POX scenario even deeper consideration within the team.

REST/POX is a scenario we think about as a first-class scenario alongside SOAP-based messaging - we are working with the ASP.NET Atlas team to integrate WCF with their AJAX story and we continue to tweak the core WCF product to enable those scenarios in a more straightforward fashion. Proof for that is that my talk (PPT here) at the MIX06 conference in Las Vegas two weeks ago was entirely dedicated to the non-SOAP scenarios.

What does that say about SOAP? Nothing. There are two parallel worlds of application-level network communication that live in peaceful co-existence:

The Faceless Web

The first scenario is the web as we know it. Almost. HTTP is an incredibly rich application protocol once you dig into RFC2616 and look at the methods in detail and consider response codes beyond 200 and 404. HTTP is strong because it is well-defined, widely supported and designed to scale, HTTP is weak because it is effectively constrained to request/response, there is no story for server-to-client notifications and it abstracts away the inherent reliability of the transmission-control protocol (TCP). These pros and cons lists are not exhaustive.

What REST/POX does is to elevate the web model above the "you give me text/html or */* and I give you application/x-www-form-urlencoded" interaction model. Whether the server punts up markup in the form of text/html or text/xml or some other angle-bracket dialect or some raw binary isn't too interesting. What's changing the way applications are built and what is really creating the foundation for, say, AJAX is that the path back to the server is increasingly XML'ised. PUT and POST with a content-type of text/xml is significantly different from application/x-www-form-urlencoded. What we are observing is the emancipation of HTTP from HTML to a degree that the "HT" in HTTP is becoming a misnomer. Something like IXTP ("Interlinked XML Transport Protocol" - I just made that up) would be a better fit by now.

The astonishing bit in this is that there has been been no fundamental technology change that has been driving this. The only thing I can identify is that browsers other than IE are now supporting XMLHTTP and therefore created the critical mass for broad adoption. REST/POX rips the face off the web and enables a separation of data and presentation in a way that mashups become easily possible and we're driving towards a point where the browser cache becomes more of an application repository than merely a place that holds cacheable collateral. When developing the newtellivision application I have spent quite a bit of time on tuning the caching behavior in a way that HTML and script are pulled from the server only when necessary and as static resources and all actual interaction with the backend services happens through XMLHTTP and in REST/POX style. newtellivision is not really a hypertext website, it's more like a smart client application that is delivered through the web technology stack.

Distributed Enterprise Computing

All that said, the significant investments in SOAP and WS-* that were made my Microsoft and industry partners such as Sun, IBM, Tibco and BEA have their primary justification in the parallel universe of highly interoperable, feature-rich intra and inter-application communication as well as in enterprise messaging. Even though there was a two-way split right through through the industry in the 1990s with one side adopting the Distributed Computing Environment (DCE) and the other side driving the Common Object Request Broker Architecture (CORBA), both of these camps made great advances towards rich, interoperable (within their boundaries) enterprise communication infrastructures. All of that got effectively killed by the web gold-rush starting in 1994/1995 as the focus (and investment) in the industry turned to HTML/HTTP and to building infrastructures that supported the web in the first place and everything else as a secondary consideration. The direct consequence of the resulting (even if big) technology islands hat sit underneath the web and the neglect of inter-application communication needs was that inter-application communication has slowly grown to become one of the greatest industry problems and cost factors. Contributing to that is that the average yearly number of corporate mergers and acquisitions has tripled compared to 10-15 years ago (even though the trend has slowed in recent years) and the information technology dependency of today's corporations has grown to become one of the deciding if not the deciding competitive factor for an ever increasing number of industries.

What we (the industry as a whole) are doing now and for the last few years is that we're working towards getting to a point where we're both writing the next chapter of the story of the web and we're fixing the distributed computing story at the same time by bringing them both onto a commonly agreed platform. The underpinning of that is XML; REST/POX is the simplest implementation. SOAP and the WS-* standards elevate that model up to the distributed enterprise computing realm.

If you compare the core properties of SOAP+WS-Adressing and the Internet Protocol (IP) in an interpretative fashion side-by-side and then also compare the Transmission Control Protocol (TCP) to WS-ReliableMessaging it may become quite clear to you what a fundamental abstraction above the networking stacks and concrete technology coupling the WS-* specification family has become. Every specification in the long list of WS-* specs is about converging and unifying formerly proprietary approaches to messaging, security, transactions, metadata, management, business process management and other aspects of distributed computing into this common platform.

Convergence

The beauty of that model is that it is an implementation superset of the web. SOAP is the out-of-band metadata container for these abstractions. The key feature of SOAP is SOAP:Header, which provides a standardized facility to relay the required metadata alongside payloads. If you are willing to constrain out-of-band metadata to one transport or application protocol, you don't need SOAP.

There is really very little difference between SOAP and REST/POX in terms of the information model. SOAP carries headers and HTTP carries headers. In HTTP they are bolted to the protocol layer and in SOAP they are tunneled through whatever carries the envelope. [In that sense, SOAP is calculated abuse of HTTP as a transport protocol for the purpose of abstraction.] You can map WS-Addressing headers from and to HTTP headers.

The SOAP/WS-* model is richer, more flexible and more complex. The SOAP/WS-* set of specifications is about infrastructure protocols. HTTP is an application protocol and therefore it is naturally more constrained - but has inherently defined qualities and features that require an explicit protocol implementation in the SOAP/WS-* world; one example is the inherent CRUD (create, read, update, delete) support in HTTP that is matched by the explicitly composed-on-top WS-Transfer protocol in SOAP/WS-*

The common platform is XML. You can scale down from SOAP/WS-* to REST/POX by putting the naked payload on the wire and rely on HTTP for your metadata, error and status information if that suits your needs. You can scale up from REST/POX to SOAP/WS-* by encapsulating payloads and leverage the WS-* infrastructure for all the flexibility and features it brings to the table. [It is fairly straightforward to go from HTTP to SOAP/WS-*, and it is harder to go the other way. That's why I say "superset".]

Doing the right thing for a given scenario is precisely what are enabling in WCF. There is a place for REST/POX for building the surface of the mashed and faceless web and there is a place for SOAP for building the backbone of it - and some may choose to mix and match these worlds. There are many scenarios and architectural models that suit them. What we want is

One Way To Program. 

* REST=REpresentational State Transfer; POX="Plain-Old XML" or "simple XML"

I kicked off quite a discussion with my recent post on O/R mapping. Some people think I am completely wrong, some say that it resonates with their experience, some say I wrote this in mean spirit, some are jubilating. I particularly liked the "Architectural Truthiness" post by David Ing and the comment by "Scott E" in my comments section who wrote:

I've hiked up the learning curve for Hibernate (the Java flavor) only to find that what time was saved in mapping basic CRUD functionality got eaten up by out-of-band custom data access (which always seems to be required) and tuning to get performance close to what it would have been with a more specialized, hand-coded DAL.

As always, it's a matter of perspective. Here is mine: I went down the O/R mapping route in a project in '98/'99 when my group at the company I was working for at the time was building a new business framework. We wrote a complete, fully transparent O/R mapper in C++. You walked up to a factory which dehydrated objects and you could walk along the association links and the object graph would either incrementally dehydrate or dehydrate in predefined segments. We had filtering capabilities that allowed to constrain 1:N collections with large N's, we could auto-resolve N:M relationships, had support for inheritance, and all that jazz. The whole framework was written with code generation in mind. Our generators were fed with augmented UML class diagrams and spit out the business layer, whereby we had a "partial classes" concept where we'd keep the auto-gen'd code in one tree and the parts that were supposed to be filled manually in another part of the code tree. Of course we'd preserve changes across re-gen's. Pure OO nirvana.

While the platforms have evolved substantially in the past 7 years, the fundamental challenges for transparent (fully abstracted) mapping of data to objects remain essentially the same.

• Given metadata to do the mapping, implementing CRUD functionality with an O/R mapper is quite easy. We had to put lots of extra metadata into our C++ classes back in the day, but with .NET and Java the metadata is all there and therefore CRUD O/R mapping is very low-hanging fruit on both platforms. That's why there's such a large number of projects and products.
• Defining and resolving associations is difficult. 1:N is hard, because you need to know what your N looks like. You don't want to dehydrate 10000 objects to find a value in one of them or to calculate a sum over a column. That's work that's, quite frankly, best left in the database. I realize that some people worry how that leads to logic bleeding into the database, but for me that's a discussion about pureness vs. pragmatism. If the N is small, grabbing all related objects is relatively easy - unless you support polymorphism, which forces the mapper into all sorts of weird query trees. 1:N is so difficult because an object model is inherently about records, while SQL is about sets. N:M is harder.
• "Object identity" is a dangerous lure. Every object has its own identifier. In memory that is its address, on disk that's some form of unique identifier. The idea of making the persistent identifier also the in-memory identifier often has the design consequence of an in-memory "running object table" with the goal of avoiding to load the same object twice but rather linking it appropriately into the object graph. That's a fantastic concept, but leads to all sort of interesting concurrency puzzles: What do you do if you happen to find an object you have already loaded as you resolve an 1:N association and realize that the object has meanwhile changed on disk? Another question is what the scope of the object identity is. Per appdomain/process, per machine or even a central object server (hope not)?
• Transactions are hard. Databases are doing a really good job with data concurrency management, especially with stored procedures. If you are loading and managing data as object-graphs, how do you manage transaction isolation? How do you identify the subtree that's being touched by a transaction? How do you manage rollbacks? What is a transaction, anyways?
• Changing the underlying data model is hard. I've run into several situations where existing applications had to be, with the customer willing to put money on the table, be integrated with existing data models. O/R mapping is relatively easy of the data model falls out of the object model. If an existing data model bubbles up against an object model, you often end up writing a DAL or the O/R in stored procedures.
• Reporting and data aggregation is hard. I'll use an analogy for that: It's really easy to write an XPath query against an XML document, but it is insanely difficult to do the same navigating the DOM.

That said, I am not for or against O/R mapping. There are lots of use cases with a lot of CRUD work where O/R saves a lot of time. However, it is a leaky abstraction. In fact is is so leaky that we ended up not using all that much of the funkyness we put into our framework, because "special cases" kept popping up. I am pointing out that there are a lot of fundamental differences between what an RDBMS does with data and how OOP treats data. The discussion is in part a discussion about ISAM vs. RDBMS.

The number of brain cycles that need to be invested for a clean O/R mapping of a complex object model in the presence of the fundamental challenges I listed here (and that list isn't exhaustive) are not automatically less than for a plain-old data layer. It may be more. YMMV.

Now you can (and some already have) ask how all of that plays with LINQ and, in particular, DLINQ. Mind that I don't work in the LINQ team, but I think to be observing a subtle but important difference between LINQ and O/R*: 

• O/R is object->relational mapping.
• LINQ is relational->object mapping.

LINQ acknowledges the relational nature of the vast majority of data, while O/R attempts to deny it. LINQ speaks about entities, relations and queries and maps result-sets into the realm of objects, even cooking up classes on the fly if it needs to. It's bottom up and the data (from whatever source) is king. Objects and classes are just tooling. For O/R mapping, the database is just tooling.