This session is a followup to the Service Bus session that I did at the build conference and explains advanced usage patterns:

From //build in Anaheim

Our team’s Development Manager MK (Murali Krishnaprasad) and me were interviewed by Michael Washam on May 2011 CTP release of Windows Azure AppFabric. We discuss new technologies such as Topics, Queues, Subscriptions and how this relates to doing async development in the cloud.

Republished from Channel 9

My PDC10 session is available online (it was pre-recorded). I talk about the new ‘Labs’ release that we released into the datacenter this week and about a range of future capabilities that we’re planning for Service Bus. Some of those future capabilities that are a bit further out are about bringing back some popular capabilities from back in the .NET Services incubation days (like Push and Service Orchestration), some are entirely new.

One important note about the new release at http://portal.appfabriclabs.com – for Service Bus, this is a focused release that provides mostly only new features and doesn’t provide the full capability scope of the production system and SDK. The goal here is to provide insight into an ongoing development process and opportunity for feedback as we’re continuing to evolve AppFabric. So don’t derive any implications from this release on what we’re going to do with the capabilities already in production.

Click here to go to the talk.

Juval Löwy’s very successful WCF book is now available in its third edition – and Juval asked me to update the foreword this time around. It’s been over three years since I wrote the foreword to the first edition and thus it was time for an update since WCF has moved on quite a bit and the use of it in the customer landscape and inside of MS has deepened where we’re building a lot of very interesting products on top of the WCF technology across all businesses – not least of which is the Azure AppFabric Service Bus that I work on and that’s entirely based on WCF services.

You can take a peek into the latest edition at the O’Reilly website and read my foreword if you care. To be clear: It’s the least important part of the whole book :-)

In case you need a refresher or update about the things me and our team work on at Microsoft, go here for a very recent and very good presentation by my PM colleague Maggie Myslinska from TechEd Australia 2010 about Windows Azure AppFabric with Service Bus demos and a demo of the new Access Control V2 CTP

I put the slides for my talks at NT Konferenca 2010 on SkyDrive. The major difference from my APAC slides is that I had to put compute and storage into one deck due to the conference schedule, but instead of purely consolidating and cutting down the slide count,  I also incorporated some common patterns coming out from debates in Asia and added slides on predictable and dynamic scaling as well as on multitenancy. Sadly, I need to rush through all that in 45 minutes today.

My office neighbor, our Service Bus Test Lead Vishal Chowdhary put together a bundle of code and documentation for how to use Service Bus with Server AppFabric and IIS 7.5. Here: http://code.msdn.microsoft.com/ServiceBusDublinIIS

… seht Euch mal die Wa an, wie die Wa ta kann. Auf der Mauer auf der Lauer sitzt ‘ne kleine Wa!.

It’s a German children’s song. The song starts out with “… sitzt ‘ne kleine Wanze” (bedbug) and with each verse you leave off a letter: Wanz, Wan, Wa, W, – silence.

I’ll do the same here, but not with a bedbug:

Let’s sing:

<soap:Envelope xmlns:soap=”” xmlns:wsaddr=”” xmlns:wsrm=”” xmlns:wsu=”” xmlns:app=””>
   <soap:Header>
         <addr:Action>http://tempuri.org/1.0/Status.set</addr:Action>
         <wsrm:Sequence>
              <wsrm:Identifier>urn:session-id</wsrm:Identifier>
              <wsrm:MessageNumber>5</wsrm:MessageNumber>
          </wsrm:Sequence>
          <wsse:Security xmlns:wsse=”…”>
               <wsse:BinarySecurityToken ValueType=" http://tempuri.org#CustomToken "
                                         EncodingType="...#Base64Binary" wsu:Id=" MyID ">
                          FHUIORv...
                </wsse:BinarySecurityToken>
               <ds:Signature>
                  <ds:SignedInfo>
                      <ds:CanonicalizationMethod Algorithm=" http://www.w3.org/2001/10/xml-exc-c14n#"/>
                      <ds:SignatureMethod Algorithm=" http://www.w3.org/2000/09/xmldsig#md5"/ > 
                      <ds:Reference URI="#MsgBody">
                            <ds:DigestMethod  Algorithm=" http://www.w3.org/2000/09/xmldsig#md5"/>  
                            <ds:DigestValue>LyLsF0Pi4wPU...</ds:DigestValue>
                      </ds:Reference>
                 </ds:SignedInfo>  
                 <ds:SignatureValue>DJbchm5gK...</ds:SignatureValue>
                 <ds:KeyInfo> 
                  <wsse:SecurityTokenReference> 
                    <wsse:Reference URI="#MyID"/>
                   </wsse:SecurityTokenReference>
               </ds:KeyInfo>
             </ds:Signature>
         </wsse:Security>
         <app:ResponseFormat>Xml</app:ResponseFormat>
         <app:Key wsu:Id=”AppKey”>27729912882….</app:Key>
    <soap:Header>
    <soap:Body wsu:Id=”MyId”>
          <app:status>Hello, I’m good</app:status>
     </soap:Body>
</soap:Envelope>

Not a very pretty song, I’ll admit. Let’s drop a some stuff. Let’s assume that we don’t need to tell the other party that we’re looking to give it an MD5 signature, but let’s say that’s implied and so were the canonicalization algorithm. Let’s also assume that the other side already knows the security token and the key. Since we only have a single signature digest here and yield a single signature we can just collapse to the signature value. Heck, you may not even know about what that all means. Verse 2:

<soap:Envelope xmlns:soap=”” xmlns:wsaddr=”” xmlns:wsrm=”” xmlns:wsu=”” xmlns:app=””>
   <soap:Header>
         <addr:Action>http://tempuri.org/1.0/Status.set</addr:Action>
         <wsrm:Sequence>
              <wsrm:Identifier>urn:session-id</wsrm:Identifier>
              <wsrm:MessageNumber>5</wsrm:MessageNumber>
          </wsrm:Sequence>
          <wsse:Security xmlns:wsse=”…”>
               <ds:Signature>
                  <ds:SignatureValue>DJbchm5gK...</ds:SignatureValue>
             </ds:Signature>
         </wsse:Security>
         <app:ResponseFormat>Xml</app:ResponseFormat>
         <app:Key wsu:Id=”AppKey”>27729912882….</app:Key>
    <soap:Header>
    <soap:Body wsu:Id=”MyId”>
          <app:status>Hello, I’m good</app:status>
     </soap:Body>
</soap:Envelope>

Better. Now let’s strip all these extra XML namespace decorations since there aren’t any name collisions as far as I can see. We’ll also collapse the rest of the security elements into one element since there’s no need for three levels of nesting with a single signature. Verse 3:

<Envelope>
   <Header>
         <Action>http://tempuri.org/1.0/Status.set</Action>
         <Sequence>
              <Identifier>urn:session-id</Identifier>
              <MessageNumber>5</MessageNumber>
          </Sequence>
          <SignatureValue>DJbchm5gK...</SignatureValue>
          <ResponseFormat>Xml</ResponseFormat>
          <Key>27729912882….</Key>
    <Header>
    <Body>
       <status>Hello, I’m good</status>
     </Body>
</Envelope>

Much better. The whole angle-bracket stuff and the nesting seems semi-gratuitous and repetitive here, too. Let’s make that a bit simpler. Verse 4:

         Action=http://tempuri.org/1.0/Status.set
         Sequence-Identifier=urn:session-id
         Sequence-MessageNumber=5
         SignatureValue=DJbchm5gK...
         ResponseFormat=Xml
         Key=27729912882….
         status=Hello, I’m good

Much, much better. Now let’s get rid of that weird URI up there and split up the action and the version info, make some of these keys are little more terse and turn that into a format that’s easily transmittable over HTTP. By what we have here application/www-form-urlencoded would probably be best. Verse 5:

         method=Status.set
         &v=1.0
         &session_key=929872172..
         &call_id=5
         &sig=DJbchm5gK...
         &format=Xml
         &api_key=27729912882….
         &status=Hello,%20I’m%20good

Oops. Facebook’s Status.set API. How did that happen? I thought that was REST?

Now play the song backwards. The “new thing” is largely analogous to where we started before the WS* Web Services stack and its CORBA/DCE/DCOM predecessors came around and there are, believe it or not, good reasons for having of that additional “overhead”. A common way to frame message content and the related control data, a common way to express complex data structures and distinguish between data domains, a common way to deal with addressing in multi-hop or store-and-forward messaging scenarios, an agreed notion of sessions and message sequencing, a solid mechanism for protecting the integrity of messages and parts of messages. This isn’t all just stupid.

It’s well worth discussing whether messages need to be expressed as XML 1.0 text on the wire at all times. I don’t think they need to and there are alternatives that aren’t as heavy. JSON is fine and encodings like the .NET Binary Encoding or Fast Infoset are viable alternatives as well. It’s also well worth discussing whether WS-Security and the myriad of related standards that were clearly built by security geniuses for security geniuses really need to be that complicated or whether we could all live with a handful of simple profiles and just cut out 80% of the options and knobs and parameters in that land.

I find it very sad that the discussion isn’t happening. Instead, people use the “REST” moniker as the escape hatch to conveniently ignore any existing open standard for tunnel-through-HTTP messaging and completely avoid the discussion.

It’s not only sad, it’s actually a bit frustrating. As one of the people responsible for the protocol surface of the .NET Service Bus, I am absolutely not at liberty to ignore what exists in the standards space. And this isn’t a mandate handed down to me, but something I do because I believe it’s the right thing to live with the constraints of the standards frameworks that exist.

When we’re sitting down and talk about a REST API, were designing a set of resources – which may result in splitting a thing like a queue into two resources, head and tail - and then we put RFC2616 on the table and try to be very precise in picking the appropriate predefined HTTP method for a given semantic and how the HTTP 2xx, 3xx, 4xx, 5xx status codes map to success and error conditions. We’re also trying to avoid inventing new ways to express things for which standards exists. There’s a standard for how to express and manage lists with ATOM and APP and hence we use that as a foundation. We use the designed extension points to add data to those lists whenever necessary.

When we’re designing a RPC SOAP API, we’re intentionally trying to avoid inventing new protocol surface and will try to leverage as much from the existing and standardized stack as we possibly can – at a minimum we’ll stick with established patterns such as the Create/GetInfo/Renew/Delete patterns for endpoint factories with renewal (which is used in several standards). I’ll add that we are – ironically - a bit backlogged on the protocol documentation for our SOAP endpoints and have more info on the REST endpoint in the latest SDK, but we’ll make that up in the near future.

So - can I build “REST” (mind the quotes) protocols that are as reduced as Facebook, Twitter, Flickr, etc? Absolutely. There wouldn’t be much new work. It’s just a matter of how we put messages on and pluck message off the wire. It’s really mostly a matter of formatting and we have a lot of the necessary building blocks in the shipping WCF bits today. I would just omit a bunch of decoration as things go out and make a bunch of assumptions on things that come in.

I just have a sense that I’d be hung upside down from a tree by the press and the blogging, twittering, facebooking community if I, as someone at Microsoft, wouldn’t follow the existing open and agreed standards or at least use protocols that we’ve published under the OSP and instead just started to do my own interpretative dance - even if that looked strikingly similar to what the folks down in the Valley are doing. At the very least, someone would call it a rip-off.

What do you think? What should I/we do?

Earlier today I hopefully gave a somewhat reasonable, simple answer to the question "What is a Claim?" Let's try the same with "Token":

In the WS-* security world, "Token" is really just a another name the security geniuses decided to use for "Handy package for all sorts of security stuff". The most popular type of token is the SAML (just say "samel") token. If the ladies and gentlemen designing and writing security platform infrastructure and frameworks are doing a good job you might want to know about the existence of such a thing, but otherwise be blissfully ignorant of all the gory details.

Tokens are meant to be a thing that you need to know about in much the same way you need to know about ... ummm... rebate coupons you can cut out of your local newspaper or all those funny books that you get in the mail. I have really no idea how the accounting works behind the scenes between the manufacturers and the stores, but it really doesn't interest me much, either. What matters to me is that we get $4 off that jumbo pack of diapers and we go through a lot of those these days with a 9 month old baby here at home. We cut out the coupon, present it at the store, four bucks saved. Works for me.

A token is the same kind of deal. You go to some (security) service, get a token, and present that token to some other service. The other service takes a good look at the token and figures whether it 'trusts' the token issuer and might then do some further inspection; if all is well you get four bucks off. Or you get to do the thing you want to do at the service. The latter is more likely, but I liked the idea for a moment.

Remember when I mentioned the surprising fact that people lie from time to time when I wrote about claims? Well, that's where tokens come in. The security stuff in a token is there to keep people honest and to make 'assertions' about claims. The security dudes and dudettes will say "Err, that's not the whole story", but for me it's good enough. It's actually pretty common (that'll be their objection) that there are tokens that don't carry any claims and where the security service effectively says "whoever brings this token is a fine person; they are ok to get in". It's like having a really close buddy relationship with the boss of the nightclub when you are having troubles with the monsters guarding the door. I'm getting a bit ahead of myself here, though.

In the post about claims I claimed that "I am authorized to approve corporate acquisitions with a transaction volume of up to $5Bln". That's a pretty obvious lie. If there was such a thing as a one-click shopping button for companies on some Microsoft Intranet site (there isn't, don't get any ideas) and I were to push it, I surely should not be authorized to execute the transaction. The imaginary "just one click and you own Xigg" button would surely have some sort of authorization mechanism on it.

I don't know what Xigg is assumed to be worth these days, but there is actually be a second authorization gate to check. I might indeed be authorized to do one-click shopping for corporate acquisitions, but even with my made-up $5Bln limit claim, Xigg may just be worth more that I'm claiming I'm authorized to approve. I digress.

How would the one-click-merger-approval service be secured? It would expect some sort of token that absolutely, positively asserts that my claim "I am authorized to approve corporate acquisitions with a transaction volume of up to $5Bln" is truthful and the one-click-merger-approval service would have to absolutely trust the security service that is making that assertion. The resulting token that I'm getting from the security service would contain the claim as an attribute of the assertion and that assertion would be signed and encrypted in mysterious (for me) yet very secure and interoperable ways, so that I can't tamper with it as much as I look at the token while having it in hands.

The service receiving the token is the only one able to crack the token (I'll get to that point in a later post) and look at its internals and the asserted attributes. So what if I were indeed authorized to spend a bit of Microsoft's reserves and I were trying to acquire Xigg at the touch of a button and, for some reason I wouldn't understand, the valuation were outside my acquisition limit? That's the service's job. It'd look at my claim, understand that I can't spend more than $5Bln and say "nope!" - and it would likely send email to SteveB under the covers. Trouble.

Bottom line: For a client application, a token is a collection of opaque (and mysterious) security stuff. The token may contain an assertion (saying "yep, that's actually true") about a claim or a set of claims that I am making. I shouldn't have to care about the further details unless I'm writing a service and I'm interested in some deeper inspection of the claims that have been asserted. I will get to that.

Before that, I notice that I talked quite a bit about some sort of "security service" here. Next post...

If you ask any search engine "What is a Claim?" and you mean the sort of claim used in the WS-* security space, you'll likely find an answer somewhere, but that answer is just as likely buried in a sea of complex terminology that is only really comprehensible if you have already wrapped your head around the details of the WS-* security model. I would have thought that by now there would be a simple and not too technical explanation of the concept that's easy to find on the Web, but I haven't really had success finding one. 

So "What is a Claim?" It's really simple.

A claim is just a simple statement like "I am Clemens Vasters", or "I am over 21 years of age", or "I am a Microsoft employee", or "I work in the Connected Systems Division", or "I am authorized to approve corporate acquisitions with a transaction volume of up to $5Bln". A claim set is just a bundle of such claims.

When I walk up to a service with some client program and want to do something on the service that requires authorization, the client program sends a claim set along with the request. For the client to know what claims to send along, the service lets it know about its requirements in its policy.

When a request comes in, this imaginary (U.S.) service looks at the request knowing "I'm a service for an online game  promoting alcoholic beverages!". It then it looks at the claim set, finds the "I am over 21 years of age" claim and thinks "Alright, I think we got that covered".

The service didn't really care who was trying to get at the service. And it shouldn't. To cover the liquor company's legal behind, they only need to know that you are over 21. They don't really need to know (and you probably don't want them to know) who is talking to them. From the client's perspective that's a good thing, because the client is now in a position to refuse giving out (m)any clues about the user's identity and only provide the exact data needed to pass the authorization gate. Mind that the claim isn't the date of birth for that exact reason. The claim just says "over 21".

Providing control over what claims are being sent to a service (I'm lumping websites, SOAP, and REST services all in the same bucket here) is one of the key reasons why Windows CardSpace exists, by the way. The service asks for a set of claims, you get to see what is being asked for, and it's ultimately your personal, interactive decision to provide or refuse to provide that information.

The only problem with relying on simple statements (claims) of that sort is that people lie. When you go to the Jack Daniel's website, you are asked to enter your date of birth before you can proceed. In reality, it's any date you like and an 10-year old kid is easily smart enough to figure that out.

All that complex security stuff is mostly there to keep people honest. Next time ...

We love WS-* as much as we do love Web-Style services. I say "Web-style", full knowing that the buzzterm is REST. Since REST is an architectural style and not an implementation technology, it makes sense to make a distinction and, also, claiming complete RESTfulness for a system is actually a pretty high bar to aspire to. So in order to avoid monikers like POX or Lo-REST/Hi-REST, I just call it what it what this is all about to mere mortals whose don't have an advanced degree in HTTP Philosophy: Services that work like the Web - or Web-Style. That's not to say that a Web-Style service cannot be fully RESTful. It surely can be. But if all you want to do is GET to serve up data into mashups and manipulate your backend resources in some other way, that's up to you. Anyways....

Tomorrow at 10:00am (Session DEV03, Room Delfino 4101A), our resident Lo-REST/Hi-REST/POX/Web-Style Program Manager Steve Maine and our Architect Don Box will explain to you how to use the new Web-Style "Programmable Web" features that we're adding to the .NET Framework 3.5 to implement the server magic and the service-client magic to power all the user experience goodness you've seen here at MIX.

Navigating the Programmable Web

Speaker(s): Don Box - Microsoft, Steve Maine

Audience(s): Developer

RSS. ATOM. JSON. POX. REST. WS-*. What are all these terms, and how do they impact the daily life of a developer trying to navigate today’s programmable Web? Join us as we explore how to consume and create Web services using a variety of different formats and protocols. Using popular services (Flickr, GData, and Amazon S3) as case studies, we look at what it takes to program against these services using the Microsoft platform today and how that will change in the future.

Christian Weyer shows off the few lines of pretty straightforward WCF code & config he needed to figure out in order to set up a duplex conversation through BizTalk Services.

Steve has a great analysis of what BizTalk Services means for Corzen and how he views it in the broader industry context.

I've spent the last 1 1/2 weeks doing one of the most fun (seriously) work assignments that each Program Manager of our team gets to do every once in a while: Servicing. So until yesterday night (I'm flying home to Germany today) I was in charge of ASP.NET Web Services and Remoting. An even though these technologies have been out there for quite a while now, there are still situations where stuff breaks and people are scratching their heads wondering what's going on. Overall, it was a very, very quiet time on the bug front though.

The one issue that we found on my watch is that you can configure ASP.NET Web Forms in a way that it breaks ASP.NET Web Services (ASMX). We are shipping one ASP.NET Web Page (.aspx) with ASMX and that unfortunate interaction manages to break that exact page with an error that's hard to figure out unless you have substantial ASP.NET knowledge and you have enough confidence in that knowledge to not trust us ;-)

If you globally override the autoEventWireup setting in the <page/> config element in the ASP.NET web.config and set that to "false", the DefaultWsdlHelpGenator.aspx page (which sits in the CONFIG directory of the Framework) becomes very unhappy and fails with a NullReferenceException, stating "Object reference not set to an instance of an object." and showing you some code that's definitely not yours.

What happened? Well, the file is missing a directive that overrides the override of the default. The fix is to go edit the DefaultWsdlHelpGenerator.aspx file and add the line:

<%@ Page AutoEventWireup="true" %>

That will fix the problem.

Now, the big question is: "Will you put that into a service pack?". While there's obviously a bug here, the answer is, in this particular case, "don't know yet". Replacing or editing that particular file is a potentially very impactful surgery done on the patched system given that the file is there in source code and in the config directory because you are supposed to be able to change it. Could we touch changed files? Probably not. Could we touch unchanged files? Probably? So how would you surface the difference and make sure that the systems we couldn't patch would not suffer from the particular bug? What's the test impact for the code and for the service pack or patch installer? How many people are actually using that ASP.NET config directive AND are hosting ASMX services in the same application and/or scope? Is it actually worth doing that? Making changes in code that has already shipped and is part of the Framework is serious business, since you are potentially altering the behavior of millions of machines all at once. So that part is definitely not done in an "agile" way, but takes quite a bit of consideration, while it takes just 10 seconds and notepad.exe for you.

It's been slashdotted and also otherwise widely discussed that Google has deprecated their SOAP API. A deadly blow for SOAP as people are speculating? Guess not.

What I find striking are the differences in the licenses between the AJAX API and the SOAP API. That's where the beef is. While the results obtained through the SOAP API can be used (for non-commercial purposes) practically in any way except that "you may not use the search results provided by the Google SOAP Search API service with an existing product or service that competes with products or services offered by Google.", the AJAX API is constrained to use with web sites with the terms of use stating that "The API is limited to allowing You to host and display Google Search Results on your site, and does not provide You with the ability to access other underlying Google Services or data."

The AJAX API is a Web service that works for Google because its terms of use are very prescriptive for how to build a service that ensures Google's advertising machine gets exposure and clicks. That's certainly a reasonable business decision, but has nothing to do with SOAP vs. REST or anything else technical. There's just no money in application-to-application messaging for Google (unless they'd actually set up an infrastructure to charge for software as a service and provide support and proper SLAs for it that is saying more than "we don't make any guarantees whatsoever") while there's a lot of money for them in being able to get lots and lots of people to give them a free spot on their own site onto which they can place their advertising. That's what their business is about, not software.

Indigo The Windows Communication Foundation's RC1 bits are now live. RC means "Release Candidate" and our team is really, really serious about this release being as close to what we intend to ship as we can ever get. Our database view with unresolved code-defects is essentially empty (there is a not more of a handful of small fixes for very esoteric scenarios that we're still doing for RTM). The time of breaking changes is absolutely and finally over for "WCF Version 1".

The team is very excited about this. There's lots of joy in the hallways. We're getting close to being done. Remember when you saw the first WS-* specs popping up out there some 6 years ago? That's when this thing was started. You can just imagine how pumped the testers, developers and program managers are around here. And even though I am new to the family, I get to celebrate a little too. Greatness.

Get the RC1 for the .NET Framework 3.0 with the WCF bits from here:
http://www.microsoft.com/downloads/details.aspx?FamilyId=19E21845-F5E3-4387-95FF-66788825C1AF&displaylang=en 

There's one little issue with the Visual Studio Tools aligned with that version, so it will take another day or so until those get uploaded.

As always, if you find problems, tell us: http://connect.microsoft.com/wcf

I've been quoted as to have said so at TechEd and I'll happily repeat it: "XML is the assembly language of Web 2.0", even though some (and likely some more) disagree. James Speer writes "Besides, Assembly Language is hard, XML isn’t." , which I have to disagree with.

True, throwing together some angle brackets isn't the hardest thing in the world, but beating things into the right shape is hard and probably even harder than in assembly. Yes, one can totally, after immersing oneself in the intricacies of Schema, write complex types and ponder for days and months about the right use of attributes and elements. It's absolutely within reach for a WSDL zealot to code up messages, portTypes and operations by hand. But please, if you think that's the right way to do things, I also demand that you write and apply your security policy in angle bracket notation from the top of your head and generate WCF config from that using svcutil instead of just throwing a binding together, because XML is so easy. Oh? Too hard? Well, it turns out that except for our developers and testers who are focusing on getting these mappings right, nobody on our product team would probably ever even want to try writing such a beast by hand for any code that sits above the deep-down guts of our stack. This isn't the fault of the specifications (or people here being ignorant), but it's a function of security being hard and the related metadata being complex. Similar things, even though the complexity isn't quite as extreme there, can be said about the other extensions to the policy framework such as WS-RM Policy or those for WS-AT.

As we're getting to the point where full range of functionality covered by WS-* specifications is due to hit the mainstream by us releasing WCF and our valued competitors releasing their respective implementations, hand-crafted contracts will become increasingly meaningless, because it's beyond the capacity of anyone whose job it is to build solutions for their customers to write complete set of contracts that not only ensures simple data interop but also protocol interop. Just as there were days that all you needed was assembly and INT21h to write a DOS program (yikes) or knowledge of "C" alongside stdio.h and fellows to write anything for everthing, things are changing now in the same way in Web Services land. Command of XSD and WSDL is no longer sufficient, all the other stuff is just as important to make things work.

Our WCF [DataContract] doesn't support attributes. That's a deliberate choice because we want to enforce simplicity and enhance interoperability of schemas. We put an abstraction over XSD and limit the control over it, because we want to simplify the stuff that goes across the wire. We certainly allow everyone to use the XmlSerializer with all of it's attribute based fine-grained control over schema, even though there are quite a few Schema constructs that even that doesn't support when building schema from such metadata. If you choose to, you can just ignore all of our serialization magic and fiddle with the XML Infoset outright and supply your own schema. However, XML and Schema are specifications that everyone and their dog wanted to get features into and Schema is hopelessly overengineered. Ever since we all (the industry, not only MS) boarded the SOAP/WS train, we're debating how to constrain the features of that monster to a reasonable subset that makes sense and the debate doesn't want to end.

James writes that he "take[s] a lot of care in terms of elements vs. attributes and mak[es] sure the structure of the XML is business-document-like", which only really makes sense if XML documents used in WS scenarios were meant for immediate human consumption, which they're not.

We want to promote a model that is simple and consistent to serialize to and from on any platform and that things like the differentiation between attributes and elements doesn't stand in the way of allowing a 1:1 mapping into alternate, non-XML serialization formats such as JSON or what-have-you (most of which don't care about that sort of differentiation).   James' statement about "business-document-like" structures is also interesting considering EDIFACT, X.12 or SWIFT, all of which only know records, fields and values, and don't care about that sort of subtle element/attribute differentation, either. (Yes, no of those might be "hip" any more, but they are implemented and power a considerable chunk of the world economy's data exchange).

By now, XML is the foundation for everything that happens on the web, and I surely don't want to have it go away. But have arrived at the point where matters have gotten so complicated that a layer of abstraction over pretty much all things XML has become a necessity for everyone who makes their money building customer solutions and not by teaching or writing about XML. In my last session at TechEd, I asked a room of about 200 people "Who of you hand-writes XSLT transforms?" 4 hands. "Who of you used to hand-write XSLT transforms?" 40+ hands. I think it's safe to assume that a bunch of those folks who have sworn off masochism and no longer hand-code XSLT are now using tools like the BizTalk Mapper or Altova's MapForce, which means that XSL/T is alive and kicking, but only downstairs in the basement. However, the abstractions that these tools provide also allow bypassing XSLT altogether and generate the transformation logic straight into compiled C++, Java, or C# code, which is what MapForce offers.  WSDL is already walking down that path.

Mark, I care deeply about the hobbyist who writes some code on the side, the programmer who works from 9-5 and has a life and just as deeply about those who work 24/7 and about everybody in between ;-)

That said: now that we're getting close to being done with the "this vs. that" debate, we can most certainly figure out the "how can we optimize the programming experience" story. For very many people I've talked to in the past 4 years or so, reducing complexity is an important thing. I firmly believe that we can do enterprise messaging and Web-Style/Lo-REST/POX with a single technology stack that scales up and down in terms of its capabilities.  

Since I take that you are worried about code-bloat on the app-level, how would you think about the following client-side one-liners?

• T data = Pox.Get<T>("myCfg")
• T data = Pox.Get<T>("myCfg", new Uri("/customer/8929", UriKind.Relative));
• T data = Pox.Get<T>("myCfg", new Uri("http: //example.com/customer/8929"));
• T data = Pox.Get<T>(new Uri("http: //example.com/customer/8929"));
• U reply = Pox.Put<T,U>( new Uri("http: //example.com/customer/8929"), data, ref location));
• U reply = Pox.Post<T,U>( new Uri("http: //example.com/customer/"), data, out location));
• Pox.Delete(settings, new Uri("http: //example.com/customer/8929"));

Whereby "myCfg" refers to a set of config to specify security, proxies, and so forth; settings would refer to an in-memory object with the same reusable info. Our stack lets me code that sort of developer experience in a quite straightforward fashion and I can throw SOAPish WS-Transfer under it and make the call flow on a reliable, routed TCP session with binary encoding without changing the least bit.

If I am still missing your point in terms of ease of use and line count, make a wish, Mark. :-)

Inside the big house....

Back in December of last year and about two weeks before I publicly announced that I will be working from Microsoft, I started a nine-part series on REST/POX* programming with Indigo WCF. (1, 2, 3, 4, 5, 6, 7, 8, 9). Since then, the WCF object model has seen quite a few feature and usability improvements across the board and those are significant enough to justify that I rewrite the entire series to get it up to the February CTP level and I will keep updating it through Vista/WinFX Beta2 and as we are marching towards our RTM. We've got a few changes/extensions in our production pipeline to make the REST/POX story for WCF v1 stronger and I will track those changes with yet another re-release of this series.

Except in one or two occasions, I haven't re-posted a reworked story on my blog. This here is quite a bit different, because of it sheer size and the things I learned in the process of writing it and developing the code along the way. So even though it is relatively new, it's already due for an end-to-end overhaul to represent my current thinking. It's also different, because I am starting to cross-post content to http://blogs.msdn.com/clemensv with this post; however http://friends.newtelligence.net/clemensv remains my primary blog since that runs my engine ;-)

Listening

The "current thinking" is of course very much influenced by now working for the team that builds WCF instead of being a customer looking at things from the outside. That changes the perspective quite a bit. One great insight I gained is how non-dogmatic and customer-oriented our team is. When I started the concrete REST/POX work with WCF back in last September (on the customer side still working with newtelligence), the extensions to the HTTP transport that enabled this work were just showing up in the public builds and they were sometimes referred to as the "Tim/Aaaron feature". Tim Ewald and Aaron Skonnard had beat the drums for having simple XML (non-SOAP) support in WCF so loudly that the team investigated the options and figured that some minimal changes to the HTTP transport would enable most of these scenarios**. Based on that feature, I wrote the set of dispatcher extensions that I've been presenting in the V1 of this series and newtellivision as the applied example did not only turn out to be a big hit as a demo, it also was one of many motivations to give the REST/POX scenario even deeper consideration within the team.

REST/POX is a scenario we think about as a first-class scenario alongside SOAP-based messaging - we are working with the ASP.NET Atlas team to integrate WCF with their AJAX story and we continue to tweak the core WCF product to enable those scenarios in a more straightforward fashion. Proof for that is that my talk (PPT here) at the MIX06 conference in Las Vegas two weeks ago was entirely dedicated to the non-SOAP scenarios.

What does that say about SOAP? Nothing. There are two parallel worlds of application-level network communication that live in peaceful co-existence:

The Faceless Web

The first scenario is the web as we know it. Almost. HTTP is an incredibly rich application protocol once you dig into RFC2616 and look at the methods in detail and consider response codes beyond 200 and 404. HTTP is strong because it is well-defined, widely supported and designed to scale, HTTP is weak because it is effectively constrained to request/response, there is no story for server-to-client notifications and it abstracts away the inherent reliability of the transmission-control protocol (TCP). These pros and cons lists are not exhaustive.

What REST/POX does is to elevate the web model above the "you give me text/html or */* and I give you application/x-www-form-urlencoded" interaction model. Whether the server punts up markup in the form of text/html or text/xml or some other angle-bracket dialect or some raw binary isn't too interesting. What's changing the way applications are built and what is really creating the foundation for, say, AJAX is that the path back to the server is increasingly XML'ised. PUT and POST with a content-type of text/xml is significantly different from application/x-www-form-urlencoded. What we are observing is the emancipation of HTTP from HTML to a degree that the "HT" in HTTP is becoming a misnomer. Something like IXTP ("Interlinked XML Transport Protocol" - I just made that up) would be a better fit by now.

The astonishing bit in this is that there has been been no fundamental technology change that has been driving this. The only thing I can identify is that browsers other than IE are now supporting XMLHTTP and therefore created the critical mass for broad adoption. REST/POX rips the face off the web and enables a separation of data and presentation in a way that mashups become easily possible and we're driving towards a point where the browser cache becomes more of an application repository than merely a place that holds cacheable collateral. When developing the newtellivision application I have spent quite a bit of time on tuning the caching behavior in a way that HTML and script are pulled from the server only when necessary and as static resources and all actual interaction with the backend services happens through XMLHTTP and in REST/POX style. newtellivision is not really a hypertext website, it's more like a smart client application that is delivered through the web technology stack.

Distributed Enterprise Computing

All that said, the significant investments in SOAP and WS-* that were made my Microsoft and industry partners such as Sun, IBM, Tibco and BEA have their primary justification in the parallel universe of highly interoperable, feature-rich intra and inter-application communication as well as in enterprise messaging. Even though there was a two-way split right through through the industry in the 1990s with one side adopting the Distributed Computing Environment (DCE) and the other side driving the Common Object Request Broker Architecture (CORBA), both of these camps made great advances towards rich, interoperable (within their boundaries) enterprise communication infrastructures. All of that got effectively killed by the web gold-rush starting in 1994/1995 as the focus (and investment) in the industry turned to HTML/HTTP and to building infrastructures that supported the web in the first place and everything else as a secondary consideration. The direct consequence of the resulting (even if big) technology islands hat sit underneath the web and the neglect of inter-application communication needs was that inter-application communication has slowly grown to become one of the greatest industry problems and cost factors. Contributing to that is that the average yearly number of corporate mergers and acquisitions has tripled compared to 10-15 years ago (even though the trend has slowed in recent years) and the information technology dependency of today's corporations has grown to become one of the deciding if not the deciding competitive factor for an ever increasing number of industries.

What we (the industry as a whole) are doing now and for the last few years is that we're working towards getting to a point where we're both writing the next chapter of the story of the web and we're fixing the distributed computing story at the same time by bringing them both onto a commonly agreed platform. The underpinning of that is XML; REST/POX is the simplest implementation. SOAP and the WS-* standards elevate that model up to the distributed enterprise computing realm.

If you compare the core properties of SOAP+WS-Adressing and the Internet Protocol (IP) in an interpretative fashion side-by-side and then also compare the Transmission Control Protocol (TCP) to WS-ReliableMessaging it may become quite clear to you what a fundamental abstraction above the networking stacks and concrete technology coupling the WS-* specification family has become. Every specification in the long list of WS-* specs is about converging and unifying formerly proprietary approaches to messaging, security, transactions, metadata, management, business process management and other aspects of distributed computing into this common platform.

Convergence

The beauty of that model is that it is an implementation superset of the web. SOAP is the out-of-band metadata container for these abstractions. The key feature of SOAP is SOAP:Header, which provides a standardized facility to relay the required metadata alongside payloads. If you are willing to constrain out-of-band metadata to one transport or application protocol, you don't need SOAP.

There is really very little difference between SOAP and REST/POX in terms of the information model. SOAP carries headers and HTTP carries headers. In HTTP they are bolted to the protocol layer and in SOAP they are tunneled through whatever carries the envelope. [In that sense, SOAP is calculated abuse of HTTP as a transport protocol for the purpose of abstraction.] You can map WS-Addressing headers from and to HTTP headers.

The SOAP/WS-* model is richer, more flexible and more complex. The SOAP/WS-* set of specifications is about infrastructure protocols. HTTP is an application protocol and therefore it is naturally more constrained - but has inherently defined qualities and features that require an explicit protocol implementation in the SOAP/WS-* world; one example is the inherent CRUD (create, read, update, delete) support in HTTP that is matched by the explicitly composed-on-top WS-Transfer protocol in SOAP/WS-*

The common platform is XML. You can scale down from SOAP/WS-* to REST/POX by putting the naked payload on the wire and rely on HTTP for your metadata, error and status information if that suits your needs. You can scale up from REST/POX to SOAP/WS-* by encapsulating payloads and leverage the WS-* infrastructure for all the flexibility and features it brings to the table. [It is fairly straightforward to go from HTTP to SOAP/WS-*, and it is harder to go the other way. That's why I say "superset".]

Doing the right thing for a given scenario is precisely what are enabling in WCF. There is a place for REST/POX for building the surface of the mashed and faceless web and there is a place for SOAP for building the backbone of it - and some may choose to mix and match these worlds. There are many scenarios and architectural models that suit them. What we want is

One Way To Program. 

* REST=REpresentational State Transfer; POX="Plain-Old XML" or "simple XML"

The stack trace below (snapshot taken at a breakpoint in [WebMethod] "HelloWorld") shows that I am having quite a bit of programming fun these days. Server-side ASP.NET hooked up to a MSMQ listener.  

simpleservicerequestinweb.dll!SimpleServiceRequestInWeb.Hello.HelloWorld() Line 53 C#
system.web.services.dll!System.Web.Services.Protocols.LogicalMethodInfo.Invoke(System.Object target, System.Object[] values) + 0x92 bytes 
system.web.services.dll!System.Web.Services.Protocols.WebServiceHandler.Invoke() + 0x9e bytes 
system.web.services.dll!System.Web.Services.Protocols.WebServiceHandler.CoreProcessRequest() + 0x142 bytes 
system.web.services.dll!System.Web.Services.Protocols.SyncSessionlessHandler.ProcessRequest(System.Web.HttpContext context) + 0x6 bytes 
system.web.dll!CallHandlerExecutionStep.System.Web.HttpApplication+IExecutionStep.Execute() + 0xb4 bytes 
system.web.dll!System.Web.HttpApplication.ExecuteStep(System.Web.HttpApplication.IExecutionStep step, bool completedSynchronously) + 0x58 bytes 
system.web.dll!System.Web.HttpApplication.ResumeSteps(System.Exception error) + 0xfa bytes 
system.web.dll!System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(System.Web.HttpContext context, System.AsyncCallback cb, System.Object extraData) + 0xe3 bytes 
system.web.dll!System.Web.HttpRuntime.ProcessRequestInternal(System.Web.HttpWorkerRequest wr) + 0x1e7 bytes 
system.web.dll!System.Web.HttpRuntime.ProcessRequest(System.Web.HttpWorkerRequest wr) + 0xb0 bytes 
newtelligence.enterprisetools.dll!newtelligence.EnterpriseTools.Msmq.MessageQueueAsmxDispatcher.MessageReceived(System.Object sender = {newtelligence.EnterpriseTools.Msmq.MessageQueueListener}, newtelligence.EnterpriseTools.Msmq.MessageReceivedEventArgs ea = {newtelligence.EnterpriseTools.Msmq.MessageReceivedEventArgs}) Line 33 C#
newtelligence.enterprisetools.dll!newtelligence.EnterpriseTools.Msmq.MessageQueueListener.ReceiveLoop() Line 305 + 0x2b bytes C#

The little series I am currently writing here on my blog has inspired me to write way too more code than actually necessary to get my point across ;-) So by now I've got my own MSMQ transport for WSE 2.0 (yes, I know that others have written that already, but I am shooting for a "enterprise strength" implementation), a WebRequest/WebResponse pair to smuggle under arbitrary ASMX proxies and I am more than halfway done with a server-side host for MSMQ-to-ASMX (spelled out: ASP.NET Web Services).

What bugs me is that WSE 2.0's messaging model is "asynchronous only" and that it always performs a push/pull translation and that there is no way to push a message through to a service on the receiving thread. Whenever I grab a message from the queue and put it into my SoapTransport's "Dispatch()" method, the message gets queued up in an in-memory queue and that is then, on a concurrent thread, pulled (OnReceiveComplete) by the SoapReceivers collection and submitted into ProcessMessage() of the SoapReceiver (like any SoapService derived implementation) matching the target endpoint. So while I can dequeue from MSMQ within a transaction scope (ServiceDomain), that transaction scope doesn't make it across onto the thread that will actually execute the action inside the SoapReceiver/SoapService.

So now I am sitting here, contemplating and trying to figure out a workaround that doesn't require me to rewrite a big chunk of WSE 2.0 (which I am totally not shy of if that is what it takes). Transaction marshaling, thread synchronization, ah, I love puzzles. Once I am know how to solve this and have made the adjustments, I'll post the queue listener I promised to wrap up the series. The other code I've written in the process will likely surface in some other way.

"My Lists", "My Photos", "My Profile" .... sounds all very familiar over there in MSN Spaces. So ... roll in the Web service interfaces, please.

I get emails like that very frequently. I have some news.

Short story: Microsoft is still willing and working to publish the application that I presented at TechEd Europe (see Benjamin's report) and they keep telling me that it will come out. Apparently there is a lot of consensus building to be done to get a big sample application out of the door. So there's nothing to be found on msdn, yet.

Little known secret: There are 15 lucky indivduals who have already received (hand-delivered) the Proseware code as a technical preview under a non-disclosure agreement. Because we (newtelligence) designed and wrote the sample application, we have permission to distribute the complete sample to participants of our SOA workshops and seminars.

So if you want to get yours hands on it, all you need to do is to send mail to training@newtelligence.com to sign up for one of the public events [Next published date is Dec 1-3, and the event is held in German, unless we get swamped with international inquiries] or you send email to the same address asking for an on-site workshop delivery. At this time, we (and MS) bind the code sample to workshop attendance so that you really understand why the application was built like it's built and that you fully understand the guidance that the application implicitly and explicitly carries (and doesn't carry).

Perspective 1:

I am not a fan of the way WS-Eventing is relying on WS-Addressing's EPR binding rules to correlate subscription responses (containing a parameterized "wse:SubscriptionManager" EPR holding the subscription identifier in wsa:ReferenceParameters, see table 5, line 24-26 in WS-Eventing) with subsequent renewals, unsubscribe and status inquiry operations and forces the most central information bit for the respective operation ("Which subscription do you wish to renew?") into the header of the respective message (wse:Identifier) instead of having it in the body.

So for the "GetStatus" operation, you end up with a single body element "GetStatus" that doesn't carry any content [see table 8, lines 19-21 and line 24 in WS-Eventing]. I know that XML lost everything intuitive about it a long time back, but the way this works is really counterintuitive and just doesn't look right. I would expect <GetStatus>uri:Subscription-Identifier</GetStatus>.

Perspective 2:

Now on the other hand! this is actually a sound approach if I were looking at the said SubscriptionManager EPR as a resource. What the wsa:ReferenceParameter does to the EPR is that it binds it uniquely to the respective subscription (section 1.3 of the spec mandates this).

What's confusing here (and not really a good name choice in my view) is that the wse:SubscriptionManager EPR does NOT only point to the subscription manager endpoint, but rather bindsall the way through to a specific subscription. Once the binding process to that subscription is done, the requsted action is then executed on the bound resource.

Ok ... I am sorry ... too abstract? I'll rephrase.

What I am saying is that WS-Eventing is an example showing how messages aren't necessarily targeted at the thing with [WebMethod] on top of it, but they may indeed be targeted at something more specific like a database record. So the binding of the endpoint reference is not a matter of the client stopping at http://somewhere/blahblah.asmx but is only complete when the wse:Identifier header is evaluated on the service-side and insideblahblah.asmx, resolved against the subscription database and the actual message target, the respective subscription record, is found. Once the EPR is fully resolved to yield the message target (the record), <GetStatus/> indeed becomes a parameterless operation and the body does not have to carry further content.

EPR = Moniker ;-)

Whenever you start thinking stuff is stable, it turns out that it is not the case. I am trying to implement a WS-Eventing compliant service and of course I ran into the issue that that specification sits on the August 2004 edition (and W3C submission) of WS-Addressing while WSE 2.0 sits on the March 2004 edition of WS-Addressing. To implement WS-Eventing correctly, I would now have to write a WS-Adressing implementation parallel to the one existing in WSE 2.0, because - of course - the August 2004 edition sports a new namespace and has a subtly different schema. Unfortunately, the March 2004 edition of WS-Addressing  is so fundamental for WSE 2.0 that routing and security and everything would sit on the March version while my own eventing functionality and nothing else would ride on the August version at the same time and in the same message. Of course that seems just totally wrong.  WSE 2.1, please!

I might be blind to not have seen that before, but this here hit me over the third Guinness at an Irish Pub while answering a sudden technical question from my buddy Bart:

<wsa:ReplyTo xmlns:wsa="http://schemas.xmlsoap.org/ws/2004/03/addressing">
 <wsa:Address>http://server/service_second_intermediary</wsa:Address>
 <wsa:ReferenceProperties>
  <wsa:ReplyTo>
   <wsa:Address>http://server/service_first_intermediary</wsa:Address>
   <wsa:ReferenceProperties>
    <wsa:ReplyTo>
     <wsa:Address>http://server/service_outer_caller</wsa:Address>
    </wsa:ReplyTo>
   </wsa:ReferenceProperties>
  </wsa:ReplyTo>
 </wsa:ReferenceProperties>
</wsa:ReplyTo>

Read the EPR binding rules section 2.3 in the WS-Addressing spec and you'll find out just like me how distributed "call-stacks" work with WS-Addressing, if your choice of communication pattern is the far more flexible duplex (or here) pattern for datagram-based message conversations instead of the rather simplistic request/response model. Of course, any endpoint-reference can be stacked in the same way. I always wondered where the (deprecated) WS-Routing "path" went, which allowed specifying source routes. I think I ran into it.

We've built FABRIQ, we've built Proseware. We have written seminar series about Web Services Best Practices and Service Orientation for Microsoft Europe. I speak about services and aspects of services at conferences around the world. And at all events where I talk about Services, I keep hearing the same question: "Enough of the theory, how do I do it?"

Therefore we have announced a seminar/workshop around designing and building service oriented systems that puts together all the things we've found out in the past years about how services can be built today and on today's Microsoft technology stack and how your systems can be designed for with migration to the next generation Microsoft technlogy stack in mind. Together with our newtelligence Associates, we are offering this workshop for in-house delivery at client sites world-wide and are planning to announce dates and locations for central, "open for all" events soon.

If you are interested in inviting us for an event at your site, contact Bart DePetrillo, or write to sales@newtelligence.com. If you are interested in participating at a central seminar, Bart would like to hear about it (no obligations) so that we can select reasonable location(s) and date(s) that fit your needs.