The basic idea of the Enterprise Service Bus paints a wonderful picture of a harmonious coexistence, integration, and collaboration of software services. Services for a particular general cause are built or procured once and reused across the Enterprise by ways of publishing them and their capabilities in a corporate services repository from where they can be discovered. The repository holds contracts and policy that allows dynamically generating functional adapters to integrate with services. Collaboration and communication is virtualized through an intermediary layer that knows how to translate messages from and to any other service hooked into the ESB like a babel fish in the Hitchhiker’s Guide to the Galaxy. The ESB is a bus, meaning it aspires to be a smart, virtualizing, mediating, orchestrating messaging substrate permeating the Enterprise, providing uniform and mediated access anytime and anywhere throughout today’s global Enterprise. That idea is so beautiful, it rivals My Little Pony. Sadly, it’s also about as realistic. We tried regardless.

As with many utopian ideas, before we can get to the pure ideal of an ESB, there’s some less ideal and usually fairly ugly phase involved where non-conformant services are made conformant. Until they are turned into WS-* services, any CICS transaction and SAP BAPI is fronted with a translator and as that skinning renovation takes place, there’s also some optimization around message flow, meaning messages get batched or de-batched, enriched or reduced. In that phase, there was also learning of the value and lure of the benefits of central control. SOA Governance is an interesting idea to get customers drunk on. That ultimately led to cheating on the ‘B’. When you look around and look at products proudly carrying the moniker ‘Enterprise Service Bus’ you will see hubs. In practice, the B in ESB is mostly just a lie. Some vendors sell ESB servers, some even sell ESB appliances. If you need to walk to a central place to talk to anyone, it’s a hub. Not a bus.

Yet, the bus does exist. The IP network is the bus. It turns out to suit us well on the Internet. Mind that I’m explicitly talking about “IP network” and not “Web” as I do believe that there are very many useful protocols beyond HTTP. The Web is obviously the banner example for a successful implementation of services on the IP network that does just fine without any form of centralized services other than the highly redundant domain name system.

Centralized control over services does not scale in any dimension. Intentionally creating a bottleneck through a centrally controlling committee of ESB machines, however far scaled out, is not a winning proposition in a time where every potential or actual customer carries a powerful computer in their pockets allowing to initiate ad-hoc transactions at any time and from anywhere and where we see vehicles, machines and devices increasingly spew out telemetry and accept remote control commands. Central control and policy driven governance over all services in an Enterprise also kills all agility and reduces the ability to adapt services to changing needs because governance invariably implies process and certification. Five-year plan, anyone?

If the ESB architecture ideal weren’t a failure already, the competitive pressure to adopt direct digital interaction with customers via Web and Apps, and therefore scale up not to the scale of the enterprise, but to scale up to the scale of the enterprise’s customer base will seal its collapse.

Service Orientation

While the ESB as a concept permeating the entire Enterprise is dead, the related notion of Service Orientation is thriving even though the four tenets of SOA are rarely mentioned anymore. HTTP-based services on the Web embrace explicit message passing. They mostly do so over the baseline application contract and negotiated payloads that the HTTP specification provides for. In the case of SOAP or XML-RPC, they are using abstractions on top that have their own application protocol semantics. Services are clearly understood as units of management, deployment, and versioning and that understanding is codified in most platform-as-a-service offerings.

That said, while explicit boundaries, autonomy, and contract sharing have been clearly established, the notion of policy-driven compatibility – arguably a political addition to the list to motivate WS-Policy as the time – has generally been replaced by something even more powerful: Code. JavaScript code to be more precise. Instead of trying to tell a generic client how to adapt to service settings by ways of giving it a complex document explaining what switches to turn, clients now get code that turns the switches outright. The successful alternative is to simply provide no choice. There’s one way to gain access authorization for a service, period. The “policy” is in the docs.

The REST architecture model is service oriented – and I am not meaning to imply that it is so because of any particular influence. The foundational principles were becoming common sense around the time when these terms were coined and as the notion of broadly interoperable programmable services started to gain traction in the late 1990s – the subsequent grand dissent that arose was around whether pure HTTP was sufficient to build these services, or whether the ambitious multi-protocol abstraction for WS-* would be needed. I think it’s fairly easy to declare the winner there.

Federated Autonomous Services

Windows Azure, to name a system that would surely be one to fit the kind of solution complexity that ESBs were aimed at, is a very large distributed system with a significant number of independent multi-tenant services and deployments that are spread across many data centers. In addition to the publicly exposed capabilities, there are quite a number of “invisible” services for provisioning, usage tracking and analysis, billing, diagnostics, deployment, and other purposes.  Some components of these internal services integrate with external providers. Windows Azure doesn’t use an ESB. Windows Azure is a federation of autonomous services.

The basic shape of each of these services is effectively identical and that’s not owing, at least not to my knowledge, to any central architectural directive even though the services that shipped after the initial wave certainly took a good look at the patterns that emerged. Practically all services have a gateway whose purpose it is to handle and dispatch and sometimes preprocess incoming network requests or sessions and a backend that ultimately fulfills the requests. The services interact through public IP space, meaning that if Service Bus wants to talk to its SQL Database backend it is using a public IP address and not some private IP. The Internet is the bus. The backend and its structure is entirely a private implementation matter.  It could be a single role or many roles.

Any gateway’s job is to provide network request management, which includes establishing and maintaining sessions, session security and authorization, API versioning where multiple variants of the same API are often provided in parallel, usage tracking, defense mechanisms, and diagnostics for its areas of responsibility. This functionality is specific and inherent to the service. And it’s not all HTTP. SQL database has a gateway that speaks the Tabular Data Stream protocol (TDS) over TCP, for instance, and Service Bus has a gateway that speaks AMQP and the binary proprietary Relay and Messaging protocols.

Governance and diagnostics doesn’t work by putting a man in the middle and watching the traffic coming by, which is akin to trying the tell whether a business is healthy by counting the trucks going to their warehouse. Instead we are integrating the data feeds that come out of the respective services and are generated fully knowing the internal state, and concentrate these data streams, like the billing stream, in yet other services that are also autonomous and have their own gateways. All these services interact and integrate even though they’re built by a composite team far exceeding the scale of most Enterprise’s largest projects, and while teams run on separate schedules where deployments into the overall system happen multiple times daily. It works because each service owns its gateway, is explicit about its versioning strategy, and has a very clear mandate to honor published contracts, which includes explicit regression testing. It would be unfathomable to maintain a system of this scale through a centrally governed switchboard service like an ESB.

Well, where does that leave “ESB technologies” like BizTalk Server? The answer is simply that they’re being used for what they’re commonly used for in practice. As a gateway technology. Once a service in such a federation would have to adhere to a particular industry standard for commerce, for instance if it would have to understand EDIFACT or X.12 messages sent to it, the Gateway would employ an appropriate and proven implementation and thus likely rely on BizTalk if implemented on the Microsoft stack. If a service would have to speak to an external service for which it would have to build EDI exchanges, it would likely be very cost effective to also use BizTalk as the appropriate tool for that outbound integration. Likewise, if data would have to be extracted from backend-internal message traffic for tracking purposes and BizTalk’s BAM capabilities would be a fit, it might be a reasonable component to use for that. If there’s a long running process around exchanging electronic documents, BizTalk Orchestration might be appropriate, if there’s a document exchange involving humans then SharePoint and/or Workflow would be a good candidate from the toolset.

For most services, the key gateway technology of choice is HTTP using frameworks like ASP.NET, Web API, probably paired with IIS features like application request routing and the gateway is largely stateless.

In this context, Windows Azure Service Bus is, in fact, a technology choice to implement application gateways. A Service Bus namespace thus forms a message bus for “a service” and not for “all services”. It’s as scoped to a service or a set of related services as an IIS site is usually scoped to one or a few related services. The Relay is a way to place a gateway into the cloud for services where the backend resides outside of the cloud environment and it also allows for multiple systems, e.g. branch systems, to be federated into a single gateway to be addressed from other systems and thus form a gateway of gateways. The messaging capabilities with Queues and Pub/Sub Topics provide a way for inbound traffic to be authorized and queued up on behalf of the service, with Service Bus acting as the mediator and first line of defense and where a service will never get a message from the outside world unless it explicitly fetches it from Service Bus. The service can’t be overstressed and it can’t be accessed except through sending it a message.

The next logical step on that journey is to provide federation capabilities with reliable handoff of message between services, meaning that you can safely enqueue a message within a service and then have Service Bus replicate that message (or one copy in the case of pub/sub) over to another service’s Gateway – across namespaces and across datacenters or your own sites, and using the open AMQP protocol. You can do that today with a few lines of code, but this will become inherent to the system later this year.

This session explains how to secure Service Bus using the Access Control Service. This is also an extension session for my session at BUILD, but watching the BUILD session is not a strict prerequisite.

This session is a followup to the Service Bus session that I did at the build conference and explains advanced usage patterns:

My PDC10 session is available online (it was pre-recorded). I talk about the new ‘Labs’ release that we released into the datacenter this week and about a range of future capabilities that we’re planning for Service Bus. Some of those future capabilities that are a bit further out are about bringing back some popular capabilities from back in the .NET Services incubation days (like Push and Service Orchestration), some are entirely new.

One important note about the new release at http://portal.appfabriclabs.com – for Service Bus, this is a focused release that provides mostly only new features and doesn’t provide the full capability scope of the production system and SDK. The goal here is to provide insight into an ongoing development process and opportunity for feedback as we’re continuing to evolve AppFabric. So don’t derive any implications from this release on what we’re going to do with the capabilities already in production.

Click here to go to the talk.

We know that there’s a number of you out there who have outbound firewall rules in place on your corporate infrastructures that are based on IP address whitelisting. So if you want to make Service Bus or Access Control work, you need to know where our services reside. Below is the current list of where the services are deployed as of today, but be aware that it’s in the nature of cloud infrastructures that things can and will move over time.IP address whitelisting strategy isn’t really the right thing to do when the other side is a massively multi-tenant infrastructure such as Windows Azure (or any other public cloud platform, for that matter)

• Asia (SouthEast): 207.46.48.0/20, 111.221.16.0/21, 111.221.80.0/20
• Asia (East): 111.221.64.0/22, 65.52.160.0/19
• Europe (West): 94.245.97.0/24, 65.52.128.0/19
• Europe (North): 213.199.128.0/20, 213.199.160.0/20, 213.199.184.0/21, 94.245.112.0/20, 94.245.88.0/21, 94.245.104.0/21, 65.52.64.0/20, 65.52.224.0/19
• US (North/Central): 207.46.192.0/20, 65.52.0.0/19, 65.52.48.0/20, 65.52.192.0/19, 209.240.220.0/23
• US (South/Central): 65.55.80.0/20, 65.54.48.0/21, 65.55.64.0/20, 70.37.48.0/20, 70.37.64.0/18, 65.52.32.0/21, 70.37.160.0/21

As our team was starting to transform our parts of the Azure Services Platform from a CTP ‘labs’ service exploring features into a full-on commercial service, it started to dawn on us that we had set ourselves up for writing a bunch of ‘enterprise apps’. The shiny parts of Service Bus and Access Control that we parade around are all about user-facing features, but if I look back at the work we had to go from a toy service to a commercial offering, I’d guess that 80%-90% of the effort went into aspects like infrastructure, deployment, upgradeability, billing, provisioning, throttling, quotas, security hardening, and service optimization. The lesson there was: when you’re boarding the train to shipping a V1, you don’t load new features on that train –  you rather throw some off.

The most interesting challenge for these infrastructure apps sitting on the backend was that we didn’t have much solid ground to stand on. Remember – these were very early days, so we couldn’t use SQL Azure since the folks over in SQL were on a pretty heroic schedule themselves and didn’t want to take on any external dependencies even from close friends. We also couldn’t use any of the capabilities of our own bits because building infrastructure for your features on your features would just be plain dumb. And while we could use capabilities of the Windows Azure platform we were building on, a lot of those parts still had rough edges as those folks were going through a lot of the same that we went through. In those days, the table store would be very moody, the queue store would sometimes swallow or duplicate messages, the Azure fabric controller would occasionally go around and kill things. All normal –  bugs.

So under those circumstances we had to figure out the architecture for some subsystems where we need to do a set of coordinated action across a distributed set of resources – a distributed transaction or saga of sorts. The architecture had a few simple goals: when we get an activation request, we must not fumble that request under any circumstance, we must run the job to completion for all resources and, at the same time, we need to minimize any potential for required operator intervention, i.e. if something goes wrong, the system better knows how to deal with it – at best it should self-heal.

My solution to that puzzle is a pattern I call “Scheduler-Agent-Supervisor Pattern” or, short, “Supervisor Pattern”. We keep finding applications for this pattern in different places, so I think it’s worth writing about it in generic terms – even without going into the details of our system.

The pattern foots on two seemingly odd and very related assumptions: ‘the system is perfect’and‘all error conditions are transient’. As a consequence, the architecture has some character traits of a toddler. It’s generally happily optimistic and gets very grumpy, very quickly when things go wrong – to the point that it will simply drop everything and run away screaming. It’s very precisely like that, in fact.

The first picture here shows all key pieces except the Supervisor that I’ll introduce later. At the core we have a Scheduler that manages a simple state machine made up of Jobs and those jobs have Steps. The steps may have a notion of interdependency or may be completely parallelizable. There is a Job Store that holds jobs and steps and there are Agents that execute operations on some resource.  Each Agent is (usually) fronted by a queue and the Scheduler has a queue (or service endpoint) through which it receives reply messages from the Agents.

Steps are recorded in a durable storage table of some sort that has at least the following fields: Current State (say: Disabled, Active), Desired State (say: Disabled, Active), LockedUntil (Date/Time value), and Actor plus any step specific information you want to store and eventually submit with the job to the step agent.

When Things Go Right

The initial flow is as follows:

(1)a – Submit a new job into the Scheduler (and wait)
(2)a – The Scheduler creates a new job and steps with an initial current state (‘Disabled’) in the job store 
(2)b – The Scheduler sets ‘desired state’ of the job and of all schedulable steps (dependencies?) to the target state (‘Active’) and sets the ‘locked until’ timeout of the step to a value in the near future, e.g. ‘Now’ + 2 minutes.
(1)b – Job submission request unblocks and returns

If all went well, we now have a job record and, here in this example, two step records in our store. They have a current state of ‘Disabled’ and a desired state of ‘Active’. If things didn’t go well, we’d have incomplete or partially wedged records or nothing in the job store, at all. The client would also know about it since we’ve held on to the reply until we have everything done – so the client is encouraged to retry. If we have nothing in the store and the client doesn’t retry – well, then the job probably wasn’t all that important, after all. But if we have at least a job record, we can make it all right later. We’re optimists, though; let’s assume it all went well.

For the next steps we assume that there’s a notion of dependencies between the steps and the second steps depends on the first. If that were not the case, the two actions would just be happening in parallel.

(3) – Place a step message into the queue for the actor for the first step; Agent 1 in this case. The message contains all the information about the step, including the current and desired state and also the LockedUntil that puts an ultimatum on the activity. The message may further contain an action indicator or arguments that are taken from the step record.
(4) – After the agent has done the work, it places a completion record into the reply queue of the Scheduler.
(5) – The Scheduler records the step as complete by setting the current state from ‘Disabled’ to ‘Active’; as a result the desired and the current state are now equal.
(6) – The Scheduler sets the next step’s desired state to the target state (‘Active’) and sets the LockedUntil timeout of the step to a value in the near future, e.g. ‘Now’ + 1 minute. The lock timeout value is an ultimatum for when the operation is expected to be complete and reported back as being complete in a worst-case success case. The actual value therefore depends on the common latency of operations in the system. If operations usually complete in milliseconds and at worst within a second, the lock timeout can be short – but not too short. We’ll discuss this  value in more detail a bit later.
(7), (8), (9) are equivalent to (3), (4), (5).

Once the last step’s current state is equal to the current state, the job’s current state gets set to the desired state and we’re done. So that was the “99% of the time” happy path.

When Things Go Wrong

So what happens when anything goes wrong? Remember the principle ‘all errors are transient’. What we do in the error case – anywhere – is to log the error condition and then promptly drop everything and simply hope that time, a change in system conditions, human or divine intervention, or – at worst – a patch will heal matters. That’s what the second principle ‘the system is perfect’ is about; the system obviously isn’t really perfect, but if we construct it in a way that we can either wait for it to return from a wedged state into a functional state or where we enable someone to go in and apply a fix for a blocking bug while preserving the system state, we can consider the system ‘perfect’ in the sense that pretty much any conceivable job that’s already in the system can be driven to completion.

In the second picture, we have Agent 2 blowing up as it is processing the step it got handed in (7). If the agent just can’t get its work done since some external dependency isn’t available – maybe a database can’t be reached or a server it’s talking to spews out ‘server too busy’ errors – it may be able to back off for a moment and retry. However, it must not retry past the LockedUntil ultimatum that’s in the step record. When things fail and the agent is still breathing, it may, as a matter of courtesy, notify the scheduler of the fact and report that the step was completed with no result, i.e. the desired state and the achieved state don’t match. That notification may also include diagnostic information. Once the LockedUntil ultimatum has passed, the Agent no longer owns the job and must drop it. It must even not report failure state back to the Scheduler past that point.

If the agent keels over and dies as it is processing the step (or right before or right after), it is obviously no longer in a position to let the scheduler know about its fate. Thus, there won’t be any message flowing back to the scheduler and the job is stalled. But we expect that. In fact, we’re ok with any failure anywhere in the system. We could lose or fumble a queue message, we could get a duplicate message, we could have the scheduler die a fiery death (or just being recycled for patching at some unfortunate moment) – all of those conditions are fine since we’ve brought the doctor on board with us: the Supervisor. 

The Supervisor

The Supervisor is a schedule driven process (or thread) of which one or a few instances may run occasionally. The frequency depends on much on the average duration of operations and the expected overall latency for completion of jobs.

The Supervisor’s job is to recover steps or jobs that have failed – and we’re assuming that failures are due to some transient condition. So if the system would expect a transient resource failure condition that prevented a job from completing just a second ago to be healed two seconds later, it’d depend on the kind of system and resource whether that’d be a good strategy.  What’s described here is a pattern, not a solution, so it depends on the concrete scenario to get the  timing right for when to try operations again once they fail.

This desired back-off time manifests in the LockedUntil value.  When a step gets scheduled, the Scheduler needs to state how long it is willing to wait for that step to complete; this includes some back-off time padding. Once that ultimatum has passed and the step is still in an inconsistent state (desired state doesn’t equal the current state)  the Supervisor can pick it up at any time and schedule it.

(1) – Supervisor queries the job store for any inconsistent steps whose LockedUntil value has expired.
(2) – The Supervisor schedules the step again by setting the LockedUntil value to a new timeout and submitting the step into the target actor’s queue
(3) – Once the step succeeds, the step is reported as complete on the regular path back to the Scheduler  where it completes normally as in steps (8), (9) from the happy-path scenario above. If it fails, we simply drop it again. For failures that allow reporting an error back to the Scheduler it may make sense to introduce an error counter that round-trips with the step so that the system could detect poisonous steps that fail ‘forever’ and have the Supervisor ignore those after some threshold.

The Supervisor can pursue a range of strategies for recovery. It can just take a look at individual steps and recover them by rescheduling them – assuming the steps are implemented as idempotent operations. If it were a bit cleverer, it may consider error information that a cooperative (and breathing) agent has submitted back to the Scheduler and even go as far as to fire an alert to an operator if the error condition were to require intervention and then take the step out of the loop by marking it and setting the LockedUntil value to some longer timeout so it’s taken out of the loop and someone can take a look.

At the job-scope, the Supervisor may want to perform recovery such that it first schedules all previously executed steps to revert back to the initial state by performing compensation work (all resources that got set to active are getting disabled again here in our example) and then scheduling another attempt at getting to the desired state.

In step (2)b up above, we’ve been logging current and desired state at the job-scope and with that we can also always find inconsistent jobs where all steps are consistent and wouldn’t show up in the step-level recovery query. That situation can occur if the Scheduler were to crash between logging one step as complete and scheduling the next step. If we find inconsistent jobs with all-consistent steps, we just need to reschedule the next step in the dependency sequence whose desired state isn’t matching the desired state of the overall job.

To be thorough, we could now take a look at all the places where things can go wrong in the system. I expect that survey to yield that at as long we can successfully get past step (2)b from the first diagram, the Supervisor is always in a position to either detect that a job isn’t making progress and help with recovery or can at least call for help. The system always knows what its current intent is, i.e. which state transitions it wants to drive, and never forgets about that intent since that intent is logged in the job store at all times and all progress against that intent is logged as well.  The submission request (1) depends on the outcome of (2)a/b to guard against failures while putting a job and its steps into the system so that a client can take corrective action. In fact, once the job record is marked as inconsistent in step (2)b, the scheduler could already report success back to the submitting party even before the first step is scheduled, because the Supervisor would pick up that inconsistency eventually.

Juval Löwy’s very successful WCF book is now available in its third edition – and Juval asked me to update the foreword this time around. It’s been over three years since I wrote the foreword to the first edition and thus it was time for an update since WCF has moved on quite a bit and the use of it in the customer landscape and inside of MS has deepened where we’re building a lot of very interesting products on top of the WCF technology across all businesses – not least of which is the Azure AppFabric Service Bus that I work on and that’s entirely based on WCF services.

You can take a peek into the latest edition at the O’Reilly website and read my foreword if you care. To be clear: It’s the least important part of the whole book :-)

In case you need a refresher or update about the things me and our team work on at Microsoft, go here for a very recent and very good presentation by my PM colleague Maggie Myslinska from TechEd Australia 2010 about Windows Azure AppFabric with Service Bus demos and a demo of the new Access Control V2 CTP

Room 265, Thursday June 10
9:45AM – 11:00 AM

Session Type: Breakout Session
Track: Application Server & Infrastructure
Speaker(s): Clemens Vasters, Juval Lowy
Level: 300 - Advanced

The availability of the Service Bus in Windows Azure AppFabric is disruptive since it enables new design and deployment patterns that are simply inconceivable without it, opening new horizons for architecture, integration, interoperability, deployment, and productivity. In this unique session organized especially for Tech·Ed, Clemens Vasters and Juval Lowy share their perspective, techniques, helper classes, insight, and expertise in architecting solutions using the service bus. Learn how to manage discrete events, how to achieve structured programming over the Service Bus buffers, what options you have for discovery and even how to mimic WCF discovery, what are the recommended options for transfer security and application authentication, and how to use AppFabric Service Bus for tunneling for diagnostics or logging, to enabling edge devices. The session ends with a glimpse at what is in store for the next versions of the service bus and the future patterns.

Yes, that's Juval and myself on the same stage. That'll be interesting. 

I put the slides for my talks at NT Konferenca 2010 on SkyDrive. The major difference from my APAC slides is that I had to put compute and storage into one deck due to the conference schedule, but instead of purely consolidating and cutting down the slide count,  I also incorporated some common patterns coming out from debates in Asia and added slides on predictable and dynamic scaling as well as on multitenancy. Sadly, I need to rush through all that in 45 minutes today.

I'm on a tour through several countries right now and I'm talking to ISVs about the Windows Azure platform, its capabilities and the many opportunities ISVs have to transform the way they do business by moving to the cloud. The first day of the events is an introduction to the platform at the capability level; it's not a coding class, that would be impossible to fit.

I've shared the slides on SkyDrive. Steal liberally if you find the material useful.

My office neighbor, our Service Bus Test Lead Vishal Chowdhary put together a bundle of code and documentation for how to use Service Bus with Server AppFabric and IIS 7.5. Here: http://code.msdn.microsoft.com/ServiceBusDublinIIS

At the bottom of this post you’ll find the DinnerNow version that I’ve been using for my PDC09 talk. The video of that talk is now available at http://microsoftpdc.com/Sessions/SVC18 and I recommend that you listen to the talk for context.

The DinnerNow drop I’m sharing here is a customized version of the DinnerNow 3.1 version that’s up on CodePlex. If I were you, I’d install the original version and then unpack my zip file alongside of it and then use some kind of diff tool (the Windows SDK’s WinDiff tool is a start) to look at the differences between the versions. That will give you a raw overview of what I had to do. You’ll find that I had to add and move a few things, but that the app didn’t change in any radical way.

Remember that looking at the code is more important that making it run. There’s one particular challenge you’d have right now with the Windows Azure CTP and that’s getting the two (!) Windows Azure compute tokens needed for separating out the web and the service tier as I’ve done here. It’s not difficult to consolidate the Web and the Web Service tier into a single role, but since I had to do the migration within a short period of time, I chose to split them up.

FWIW, I time-boxed the migration to 3 work days – which included learning about what our buddies over in SQL Azure had done in the past months — and that turned out to be a comfortable fit in terms of time.

Another function of time-boxing is that you’re finding me disabling security on most endpoints, including disabling the Access Control integration with Service Bus for most endpoints by setting the relayClientAuthenticationType attribute on the respective binding elements to None.

I know that’s a sin, but I didn’t want to cause too much churn in the first iteration. The original version of DinnerNow is conveniently using Windows authentication/authorization for its communication paths. While that’s ok for a LAN setup, things get more complicated for an actual WAN setup that the DinnerNow scenario calls for. That would spawn a wholly different discussion that shines the spotlight on our Access Control service and why it’s useful – even required – for that scenario. In order not to overwhelm everyone, I left that out for this round and will revisit that aspect in the next weeks – or maybe one of our (aspiring?) MVPs or RDs will beat me to it.

I’m also going to work with the guys who wrote DinnerNow to find a way to host this modified version of Dinner Now with the on-premise runtime bits expressly not on my primary dev machine, where they’d live now.

Here what you need to do to get it to run

I know this is rough. Writing up the long version of this is going to take some time and I prefer getting the bits to you early over me sitting here writing pages of docs. Maybe you can even help ;-)

1. First, you’ll need to go to the Windows Azure portal and get the SDKs and tokens/accounts. The Getting Started page has all the data and links you need so I’m not going to repeat them here in much detail. You will need at least one Windows Azure compute account (apply here), one SQL Azure account (apply here), and an AppFabric account (no application needed, just log in w/ LiveID). 
2. Download and install the regular version DinnerNow 3.1 from Codeplex. This will drop a “Configure DinnerNow 3.1” shortcut on your desktop. Run that, install all prerequisites and make sure DinnerNow runs locally before you proceed.
3. You will later need the databases that the setup created in your local SQLEXPRESS instance by setup. You’ll have to make a few changes, though.
   1. First, (download, install, and) open SQL Server Management Studio, connect to your SQL Server Express instance and switch to “SQL Server and Windows Authentication mode” on the Server Properties under Security. Then you’ll need to go to to the Security settings and either create a new account and grant it all rights on the aspnetdb database or just enable the ‘sa’ account and set its password. 
   2. Then you need to find the “SQL Server Configuration Manager” and enable TCP for your SQLEXPRESS instance like this. The default port will be 1433. If you have a full SQL Server instance on your dev machine and it’s configured for TCP the easiest is to suspend that for the moment and allow the SQLEXPRESS instance to squat the port.
4. Unpack the ZIP file appended below into a directory on your machine. At this point it should be ok to override the existing DinnerNow directory, but I’d keep things side-by-side for reference. If you copy side-by-side, grab the ./solution/DinnerNow – Web/DinnerNow.WebUX/images/’ directory from your local installation and copy it into the location where you unzipped the file here. I left out the images due to their size. And just as with the normal DinnerNow installation you’ll find a solution file named “DinnerNow  - Main.sln” in the unpacked directory – open that in Visual Studio 2008 (not 2010!) because you’ll have to make some changes and edits.
5. If you are lucky enough to have two Windows Azure compute accounts, you can skip this step. Otherwise, you will have to restructure the application a bit: 
   1. In the “DinnerNow – WA” solution branch where the Windows Azure deployment project reside you’ll have to consolidate the DinnerNow.WindowsAzure project and the DinnerNow.WindowsAzureAppSrv projects into one by replicating the DinnerNow.DBBridge reference into the DinnerNow.WindowsAzure project and abandoning/deleting the rest.
   2. In the “DinnerNow – Web” solution branch you will have to modify the DinnerNow.WebUX project by merging the DinnerNow.ServiceHost project from the “DinnerNow -ServicePortfolio2” branch into it, including merging the config files. In the original DinnerNow the hosting default is that the ServiceHost  project lives in the ./services subdirectory of the WebUX app. You can also do it that way, but you’ll have to change the respective client URIs to point to the right path.
6. In the ./database directory is a file called SQLAzureImport.sql. That’s the exported and customized script for the DinnerNow restaurants and menus database. Create a new database (1GB is enough) and load the DB with this script. You can do this with the command line or with SQL Management Studio. The SQL Azure docs will tell you how.
7. Now you’ll need to do a range of search/replace steps across the whole project. These are mostly in *.config files - a few places are in the code, which I count as bugs, but those are faithfully carried over from the original:
   1. Find all occurrences of sqlazure-instance and replace them with your unqualified SQL Azure server name (might look like this: tn0a1b2c3d)
   2. Find all occurrences of sqlazure-dbname and replace them with your SQL Azure database name
   3. Find all occurrences of sqlazure-acct and replace them with your SQL Azure administrator username
   4. Find all occurrences of sqlazure-password and replace them with your SQL Azure administrator password
   5. Find all occurrences of appfabricservicebus-ns and replace them with your unqualified AppFabric namespace name
   6. Find all occurrences of appfabricservicebus-key and replace them with your AppFabric Service Busissuer key
   7. Find all occurrences of windowsazuresvcrole-acct and replace them with the name of your Windows Azure compute account. If you have just one, use that (given you’ve done the rework in step 4), if you have two use the account-name where you will host the service tier.
   8. Find all occurrences of sqlserver-password and replace them with your localSQL Server Express instance’s ‘sa’ account password.
8. Do a full batch Rebuild of the whole project
9. Go to the “DinnerNow –WA” solution and publish the project(s) to your Windows Azure compute account(s). If you had to consolidate them you’ll have one package to deploy, if you left things as they are you’ll have two packages to deploy. You can also run these packages in the local DevFabric to test things out.
10. The executables you need to run are going to be dropped into the .\bin directory by the build. You need to run all 6 apps – but you could run them on 6 different machines – the two workflow hosts each assume the local presence of the DinnerNowWF database:
    1. CloudTraceRecorder.exe – this is the simple event listener app. You can run this right away to observe the apps starting up inside of Azure as they write events to the event listener. You can and should run this as you deploy. You can run any number of instances of CloudTraceRecorder anywhere.
    2. PortBridge.exe – this is the on-premise bridge-head for bridging to your local SQL Server Express instance so that the cloud application can get at its membership database that you host for it on your machine. After the search/replace steps you will notice that you have modified connection strings that point to a SQL Server role peeking out of your *AppSrv role. The secret ingredient is in the DinnerNow.DBBridge role that’s listening for TCP connections on behalf of your on-premise SQL Server and that connects them down to your local server with the logic in Microsoft.Samples.ServiceBus.Connections. This is the same code that’s in PortBridge.
    3. DinnerNow.OrderProcessingHost.exe is the (new) host application for the workflow that handles the order process.
    4. DinnerNow.RestaurantProcessingHost.exe is the (new) host application for the workflow that handles the restaurant process.
    5. DinnerNowKiosk.exe is the only slightly modified version of the DinnerNow in-restaurant kiosk
    6. Not in .\bin but rather to be started/deployed from VS is the also just slightly modified Windows Mobile app for the delivery app

Please also mind that the DinnerNow Powershell support and the other test and diagnostics capabilities haven’t been touched here, yet.

Oh, and … this is provided as-is … I’ll do my best to discuss some of the patterns over the next several weeks, but I don’t have time to provide 1:1 support.

Here’s the code:

Building “hybrid” cloud applications where parts of an an app lives up in a cloud infrastructure and other parts of the infrastructure live at a hosting site, or a data center, or even in your house ought to be simple – especially in this day and age of Web services. You create a Web service, make it accessible through your firewall and NAT, and the the cloud-hosted app calls it. That’s as easy as it ought to be.

Unfortunately it’s not always that easy. If the server sits behind an Internet connection with dynamically assigned IP addresses, if the upstream ISP is blocking select ports, if it’s not feasible to open up inbound firewall ports, or if you have no influence over the infrastructure whatsoever, reaching an on-premise service from the cloud (or anywhere else) is a difficult thing to do. For these scenarios (and others) our team is building the Windows Azure platform AppFabric Service Bus (friends call us just Service Bus).

Now – the Service Bus and the client bits in the Microsoft.ServiceBus.dll assembly are great if you have services can can be readily hooked up into the Service Bus because they’re built with WCF. For services that aren’t built with WCF, but are at least using HTTP, I’ve previously shown a way to hook them into Service Bus and have also demoed an updated version of that capability at Sun’s Java One. I’ll release an update for those bits tomorrow after my talk at PDC09 – the version currently here on my blog (ironically) doesn’t play well with SOAP and also doesn’t have rewrite capabilities for WSDL. The new version does.

But what if your service isn’t a WCF service or doesn’t speak HTTP? What if it speaks SMTP, SNMP, POP, IMAP, RDP, TDS, SSH, ETC?

Introducing Port Bridge

“Port Bridge” – which is just a descriptive name for this code sample, not an attempt at branding – is a point-to-point tunneling utility to help with these scenarios. Port Bridge consists of two components, the “Port Bridge Service” and the “Port Bridge Agent”. Here’s a picture:

The Agent’s job is to listen for and accept TCP or Named Pipe connections on a configurable port or local pipe name. The Service’s job is to accept for incoming connections from the Agent, establish a duplex channel with the Agent, and pump the data from the Agent to the actual listening service – and vice versa. It’s actually quite simple. In the picture above you see that the Service is configured to connect to a SQL Server listening at the SQL Server default port 1433 and that the Agent – running on a different machine, is listening on port 1433 as well, thus mapping the remote SQL Server onto the Agent machine as if it ran there. You can (and I think of that as to be more common) map the service on the Agent to any port you like – say higher up at 41433.

In order to increase the responsiveness and throughput for protocols that are happy to kill and reestablish connections such as HTTP does, “Port Bridge” is always multiplexing concurrent traffic that’s flowing between two parties on the same logical socket. When using Port Bridge to bridge to a remote HTTP proxy that the Service machine can see, but the Agent machine can’t see (which turns out to be the at-home scenario that this capability emerged from) there are very many and very short-lived connections being tunneled through the channel. Creating a new Service Bus channel for each of these connections is feasible – but not very efficient. Holding on to a connection for an extended period of time and multiplexing traffic over it is also beneficial in the Port Bridge case because it is using the Service Bus Hybrid connection mode by default. With Hybrid, all connections are first established through the Service Bus Relay and then our bits do a little “NAT dance” trying to figure out whether there’s a way to connect both parties with a direct socket – if that works the connection gets upgraded to the most direct connections in-flight. The probing, handshake, and upgrade of the socket may take 2-20 seconds and there’s some degree of luck involved to get that direct socket established on a very busy NAT – and thus we want to maximize the use of that precious socket instead of throwing it away all the time.

That seems familiar?!

You may notice that SocketShifter (built by our friends at AWS in the UK) is quite similar to Port Bridge. Even though the timing of the respective releases may not suggest it, Port Bridge is indeed Socketshifter’s older brother. Because we couldn’t make up our mind on whether to release Port Bridge for a while, I had AWS take a look at the service contract shown below and explained a few principles that I’m also explaining here and they had a first version of Socketshifter running within a few hours. There’s nothing wrong with having two variants of the same thing.

How does it work?

Since I’m publishing this as a sample, I obviously need to spend a little time on the “how”, even I’ll limit that here and will explain that in more detail in a future post. At the heart of the app, the contract that’s used between the Agent and the Service is a simple duplex WCF contract:

    [ServiceContract(Namespace="n:",
Name="idx", CallbackContract=typeof(IDataExchange),
SessionMode=SessionMode.Required)] public
interface IDataExchange { [OperationContract(Action="c",
IsOneWay = true, IsInitiating=true)] void Connect(string i);
[OperationContract(Action = "w",
IsOneWay = true)] void Write(TransferBuffer d);
[OperationContract(Action = "d",
IsOneWay = true, IsTerminating = true)] void Disconnect();
}

There’s a way to establish a session, send data either way, and close the session. The TransferBuffer type is really just a trick to avoid extra buffer copies during serialization for efficiency reasons. But that’s it. The rest of Port Bridge is a set of queue-buffered streams and pumps to make the data packets flow smoothly and to accept inbound sockets/pipes and dispatch them out to the proxied services. What’s noteworthy is that Port Bridge doesn’t use WCF streaming, but sends data in chunks – which allows for much better flow control and enables multiplexing.

Now you might say You are using a WCF ServiceContract? Isn’t that using SOAP and doesn’t that cause ginormous overhead? No, it doesn’t. We’re using the WCF binary encoder in session mode here. That’s about as efficient as you can get it on the wire with serialized data. The per-frame SOAP overhead for net.tcp with the binary encoder in session mode is in the order of 40-50 bytes per message because of dictionary-based metadata compression. The binary encoder also isn’t doing any base64 trickery but treats binary as binary – one byte is one byte. Port Bridge is using a default frame size of 64K (which gets filled up in high-volume streaming cases due to the built-in Nagling support) and so we’re looking at an overhead of far less than 0.1%. That’s not shabby.

How do I use it?

This is a code sample and thus you’ll have to build it using Visual Studio 2008. You’ll find three code projects: PortBridge (the Service), PortBridgeAgent (the Agent), and the Microsoft.Samples.ServiceBus.Connections assembly that contains the bulk of the logic for Port Bridge. It’s mostly straightforward to embed the agent side or the service side into other hosts and I’ll show that in a separate post.

Service

The service’s exe file is “PortBridge.exe” and is both a console app and a Windows Service. If the Windows Service isn’t registered, the app will always start as a console app. If the Windows Service is registered (with the installer or with installutil.exe) you can force console-mode with the –c command line option.

The app.config file on the Service Side (PortBridge/app.config, PortBridge.exe.config in the binaries folder) specifies what ports or named pipes you want to project into Service Bus:

           <
          portBridge 
          serviceBusNamespace
          ="mynamespace" serviceBusIssuerName="owner" serviceBusIssuerSecret="xxxxxxxx" localHostName="mybox">
<hostMappings>
<add targetHost="localhost" allowedPorts="3389" />
</hostMappings>
</portBridge>

The serviceBusNamespace attribute takes your Service Bus namespace name, and the serviceBusIssuerSecret the respective secret. The serviceBusIssuerName should remain “owner” unless you know why you want to change it. If you don’t have an AppFabric account you might not understand what I’m writing about: Go make one. 

The localHostName attribute is optional and when set, it’s the name that’s being used to map “localhost” into your Service Bus namespace. By default the name that’s being used is the good old Windows computer-name.

The hostMappings section contains a list of hosts and rules for what you want to project out to Service Bus. Mind that all inbound connections to the endpoints generated from the host mappings section are protected by the Access Control service and require a token that grants access to your namespace – which is already very different from opening up a port in your firewall. If you open up port 3389 (Remote Desktop) through your firewall and NAT, everyone can walk up to that port and try their password-guessing skills. If you open up port 3389 via Port Bridge, you first need to get through the Access Control gate before you can even get at the remote port.

New host mappings are added with the add element. You can add any host that the machine running the Port Bridge service can “see” via the network. The allowedPorts and allowedPipes attributes define with TCP ports and/or which local named pipes are accessible. Examples:

• < add targetHost ="localhost" allowedPorts="3389" /> project the local machine into Service Bus and only allow Remote Desktop (3389)
• < add targetHost ="localhost" allowedPorts="3389,1433" /> project the local machine into Service Bus and allow Remote Desktop (3389) and SQL Server TDS (1433)
• < add targetHost ="localhost" allowedPorts="*" /> project the local machine into Service Bus and only allow any TCP port connection
• < add targetHost ="localhost" allowedPipes="sql/query" /> project the local machine into Service Bus and allow no TCP connections but all named pipe connections to \.\pipes\sql\query
• < add targetHost ="otherbox" allowedPorts="1433" /> project the machine “otherbox” into Service Bus and allow SQL Server TDS connections via TCP

Agent

The agent’s exe file is “PortBridgeAgent.exe” and is also both a console app and a Windows Service.

The app.config file on the Agent side (PortBridgeAgent/app.config, PortBridgeAgent.exe.config in the binaries folder) specifies which ports or pipes you want to project into the Agent machine and whether and how you want to firewall these ports. The firewall rules here are not interacting with your local firewall. This is an additional layer of protection.

           <
          portBridgeAgent 
          serviceBusNamespace
          ="mysolution" serviceBusIssuerName="owner" serviceBusIssuerSecret="xxxxxxxx">
<portMappings>
<port localTcpPort="13389" targetHost="mymachine" remoteTcpPort="3389">
<firewallRules>
<rule source="127.0.0.1" />
<rule sourceRangeBegin="10.0.0.0" sourceRangeEnd="10.255.255.255" />
</firewallRules>
</port>
</portMappings>
</portBridgeAgent>

Again, the serviceBusNamespace attribute takes your Service Bus namespace name, and the serviceBusIssuerSecret the respective secret.

The portMappings collection holds the individual ports or pipes you want to bring onto the local machine. Shown above is a mapping of Remote Desktop (port 3389 on the machine with the computer name or localHostName ‘mymachine’) to the local port 13389. Once Service and Agent are running, you can connect to the agent machine on port 13389 using the Remote Desktop client – with PortBridge mapping that to port 3389 on the remote box.

The firewallRules collection allows (un-)constraining the TCP clients that may connect to the projected port. By default, only connections from the same machine are permitted.

For named pipes, the configuration is similar, even though there are no firewall rules and named pipes are always constrained to local connectivity by a set of ACLs that are applied to the pipe. Pipe names must be relative. Here’s how a named pipe projection of a default SQL Server instance could look like:

     < port localPipe ="sql/remote" targetHost="mymachine" remotePipe="sql/query"/>

There’s more to write about this, but how about I let you take a look at the code first. I’ve also included two setup projects that can easily install Agent and Service as Windows Services. You obviously don’t have to use those.

[Updated archive (2010-06-10) fixing config issue:]

In Parts 3 and 4 of this series I’ve explained the REST protocol for the .NET Service Bus Queue capability. If that looked a little too complicated for your taste and you’d rather clean with a simple API surface, here’s the API that you’ll prefer.

We’ve got two simple classes in the SDK (in the Microsoft.ServiceBus assembly/namespace) that do the job. The QueueManagementClient allows you to create, renew and delete queues in/from the .NET Service Bus namespace:

          
  1: publicstaticclass QueueManagementClient 
          
  2: {

          
  3:    publicstatic QueueClient
CreateQueue(TransportClientEndpointBehavior credential, 
          
  4:                                          Uri queueUri, QueuePolicy queuePolicy);

          
  5:    publicstaticvoid DeleteQueue(TransportClientEndpointBehavior
credential, 
          
  6:                                   Uri queueUri);

          
  7:    publicstatic QueueClient
GetQueue(TransportClientEndpointBehavior credential, 
          
  8:                                       Uri queueUri);

          
  9:    publicstatic QueuePolicy
GetQueuePolicy(TransportClientEndpointBehavior credential, 
          
 10:                                             Uri queueUri);

          
 11:    publicstatic DateTime
RenewQueue(TransportClientEndpointBehavior credential, 
          
 12:                                      Uri queueUri, TimeSpan requestedExpiration);

          
 13: }
        

The most striking difference between this API (and the underlying SOAP protocol for which we still owe you some docs) and the REST API/Protocol is that there’s only a single URI to deal with. The WS-Transfer aligned protocol to manage the policy, the “Queue Transfer” protocol used to dequeue messages, and the “Enqueue” protocol to add messages to the queue are all overlaid over the exact same URI.

The only somewhat ugly element here is the TransportClientEndpointBehavior that’s our awkwardly named credentials container. That class was meant to evolve into something else and then we changed our mind at some point, which is how the baby got stuck with that name. We’ll give that a prettier moniker in one of the next CTPs and as part of an overhaul of the .NET Access Control integration into Service Bus.

The function of the methods themselves should be quite obvious. You can create a queue by applying a policy, you can delete a queue, attach to an existing queue, get the latest queue policy, and renew (extend the lifetime) of a queue. For completeness, here’s CreateQueue in context:

          
  1: Uri queueUri = ServiceBusEnvironment.CreateServiceUri("sb",
solutionName, "/MyQueue/"); 
          
  2: 

          
  3: QueuePolicy queuePolicy = new QueuePolicy(); 
          
  4: queuePolicy.ExpirationInstant = DateTime.UtcNow + TimeSpan.FromHours(1);

          
  5: 

          
  6: QueueClient client = QueueManagementClient.CreateQueue(credential, queueUri, queuePolicy);

          
  7: queuePolicy = client.GetPolicy(); //
get effective policy
        

The QueueClient class allows interaction with a Queue. QueueClient instances cannot be created directly, but must be created via the QueueManagementClient factory.

          
  1: publicsealedclass QueueClient 
          
  2: {

          
  3:     publicvoid DeleteLockedMessage(Message
message); 
          
  4:     publicvoid DeleteQueue(); 
          
  5:     public DateTime
GetExpiration(); 
          
  6:     public QueuePolicy
GetPolicy(); 
          
  7:     public Message
PeekLock(); 
          
  8:     public Message
PeekLock(TimeSpan timeout); 
          
  9:     public IEnumerable<Message>
PeekLockMultiple(int maxMessages); 
          
 10:     public IEnumerable<Message>
PeekLockMultiple(int maxMessages, TimeSpan timeout); 
          
 11:     publicvoid Purge(); 
          
 12:     publicvoid ReleaseLock(Message
message); 
          
 13:     public DateTime
Renew(TimeSpan requestedExpiration); 
          
 14:     public Message
Retrieve(); 
          
 15:     public Message
Retrieve(TimeSpan timeout); 
          
 16:     public IEnumerable<Message>
RetrieveMultiple(int maxMessages); 
          
 17:     public IEnumerable<Message>
RetrieveMultiple(int maxMessages, TimeSpan timeout); 
          
 18:     publicvoid Send(Message
message, TimeSpan timeout); 
          
 19:     public RouterSubscriptionClient
SubscribeToRouter(RouterClient routerClient, TimeSpan requestedTimeout); 
          
 20: }
        

As you can tell, the class is – well – a queue client. It’s using WCF’s Message class as its message abstraction and supports sending messages into the queue, reading messages off the queue in a destructive fashion (Retrieve), and reading messages off the queue using the Peek/Lock pattern which provides resilience against message loss if the message were to be lost in transfer or the receiver fumbled the message. What you’ll also notice is that there are RetrieveMultiple and PeekLockMultiple variants of the retrieval functions which allow for getting more data with fewer network roundtrips.

Again, there should be no surprises using the API. Here’s how you send:

          
  1: queueClient.Send(Message.CreateMessage(MessageVersion.Default, "Hello",
input), TimeSpan.MaxValue);
        

and here’s how you do a destructive read:

          
  1: Message message = queueClient.Retrieve();
        

and here’s how you use the Peek/Lock pattern:

          
  1: Message message = queueClient.PeekLock();

          
  2: string content
= message.GetBody<string>(); 
          
  3: queueClient.DeleteLockedMessage(message);

        

There are several examples in the SDK showing how to use the API for Queues. The OnewayQueueSender and SoapHttpQueueSender samples are particularly interesting since they just use WCF channels to enqueue, including all the bells and whistles you get from using a WCF channel. Here’s how the OnewayQueueSender does it:

          
  1: ChannelFactory<IOnewayChannel> channelFactory = 

          
  2:      new ChannelFactory<IOnewayChannel>(new NetOnewayRelayBinding(), 
          
  3:                                         new EndpointAddress(queueUri)); 
          
  4: channelFactory.Endpoint.Behaviors.Add(userNamePasswordServiceBusCredential);

          
  5: IOnewayChannel onewayChannel = channelFactory.CreateChannel();

          
  6: 

          
  7: string input
= Console.ReadLine(); 
          
  8: onewayChannel.Hello(new HelloMessage(input)); 
        

The logical next question is: So why is the Queue not hooked up to a WCF listener? Answer: That’s what we’ve got “buffered Routers” for. The queue very explicitly provides a ‘pull’ model and the WCF listener would abstract that away and turn it into ‘push’. Routers provide ‘push’ natively and they can embed a Queue. More on Routers in the next set of posts in this series.

After I’ve gone through the dry facts of the REST Queue Protocol in Part 3 of this series, here’s some code to look at. The code snippets are ripped from the HTTP Queue sample we’ve got in the .NET Services SDK and you can get the copy/paste ready code from there.

The most straightforward way to acquire the required hashed security token (see my comments to Stefan Tilkov) using HttpWebRequest looks like this:

          
  1: staticstring HttpGetAuthenticationToken(string username, string password) 
          
  2: {

          
  3:     string tokenUri
= string.Format("https://{0}/issuetoken.aspx?u={1}&p={2}", 
          
  4:                                     ServiceBusEnvironment.DefaultIdentityHostName, 

          
  5:                                     username, Uri.EscapeDataString(password));

          
  6: 

          
  7:     HttpWebRequest tokenRequest = (HttpWebRequest)WebRequest.Create(tokenUri);

          
  8:     using (var
tokenResponse = tokenRequest.GetResponse()) 
          
  9:     {

          
 10:         using (var
tokenResponseStream = tokenResponse.GetResponseStream()) 
          
 11:         {

          
 12:             byte[]
tokenBody = newbyte[500]; 
          
 13:             int tokenBodyLength
= tokenResponseStream.Read(tokenBody, 0, 500); 
          
 14:             return Encoding.UTF8.GetString(tokenBody,
0, tokenBodyLength); 
          
 15:         }

          
 16:     }

          
 17: }

          
 18: 
        

Once you’ve got the token, you need to pick a place in the Service Bus namespace. We’ve got a built-in helper function in the SDK that sits on ServiceBusEnvironment and which knows all the right incantations:

          
  1:  Uri queueUri = ServiceBusEnvironment.CreateServiceUri(

          
  2:                    Uri.UriSchemeHttps, 

          
  3:                    solutionName, 

          
  4:                    "/MyHttpQueue/"); 
        

Invoked in this way, the method yields the URI https://yourproject.servicebus.windows.net/MyHttpQueue/ so there’s spectacularly little magic to it. The reason why we recommend that you use the method in .NET applications is that we’ve broken everyone’s apps going from the previous CTP to the current CTP due to the namespace restructuring and we’d like to avoid doing that again as we improve the namespace story. I don’t foresee any further change of the magnitude we had this time, however.

To create a new Queue, you first create a Queue policy and then apply it to the Service Bus namespace name. We’ve got a Queue Policy object in the SDK that I’ll use here to keep things short. I’m accepting all the defaults that are spelled out in Part 2 of the series, but explicitly I’m setting the expiration to 1 hour. The Queue Policy XSD schema in the the HTTP Queue sample’s readme document, by the way.

          
  1: QueuePolicy queuePolicy = new QueuePolicy(); 
          
  2: queuePolicy.ExpirationInstant = DateTime.UtcNow + TimeSpan.FromHours(1);

          
  3: queueManageUri = HttpCreateQueue(token, queueUri, queuePolicy);
        

The code for creating the Queue shouldn’t be very surprising. It’s a POST of an Atom 1.0 entry for which I’m using HttpWebRequest and the WCF 3.5 Syndication API. The SDK sample is using a rather terse version of the ‘correct’ way of doing things. I’m quoting the expanded variant from the Text140 sample that I showed at MIX’09:

          
  1: publicstatic QueuePolicy
HttpCreateQueue(string token, Uri queueUri, QueuePolicy
policy, 
          
  2:                                           out Uri
queueSelfUri, out Uri queueHeadUri) 
          
  3: {

          
  4:     HttpWebRequest webRequest;

          
  5:     HttpWebResponse webResponse;

          
  6: 

          
  7:     queueSelfUri = null; 
          
  8:     queueHeadUri = null; 
          
  9: 

          
 10:     //
Create a new syndication item and add the queue policy as an extension
          
 11:     SyndicationItem syndicationItem = new SyndicationItem(); 
          
 12:     syndicationItem.ElementExtensions.Add(

          
 13:         new SyndicationElementExtension("QueuePolicy", 
          
 14:                                         "http://schemas.microsoft.com/ws/2007/08/connect", 
          
 15:                                         policy));

          
 16: 

          
 17:     //
create a new POST request and set the Content-Type to represent an Atom 1.0 entry
          
 18:     webRequest = HttpWebRequest.Create(queueUri) as HttpWebRequest; 
          
 19:     webRequest.ConnectionGroupName = "queueclient"; 
          
 20:     webRequest.KeepAlive = false; 
          
 21:     webRequest.Method = "POST"; 
          
 22:     webRequest.Headers.Add("X-MS-Identity-Token",
token); 
          
 23:     webRequest.ContentType = "application/atom+xml;type=entry;charset=utf-8"; 
          
 24: 

          
 25:     //
write the item to the request stream
          
 26:     using (var
requestStream = webRequest.GetRequestStream()) 
          
 27:     {

          
 28:         using (var
textWriter = new XmlTextWriter(requestStream,
Encoding.UTF8)) 
          
 29:         {

          
 30:             syndicationItem.GetAtom10Formatter().WriteTo(textWriter);

          
 31:             textWriter.Flush();

          
 32:         }

          
 33:     }

          
 34: 

          
 35:     //
get the response
          
 36:     using (webResponse
= webRequest.GetResponse() as HttpWebResponse) 
          
 37:     {

          
 38:         return GetQueueInfoFromResponse(webResponse,
queueUri, out queueSelfUri, out queueHeadUri); 
          
 39:     }

          
 40: }

          
 41: 

          
 42: static QueuePolicy
GetQueueInfoFromResponse(HttpWebResponse webResponse, Uri queueUri, 
          
 43:                                                     out Uri
queueSelfUri, out Uri queueHeadUri) 
          
 44: {

          
 45:     queueHeadUri = null; 
          
 46:     queueSelfUri = null; 
          
 47: 

          
 48:     if (webResponse.ContentType.StartsWith("application/atom+xml;type=entry",
StringComparison.OrdinalIgnoreCase)) 
          
 49:     {

          
 50:         Atom10ItemFormatter atomItemFormatter = new Atom10ItemFormatter(); 
          
 51:         using (var
responseReader = new XmlTextReader(webResponse.GetResponseStream())) 
          
 52:         {

          
 53:             atomItemFormatter.ReadFrom(responseReader);

          
 54:             //
we found it. let's get the policy and the queue head URI
          
 55:             foreach (SyndicationLink
queueSelfLink in
          
 56:                            from link in atomItemFormatter.Item.Links 
          
 57:                            where link.RelationshipType.Equals("self",
StringComparison.OrdinalIgnoreCase) 
          
 58:                            select link)

          
 59:             {

          
 60:                 queueSelfUri = queueSelfLink.Uri;

          
 61:                 break; 
          
 62:             }

          
 63: 

          
 64:             foreach (SyndicationLink
queueHeadLink in
          
 65:                            from link in atomItemFormatter.Item.Links 
          
 66:                            where link.RelationshipType.Equals("queuehead",
StringComparison.OrdinalIgnoreCase) 
          
 67:                            select link)

          
 68:             {

          
 69:                 queueHeadUri = queueHeadLink.Uri;

          
 70:                 break; 
          
 71:             }

          
 72: 

          
 73:             return atomItemFormatter.Item.ElementExtensions.ReadElementExtensions<QueuePolicy>( 
          
 74:                  "QueuePolicy", 
          
 75:                  "http://schemas.microsoft.com/ws/2007/08/connect").FirstOrDefault(); 
          
 76:         }

          
 77:     }

          
 78:     else
          
 79:     {

          
 80:         Atom10FeedFormatter atomFeedFormatter = new Atom10FeedFormatter(); 
          
 81:         using (var
responseReader = new XmlTextReader(webResponse.GetResponseStream())) 
          
 82:         {

          
 83:             atomFeedFormatter.ReadFrom(responseReader);

          
 84: 

          
 85:             //
look for an item whose alternate-link equals the desired queue URI. The alternate 
          
 86:             //
URI is the 'tail' endpoint by which the queue is mapped into the namespace. We are 
          
 87:             //
comparing Authority and Path since the scheme might be http or https and we want to
match
          
 88:             //
either.
          
 89:             foreach (SyndicationItem
requestedItem in
          
 90:                          from item in atomFeedFormatter.Feed.Items 
          
 91:                          from link in item.Links 
          
 92:                          where 

          
 93:                            (link.RelationshipType.Equals("alternate",
StringComparison.OrdinalIgnoreCase) && 
          
 94:                             link.Uri.Authority.Equals(queueUri.Authority, StringComparison.OrdinalIgnoreCase) &&

          
 95:                             link.Uri.AbsolutePath.Equals(queueUri.AbsolutePath, StringComparison.OrdinalIgnoreCase)) ||

          
 96:                            (link.RelationshipType.Equals("self",
StringComparison.OrdinalIgnoreCase) && 
          
 97:                             link.Uri.Equals(queueUri))

          
 98:                          select item)

          
 99:             {

          
100:                 //
we found it. let's get the policy and the queue head URI
          
101:                 foreach (SyndicationLink
queueSelfLink in
          
102:                                from link in requestedItem.Links 
          
103:                                where link.RelationshipType.Equals("self",
StringComparison.OrdinalIgnoreCase) 
          
104:                                select link)

          
105:                 {

          
106:                     queueSelfUri = queueSelfLink.Uri;

          
107:                     break; 
          
108:                 }

          
109: 

          
110:                 foreach (SyndicationLink
queueHeadLink in
          
111:                                from link in requestedItem.Links 
          
112:                                where link.RelationshipType.Equals("queuehead",
StringComparison.OrdinalIgnoreCase) 
          
113:                                select link)

          
114:                 {

          
115:                     queueHeadUri = queueHeadLink.Uri;

          
116:                     break; 
          
117:                 }

          
118: 

          
119:                 return requestedItem.ElementExtensions.ReadElementExtensions<QueuePolicy>( 
          
120:                      "QueuePolicy", 
          
121:                      "http://schemas.microsoft.com/ws/2007/08/connect").FirstOrDefault(); 
          
122:             }

          
123:         }

          
124:     }

          
125:     returnnull; 
          
126: }
        

The “magic” that isn’t straight up Atom Pub is in lines 10-15 above. I’m attaching the policy object to the SyndicationItem abstraction that will be written out as an Atom 1.0 entry. [I know that copy/paste is difficult using this format – I need to clean up that expanded utility class and will post it in downloadable form within the next few days]

The method returns the effective policy that the Service Bus is using, and yields the management URI for the queue where the Atom entry resides (self-link) as well as the head of the queue as output arguments. The queueUri to which you applied the entry has morphed into the Queue’s tail when the method returns successfully. Having the Queue by its tail, submitting messages into it is very simple. Just do a plain POST. Here we’re just writing a simple string from the local variable input as the entity body and send it.

          
  1: //
send
          
  2: HttpWebRequest sendRequest = HttpWebRequest.Create(queueUri) as HttpWebRequest; 
          
  3: sendRequest.Method = "POST"; 
          
  4: sendRequest.Headers.Add("X-MS-Identity-Token",
token); 
          
  5: sendRequest.ContentType = "text/plain;charset=utf-8"; 
          
  6: using (var
sendStream = sendRequest.GetRequestStream()) 
          
  7: {

          
  8:     using (var
writer = new StreamWriter(sendStream, Encoding.UTF8)) 
          
  9:     {

          
 10:         writer.Write(input);

          
 11:         writer.Flush();

          
 12:     }

          
 13: }

          
 14: sendRequest.GetResponse().Close();
        

For a simple destructive read, you use the ‘queue head’ URI and do a DELETE:

          
  1: HttpWebRequest dequeueRequest = 

          
  2:     HttpWebRequest.Create(queueHeadUri.AbsoluteUri+

          
  3:                           "?encoding=asreply&maxmessages=1&timeout=60") as HttpWebRequest; 
          
  4: dequeueRequest.ConnectionGroupName = "dequeue"; 
          
  5: dequeueRequest.Method = "DELETE"; 
          
  6: dequeueRequest.ContentLength = 0;

          
  7: dequeueRequest.Headers.Add("X-MS-Identity-Token",
token); 
          
  8: using (HttpWebResponse
response = dequeueRequest.GetResponse() as HttpWebResponse) 
          
  9: {

          
 10:     if (response.StatusCode
== HttpStatusCode.OK) 
          
 11:     {

          
 12:         using (var
responseBody = response.GetResponseStream()) 
          
 13:         {

          
 14:             using (var
responseReader = new StreamReader(responseBody,
Encoding.UTF8)) 
          
 15:             {

          
 16:                 string data
= responseReader.ReadToEnd(); 
          
 17:                 Console.WriteLine(data);

          
 18:             }

          
 19:         }

          
 20:     }

          
 21: }
        

For a peek/lock read where you first lock the message and then delete the lock when you want to keep the message, you do a POST on the head and remember the X-MS-Message-Lock header value:

          
  1: HttpWebRequest dequeueRequest = 

          
  2:      HttpWebRequest.Create(queueHeadUri.AbsoluteUri+

          
  3:                            "?encoding=asreply&maxmessages=1&timeout=60") as HttpWebRequest; 
          
  4: dequeueRequest.ConnectionGroupName = "dequeue"; 
          
  5: dequeueRequest.Method = "POST"; 
          
  6: dequeueRequest.ContentLength = 0;

          
  7: dequeueRequest.Headers.Add("X-MS-Identity-Token",
token); 
          
  8: using (HttpWebResponse
response = dequeueRequest.GetResponse() as HttpWebResponse) 
          
  9: {

          
 10:     if (response.StatusCode
== HttpStatusCode.OK) 
          
 11:     {

          
 12:         lockUri = new Uri(response.Headers["X-MS-Message-Lock"]); 
          
 13:         using (var
responseBody = response.GetResponseStream()) 
          
 14:         {

          
 15:             using (var
responseReader = new StreamReader(responseBody,
Encoding.UTF8)) 
          
 16:             {

          
 17:                 string data
= responseReader.ReadToEnd(); 
          
 18:                 Console.WriteLine(data);

          
 19:             }

          
 20:         }

          
 21:     }

          
 22: }
        

If you decide to keep the message (i.e. your app didn’t puke processing it), you DELETE the lock, otherwise you do a PUT. The code is practically the same except for the method so I’ll just quote DELETE:

          
  1: HttpWebRequest deleteLockedMessageRequest = HttpWebRequest.Create(lockUri) as HttpWebRequest; 
          
  2: deleteLockedMessageRequest.ConnectionGroupName = "lock"; 
          
  3: deleteLockedMessageRequest.Method = "DELETE"; 
          
  4: deleteLockedMessageRequest.KeepAlive=false; 
          
  5: deleteLockedMessageRequest.ContentLength = 0;

          
  6: deleteLockedMessageRequest.Headers.Add("X-MS-Identity-Token",
token); 
          
  7: deleteLockedMessageRequest.ContentLength = 0;

          
  8: deleteLockedMessageRequest.GetResponse().Close();
        

That’s it. Create, Enqueue, Destructive Read, Peek/Lock Read, and .. oh .. yes.. forgot Delete the Queue. Take the snippet above and swap out the lockUri for the queueSelfUri. Done. 

Ah, yes, and please … don’t blame me for HTTP or HttpWebRequest or Clemens’ demo coding style requiring too many lines of code. I could have applied a bit of code-compression here, but I’m intentionally trying not to abstract away too much of the protocol here. I’m generally with you, though. The guys putting together the WCF REST Starter Kit are working on making code like that shrink.

Stefan Tilkov has several interesting remarks regarding our .NET Service Bus REST Queue Protocol that are worth addressing.

      Putting a password in the URI to get an identity token seems to expose information unnecessarily

That’s an area where we know that we’re going to change the protocol. We’ve already labeled that protocol as temporary in the documentation for the PDC CTP and we didn’t get all the pieces in the .NET Access Control service together, yet. Since it’s a HTTPS call, the data doesn’t get exposed on the wire, though.

      Queue creation seems fine, even though I feel a little uneasy about wrapping this in an Atom entry

Using Atom 1.0 and the Atom Publishing Protocol as the framework for managing the namespace is very intentional - for several reasons. First of all, it’s a standardized protocol for managing generic lists and the elements in those lists. With that we have a stable and accepted protocol framework and there’s plenty of tooling and framework support around it. That’s worth something. All we need to do is to add some simple extensions – the policies – on top of that stack. Beats having to define, version, and maintain a whole protocol.

      On the other hand, Atom seems reasonable considering you get an Atom feed from the queue’s “parent” resource

That’s what I mean. All sorts of tools know how to navigate and display Atom.

      Very nice to see the use of link/rel to get to the detailed Queue URIs; it would be even better if the rel values themselves were URIs in an MS namespace

I don’t see much potential for collision here and I would find it odd to have something as simple “self” and “alternate” and then add some unsightly QName for my rel expressions. Simple is good.

      Using “alternate” for the tail seems strange

“self” refers to the location where the Atom <entry> resides. “alternate” is what the entry points to. Since the Queue gets mapped into the namespace by “sticking its tail out”, the choice of the alternate link is the simplest possible mapping I could think of.

      “The queue’s tail endpoint accepts requests with any HTTP Content-Type and any HTTP method except GET, HEAD, and OPTIONS.” The tail does not accept GET. Why not? Can’t think of a reason. Also, I don’t get why any method is allowed. It seems to me a POST/PUT combination or an idempotent POST variant would be the way to go.

The way to look at this is that the Queue’s tail is acting on behalf of the receiver/resource that’ll eventually pick up and process the messages. POST, PUT, DELETE, and BOOYAH are all operations that cleanly map to processing operations and can often be delivered asynchronously with a 202 receipt reply. GET and HEAD don’t make much sense when executed in an asynchronous fashion without getting a reply that’s backed by a response for the receiver. OPTIONS is simply reserved for future use.

      “DELETE performing a destructive read” – huh? What happens if I don’t get back a response? How can I retry (as I can’t tell whether I get the same or a different message)? Maybe it’s intended to be used only if you don’t care about losing messages.

DELETE is indeed the dequeue operation variant that you’d use if you are ok with occasional message loss and want to trade transfer reliability for fewer roundtrips.

      The POST lock/delete approach, on the other hand, is very nice. Maybe it should be made idempotent, again e.g. using POE

POST lock/delete is the dequeue variant where you are doing the trade the other way around. More reliability bought with an extra roundtrip. In my view, idempotent access to individual messages isn’t much of a practical priority for a competing consumer queue. You’ll get a message or a set of messages to look at under a lock and if you walk away from the message(s), those messages pop back into the queue for someone else to look at. There are obviously scenarios where you want to look at a message sequence in other ways than an ordered queue where you can only get at messages as they appear on the head of the sequence – direct message access and per-message idempotent access matter in those scenarios and we’re looking to give you a capability of that sort in a different kind of messaging primitive.

      “The Delete request is sent without any entity body and MUST have a Content-Length header that is set to zero (0)”; although my immediate reaction was to question whether DELETE ever carries a body, the HTTP spec indeed doesn’t say anything about this

One of the great things about working here is that there are all sorts of interesting people around. I’ve discussed the use of DELETE and whether you can provide an entity body in either direction with Henrik Frystyk Nielsen, who works as an architect on the WCF team and is one of the co-authors of HTTP 1.1. Henrik’s stance is that all operations allow entity-bodies unless it’s explicitly forbidden in the spec. I don’t have a better authority to talk to.

      “The DELETE and POST operation have a set of options that are expressed as query parameters appended to the queue’s head URI” - the wording is worse than the actual approach.

I’m sorry that my writing is so clumsy ;-)

Short answer: You can’t.

There is a range of breaking protocol changes between the December bits and the March bits. Your app won’t work until you upgrade (uninstall/install) to the March 2009 CTP client bits and/or SDK.

[If this text looks vaguely familiar you’ve read the HttpQueueSample Readme doc from the SDK. Good job!]

Here is the Service Bus Queue REST protocol. I apologize if this is a bit dry, but I want to give you the bare protocol facts first and will follow-up with code samples later. Look at the HTTP Queue SDK sample if you can’t wait for code ;-)

You can find the HTTP Queue sample in the SDK install path under .\Samples\ServiceBus\ExploringFeatures\Queues\HttpMessages\

The beauty of REST is that there’s a uniform interface. It’s all just GET, POST, PUT, DELETE, a set of resources, and a rigorous application of the RFC2616 / HTTP 1.1 rules. That doesn’t make the description particularly entertaining, but it does yield a pretty consistent interaction model:

Authorization

All Queue operations require the client to be appropriately authorized. The client must therefore acquire an lightweight identity token from the Microsoft .NET Access Control Service using the .NET Services solution name and the solution password and include the acquired token with each request. The tokens are added to requests using the X-MS-Identity-Token HTTP header. The value for this header is a short-lived token which can be used for a period of up to 8 hours and must be acquired from the .NET Access Control Service. We strongly encourage you to guard the acquired tokens in the same fashion as you would guard a credential and you should not openly expose them unprotected on the network or web-pages or embedded in a Flash or Silverlight application.

The request for acquiring the token is an HTTPS GET on the URI https://accesscontrol.windows.net/issuetoken.aspx?u={solutionName}&p={solutionPassword} whereby you replace the arguments for solution name and the solution password with your .NET Services solution credentials. If a token is successfully issued to you the request returns with a 200 (OK) status code and contains a text/plain entity body with a short, base64 encoded token hash as a single line of text, which you for the value of the X-MS-Identity-Token header.

Creating a Queue

A Queue is created on the Service Bus in four simple steps. I’ve discussed most of this in the post on Queue Policies:

1. Select a name in the Service Bus naming hierarchy where the Queue should be located, i.e. https://mysolution.servicebus.windows.net/myapp/q1

2. Create a Queue policy and define the desired properties of the Queue.

3. Embed the Queue policy into an ATOM 1.0 <atom:entry> as an extension and POST the entry to the designated Queue URI with the content-type application/atom+xml;type=entry.The request must carry an X-MS-Identity-Token header and the respective identity must have 'Manage' permission on for scope covering the Queue URI.

4. If the queue was successfully created, the POST request returns with a 201 'created' status code along with a Location header. The response entity body contains the effective Queue policy as applied to the name by the Service Bus.
   
   The location header contains the queue's management URI that you need to retain in your application state to have access to the queue's metadata and management functions. If the queue could not be created successfully, the request may yield the following status codes:

   • 400 - Bad Request. The policy was malformed or invalid.

   • 403 - Forbidden. The client did not provide a X-MS-Identity-Token header, the provided token is no longer valid for use, or the provided identity is not authorized to create a Queue at this location.

   • 409 - Conflict.  There is already a Queue with an incompatible policy at the given location or the location is occupied by a Router or a Service Bus listener.

   • 415 - Unsupported Media Type. The request did not carry the required content-type header.

   • 500 - Internal Server Error. The processing failed to to a condition internal to the Service Bus service.

REST Anatomy of a Queue

Each Queue has five distinct types of resources: Policy, Tail, Head, Locks, and Control. The concrete URIs are discoverable using the Service Registry's Atom Publishing protocol. you should not make any assumptions about the particular format of the URIs spelled out below since the particular format might change.

The Atom 1.0 entry shown below is also enclosed in the response entity body of the create request (POST) explained above. A Queue's representation in the registry feed as seen when doing a GET (discovery-) request on the queue's parent URI is commonly as follows:

<feed xmlns="http://www.w3.org/2005/Atom">
  ...
  <entry>
    <id>{id}</id>
    <title type="text">MyHttpQueue</title>
    <updated>{date}</updated>
    <link rel="alternate" href="https://solution.servicebus.windows.net/myapp/q1" />
    <link rel="self" href="https://solution.servicebus.windows.net/myapp/q1!(queue)" />
    <link rel="queuehead" href="https://solution.servicebus.windows.net/myapp/q1!(queue/head)" />
    <link rel="queuecontrol" href="https://solution.servicebus.windows.net/myapp/q1!(queue/control)" />
    <QueuePolicy xmlns="http://schemas.microsoft.com/ws/2007/08/connect">
      <Discoverability>Public</Discoverability>
      <ExpirationInstant>{expiration}</ExpirationInstant>
    </QueuePolicy>
  </entry>
</feed>

Managing The Queue

<link rel="self" href="https://solution.servicebus.windows.net/myapp/q1!(queue)" />

The "self" link in the entry above is the queue's management URI that allows you to interact with the queue's policy.

• To renew a queue and extend its expiration you PUT an updated Atom 1.0 entry with the effective policy and an updated ExpirationInstant value to the "self" URI.  The "self" URI is the same URI as the one returned in the Location header returned by the create POST explained above. The request will yield a 200 status code if the renewal is successful. It will yield a 404 if the queue no longer exists.

• To delete a queue you issue a DELETE request on  the queue's management URI. The Delete request is sent without any entity body and MUST have a Content-Length header that is set to zero (0). Not setting this header will yield a 411 status code. The request yields a 204 response if the queue was deleted successfully.

Enqueue

<link rel="alternate" href="https://solution.servicebus.windows.net/myapp/q1" />

The "alternate" link refers to the tail of the queue. The Queue's tail resource is where senders submit messages into the Queue. The queue's tail URI is identical to the URI where the queue was initially created. The queue's tail endpoint accepts requests with any HTTP Content-Type and any HTTP method except GET, HEAD, and OPTIONS.

The endpoint should be considered a 'delegate' of the message recipient and not the resource that the respective operations have an immediate effect on. That means that the HTTP method applied to the Queue tail doesn’t have any particular semantic. If a message has been successfully accepted into the queue, the request returns a 202 status code. Otherwise it will return a 500

Dequeue and Peek/Lock

<link rel="queuehead" href="https://solution.servicebus.windows.net/myapp/q1!(queue/head)" />

The "queuehead" link refers to the head of the Queue. The Queue's head resource is where receivers retrieve messages from the queue. It is modeled as a resource whose current representation corresponds to the message(s) that reside at the head of the queue and which is/are next in line to be retrieved by consumers. Permitted operations on the head are:

• DELETE:  Delete performs a destructive read on the queue whereby the message(s) at the head of the queue are permanently deleted and returned as the entity body of the response to the request. The content type and formatting of the response depends on a set of query parameters that are discussed below. The DELETE request's response status code is 200 if at least one message could be retrieved and deleted from the queue and the message content is enclosed in the response's entity body. The response status code is 204if no message could be retrieved/deleted.
• POST: Post creates a lock on the message(s) at the head of the queue and returns the locked messages. Locked messages are temporarily removed from the head of the queue for a period of 1 minute during which time the receiver can decide whether to return the message to the head of the queue or whether to permanently delete it. The POST request's response status code is 200 if at least one message could be retrieved and locked on the queue and the message content is enclosed in the response's entity body. The response status code is 204if no message could be retrieved/locked.

Releasing and Deleting Peek/Lock Message Locks

Any retrieved, peek/locked message (POST on “queuehead”) contains an HTTP header X-MS-Message-Lock whose value is a URI. The following operations can be performed on this URI:

• DELETE: The message has been accepted/processed by the receiver and the lock shall be deleted. The deletion of the lock discards the message permanently from the queue. The Delete request is sent without any entity body and MUST have a Content-Length header that is set to zero (0). Not setting this header will yield a 411 status code. The request yields a 204 response if the lock was deleted successfully.
• PUT:  The message has been not been accepted/processed by the receiver and the message shall be put back at the head of the queue. This operation transitions the lock from the locked state into the unlocked state which releases the message back into the queue. The lock itself is discarded afterwards. The Put request is sent without any entity body and MUST have a Content-Length header that is set to zero (0). Not setting this header will yield a 411 status code. The request yields a 204 response if the lock was released successfully.

Options for Dequeue and Peek/Lock

The DELETE and POST operation have a set of options that are expressed as query parameters appended to the queue's head URI. The options are the same for both operations:

• timeout= value - The value is a numerical value (expressed in seconds) that indicates how long the client is willing to wait for the polling request to complete. The value should not exceed 60 seconds and a value of 30 seconds is a safer choice with intermediate proxies in place. If the timeout expires and no message is available, the request completes with a 204 response code. The default value is zero (0), which means that the request returns immediately.
  

• maxmessages=value - Indicates how many messages the client is willing to accept in a single response. The default is 1. The service will return at most 10 messages at a time and a value greater than zero requires encoding=multipart (see below)
  

• encoding=value - There are three supported encoding modes for the REST retrieval model

  • asreply - The request stored in the Queue is mapped onto the response to the DELETE/POST request returning the message from the queue. This option requires maxmessages=1 or omission of the maxmessages option.

  • single - The request stored in the queue is returned as a complete HTTP request frame on the response to the DELETE/POST request using Content-Type application/http. This option requires maxmessages=1 or omission of the maxmessages option.

  • multipart - The request(s) stored in the queue are returned as a complete HTTP request frames on the response to the DELETE/POST request using Content-Type multipart/mixed with MIME parts of Content-Type application/http.

Queue Content Control

<link rel="queuecontrol" href="https://solution.servicebus.windows.net/myapp/q1!(queue/control)" />

The "queuecontrol" link refers to the control resource provides information about the status of the queue and allows for purging all of the queue's contents. The control resource is present in the current CTP, but you cannot perform any operations on it.

In the previous post in this series I’ve discussed some of the foundational principles of the new Queue and Router features of the .NET Services Bus and the role that policies play in turning names in the namespace into messaging primitives. In this post I’ll start drilling into the capabilities of the Queue feature and will discuss what’s in the queue policy.

The capabilities of the Queue are best explored by looking at the Queue policy. Policies can be expressed in two ways: You can either you use the .NET Services SDK’s Microsoft.ServiceBus.QueuePolicy class with a .NET application or you can just whip up a bit of XML. The structure and the serialized representation of that class is exactly equivalent to an XML queue policy you’d wire up “by hand” and thus I’ll discuss the policy in terms of its XML elements:

The queue policy has the element name “QueuePolicy” within the XML namespace “http://schemas.microsoft.com/ws/2007/08/connect”. That’s the namespace we use for most protocols and data structures in the .NET Service Bus at this development stage and you should expect that we’re migrating to the “final” V1 namespaces in one of the next CTPs.  If you use the .NET class you’ll just have to recompile to snap to new namespaces once they come around.

Below is a list of the policy’s elements and the permitted values. All elements are optional and can appear at most once. The default value applies when an element is omitted. It’s perfectly ok to send an empty QueuePolicy and accept the default values to keep things simple. I’d recommend to always set the ExpirationInstant value, however.

One required forward reference: Don’t get confused or distracted trying to figure out from here what it means “to provide an authorization token”. The docs explain this and I’ll also address this in the next blog post when I dive into the protocol.

• Authorization – The authorization setting indicates whether sender and/or consumers must provide a security token when interacting with the queue. You can explicitly permit the .NET Service Bus to allow anonymous consumers and/or anonymous senders on a Queue. If a token is required, a sender must present a token with “Send” permission that is issued by the .NET Access Control service and the consumer must present a token with “Listen” permission that is issued by the .NET Access Control service. The respective rules are set up in the .NET Access Control service on your project’s “Service Bus” scope. Mind that all anonymous traffic is attributed to the “manager” of a Queue, i.e. to whoever sets the policy; all traffic is metered and accounted for. The permitted values for this element are:
  • Required – (default) Senders and Consumers must provide an authorization token to send/retrieve messages.
  • RequiredToSend – Senders must provide an authorization token with a “Send” claim to send messages, consumers don’t.
  • RequiredToReceive – Consumers must provide an authorization token with a “Listen” claim to receiver messages, senders don’t.
  • NotRequired – Neither senders nor consumers must provide a token, i.e. the queue is set up for anonymous traffic.
    
• Discoverability – The discoverability setting defines under which conditions the Queue and its policy are visible in the Service Registry Atom feed and any forthcoming discoverability mechanisms. When you are browsing the Service Registry feed from within a browser, the only setting that will make the Queue visible is Public since the browser isn’t able to attach an authorization token without some scripting assistance. 
  • Managers - (default) The Queue and its policy are only discoverable if you provide an authorization token with the request and the token carries a “Manage” claim.
  • ManagersListeners – The Queue and its policy are only discoverable if you provide and authorization token with the request and the token carries a “Manage” and/or “Listen” claim.
  • ManagersListenersSenders – The Queue and its policy are only discoverable if you provide and authorization token with the request and the token carries a “Manage” and/or “Listen” and/or “Send” claim.
  • Public – The Queue and its policy are discoverable without any authorization requirement, i.e. the general public.
    
• ExpirationInstant – This is an XML dateTime value indicating the TTL (time-to-live) for the Queue. You can make Queues long-lived or short-lived. A Queue’s lifetime may be extended by changing this policy value and reapplying the policy. Once a Queue expires it is removed from the system along with all the messages that reside in it. The system may limit the Queue lifetime in the effective policy that you get back from the create/update request, but we’ll typically allow at least 24 hours without renewal. This value has a default of 24 hours, a maximum of 21 days and a minimum of 30 seconds. The value must represent a future dateTime and must be expressed in UTC.
  
• MaxMessageSize – This is a numeric value defining the maximum size of any individual message that can be accepted into the Queue. The maximum message size is expressed in bytes and includes all infrastructure-imposed and encoding-related overhead. The actual overhead varies significantly based on whether messages are sent as SOAP 1.1, SOAP 1.2, or plain HTTP frames using text or binary encoding and whether a security token is required. A very defensive assumption is to reserve 10-12KB for protocol overhead in complex cases with WS* Security; the minimal allocation for protocol overhead should be around 4KB. The default and maximum values for the maximum message size is 60KB (60*1024 bytes). The minimum value is 8KB. We suggest that the payload size for an individual message does not exceed48KB, even though you can try to push it a bit. The expectation is that these values will trend up but I don’t see them more than doubling due to throughput, timeliness and scale considerations. If your data is larger you should consider how you can chunk it up.
  
• TransportProtection – The transport protection setting defines whether senders and consumers must use HTTPS to interact with the Queue. This is the default setting. You would modify this to accommodate clients – typically on devices - that don’t do HTTPS all that well. Permitted values: 
  • AllPaths - (default) Any interaction with the queue requires HTTPS.
  • None – Any interaction may be performed using either HTTP or HTTPS.
    
• EnqueueTimeout – The enqueue timeout is an XML duration value that specifies how long an enqueue operation will hang if the queue is at capacity (full). The default value is 10 seconds, the minimum is 0 seconds, and the maximum is 60 seconds. If the timeout expires and the message could not be added to the queue during that time, the queue will act according to the Overflow policy setting.
  
• MaxConcurrentReaders - [this setting not supported/enforced in the March 2009 CTP] The maximum concurrent readers value defines how many concurrent readers are permitted on the queue. If this numeric value is smaller than the default value of 2^31 (max int), the queue protocol will switch into ‘session mode’ that grants a limited number of read-locks on the queue. The minimum value is 1.
  
• MaxDequeueRetries - [this setting not supported/enforced in the March 2009 CTP] The maximum dequeue retries value defines how often a message may be peek/locked and put back into the queue until it is considered poisonous, i.e. after how many retries it should be expected the the consumer will not be able to ever consume the message successfully, because it is malformed or the consumer experiences an error condition that requires some form of manual intervention (including a bug fix). If the message is found to be poisonous it will be sent once and without any retries to the endpoint specified in the policy’s PoisonMessageDrop value.
  
• MaxMessageAge – The maximum message age is an XML duration value indicating after what time any enqueued message is considered ‘stale” and will be automatically dropped and removed from the queue. The default value is 10 minutes (600 seconds), the minimum value is 0 seconds (which effectively means that all incoming messages get dropped), the maximum value is 7 days.
  
• MaxQueueCapacity – This numeric value indicates the maximum size of the Queue in bytes. This is a system-calculated value that cannot be set on the .NET class and should not be set by clients creating policies from scratch. The default and maximum capacity of any queue is currently capped at 2MB. This is a limitation specific to the March 2009 CTP , it’s been painful to impose this constraint, and it’s absolutely expected that this limit will be expanded significantly. 2MB still allow for a several hundred notification messages of a 2-4KB. If you need to store more data you can absolutely create several queues and partition data across them either explicitly or using Routers with a load-balancing message distribution policy (more on that in a subsequent post).
  
• MaxQueueLength – This numeric value indicates the maximum queue length. The maximum and default value is 2^31, the minimum value is 1. We’re not enforcing the exact queue length in the March 2009 CTP, but your code should assume that we do. The queue length value is the basis for the calculation of the MaxQueueCapacity, with MaxQueueCapacity = min(MaxQueueLength * MaxMessageSize, 2MB) enforcing the hard 2MB limit that is currently in effect. You don’t need to touch this value unless you’d really want a Queue that’s even more size-constrained.
  
• Overflow – The overflow policy setting becomes relevant when the queue is at capacity and the EnqueueTimeout has expired. It tells the Queue what to do with the message that just came in and that can’t be put into the queue because it’s full. Permitted values: 
  • RejectIncomingMessage - (default) The message will be rejected and an error status code or SOAP fault will be sent to the sender.
  • DiscardIncomingMessage – The sender will get an indication that the message has been accepted, but the message will be dropped on the floor.
  • DiscardExistingMessage – The Queue will remove and discard messages from the head of the Queue until the new, incoming message fits in the Queue.
    
• PoisonMessageDrop – [this setting not supported in the March 2009 CTP] This value is a WS-Addressing 1.0 EndpointReference referring to an endpoint that any poison messages will be sent to once the MaxDequeueRetries limit has been exceeded. The EndpointReference may point to any SOAP or plain HTTP endpoint that can accept ‘any’ message.

Phew! Lots of options. The good thing is that most apps should be ok with the defaults.

I know that the 2MB capacity limit is somewhat disappointing and I’m certainly not happy with it. There’s a particular behavior (with bug number) that may occur under very rare circumstances in the underlying replication system, which caused us to play it very safe instead of risking data loss. I don’t think the limit is a showstopper for apps that send notifications and events around – it is a showstopper for apps that want to exchange larger payloads and we’re working to relax that limit and make Queues much, much larger as soon as we can. You can obviously always spool larger data into SDS or one of the Azure storage systems and then send a reference to that data as a message, but it’d be strange for a messaging system to make that a required pattern for data of all sizes. If we’re talking hundreds of megabytes it makes sense, though.

With the due apology out of the way, let’s look at how a policy may be applied to a namespace name – or in other words, how a queue is created in the simplest case (this must be done via HTTPS). The model here is that the client proposes a policy and Service Bus is at liberty to adjust the policy.

If you are intimately familiar with Atom, you’ll notice that the <QueuePolicy> is an extension and isn’t carried as <content>. That’s by intent. <content> is for people, extensions are for apps. We’ll start using <content> in a later milestone, so consider that being reserved.

HTTP/1.1 POST /myapp/q1
Host: clemensv.servicebus.windows.net
X-MS-Identity-Token: A6hbJklu18hsnHRql61k1==
Content-Type: application/atom+xml;type=entry;charset=utf-8
Content-Length: nnn

<entry xmlns=”http://www.w3.org/2005/Atom”>
   <QueuePolicy xmlns=” http://schemas.microsoft.com/ws/2007/08/connect”>
      <ExpirationInstant>2009-04-03T12:00:00</ExpirationInstant>
   </QueuePolicy>
</entry>

If the queue can be created, the response is a 200 and you get the policy back along with any adjustments that the service may make. This is called the “effective” policy, i.e. that’s what the server is using. You also learn about where you can modify the policy, since – as I explained before – the endpoint where you POST the policy to is morphing into the Queue’s tail.

HTTP/1.1 200 OK
Location: https://project-name.servicebus.windows.net/myapp/q1!(queue)
Content-Type: application/atom+xml;type=entry;charset=utf-8
Content-Length: nnn

<entry xmlns=”http://www.w3.org/2005/Atom”>
   <link rel=”self” href=”https://project-name.servicebus.windows.net/myapp/q1!(queue)” />
   <link rel=”alternate” href=”https://project-name.servicebus.windows.net/myapp/q1” />
   <link rel=”queuehead” href=”https://project-name.servicebus.windows.net/myapp/q1!(queue/head)” />
   <QueuePolicy xmlns=” http://schemas.microsoft.com/ws/2007/08/connect” >
         <ExpirationInstant>2009-04-03T12:00:00</ExpirationInstant>
         <MaxQueueCapacity>2097152</MaxQueueCapacity>
   </QueuePolicy>
</entry>

If you want to renew the Queue (extend the expiration), take the effective policy, adjust ExpirationInstant and do a PUT to the “self” location.

HTTP/1.1 PUT /myapp/q1!(queue)
Host: project-name.servicebus.windows.net
X-MS-Identity-Token: A6hbJklu18hsnHRql61k1==
Content-Type: application/atom+xml;type=entry;charset=utf-8
Content-Length: nnn

<entry xmlns=”http://www.w3.org/2005/Atom”>
   <link rel=”self” href=”https://project-name.servicebus.windows.net/myapp/q1!(queue)” />
   <link rel=”alternate” href=”https://project-name.servicebus.windows.net/myapp/q1” />
   <link rel=”queuehead” href=”https://project-name.servicebus.windows.net/myapp/q1!(queue/head)” />
   <QueuePolicy xmlns=” http://schemas.microsoft.com/ws/2007/08/connect” >
         <ExpirationInstant>2009-04-04T12:00:00</ExpirationInstant>
         <MaxQueueCapacity>2097152</MaxQueueCapacity>
   </QueuePolicy>
</entry>

[Bug note: The PUT will return the new effective policy just like the POST response shown above, but the returned Atom <links> aren’t correctly formed. Keep the ones returned from the POST]

If you want to delete the queue, just nuke the policy:

HTTP/1.1 DELETE /myapp/q1!(queue)
Host: project-name.servicebus.windows.net
X-MS-Identity-Token: A6hbJklu18hsnHRql61k1==
Content-Length: 0

So that’s the policy story for Queues. In the next posts I’ll discuss the REST Queue protocol and the SOAP Queue protocol for how you send message to the queue and get messages out. REST I’ll explain in protocol terms, the SOAP model in .NET programming model terms.

… seht Euch mal die Wa an, wie die Wa ta kann. Auf der Mauer auf der Lauer sitzt ‘ne kleine Wa!.

It’s a German children’s song. The song starts out with “… sitzt ‘ne kleine Wanze” (bedbug) and with each verse you leave off a letter: Wanz, Wan, Wa, W, – silence.

I’ll do the same here, but not with a bedbug:

Let’s sing:

<soap:Envelope xmlns:soap=”” xmlns:wsaddr=”” xmlns:wsrm=”” xmlns:wsu=”” xmlns:app=””>
   <soap:Header>
         <addr:Action>http://tempuri.org/1.0/Status.set</addr:Action>
         <wsrm:Sequence>
              <wsrm:Identifier>urn:session-id</wsrm:Identifier>
              <wsrm:MessageNumber>5</wsrm:MessageNumber>
          </wsrm:Sequence>
          <wsse:Security xmlns:wsse=”…”>
               <wsse:BinarySecurityToken ValueType=" http://tempuri.org#CustomToken "
                                         EncodingType="...#Base64Binary" wsu:Id=" MyID ">
                          FHUIORv...
                </wsse:BinarySecurityToken>
               <ds:Signature>
                  <ds:SignedInfo>
                      <ds:CanonicalizationMethod Algorithm=" http://www.w3.org/2001/10/xml-exc-c14n#"/>
                      <ds:SignatureMethod Algorithm=" http://www.w3.org/2000/09/xmldsig#md5"/ > 
                      <ds:Reference URI="#MsgBody">
                            <ds:DigestMethod  Algorithm=" http://www.w3.org/2000/09/xmldsig#md5"/>  
                            <ds:DigestValue>LyLsF0Pi4wPU...</ds:DigestValue>
                      </ds:Reference>
                 </ds:SignedInfo>  
                 <ds:SignatureValue>DJbchm5gK...</ds:SignatureValue>
                 <ds:KeyInfo> 
                  <wsse:SecurityTokenReference> 
                    <wsse:Reference URI="#MyID"/>
                   </wsse:SecurityTokenReference>
               </ds:KeyInfo>
             </ds:Signature>
         </wsse:Security>
         <app:ResponseFormat>Xml</app:ResponseFormat>
         <app:Key wsu:Id=”AppKey”>27729912882….</app:Key>
    <soap:Header>
    <soap:Body wsu:Id=”MyId”>
          <app:status>Hello, I’m good</app:status>
     </soap:Body>
</soap:Envelope>

Not a very pretty song, I’ll admit. Let’s drop a some stuff. Let’s assume that we don’t need to tell the other party that we’re looking to give it an MD5 signature, but let’s say that’s implied and so were the canonicalization algorithm. Let’s also assume that the other side already knows the security token and the key. Since we only have a single signature digest here and yield a single signature we can just collapse to the signature value. Heck, you may not even know about what that all means. Verse 2:

<soap:Envelope xmlns:soap=”” xmlns:wsaddr=”” xmlns:wsrm=”” xmlns:wsu=”” xmlns:app=””>
   <soap:Header>
         <addr:Action>http://tempuri.org/1.0/Status.set</addr:Action>
         <wsrm:Sequence>
              <wsrm:Identifier>urn:session-id</wsrm:Identifier>
              <wsrm:MessageNumber>5</wsrm:MessageNumber>
          </wsrm:Sequence>
          <wsse:Security xmlns:wsse=”…”>
               <ds:Signature>
                  <ds:SignatureValue>DJbchm5gK...</ds:SignatureValue>
             </ds:Signature>
         </wsse:Security>
         <app:ResponseFormat>Xml</app:ResponseFormat>
         <app:Key wsu:Id=”AppKey”>27729912882….</app:Key>
    <soap:Header>
    <soap:Body wsu:Id=”MyId”>
          <app:status>Hello, I’m good</app:status>
     </soap:Body>
</soap:Envelope>

Better. Now let’s strip all these extra XML namespace decorations since there aren’t any name collisions as far as I can see. We’ll also collapse the rest of the security elements into one element since there’s no need for three levels of nesting with a single signature. Verse 3:

<Envelope>
   <Header>
         <Action>http://tempuri.org/1.0/Status.set</Action>
         <Sequence>
              <Identifier>urn:session-id</Identifier>
              <MessageNumber>5</MessageNumber>
          </Sequence>
          <SignatureValue>DJbchm5gK...</SignatureValue>
          <ResponseFormat>Xml</ResponseFormat>
          <Key>27729912882….</Key>
    <Header>
    <Body>
       <status>Hello, I’m good</status>
     </Body>
</Envelope>

Much better. The whole angle-bracket stuff and the nesting seems semi-gratuitous and repetitive here, too. Let’s make that a bit simpler. Verse 4:

         Action=http://tempuri.org/1.0/Status.set
         Sequence-Identifier=urn:session-id
         Sequence-MessageNumber=5
         SignatureValue=DJbchm5gK...
         ResponseFormat=Xml
         Key=27729912882….
         status=Hello, I’m good

Much, much better. Now let’s get rid of that weird URI up there and split up the action and the version info, make some of these keys are little more terse and turn that into a format that’s easily transmittable over HTTP. By what we have here application/www-form-urlencoded would probably be best. Verse 5:

         method=Status.set
         &v=1.0
         &session_key=929872172..
         &call_id=5
         &sig=DJbchm5gK...
         &format=Xml
         &api_key=27729912882….
         &status=Hello,%20I’m%20good

Oops. Facebook’s Status.set API. How did that happen? I thought that was REST?

Now play the song backwards. The “new thing” is largely analogous to where we started before the WS* Web Services stack and its CORBA/DCE/DCOM predecessors came around and there are, believe it or not, good reasons for having of that additional “overhead”. A common way to frame message content and the related control data, a common way to express complex data structures and distinguish between data domains, a common way to deal with addressing in multi-hop or store-and-forward messaging scenarios, an agreed notion of sessions and message sequencing, a solid mechanism for protecting the integrity of messages and parts of messages. This isn’t all just stupid.

It’s well worth discussing whether messages need to be expressed as XML 1.0 text on the wire at all times. I don’t think they need to and there are alternatives that aren’t as heavy. JSON is fine and encodings like the .NET Binary Encoding or Fast Infoset are viable alternatives as well. It’s also well worth discussing whether WS-Security and the myriad of related standards that were clearly built by security geniuses for security geniuses really need to be that complicated or whether we could all live with a handful of simple profiles and just cut out 80% of the options and knobs and parameters in that land.

I find it very sad that the discussion isn’t happening. Instead, people use the “REST” moniker as the escape hatch to conveniently ignore any existing open standard for tunnel-through-HTTP messaging and completely avoid the discussion.

It’s not only sad, it’s actually a bit frustrating. As one of the people responsible for the protocol surface of the .NET Service Bus, I am absolutely not at liberty to ignore what exists in the standards space. And this isn’t a mandate handed down to me, but something I do because I believe it’s the right thing to live with the constraints of the standards frameworks that exist.

When we’re sitting down and talk about a REST API, were designing a set of resources – which may result in splitting a thing like a queue into two resources, head and tail - and then we put RFC2616 on the table and try to be very precise in picking the appropriate predefined HTTP method for a given semantic and how the HTTP 2xx, 3xx, 4xx, 5xx status codes map to success and error conditions. We’re also trying to avoid inventing new ways to express things for which standards exists. There’s a standard for how to express and manage lists with ATOM and APP and hence we use that as a foundation. We use the designed extension points to add data to those lists whenever necessary.

When we’re designing a RPC SOAP API, we’re intentionally trying to avoid inventing new protocol surface and will try to leverage as much from the existing and standardized stack as we possibly can – at a minimum we’ll stick with established patterns such as the Create/GetInfo/Renew/Delete patterns for endpoint factories with renewal (which is used in several standards). I’ll add that we are – ironically - a bit backlogged on the protocol documentation for our SOAP endpoints and have more info on the REST endpoint in the latest SDK, but we’ll make that up in the near future.

So - can I build “REST” (mind the quotes) protocols that are as reduced as Facebook, Twitter, Flickr, etc? Absolutely. There wouldn’t be much new work. It’s just a matter of how we put messages on and pluck message off the wire. It’s really mostly a matter of formatting and we have a lot of the necessary building blocks in the shipping WCF bits today. I would just omit a bunch of decoration as things go out and make a bunch of assumptions on things that come in.

I just have a sense that I’d be hung upside down from a tree by the press and the blogging, twittering, facebooking community if I, as someone at Microsoft, wouldn’t follow the existing open and agreed standards or at least use protocols that we’ve published under the OSP and instead just started to do my own interpretative dance - even if that looked strikingly similar to what the folks down in the Valley are doing. At the very least, someone would call it a rip-off.

What do you think? What should I/we do?

[Update 2010-08-25: Wade Wegner  now shows a solution on his blog ]

We’ve been getting some questions along the lines of “I am hosting a service as xyz.svc in IIS and have changed the config to use on of the Service Bus bindings, but the service never gets called?”

That’s right. It doesn’t. The reason for that is that we don’t yet have WAS/IIS integration for any of the Service Bus bindings in the November 2008 CTP. Enabling the WCF WAS activation scenario that puts the NetTcpRelayBinding and friends on par with their WCF siblings is on our work backlog for the next major milestone.

It’s worth considering for a moment what that integration requires. Fundamentally, all of the Relay bindings replace the local TCP or HTTP listener with a listener that sits up in the cloud and services then connect up to that listener to create an inbound route for received messages. That’s similar to how local services interact with WCF’s shared TCP listener or HTTP.SYS, but there are quite a few important differences. First, all Relay listeners need to acquire and present an Access Control token when they start listening on the Service Bus. In contrast, the local listener facilities are ACL’d using the local or domain account system and use the Windows process identity to decide on whether a process may or may not listen on a particular port and/or namespace. Second, since the actual listener is off-machine, we need to spin up the connection as the IIS/WAS host spins up and need to make sure that the connection is kept alive and aggressively reconnects when dropped for any reason. That’s something you don’t really have to worry much about when the listener sits right there on the same machine as your own service and the connection is a named pipe. Third, the local listeners listen on a particular host address and port; the Relay listeners listen on a leaf of a namespace tree and that namespace may be shared amongst many listeners living on a multitude of different machines in different locations.  Fourth,   ... well you get the picture.

Bottom line: Not having support for WAS activation and xyz.svc service endpoints is by no means an oversight. It’s on the list.

The MSDN Developer Center for .NET Services is the first stop to go to for technical information on the Service Bus, the Access Control Service and the Workflow Service.

There quite a bit of documentation for “my” feature area, the .NET Service Bus, including description of all the bindings and most of the object model surface area. Since we had quite a bit of object model churn up until a few weeks before PDC as we’ve exploded the former, singular RelayBinding into two handful of WCF-aligned bindings, the reference documentation isn’t yet in the familiar MSDN reference format and also doesn’t yet work with Visual Studio’s “F1”. We’re obviously going to address that in the next major milestone now that the dust is settling a bit and the programming model is already quite a bit closer to what we want it to be for our “V1” release.

In the sea of PDC 2008 announcements you may have missed the following two signficant developments:

• .NET Services for Java
• .NET Services for Ruby

For the past 2 months our team has worked very closely with our partners at Schakra on the Java SDK parts and with ThoughtWorks on the Ruby parts. These are the first baby steps and these two SDKs cover only a small subset of the capabilities of the .NET SDK so far. That's merely a function of when we started with these projects and how far we've gotten with the required protocol support; we want and we will take this a lot further over the next development milestones. In the end, the .NET Services fabric ought not to care much what language the senders and listeners are written in and what platform they run on. We're building a universal services platform. We're taking Java and Ruby very seriously and have a few more platforms on the list for which we want to add explicit support.