From //build in Anaheim

Our team’s Development Manager MK (Murali Krishnaprasad) and me were interviewed by Michael Washam on May 2011 CTP release of Windows Azure AppFabric. We discuss new technologies such as Topics, Queues, Subscriptions and how this relates to doing async development in the cloud.

Republished from Channel 9

In case you need a refresher or update about the things me and our team work on at Microsoft, go here for a very recent and very good presentation by my PM colleague Maggie Myslinska from TechEd Australia 2010 about Windows Azure AppFabric with Service Bus demos and a demo of the new Access Control V2 CTP

Building “hybrid” cloud applications where parts of an an app lives up in a cloud infrastructure and other parts of the infrastructure live at a hosting site, or a data center, or even in your house ought to be simple – especially in this day and age of Web services. You create a Web service, make it accessible through your firewall and NAT, and the the cloud-hosted app calls it. That’s as easy as it ought to be.

Unfortunately it’s not always that easy. If the server sits behind an Internet connection with dynamically assigned IP addresses, if the upstream ISP is blocking select ports, if it’s not feasible to open up inbound firewall ports, or if you have no influence over the infrastructure whatsoever, reaching an on-premise service from the cloud (or anywhere else) is a difficult thing to do. For these scenarios (and others) our team is building the Windows Azure platform AppFabric Service Bus (friends call us just Service Bus).

Now – the Service Bus and the client bits in the Microsoft.ServiceBus.dll assembly are great if you have services can can be readily hooked up into the Service Bus because they’re built with WCF. For services that aren’t built with WCF, but are at least using HTTP, I’ve previously shown a way to hook them into Service Bus and have also demoed an updated version of that capability at Sun’s Java One. I’ll release an update for those bits tomorrow after my talk at PDC09 – the version currently here on my blog (ironically) doesn’t play well with SOAP and also doesn’t have rewrite capabilities for WSDL. The new version does.

But what if your service isn’t a WCF service or doesn’t speak HTTP? What if it speaks SMTP, SNMP, POP, IMAP, RDP, TDS, SSH, ETC?

Introducing Port Bridge

“Port Bridge” – which is just a descriptive name for this code sample, not an attempt at branding – is a point-to-point tunneling utility to help with these scenarios. Port Bridge consists of two components, the “Port Bridge Service” and the “Port Bridge Agent”. Here’s a picture:

The Agent’s job is to listen for and accept TCP or Named Pipe connections on a configurable port or local pipe name. The Service’s job is to accept for incoming connections from the Agent, establish a duplex channel with the Agent, and pump the data from the Agent to the actual listening service – and vice versa. It’s actually quite simple. In the picture above you see that the Service is configured to connect to a SQL Server listening at the SQL Server default port 1433 and that the Agent – running on a different machine, is listening on port 1433 as well, thus mapping the remote SQL Server onto the Agent machine as if it ran there. You can (and I think of that as to be more common) map the service on the Agent to any port you like – say higher up at 41433.

In order to increase the responsiveness and throughput for protocols that are happy to kill and reestablish connections such as HTTP does, “Port Bridge” is always multiplexing concurrent traffic that’s flowing between two parties on the same logical socket. When using Port Bridge to bridge to a remote HTTP proxy that the Service machine can see, but the Agent machine can’t see (which turns out to be the at-home scenario that this capability emerged from) there are very many and very short-lived connections being tunneled through the channel. Creating a new Service Bus channel for each of these connections is feasible – but not very efficient. Holding on to a connection for an extended period of time and multiplexing traffic over it is also beneficial in the Port Bridge case because it is using the Service Bus Hybrid connection mode by default. With Hybrid, all connections are first established through the Service Bus Relay and then our bits do a little “NAT dance” trying to figure out whether there’s a way to connect both parties with a direct socket – if that works the connection gets upgraded to the most direct connections in-flight. The probing, handshake, and upgrade of the socket may take 2-20 seconds and there’s some degree of luck involved to get that direct socket established on a very busy NAT – and thus we want to maximize the use of that precious socket instead of throwing it away all the time.

That seems familiar?!

You may notice that SocketShifter (built by our friends at AWS in the UK) is quite similar to Port Bridge. Even though the timing of the respective releases may not suggest it, Port Bridge is indeed Socketshifter’s older brother. Because we couldn’t make up our mind on whether to release Port Bridge for a while, I had AWS take a look at the service contract shown below and explained a few principles that I’m also explaining here and they had a first version of Socketshifter running within a few hours. There’s nothing wrong with having two variants of the same thing.

How does it work?

Since I’m publishing this as a sample, I obviously need to spend a little time on the “how”, even I’ll limit that here and will explain that in more detail in a future post. At the heart of the app, the contract that’s used between the Agent and the Service is a simple duplex WCF contract:

    [ServiceContract(Namespace="n:",
Name="idx", CallbackContract=typeof(IDataExchange),
SessionMode=SessionMode.Required)] public
interface IDataExchange { [OperationContract(Action="c",
IsOneWay = true, IsInitiating=true)] void Connect(string i);
[OperationContract(Action = "w",
IsOneWay = true)] void Write(TransferBuffer d);
[OperationContract(Action = "d",
IsOneWay = true, IsTerminating = true)] void Disconnect();
}

There’s a way to establish a session, send data either way, and close the session. The TransferBuffer type is really just a trick to avoid extra buffer copies during serialization for efficiency reasons. But that’s it. The rest of Port Bridge is a set of queue-buffered streams and pumps to make the data packets flow smoothly and to accept inbound sockets/pipes and dispatch them out to the proxied services. What’s noteworthy is that Port Bridge doesn’t use WCF streaming, but sends data in chunks – which allows for much better flow control and enables multiplexing.

Now you might say You are using a WCF ServiceContract? Isn’t that using SOAP and doesn’t that cause ginormous overhead? No, it doesn’t. We’re using the WCF binary encoder in session mode here. That’s about as efficient as you can get it on the wire with serialized data. The per-frame SOAP overhead for net.tcp with the binary encoder in session mode is in the order of 40-50 bytes per message because of dictionary-based metadata compression. The binary encoder also isn’t doing any base64 trickery but treats binary as binary – one byte is one byte. Port Bridge is using a default frame size of 64K (which gets filled up in high-volume streaming cases due to the built-in Nagling support) and so we’re looking at an overhead of far less than 0.1%. That’s not shabby.

How do I use it?

This is a code sample and thus you’ll have to build it using Visual Studio 2008. You’ll find three code projects: PortBridge (the Service), PortBridgeAgent (the Agent), and the Microsoft.Samples.ServiceBus.Connections assembly that contains the bulk of the logic for Port Bridge. It’s mostly straightforward to embed the agent side or the service side into other hosts and I’ll show that in a separate post.

Service

The service’s exe file is “PortBridge.exe” and is both a console app and a Windows Service. If the Windows Service isn’t registered, the app will always start as a console app. If the Windows Service is registered (with the installer or with installutil.exe) you can force console-mode with the –c command line option.

The app.config file on the Service Side (PortBridge/app.config, PortBridge.exe.config in the binaries folder) specifies what ports or named pipes you want to project into Service Bus:

           <
          portBridge 
          serviceBusNamespace
          ="mynamespace" serviceBusIssuerName="owner" serviceBusIssuerSecret="xxxxxxxx" localHostName="mybox">
<hostMappings>
<add targetHost="localhost" allowedPorts="3389" />
</hostMappings>
</portBridge>

The serviceBusNamespace attribute takes your Service Bus namespace name, and the serviceBusIssuerSecret the respective secret. The serviceBusIssuerName should remain “owner” unless you know why you want to change it. If you don’t have an AppFabric account you might not understand what I’m writing about: Go make one. 

The localHostName attribute is optional and when set, it’s the name that’s being used to map “localhost” into your Service Bus namespace. By default the name that’s being used is the good old Windows computer-name.

The hostMappings section contains a list of hosts and rules for what you want to project out to Service Bus. Mind that all inbound connections to the endpoints generated from the host mappings section are protected by the Access Control service and require a token that grants access to your namespace – which is already very different from opening up a port in your firewall. If you open up port 3389 (Remote Desktop) through your firewall and NAT, everyone can walk up to that port and try their password-guessing skills. If you open up port 3389 via Port Bridge, you first need to get through the Access Control gate before you can even get at the remote port.

New host mappings are added with the add element. You can add any host that the machine running the Port Bridge service can “see” via the network. The allowedPorts and allowedPipes attributes define with TCP ports and/or which local named pipes are accessible. Examples:

• < add targetHost ="localhost" allowedPorts="3389" /> project the local machine into Service Bus and only allow Remote Desktop (3389)
• < add targetHost ="localhost" allowedPorts="3389,1433" /> project the local machine into Service Bus and allow Remote Desktop (3389) and SQL Server TDS (1433)
• < add targetHost ="localhost" allowedPorts="*" /> project the local machine into Service Bus and only allow any TCP port connection
• < add targetHost ="localhost" allowedPipes="sql/query" /> project the local machine into Service Bus and allow no TCP connections but all named pipe connections to \.\pipes\sql\query
• < add targetHost ="otherbox" allowedPorts="1433" /> project the machine “otherbox” into Service Bus and allow SQL Server TDS connections via TCP

Agent

The agent’s exe file is “PortBridgeAgent.exe” and is also both a console app and a Windows Service.

The app.config file on the Agent side (PortBridgeAgent/app.config, PortBridgeAgent.exe.config in the binaries folder) specifies which ports or pipes you want to project into the Agent machine and whether and how you want to firewall these ports. The firewall rules here are not interacting with your local firewall. This is an additional layer of protection.

           <
          portBridgeAgent 
          serviceBusNamespace
          ="mysolution" serviceBusIssuerName="owner" serviceBusIssuerSecret="xxxxxxxx">
<portMappings>
<port localTcpPort="13389" targetHost="mymachine" remoteTcpPort="3389">
<firewallRules>
<rule source="127.0.0.1" />
<rule sourceRangeBegin="10.0.0.0" sourceRangeEnd="10.255.255.255" />
</firewallRules>
</port>
</portMappings>
</portBridgeAgent>

Again, the serviceBusNamespace attribute takes your Service Bus namespace name, and the serviceBusIssuerSecret the respective secret.

The portMappings collection holds the individual ports or pipes you want to bring onto the local machine. Shown above is a mapping of Remote Desktop (port 3389 on the machine with the computer name or localHostName ‘mymachine’) to the local port 13389. Once Service and Agent are running, you can connect to the agent machine on port 13389 using the Remote Desktop client – with PortBridge mapping that to port 3389 on the remote box.

The firewallRules collection allows (un-)constraining the TCP clients that may connect to the projected port. By default, only connections from the same machine are permitted.

For named pipes, the configuration is similar, even though there are no firewall rules and named pipes are always constrained to local connectivity by a set of ACLs that are applied to the pipe. Pipe names must be relative. Here’s how a named pipe projection of a default SQL Server instance could look like:

     < port localPipe ="sql/remote" targetHost="mymachine" remotePipe="sql/query"/>

There’s more to write about this, but how about I let you take a look at the code first. I’ve also included two setup projects that can easily install Agent and Service as Windows Services. You obviously don’t have to use those.

[Updated archive (2010-06-10) fixing config issue:]

… seht Euch mal die Wa an, wie die Wa ta kann. Auf der Mauer auf der Lauer sitzt ‘ne kleine Wa!.

It’s a German children’s song. The song starts out with “… sitzt ‘ne kleine Wanze” (bedbug) and with each verse you leave off a letter: Wanz, Wan, Wa, W, – silence.

I’ll do the same here, but not with a bedbug:

Let’s sing:

<soap:Envelope xmlns:soap=”” xmlns:wsaddr=”” xmlns:wsrm=”” xmlns:wsu=”” xmlns:app=””>
   <soap:Header>
         <addr:Action>http://tempuri.org/1.0/Status.set</addr:Action>
         <wsrm:Sequence>
              <wsrm:Identifier>urn:session-id</wsrm:Identifier>
              <wsrm:MessageNumber>5</wsrm:MessageNumber>
          </wsrm:Sequence>
          <wsse:Security xmlns:wsse=”…”>
               <wsse:BinarySecurityToken ValueType=" http://tempuri.org#CustomToken "
                                         EncodingType="...#Base64Binary" wsu:Id=" MyID ">
                          FHUIORv...
                </wsse:BinarySecurityToken>
               <ds:Signature>
                  <ds:SignedInfo>
                      <ds:CanonicalizationMethod Algorithm=" http://www.w3.org/2001/10/xml-exc-c14n#"/>
                      <ds:SignatureMethod Algorithm=" http://www.w3.org/2000/09/xmldsig#md5"/ > 
                      <ds:Reference URI="#MsgBody">
                            <ds:DigestMethod  Algorithm=" http://www.w3.org/2000/09/xmldsig#md5"/>  
                            <ds:DigestValue>LyLsF0Pi4wPU...</ds:DigestValue>
                      </ds:Reference>
                 </ds:SignedInfo>  
                 <ds:SignatureValue>DJbchm5gK...</ds:SignatureValue>
                 <ds:KeyInfo> 
                  <wsse:SecurityTokenReference> 
                    <wsse:Reference URI="#MyID"/>
                   </wsse:SecurityTokenReference>
               </ds:KeyInfo>
             </ds:Signature>
         </wsse:Security>
         <app:ResponseFormat>Xml</app:ResponseFormat>
         <app:Key wsu:Id=”AppKey”>27729912882….</app:Key>
    <soap:Header>
    <soap:Body wsu:Id=”MyId”>
          <app:status>Hello, I’m good</app:status>
     </soap:Body>
</soap:Envelope>

Not a very pretty song, I’ll admit. Let’s drop a some stuff. Let’s assume that we don’t need to tell the other party that we’re looking to give it an MD5 signature, but let’s say that’s implied and so were the canonicalization algorithm. Let’s also assume that the other side already knows the security token and the key. Since we only have a single signature digest here and yield a single signature we can just collapse to the signature value. Heck, you may not even know about what that all means. Verse 2:

<soap:Envelope xmlns:soap=”” xmlns:wsaddr=”” xmlns:wsrm=”” xmlns:wsu=”” xmlns:app=””>
   <soap:Header>
         <addr:Action>http://tempuri.org/1.0/Status.set</addr:Action>
         <wsrm:Sequence>
              <wsrm:Identifier>urn:session-id</wsrm:Identifier>
              <wsrm:MessageNumber>5</wsrm:MessageNumber>
          </wsrm:Sequence>
          <wsse:Security xmlns:wsse=”…”>
               <ds:Signature>
                  <ds:SignatureValue>DJbchm5gK...</ds:SignatureValue>
             </ds:Signature>
         </wsse:Security>
         <app:ResponseFormat>Xml</app:ResponseFormat>
         <app:Key wsu:Id=”AppKey”>27729912882….</app:Key>
    <soap:Header>
    <soap:Body wsu:Id=”MyId”>
          <app:status>Hello, I’m good</app:status>
     </soap:Body>
</soap:Envelope>

Better. Now let’s strip all these extra XML namespace decorations since there aren’t any name collisions as far as I can see. We’ll also collapse the rest of the security elements into one element since there’s no need for three levels of nesting with a single signature. Verse 3:

<Envelope>
   <Header>
         <Action>http://tempuri.org/1.0/Status.set</Action>
         <Sequence>
              <Identifier>urn:session-id</Identifier>
              <MessageNumber>5</MessageNumber>
          </Sequence>
          <SignatureValue>DJbchm5gK...</SignatureValue>
          <ResponseFormat>Xml</ResponseFormat>
          <Key>27729912882….</Key>
    <Header>
    <Body>
       <status>Hello, I’m good</status>
     </Body>
</Envelope>

Much better. The whole angle-bracket stuff and the nesting seems semi-gratuitous and repetitive here, too. Let’s make that a bit simpler. Verse 4:

         Action=http://tempuri.org/1.0/Status.set
         Sequence-Identifier=urn:session-id
         Sequence-MessageNumber=5
         SignatureValue=DJbchm5gK...
         ResponseFormat=Xml
         Key=27729912882….
         status=Hello, I’m good

Much, much better. Now let’s get rid of that weird URI up there and split up the action and the version info, make some of these keys are little more terse and turn that into a format that’s easily transmittable over HTTP. By what we have here application/www-form-urlencoded would probably be best. Verse 5:

         method=Status.set
         &v=1.0
         &session_key=929872172..
         &call_id=5
         &sig=DJbchm5gK...
         &format=Xml
         &api_key=27729912882….
         &status=Hello,%20I’m%20good

Oops. Facebook’s Status.set API. How did that happen? I thought that was REST?

Now play the song backwards. The “new thing” is largely analogous to where we started before the WS* Web Services stack and its CORBA/DCE/DCOM predecessors came around and there are, believe it or not, good reasons for having of that additional “overhead”. A common way to frame message content and the related control data, a common way to express complex data structures and distinguish between data domains, a common way to deal with addressing in multi-hop or store-and-forward messaging scenarios, an agreed notion of sessions and message sequencing, a solid mechanism for protecting the integrity of messages and parts of messages. This isn’t all just stupid.

It’s well worth discussing whether messages need to be expressed as XML 1.0 text on the wire at all times. I don’t think they need to and there are alternatives that aren’t as heavy. JSON is fine and encodings like the .NET Binary Encoding or Fast Infoset are viable alternatives as well. It’s also well worth discussing whether WS-Security and the myriad of related standards that were clearly built by security geniuses for security geniuses really need to be that complicated or whether we could all live with a handful of simple profiles and just cut out 80% of the options and knobs and parameters in that land.

I find it very sad that the discussion isn’t happening. Instead, people use the “REST” moniker as the escape hatch to conveniently ignore any existing open standard for tunnel-through-HTTP messaging and completely avoid the discussion.

It’s not only sad, it’s actually a bit frustrating. As one of the people responsible for the protocol surface of the .NET Service Bus, I am absolutely not at liberty to ignore what exists in the standards space. And this isn’t a mandate handed down to me, but something I do because I believe it’s the right thing to live with the constraints of the standards frameworks that exist.

When we’re sitting down and talk about a REST API, were designing a set of resources – which may result in splitting a thing like a queue into two resources, head and tail - and then we put RFC2616 on the table and try to be very precise in picking the appropriate predefined HTTP method for a given semantic and how the HTTP 2xx, 3xx, 4xx, 5xx status codes map to success and error conditions. We’re also trying to avoid inventing new ways to express things for which standards exists. There’s a standard for how to express and manage lists with ATOM and APP and hence we use that as a foundation. We use the designed extension points to add data to those lists whenever necessary.

When we’re designing a RPC SOAP API, we’re intentionally trying to avoid inventing new protocol surface and will try to leverage as much from the existing and standardized stack as we possibly can – at a minimum we’ll stick with established patterns such as the Create/GetInfo/Renew/Delete patterns for endpoint factories with renewal (which is used in several standards). I’ll add that we are – ironically - a bit backlogged on the protocol documentation for our SOAP endpoints and have more info on the REST endpoint in the latest SDK, but we’ll make that up in the near future.

So - can I build “REST” (mind the quotes) protocols that are as reduced as Facebook, Twitter, Flickr, etc? Absolutely. There wouldn’t be much new work. It’s just a matter of how we put messages on and pluck message off the wire. It’s really mostly a matter of formatting and we have a lot of the necessary building blocks in the shipping WCF bits today. I would just omit a bunch of decoration as things go out and make a bunch of assumptions on things that come in.

I just have a sense that I’d be hung upside down from a tree by the press and the blogging, twittering, facebooking community if I, as someone at Microsoft, wouldn’t follow the existing open and agreed standards or at least use protocols that we’ve published under the OSP and instead just started to do my own interpretative dance - even if that looked strikingly similar to what the folks down in the Valley are doing. At the very least, someone would call it a rip-off.

What do you think? What should I/we do?

We've got a discussion forum up on MSDN where you can ask questions about Microsoft .NET Services (Service Bus, Workflow, Access Control): http://social.msdn.microsoft.com/Forums/en-US/netservices/threads/

We’re thrilled to announce that the BizTalk Services “R12” Community Technology Preview (CTP) is now available for general use.

“BizTalk Services” is the code-name for a platform-in-the-cloud offering from Microsoft.  Currently in active development, BizTalk Services provides Messaging, Workflow, and Identity functionality to enable disparate applications to connect quickly and easily.
Combined together in an integrated offering, these capabilities deliver a Service Bus architectural pattern that is immediately usable by applications that need to connect across the Internet. 

Many enterprises employ the ‘Enterprise Service Bus’ pattern to interconnect disparate systems within an organizational domain. Built on Microsoft platform technology, an ESB might include building blocks such as Windows Server, Active Directory, BizTalk Server, as well as the Windows Communication Foundation and Windows Workflow Foundation technologies included in the .NET Framework.  “BizTalk Services” extends the concept of an ESB to truly exploit the Internet, for instance by exposing individual service endpoints in a secure fashion or by selectively federating elements of distinct identity systems to facilitate cross-company collaboration.
 
For ISVs and Solution Providers creating specialized business solutions that enable collaboration and information exchange across increasingly mobile and distributed work-forces, “BizTalk Services” provides the cloud-based platform building blocks to create sophisticated (Internet-) Service Bus solutions with broad reach that could otherwise only be realized by operating dedicated Data Centers of significant complexity – which is often out of reach for both, ISVs and their customers.

Major Changes

With the release of BizTalk Services “R12”, developers must update all clients and SDK installations to the new release. 

New in R12 - Workflow

The most exciting new capability we’ve added in the “R12” CTP is Workflow. These new cloud-based Workflow capabilities enable ‘service orchestration’ from the cloud.  This specialized cloud-based, or hosted, Windows Workflow Foundation runtime can orchestrate services that connect to systems in your enterprise, or to systems running anywhere on the Internet via Web services messages.  This new power and capability will enable an entirely new set of application scenarios, and we’re very excited to see what people will do with it.

In the SDK you will find  samples showing how to create and control Workflow instances hosted on the BizTalk Services cloud, including a sample Workflow implementation that monitors the availability of a website and fires multicast events into the service bus indicating the state.

New in R12 - Identity

For R12, the BizTalk Services Identity Service has been expanded and enhanced to enable more flexibility for scenarios demanded by our customers.  R12 introduces a new approach for creating, viewing, and managing access control rules. This approach relies on a few key principles outlined below:

• Every Identity Service account owns a Security Token Service (STS).
• An STS is composed of one or more scopes.
• A scope contains zero or more access control rules.
• An STS owner can grant another Identity Service account permission to edit the access control rules in a scope

A practical illustration to clarify:. The Messaging Service owns an STS whose root scope is http://connect.biztalk.net/services/. When you create a new account (newaccount) in the Identity Service, the messaging service creates a new scope http://connect.biztalk.net/services/newaccount. The Messaging Service then grants (newaccount) the permission to create access control rules in that scope. Any communication endpoints hosted there can thus be secured by the owner of the scope.  Rules from R11 accounts have been migrated to the “root” scope of the new account.

On the protocols front, we’ve added several new capabilities for ‘REST’ services. We now support integration with Windows Live ID and have added RFC2617 Basic and HTTPS/Client Certificate support for acquiring security tokens using simple HTTP GET requests.

New in R12 - Messaging

Connectivity Modes

The most fundamental new feature area in the Messaging service are the new ‘connectivity mode’ settings on the RelayBinding. Before this release, BizTalk Services clients and listeners always required outbound TCP ports 808 and 818 to be available for connecting to the BizTalk Services cloud for all connection modes except the clients of a listener running with ConnectionMode.RelayedHttp. 

In this release we are introducing three different connectivity modes: Tcp, Http, and AutoDetect. The connectivity mode can be set on a static property of the RelayBinding. The Communication\ExploringFeatures\ConnectionModes\Multicast sample shows how. For clarity: ‘Connection Mode’ defines the type of end-to-end connection that is to be established through the Relay. ‘Connectivity Mode’ defines how a particular endpoint connects up to the Relay.

The ‘Tcp’ connectivity mode is the most efficient one and works as in previous releases. The ‘Http’ mode is new. It creates a volatile FIFO buffer for messages in the BizTalk Services cloud and polls for messages using HTTP ‘parked requests’.  The Http model exhibits delivery latency characteristics similar to Tcp mode, albeit with slightly higher bandwidth consumption on idle connections. The ‘AutoDetect’ mode will check whether TCP connectivity is available and will choose ‘Tcp’ if that’s the case and ‘Http’ otherwise.

The new HTTP-based connectivity option is only effective for the RelayedOneway, RelayedMulticast and RelayedDuplex connection modes. RelayedDuplexSession, HybridDuplexSession, and RelayedHttp (listener only) still require TCP connectivity at this time. 

Transport Credentials and Unauthenticated Access

Also, in the “R12” release, the model for specifying the client credentials for the Relay has now been closely aligned with the standard WCF client credentials model. Instead of picking and instantiating token providers, there is now a TransportClientEndpointBehavior that holds all credential information and credential types. The samples in the Communication\ExploringFeatures\RelayAuthentication of the SDK download clarify the use of this new behavior.

We have added a pair of ‘WebNoAuth’ samples which introduce a new capability that we had a lot of requests for: Unauthenticated client access. When registering a service listener you can now explicitly waive the authentication requirement for clients connecting to your service. This is very useful in Web scenarios where you want to enable any HTTP client to connect to your service and don’t want them to authenticate in any way. For the time being we suggest that you always use this new  unauthenticated access mode for RelayedHttp services until we release the update for the ‘Web’ client authentication capability.

For R12, we have omitted the ‘Web’ (REST) samples for Relay authentication since that area is undergoing some substantial protocol changes.  The update for this will be released soon. In the interim, existing applications that were built on a prior release of the BizTalk Services SDK to use the authentication technique shown in the R11 ‘Web’ sample must be modified to use unauthenticated access as shown in the new ‘WebNoAuth’ sample.  

Give it a try

The new BizTalk Services “R12” CTP is online and available now for your use.  The SDK is available at http://labs.biztalk.net. If you already have an account for BizTalk Services, your accounts and settings have been migrated to the new environment. If you don’t have an account yet, just sign up, download the SDK, and get started creating the new generation of connected applications.  

The BizTalk Services CTP will be switched from the "R11" to the "R12" release starting in about 30 minutes and we expect to have a 2 hour time window (1400h-1600h PT/2300h-0100h UTC) where existing service accounts are being rolled over to the new release. We're expecting to be done with the migration by 1600h. Once the migration is done we'll give you an update on what's new in R12.

Heads up:  If things go as planned, the BizTalk Services cloud will be unavailable for a few hours during the day on Tuesday 7/15 (U.S. Pacific Time) since we're doing an update to the services and to the SDK. I will post an update with the exact time window some time on Monday. Once we're back up and have verified that everything is working as intended we'll let you know about it and tell you what's new.

Applications built on the R11 release (the release currently running in the data center) will have to be recompiled (and in some instances slightly changed) against the new R12 release's assemblies to run with R12. We've done some protocol adjustments in R12 that make this necessary - mind that we're still in "experimentation-only preview" mode here. Theory suggests that some compiled R11 applications will work against the R12 cloud, but it's not a combination we're explicitly testing as of yet. We obviously have that sort of backwards compatibility on the radar (it's SOA, should be easy, right?) but it'll likely take us a couple more revisions before we're happy enough with the baseline protocols.

[Update: The switch to R12 will happen between 1400-1600 PT/2300-0100 UTC. More later]

Didn't I write that I wanted to blog more this year? It's June, you see what came out of that.

First things, first; I'm flying to Orlando tomorrow for TechEd. Looking back at what my conference schedule looked like up until 2 years ago, it's hard to believe that this is my first (!) scheduled conference talk this year. I actually do miss the life on the road a little bit. The compensation for it is that I get to see my family every day (my daughter Eva's first birthday is coming up on June 25th) and that I'm getting to work on and define the stuff that I 'just' used to be talking about. This really is the first time that I do a talk about a Microsoft technology that I own; so that's a bit of a thing:

SOA 403 Building Federated Solutions on the Internet Service Bus
Thursday, June 5, 2008 10:15AM-11:30AM
Room: S220 C (DEV)

'Own' means here that I'm the responsible Program Manager for the entire 'Messaging' feature area of BizTalk Services in what we call the '.NET Online Services' team around here. The PM title isn't entirely accurate, because I'm also writing pretty substantial amounts of product code these days. The ability to write and contribute code into the product was the primary reason why I switched jobs and joined the team I'm now in, but it turned out that the PM role was the overall better fit for me. So I'm 60% PM and 40% Dev. Or something like that.

Back to TechEd. There are two talk about what we're building. The first one is 'today' (I'm still on Pacific Time so I realize that may be a bit late); Justin Smith will provide a broad overview on the services we're building:

SOA206 Messaging, Identity, and Workflow in the Cloud
Tuesday, June 3 10:30 AM - 11:45 AM
Room: S220 C  

I highly recommend reading Vittorio's most excellent and illuminating blog entry for how to use the new features we've added to BizTalk Identity Services for allowing you to use 3rd Party Managed Cards.

A flock of pigs has been doing aerobatics high up over Microsoft Campus in Redmond in the past three weeks. Neither City of Redmond nor Microsoft spokespeople returned calls requesting comments in time for this article. An Microsoft worker who requested anonymity and has seen the pigs flying overhead commented that "they are as good as the Blue Angels at Seafair, just funnier" and "they seem to circle over building 42 a lot, but I wouldn't know why".

In related news ...

We wrapped up the BizTalk Services "R11" CTP this last Thursday and put the latest SDK release up on http://labs.biztalk.net/. As you may or may not know, "BizTalk Services" is the codename for Microsoft's cloud-based Identity and Connectivity services - with a significant set of further services in the pipeline. The R11 release is a major milestone for the data center side of BizTalk Services, but we've also added several new client-facing features, especially on the Identity services. You can now authenticate using a certificate in addition to username and CardSpace authentication, we have enabled support for 3rd party managed CardSpace cards, and there is extended support for claims based authorization.

Now the surprising bit:

Only about an hour before we locked down the SDK on Thursday, we checked a sample into the samples tree that has a rather unusual set of prerequisites for something coming out of Microsoft:

Runtime: Java EE 5 on Sun Glassfish v2 + Sun WSIT/Metro (JAX-WS extensions), Tool: Netbeans 6.0 IDE.

The sample shows how to use the BizTalk Services Identity Security Token Service (STS) to secure the communication between a Java client and a Java service providing federated authentication and claims-based authorization.

The sample, which you can find in ./Samples/OtherPlatforms/StandaloneAccessControl/JavaEE5 once you installed the SDK, is a pure Java sample not requiring any of our bits on either the service or client side. The interaction with our services is purely happening on the wire.

If you are a "Javahead", it might seem odd that we're shipping this sample inside a Windows-only MSI installer and I will agree that that's odd. It's simply a function of timing and the point in time when we knew that we could get it done (some more on that below). For the next BizTalk Services SDK release I expect there to be an additional .jar file for the Java samples.

It's important to note that this isn't just a thing we did as a one-time thing and because we could. We have done a significant amount of work on the backend protocol implementations to start opening up a very broad set of scenarios on the BizTalk Services Connectivity services for platforms other than .NET. We already have a set of additional Java EE samples lined up for when we enable that functionality on the backend. However, since getting security and identity working is a prerequisite for making all other services work, that's where we started. There'll be more and there'll be more platform and language choice than Java down the road.

Just to be perfectly clear: Around here we strongly believe that .NET and the Windows Communication Foundation in particular is the most advanced platform to build services, irrespective of whether they are of the WS-* or REST variety. If you care about my personal opinion, I'll say that several months of research into the capabilities of other platforms has only reaffirmed that belief for me and I don't even need to put a Microsoft hat on to say that.

But we recognize and respect that there are a great variety of individual reasons why people might not be using .NET and WCF. The obvious one is "platform". If you run on Linux or Unix and/or if your deployment target is a Java Application Server, then your platform is very likely not .NET. It's something else. If that's your world, we still think that our services are something that's useful for your applications and we want to show you why. And it is absolutely not enough for us to say "here is the wire protocol documentation; go party!". Only Code is Truth.

I'm also writing "Only Code is Truth" also because we've found - perhaps not too surprisingly - that there is a significant difference between reading and implementing the WS-* specs and having things actually work. And here I get to the point where a round of public "Thank You" is due:

The Metro team over at Sun Microsystems has made a very significant contribution to making this all work. Before we started making changes to accommodate Java, there would have been very little hope for anyone to get this seemingly simple scenario to work. We had to make quite a few changes even though our service did follow the specs.

While we were adjusting our backend STS accordingly, the Sun Metro team worked on a set of issues that we identified on their end (with fantastic turnaround times) and worked those into their public nightly builds. The Sun team also 'promoted' a nightly build of Metro 1.2 to a semi-permanent download location (the first 1.2 build that got that treatment), because it is the build tested to successfully interop with our SDK release, even though that build is known to have some regressions for some of their other test scenarios. As they work towards wrapping up their 1.2 release and fix those other bugs, we’ll continue to test and talk to help that the interop scenarios keep working.

As a result of this collaboration, Metro 1.2 is going to be a better and more interoperable release for the Sun's customers and the greater Java community and BizTalk Services as well as our future identity products will be better and more interoperable, too. Win-Win. Thank you, Sun.

As a goodie, I put some code into the Java sample that might be useful even if you don't even care about our services. Since configuring the Java certificate stores for standalone applications can be really painful, I added some simple code that's using a week-old feature of the latest Metro 1.2 bits that allows configuring the Truststores/Keystores dynamically and pull the stores from the client's .jar at runtime. The code also has an authorization utility class that shows how to get and evaluate claims on the service side by pulling the SAML token out of the context and pulling the correct attributes from the token.

Have fun.

[By the way, this is not an April Fool's joke, in case you were wondering]

Reading Techmeme I'm seeing that Yahoo! went the way of laying off in the order of a thousand heads by salary ranges rather than experience level. 

Look, Seattle's reuptation for weather may be well deserved, but - on the upside - we actually have weather. You know - rain, snow, wind, freezing - all that stuff that reminds you that you are alive - PLUS 5 months of glorious spring/summer weather where it never gets too hot and never too cold. And we've got the water and the mountains. It's gorgeous here.

Oh, and, we're really not that evil here. I'm having great fun working here all day. My boss looks or acts nowhere near like Darth Vader (he's actually a former Silicon Valley dude who's still adjusting to the climate). I'm learning from people smarter than me every day. I even get to write code on platforms and runtime I've never thought I'd ever touch; and I'm not talking about VB6 or Fortran.

So if you are up for a challenge and want to extend the reach of the .NET Framework into the "cloud" like we're doing here at http://labs.biztalk.net (bigger, bigger stuff ahead), I'll stick my head out - you can write me email at clemensv@microsoft.com with your resume and I will connect you to the right folks.  

And for the long-time readers of my blog: You can write me too. If you want to make an big impact in the industry, now's the time. Oh, and, FWIW, you'll likely get to work with me, but you should rather look forward to work with the gusy I work with.

2007 I've posted some 30 entries on my blog. That's what some of the "Whoa, listen to me, I am so awesome!" blogging crowd of today typically does in a day or two. 2008 promises to be so interesting that it would be a shame not to be blogging, and hence I do. There'll be lots of things going on in tech and in the world. 

Over the past year I've been very deeply involved in the still rather stealthy project 'Oslo' about which we'll talk about in MUCH more detail throughout this year than we have at the recent conferences. When you are in a project with tight disclosure constraints there's really nothing of any substance to talk or blog about. Hence I didn't.

However, since Wednesday I have a new job. I'm now getting my hands dirty by writing code for our Internet Service Bus infrastructure that's currently code-named 'BizTalk Services'. Here, the rules of the game are very different. We're actually building most of the stuff out in the open and are inviting people to play with it. That's really more in the spirit of how I've been working with the community in the past and therefore I'm looking forward to the fun that's to be had in this new team.

Beware; since I gather that I've lost about 95% of my readership of my main at http://vasters.com/clemensv blog due to my inactivity I will use the opportunity to adjust the agenda and make it a "everything that I find interesting" place. Expect political opinion. My MSDN blog at http://blogs.msdn.com/clemensv will get mirrored copies of the tech topics as I've done that since I work here at MSFT. If you just care about the tech stuff read the MSDN mirror. 

Having an Internet Service Bus up in the cloud is not very entertaining unless there are services in the bus. Therefore, I built one (and already showed some of the code basics) that’s hopefully fun to play with and will soon share the first version with you after some scrubbing and pending a few updates to the ISB that will optimize the authentication process. It’s a 0.1 version and an experiment. The code download should be ready in the next two weeks, including those adjustments. But you can actually play with parts of it today without compiling or installing anything. The info is at the bottom of this post.

To make matters really interesting, this sample not only shows how to plug a service into the cloud and call it from some Console app, but is a combo of two rather unusual hosts for WCF services: A Windows Live Messenger Add-In that acts as the server, and a Windows Vista Sidebar gadget that acts as the client.

Since the Silicon Valley scene is currently all over Twitter and clones of Twitter are apparently popping up somewhere every day, I thought I could easily provide fodder to the proponents of the alleged Microsoft tradition of purely relying on copying other’s ideas and clone them as well ;-)  Well, no, maybe not. This is a bit different.

TweetieBot is an example of a simple personal service. If you choose to host it, you own it, you run it, you control it. The data is held nowhere but on your personal machine and it’s using the BizTalk Services ISB to stick its head up into the cloud and at a stable endpoint so that its easily reachable for a circle of friends, bridging the common obstacles of dynamic IPs, firewalls and NAT. No need to use UPnP or open up ports on your router. If you choose to do so, you can encrypt traffic so that there’s no chance that anyone looking at our ISB nor anyone else can see the what’s actually going across the wire.

Right now, lots of the Web 2.0 world lives on the assumption that everything needs to live at central places and that community forms around ad-driven hubs. The mainframe folks had a similar stance in the 70s and 80s and then Personal Computers came along. The pendulum is always swinging and I have little doubt that it will swing back to “personal” once more and that the federation of personal services will seriously challenge the hub model once more.

So what does the sample do? As indicated, TweetieBot is a bot that plugs into a Windows Live Messenger using a simple Add-In. Bart De Smet has a brilliant summary for how to build such Add-Ins. When the Add-In is active and someone chats the bot, it answers politely and remembers the chat line, time and sender. The bird has a leaky long term memory, though. It forgets everything past the last 40 lines.

Where it gets interesting is that the Add-In can stick three endpoints into the BizTalk Services ISB:

• A Request/Response Web Service that allows retrieving the list of the last 40 (or less) “tweets” and also allows client to submit tweets programmatically.
• An RSS service that allows (right now) anyone to peek in to the chat log of the last 40 tweets.
• An Event service that allows subscribers to get real-time notifications whenever a new tweet is recorded.

The accompanying Sidebar Gadget, which is implemented using WPF, is a client for two of these services.

 When you drop the Gadget on the Sidebar, it will prompt for the IM address of the TweetieBot service you’d like to subscribe to. Once you’ve authenticated at the relay using your registered Information Card, the gadget will pull and show the current list of Tweets and subscribe to the Events service for real-time updates. And whenever someone chats the bot, the Sidebar gadget will immediately show the new entry. So even though the Gadget lives on some client machine that’s hidden between several layers of firewalls and behind NAT, it can actually get push-style event notifications through the cloud!

“How do I send events to clients?” must be one of the most frequent questions that I’ve been asked about Web Services in the past several years. Well, this is your answer right here.

While I’m still toying around with the code and the guys on the 1st floor in my building are doing some tweaks on the ISB infrastructure to make multi-endpoint authentication simpler, you can already play with the bot and help me a bit:

Using Windows Live Messenger you can chat (click here) tweetiebot@hotmail.comnow. Drop a few lines. If the bot is online (which means that I’m not tinkering with it) it will reply. Then look at this RSS feed [1] and you can see what you and everyone else have been telling the bot recently. Enjoy.

[1] http://connect.biztalk.net/services/tweetiebot/tweetiebot%40hotmail.com/rss

Steve Maine explains what's in the newest revision of the BizTalk Services SDK, including quite a few (standalone-) surprises for WCF and WF developers. In case you haven't noticed, we've dropped a new and substantially expanded build of the SDK just a week after we published the first SDK.

Stop. Don't leave yet. Before you say "What do I care about BizTalk?", you should know that while BizTalk has been more or less associated with the BizTalk Server 200x product line in the past few years, (Codename-) BizTalk Services is a complementary set of functionality that's not only interesting to BizTalk Server customers, but really to all .NET developers.

Weird? Flip flopping? Confusing? No. The fact that BizTalk is not only BizTalk Server isn't really new. When BizTalk came out back in 2000 and I was very closely looking at what's going on (get it used for $2), the definition read like this in the press release:

The BizTalk Initiative represents the collective set of investments that Microsoft is making to facilitate business process integration within and between organizations using Internet-standard protocols and formats. It includes the BizTalk Framework, the BizTalk.org community and business document library, as well as BizTalk Server 2000, a business process orchestration server and tools for developing, executing and managing distributed business processes. These investments are being made in conjunction with industry standards groups, technology and service providers, as well as key global organizations.

While the envisioned schema exchange BizTalk.org fell flat since industry-wide message-level-schema standardization for "everything" more or less didn't happen in the way people initially expected, what came out of this initiative as a significant element was that the set of specifications then known as the BizTalk Framework 2.0 that acted as a foundation for quite a few of the WS-* specifications and the BizTalk Server product which evolved into a very successful and leading SOA/BPM suite that's soon seeing its next release, BizTalk Server 2006 R2. Fast forward, read Steven Martin's blog entry where he writes:

[...] We see BizTalk Services as a complement to "traditional" BizTalk Server uses on premise. As you need to coordinate SOA on a broader scale beyond the organization, we see the introduction of hosted services as one way to help support federation of business process, messaging, and identity across boundaries. Over time, we want to ensure that BizTalk Server customers will be able to easily use the cloud services in conjunction with their premise technology. [...]

So all in all, a very sane way to think about BizTalk is that the software and services we publish under that name are providing functionality for messaging, process management and connectivity that go beyond the capability of the core .NET Framework.

I wrote a slightly Twitter-inspired, fun app over the weekend that's using the BizTalk Services Connectivity service and relay. In the spirit of Software+Services I'm going to give you half of it [for now] ;-)   You must have the BizTalk Services SDK installed to run the sample.

The server app, which I'm keeping to myself for the next few days as part of the experiment, is an extension (add-in) to Windows Live Messenger. The Messenger add-in monitors all chats with tweetiebot@hotmail.com and keeps circular buffer with the last 40 incoming messages. Using the client (which is in the attached archive), you can get a list of "Tweets" and add a new one (same as chatting)

[ServiceContract(Name = "TweetieBot", Namespace = http://samples.vasters.com/2007/05/tweetiebot)]
publicinterfaceITweetieBot
{
  [OperationContract]
  IList<Tweet> GetTweets(DateTime? since);
  [OperationContract]
  void Tweet(string nickname, string text);
}

or you can subscribe to new tweets and get them as they arrive

[ServiceContract(Name = "TweetieEvents", Namespace = http://samples.vasters.com/2007/05/tweetiebot)]
publicinterfaceITweetieEvents
{
  [OperationContract(IsOneWay=true)]
  void OnTweet(Tweet tweet);
}

The client application hooks up to the client (that lives right on my desktop machine) through the BizTalk Services ISB and the server fires events back through the ISB relay into the client as new tweets arrive. So when you run the attached client app, you'll find that it starts with a dump of the current log of the bot and then keeps spitting out events as they arrive.

The client is actually pretty simple. The EventsClient is the subscriber for the pub/sub service (ConnectionMode.RelayMulticast) that writes out the received events to the console. The rest all happens in Main (parsing an validating the command line argument) and in Run.

    class Program
    {
        class EventsClient : ITweetieEvents
        {
            public void OnTweet( Tweet tweet)
            {
                Console .WriteLine( "[{0}] {1}:{2}" , tweet.Time, tweet.User, tweet.Text);
            }
        }

        static void Main( string [] args)
        {
            string usageMessage = "Usage: IMBotClient <messenger-email-address>" ;
            if (args.Length == 0)
            {
                Console .WriteLine(usageMessage);
            }
            else
            {
                if (! Regex .IsMatch(args[0], @"^([\w\-\.]+)@((\[([0-9]{1,3}\.){3}[0-9]{1,3}\])|(([\w\-]+\.)+)([a-zA-Z]{2,4}))$" ))
                {
                    Console .WriteLine(usageMessage);
                    Console .WriteLine( "'{0}' is not a valid email address" );
                }
                else
                {
                    Run(args[0]);
                }
            }
        }

        private static void Run( string emailAddress)
        {
            EndpointAddress serviceAddress =
                new EndpointAddress ( String .Format( String .Format( "sb://{0}/services/tweetiebot/{1}/service" , 
                                     RelayBinding .DefaultRelayHostName, Uri .EscapeDataString(emailAddress))));
            EndpointAddress eventsAddress =
                new EndpointAddress ( String .Format( String .Format( "sb://{0}/services/tweetiebot/{1}/events" ,
                                    RelayBinding .DefaultRelayHostName, Uri .EscapeDataString(emailAddress))));

The URI scheme for services that hook into the ISB is "sb:" and the default address of the relay is encoded in the SDK assemblies. We set up two endpoints here. One for the client channel to fetch the initial list and one for the event subscriber. 

            RelayBinding relayBinding = new RelayBinding ();
     
              ServiceHost eventsHost = new ServiceHost ( typeof ( EventsClient ));
            RelayBinding eventBinding = new RelayBinding ( RelayConnectionMode .RelayedMulticast);
            eventsHost.AddServiceEndpoint( typeof ( ITweetieEvents ), eventBinding, eventsAddress.ToString());
            eventsHost.Open();

            ChannelFactory < TweetieBotChannel > channelFactory = new ChannelFactory < TweetieBotChannel >(relayBinding, serviceAddress);
            TweetieBotChannel channel = channelFactory.CreateChannel();
            channel.Open();

The two *.Open() calls will each prompt for a CardSpace authentication, so you will have to be registered to run the sample. Once you have opened the channels (and my service is running), you'll be able to pull the list of current tweets. Meanwhile, whenever a new event pops up, the EventsClient above will write out a new line.

            IList < Tweet > tweets = channel.GetTweets(lastTime);
            foreach ( Tweet tweet in tweets)
            {
                Console .WriteLine( "[{0}] {1}:{2}" , tweet.Time, tweet.User, tweet.Text);
            }

            Console .WriteLine( "Press ENTER to quit at any time" );
            Console .ReadLine();

            eventsHost.Close();
            channel.Close();
            channelFactory.Close();
        }