December 23, 2003
@ 09:32 AM

The suggested answer to the first question is incorrect and illustrates a security problem.

Categories: Technology

Slowly, slowly I am seeing some light at the end of the tunnel designing the FABRIQ. It’s a very challenging project and although I am having a lot of fun, it’s really much harder work than I initially thought.

Today I’d like to share some details with you on how I am employing the lightweight transaction coordinator “WorkSet” that Steve Swartz and I wrote during this year’s Scalable Applications tour inside the FABRIQ.

The obvious problem with one-way pipeline processing (and a problem with the composition of independent cross-cutting concerns in general) is that failure management is pretty difficult. Once one of the pipeline components fails, other components may already have done work that might not be valid if the processing fails further down through the pipeline. The simplest example of that is, of course, logging. If you log a message as the first stage of a pipeline and a subsequent stage fails, do you want the log entry to remain where it is? The problem is: it depends. So although you might need to see the message before it is being processed by stages further down the pipeline, you can only find out whether it is flagged as success or failure once processing is complete or you may want to discard the log entry altogether on failure.

Before I go into details, I’ll clarify some of the terminology I am using:

·         A message handler is an object that typically implements a ProcessMessage() method and a property Next pointing to the handler that immediately follows it in a chain of handlers.

·         A pipeline hosts a chain of message handlers and has a “head” and a “tail” message handler which link the pipeline with that chain of handlers. The pipeline itself is a message handler itself, so that pipelines can be nested inside pipelines. The FabriqPipeline is a concrete implementation of such a pipeline that has, amongst other things, support for the mechanism described here.

·         A message is an object representing a SOAP message and has a collection of headers, a body (as an XmlReader) and a transient collection of message properties that are only valid as long as the message is in memory.

·         A work set is a lightweight, in-memory 2PC transaction that provides really only the “atomicity” and “consistency” properties out of the well-known “ACID” transaction property set. “Durability” is not a goal here and “isolation” sort of guaranteed, because messages are not shared resources. If external resources are touched, isolation needs to be guaranteed by the enlisted workers. A worker is a lightweight resource manager that can enlist into a work set and provides Prepare/Commit/Abort entry points.

Whenever a new message arrives at a FabriqPipeline, a new work set is created that governs the fault management for processing the respective message. The work set is associated with the message by creating a “@WorkSet” property on the message that references the WorkSet object. The pipeline itself maintains no immediate reference to the work set – it is message-bound.

public class FabriqWorker : IWorker
{
   private Message msg;
   private FabriqMessageHandler handler;
       
    public FabriqWorker(Message msg, FabriqMessageHandler handler)
    {
         msg = msg;
         handler = handler;
    }
       
     bool IWorker.Prepare(bool vote)
    {
      return handler.Prepare(vote, msg );
    }

    void IWorker.Abort()
    {
       handler.Abort( msg );
    }

    void IWorker.Commit()
    {
       handler.Commit( msg );
    }
}

 

The FabriqPipeline does not enlist any workers into the work set directly. Instead, message handlers enlist their workers into the work set as the message flows through the pipeline. A “worker” is an implementation of the IWorker interface that can be enlisted into a work set as a participant. Because the pipeline instance along with all message handler instances shall be reusable and shall be capable of processing several messages concurrently, the worker is not implemented on the handler itself. Instead, workers are implemented as a separate helper class (FabriqWorker). Instances of these worker classes are enlisted into the message’s work set. The worker instance gets a reference to the message it deals with and to the handler which enlisted it into the work set; once the worker is called during the 2 phase commit protocol phases, it calls the message handler’s implementation of Prepare/Abort/Commit.

This way, we can have one “all in one place” implementation of all message-handling on the message handler, but are keeping the transaction dependent state in a very lightweight object; therefore we can share the entire (likely complex) pipeline and handlers for many concurrent transactions, because none of the pipeline is made dependent on the message or transaction state.

public abstract class FabriqMessageHandler :
   IMessageHandler
{
   IMessageHandler next = null;
   public FabriqMessageHandler()
   {
   }
  
   public virtual IMessageHandler Next  
   { get { return next; } set { next = value; }}
  
   bool IMessageHandler.Process(Message msg)
   {
      bool result = this.Preprocess(msg);
      WorkSet workSet = msg.Properties["@WorkSet"] as WorkSet;
      if ( workSet != null )
      {
          workSet.Register(new FabriqWorker( msg, this ) );
      }
      return result;           
   }

   protected bool Forward( Message msg )
   {
      if ( next != null )
      {
         return next.Process( msg );
      }
      else
      {
         return false;
      }
   }

   public virtual bool Preprocess( Message msg )
   {
       return false;
   }

   public abstract bool Prepare( bool vote, Message msg );

   public virtual void Commit( Message msg )
   {
   }

   public virtual void Abort( Message msg )
   {
   }
}

 

 

 

When a message flows into the pipeline, all a transactional message handler does when it gets called in ProcessMessage() is to enlist its worker and return. If the handler is not transactional, it must never fail (such things exist), can ignore the whole work set story and simply forward the message to the Next handler. So, in fact, a transactional message handler will never forward the message in the (non-transactional) ProcessMessage() method.

One problem that the dependencies between message handlers create is that it may be impossible to forward a message to the next message handler in the chain before the message is processed; at least you can’t make a Prepare==true promise for the transaction outcome until you’ve done most work on the message and have verified that all resultant work will very likely succeed. Messages may even be transformed into new messages or split into multiple messages inside the pipeline, so that you can’t do anything meaningful until you are at least preparing.

The resulting contradiction is that a transaction participant cannot perform all work resulting from on a message before it is asked to commit work, but that message handlers following in the sequence may not have received the resulting message until then and may not even be enlisted into the transaction.

To resolve this problem, the FABRIQ pipeline’s transaction management is governed by some special transaction handling rules that are more liberal than those of traditional transaction coordinators.

·         During the first (prepare) phase of the 2-phase commit protocol, workers may still enlist into the transaction. This allows a message handler to forward messages to a not-yet-enlisted message handler during the prepare phase. The worker(s) that is/are enlisted by a subsequent handler because the currently preparing message handler is forwarding one (or multiple) messages to it, is/are appended to the list of workers in the work set and asked to prepare their work once the current message handler is done preparing. We call this method a “rolling enlistment” during prepare.

·         Inside the pipeline, messages are considered to be transient data. Therefore, they may be manipulated and passed on during the Prepare phase, independent of the overall transaction outcome. The tail of the transaction controller pipeline (which is the outermost pipeline object) always enlists a worker into the transaction that will only forward messages to outside parties on Commit() and therefore takes care of hiding the transaction work to guarantee isolation.

·         Changes to any resources external to the message (so, anything that is not contained in message properties or message headers) must be guarded by the transaction workers. This means that all usual rules about guarding intermediate transaction state and transaction resources apply: The ability to make changes must be verified by tentative actions during Prepare() and changes may only be finally performed in Commit(). In case the external resources do not permit tentative actions, the Abort() method must take the necessary steps to undo actions performed during Prepare().

Whenever new messages get created during processing, the message properties (which hold the reference to the work set and, hence, to the current transaction) may be propagated into the newly created message, which causes the processing of these messages to be enlisted in the transaction, or a new or no work set can be created so that further processing of these messages is separate from the ongoing transaction. That’s what we do for failure messages.

During prepare, participants can log failure information to a message property called “@FaultInfo” that contains a collection of FaultInfo objects. If message processing fails, this information is logged and is, if possible, relayed to the message sender’s WS Addressing wsa:FaultTo, wsa:ReplyTo or wsa:From destination (in that order of preference) in a SOAP fault message.

For integration with “real” transactions, the entire work set may act as a DTC resource manager. If that’s so, the 2PC management is done by DTC and the work set acts as an aggregating proxy for the workers toward DTC. It collects its vote from its own enlistments and forwards the Commit/Abort to its enlistments.

Categories: Architecture | FABRIQ

December 13, 2003
@ 07:32 AM

There's a free weblog hosting site under the domain dasblog.com. I'd like to inform you (my readers) that neither me nor newtelligence AG are associated with that site and that we are not providing that service. The operator of the site does certainly nothing to make that clear. We're currently investigating the situation.

Categories: newtelligence | dasBlog

Yesterday, at the Microsoft EMEA Architect Forum in Portugal, I found out yet again that writing slide decks that work for all of Europe is a bit challenging. To make a point about some of the benefits of using an EAI/B2B tool, I was citing some points from a BizTalk case study talking about how the Swedish company Svenska Foder implemented their B2B infrastructure. Guess what? "Foder" isn't something you use to feed animals in Portuguese; it's rather a pretty strong word describing what people do when they're trying (or not trying) to make babies.

Categories: Talks

December 8, 2003
@ 11:38 AM

Die Harald Schmidt Show wird eingestellt. Höchste Zeit, SAT.1 von der Fernbedienung zu verbannen. Mensch, das versaut einem doch direkt den ganzen Wochenablauf. "Freiwillige Bildschirmpause"; dass ich nicht lache.

Categories: Other Stuff

December 6, 2003
@ 06:27 PM

„Software architecture is a tough thing - a vast, interesting and largely unexplored subject area. And of course everyone has something to say about it!”

Go, get and read the first issue of the Microsoft EMEA Architect’s JOURNAL. You will be surprised in many ways.

*(I wrote an article on blogging in general and dasBlog in particular for this issue. Here’s the shortcut.)

 

Categories: Architecture

December 6, 2003
@ 03:33 PM

The Releases section over at the GotDotNet workspace has three variants of v1.5:

"Source ZIP" with the code, "Web Files" with the runtime files (to update existing installs and do manual installs) and "Web Setup" which is an MSI to install the web site. Because the feature list keep growing, but we haven't done a language update for the various local languages, yet, updates to the string tables are welcome in the workspace source control system. And before you update: make a backup.

I am running v1.5 here without any problems. Let us know in the GotDotNet workspace message boards if you find any. I likely won't be able to answer any support questions this or next week.

Please note that you must not have the Whidbey Alpha version of ASP.NET mapped to the web into which you install this version; it can be present on the box, but not on the web for dasBlog. Otherwise you will get all sorts of assertions and error messages that aren't my fault ;)

 

Categories: dasBlog

December 5, 2003
@ 06:35 PM

I had too little time today to make the necessary corrections on the few install issues that I found due to the workspace fellows not adding the files where they need to be (It's tricky so not necessarily their fault). The problem is not fixing, the problem is testing: Install the source MSI, build, find problem. Go back to original version, fix, full build, install source MSI ... you get the picture. 

Categories: dasBlog

I think I am done with v1.5 of dasBlog. However, experience tells me that I'll usually find a tiny problem just after I've packaged it all up and posted it, so I'll keep the source tree checked out and the files sitting here for another 24 hours until I am sure that it works well here.

Omar and Harry have done a fantastic job cleaning up some of my mess and they've added some cool new things.

We've got a simple search facility now, comments can be deleted, the administrator UI got a bit better organized, a bunch of annoying little bugs were fixed, the stylish "dasBlog" theme does work now and is the default, you can finally change your password through the UI, a corrupt login-cookie won't crash the site anymore, comments now show up in the correct time zone, the SMTP server settings for notifications can now be tested, we have a plugin for NewsGator, etc.

 

Categories: dasBlog

"...by the way my colleague on the shit project at work said today it might be easier if we just killed the customer."

Categories: Other Stuff

December 3, 2003
@ 11:12 PM
Arvindra Sehmi, who is “Senior Architect” at Microsoft EMEA, is indeed one of the most brilliant architects I know and also happens to be the project manager and “owner” of the project I am working on as the lead architect at the moment (I’ve hinted at it here and here) has finally allowed me to say bit more about what we’re up to. The goal of this project, code-named “FABRIQ”, is to create a special-purpose, high-performance, service-oriented, one-way messaging infrastructure for queuing networks, agents and agile computing. It’s not a Microsoft product. It’s an architecture blue-print backed by code that we write so that customers don’t need to – at least that’s the plan. In case that doesn’t tell you anything, I’ll try to give you a little bit of an idea (It’s long, but it’s hopefully worth it) ....
Categories: Architecture

I am very much looking forward to the “EMEA Microsoft Longhorn Developer Preview Tour” that’s going to happen in a very dense 3 week stretch in late January / early February 2004. I feel honored to have been invited again to present the highlights of the PDC on a speaking tour throughout Europe (as in 2002) with David Chappell and an excellent group of Microsoft EMEA technical evangelists (Lester Madden, Nigel Watling, and Hans Verbeeck). We are going to be in 13 countries within 3 weeks – or 15 workdays. I will post links to the individual country’s event sites as I learn about them. In one day, we’ll take you through the best of Longhorn, WinFS, Avalon, the Visual Studio Whidbey release and Indigo (my part). If you weren’t at PDC, you should go. If you were at PDC, you should still go just to hear David speak. :-D

Here’s the first event I know the official site of. The Developer and ITPro days in Belgium are, however, much bigger than “just” our tour. We’ll be there on the second day (Feb 11th), but there’s a very exciting program on the first day already and the array of speakers is nothing less than impressive. (I just wonder why some of the speakers look like lizards right now)

Developer and ITPro Days 2004. February 10th-11th 2004, Ghent, Belgium. I’ll be there.

 

Categories: Talks | EMEA Longhorn Preview