See Part 1

Before we can do anything about deadlocks or deal with similar troubles, we first need to be able to tell that we indeed have a deadlock situation. Finding this out is a matter of knowing the respective error codes that your database gives you and a mechanism to bubble that information up to some code that will handle the situation. So before we can think about and write the handling logic for failed/failing but safely repeatable transactions, we need to build a few little things. The first thing we’ll need is an exception class that will wrap the original exception indicating the reason for the transaction failure. The new exception class’s identity will later serve to filter out exceptions in a “catch” statement and take the appropriate actions.

using System;
using System.Runtime.Serialization;

namespace newtelligence.EnterpriseTools.Data
{
   [Serializable]
   public class RepeatableOperationException : Exception
   {
       public RepeatableOperationException():base()
       {
       }

       public RepeatableOperationException(Exception innerException)
           :base(null,innerException)
       {
       }

       public RepeatableOperationException(string message, Exception innerException)
           :base(message,innerException)
       {
       }

       public RepeatableOperationException(string message):base(message)
       {
       }

        public RepeatableOperationException(
          SerializationInfo serializationInfo,
          StreamingContext streamingContext)
            :base(serializationInfo,streamingContext)
        {
        }

        public override void GetObjectData(
           System.Runtime.Serialization.SerializationInfo info,
           System.Runtime.Serialization.StreamingContext context)
        {
            base.GetObjectData (info, context);
        }
   }
}

Having an exception wrapper with the desired semantics, we know need to be able to figure out when to replace the original exception with this wrapper and re-throw it up on the call stack. The idea is that whenever you execute a database operation – or, more generally, any operation that might be repeatable on failure – you will catch the resulting exception and run it through a factory, which will analyze the exception and wrap it with the RepeatableOperationException if the issue at hand can be resolved by re-running the transaction. The (still a little naïve) code below illustrates how to such a factory in the application code. Later we will flesh out the catch block a little more, since we will lose the original call stack if we end up re-throwing the original exception like shown here:

Try
{
   dbConnection.Open();
   sprocUpdateAndQueryStuff.Parameters["@StuffArgument"].Value = argument;
   result = this.GetResultFromReader( sprocUpdateAndQueryStuff.ExecuteReader() );
}
catch( Exception exception )
{
   throw RepeatableOperationExceptionMapper.MapException( exception );                           
}
finally
{
   dbConnection.Close();
}

The factory class itself is rather simple in structure, but a bit tricky to put together, because you have to know the right error codes for all resource managers you will ever run into. In the example below I put in what I believe to be the appropriate codes for SQL Server and Oracle (corrections are welcome) and left the ODBC and OLE DB factories (for which would have to inspect the driver type and the respective driver-specific error codes) blank. The factory will check out the exception data type and delegate mapping to a private method that is specialized for a specific managed provider.

using System;
using System.Data.SqlClient;
using System.Data.OleDb;
using System.Data.Odbc;
using System.Data.OracleClient;

namespace newtelligence.EnterpriseTools.Data
{
   public class RepeatableOperationExceptionMapper
   {
        /// <summary>
        /// Maps the exception to a Repeatable exception, if the error code
        /// indicates that the transaction is repeatable.
        /// </summary>
        /// <param name="sqlException"></param>
        /// <returns></returns>
        private static Exception MapSqlException( SqlException sqlException )
        {
            switch ( sqlException.Number )
            {
                case -2: /* Client Timeout */
                case 701: /* Out of Memory */
                case 1204: /* Lock Issue */
                case 1205: /* Deadlock Victim */
                case 1222: /* Lock Request Timeout */
                case 8645: /* Timeout waiting for memory resource */
                case 8651: /* Low memory condition */
                    return new RepeatableOperationException(sqlException);
                default:
                    return sqlException;
            }
        }

        private static Exception MapOleDbException( OleDbException oledbException )
        {
            switch ( oledbException.ErrorCode )
            {
                default:
                    return oledbException;
            }
        }

        private static Exception MapOdbcException( OdbcException odbcException )
        {
            return odbcException;           
        }

        private static Exception MapOracleException( OracleException oracleException )
        {
            switch ( oracleException.Code )
            {
                case 104:  /* ORA-00104: Deadlock detected; all public servers blocked waiting for resources */
                case 1013: /* ORA-01013: User requested cancel of current operation */
                case 2087: /* ORA-02087: Object locked by another process in same transaction */
                case 60:   /* ORA-00060: Deadlock detected while waiting for resource */
                    return new RepeatableOperationException( oracleException );
                default:
                    return oracleException;
            }
        }

        public static Exception MapException( Exception exception )
        {
            if ( exception is SqlException )
            {
                return MapSqlException( exception as SqlException );
            }
            else if ( exception is OleDbException )
            {
                return MapOleDbException( exception as OleDbException );
            }
            else if (exception is OdbcException )
            {
                return MapOdbcException( exception as OdbcException );
            }
            else if (exception is OracleException )
            {
                return MapOracleException( exception as OracleException );
            }
            else
            {
                return exception;
            }
        }
   }
}

With that little framework of two classes, we can now selectively throw exceptions that convey whether a failed/failing transaction is worth repeating. Next step: How do we do actually run such repeats and make sure we neither lose data nor make the user unhappy in the process? Stay tuned.

Categories: Architecture | SOA | Enterprise Services | MSMQ

Deadlocks and other locking conflicts that cause transactional database operations to fail are things that puzzle many application developers. Sure, proper database design and careful implementation of database access (and appropriate support by the database engine) should take care of that problem, but it cannot do so in all cases. Sometimes, especially under stress and other situations with high lock contention, a database just has not much of a choice but picking at least one of the transactions competing for the same locks as the victim in resolving the deadlock situation and then aborts the chosen transaction. Generally speaking, transactions that abort and roll back are a good thing, because this behavior guarantees data integrity. In the end, we use transaction technology for those cases where data integrity is at risk. What’s interesting is that even though transactions are a technology that is explicitly about things going wrong, the strategy for dealing with failing transaction is often not much more than to bubble the problem up to the user and say “We apologize for the inconvenience. Please press OK”.

The appropriate strategy for handling a deadlock or some other recoverable reason for a transaction abort on the application level is to back out of the entire operation and to retry the transaction. Retrying is a gamble that the next time the transaction runs, it won’t run into the same deadlock situation again or that it will at least come out victorious when the database picks its victims. Eventually, it’ll work. Even if it takes a few attempts. That’s the idea. It’s quite simple.

What is not really all that simple is the implementation. Whenever you are using transactions, you must make your code aware that such “good errors” may occur at any time. Wrapping your transactional ODBC/OLEDB/ADO/ADO.NET code or calls to transactional Enterprise Services or COM+ components with a try/catch block, writing errors to log-files and showing message boxes to users just isn’t the right thing to do. The right thing is to simply do the same batch of work again and until it succeeds.

The problem that some developers seem to have with “just retry” is that it’s not so clear what should be retried. It’s a problem of finding and defining the proper transaction scope. Especially when user interaction is in the picture, things easily get very confusing. If a user has filled in a form on a web page or some dialog window and all of his/her input is complete and correct, should the user be bothered with a message that the update transaction failed due to a locking issue? Certainly not. Should the user know when the transaction fails because the database is currently unavailable? Maybe, but not necessarily. Should the user be made aware that the application he/she is using is for some sudden reason incompatible with the database schema of the backend database? Maybe, but what does Joe in the sales department do with that valuable piece of information?

If stuff fails, should we just forget about Joe’s input and tell him to come back when the system is happier to serve him? So, in other words, do we have Joe retry the job? That’s easy to program, but that sort of strategy doesn’t really make Joe happy, does it?

So what’s the right thing to do? One part of the solution is a proper separation between the things the user (or a program) does and the things that the transaction does. This will give us two layers and “a job” that can be handed down from the presentation layer down to the “transaction layer”. Once this separation is in place, we can come up with a mechanism that will run those jobs in transactions and will automate how and when transactions are to be retried. Transactional MSMQ queues turn out to be a brilliant tool to make this very easy to implement. More tomorrow. Stay tuned.

Categories: Architecture | SOA | Enterprise Services | MSMQ

November 29, 2004
@ 03:00 PM

If I were really good at writing about life, love, happiness and tragedy, weird relationships, drama, grand obstacles, successes and defeats, and all those sudden unexpected turns and twists that a story could possibly have, and I had been willing to share what went on with and around me in real life in the last six months -- my blog would now have an entirely different audience and I could easily sell the movie rights by now. So the actual reason why you haven't seen much happening here is simply that a dramatic surge in "personal life activity" (no, not starting at "no life") took over the "blogging timeslice" and had, frankly, some adverse effects on my work morale at times. The good news is that there is definitely light at the end of that tunnel and the better news (for you as a reader) is that this place here won't be as quiet as it has been in the recent months. I've got some interesting stuff cooking.

Categories: Other Stuff

I am presently doing some intense research on services, service patterns, message exchange patterns and many other issues related to services (No surprise there). However, I can't do that without external help and since many people are reading my blog, I can just as well start asking around right here:

I would like to get in touch with companies (preferrably insurances and banks) who afford a corporate history department. The ambitious goal I have is to reconstruct a few banking or insurance or purchasing business processes of ca. 1955-1965. I have come to believe that there is a lot, a lot to be learned there that will be very useful to what we're all doing. The deal is that if you share, I share whatever I have as soon as I have it. My contact address is clemensv@newtelligence.com

Categories: SOA

November 18, 2004
@ 07:59 AM

The good news is that the V|@gr@ spam is getting less, but what scares me is that I start getting lots and lots of religious spam from Jesusland. 

Categories: Other Stuff

November 7, 2004
@ 08:27 AM

In two hours I'll be back on the road (well, airport, to be precise). Today I will fly out to Reykjavik in Iceland where Achim and I will do the first of a series of SOA workshops with Microsoft EMEA from Monday to Wednesday, explaining principles of Service Oriented Architectures and the application of those principles in real applications with today's technologies. Other stops on the tour will be in Denmark in early December and, early next year, in Poland, Belgium and the Netherlands (AFAIK, all of these events are invite-only Microsoft customer events). A German-language, newtelligence-branded edition of that workshop will take place December 1-3 in Düsseldorf and we plan a newtelligence event in South Africa in early March 2005.

When I come back from Iceland Wednesday night, I'll stay home for less than 12 hours and will then fly out to Denver for a long weekend and when I come back from there the following Wednesday I'll go straight at our own TornadoCamp event held in Bad Ems (half way between Frankfurt and Düsseldorf). Coming back it'll be another short turnaround of likely less than a day before I will leave for the Microsoft EastMed Developer Conference in Amman. (Very much looking forward to that)

So with that schedule and a few customer engagements in between, I have plenty of days on the road and only two days left at the newtelligence office, before I'll move my office desk to Denver on December 11 for the rest of the year and into the new year so that I can spend Christmas with Jen, get some better traction with Visual Studio 2005 and do some writing. And for when I come back on January 10, the schedule looks just as busy for the following weeks and months.

Categories: Other Stuff | Travel

November 5, 2004
@ 09:11 AM

I've got nothing against advertising on websites. However, there are two things that are completely annoying. The first are popup windows and my popup blocker is taking care of those. As an alternative, the advertising people have invented the Macromedia Flash-popup that pops up on the page and obscures the content for a little while. That's annoying but something I can absolutely deal with. What I cannot deal with, and that's the second annoying thing, is that some advertising twits try to entertain me with music and or other sorts of 30-second audio/video shows. You people might find that cool, but I don't. Sound effects and music are strictly an "opt-in" feature on my computer and at my work desk. I just uninstalled the Flash player. Silence has returned and websites became instantly more useful. Try it.

[Usually it's as easy as this: In Internet Explorer select "Tools/Internet Options" on the menu, then click the "Settings..." button in the "Temporary Internet Files" box on the "General" tab of the dialog that opens up. Click the "View Objects" button in the "Settings" dialog that opens up. There will be another window opening. Find "Shockwave Flash" and delete the object. Close IE. Done.]

Categories: Other Stuff

November 4, 2004
@ 11:04 PM

Wenn eine Regierung soweit ist, daß sie den nationalen Feiertag aus kurzfristigen, taktischen Erwägungen abschaffen und für die eingebildete Hoffnung auf ein halbes Prozentchen Wachstum im nächsten Jahr verhökern will, dann ist das die bedingungslose Kapitulation vor der eigenen Inkompetenz. Und obwohl ich ganz sicherlich nicht dem rechten, nationalen politischen Rand zuneige, fällt mir nur ein einziges passendes Wort dazu ein: "Vaterlandsverrat".

Categories: Other Stuff