Clemens Vasters - Technology|CLR

Are you catching falling knives?

Thu, 06 Sep 2012 19:08:10 GMT

as I thumb through some people's code on Github, I see a fairly large number of "catch all" exception handling cases. It's difficult to blame folks for that, since there's generally (and sadly) very little discipline about exception contracts and exception masking, i.e. wrapping exceptions to avoid bubbling through failure conditions of underlying implementation details.

If you're calling a function and that sits on a mountain of dependencies and folks don't care about masking exceptions, there are many dozens of candidate exceptions that can bubble back up to you and there's little chance to deal with them all or even knowing them. Java has been trying to enforce more discipline in that regards, but people cheat there with "catch all" as well. There's also a question what the right way tot deal with most exceptions is. In many cases, folks implement "intercept, shrug and log" and mask the failure by telling users that something went wrong. In other common cases, folks implement retries. It's actually fairly rare to see deeply customized and careful reactions to particular exceptions. Again - things are complicated and exceptions are supposed to be exceptional (reminder: throwing exceptions as part of the regular happy path is horrifingly bad for performance and terrible from a style perspective), so these blanket strategies are typically an efficient way of dealing with things.

That all said ...

Never, never ever do this:

try {
Work(); } catch { }

And not even this:

try {
Work(); } catch(Exception e) { Trace.TraceError(e.ToString());
}

Those examples are universally bad. (Yes, you will probably find examples of that type even in the archive of this blog and some of my public code. Just goes to show that I've learned some better coding practices here at Microsoft in the past 6 1/2 years.)

The problem with them is that they catch not only the benign stuff, but they also catch and suppress the C# runtime equivalent of the Zombie Apocalypse. If you get thread-abort, out-of-memory, or stack-overflow exceptions thrown back at you, you don't want to suppress those. Once you run into these, your code has ignored all the red flags and exhausted its resources and whatever it was that you called didn't get its job done and likely sits there as a zombie in an undefined state. That class of exceptions is raining down your call stack like a shower of knife blades. They can't happen. Your code must be defensively enough written to never run into that situation and overtax resources in that way; if it does without you knowing what the root cause is, this is an automatic "Priority 0", "drop-everything-you're-working-on" class bug. It certainly is if you're writing services that need to stay up 99.95%+.

What do we do? if we see any of those exceptions, it's an automatic death penalty for the process. Once you see an unsafe out-of-memory exception or stack overflow, you can't trust the state of the respective part of the system and likely not the stability of the system. Mind that there's also a "it depends" here; I would follow a different strategy if I was talking about software for an autonomous Mars Rover that can't crash even if its gravely ill. There I would likely spend a few months on the exception design and "what could go wrong here" before even thinking about functionality, so that's a different ballgame. In a cloud system, booting a cluster machine that has the memory flu is a good strategy.

Here's a variation of the helper we use:

public static bool IsFatal(this Exception
exception) { while (exception != null)
{ if (exception as OutOfMemoryException
!= null && exception as InsufficientMemoryException
== null || exception as ThreadAbortException
!= null || 


exception as AccessViolationException != null ||
exception as SEHException != null ||
exception as StackOverflowException != null)
{ return true; } else { if (exception as TypeInitializationException
== null && exception as TargetInvocationException
== null) { break; } exception = exception.InnerException;
} } return false; }

If you put this into a static utility class, you can use this on any exception as an extension. And whenever you want to do a "catch all", you do this:

try {
DoWork();

} catch (Exception e) { if (e.IsFatal())
{ throw; } Trace.TraceError(..., e); }

If the exception is fatal, you simply throw it up as high as you can. Eventually it'll end up on the bottom of whatever thread they happen on (where you might log and rethrow) and will hopefully take the process with it. Threads marked as background threads don't do that, so it's actually not a good idea to use those. These exceptions are unhandled, process-terminating disasters with a resulting process crash-dump you want to force in a 24/7 system so that you can weed them out one by one.

(Update) As Richard Blewett pointed out after reading this post, the StackOverflowException can't be caught in .NET 2.0+, at all, and the ThreadAbortException automatically rethrows even if you try to suppress it. There are two reasons for them to be on the list: first, to shut up any code review debates about which of the .NET stock exceptions are fatal and ought to be there; second, because code might (ab-)use these exceptions as fail-fast exceptions and fake-throw them, or the exceptions might be blindly rethrown when marshaled from a terminated background thread where they were caught at the bottom of the thread. However they show up, it's always bad for them to show up.

If you catch a falling knife, rethrow.

The Magical Input Queue Of T (aka InputQueue<T>)

Sun, 12 Sep 2010 07:29:23 GMT

This post explains an essential class for asynchronous programming that lurks in the depths of the WCF samples: InputQueue<T>. If you need to write efficient server-side apps, you should consider reading through this and add InputQueue<T> to your arsenal.

Let me start with: This blog post is 4 years late. Sorry! – and with that out of the way:

The WCF samples ship with several copies of a class that’s marked as internal in the System.ServiceModel.dll assembly: InputQueue<T>. Why are these samples – mostly those implementing channel-model extensions – bringing local copies of this class with them? It’s an essential tool for implementing the asynchronous call paths of many aspects of channels correctly and efficiently.

If you look closely enough, the WCF channel infrastructure resembles the Berkeley Socket model quite a bit – especially on the server side. There’s a channel listener that’s constructed on the server side and when that is opened (usually under the covers of the WCF ServiceHost) that operation is largely equivalent to calling ‘listen’ on a socket – the network endpoint is ready for business. On sockets you’ll then call ‘accept’ to accept the next available socket connection from a client, in WCF you call ‘AcceptChannel’ to accept the next available (session-) channel. On sockets you then call ‘receive’ to obtain bytes, on a channel you call ’Receive’ to obtain a message.

Before and between calls to '’AcceptChannel’ made by the server-side logic, client-initiated connections – and thus channels – may be coming in and queue up for a bit before they handed out to the next caller of ‘AcceptChannel’, or the asynchronous equivalent ‘Begin/EndAcceptChannel’ method pair. The number of channels that may be pending is configured in WCF with the ‘ListenBacklog’ property that’s available on most bindings.

I wrote ‘queue up’ there since that’s precisely what happens – those newly created channels on top of freshly accepted sockets or HTTP request channels are enqueued into an InputQueue<T> instance and (Begin-)Accept is implemented as a dequeue operation on that queue. There are two particular challenges here that make the regular Queue<T> class from the System.Collections.Generic namespace unsuitable for use in the implementation of that mechanism: Firstly, the Dequeue method there is only available as a synchronous variant and also doesn’t allow for specifying a timeout. Secondly, the queue implementation doesn’t really help much with implementing the ListenBacklog quota where not only the length of the queue is limited to some configured number of entries, but accepting further connections/channels from the underlying network is also suspended for as long as the queue is at capacity and needs to resume as soon as the pressure is relieved, i.e. a caller takes a channel out of the queue.

To show that InputQueue<T> is a very useful general purpose class even outside of the context of the WCF channel infrastructure, I’ve lifted a version of it from one of the most recent WCF channel samples, made a small number of modifications that I’ll write about later, and created a little sample around it that I’ve attached to this post.

The sample I’ll discuss here is simulating parsing/reading IP addresses from a log-file and then performing a reverse DNS name resolution on those addresses – something that you’d do in a web-server log-analyzer or as the background task in a blog engine wile preparing statistics.

Reverse DNS name resolution is quite interesting since it’s embarrassingly easy to parallelize and each resolution commonly takes a really long time (4-5 seconds) –whereby all the work is done elsewhere. The process issuing the queries is mostly sitting around idle waiting for the response. Therefore, it’s a good idea to run a number of DNS requests in parallel, but it’s a terrible idea to have any of these requests execute as a blocking call and burning a thread. Since we’re assuming that we’re reading from a log file that requires some parsing, it would also be a spectacularly bad idea to have multiple concurrent threads compete for access to that file and get into each other’s way. And since it is a file and we need to lift things up from disk, we probably shouldn’t do that ‘just in time’ as a DNS resolution step is done, but there should rather be some data readily waiting for processing. InputQueue<T> is enormously helpful in such a scenario.

The key file of the sample code – the implementation of the queue itself aside – is obviously Program.cs. Here’s Main() :

static void Main(string[]
args)

    int maxItemsInQueue = 10;

    InputQueue<IPAddress> logDataQueue = new InputQueue<IPAddress>();

    int numResolverLoops = 20;

    ManualResetEvent shutdownCompleteEvent = new ManualResetEvent(false);

    List<IPAddressResolverLoop> resolverLoops = new List<IPAddressResolverLoop>();

    Console.WriteLine("You can stop the program by
pressing ENTER.");

We’re setting up a new InputQueue<IPAddress> here into which we’ll throw the parsed addresses from our acquisition loop that simulates reading from the log. The queue’s capacity will be limited to just 10 entries (maxItemsInQueue is the input value) and we will run 20 'resolver loops’, which are logical threads that process IP-to-hostname resolution steps.

    Console.WriteLine("You can
stop the program by pressing ENTER.");

    // set up the loop termination callback

    WaitCallback loopTerminationCallback = o =>

        if (Interlocked.Decrement(ref numResolverLoops)
== 0)

            shutdownCompleteEvent.Set();

};

    // set up the resolver loops

    for (int loop =
0; loop < numResolverLoops; loop++)

        // add the resolver loop 'i' and set the done
flag when the

        // last of them terminates

        resolverLoops.Add(

            new IPAddressResolverLoop(

                logDataQueue, loop,

                loopTerminationCallback, null));

Next we’re kicking off the resolver loops – we’ll look at these in detail a bit later. We’ve got a ManualResetEvent lock object that guards the program’s exit until all these loops have completed and we’re going to set that to signaled once the last loop completes – that’s what the loopTerminationCallback anonymous method is for. We’re registering the method with each of the loops and as they complete the method gets called and the last call sets the event. Each loop gets a reference to the logDataQueue from where it gets its work.

   // set up the acquisition
loop; the loop auto-starts

    using (LogDataAcquisitionLoop acquisitionLoop =

        new LogDataAcquisitionLoop(logDataQueue,
maxItemsInQueue))

        // hang main thread waiting for ENTER

        Console.ReadLine();

        Console.WriteLine("*** Shutdown initiated.");

Finally we’re starting the acquisition loop that gets the data from the log file. The loop gets a reference to the logDataQueue where it places the acquired items and it’s passed the maxItemsInQueue quota that governs how many items may be read ahead into the queue. Once the user presses the ENTER key, the acquisition loop object is disposed by ways of exiting the using scope, which stops the loop.

 // shut down the queue; the
resolvers will auto-close

    // as the queue drains. We don't need to close them here.

    logDataQueue.Shutdown();

    // wait for all work to complete

    shutdownCompleteEvent.WaitOne();

Lastly, the queue is shut down (by fittingly calling Shutdown). Shutdown closes the queue (all further enqueue operations are absorbed) and causes all pending readers for which no more entries are available on the queue to unblock immediately and return null. The resolver loops will complete their respective jobs and will terminate whenever they dequeue null from the queue. As they terminate, they call the registered termination callback (loopTerminationCallback from above) and that will eventually cause shutdownCompletedEvent to become signaled as discussed above.

The log-reader simulator isn’t particularly interesting for this sample, even though one of the goodies is that the simulation executes on an I/O completion port instead of a managed thread-pool thread – that’s another blog post. The two methods of interest are Begin/EndGetLogData – all that’s of interest here is that EndGetLogData returns an IPAddress that’s assumed to be parsed out of a log.

class IPAddressLogReaderSimulator

    public IAsyncResult BeginGetLogData(AsyncCallback
callback, object data);

    public IPAddress EndGetLogData(IAsyncResult result);

The simulator is used internally by the LogDataAcquisitionLoop class – which we’ll drill into because it implements the throttling mechanism on the queue.

class LogDataAcquisitionLoop
: IDisposable

    readonly IPAddressLogReaderSimulator ipAddressLogReaderSimulator;

    readonly InputQueue<IPAddress> logDataQueue;

    int maxItemsInQueue;

    int readingSuspended;

    bool shuttingDown;

    public LogDataAcquisitionLoop(InputQueue<IPAddress>
logDataQueue, int maxItemsInQueue)

        this.logDataQueue = logDataQueue;

        this.maxItemsInQueue = maxItemsInQueue;

        this.shuttingDown = false;

        this.ipAddressLogReaderSimulator = new IPAddressLogReaderSimulator();

        this.ipAddressLogReaderSimulator.BeginGetLogData(this.LogDataAcquired, null);

The constructor sets up the shared state of the loop and kicks off the first read operation on the simulator. Once BeginGetLogData has acquired the first IPAddress (which will happy very quickly), the LogDataAcquired callback method will be invoked.

    void LogDataAcquired(IAsyncResult
result)

        IPAddress address = this.ipAddressLogReaderSimulator.EndGetLogData(result);

        Console.WriteLine("-- added {0}", address);

        this.logDataQueue.EnqueueAndDispatch(address, this.LogDataItemDequeued);

        if (!this.shuttingDown
&& this.logDataQueue.PendingCount < this.maxItemsInQueue)

            this.ipAddressLogReaderSimulator.BeginGetLogData(this.LogDataAcquired, null);

        else

            // the queue will be at the defined capacity,
thus abandon

            // the read loop - it'll be picked up by LogDataItemDequeued

            // as the queue pressure eases

            Interlocked.Exchange(ref this.readingSuspended,
1);

            Console.WriteLine("-- suspended reads");

The callback method gets the IPAddress and puts it into the queue – using the InputQueue<T>.EnqueueAndDispatch(T, Action) method. There are two aspects that are quite special about that method when compared to the regular Queue<T>.Enqueue(T) method. First, it does take a callback as the second argument alongside the item to be enqueued; second, the method name isn’t just Enqueue, it also says Dispatch.

When EnqueueAndDispatch() is called, the item and the callback get put into an internal item queue – that’s the ‘enqueue’ part. As we will see in context a bit later in this post, the ‘dequeue’ operation on the queue is the BeginDequeue/EndDequeue asynchronous method call pair. There can be any number of concurrent BeginDequeue requests pending on the queue. ‘Pending’ means that the calls – rather their async callbacks and async state – are registered in another queue internal to InputQueue<T> that preserves the call order. Thus, BeginDequeue always only puts the async callback and async state into that queue and returns afterwards. There is no thread spun or hung. That’s all it does.

As things go, the best opportunity to service a pending dequeue operation on a queue is when an item is being enqueued. Consequently, EnqueueAndDispatch() will first put the item into the internal queue and will then look whether there are registered waiters and/or readers – waiters are registered by ‘(Begin-)WaitForItem’, readers are registered by ‘(Begin-)Dequeue’. Since it’s known that there a new item in the queue now, the operation will iterate overall waiters and complete them – and does so by invoking their async callbacks, effectively lending the enqueue operation’s thread to the waiters. If there’s at least one pending reader, it’ll then pop a message from the head of the internal item queue and call the reader’s async callback, lending the enqueue operation’s thread to processing of the dequeue operation. If that just made your head spin – yes, the item may have been dequeued and processed as EnqueueAndDispatch returns.

There is an overload for EnqueueAndDispatch() that takes an extra boolean parameter that lets you cause the dispatch operation to happen on a different thread, and there is also a EnqueueWithoutDispatch() method that just won’t dispatch through and a standalone Dispatch() method.

The callback supplied to EnqueueAndDispatch(), here the LogDataItemDequeued method, is am Action delegate. The queue will call this callback as the item is being dequeued and, more precisely, when the item has been removed from the internal item queue, but just before it is returned to the caller. That turns out to be quite handy. If you take another look at the LogDataAcquired method you’ll notice that we’ve got two alternate code paths after EnqueueAndDispatch(). The first branch is called when the queue has not reached capacity and it’s not shutting down. When that’s so, we’re scheduling getting the next log item – otherwise we don’t. Instead, we set the readingSuspended flag and quit – effectively terminating and abandoning the loop. So how does that get restarted when the queue is no longer at capacity? The LogDataItemDequeued callback!

    void LogDataItemDequeued()

        // called whenever an item is dequeued. First
we check

        // whether the queue is no longer full after this

        // operation and the we check whether we need
to resume

        // the read loop.

        if (!this.shuttingDown
&&

            this.logDataQueue.PendingCount < this.maxItemsInQueue
&&

            Interlocked.CompareExchange(ref this.readingSuspended,
0, 1) == 1)

            Console.WriteLine("-- resuming reads");

            this.ipAddressLogReaderSimulator.BeginGetLogData(this.LogDataAcquired, null);

The callback gets called for each item that gets dequeued. Which means that we’ll get an opportunity to restart the loop when it’s been stalled because the queue reached capacity. So we’re checking here whether the queue isn’t shuttong down and whether it’s below capacity and if that’s so and the readingSuspended flag is set, we’re restarting the read loop. And that’s how the throttle works.

So now we’ve got the data from the log in the queue and we’re throttling nicely so that we don’t pull too much data into memory. How about taking a look at the DNS resolver loops that process the data?

class IPAddressResolverLoop
: IDisposable

    readonly InputQueue<IPAddress> logDataQueue;

    readonly int loop;

    readonly WaitCallback loopCompleted;

    readonly object state;

    bool shutdown;

    public IPAddressResolverLoop(InputQueue<IPAddress>
logDataQueue, int loop, WaitCallback loopCompleted, object state)

        this.logDataQueue = logDataQueue;

        this.loop = loop;

        this.loopCompleted = loopCompleted;

        this.state = state;

        this.logDataQueue.BeginDequeue(TimeSpan.MaxValue, this.IPAddressDequeued, null);

This loop is also implemented as a class and the fields hold shared that that’s initialized in the constructor. This loop also auto-starts and does so by calling BeginDequeue on the input queue. As stated above, BeginDequeue commonly just parks the callback and returns.

    void IPAddressDequeued(IAsyncResult
ar)

        IPAddress address = this.logDataQueue.EndDequeue(ar);

        if (!this.shutdown &&
address != null)

            Console.WriteLine("-- took {0}", address);

            Dns.BeginGetHostEntry(address, this.IPAddressResolved, new object[]
{ Stopwatch.StartNew(), address });

        else

            this.loopCompleted(this.state);

As an IPAddress is becomes available on the queue, the callback is being invoked and that’s quite likely on a thread lent by EnqueueAndDispatch() and therefore sitting on the thread the log file generator is using to call back for completion of the BeginGetLogData method if you trace things back. If we get an address and the value isn’t null, we’ll then proceed to schedule the DNS lookup via Dns.BeginGetHostEntry. Otherwise we’ll terminate the loop and call the loopCompleted callback. In Main() that’s the anonymous method that counts down the loop counter and signals the event when it falls to zero.

    void IPAddressResolved(IAsyncResult
ar)

        var args = ((object[])ar.AsyncState);

        var stopwatch = (Stopwatch)args[0];

        var address = (IPAddress)args[1];

        stopwatch.Stop();

        double msecs = stopwatch.ElapsedMilliseconds;

try

            IPHostEntry entry = Dns.EndGetHostEntry(ar);

            Console.WriteLine("{0}: {1} {2}ms", this.loop,
entry.HostName, msecs);

        catch (SocketException)

            // couldn't resolve. print the literal
address

            Console.WriteLine("{0}: {1} {2}ms", this.loop,
address, msecs);

        // done with this entry, get the next

        this.logDataQueue.BeginDequeue(TimeSpan.MaxValue, this.IPAddressDequeued, null);

The IPAddressResolved method just deals with the mechanics of printing out the result of the lookup and then schedules another BeginDequeue call to start the next iteration.

Summary: The enabler for and the core piece of the implementation of this scenario is InputQueue<T> – the dequeue-callback enables implementing throttling effectively and the dispatch logic provides an efficient way to leverage threads in applications that leverage asynchronous programming patterns, especially in I/O driven situations as illustrated here.

And last but not least – here’s teh codez; project file is for VS2010, throw the files into a new console app for VS2008 and mark the project to allow unsafe code (for the I/O completion thread pool code).

UsingInputQueue.zip (13.85 KB)

or if you'd rather have a version of InputQueue that is using the regular thread pool, download the WCF samples and look for InputQueue.cs.

[The sample code posted here is subject to the Windows SDK sample code license]

Code-Name WinFX vs .NET Framework 3.0

Sun, 18 Jun 2006 12:39:48 GMT

I was sad when "Indigo" and "Avalon" went away. It'd be great if we'd have a pool of cool legal-approved code-names for which we own the trademark rights and which we could stick to. Think Delphi or Safari. "Indigo" was cool insofar as it was very handy to refer to the technology set, but was removed far enough from the specifics that it doesn't create a sharply defined, product-like island within the larger managed-code landscape or has legacy connotations like "ADO.NET". Also, my talks these days could be 10 minutes shorter if I could refer to Indigo instead of "Windows Communications Foundation". Likewise, my job title wouldn't have to have a line wrap on the business card of I ever spelled it out in full.

However, when I learned about the WinFX name going away (several weeks before the public announcement) and the new "Vista Wave" technologies (WPF/WF/WCF/WCS) being rolled up under the .NET Framework brand, I was quite happy. Ever since it became clear in 2004 that the grand plan to put a complete, covers-all-and-everything managed API on top (and on quite a bit of the bottom) of everything Windows would have to wait until siginificantly after Vista and that therefore the Win16>Win32>WinFX continuity would not tell the true story, that name made only limited sense to stick to. The .NET Framework is the #1 choice for business applications and a well established brand. People refer to themselves as being "dotnet" developers. But even though the .NET Framework covers a lot of ground and "Indigo", "Avalon", "InfoCard", and "Workflow" are overwhelmingly (or exclusively) managed-code based, there are still quite a few things in Windows Vista that still require using P/Invoke or COM/Interop from managed code or unmanaged code outright. That's not a problem. Something has to manage the managed code and there's no urgent need to rewrite entire subsystems to managed code if you only want to add or revise features.

So now all the new stuff is now part of the .NET Framework. That is a good, good, good change. This says what it all is.

Admittedly confusing is the "3.0" bit. What we'll ship is a Framework 3.0 that rides on top of the 2.0 CLR and includes the 2.0 versions of the Base-Class Library, Windows Forms, and ASP.NET. It doesn't include the formerly-announced-as-to-be-part-of-3.0 technologies like VB9 (there you have the version number consistency flying out the window outright), C# 3.0, and LINQ. Personally, I think that it might be a tiny bit less confusing if the Framework had a version-number neutral name such as ".NET Framework 2006" which would allow doing what we do now with less potential for confusion, but only a tiny bit. Certainly not enough to stage a war over "2006" vs. "3.0".

It's a matter of project management reality and also one of platform predictability that the ASP.NET, or Windows Forms teams do not and should not ship a full major-version revision of their bits every year. They shipped Whidbey (2.0) in late 2005 and hence it's healthy for them to have boarded the scheduled-to-arrive-in-2007 boat heading to Orcas. We (the "WinFX" teams) subscribed to the Vista ship docking later this year and we bring great innovation which will be preinstalled on every copy of it. LINQ as well as VB9 and C# incorporating it on a language-level are very obviously Visual Studio bound and hence they are on the Orcas ferry as well. The .NET Framework is a steadily growing development platform that spans technologies from the Developer Division, Connected Systems, Windows Server, Windows Client, SQL Server, and other groups, and my gut feeling is that it will become the norm that it will be extended off-cycle from the Developer Division's Visual Studio and CLR releases. Whenever a big ship docks in the port, may it be Office, SQL, BizTalk, Windows Server, or Windows Client, and as more and more of the still-unmanaged Win32/Win64 surface area gets wrapped, augmented or replaced by managed-code APIs over time and entirely new things are added, there might be bits that fit into and update the Framework.

So one sane way to think about the .NET Framework version number is that it merely labels the overall package and not the individual assemblies and components included within it. Up to 2.0 everything was pretty synchronized, but given the ever-increasing scale of the thing, it's good to think of that being a lucky (even if intended) coindicence of scheduling. This surely isn't the first time that packages were versioned independently of their components. There was and is no reason for the ASP.NET team to gratuitously recompile their existing bits with a new version number just to have the GAC look pretty and to create the illusion that everything is new - and to break Visual Studio compatibility in the process.

Of course, once we cover 100% of the Win32 surface area, we can rename it all into WinFX again ;-) (just kidding)

[All the usual "personal opinion" disclaimers apply to this post]

Update: Removed reference to "Win64".

Managed TCP port-forwarder

Mon, 02 Jan 2006 11:08:46 GMT

I recently needed a TCP port-forwarder that sits on a socket connection and monitors it. My concrete use-case is that I need to front the backend-server of my TV application with such a port forwarder in order to create live-TV streaming sessions as soon as a client requests them and also tears them down shortly after the client disconnects so that the session doesn’t need to time out and blocks the tuner until then. The backend also requires that I do a periodical “keep-alive” ping every 30-40 seconds, which isn’t a very practical requirement for some of my client scenarios. Therefore, I needed, generally speaking, something that would sit between the client and the backend server, monitors the data stream and would let me run some code (set up the live session and start the keep-alive timer) when I get a new client connection and just before I connect through to the target and which would let me run some code (shut down the session and stop the keep-alive) as soon as the connection is torn down.

Since, I didn’t find one (or was too blind or too lazy, you know how that goes), I wrote one. It’s a fully asynchronous TcpListener/TcpClient based implementation, it’s fast and stable enough for my purposes and it might or might not be for yours, it has a bit of tolerance for targets that don’t accept a connection on the first try, and you can hook up events to “before target connect” and “after target disconnect”. Since all the bytes fly by, you can instrument the thing further or monitor the stream as you like.

The code is pretty straightforward, even though the asynchronous calls/callbacks admittedly make the execution paths in the implementation a bit challenging to follow, and should not require much further explanation. You construct an instance of TcpPortForwader passing the local port and the target port and host to forward to, call Start() and the listener starts listening. Stop() stops the listener. You can call Start() from any thread; the listener will implicitly use thread-pool threads to run on its own. Hook up the events and they are being raised. Simple enough. Download below.

Download: TcpPortForwarder.zip

Naming and Continuations (continued)

Wed, 21 Jul 2004 08:24:16 GMT

I was a little off when I compared my problem here to a tail call. Gordon Weakliem corrected me with the term "continuation".

The fact that the post got 28 comments shows that this seems to be an interesting problem and, naming aside, it is indeed a tricky thing to implement in a framework when the programming language you use (C# in my case) doesn't support the construct. What's specifically tricky about the concrete case that I have is that I don't know where I am yielding control to at the time when I make the respective call.

I'll recap. Assume there is the following call

CustomerService cs = new CustomerService();
cs.FindCustomer(customerId);

FindCustomer is a call that will not return any result as a return value. Instead, the invoked service comes back into the caller's program at some completely different place such this:

[WebMethod]
public void FindCustomerReply(Customer[] result)
{
...
}

So what we have here is a "duplex" conversation. The result of an operation initiated by an outbound message (call) is received, some time later, through an inbound message (call), but not on the same thread and not on the same "object". You could say that this is a callback, but that's not precisely what it is, because a "callback" usually happens while the initiating call (as above FindCustomer) has not yet returned back to its scope or at least while the initiating object (or an object passed by some sort of reference) is still alive. Here, instead, processing of the FindCustomer call may take a while and the initiating thread and the initiating object may be long gone when the answer is ready.

Now, the additional issue I have is that at the time when the FindCustomer call is made, it is not known what "FindCustomerReply" message handler it going to be processing the result and it is really not know what's happening next. The decision about what happens next and which handler is chosen is dependent on several factors, including the time that it takes to receive the result. If the FindCustomer is called from a web-page and the service providing FindCustomer drops a result at the caller's doorstep within 2-3 seconds [1], the FindCustomerReply handler can go and hijack the initial call's thread (and HTTP context) and render a page showing the result. If the reply takes longer, the web-page (the caller) may lose its patience [2] and choose to continue by rendering a page that says "We are sending the result to your email account." and the message handler with not throw HTML into an HTTP response on an open socket, but rather render it to an email and send it via SMTP and maybe even alert the user through his/her Instant Messenger when/if the result arrives.

[1] HTTP Request => FindCustomer() =?> "FindCustomerReply" => yield to CustomerList.aspx => HTTP Response
[2] HTTP Request => FindCustomer() =?> Timeout!            => yield to YouWillGetMail.aspx => HTTP Response
                               T+n =?> "FindCustomerReply" => SMTP Mail
                                                           => IM Notification

So, in case [1] I need to correlate the reply with the request and continue processing on the original thread. In case [2], the original thread continues on a "default path" without an available reply and the reply is processed on (possibly two) independent threads and using two different notification channels.

A slightly different angle. Consider a workflow application environment in a bank, where users are assigned tasks and simply fetch the next thing from the to-do list (by clicking a link in an HTML-rendered list). The reply that results from "LookupAndDoNextTask" is a message that contains the job that the user is supposed to do.

[1] HTTP Request => LookupAndDoNextTask() =?> Job: "Call Customer" => yield to CallCustomer.aspx => HTTP Response
[2] HTTP Request => LookupAndDoNextTask() =?> Job: "Review Credit Offer" => yield to ReviewCredit.aspx => HTTP Response
[3] HTTP Request => LookupAndDoNextTask() =?> Job: "Approve Mortgage" => yield to ApproveMortgage.aspx => HTTP Response
[4] HTTP Request => LookupAndDoNextTask() =?> No Job / Timeout => yield to Solitaire.aspx => HTTP Response

In all of these cases, calls to "FindCustomer()" and "LookupAndDoTask()" that are made from the code that deals with the incoming request will (at least in the theoretical model) never return to their caller and the thread will continue to execute in a different context that is "TBD" at the time of the call. By the time the call stack is unwound and the initiating call (like FindCustomer) indeed returns, the request is therefore fully processed and the caller may not perform any further actions.

So the issue at hand is to make that fact clear in the programming model. In ASP.NET, there is a single construct called "Server.Transfer()" for that sort of continuation, but it's very specific to ASP.NET and requires that the caller knows where you want to yield control to. In the case I have here, the caller knows that it is surrendering the thread to some other handler, but it doesn't know to to whom, because this is dynamically determined by the underlying frameworks. All that's visible and should be visible in the code is a "normal" method call.

cs.FindCustomer(customerId) might therefore not be a good name, because it looks "too normal". And of course I don't have the powers to invent a new statement for the C# language like continue(cs.FindCustomer(customerId)) that would result in a continuation that simply doesn't return to the call location. Since I can't do that, there has to be a different way to flag it. Sure, I could put an attribute on the method, but Intellisense wouldn't show that, would it? So it seems the best way is to have a convention of prefixing the method name.

There were a bunch of ideas in the comments for method-name prefixes. Here is a selection:

cs.InitiateFindCustomer(customerId)
cs.YieldFindCustomer(customerId)
cs.YieldToFindCustomer(customerId)
cs.InjectFindCustomer(customerId)
cs.PlaceRequestFindCustomer(customerId)
cs.PostRequestFindCustomer(customerId)

I've got most of the underlying correlation and dispatch infrastructure sitting here, but finding a good programming model for that sort of behavior is quite difficult.

[Of course, this post won't make it on Microsoft Watch, eWeek or The Register]

Scusi? Visual C++ Compiler for free? Woo!

Sun, 16 May 2004 07:46:49 GMT

I didn't spend much time for anything except writing, coding, travel, speaking and being at geek parties in the past weeks. Hence, I am sure I am the last one to notice, but I find it absolutely revolutionary that the Microsoft Visual C++ 2003 command line compiler (Microsoft C/C++ Version 13.1) is now a freebie.

public, protected, private

Wed, 01 Oct 2003 13:43:34 GMT

We just had a short discussion here at the office on the goodness and badness of using Reflection to cheat around private and protected and cases where it does and doesn't work (it's of course a System.Security.Permissions.ReflectionPermission thing). The discussion brought back memories of that old C/C++ hack that I've been using for almost any application back in my Borland C++ and OWL days:

#define private public
#define protected public
#include <owl.h>

dllhost.exe.config dilemma fixed, at last -- by Microsoft.

Fri, 26 Sep 2003 11:37:36 GMT

A long while back, I wrote about a hack to fix the dllhost.exe.config dilemma of Enterprise Services. That hack no longer works due to changes in the Framework, but the good news is there is an “official” and very easy solution for this now. Unfortunately there is no documentation on this (or at least it’s not easy enough to find that I could locate it) and Google only yields a few hints if you know exactly what you are looking for. So, index this, Google!

What I call the “config dilemma” of Enterprise Services is that because all out-of-process ES applications are executed using the surrogate process provided by %SystemRoot%\System32\dllhost.exe and the runtime is loaded into that process, the default application configuration file is dllhost.exe.config, must reside just next to dllhost.exe (in System32) and is therefore shared across all out-of-process Enterprise Services applications.

That makes using the XML configuration infrastructure for Enterprise Services very unattractive, to say the least.

Now, with COM+ 1.5 (Windows Server 2003) and the .NET Framework 1.1, things did change in a big way.

To use per-application application configuration files, all you have to do is to create an (possibly otherwise empty) “application root directory” for your application in which you place two files: An application.manifest file (that exact name) and an application.config file. Once your application is registered (lazy or using the RegistrationHelper class or through regsvcs.exe), you will have to configure the application’s root directory in the catalog – that can be done either programmatically using the catalog admin API (ApplicationDirectory property) or through the Component Services explorer as shown above.

The picture shows that the example that you can download using the link below is installed at “c:\Development\ES\ConfigTest\ESTest” on my machine and has these two said files sitting right there.

The application.manifest file content is embarrassingly simple

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0">
</assembly>

and the application.config isn’t complicated either:

<?xml version="1.0"?>
<configuration>
  <appSettings>
     <add key="configBit" value="This rocks!"/>
  </appSettings>
</configuration>

These two files, placed into the same directory and properly configured as shown in the above picture, let this class

       public class SimpleComponent : ServicedComponent
       {
       public string GetConfigBit()
        {
            return ConfigurationSettings.AppSettings["configBit"];
        }
       }

yield the expected result for GetConfigBit(): “This rocks!”

Download: ESTest.zip

PDC speculations: Avalon

Fri, 26 Sep 2003 05:59:39 GMT

Lots of PDC hype these days. Here's a piece an Avalon by Wesner Moise that's still leaving quite a bit in the fog.

My translation of what I am reading from the abstracts is:

Imagine Microsoft would drop the entire USER32 subsystem of Windows and replace it with a brand-spanking-new windowing and I/O engine and a fully object-oriented, managed API, finally doing the long-overdue overhaul and replacement of the foundation of the Windows UI technologies that have, in essence, been with us since Windows 1.0.

.... and create a WOW ("Windows on Windows") subsystem layer, not dissimilar to what we saw in NT for Win16 apps, to support existing apps.

Composite Metadata (this will change your life)

Thu, 25 Sep 2003 20:19:55 GMT

I am in a blogging mood today … Here are some thoughts around composite metadata. Sorry for the bold title ;)

* * *

Whenever I am asked what I consider the most important innovation of the CLR, I don’t hesitate to respond “extensible metadata” coming in the form of custom attributes. Everyone who has followed this blog for a while and looked at some of the source code I published knows that I am seriously in love with attributes. In fact, very few of the projects I write don’t include at least one class derived from Attribute and once you use the XmlSerializer, Enterprise Services or ASMX, there’s no way around using them.

In my keynote on contracts and metadata at the Norwegian Visual Studio .NET 2003 launch earlier this year, I used the sample that’s attached at the bottom of this article. It illustrates how contracts can be enforced by both, schema validation and validation of object graphs based on the same set of constraints. In schema, the constraints are defined using metadata (restrictions) inside element or type definitions, and in classes, the very same restrictions can be applied using custom attributes, given you have a sufficient set of attributes and the respective validation logic. In both cases, the data is run through a filter that’s driven by the metadata information. If either filter is used at the inbound and outbound channels of a service, contract enforcement is automatic and “contract trust” between services, as defined in my previous article, can be achieved. So far, so good.

In my example, the metadata instrumentation for a CLR type looks like this:

      [System.Xml.Serialization.XmlTypeAttribute(
           Namespace="urn:schemas-newtelligence-com:transactionsamples:customerdata:v1")]
       public class addressType
       {
              [Match(@"\p{L}[\p{L}\p{P}0-9\s]*"),MaxLength(80)]
              public string City;
              public countryNameType Country;
              public countryCodeType CountryCode;
              [MaxLength(10)]
              public string PostalCode;
              [MaxLength(160)]
              public string AddressLine;
      }

… while the corresponding schema is a bit better factored and looks like this:

    <xsd:simpleType name="nameType">
              <xsd:restriction base="xsd:string">
                     <xsd:pattern value="\p{L}[\p{L}\p{P}0-9\s]*" />
              </xsd:restriction>
    </xsd:simpleType>
    <xsd:complexType name="addressType">
              <xsd:sequence>
                     <xsd:element name="City">
                            <xsd:simpleType>
                                   <xsd:restriction base="nameType">
                                          <xsd:maxLength value="80" />
                                   </xsd:restriction>
                            </xsd:simpleType>
                     </xsd:element>
                     <xsd:element name="Country" type="countryNameType" />
                     <xsd:element name="CountryCode" type="countryCodeType" />
                     <xsd:element name="PostalCode">
                            <xsd:simpleType>
                                   <xsd:restriction base="xsd:string">
                                          <xsd:maxLength value="10" />
                                   </xsd:restriction>
                            </xsd:simpleType>
                     </xsd:element>
                     <xsd:element name="AddressLine">
                            <xsd:simpleType>
                                   <xsd:restriction base="xsd:string">
                                          <xsd:maxLength value="160" />
                                   </xsd:restriction>
                            </xsd:simpleType>
                     </xsd:element>
              </xsd:sequence>
       </xsd:complexType>

The restrictions are expressed differently, but they are aspects of type in both cases and semantically identical. And both cases work and even the regular expressions are identical. All the sexiness of this example aside, there’s one thing that bugs me:

In XSD, I can create a new simple type by extending a base type with additional metadata like this

<xsd:simpleType name="nameType">
       <xsd:restriction base="xsd:string">
              <xsd:pattern value="\p{L}[\p{L}\p{P}0-9\s]*" />
       </xsd:restriction>
</xsd:simpleType>

which causes the metadata to be inherited by the subsequent element definition that again uses metadata to further augment the type definition with metadata rules:

<xsd:element name="City">
       <xsd:simpleType>
              <xsd:restriction base="nameType">
                     <xsd:maxLength value="80" />
              </xsd:restriction>
       </xsd:simpleType>
</xsd:element>

So, XSD knows how to do metadata inheritance on simple types. The basic storage type (xsd:string) isn’t changed by this augmentation, it’s just the validation rules that change, expressed by adding metadata to the type. The problem is that the CLR model isn’t directly compatible with this. You can’t derive from any of the simple types and therefore you can’t project this schema directly onto a CLR type definition. Therefore I will have to apply the metadata onto every field/property, which is the equivalent of the XSD’s element declaration. The luxury of the <xsd:simpleType/> definition and inheritable metadata doesn’t exist. Or does it?

Well, using the following pattern it indeed can. Almost.

Let’s forget for a little moment that the nameType simple type definition above is a restriction of xsd:string, but let’s focus on what it really does for us. It encapsulates metadata. When we inherit that into the City element, an additional metadata item is added, resulting in a metadata composite of two rules – applied to the base type xsd:string.

So the about equivalent of this expressed in CLR terms could look like this:

    [AttributeUsage(AttributeTargets.Field)]
    [Match(@"\p{L}[\p{L}\p{P}0-9\s]+")]
    public class NameTypeStringAttribute : Attribute
    {
    }

    [System.Xml.Serialization.XmlTypeAttribute(
       Namespace="urn:schemas-newtelligence-com:transactionsamples:customerdata:v1")]
    public class addressType
    {
        [NameTypeString,MaxLength(80)]
        public string City;

        …
    }

Now we have an attribute NameTypeString(Attribute) that fulfills the same metadata containment function. The attribute has an attribute. In fact, we could even go further with this and introduce a dedicated “CityString” meta-type either by composition:

   [AttributeUsage(AttributeTargets.Field)]
   [NameTypeString,MaxLength(80)]
      public class CityStringAttribute : Attribute
    {

}

… or by inheritance

   [AttributeUsage(AttributeTargets.Field)]
   [MaxLength(80)]
      public class CityStringAttribute : NameTypeStringAttribute
    {
    }

Resulting in the simple field declaration

[CityString] public string City;

The declaration essentially tells us “stored as a string, following the contract rules as defined in the composite metadata of [CityString]”.

Having that, there is one thing that’s still missing. How does the infrastructure tell if an attribute is indeed a composite and that the applicable set of metadata is a combination of all attributes found on this attribute and attributes that are declared on itself?

The answer is the following innocent looking marker interface:

public interface ICompositeAttribute
{ }

If that marker interface is found on an attribute, the attribute is considered a composite attribute and the infrastructure must (potentially recursively) consider attributes defined on this attribute in the same way as attributes that exist on the originally inspected element – for instance, a field.

    [AttributeUsage(AttributeTargets.Field)]
    [Match(@"\p{L}[\p{L}\p{P}0-9\s]+")]
    public class NameTypeStringAttribute : Attribute, ICompositeAttribute
    {   }

Why a marker interface and not just another attribute on the attribute? The answer is quite simple: Convenience. Using the marker interface, you can find composites simply with the following expression: *.GetCustomAttributes(typeof(ICompositeAttribute),true)

And why not use a base-class “CompositeAttribute”? Because that would be an unnecessary restriction for the composition of attributes. If only the marker interface is used, the composite can have any base attribute class, including those built into the system.

But wait, this is just one side of the composition story for attributes. There’s already a hint on an additional composition quality two short paragraphs up: *.GetCustomAttributes(typeof(ICompositeAttribute),true). The metadata search algorithm doesn’t only look for concrete attribute types, but also looks for interfaces, allowing the above expression to work.

So how would it be if an infrastructure like Enterprise Services would not use concrete attributes, but would also support composable attributes as illustrated here …

    public interface ITransactionAttribute
    {
        public TransactionOption TransactionOption
        {
            get;
        }
    }

    public interface IObjectPoolingAttribute
    {
        public int MinPoolSize
        {
            get;
        }

        public int MaxPoolSize
        {
            get;
        }
    }

In that case, you would also be able to define composite attributes that define standardized behavior for a certain class of ServicedComponents that you have in your application and should all behave in a similar way, resulting in a declaration like this:

      public class StandardTransactionalPooledAttribute :
        Attribute, ITransactionAttribute, IObjectPoolingAttribute
    {
    }

      [StandardTransactionalPooled]
    public class MyComponent : ServiceComponent
    {

}

While it seems to be an “either/or” thing at first, both illustrated composition patterns, the one using ICompositeAttribute and the other that’s entirely based on the inherent composition qualities of interface are useful. If you want to reuse a set of pre-built attributes like the ones that I am using to implement the constraints, the marker interface solution is very cheap, because the coding effort is minimal. If you are writing a larger infrastructure and want to allow your users more control over what attributes do and allow them to provide their own implementation, “interface-based attributes” may be a better choice.

Download: MetadataTester.zip

XmlSerializer & UTC once more

Wed, 20 Aug 2003 13:43:30 GMT

While I wasn't looking, Doug Purdy, who is the PM for the team in charge of the XmlSerializer has posted this little cryptic answer to my initial question:

String serializedDateTime = XmlConvert.ToString( someDateTime );
DateTime deserializedDateTime = XmlConvert.ToDateTime( serializedDateTime );

What Doug is showing is really what the serializer is doing with dates under the hood and that I am incorrectly blaming the XmlSerializer for the lack of UTC support. [XmlConvert.ToString() also does nothing more than calling DateTime.ToString() with the appropriate format string]. As I was already saying in yesterday's post, but what I want to make a bit clearer here again is that the actual problem is the lack of time-zone awareness in DateTime.

So the proper thing more me to do is to ask the base-class library team for time-zone support in the base-class library for Whidbey so that Doug can fix this for us ;)

XmlSerializer Quiz: The solution

Tue, 19 Aug 2003 18:41:11 GMT

I posted a little question here, just asking "Why?". For your reading convenience, I pull the code in here once more:

DateTime issued;

[XmlIgnore]
public DateTime IssuedUtc
{ get { return issued; } set { issued = value; } }

[XmlElement("issued")]
public DateTime IssuedLocalTime
{ get { return Issued.ToLocalTime(); } set { Issued = value.ToUniversalTime(); } }

So, indeed, why am I doing this? Well, when I decided to normalize all times handled by the backend engine of dasBlog into the UTC timezone, I really didn't think of the XML Serializer being a problem at first. It turned out to be one.

The reason why I wanted all times to be handled internally as UTC is quite simple: Too many time zones to deal with and I need to have a proper reference to do forward and backward time calculations. dasBlog deals with four time-zones:

"Reader Time": The "<%userWhen%> macro emits a block of script that will cause the browser to emit the time of a post local to the reader's time zone. That one is easy, because the calculation happens on the client, but I need to feed it UTC (GMT).
"Display Time": This is the time zone the blog owner selects for his/her blog. All times displayed on the weblog pages are shown in that time zone (complete with the TZ name and the GMT offset). This also applies to the "admin pages" such as referrals and events (which both roll over in synch with the display time zone). Display time is calculated dynamically and you will notice that it also automatically adjusts for daylight savings time. The display time zone is also by no means fixed. If the blog author travels (and you will see this on my blog starting next week), he/she can adjust the blog to his/her present time zone. When I am going to be at TechEd Malaysia, my blog will show UTC+0800. To make this time-zone shifting work, the absolute time must be stored in UTC.
"Engine Time": This is UTC. All of the dasBlog runtime handles everything in UTC.
"Server Time": Now were getting to the point. The engine runs on a server that has it's own local time zone setting: "server local time". That's something that the user who's running his engine at some ISP can't control and that's the one of all the time zones that really nobody is ever interested in. You shouldn't care whether your blog is hosted in Germany, the U.S. East Coast or Singapore. That's even more of an issue because one could expect that hosted sites may get moved around between ISP locations. The only little thing we're interested in is that the server knows its offset to UTC.

So ... I was thinking.... ask for DateTime.Now.ToUniversalTime(), handle everything in UTC from there on and everything's good. (Btw, I know about DateTime.UtcNow, but I like the expressiveness of this better).

What I wasn't considering is the way the XmlSerializer works, which I am using both for storage and for the various web services (including the Atom feed). What I also found is that the DateTime class in the framework isn't time-zone aware.

The workaround above is used because the XML Serialization infrastructure always assumes local time ("Server time") for serialization and will always emit ISO 8601 dates with a time-zone suffix like this: 2003-08-19T15:15:58.0781250+02:00. So when you are handling UTC times internally and use them blindly with XML serialization for both storage and web services, the serializer will assume you are using local time and throw you off by your own time-zone difference to UTC.

This isn't "too bad" when you store stuff in local XML files, because when you write something wrong from the same place you read it back into and your time-zones don't change you are in ok in memory, but you are nevertheless wrong on disk. What happened to me in dasBlog version 1.1 was that I was thinking that I stored UTC, but in fact I stored everything in local time. My UTC time 2003-08-19T15:15:58 always turned into 2003-08-19T15:15:58+02:00, because the DateTime class doesn't keep time-zone information around that the serializer could use. Therefore the serializer must always assume local time and that causes the offset to be emitted. That's of course much worse for UTC+12.

The fix:

The field DateTime issued; holds the "engine time", which is always UTC.

The property

[XmlIgnore]
public DateTime IssuedUtc
{ get { return issued; } set { issued = value; } }

wraps this value and is the property that the engine works with internally. The XmlSerializer is instructed to ignore this value in the serialization process by declaration of the [XmlIgnore] attribute. Instead, we tell the serializer to look at the following property, which is not used by the engine itself and also indicates that by its name suffix "LocalTime", which essentially declares it as "off limits" for direct access to everyone knowing the project:

[XmlElement("issued")]
public DateTime IssuedLocalTime
{ get { return Issued.ToLocalTime(); } set { Issued = value.ToUniversalTime(); } }

This property is the one that the serializer grabs and it does the proper conversion to and from server local time that the serializer requires. The actual dasBlog code is using a variant of this property that is a bit larger in code size because it also checks for DateTime.MinValue and DateTime.MaxValue occurrences, which, depending on the server time zone, would cause the time zone shifting to fail with an overflow/underflow exception (and no, I am not checking near MinValue and MaxValue):

[XmlElement("Date")]
public DateTime DateLocalTime
{
    get
    {
       return (DateUtc==DateTime.MinValue||DateUtc==DateTime.MaxValue)?
               DateUtc:DateUtc.ToLocalTime();
    }
    set
    {
        DateUtc = (value==DateTime.MinValue||value==DateTime.MaxValue)?
                  value:value.Date.ToUniversalTime();
    }
}>

So, that's why.

Be aware that all of this applies to ASP.NET Web Services, too, and if you are dealing with multiple time zones are you are using UTC normalized times in your app, you will have to deal with this. If "server time" makes you happy, you won't need to worry.

A little XmlSerializer Quiz.

Thu, 14 Aug 2003 17:51:49 GMT

DateTime issued;

Why?

Regular expressions for SQL Server, 2nd try

Mon, 11 Aug 2003 03:55:15 GMT

Dan Farino, who wrote the CLR based, Regex extended stored procedure on which I put a warning sign yesterday, wrote me an email back (I notified him that I wrote that blog entry) and told me that he just uploaded an unmanaged version. Haven't downloaded it, yet, but it seems to be functionally equivalent. If it's stable and quick, I can think of 2 bazillon uses for this -- including, of course, Regex based XML parsing inside SQL Server (while we wait for Yukon).

Next steps and why some cool things are just cool at first sight, sigh!

Sun, 10 Aug 2003 17:07:46 GMT

I guess my next little toy project will be around aggregation. Not that I won't continue adding stuff to "dasBlog", but I am quite happy with the status as of now and once we've got Stephen's caching work in the project, we'll make it a 1.2 .

So, the other bigger chunk of code that I have sitting around here from the blogging infrastructure that I wrote but didn't get done in the first quarter of this year (and from which the rendering engine made it into dasBlog) is a pretty serious, extensible syndication engine based on Enterprise Services and SQL Server. Pulling in RSS feeds is, in my view, a background server task. A common infrastructure for subscribing, queuing, requesting, transforming, filtering, caching and pushing out results (via LCE) is something that could be very handy, makes a great demo and there's lots of sample data to gather in a short time ;)

I am not at all interested in writing yet another aggregation GUI. I may grab one of the available aggregators like SharpReader or RSS Bandit and hook it up for demo purposes, but that's about it.

So ... while looking around for tools I could use, I found this cool extended stored procedure for using .NET Framework regular expressions straight from within SQL Server. Hold! What I also found (and I actually looked for it) was this Microsoft KB article (Q322884) that states that extended stored procs written in managed code "is not supported".

Based on some discussions I had with members of the SQL team about the next release of SQL Server (Yukon) earlier this year about their work around incorporating the CLR (the fact that they're doing that is public so I am not violating any of the papers I signed), this absolutely doesn't come as a surprise to me and the bottom line is: This stored procedure looks cool, but shouldn't get near a SQL Server 2000 production server, ever.

The reason for that is that SQL Server is essentially an operating system by itself. It does its own memory management, can do its own thread scheduling (it makes use of fibers, which are self-scheduled threads) and has, of course, its own I/O system sitting on low-level OS functions. The CLR doesn't know about any of that and SQL Server 2000 doesn't know about the CLR. If you are writing a stored procedure in unmanaged C/C++, you create just a DLL that runs on top of Win32 and if you follow a couple of rules, you should be ok. If you are writing in C# or VB.NET, you are bringing along an environment that doesn't know anything about these rules around SQL's use of threading and thread local storage, probably wants to launch additional threads under the covers and whose garbage collector hasn't been designed to cooperate with SQL's memory management. Yes, it may run, but what happens under stress or low memory conditions? How does it affect transactions? What's the story around performance once the CLR is present in the process and is probably used in only one of 500 queries? I know that putting the CLR into Yukon is nothing that a someone at Microsoft does in a day or two and therefore I doubt that putting the CLR into SQL 2000 within half a day is a good idea for a real system. Things are different if you are running inside what essentially is an OS kernel.

Why 3219?

Thu, 07 Aug 2003 20:19:51 GMT

No, I haven’t built the stuff 3219 times. It’s “3” for 2003 and “219” is today’s day of the year. Here’s the compiled one-liner that sits in my system32 directory and tells me …

Download: buildday.zip

Ingo says that "sealed" is good. Is it?

Thu, 24 Jul 2003 14:49:31 GMT

Ingo says that "sealed" classes are a good thing and goes great lengths to explain why he thinks so in this article.

Now, I do have a little problem with his conclusion (which I still partially share, because I do write sealed classes every once in a while, but for a slightly different reason, see below) because the example he's using isn't really fitting the problem and he actually makes some assumptions about the ImageList class that aren't accurate in this context.

First off, the property ImageList.Images isn't virtual and therefore cannot be overridden, at all. So that by itself is no reason to make the class sealed. The property has been introduced by the class, the class designer chose not to make it virtual. Can't be overriden, done. The same is true for all other properties except Container and Site, which are inherited from System.ComponentModel.Component. I fail to see anything on the ImageList class (not even a method) that really justifies the lockup using sealed considering the current version of the Framework. The bad that sealed does here is that I can't create a wrapper around the ImageList that simplifies setting up such a list in my specific environment.

Using sealed on a class is a very brutal thing to do. A more gentle way of using the sealed keyword is to say [void sealed override myMethod() { }], which essentially says: "Within my own class space I am using inheritance for this method, but since outsiders don't know what I am doing I won't let them override the behavior from here downwards in the class hierarchy." That doesn't affect the inheritability of the entire class and hence doesn't adversely affect the ability to wrap the existing functionality.

Now ... is sealed on a class just plain evil? There are two answers:

Yes. The keyword sealed on a class usually doesn't have much or any of a technical justification in the "here and now" version of a class library.
No. The keyword sealed conveys a very clear message that the class designer reserves the right to change the class so dramatically in a future release of the product that you would be tremendously unhappy if you had derived from it to implement your own functionality, once that new version comes along. Sealed says: This is going to change in a big way.

Quake II on .NET

Fri, 11 Jul 2003 09:07:09 GMT

Quake II on .NET. Scott Stanfield and his team at Vertigo Software show yet another nice trick: Quake II in managed code.