Crazy benchmarks

There are a lot of JVM vs. CLR "benchmarks" popping up all over the place, which are used as arguments for or against the respective environments. I am not talking about the "Petshop wars" here; I have rather looked at a few of the "little ones" that are used as ammunition in advocacy forums. The problem: Almost all of these "benchmarks" really only show how hard it is to write good and fair performance tests that give others some actual guidance.

I picked out two: On Javalobby.com, Dan Benanav shows how Swing outperforms Windows Forms by comparing the performance of the grid controls of Swing and Windows Forms. The Swing version wins easily in his version -- further down in the discussion thread someone rewrote the scenario and Windows Forms blows Swing away. The issue is that this test is so singularly focused on a specific feature that it doesn't tell anybody anything good about either environment. The same is true for Daniel Mettlers Mono benchmark. The "benchmark" tries to illustrate performance using 100000 iterations through a loop printing "Hello World" to the console. Again, this test may say a lot about console output performance but nothing about anything else.

In a time, where any desktop machine that you can buy at a supermarket has more firepower than 98% of all desktop applications will ever need in terms of GUI performance, client-side benchmarks are really "out". Even a Java application without any JIT will work more than satisfactory for most users that have >1GHz under their fingertips. Good server-side benchmarks are a different issue and still needed. But those require a lot of time, architectural thought, knowledge of proper algorithms and, more importantly, the money to buy/lease/borrow the required hardware.

The biggest "performance problem" is typically not the underlying platform, it's thoughtless programming where (proper choices of) algorithms don't play any role, where the choice between synchronous and asynchronous operations are not considered, where time is burned up in parsing and compiling query-plans for ad-hoc SQL queries instead of using stored procedures, where complex service constructs like EJB or COM+ are mindlessly used for "code consistency" reasons and not for the services they provide and so on and so on.

Only if all these things are considered, comparing platforms for raw speed may begin to make sense. However, the result is typically: "Your mileage may vary" and "It still depends on what you do".  The great side-effect of benchmark wars like those happening around the .NET/J2EE Petshop battle is that there are a lot of architectural best practices evolving in the process: These are the interesting bits, the raw numbers are of much lesser value.

All numbers are worthless if you don't look at how they were achieved and learn from the differences in choice of platform, architecture and implementation technique.

However, it'd be much more important to look at a different kind of speed: How long does it take the average developer to complete a certain task, achieveing set quality goals? How long does it take an average developer to understand code he or she needs to maintain? I don't see such benchmarks all that often.

Pioneer 10 lives.

WS-Policy is public. This spec is significant as it is a step towards separating message content declaration (aka "what we have now in WSDL") from the services and QoS declaration. There are also significant new specs around WS-Security. WS-Trust and WS-SecureConversation are specifically interesting for things as those that I have been doing with my Kerberos integration with WS-Security, because it provides a framework for establishing trust relationships and secure, "connection-like" bidirectional conversations (and not only one-shot messages).

Microsoft Watch: "One source close to the company said that Microsoft has held internal discussions to kick around ideas for XML-specific language, referred to internally as 'X#'." (via Managed Space) [Brian Jepson's Radio Weblog]

(a) Dig out your PDC2001 CD.
(b) Look for the "XLANG/s" sample
(c) Look at the assemblies with ILDASM

I think it's funny that this blog really made it onto the top 10 Google hits for "Alien Abductions". In all fairness I should probably rename it into "Enterprise Development & Other Stuff", but I like it better this way ;)

I am back online after this. Luckily, the nerve responsible for making this "ouch" (I am no doctor...) wasn't really happy about all this and went on strike just long enough to make this a no-pain-whatsoever experience. However, the doctor ordered that I stay at home and don't work and so I really didn't. Instead I finally got through the crazy final sequence of Halo (I am not even close to be as good at this as these guys), worked on my Rallisport driving skills, ruined some expensive cars and otherwise just took a break. Meanwhile, my notebook got a new LCD, keyboard and modem. It also doesn't complain about post-surgery pain in any way.

One more week to go. Then the whole christmas craziness happens and after that Patricia and myself fly over to New York City for New Year's.

Why you want to use Enterprise Services for your .NET application
Part 1: Introduction
Part 2: Basic Architectural Considerations and the benefits of Processes and Process Models 

Part 3: Managing limited and expensive resources
Mr. Miller and the Data-Center

(Parental Advisory: May contain ironic sequences, dramatizations, mild simplifications for educational purposes). Meet Mr. Miller. Mr. Miller's job is to run a "pilot" .NET web-portal project that lets insurance clients and agents submit claims over the web, buy products and do some limited self-management and analysis of their current contracts portfolio. As with many major insurers, most of the actual data lives in an mainframe system and pretty much all business logic is implemented there, because ... it is that way (for many good current and historical reasons). So once he's getting to the mainframe integration part of his planning efforts, he picks up the phone and calls Mr. Petterson, who's managing the data center ....

 

LH3003 Oslo-Frankfurt, Departure 1835 scheduled, 1845 actual, arrival ca. 2000-2010. Connection flight to Düsseldorf leaves Frankfurt at 2200, boarding at 2130. Traveling. The more traveling I do, the harder it is to tell whether I like it or hate it. It's probably both. I have definitely understood why five-star hotels exist. If you spend much time traveling, you also spend a lot of time in hotels -- they really become your second home. And "home" is supposed to be comfortable, isn't it? The same applies to things like flying business class or sometimes having a very good (read: expensive) meal at a restaurant. Much of these "luxury" things are for the "upper class" in the eyes of most people who don't travel very frequently. I don't consider myself being part of that "upper class", really. Still, I spend a lot of time traveling and therefore I don't like to be squeezed into an overcrowded "cattle-class" segment on a long-haul flight for the same reasons as many people rather choose the luxury of their own car to go to work instead of an overcrowded city bus. Also, I don't mind going to McDonald's or some other fast food place, but not every day! For the record: I still fly economy on this flight -- it's just 90 minutes. ;)

Second and last day of this stop of the Microsoft EMEA Architect's Tour in Oslo, Norway. Today I am going to talk about Aspect-Oriented Programming, which is fun.

From tomorrow the blog will likely be silent for about a week, because both I and my notebook will have to undergo surgery and we'll both be out-of-service of about a week. I need to get some pretty ugly dental surgery done and the notebook needs to get some pretty ugly electronic surgery done. Patricia and my partners decided that this would be a good time to make sure that I definitely won't work when I shouldn't. I guess I will take it easy for a change and possibly work on my XBox skills if I can. ;)

Lunchtime! IBM buys Rational.

I updated my copyright notice at the bottom of the blog. The reason for that is that I found my blog to be replicated in full to another website in their design and with their copyright notice, without any backlinks to this blog and without having been asked for permission. 

I may offend a few friends over this, but there are friendly rules of conduct which I follow and on which I must insist: Quote liberally, aggregate, combine, analyze, but do not republish large portions of content in unaltered form and don't quote without linking back to the source. Brief legal excursion: There are explicit rules about what's permitted in terms of quoting other's works in just about every copyright law. The German Copyright Act and related international treaties are applicable here. For instance, it's no problem to take this entire posting and quote it elsewhere as-is. It is a problem to simply publicly republish the whole rss-feed content elsewhere. Quoting and commenting is a good thing, plain theft of intellectual property isn't.

Thoughts from integrating with WSDK. 

It shouldn't be very surprising that I am currently integrating my web services extensions with the WSDK with the goal of making that available shortly after WSDK ships. However, it's no as easy as I thought it would be, because the extensibility pipeline model that MS layered on top of the SoapExtensions requires a different approach for quite a few things that I've done. I could just leave my things the way they are and have them function side-by-side with the WSDK's pipeline (that works), but I want to take advantage of the pipeline's relative independence of the ASP.NET request/response cage, which I find increasingly limiting. Also, integrating Kerberos into the WSDK's WS-Security framework is a real challenge, because I may have to go all the way down into writing an algorithm wrapper based on the stuff in the System.Security.Cryptography namespace. However -- I really want to get this working and therefore I will ;)

A new terminology set.

While working on this, I have found that a lot of terminology that I am using here is massively confusing: "client", "server", "input", "output", "request", "response", "proxy", "stub", "endpoint", "router" and quite a few more terms are ambiguous and depend on your current point of view and on what you are just building.

I find it more helpful to focus on the message and look at it as something that starts empty, grows and shrinks (and grows and shrinks) as it goes through a unidirectional pipeline of message-part producers and message-part-consumers and whose processing ends when the message is empty again. Request/response is a special case where the message flow originates and ends at the same place. The pipeline elements (producers/consumers) are connected by local ("through the stack") and remote ("over the wire")  connectors. A server-side "request handler" as it's currently known is simply a message-part consumer taking stuff from the body and sticking stuff back into the body.

Here's the terminology set I am currently playing around with:

• Message Producer: Acts on information from the system environment or state and possibly adds information to a message.
• Message Consumer: Acts on and removes information from the message and possibly alters the system environment or state.
• Service Acquisition Filter: A message producer that acquires system state to flow information about a service (eg. picks up a COM+ transaction context or the current security principal)
• Service Enlistment Filter: A message consumer that reads and service information from a message and enlists into a service or creates a new service (eg. picks up a TIP URI and creates a service domain or creates a security context from a serialized authentication package)
• Fork: In case message processing cannot proceed without additional information or in case a filter wants to split the message into separate flows, any pipeline can fork off new pipeline instances to request information or flow information to additional destinations. A fork is blocking if it can't proceed in its current state and otherwise non-blocking.

A proxy is a message producer. A stub is a message consumer. A logging sink is a message consumer. A filter that adds timestamp information is a message producer. A filter that reads timestamp information is a message consumer. A client-side authentication extension (sending path) is a service acquisition filter. A server-side authentication extension (receiving path) is a service enlistment filter. The inverse of a "service enlistment filter" is just another "service acquisition filter", which acquires the service state and possibly disassociates the action from the service -- reversely, the same relationship exists.

Anyone still with me?

[Disregard the first two comments showing up below. The comments feature of radio is confused with my blog]

Three related links.

The next installment of my little blog-series "Why you want to use Enterprise Services for your .NET application" has the clumsy working title "Object services and management of expensive and limited resources". I just need to make the time to finish it; hopefully that works today or tomorrow. The first two parts are also in my stories list.

Once System.Diagnostics.StackWalk and "__asm mov [__EBP], ebp;" become useful tools, you realize that you've exhausted an infrastructure's extensibility hooks. Sigh!

COM was love

This isn't an Enterprise Services crusade. There's a lot of very useful stuff and I am pointing it out. That's all ;)

However, I think I should admit that I did indeed have a very long affair with the entire ES/COM+/MTS/COM family. {Is that a nasty thing to admit?}. The love went away, now we're just good friends.  

October 1992. The first time COM got into customer's hands. I got this CD and I was like "man, that's nuts! Way too complicated!". The first shipment came with docs that dealt in very deep and great detail with IUnknown, structured storage and especially IMoniker. I didn't understand the moniker stuff until two years later. What I understood was AddRef()/Release() and QueryInterface(). An implementation of IUnknown became my default-base class for anything in November 1992. Only about 3 months later I understood how "outer IUnknowns" and aggregation worked. A lot of code-rewriting followed ;). It took about a year to realize that not *everything* should be a COM class. And so it went on and on....

And I just realize... there was no 10th birthday party;) 

My friend Morten Abrahamsen from Norway, an exceptionally smart software architect with a very practical view on things and a lot of experience building large transactional systems, has written a comment very worth reading here in the blog that deserves to seen not only by the folks who accidentially click the comments link. Morten says: "Interesting how the struggle for singular performance clouds the overview of a flexible architecture that is resilient to change, versioning and implementation details.".

LRPC.

Shameless sales pitch: Little known, 18 month old book with an unfitting title about an interesting product that obviously doesn't really sell well. I am considering to go back to that material and do a rewrite which is going to be less (or not at all) product focused and instead centers around service-oriented architectures and document-driven designs. Chapters 1 & 2 can be read online at amazon.

Hey, Greg, what are you doing with that method signature [1] when you're porting that app to another data provider? What if a future revision of that method wants to add that customer to the database asynchronously and does so by stuffing it into a transactional queue, first?

[1] public int AddCustomer(CustInfo custInfo, SqlTransaction trans)