Craig Andera on why AOP is broken & Why I surprisingly agree and what I am doing about it

Craig Andera has some interesting thoughts around AOP and specifically mentions the stuff that I have been doing in that area. And he say that it doesn't work and never has, because services are never truly orthogonal and have various interdependencies. In essence he's saying (I guess) that because the interdependencies just create a whole new level of complexity, the AOP approach is broken and it's better to generate explicit code instead of using interception techniques. I partially agree and always put a warning at the end of all of my talks around this issue: There is a limited set of use-cases for which an aspect'ish approach is useful. Security, logging, monitoring, billing, transaction enlistment, and a few others.

One of the biggest problems is service-order. You need to run the decryption and signature verification services before you can even evaulate a header that any other service can use. And even then, when you have something like a transaction-enlistment filter, do you open the transaction before or after a logging service wants to write something to a database? Does the logged data need to stay in the logging store when a transaction aborts? Yes? What if the log is used for billing? No? What if the log is used for diagnostics?

However, being explicit when chaining services together doesn't make things any better than using interception:

try
{
   handleServiceA(msg);
   handleServiceB(msg);
   handleServiceC(msg);
}
catch( Exception e )
{
   // do proper handling
}

is just as broken. I don't think it fundamentally matters much how code gets woven into the call chain. Setting up contexts is just one issue. What's even more difficult is to find a way to deal with errors in the presence of cooperating aspects (or, in more general terms, interception services). What's clear is that there's no way around interception-driven services in a web services world. It's all pipeline-based and, even worse, the pipelines are distributed pipelines of pipelines. It's too simple to say "it's broken, get over it". That doesn't help solving what is an actual problem.

A promising approach is to make aspects/interceptors act like resource managers and coordinate their work using a very lightweight 2PC protocol ("AC" guarantee only; no "ID"). Using 2PC for this approach allows interceptors/aspects to coordinate their work and know about each other before any work actually gets done. I have discussed these issues with a couple of people in depth we put some code together that essentially implements a little, in-memory "DTC" for that purpose. We call it a "WorkSet" instead of a transaction.  There's still some work to be done there, but I think I'll be able to post an example in a little while. Maybe around TechEd Europe time.

Sam on Perf

Sam outs himself as a fan of low-level performance optimization. That's all good and fair, but often micro-optimization just takes way too much time with way too little of a result for the overall application throughput and its scalability. For distributed apps, the true optimization happens during the architecture phase. Or, as my friend Steve Swartz put it during our "Scalable Apps" tour: When you are stuck in a traffic jam with a Porsche, all you do is burn more gas in idle. Scalability is about building wider roads, not about building faster cars.

On 400 level sessions and scores

Samer Ibrahim writes "I believe that a 400 level session should present 400 level material regardless of how many people have never wrote a single line of code in their entire life.  That's not my problem and that's not fair to those of us who are here to get an edge.  Find 100-200 level sessions instead."

My WEB404 session at TechEd US was probably a 500 because I really had lots (too much) of code. The downside of doing 400 level sessions at an event with a very broad audience spectrum is that you are getting killed in the feedback and scores after the talk, no matter what you do. Either you're too shallow for some or you are too deep down in the bits for others. Now, what needs to be understood is that speakers will often scale back on content if they feel that the content is too deep for the audience they have, just because it'll kill their average score. There's lots of competition behind the scenes on that.

What was new at this TechEd was that the written comments are now available to MS in "softcopy", which means that they get printed up with the numbers. And if you have only 10 people in an audience of 300 who write "Thank you, this session was really helpful for me", you feel like you have done your job right and MS sees that too, which is of much higher importance for "us" external speakers than the average score.

So, here's a hint: My understanding is that the scoring system is still open over the weekend at www.mymsevents.com. If you attended a session that you found helpful and on which you haven't given a score so far, do so and don't forget to write a comment stating what you liked or what you would like to see improved. That's especially true for sessions with deep and focused technical content and lots of people in the audience. These will typically get comparatively bad scores, because it's nearly impossible that the content is absolutely relevant for 400 or 600 people in a room at a conference like that. So, if you think that the speaker did a good job, say so. You'll be heard.

(I should add that I am fairly happy with my scores already and I am not begging ;)

Andres observes that Steve and I are in agreement on very many things, including what to put on slides in talks covering services, layering and tiering.  ;)

Clemens - I attended your two sessions, AOP and "I dont know you could do that". Excellent stuff. Couple of questions I have:
1. I heard, and I might be wrong(and please correct me if I am), that you have serious issues with .Net Remoting. Is that true, and if it is, why? 2. In an app where you want to cache objects, would you use com+ object pooling, or are there better ways to cache your objects?
And last, but not least, have u written any papers? And can you tell me any good book to go deep into the stuff that you talked about?

Thanks Ali / Ali Khawaja • 6/6/03; 5:53:13 PM

1. I don't have serious issues with Remoting as such. I am just saying that it is the successor of Automation and not of the full blown DCOM model. Hence, it is useful in all the scenarios (mostly on-machine) where Automation is useful in the unmanaged world. Once you go across machines where security plays a role and when you need an appropriate hosting and process model for your objects, there is Enterprise Services. Whenever you see a need to add a custom channel sink to Remoting for authentication, authorization, encryption, or signature, there is a fair chance that you are using the wrong technology set. Whenever you think you need to write a custom host for you app in order to tune the thread pool and up the number of available threads for Remoting, you are using the wrong technology set. There's nothing fundamentally wrong about Remoting -- there's just a limited set of use-cases where it is applicable. My issue with it is only how many people are using it and how it is being portrayed as the successor to DCOM, which it is not.

One thing is important to keep in mind: The COM transport sits on top of Microsoft RPC, which is, in turn, the core technology stack that essentially powers most call-level communication between the components of Windows and hence has had full kernel support ever since the NT kernel saw the light of day. RPC supports virtually all network protocols as well as shared-memory marshaled L(R)PC [read!] for on-machine calls. Remoting sits on top of the CLR and on top of the Framework, which, in turn, sits on the Win32 user-level API. That's a wholly different ballgame.

Enterprise Services has a very elegant solution for mixing the two models in that it uses Remoting to do almost all marshaling work (with two exceptions: QC and calls with isomorphic call sigs) and then tunnels the serialized IMessage through DCOM transport, which means that you get full CLR type fidelity while using a rock solid transport that has been continuously optimized ever since 1993. I understand that some people consider a 10 year old protocol boring; I just call it "stable". Also I see people complaining about COM being hard to deploy, because it requires use of the registry and distribution of proxies. Admittedly, there's some truth to that, but in the end, you will also have to deploy and customize config files for Remoting and distribute proxies there. That's true for any RPC-type technology and is, as per current practice, even true for most Web Services. For distributed systems of any scale, "xcopy deployment" is a sweet dream. There's work to do.

2. Yes. Enterprise Services object pooling is great to pool object instances and guard access to limited resources.

Finally, I have written a book on Enterprise Services, which is, for a variety of historic reasons, in German. However, I am talking to a publisher for translation and once that happens I will definitely rev it so that it incoporates all of my "current" thinking (of course).

PerfectTheft.

PerfectXml.com has a redirect tool up that presents this blog (and everyone else's blog) on their site: http://www.perfectxml.com/RSSConnect/RR.asp?u=http://radio.weblogs.com/0108971/rss.xml. Since they didn't get back on my email, I'll have to tell them in public:

Any content of this blog is my property. The RSS feed is available for aggregation and personal information by anyone, BUT if you republish my weblog on your website without my permission, you are stealing intellectual property and you are violating my copyright.  Take that redirector down or block my blog.

Thank you, Julia. I am glad you liked my TechEd sessions and thank you for the kind words :)  However ...

What is so strange is that I cannot get  used to seeing him open up and work in Visual Studio. Why on earth is that? Perhaps it is something to do with the level of what he is talking about that it is bigger than coding, so though he obviously needs to code to put the concepts in action, it just seems almost mundane in comparison to the concepts.

Hmmm .... I am not sure whether I agree here. Most of the things that I talked about were really about code all the way and then I can just as well show some (or flood the audience with codce as in WEB404). The takeaway are the concepts. My job is to bring lesser known things into the limelight. In that I do agree.

TechEd / DEV359, WEB404 related code

Earlier builds and some explanation of the stuff that I have been showing in the talks can be found here (Enterprise Services AOP) and there (Web Services Extensibility). These builds are for Visual Studio .NET 2002. The builds for the new version will -- as said -- be available some time next week. Don't complain... it's free stuff, after all ;)

TechEd / Getting stuff out the door: Code for DEV357

Pending a more polished and documented version (which I'll publish some time next week), here's "just" the zipped up directory with the code from the DEV357 session (Building distributed apps). 41 C# files, 375KB of source code. Way too much ;)

The code for DEV359 and WEB404 is a bit more difficult to pack up, because it's much harder to deploy and get to work without a proper installer. Unfortunately all WMI support for the Framework died on this machine this week ("Provider load failure") and is fubar and therefore I can't test the installation procedures to put stuff into machine.config. So that may have to wait until next week :(

Demos, demos, demos

I should probably stop writing more stuff for my Thursday demos. I think that some 10000 lines of "giveaway" source code should be enough ... but somehow I feel like I am still not done yet. Here's a quick list of the stuff that I have with me to show.

• Aspect oriented programming with Remoting and Enterprise Services and Web Services using the same extensibility model
• An attribute driven validator for object-graphs of structures and classes of arbitrary depth (sort of like schema validation for XML, but on objects)
• Schema facets (maxLength, minLength, pattern, etc.) generated into the WSDL from [WebMethod] parameters and data structures
• Just in Time Activation pooling as a the ultimate performance booster for Enterprise Services
• A multi-tier "cascade" for data services for that can serve up read-only data from cache (memory) or isolated storage or a remote web service or straight from a SQL store through the flip of a config file entry and which are connected in a way that the isolated storage refreshed through the web service which then walks up to SQL.

What I will show in which detail largely depends on how the talks go in terms of timing. Since my DEV357, DEV359 and WEB404 talks are all back-to-back-to-back (different rooms, though), I will essentially be using one larger demo for all of them (and I am still putting it together right now .. cough!). In DEV357, I'll primarily talk about the relationship between ASMX Web Services and Enterprise Services and how to use ES efficiently as a backend for ASMX. In DEV359 I'll drill down into the "aspectish" elements of the demo application and talk about separation of primary concerns ("why you write the app") and secondary concerns ("stuff that needs to be done, too"). In WEB404 I will show how I teach "Add Web Reference" to generate code that has references to stuff in "newtelligence.Web.Services" in it and how I can make the schema in ASMX's generated WSDL a bit better.

I will try to post links to as much of the actual source code for the demos and its support libraries here until the end of the week. Don't expect anything before Thursday, though. Right now I am writing installers, because I don't want to make it unneccessarily hard for all of you to try the stuff at home.

TechEd: Meet Juval Löwy and me at the INETA booth

Juval Löwy and myself will be at the INETA booth (Aisle 600) in the expo area at TechEd today between Noon and 1:30pm and between 3:15pm and 5:00pm. So, if you have any questions about Enterprise Services or Web Services just come over and I am sure that Juval will have an appropriate answer for you ;)

"Power Lunch with Don Box and Friends"

As said before, I got invited to a fun lunch panel discussion with Don, Yasser Shohoud, and Steve Swartz. We chatted about 45 minutes about things we all like and dislike about the .NET Framework as it ships today, about XML and SOAP standards, how to build Web Services "right" in .NET, about the unfortunate split between infrastructures for Remoting, Enterprise Services and ASMX and plenty other little things.

Quote of the day:

• Don: "So, Steve, is COM dead?"
• Steve: "There's a time when you are growing up and everything is exciting at the time. There's always new things, new stuff to look at, it's all cool. And then at some point you're grown up and it's not that you die when you're a grownup, right? So, COM is a grownup now. It just lives." 

Ping! .... from TechEd 2003 Dallas

I haven't been blogging for more than  two weeks now because of (a) being very busy on the road and (b) being sick all last week. After 8 or so weeks on the road, my body just went on strike and punished me for all the stress in many horrible ways.

Anyways, I am back online now, sitting in the speaker's lounge at TechEd Dallas checking some email, working on the samples for Thursday (details on that later) and just got invited by Don to participate in the "Web Services Roundtable: Power Lunch with Don Box and Friends" session that's at 12:15pm in the Arena. I don't know what's on the agenda, but Don said I should just come up. I am sure it's going to be fun.

Still on the road.

Friday was my first scheduled office day after 2 1/2 months on the road. Now I am sitting in the CSA airport lounge in Prague (free Internet, bring cable) waiting for my flight to Bucharest. Tuesday I go to Frankfurt, then on to our TornadoCamp.NET event in Bad Ems (Germany) where I'll be for the rest of the week. As per current planning, I'll have a couple of days at home next week and then I am going to Dallas, TX for TechEd. The U.S. is country #18 on this year's list -- no, wait, been there already this year.

For TechEd I am prepping a set of new extensions for ASMX. I think ASMX doesn't do as well in regards to its support for XML Schema as it could.

My talks at Microsoft TechEd in Dallas will be:

WEB404: ASP.NET Web Services Extensibility: "I didn't know you could do that!"
DEV359: Aspect Oriented Programming
DEV357: Building Distributed .NET Applications

The last one (which is likely going to be rescheduled to earlier in the week than Thursday) is the most important and ambitious one. I will try to give some guidance around the "appropriate use" of .NET Remoting, Enterprise Services and Web Services and show that there that there's no "vs." or "or" between those three technologies but an "and". I'll talk about layers and tiers, process models, security, data, objects, services -- all in a bit more than an hour.

A 3-day workshop based on the Microsoft EMEA Building Scalable Applications Tour 2003.  The first day is direct from the tour and days 2 and 3 are putting it all into practice in a hands-on workshop! Send mail to info@newtelligence.com for details.

Maximum travel -- This past week

This week was a crazy travel week and I can't wait to be back home tonight. It's a miracle that I still have my bag and didn't miss any connections and everything worked out. Here's a brief summary of the travel ordeal:

• Sunday afternoon: Drove from Meerbusch (Düsseldorf) to Frankfurt (2 hrs) for the first stop on the German roadshow with Jörg. Jörg took my car from there to Munich for another event.
• Monday: Frankfurt event, went to the aiport in the afternoon to catch a flight to Oslo for the Norwegian Developer Days.
• Tuesday: Oslo event, then to the airport to catch a 1845 flight to Frankfurt, got into Frankfurt with a 30 minute delay into the "B" gates, with my connection to Venice already boarding for 10 minutes in the "A" gates. Frankfurt is big. Ran from "B" to "A" an got to the gate just before they were closing the doors. Of course, the Venice flight ended up being in line for takeoff about 30 minutes. Got into Venice late (and miraculously still got my bag) and made it to the rental car booth about 5 minutes before the lady there would have closed shop. Got a car and drove to Trieste and then on to Portoroz in Slovenia. Got into the hotel at 0200.
• Wednesday: Slovenian event
• Thursday: Got up at 0500, got into the car at 0530 and drove back to Trieste airport (which isn't called Trieste Airport). Caught an 0820 plane to Munich, got there at 0930, got into a cab and was at the event location for the Munich stop of the German launch 20 minutes before my talk was due to start. At 1815 got into a cab to the aiport, boarded at 1950 for a flight to Berlin. Arrived at the hotel at around 2200
• Friday: Berlin event, tonight at around 2000 going home to Düsseldorf. I'll be using the 19th and last flight coupon of the ticket that I had on me in the last three weeks, Scalable Applications Tour included. Next week: next ticket.

That much on "all this travel must be great fun". Being at events and talking to developers and architects is great fun, the travel part isn't, really.

Enterprise Services Beispiel von der deutschen VS.NET/Server Roadshow 2003
Enterprise Services Sample from the Slovenian NT Konferenca 2003, Norway Developer Days

Dieses Archiv beinhaltet das in meinem Enterprise Services Vortrag gezeigte Beispiel zu JustInTimeActivation und Pooling von der deutschen VS.NET/Windows Server 2003 Roadshow.

This sample around JustInTimeActivation and Pooling has also been shown at the DevDays in Norway last Tuesday and the NTKonferenca in Portoroz on Wednesday and is a slight variation of the DataSet caching example that I posted earlier.