At the SDC conference in Arnhem (NL), Chris Anderson entered the following task into his Pocket Outlook: "Turn Clemens from a server developer into a smart client developer within the next 15 months." We'll see how that goes. Chris: For that to happen, you'll have to give me something that I can seriously fall in love with.
It's interesting that I get far more than 10000 unique daily unique page views on the site along with a similar number of aggregator views daily without even posting much. At least that was true for the last couple of months. Today is my "get back to blogging day". At the same time, the number of tracked direct referrals that I get when someone navigates to an entry via a link on another site is relatively low and accounts for less than 3% of the daily traffic.
I am sure I am the last to realize this phenomenon, but: I conclude that I must have a "root blog". That means that the overwhelming majority of readers don't find me via links; instead, I am on their daily reading list or in their RSS aggregator. I don't really get many on-topic inbound links, but I give links. Other great examples for "higher order" root blogs are those of Robert Scoble and Don Box, because once they link to me, the number of direct referrals rises significantly.
When I started blogging (when blogosphere was much smaller), I had a "leaf blog" that wouldn't get many reads except through other people's links. It's interesting to observe how those things change.
 The two biggest conferences in Microsoft space (save PDC)
are coming up and I am already looking forward to be in San Diego
in two weeks and in Amsterdam
four weeks later. Those two events are always very special because they are
big, because they are really well organized and because I get to meet and party
with very many good friends who I see regularly at some place somewhere on
earth, but only once a year we’re all together.
As much as I value the technical education aspect of events like that (yes,
I do attend sessions, too),  the
primary reason for me to go to TechEd is too meet friends and make new friends.
And the “networking” on the professional level that goes on at
TechEd is very important as well: There’s nothing in this industry as
valuable as learning from other people.
What I am also looking forward to is some time off when TechEd Amsterdam is
over. At that time, I will have been to 25 countries since January of this year
(several of them twice or even more often) and I would have to do some serious
analysis of my calendar to assess how many events it were. My friend Lester
Madden made the best comment on that sort of traveling lifestyle some time back
in February. We boarded one of those planes together and he threw himself into
the seat grinning sarcastically “Ah! Home, sweet home”.
So with the somewhat slow summer time ahead, I’d like to say “Thank
you for all the beer”, because Microsoft (most, but not all events were
hosted by them) certainly knows how to throw great parties. So here are my “Feierabend Awards”
for the first half of 2004 and before the “big two” events:
My “Winter/Spring 2004 Best Conference Party Award” goes
to: The Beach Party at Microsoft
TechEd Israel (Elat, Israel). Close runners up are the Arabian Night at the
North Africa
Developer Conference 2004 (Casablanca, Morocco) and the “Wild West”
party at the NT
Konferenca 2004 in Portoroz, Slovenija.
My “Winter/Spring 2004 Best Organized-After-Work-Activity Award”
goes – hands down – to Microsoft Finland and their Architecture
Bootcamp in Ruka, where we did a 25km snow
mobile ride in beautiful northern Finland and afterwards had a very Finnish “now
let’s get naked with all the customers and go to Sauna” experience.
Runner up is a great evening hosted by Microsoft Turkey at Galata Tower in
Istanbul. The restaurant up there is an absolute tourist trap, but we had a fun
night and the views from up there can’t be beat.
My “Winter/Spring 2004 Best Beer Award” must
of course go to Dublin. Not much (except our
local beer in and around
Düsseldorf) beats a fresh Guinness. Along
with that goes the sub-award for “most inappropriate workplace
discussion” about how cleavage (Def. 6) is most
effectively used in business.
The “Winter/Spring 2004 Best Restaurant Award”
goes to the Vilamoura
Restaurant (Portuguese) at the Intercontinental Hotel in Sandton/Johannesburg for
absolutely awesome shellfish. Runner up is another Portuguese restaurant: the Doca
Peixe in Lisbon/Portugal. The special Best Homefood Award goes to Malek’s mother. The “Winter/Spring
2004 Best Nightclub Award” goes to the Amstrong
(sic!) Jazz Club (which it really isn’t) in Casablanca, Morocco.
The “Winter/Spring 2004 Gorgeous Event Hostesses Recruiting
Award” (sorry, but while that’s not strictly “after work”
that’s a category that I can’t leave out) has to be evenly split
between four winners: Morocco’s North Africa
Developer Conference 2004 (just ask Mr. Forte), Slovenija’s
NT Konferenca 2004
(reliable winner each year), the Longhorn Developer Preview event
in Budapest/Hungary and the MS EMEA Architect Forum
Event in Milan, Italy. Israel already won the best party event and that should
speak pretty much for itself. Therefore they’re runner up in this
category.
The “Winter/Spring 2004 Best Travel Buddy Award”
goes to Arvindra Sehmi for the EMEA Architect Tour, and Lester Madden, Nigel
Watling, Hans Verbeeck, and David Chappell for the Longhorn Developer Preview
Tour.
Finally, the “Winter/Spring 2004 Best Host Award”
goes to my great friend Malek Kemmou from
Morocco, whose house became “Speaker’s HQ” before, during and
after the NDC conference and who took us all around the country to experience Morocco
– and refused to let any of us pay for anything.
I talked about transactions on several events in the last few weeks and the sample that I use to illustrate that transactions are more than just a database technology is the little tile puzzle that I wrote a while back. For those interested who can't find it, here's the link again. The WorkSet class that is included in the puzzle is a fully functional, lightweight, in-memory 2 phase-commit transaction manager that's free for you to use.
 Sometimes you’re trying to fix a problem for ages (months in our case) and while the solution is really simple, you only find it by complete accident and while looking for something completely different.
(And yes, I do think that we need to finally get a network admin to take care of those things)
For several months, our Exchange server “randomly” denied communicating with several of our partner’s mail servers. There were several of our partners who were not able to send us email and their emails would always bounce, although we could communicate wonderfully with the rest of the world. What was stunning is that there wasn’t any apparent commonality between the denied senders and the problem came and went and sometimes it would work and sometimes it wouldn’t.
First we thought that something was broken about our DNS entries and specifically about our MX record and how it was mapped to the actual server host record. So we reconfigured that – to no avail. Then we thought it’d be some problem with the SMTP filters in the firewall and spent time analyzing that. When that didn’t go anywhere, we suspected something was fishy about the network routing – it wasn’t any of that either. I literally spent hours looking at network traces trying  to figure out what the problem was – nothing.
Yesterday, while looking for something totally different, I found the issue. Some time ago, during one of the email worm floods, we put in an explicit “deny” access control entry into the SMTP service for one Korean and one Japanese SMTP server that were sending us hundreds of messages per minute. The error that we made was to deny access by the server DNS name and not by their concrete IP address.
What happened was that because of this setting our SMTP server would turn around and try to resolve every sender’s IP address back to a host name to perform that check and that’s independent of the “Perform reverse DNS lookup on incoming messages” setting in the “Delivery”/“Advanced Delivery” dialog. It would then simply deny access to all those servers for which it could not find a host name by reverse lookup. I removed those two entries and now it all works again.
Of course, the error isn’t really ours, but the problem was. What’s broken is that the whole reverse DNS lookup story is something that seems (is) really hard to set up and that quite a few mail servers simply don’t reversely resolve into any host name. DNS sucks.
Welcome back 
On one of those flights last week I read a short article about Enterprise
Services in a developer magazine (name withheld to protect the innocent). The “teaser”
introduction of the article said: “Enterprise Services serviced
components are scalable, because they are stateless.” That statement is of
course an echo of the same claim found in many other articles about the topic
and also in many write-ups about Web services. So, is it true? Unfortunately,
it’s not.
|
public class AmIStateless
{
public int MyMethod()
{
//
do some work
return
0;
}
}
|
“Stateless” in the sense that it is being used in that
article and many others describes the static structure of a class. Unfortunately
that does not help us much to figure out how well instances of that class help
us to scale by limiting the amount of server resources they consume. More
precisely: If you look at a component and find that it doesn’t have any
fields to hold data across calls (see the code snippet) and does furthermore
not hold any data across calls in some secondary store (such as a “session
object”), the component can be thought of as being stateless with regards
to its callers, but how is the relationship with components and services that
are called from it?
But before I continue: Why do we say that “stateless” scales well?
A component (or service) that does not hold any state across invocations has
many benefits with regards to scalability. First, it lends itself well to load
balancing. If you run the same component/service on a cluster of several
machines and the client needs to make several calls, the client can walk up to
any of the cluster machines in any order. That way, you can add any number of
machines to the cluster and scale linearly. Second, components that don’t
hold state across invocations can be discarded by the server at will or can be
pooled and reused for any other client. This saves activation (construction)
cost if you choose to pool and limits the amount of resources (memory, etc.) that
instances consume on the server-end if you choose to discard components after
each call. Pooling saves CPU cycles. Discarding saves memory and other
resources. Both choices allow the server to serve more clients.
 However, looking at the “edge” of a service isn’t
enough and that’s where the problem lies.
The AmIStateless service that I am illustrating here does not stand
alone. And even though it doesn’t keep any instance state in fields as
you can see from the code snippet, it is absolutely not stateless. In fact, it
may be a horrible resource hog. When the client makes a call to a method of the
service (or otherwise sends a message to it), the service does its work by employing
the components X and Y. Y in turn delegates work to an
external service named ILiveElsewhere. All of a sudden, the
oh-so-stateless AmIStateless service might turn into a significant
resource consumer and limit scalability.
First observation: While no state is held in fields, the service does
hold state on the stack while it runs. All local variables that
are kept in on the call stack in the invoked service method, in X and in Y are
consuming resources and depending on what you do that may not be little. Also,
that memory will remain consumed until the next garbage collector run.
Second observation: If any of the secondary components takes a long
time for processing (especially ILiveElsewhere), the service consumes and
blocks a thread for a long time. Depending on how you invoke ILiveElsewhere you
might indeed consume more than just the thread you run on.
Third observation: If AmIStateless is the root of a
transaction, you consume significant resources (locks) in all backend resource
managers until the transaction completes – which may be much later than
when the call returns. If you happen to run into an unfortunate situation, the
transaction may take a significant time (minutes) to resolve.
Conclusion: Since the whole purpose of what we usually do is
data processing and we need to pass that data on between components, nothing is
ever stateless while it runs. “Stateless” is a purely static view
on code and only describes the immediate relationship between one provider  and one consumer with regards to how much information is kept across
calls. “Stateless” says nothing about what happens during a call.
Consequence: The scalability recipe isn’t to try achieving static
statelessness by avoiding holding state across calls. Using this as a pattern
certainly helps the naïve, but the actual goal is rather to keep sessions (interaction
sequence duration) as short as possible and therefore limit the resource consumption
of a single activity. A component that holds state across calls but for which
the call sequence takes only a very short time or which does not block a lot of
resources during the sequence may turn out to aid scalability much more than a
component that seems “stateless” when you look at it, but which
takes a long time for processing or consumes a lot of resources while
processing the call. One way to get there is to avoid accumulating state on call
stacks. How? Stay tuned.
I am slowly getting out of a very, very long period of "working too much". In the last 3 1/2 weeks I worked pretty much for 18 hours every day in order to get a fairly large service oriented application done (sharing the workload with my newtelligence partner Achim Oellers). The stats: 13 services, about 20 portTypes, 1.6 MB of C# code, 10 SQL Server databases (autonomy!), countless stored procedures. We have duplex (one-way with reply path), simplex (one-way) and request/response communication paths, use ObjectPooling, Just In Time Activation, Role Based Security, Compensating Resource Managers, Process Initialization, Automatic Transactions, Service Domains, Run-As-Service, and Loosely Coupled Events from Enterprise Services, we use several features from ASP.NET Web Services, we use quite a bit of the Web Service Enhancements Tools, have full instrumentation with Eventlog support an Performance Counters, have deployment tools that create domain accounts, elevate their privileges and configure all the security settings to run a service in "locked down" mode, and use SQL Server Replication. The core services were supposed to ship yesterday and we made that date.
Now I need to work on the backlog. I am late on delivering some PowerPoint decks. I have a 12 hour travel day today. That means writing PPTs on the plane.
The EMEA Architect Tour 2004 Videos from Finland are online and certainly one of the rare chances to see me speaking in a proper suit. And we speak about FABRIQ...
I am writing a very, very, very big application at the moment and I am totally swamped in a 24/7 coding frenzy that’s going to continue for the next week or so, but here’s one little bit to think about and for which I came up with a solution. It’s actually a pretty scary problem.
Let’s say you have a transactional serviced component (or make that a transactional EJB) and you call an HTTP web service from it that forwards any information to another service. What happens if the transaction fails for any reason? You’ve just produced a phantom record. The web service on the other end should never have seen that information. In fact, that information doesn’t exist from the viewpoint of your rolled back local transaction. And of course, as of yet, there is no infrastructure in place that gives you interoperable transaction flow. And if that were the case, the other web service may not support it. What should you do? Panic?
There is help right in the platform (Enterprise Services that is). Your best friend for that sort of circumstance is System.EnterpriseServices.CompensatingResourceManager.
The use case here is to call another service to allocate some items from an inventory service. The call is strictly asynchronous and I the remote service will eventually turn around and call an action on my service (they have a “duplex” conversation using asynchronous calls going back and forth). Instead of calling the service form within my transactional method, I am deferring the call until the transaction is being resolved. Only when DTC is sure that the local transaction will go through, the web service call will be made. There is no way to guarantee that the remote call succeeds, but it does at least eliminate the very horrible side effects on overall system consistency caused by phantom calls. It is in fact quite impossible to implement “Prepare” correctly here, since the remote service may fail processing the (one-way) call on a different thread and hence I might never get a SOAP fault indicating failure. Because that’s so and because I really don’t know what the other service does, I am not writing any specific recovery code in the “Commit” phase. Instead, my local state for the conversation indicates the current progress of the interaction between the two services and logs an expiration time. Once that expiration time has passed without a response from the remote service, a watchdog will pick up the state record, create a new message for the remote service and replay the call.
For synchronous call scenarios, you could implement (not shown here) a two-step call sequence to the remote service, which the remote service needs to support, of course. In “Prepare” (or in the “normal code”) you would pass the data to the remote service and hold a session state cookie. If that call succeeds, you vote “true”. In “Commit” you would issue a call to commit that data on the remote service for this session, on “Abort” (remember that the transaction may fail for any reason outside the scope of the web service call), you will call the remote service to cancel the action and discard the data of the session. What if the network connection fails between the “Prepare” phase call and the “Commit” phase call? That’s the tricky bit. You could log the call data and retry the “Commit” call at a later time or keep retrying for a while in the “Commit” phase (which will cause the transaction to hang). There’s no really good solution for that case, unless you have transaction flow. In any event, the remote service will have to default to an “Abort” once the session times out, which is easy to do if the data is kept in a volatile session store over there. It just “forgets” it.
However, all of this is much, much better than making naïve, simple web service calls that fan out intermediate data from within transactions. Fight the phantoms.
At the call location, write the call data to the CRM transaction log using the Clerk:
AllocatedItemsMessage aim = new AllocatedItemsMessage();
aim.allocatedAllocation = <<< copy that data from elsewhere>>>
Clerk clerk = new Clerk(typeof(SiteInventoryConfirmAllocationRM),"SiteInventoryConfirmAllocationRM",CompensatorOptions.AllPhases);
SiteInventoryConfirmAllocationRM.ConfirmAllocationLogRecord rec = new RhineBooks.ClusterInventoryService.SiteInventoryConfirmAllocationRM.ConfirmAllocationLogRecord();
rec.allocatedItemsMessage = aim;
clerk.WriteLogRecord( rec.XmlSerialize() );
clerk.ForceLog();
Write a compensator that picks up the call data from the log and forwards it to the remote service. In the “Prepare” phase, the minimum work that can be done is to check whether the proxy can be constructed. You could also make sure that the call URL is valid, the server name resolves and you could even try a GET on the service’s documentation page or call a “Ping” method the remote service may provide. That all serves to verify as good as you can that the “Commit” call has a good chance of succeeding:
using System.EnterpriseServices.CompensatingResourceManager;
using …
///
/// This class is a CRM compensator that will invoke the allocation confirmation
/// activity on the site inventory service if, and only if, the local transaction
/// enlisting it is succeeding. Using the technique is a workaround for the lack
/// of transactional I/O with HTTP web services. While the compensator cannot make
/// sure that the call will succeed, it can at least guarantee that we do not produce
/// phantom calls to external services.
///
public class SiteInventoryConfirmAllocationRM : Compensator
{
private bool vote = true;
[Serializable]
public class ConfirmAllocationLogRecord
{
public SiteInventoryInquiries.AllocatedItemsMessage allocatedItemsMessage;
internal string XmlSerialize()
{
StringWriter sw = new StringWriter();
XmlSerializer xs = new XmlSerializer(typeof(ConfirmAllocationLogRecord));
xs.Serialize(sw,this);
sw.Flush();
return sw.ToString();
}
internal static ConfirmAllocationLogRecord XmlDeserialize(string s)
{
StringReader sr = new StringReader(s);
XmlSerializer xs = new XmlSerializer(typeof(ConfirmAllocationLogRecord));
return xs.Deserialize(sr) as ConfirmAllocationLogRecord;
}
}
public override bool PrepareRecord(LogRecord rec)
{
try
{
SiteInventoryInquiriesWse sii;
ConfirmAllocationLogRecord calr = ConfirmAllocationLogRecord.XmlDeserialize((string)rec.Record);
sii = InventoryInquiriesInternal.GetSiteInventoryInquiries( calr.allocatedItemsMessage.allocatedAllocation.warehouseName );
vote = sii != null;
return false;
}
catch( Exception ex )
{
ExceptionManager.Publish( ex );
vote = false;
return true;
}
}
public override bool EndPrepare()
{
return vote;
}
public override bool CommitRecord(LogRecord rec)
{
SiteInventoryInquiriesWse sii;
ConfirmAllocationLogRecord calr = ConfirmAllocationLogRecord.XmlDeserialize((string)rec.Record);
sii = InventoryInquiriesInternal.GetSiteInventoryInquiries( calr.allocatedItemsMessage.allocatedAllocation.warehouseName );
try
{
sii.ConfirmAllocation( calr.allocatedItemsMessage );
}
catch( Exception ex )
{
ExceptionManager.Publish( ex );
}
return true;
}
}
Here's a life sign. I am buried under lots of work of which pretty much all will see the light of day at TechEd. We're getting close to having a first public version of the FABRIQ project with Microsoft EMEA and we're very busy here at newtelligence writing a huge SOA sample application using and combining all the good things of ASP.NET Web Services, WSE 2.0, Enterprise Services, MSMQ, SQL, and Remoting. The result is quite likely going to play some role at TechEd US and other TechEds this year. Between then and my last technical blog positing I've written several thousand lines of code again and there are several thousand more to follow. Hence the silence. Once those two projects are done or close to being done, expect a flood of explanations.
One year ago (plus 5 days), I posted this here on my blog. I just found it again through my referral stats. Of course, that post isn't about Juliet, at all. Fun.
You've gotta love this sentence from here: "Pool resources to get additional UK Government and European Union funding."
|