Dear Aiden,
I think you remember the conversation we had recently at this software
conference in Dublin. You came up to me and told me how the stuff I was talking
about was mostly useless, because it is closed-source, people need to pay for
it and that companies charging for software are evil anyways – especially
Microsoft. Unfortunately I don’t have your email, but I am sure this
will reach you.
First, I would like to thank you for the interesting conversation that
developed and to make sure that none of what was said just fades away,
I’ll tell you here once again what I am thinking about what you do, what
you think and – most importantly about your future.
When I was 21 – like you now – I was also at university and was
pursing a computer science master degree. Back then, I was very enthusiastic
about programming and creating stuff that mattered. And thought that I was the
best programmer the field has ever seen and everyone else was mostly worthless.
And I did indeed write some programs that mattered and made a difference. The
program I spent some 3 years writing in Turbo Pascal from when I was 18 was for
my father’s business. Because the business he’s in requires a lot
of bureaucracy, he and my mother spent about 2-3 daily hours on average doing
all of this stuff by hand. When I was done with my program and he started using
it, that time went from 3 hours to about 15 minutes a day. That was software
that absolutely improved the quality of life for the entire family! And his
friends and colleagues loved it, too. I didn’t sell many licenses at that
time (I think I had 3 customers), but each one was worth 1500 German Marks and
that was a huge heap of money for me. I mean – I was living at my
parent’s house, getting a monthly allowance of 120 German Marks and
worked as a cable grip for a couple of TV stations every once in a while
– maybe 2-3 times a month. And if I ever had 400 Marks per month I could
really consider myself massively rich at the time and for my age, because I had
very minimal additional expenses. So 4500 Marks on top of that? Fantastic.
Where did the money go? I can’t really remember where it all went, but I
guess “lot of partying” or “Girls, Drugs and
Rock’n’Roll” would be a reasonably good explanation. Hey, I
was 21 and that’s what one is supposed to do at that age, right?
That was in 1990 – let’s fast forward to 2004 and you. All
software that you and your father could possibly be interested in has already
been written. That’s probably not true, but it’s hard to think of
something, right? Ok, the software may not run on your favorite operation
system and may cost money, but what you can immediately think of is likely
there. So where do you put all your energy? Into this absolutely amazing
open-source project you co-coordinate. I mean, really, the stuff that you and
your buddies are doing there is truly impressive. There are a couple of things
I’d probably do differently in terms of design and architecture, but it
works well and that’s mostly what matters. And you do make an impact as
well. I know that hundreds of people and dozens of companies use your stuff.
That’s great.
However, I start to wonder where your benefit is. You are – out of
principle – not making any money out of this, because it is open-source
and you and your buddies insist that it must be absolutely free. So you are
putting all of that time and energy into this project for what? Fame? To found
a career? Come on.
If someone installs your work from disc 3 of some Linux distro, they
couldn’t care less who you are. The whole fame thing you are telling me
only works amongst geeks. The good looking, intelligent girl over there at the
bar that you’d really like to talk to doesn’t care much whether you
are famous amongst a group of geeks and neither does she even remotely fathom
why you’d be famous for that stuff in the first place. I mean – get
real here.
So once you get your degree from school, what’s the plan?
Right now, your university education is free like in many places in Europe and
you have plenty of time to work on your degree without too much financial
pressure. Over here in Germany things are a bit extreme in that it is not
uncommon that folks spend 6, 8 or even 10 years (!) in school until they
finally get their masters degree. So you may not have to think about this much
now and you probably don’t. But let’s talk about it anyways.
When you leave school, your parents will – honestly – be keen to
get you out of their house. They’ve spent 25 years of their life being
parents and now that they are in their early 50s, they want to enjoy their life
and I am sure that your dad is keen to play with grandchildren – but just
every once in a while. So you’ll have to take care of yourself.
How so? Well, you need to get a job that pays. And you’ll probably
want to have your own car, your own apartment and if you really want to have a
family you will have to be able to support it. All of that only works with
money. Where does it come from? If you believe that the result of your own work
must be free for everyone – who’s going to pay for it?
No – in the end you are going to settle for a job that pays for
your house, your car and your wife and children. You’ll be a developer
and, eventually, architect or project manager who produces software for money.
That’s your core skill and that’s what you invested 6 years and
more of your life into. That money will either come from some internal budget
of the company that you work for as a “corporate developer” or it
will come from the clients that license the software that your company
produces. In the end, there’s got to be money in your pocket. I know
that’s not very romantic and has very little to do with the “free
software is love” sort of thing, but it’s inevitable. Romantic is
what you can get out of that money and that’s a decent life with a house,
a car and a family.
Yes, I know the argument. Software is supposed to be free and the money is
made out of supporting it. Look around you. Read some industry magazines. Who
exactly is making money out of “free”? IBM does, HP does and the
large consulting companies do. They rake in the big bucks. But do they make the
money on open-source software? No, they make that money on outsourcing deals,
running data centers and selling hardware. That’s not the side of the IT
business that is at all concerned about creating software that you want to be
in. That is the side of the IT business that runs software.
Where money is made from creating software, software isn’t free.
Either the software is paid for directly or it is cross-subsidized from budgets
elsewhere in a company that also sells hardware or consulting services.
The whole thing about “free software” is a lie. It’s a
dream created and made popular by people who have a keen interest in having
cheap software so that they can drive down their own cost and profit more or by
people who can easily demand it, because they make their money out of speaking
at conferences or write books about how nice it is to have free software. At
the bottom of the food chain are people like you, who are easily fooled by the
“let’s make the world a better place” rhetoric and who are so
enthusiastic about technology that writing open-source – or any source
for that matter – is the absolutely best imaginable way to spend their
time. It doesn’t matter whether you love what you are doing and consider
this the hobby you want to spend 110% of your time on: It’s exploitation
by companies who are not at all interested in creating stuff. They want to use
your stuff for free. That’s why they trick you into doing it.
And I sure understand the whole altruistic aspect of this and the idea of
helping people to have better lives through free software. There’s a
saying that goes: “If you are 20 and you aren’t a communist you
have no heart.”, but it continues “if you are 30 and you still are
a communist, you lack rationality”.
In the end, Aiden, it’s your choice. Do you want to have a car, a
house and a family when you are 30? Do you love being a software engineer at
the same time? If so, you literally need to get a life. Forget the dream about
stuff being free and stop advocating it. It’s idiocy. It’s bigotry.
If you want to put your skills to work and you need to support a family, your
work and work results can’t be free. Software is the immediate result and
the manifestation of what your learned and what you know. How much is that
worth? Nothing? Think again.
With best wishes for your future
Clemens
If Sun were actually to open-source (that's a verb now, is it?) Java as IBM demands, IBM would finally own it. They've got more resources and they'd throw them at the problem, easily taking away the leadership in the Java space. Sun would just be sitting there, watching in disbelief what happens to what used to be their stuff. That's really what IBM wants and I am amazed how clever they are about it.
So, let's assume Sun would fall for it. Then there would be a core Java environment that's open-source and the J2EE (the stuff that really matters) implementations would still be closed? How much would that be worth? So the next logical call is to say "let's open up all those J2EE app servers and related infrastructures, too." And there we have IBM making WebSphere a freebie they throw into projects (isn't that done, anyways?) and killing off most of the software revenue models of their competition while happily buzzing along with their huge global services operation and their server hardware business.
Of course that's just an evil conspiracy theory.
Of course, there is really no unanimously agreed-upon definition of what’s absolutely fundamental to SOA – or even what SOA really is. But I think there are four things that most people agree on and I think I there’s even a fifth:
To me, the first four core principles are:
· Explicitness of boundaries [read: there’s stuff that is explicitly public and the rest isn’t],
· Data exchange governed by an implementation independent message contract,
· Compatibility of behaviors through negotiation of capabilities based on policies and
· Service autonomy.
Number five is:
· Locating and binding to a remote service is always indirect [read: the most important design pattern is the factory pattern]
I hear that there's quite a bit of amusement among the more senior Microsoft folks (and people like me) that there’s a lot of "reinventing COM" going on. It’s not that there’s a push into that direction. It just seems to happen. All of a sudden folks are playing with (differently named) variants of monikers, class factories and all those things. Say what you will, the IClassFactory indirection is great thing to have – one place to find a service, one place to configure a proxy, one place to sneak in a mapper/wrapper that makes the actual service you talk to look like another service.
(Note that I don’t mention SOAP here. Must a service use SOAP? How about services that fall back to something without angle brackets because their respective policies indicate that they can?)
I am currently writing the speaker notes for a service-oriented architecture workshop that Microsoft and newtelligence will run later this year. I was just working on the definitions of components and services and I think I found a reasonably short and clear definition for it:
One of the most loaded and least well defined terms in programming is "component". Unfortunately, the same is true for "service". Especially there is confusion about the terms "component" and "services" in the context of SOA.
The term component is a development and deployment concept and refers to some form of compiled code. A component might be a JVM or CLR class, a Java bean or a COM class; in short, a component is any form of a unit of potentially reusable code that can be accessed by name, deployed and activated and can be assembled to build applications. Components are typically implemented using object-oriented programming languages and components can be used to implement services.
A service is a deployment and runtime concept. A service is strictly not a unit of code; it is rather a boundary definition that might be valid for several different concrete implementations. The service definition is deployed along with the components that implement it. The communication to and from a service is governed by data contracts and services policies. From the outside, a service is considered an autonomous unit that is solely responsible for the resource it provides access to. Services are used to compose solutions that may or may not span multiple applications.
Let me repeat the one sentence that made me go “damn, I think now I finally have the topic cornered”:
A service is strictly not a unit of code; it is rather a boundary definition that might be valid for several different concrete implementations.
Tiago is already disappointed about my talk tomorrow. Easy! It's not entirely dumbed down. ;)
On our 4 hour taxi ride from Portoroz in Slovenia to Zagreb in Croatia, I decided to make some significant changes to my Indigo slide deck for the tour. David Chappell called my talk an “impossible problem”, mostly because the scope of the talks we are doing is so broad, ranging from the big picture of Longhorn over Avalon and WinFS to the Whidbey innovations and I am stuck in the middle with a technology that solves problems most event attendees don’t consider to have.
So I took a rather dramatic step: I dropped almost all of the slides that explain how Indigo works. What’s left is mostly only the Service Model’s programming surface. For the eight slides I dropped, I added and modified six slides from the “Scalability” talk written by Steve Swartz and myself for last year’s “Scalable Applications Tour”, which now front the talk. Until about 20 minutes into the “new” talk, I don’t speak about Indigo, at all. And that turned out to be a really good idea.
As I’ve written before, many people who attend the events on this tour have no or little experience in writing distributed applications. In reality, the classic 2-tier client/server model where all user-code sits on one tier (let it be Windows Forms, VB6, ASP or ASP.NET) and the other tier is the database does still rule the world. And, no, the browser doesn’t count as a tier for me; it’s just a “remote display surface” for the presentation tier.
Instead of talking about features, I now talk about motivation. Using two use-case scenarios and high-level architectural overviews modeled after Hotmail and Amazon (that everybody knows) I explain the reasons for why distributing work across multiple systems is a good thing, how such systems can be separated so that each of them can scale independently and what sort of services infrastructure is needed to implement them. And it works great. Once I have the audience nodding to the obvious goodness I can continue and map the requirements to Indigo features and explain the respective aspects of the service model. The flow of the talk is much better and the attendees get more and immediate value out of it. If I weren’t so time constrained I would probably map it to Enterprise Services (now) and Indigo (future) all in the same talk and also show to do the transition. I am sure that I can do that sort of talk at some event this year.
Lesson learned: Less features, more why. With the majority of developers the challenge isn’t about showing them how distributed systems are being improved; it’s about getting them to understand and possibly adopt the idea in the first place.
I am in Budapest today and I am just done with my Indigo talk (you can find the slides at http://codezone.info under “Talks”), having done it for the 6th time on this tour throughout Europe. After the events Den Haag, Oslo, Copenhagen, Helsinki and Geneva, I still find Indigo a very difficult topic to talk about on this tour. It’s not about technology or because my talk doesn’t work: It’s about whether people think it’s relevant to their work.
The true challenge is to explain to the developers we meet that Indigo is going to be very important for them down the road. I find that when I talk to developers on this tour or look at their evaluation forms that very many of them apparently still write fairly compact (to avoid the word monolithic) ASP.NET applications or Windows Forms applications that use a conservative client/server approach. All presentation and logic resides in one tier and the only remote component worth mentioning is the database. That means that the majority of the folks sitting in my talks hasn’t even touched one of the existing distributed technology stacks that Indigo is set to replace.
The difficulty presenting Indigo on this tour – alongside sexy stuff like declarative UI programming with spinning Windows and Videos with alpha-blending in Avalon and googlefast cross-media searches across all of your local storage media as in WinFS – is that Indigo is about things that are hidden inside applications and do not surface to the user. Stuff that drives server-applications is sometimes hard to understand without knowing the architectural background and the motivations. (Sidenote: A while ago I heard a rumor from a usually trustworthy source that the spinning balls in the COM+ Explorer exist because COM+ was horribly hard to demo as well and the spinning balls provided a good way of visualizing that stuff was happening.)
The ideal talk for an unsuspecting audience with little knowledge in distributed systems would have to sell the whole idea of distributed systems to boot, the experiences and errors made, the reasons for why Web services are a good thing, the problems creating the motivation for and the principles of service oriented architectures, a set of some tangible application examples and use cases along with the solutions that Indigo provides; all of that in the same talk and within 75 minutes. And that in a way that developers get to see code and demos, too. That sort of talk would span about 20 years of distributed computing history. I am not sure this fits in 75 minutes. Therefore I think I will have to be happy with only a fraction of the audience being interested and/or willing to appreciate the things that I am talking about here.
Very many folks think that the topics I am talking about are only relevant to “big apps” and have a hard time seeing the benefits of something like Indigo – much in the same way as it is with Enterprise Services or Web Services.
If you believe Don Box, who said at PDC that Indigo will ship at some point between Whidbey and Longhorn, and think about the implications of that, Indigo is in fact relevant to everyone writing applications that expose functionality to other applications in some way – now or at least quite soon. The first ship vehicle for Indigo will be, if Don’s statement holds water in its consequences, some service pack or upgrade pack for Windows Server 2003 and Windows XP. That means nothing less than the entire application infrastructure of Windows Server 2003 is getting a major upgrade probably in a year or so from now.
If you are writing applications using ASMX, Remoting or Enterprise Services today, the impact of Indigo’s arrival can be immediate if you want to make it so. If you code your applications cleverly today (following guidelines explained by Joe Long here or in my talk) and don’t play too many tricks on the infrastructure – for instance by using the Remoting extensibility points – you should have a fairly smooth upgrade path to Indigo. The goal is that upgrading code will be simple and mechanical in most cases.
Don says that BEA's Deputy CTO has missed the cluetrain. I absolutely agree with Don's opinion on this article and what's even worse than the things said is what the article implies. If that is BEA's official position, this is nothing less than an outing that they are passengers in the backseat of a car that is driven by IBM and Microsoft (switching drivers every once in a while) and that they're neither behind the spirit of the whole undertaking nor do they fully understand the specifications they have put their names on. Integration or standardization on the API level has failed miserably in countless attempts and any middleware company (including BEA) that is out there to compete on features must go beyond the least common denominator approach to win over customers. Does BEA have Indigo envy?
I admit: I haven't read anybody's blog in over 1 1/2 months. I haven't posted much. I actually got tired of blogging and reading blogs for a while. Multiple projects under pressure don't go well with blogging for me. I don't even have an RSS reader on this new machine yet (I am going to change that within the hour). There's a lot of stuff to write about and I plan to write lots about the FABRIQ (which is coming along nicely), Transactions and Indigo while on the road. I've got my blogging mojo back: Yeah, baby.
The evolution of in-memory concept of messages in the managed Microsoft Web Services stack(s) is quite interesting to look at. When you compare the concepts of System.Web.Services (ASMX), Microsoft.Web.Services (WSE) and System.MessageBus (Indigo M4), you'll find that this most fundamental element has undergone some interesting changes and that the Indigo M4 incarnation of "Message" is actually a bit surprising in its design.
ASMX
In the core ASP.NET Web Services model (nicknamed ASMX), the concept of an in-memory message doesn't really surface anywhere in the programming model unless you use the ASMX extensibility mechanism. The abstract SoapMessage class, which comes in concrete SoapClientMessage and SoapServerMessage flavors has two fundamental states that depend on the message stage that the message is inspected in: The message is either unparsed or parsed (some say "cracked").
If it's parsed you can get at the parameters that are being passed to the server or are about to be returned to the client, but the original XML data stream of the message is no longer available and all headers have likewise either been mapped onto objects or lumped into a "unknown headers" array. if the message is unparsed, all you get is an text stream that you'll have to parse yourself. If you want to add, remove or modify headers while processing a message in an extension, you will have to read and parse your copy of the input stream (the message text) and write the resulting mesage to an output stream that's handed onwards to the next extension or to the infrastructure. In essence that means that if you had two or three ASMX-style SOAP extensions that implement security, addressing and routing functionality, you'd be parsing the message three times and serializing it three times just so that the infrastructure would parse it yet again. Not so good.
WSE
The Web Services Enhancements (WSE) have a simple, but very effective fix for that problem. The WSE team needed to use the ASMX extensibility point but found that if they'd build all their required extensions using the ASMX model, they'd run into that obvious performance problem. Therefore, WSE has its own pipeline and its own extensibility mechanism that plugs as one big extension into ASMX and when you write extensions (handlers) for WSE, you don't get a stream but an in-memory info-set in form of a SoapEnvelope (that is derived from System.Xml.XmlDocument and therefore a DOM). Parsing the XML text just once and have all processing steps work on a shared in-memory object-model seems optimal. Can it really get any better than "parse once" as WSE does it?
Indigo
When you look at the Indigo concept of Message (the Message class in the next milestone will be the same in spirit, similar in concept and different in detail and simpler as a result), you'll find that it doesn't contain a reference to an XmlDocument or some other DOM-like structure. The Indigo message contains a collection of headers (which in the M4 milestone also come in an "in-memory only" flavor) and a content object, which has, as its most important member, an XmlReader-typed Reader property.
When I learned about this design decision a while ago, I was a bit puzzled why that's so. It appeared clear to me that if you kept the message parsed in a DOM, you'd have a good solution if you want to hand the message down a chain of extensibility points, because you don't need to reparse. The magic sentence that woke me up was "We need to support streaming". And then it clicked.
Assume you want to receive a 1GB video stream over an Indigo TCP multicast or UDP connection (even if you think that's a silly idea - work with me here). Because Indigo will represent the message containing that video as an XML Infoset (mind that this doesn't imply that we're talking about base64-encoded content in an UTF-8 angle bracket document and therefore 2GB on the wire), we've got some problems if there was a DOM based solution. A DOM like XmlDocument is only ready for business when it has seen the end tag of its source stream. This is not so good for streams of that size, because you surely would want to see the video stream as it downloads and, if the video stream is a live broadcast, there may simply be no defined end: The message may have a virtually infinite size with the "end-tag" being expected just shortly before judgment day.
There's something philosophically interesting about a message relaying a 24*7*365 video stream where the binary content inside the message body starts with the current video broadcast bits as of the time the message is generated and then never ends. The message can indeed be treated as being well-formed XML because there is always a theoretical end to it. The end-tag just happens to be a couple of "bit-years" away.
Back to the message design: When Indigo gets its hands on a transport stream it layers a Message object over the raw bits available on the message using an XmlReader. Then it peeks into the message and parses soap:Envelope and everything inside soap:Header. The headers it finds go into the in-memory header collection. Once it sees soap:Body, Indigo stops and backs off. The result of this is a partially parsed in-memory message for which all headers are available in memory and the body of the message is left sitting in an XmlReader. When the XmlReader sits on top of a NetworkStream, we now have a construct where Indigo can already work on the message and its control information (headers) while the network socket is still open and the rest of the message is still arriving (or portions haven't even been sent by the other party).
Unless an infrastructure extension must touch the body (in-message body encryption or signature do indeed spoil the party here), Indigo can process the message, just ignore the body portion and hand it to the application endpoint for processing as-is. When the application endpoint reads the message through the XmlReader it therefore pulls the bits directly off the wire. Another variant of this, and the case where it really gets interesting, is that using this technique, arbitrary large data streams can be routed over multiple Indigo hops using virtualized WS-Addressing addressing where every intermediary server just forwards the bits to the next hop as they arrive. Combine this with publish and subscribe services and Indigo's broadcasting abilities and this is getting really sexy for all sorts of applications that need to traverse transport-level obstacles such as firewalls or where you simply can't use IP.
For business applications, this support for very large messages is not only very interesting but actually vital for a lot of applications. In our BizTalk workshops we've had quite a few customers who exchange catalogs for engineering parts with other parties. These catalogs easily exceed 1GB in size on the wire. If you want to expand those messages up into a DOM you've got a problem. Consequently, neither WSE nor ASMX nor BizTalk Server nor any other DOM based solution that isn't running on a well equipped 64-bit box can successfully handle such real-customer-scenario messages. Once messages support streaming, you have that sort of flexibility.
The problem that remains with XmlReader is that once you touch the body, things get a bit more complex than with a DOM representation. The XmlReader is a "read once" construct that usually can't be reset to its initial state. That is specifically true if the reader sits on top of a network stream and returns the translated bits as they arrive. Once you touch the message content is the infrastructure, the message is therefore "consumed" and can't be used for further processing. The good news is, though, that if you buffer the message content into a DOM, you can layer an XmlNodeReader over the DOM's document element and forward the message with that reader. If you only need to read parts of the message or if you don't want to use the DOM, you can layer a custom XML reader over a combination of your buffer data and the original XmlReader.
The Microsoft Developer Days 2004 in Den Haag (The Hague) were a great event. Not so much fun was going there (the train from Utrecht was split in two trains on the way and I ended up in Rotterdam instead of Den Haag at first) and getting back (the train from Venlo to Düsseldorf simply didn't go because of "technical difficulties" so I had to take a rather expensive cab home).
I've had lots of interesting discussions and the result of one was that I might be speaking at the SDGN's CttM conference. I'll definitely be back for the second run of the Architect's Forum in Zeewolde in March 29th.
De SDGN heft gezegt dat ik nu moet genoeg Nederlands lere omdat ik mij CttM presentatie in de Nederlandse taal kan doen, maar ik weet niet of ze bereid zijn om mij zovel tijd voor een presentatie te geve zo dat ik ook lang genoeg voor iede enkele woord kan zoeke. :)
My talk on Indigo apparently went well for the audience and one of my fellow RDs even said that he learned more about Indigo in my talk than at the PDC (that's because I consolidated the PDC slides and therefore have it "all at once"), but personally I was a bit unhappy with it. Didn't flow right. Two slides too much, one slide missing (I need to explain "Dialogs"). This will be fixed for the next stop in Oslo on Monday.
|