I am not a “smart client” programmer and probably not even a smart client programmer and this trick has probably been around for ages, but …

For someone who’s been doing WPARAM and LPARAM acrobatics for years and still vividly recalls what (not necessarily good) you can do with WM NCPAINT and WM NCMOUSEMOVE (all that before I discovered the blessings of the server-side), it’s pretty annoying that Windows Forms doesn’t bubble events – mouse events specifically. It is actually hard to believe that that wouldn’t work. But I’ve read somewhere that bubbling events is “new in Whidbey”, so it is probably not my ignorance. Anyways … include the following snippet in your form (add MouseDown, MouseUp, … variants at your leisure), bind the respective events of all labels, panels and all the other “dead stuff” to this very handler (yes, all the controls share that handler) and that’ll have the events bubble up to your form in case you need them. I am just implementing custom resizing and repositioning for some user controls in a little tool and that’s how I got trapped into this. Voilá. Keep it.

protected void BubbleMouseMove(object sender, System.Windows.Forms.MouseEventArgs e)
{
      Point pt = this.PointToClient(((Control)sender).PointToScreen(new Point(e.X,e.Y)));
      MouseEventArgs me = new MouseEventArgs(e.Button,e.Clicks,pt.X,pt.Y,e.Delta);
      OnMouseMove(sender,me);
}

Ralf Westphal responded to this and there are really just two sentences that I’d like to pick out from Ralf’s response because that allows me to go a quite a bit deeper into the data services idea and might help to further clarify what I understand as a service oriented approach to data and resource management. Ralf says: There is no necessity to put data access into a service and deploy it pretty far away from its clients. Sometimes is might make sense, sometimes it doesn’t.

I like patterns that eliminate that sort of doubt and which allow one to say “data services always make sense”.

Co-locating data acquisition and storage with business rules inside a service makes absolute sense if all accessed data can be assumed to be co-located on the same store and has similar characteristics with regards to the timely accuracy the data must have. In all other cases, it’s very beneficial to move data access into a separate, autonomous data service and as I’ll explain here, the design can be made so flexible that the data service consumer won’t even notice radical architectural changes to how data is stored. I will show three quite large scenarios to help illustrating what I mean: A federated warehouse system, a partitioned customer data storage system and a master/replica catalog system.

The central question that I want to answer is: Why would you want delegate data acquisition and storage to dedicated services? The short answer is: Because data doesn’t always live in a single place and not all data is alike.

The Warehouse

The Warehouse Inventory Service (WIS) holds data about all the goods/items that are stored in warehouse. It’s a data service in the sense that it manages the records (quantity in stock, reorder levels, items on back order) for the individual goods, performs some simplistic accounting-like work to allocate pools of items to orders, but it doesn’t really contain any sophisticated business rules. The services implementing the supply order process and the order fulfillment process for customer orders implement such business rules – the warehouse service just keeps data records.

The public interface [“Explicit Boundary” SOA tenet] for this service is governed by one (or a set of) WSDL portType(s), which define(s) a set of actions and message exchanges that the service implements and understands [“Shared Contract” SOA tenet]. Complementary is a deployment-dependent policy definition for the service, which describes several assertions about the Security and QoS requirements the service makes [“Policy” SOA tenet].

The WIS controls its own, isolated store over which it has exclusive control and the only way that others can get at the content of that data store is through actions available on the public interface of the service [“Autonomy” SOA tenet].

The Customer Data Storage

This scenario for the Customer Data Storage Service (CDS) starts as simple as the Warehouse Inventory scenario and with a single service. The design principles are the same.

Now let’s assume we’re running a quite sophisticated e-commerce site where customers can customize quite a few aspects of the site, can store and reload shopping carts, make personal annotations on items, and can review their own order history. Let’s also assume that we’re pretty aggressively tracking what they look at, what their search keywords are and also what items they put into any shopping cart so that we can show them a very personalized selection of goods that precisely matches their interest profile. Let’s say that all-in-all, we need to have storage space of about 2Mbytes for the cumulative profile/tracking data of each customer. And we happen to have 2 million customers. Even in the Gigabyte age, ~4mln Mbytes (4TB) is quite a bit of data payload to manage in a read/write access database that should be reasonably responsive.

So, the solution is to partition the customer data across an array of smaller (cheaper!) machines that each holds a bucket of customer records. With that we’re also eliminating the co-location assumption.

As in the warehouse case, we are putting a dispatcher service implementing the very same CDS portType on top of the partitioned data service array and therefore hide the storage strategy re-architecture from the service consumers entirely. With this application-level partitioning strategy (and a set of auxiliary service to manage partitions that I am not discussing here), we could scale this up to 2 billion customers and still have an appropriate architecture. Mind that we can have any number of dispatcher instances as long as they implement the same rules for how to route requests to partitions. Strategies for this are a direct partition reference in the customer identifier or a locator service sitting on a customer/machine lookup dictionary.

Now you might say “my database engine does this for me”. Yes, so-called “shared-nothing” clustering techniques do exist on the database level for a while now, but the following addition to the scenario mandates putting more logic into the dispatching and allocation service than – for instance – SQL Server’s “distributed partitioned views” are ready to deal with.

What I am adding to the picture is the European Union’s Data Privacy Directive. Very simplified, the EU directives and regulations it is illegal to permanently store personal data of EU citizens outside EU territory, unless the storage operator and the legislation governing the operator complies with the respective “Safe Harbor” regulations spelled out in these EU rules.

So let’s say we’re a tiny little bit evil and want to treat EU data according to EU rules, but be more “relaxed” about data privacy for the rest of the world. Hence, we permanently store all EU customer data in a data center near Dublin, Ireland and the data for the rest of the world in a data center in Dallas, TX (not making any implications here).

In that case, we’re adding yet another service on top of the unaltered partitioning architecture that implements the same CDS contract and which internally implements the appropriate data routing and service access rules. Those rules which will most likely be based on some location code embedded in the customer identifier (“E1223344” vs. “U1223344”). Based on these rules, requests are dispatched to the right data center. To improve performance and avoid having to data travel along the complete path repeatedly or in small chunks during an interactive session with the customer (customer is logged into the web site), the dispatcher service might choose to have a temporary, non-permanent cache for customer data that is filled with a single request and allows quicker and repeat access to customer data. Changes to the customer’s data that result from the interactive session can later be replicated out to the remote permanent storage.

The Catalog System

Same picture to boot with and the same rules: Here we have a simple service fronting a catalog database. If you have millions of catalog items with customer reviews, pictures, audio and/or video clips, you might chose to partition this just like we did with the customer data.

If you have different catalogs depending on the markets you are selling into (for instance German-language books for Austria, Switzerland and Germany), you might want to partition by location just as in the warehouse scenario.

One thing that’s very special about catalog data is that very much of it rarely ever changes. Reviews are added, media might be added, but except for corrections, the title, author, ISBN number and content summary for a book really doesn’t ever change as long as the book is kept in the catalog. Such data is essentially “insert once, change never”. It’s read-only for all practical purposes.

What’s wonderful about read-only data is that you can replicate it, cache it, move it close to the consumer and pre-index it. You’re expecting that a lot of people will search for items with “Christmas” in the item description come November? Instead of running a full text search every time, run that query once, save the result set in an extra table and have the stored procedure running the “QueryByItemDescription” activity simply return the entire table if it sees that keyword. Read-only data is optimization heaven.

Also, for catalog data, timeliness is not a great concern. If a customer review or a correction isn’t immediately reflected on the presentation surface, but only 30 minutes or 3 hours after is has been added to the master catalog, it doesn’t do any harm as long as the individual adding such information is sufficiently informed of such a delay.

So what we can do to with the catalog is to periodically (every few hours or even just twice a week) consolidate, pre-index and then propagate the master catalog data to distributed read-only replicas. The data services fronting the replicas will satisfy all read operations from the local store and will delegate all write operations directly (passthrough) to the master catalog service. They might choose to update their local replica to reflect those changes immediately, but that would preclude editorial or validation rules that might be enforced by the master catalog service.

So there you have it. What I’ve described here is the net effect of sticking to SOA rules.

·         Shared Contract: Any number of services can implement the same contract (although the concrete implementation, purpose and hence their type differ). Layering contract-compatible services with gradually increasing levels of abstractions and refining rules over existing services creates very clear and simple designs that help you scale and distribute data very well

·         Explicit Boundaries: Forbidding foreign access or even knowledge about service internals allows radical changes inside and “underneath” services.

·         Autonomy allows for data partitioning and data access optimization and avoids “tight coupling in the backend”.

·         Policy: Separating out policy from the service/message contract allows flexible deployment of the compatible services across a variety of security and trust scenarios and also allows for dynamic adaptation to “far” or “near” communications paths by mandating certain QoS properties such as reliable messaging.

Service-Orientation is most useful if you don’t consider it as just another technique or tool, but embrace it as a paradigm. And very little of this thinking has to do with SOAP or XML. SOAP and XML are indeed just tools.

Rebecca Dias from Microsoft asked us to do a bit of work for her team and write a demo app for TechEd 2004. As things happen and being the serious German engineers we are, it just turned out to be a little too serious, little too big to be useful as a “and now here’s a bit of code!” demo app for TechEd (U.S.).

What we’ve built is a very serious service-oriented application and your feedback will contribute to the final decision about how Microsoft is going to make the application and code available to you. What’s already clear is that I will do a TechEd Europe talk that will cover the most important architecture and technology choices made for the application. Unfortunately the decision to have such a talk came too late to squeeze it into the TechEd U.S. agenda. Come to Amsterdam, TechEd Europe isn’t sold out, yet.

Comment on Rebecca’s blog entry here and let her know whether you rather like little samples like Duwamish or a full-blown SOA system that you can stick your head into for a week.

[This might be more a “note to self” than anything else and might not be immediately clear. If this one goes over your head on the first pass – read it once more ]

Fellow German RD Ralf Westphal is figuring out layers and data access. The “onion” he has in a recent article on his blog resembles the notation that Steve Swartz and I introduced for the Scalable Applications Tour 2003.  (See picture; get the full layers deck from Microsoft’s download site if you don’t have it already)

What Ralf describes with his “high level resource access” public interface encapsulation is in fact a “data service” as per our definition. To boot, we consider literally every unit in a program (function, class, module, service, application, system) as having three layers: the outermost layer is the publicly accessible interface, the inner layer is the hidden internal implementation and the innermost layer hides and abstracts services and resource access. The concrete implementation of this model depends on the type of unit you are dealing with. A class has public methods as public interface, protected/private methods as internal implementation and uses “new” or a factory indirection to construct references to its resource providers. A SQL database has stored procedures and views as public interface, tables and indexes as internal implementation and the resource access is the database engine itself. It goes much further than that, but I don’t want to get carried away here.

A data service is a layered unit that specializes in acquiring, storing, caching or otherwise dealing with data as appropriate to a certain scope of data items. By autonomy rules, data services do not only hide the data access methods, but also any of these characteristics. The service consumer can walk up to a data service and make a call to acquire some data and it is the data service’s responsibility to decide how that task is best fulfilled. Data might be returned from a cache, aggregated from a set of downstream services or directly acquired from a resource. Delegating resource access to autonomous services instead of “just” encapsulating it with a layer abstraction allows for several implementations of the same data service contract. One of the alternate implementations might live close to the master data copy, another might be sitting on a replica with remote update capability and yet another one may implement data partitioning across a cluster of storage services. Which variant of such a choice of data services is used for a specific environment then becomes a matter of the deployment-time wiring of the system.

Data services are the resource access layer of the “onion” model on the next higher level of abstraction. The public interface consists of presentation services (which render external data presentations of all sorts, not only human interaction), the internal implementation is made up of business services that implement the core of the application and the resource access are said data services. On the next higher level of abstraction, presentation services may very well play the role of data services to other services. And so it all repeats.

Now … Ralf says he thinks that the abstraction model works wonderfully for pulling chunks of data from underlying layers, but he’s very concerned about streaming data and large data sets – and uses reporting as a concrete example.

Now, I consider data consolidation (which reporting is) an inherent functionality of the data store technology and hence I am not at all agreeing with any part of the “read millions of records into Crystal Reports” story. A reporting rendering tool shall get pre-consolidated, pre-calculated data and turn that into a funky document; it should not consolidate data. Also, Ralf’s proposed delivery of data to a reporting engine in chunks doesn’t avoid that you’ll likely end up having to co-locate all received data into memory or onto disk to actually run the consolidation and reporting job --- in which case you end up where you started. But that’s not the point here.

Ralf says that for very large amounts of data or data streams, pull must change to push and the resource access layer must spoon-feed the business implementation (reporting service in his case) chunks of data at a time. Yes! Right on!

What Ralf leaves a bit in the fog is really how the reporting engine learns of a new reporting job, where and how results of the engine are being delivered and how he plans to deal with concurrency. Unfortunately, Ralf doesn’t mention context and how it is established and also doesn’t loop his solution back to the layering model he found. Also, the reporting service he’s describing doesn’t seem very flexible as it cannot perform autonomous data acquisition, but is absolutely dependent on being fed by the app – which might create an undesirable tightly coupled dependency between the feeder and a concrete report target.

The reporting service shall be autonomous and must be able to do its own data acquisition. It must be able to “pull” in the sense that it must be able to proactively initiate requests to data providers. At the same time, Ralf is right that the request result should be pushed to the reporting service, especially if the result set is very large.

Is that a contradiction? Does that require a different architecture? I’d say that we can’t allow very large data sets to break the fundamental layering model or that we should have to rethink the overall architectural structure in their presence. What’s needed is simply a message/call exchange pattern between the reporting service and the underlying data service that is not request/response, but duplex and which allows the callee to incrementally bubble up results to the caller. Duplex is the service-oriented equivalent of a callback interface with the difference that it’s not based on a (marshaled) interface pointer but rather on a more abstract context or correlation identifier (which might or might not be a session cookie). The requestor invokes the data service and provides a “reply-to” endpoint reference referencing itself (wsa:ReplyTo/wsa:Address), containing a correlation cookie identifying the originator context (wsa:ReplyTo/wsa:ReferenceProperties), and identifying an implemented port-type (wsa:ReplyTo/wsa:PortType) for which the data service knows how to feed back results. The port-type definition is essential, because the data service might know quite a few different port-types it can feed data to – in the given case it might be a port-type defined and exposed by the reporting service. [WS-Addressing is Total Goodness™]. What’s noteworthy regarding the mapping of duplex communication to the presented layering model is that the request originates from within the resource access layer, but the results for the requests are always delivered at the public interface.

The second fundamental difference to callbacks is that the request and all replies are typically delivered as one-way messages and hence doesn’t block any resources (threads) on the respective caller’s end.

For chunked data delivery, the callee makes repeated calls/sends messages to the “reply-to” destination and sends an appropriate termination message or makes a termination call when the request has been fulfilled. For streaming data delivery, the callee opens a streaming channel to the “reply-to” destination (something like a plain socket, TCP/DIME or – in the future -- Indigo’s TCP streaming transport) and just pumps a very long, continuous message.

Bottom line: Sometimes pull is good, sometimes push is good and duplex fits it all back into a consistent model.

People often ask me what I’ve done before Bart, Achim and I started newtelligence together with Jörg. So where do we come from? Typically, we have given somewhat “foggy” answers to those kinds of questions, but Achim and I talked about that yesterday and we’ve started to ask ourselves “why we do that”?

In fact, Achim, Bart and I had been working together for a long time before we started newtelligence. We used to work for a banking software company called ABIT Software GmbH, which then merged with two other sibling companies by the same owners to form today’s ABIT AG. We’ve only reluctantly communicated that fact publicly, because the formation of our company didn’t really get much applause from our former employer – quite the contrary was true and hence we’ve been quite cautious.

For us it was always quite frustrating that ABIT was sitting on heaps on very cool technology that my colleagues and I developed over the years (including patented things) and never chose to capitalize on the technology itself. Here are some randomly selected milestones:

We had our own SOAP 0.9 stack running in 1999, which was part of a framework that had a working and fully transparent object-relational mapping system based on COM along with an abstract, XML-based UI description language (people call those things XUL or XAML nowadays).

In 1998 we forced (with some help of our customer’s wallet) IBM into a 6 months avalanche of weekly patches for the database engine and client software that turned SQL/400 (the SQL processor for DB/400 on AS/400) from a not-quite-to-perfect product into a rather acceptable SQL database.

In 1996 we fixed well over 500 bugs and added tons of features to Borland’s OWL for OS/2 with which we must have had the pretty unique framework setup where cross-platform Windows 3.x, Windows NT and OS/2 development actually functioned on top of that shared class library.

In 1994 we already had what could be considered as the precursor to a service-oriented architecture in with collaborating, (mostly) autonomous services. The framework underlying that architecture had an ODBC C++ class library well over 6 months before Microsoft came out with their first MFC wrapper for it, had an MVC model based centered around the idea of “value-holders” that we borrowed from SmallTalk and which spoke, amongst other things, a text-validation protocol that allowed us to have a single “TextBox” control could be bound against arbitrary value holders that would handle all the text-input syntax rules as per their data type (or subtype). All of this was fully based on the nascent COM model which was then still buried in three documentation pages of OLE 2.0. I didn’t care much about linking and embedding (although I wrote my own in-place editing client from scratch), but I cared a lot about IUnknown as soon as I got my hands on it in late 1993. And all applications (and services) built on that framework supported thorough OLE Automation with Visual Basic 3.0 to a degree that you could fill out any form and press any button programmatically – a functionality that was vital for the application’s workflow engine.

And of course, during all that time, we were actively involved in project and product development for huge financial applications with literally millions of lines of production code.

None of the technology work (except the final products) was ever shared or available to anyone for licensing.  We were at a solutions company that supported great visions internally, but never figured out that the technology would be a value by itself.

newtelligence AG exists because of that pain. Years back, we’ve already designed and implemented variations of many of the technologies that are now state of the art or (in the case of XAML) not even shipping yet. At the same time, we continue to develop our vision and that’s how we can stay on top of things. So it’s really not that we’re not learning like crazy and go through constant paradigm shifts – we’re lucky that we can accumulate knowledge on top of the vast experience that we have and adjust and modernize our thinking. However, what’s different now is that we can share the essence of what we figure out with the world. That’s a fantastic thing if you’ve spent most of your professional life “hidden and locked away” and were unable to share things with peers.

So every time you’ll see a “Flashback” title here on my blog, I’ll dig into my recollection and try to remember some of the architectural thinking we had back in those times. We’ve made some horrible mistakes and had some exuberant and not necessarily wise ideas (such as the belief that persistent object-identity and object-relational mapping are good things); but we also had quite a few really bright innovative ideas. The things that really bring you forward are the grand successes and the most devastating defeats. We’ve got plenty of those under our belt and even though some of these insights date back 10 years, they are surprisingly current and the “lessons learned” very much apply to the current technology stacks and architectural patterns.

So – if you’ve ever thought that we’re “all theory” authors and “sample app” developers – nothing is further away from the truth. Also: Although I fill an “architecture consultant” role more than anything else now, I probably write more code on a monthly basis than some full-time application developers – what’s finally surfacing in talks and workshops is usually just the tip of that iceberg and often very much simplified to explain the essence of what we find.