I updated my copyright notice at the bottom of the blog. The reason for that is that I found my blog to be replicated in full to another website in their design and with their copyright notice, without any backlinks to this blog and without having been asked for permission.
I may offend a few friends over this, but there are friendly rules of conduct which I follow and on which I must insist: Quote liberally, aggregate, combine, analyze, but do not republish large portions of content in unaltered form and don't quote without linking back to the source. Brief legal excursion: There are explicit rules about what's permitted in terms of quoting other's works in just about every copyright law. The German Copyright Act and related international treaties are applicable here. For instance, it's no problem to take this entire posting and quote it elsewhere as-is. It is a problem to simply publicly republish the whole rss-feed content elsewhere. Quoting and commenting is a good thing, plain theft of intellectual property isn't.
Thoughts from integrating with WSDK.
It shouldn't be very surprising that I am currently integrating my web services extensions with the WSDK with the goal of making that available shortly after WSDK ships. However, it's no as easy as I thought it would be, because the extensibility pipeline model that MS layered on top of the SoapExtensions requires a different approach for quite a few things that I've done. I could just leave my things the way they are and have them function side-by-side with the WSDK's pipeline (that works), but I want to take advantage of the pipeline's relative independence of the ASP.NET request/response cage, which I find increasingly limiting. Also, integrating Kerberos into the WSDK's WS-Security framework is a real challenge, because I may have to go all the way down into writing an algorithm wrapper based on the stuff in the System.Security.Cryptography namespace. However -- I really want to get this working and therefore I will ;)
A new terminology set.
While working on this, I have found that a lot of terminology that I am using here is massively confusing: "client", "server", "input", "output", "request", "response", "proxy", "stub", "endpoint", "router" and quite a few more terms are ambiguous and depend on your current point of view and on what you are just building.
I find it more helpful to focus on the message and look at it as something that starts empty, grows and shrinks (and grows and shrinks) as it goes through a unidirectional pipeline of message-part producers and message-part-consumers and whose processing ends when the message is empty again. Request/response is a special case where the message flow originates and ends at the same place. The pipeline elements (producers/consumers) are connected by local ("through the stack") and remote ("over the wire") connectors. A server-side "request handler" as it's currently known is simply a message-part consumer taking stuff from the body and sticking stuff back into the body.
Here's the terminology set I am currently playing around with:
- Message Producer: Acts on information from the system environment or state and possibly adds information to a message.
- Message Consumer: Acts on and removes information from the message and possibly alters the system environment or state.
- Service Acquisition Filter: A message producer that acquires system state to flow information about a service (eg. picks up a COM+ transaction context or the current security principal)
- Service Enlistment Filter: A message consumer that reads and service information from a message and enlists into a service or creates a new service (eg. picks up a TIP URI and creates a service domain or creates a security context from a serialized authentication package)
- Fork: In case message processing cannot proceed without additional information or in case a filter wants to split the message into separate flows, any pipeline can fork off new pipeline instances to request information or flow information to additional destinations. A fork is blocking if it can't proceed in its current state and otherwise non-blocking.
A proxy is a message producer. A stub is a message consumer. A logging sink is a message consumer. A filter that adds timestamp information is a message producer. A filter that reads timestamp information is a message consumer. A client-side authentication extension (sending path) is a service acquisition filter. A server-side authentication extension (receiving path) is a service enlistment filter. The inverse of a "service enlistment filter" is just another "service acquisition filter", which acquires the service state and possibly disassociates the action from the service -- reversely, the same relationship exists.
Anyone still with me?
[Disregard the first two comments showing up below. The comments feature of radio is confused with my blog]
Once System.Diagnostics.StackWalk and "__asm mov [__EBP], ebp;" become useful tools, you realize that you've exhausted an infrastructure's extensibility hooks. Sigh!
 COM was love
This isn't an Enterprise Services crusade. There's a lot of very useful stuff and I am pointing it out. That's all ;)
However, I think I should admit that I did indeed have a very long affair with the entire ES/COM+/MTS/COM family. {Is that a nasty thing to admit?}. The love went away, now we're just good friends.
October 1992. The first time COM got into customer's hands. I got this CD and I was like "man, that's nuts! Way too complicated!". The first shipment came with docs that dealt in very deep and great detail with IUnknown, structured storage and especially IMoniker. I didn't understand the moniker stuff until two years later. What I understood was AddRef()/Release() and QueryInterface(). An implementation of IUnknown became my default-base class for anything in November 1992. Only about 3 months later I understood how "outer IUnknowns" and aggregation worked. A lot of code-rewriting followed ;). It took about a year to realize that not *everything* should be a COM class. And so it went on and on....
And I just realize... there was no 10th birthday party;)
My friend Morten Abrahamsen from Norway, an exceptionally smart software architect with a very practical view on things and a lot of experience building large transactional systems, has written a comment very worth reading here in the blog that deserves to seen not only by the folks who accidentially click the comments link. Morten says: "Interesting how the struggle for singular performance clouds the overview of a flexible architecture that is resilient to change, versioning and implementation details.".
Hey, Greg, what are you doing with that method signature [1] when you're porting that app to another data provider? What if a future revision of that method wants to add that customer to the database asynchronously and does so by stuffing it into a transactional queue, first?
[1] public int AddCustomer(CustInfo custInfo, SqlTransaction trans)
Exactly. You don't need any of COM+ (which is really what to call it) unless you are trying to use the DTC. The rest is all done by the runtime already as you point out. [Sam Gentile's Radio Weblog]
No. Show me a COM+ feature except the non-transactional incarnation of the shared property manager (AppDomain statics may do that), which is exactly or even similarly replicated in the runtime. App-specific roles aren't, object pooling isn't, JIT activation isn't, QC aren't, process model (thread pool, app pool, pause, disable, recycling) isn't, event monitoring isn't, LCE aren't, memory gates aren't, secure RPC isn't, security context isn't and this list isn't complete. It's simply not right, Sam.
John Lam shares his thoughts on Enterprise Services and says that many features of ES are replicated throughout the framework and therefore he sees little need to use ES anymore except for distributed transactions. This was my exact position about a year ago. At the time, I posted a lengthy statement with almost the same arguments to the Microsoft Regional Regional Directors (non-public) mailing list and since then I had a lot of very valuable discussions with a lot of very smart folks who mapped out the differences between what's in the framework and what's in ES and who have helped me to understand that I simply wasn't right with what I was saying.
Let me go over what John highlights:
- Load balancing. Load balancing is a filter for work where the amount of work is either not predictable or can't be handled by a single system. You load balance as close to the "topmost" client as you can, handle parts of the load and reduce the need for load balancing downstream. If you have a website or web-service, you load balance the web-tier. Will you load balance the business logic tier? Possibly, but the load generated by one web-server towards its backend is typically sufficiently predictable to eliminate the need for component load balancing and rather make not-so-dynamic assignments of backend servers to a group of web servers. Component load balancing is only really good for when you can't load-balance the presentation tier (for instance if you have GUI clients) or if you have a huge spread in execution times for a single class (if you allow users to execute ad-hoc database queries).
- ASP.NET as an application host. Hosting your business logic there is good, if you are all stateless. If you need to keep and share expensive-to-acquire application state around (such as large caches) or if you need to guard a set of limited resources that are limited for your entire web-farm, hosting there has limitations. The most important limitation of hosting business logic there is security. In ASP.NET, everything happens within the security context of the external caller (or its delegate ASPNET) and that's a problematic thing. You will want to do certain things with elevated privileges in the context of a service account and LogonUser() isn't really what you want to do in that case.
- Roles. You can make your own user and role types in the .NET framework, but not many people do. You'd have to write your own admin tools, your own infrastructure and you'd have to provide a mapping to OS roles and users for infrastructure access. If you stick with OS roles (SAM or Active Directory groups, in essence) and use the PrincipalPermissionAttribute as a replacement for ES role-based security, you will lose a level of indirection. Instead of defining a role required to access a single method on a single class in a single application right there in that application, you will have to define that in Active Directory and have it replicate throughout your AD structure. There can be very many such roles.
- Object Pool: Object pooling is a good workaround to overcome limitations of OLE DB, but that's just one aspect. It's a generic semaphore for classes. It helps implementing write access to any resource with limited or no concurrency control (the art of handling FileStream.Lock() is often long forgotten), it'll help you pre-initialize and control access to things like 3270 terminal screen-scrapers with very limited permitted concurrent sessions or maybe interfaces to physical devices of which you only have one or four (like a metal-sheet press). Nothing you couldn't do without ES, but ... it's already there and the number of folks who don't want to spend the time implementing all the required infrastructure goo is substantial.
- Transactions: Even with a single resource manager, using ES for transactions is not a bad idea for complex systems. For the simplest case that a component method creates a transaction, does work and commits the transaction, using native database connections are a good thing and the fastest choice. If a component calls another component (which may call other components itself) and the transaction shall span those components, transaction management can easily get out of hand. You will have to pass the database connection and the transaction objects around (in the case of ADO.NET), you will have to negotiate who may commit or abort and you may have to collect votes on the outcome. Also, since components are black-boxes, you will not always know whether a component you want to roll into your transaction doesn't require a second resource manager -- in which case your native database transaction can't be used. (Update: check out Ingo Rammer's comments on the same topic)
Still, I consider John's remarks as very valid for a large number of web-applications. COM+ no longer is the automatic default for hosting business logic as it was when stuff was implemented in ASP and VB6 components instead of ASP.NET. However, ASP.NET hasn't become the automatic default for hosting all things related to a web application, either. Your mileage may vary.
Why you want to use Enterprise Services for your .NET application
Part 1: Introduction
Part 2: Basic Architectural Considerations and the benefits of Processes and Process Models
UI -> BusinessLogic -> DataAccess. This three layer model is the most common way to separate functional blocks of applications. It's clear, simple and very obvious. If you're writing a web application, all stuff on top of ASP, JSP or ASP.NET is your UI layer, whatever is called by that layer is your business logic layer and that, in itself is split into a "logic" and into a "data access" layer, which serves to make your data access code more resilient against changes in data access technology, database product choice or, in the simplest and most common case, schema changes in the underlying data store. If you are writing a GUI application, all GUI-related functionality is in the top-layer and you may be able to use the same business logic layer as you are using for a web application. Ideally, all user-interface type and technology specific aspects are handled in the UI layer, and business logic shall be as resilient against changes in the UI as it is against changes in the data layer. That's why we have those layers.
I call them "layers", not "tiers". In my world (not necessarily in everyone else's), a layer is a purely logical concept. Layering is about separating functionally different areas of code. To me, a tier is a solely a physical concept. It is about how code gets distributed in a runtime environment. A GUI application, which ends up being compiled into a single EXE may be built using multiple separate layers just as much as a server application, which potentially maps each layer on a distinct physical tier. More likely is, though, that stuff from two or more layers gets mapped into a tier or that two tiers handle one layer. Example: SQL Server stored procedures and components manipulating ADO.NET DataSets using ad-hoc SQL are all things that belong to an data access layer. Still they are physically deployed in two places: Inside SQL Server and in a process that is accessing SQL Server from the outside. I would call that two separate tiers, but one layer. A "single EXE" GUI application has many layers, but possibly only one tier (if it does, for instance use the JetDB engine mapped into its own process space).
That's all well-known and very obvious to mostly everybody as long as the "UI" role is very obvious (Web front or GUI) and the "Data Store" role is very obvious (some RDBMS). Most commonly, (sub-)systems which feature this type of layers are running in "reactionary mode". They are triggered by some user activity and run one or more (potentially parallel) sequences of activities in response to the user activity. Not seldomly, architectural confusion begins whenever a system shall perform autonomous actions (for instance based on timers) or when the trigger for an activity is not a user, but some other binary lifeform. Where does that fit into the layering picture? Does it fit at all? Also, if my business code needs to invoke a remote system through a web service or needs to submit a document to a remote site using an infrastructure like BizTalk or if it simply wants to send an email via SMTP. Where does that go?
In my world, the acronym "UI" doesn't mean "user-interface", it means "use-case-interface". Everything that triggers any activity in the business logic layer is a "UI". A Web Services is a UI, a BizTalk Server application integration component (AIC) is a UI, a Windows Service process is a UI. The business code doesn't really care whether the current method call was originally triggered by a human being clicking anywhere on a remote screen. All such UIs can share a lot of code. Indeed, "business logic" is all code that is UI-agnostic in this expanded scenario.
The "Data Access" layer isn't just for databases. All code that accesses any functionality outside your own application and which is triggered by activities that root in your own application belongs there. If you call a remote web service or a remote application that's not under your own immediate control, you need to make your business code resilient against changes in those external applications. If you send a Word document attached to an Email via SMTP now, you may want to send an PDF document via other ways tomorrow. The fact that the information must be sent doesn't change, formatting and ways of sending does. So, I like to speak of "Infrastructure Access Layer" rather than "Data Access" layer to limit confusion.
What we're getting out of this are three separate layers of code. "Use-Case Interface", "Business Logic" and "Infrastructure Access". It's a good way to organize interfaces and code and works very well for working in large teams. What we're not getting out of this a consistent and reliable mapping to a runtime environment. Business logic will execute in the process space of ASP.NET, a BizTalk AIC, a Windows Service or a desktop GUI application. All these process spaces are very different. The GUI EXE runs reliably as long as the user Alt-F4's it or until the machine becomes unresponsive courtesy of Windows Exploder. BizTalk will load and unload an AIC (and hence your entire layered model) for the duration of a single action. ASP.NET will load your code, but it'll sometimes recycle the process "suddenly" for various good reasons. A Windows service has a very predicable execution profile (starts/stops at boot/shutdown), but by itself it doesn't have a concept of communication with the outside world -- you'll have to make it an RPC or Remoting server or a Message Queue listener yourself and that will involve creating and maintaining worker threads, etc.
If you want to write applications that are dealing with data efficiently and truly scale, you will want to cache large parts of those 80% of all tables in your data model that hold static or near-static data in memory. You will want to keep pools of infrastructure objects ready and initialized. You will want to have pre-activated and smart "gatekeepers" that guard access to limited or expensive external resources such as 3270 terminal sessions, remote web services with low bandwidth, etc. What you need is a predictable execution environment, which allows you coordination of such access to limited resources, which will allow you to keep caches alive and current and which provides you with a security boundary that will allow authorization for accessing services and security-identity switches that can access services with elevated privileges for such authorized users. What you want is to go "out of process".
"Going out of process" and hosting your business logic in a decicated environment is not "necessary evil", it's a carefully chosen and intended feature of your architecture. Enterprise Services/COM+ (and J2EE application servers) provide you with such a predictable hosting environment for your "Business Logic" and "Infrastructure Layer" components. "Going out of process" means that you will isolate your business code from unique behavior of your "Use-Case Interface" hosting environment.
What you get is a process with a well-defined process model. It'll create and manage thread pools, it'll manage external access, it'll provide you with a way to access this functionality from other processes. Using Enterprise Services applications "in process" is a special case for whenever you only deal with a single "Use-case Interface" and you are ready to deal with possible restrictions its process model imposes for your business logic. Hosting business logic "out of process" is the default.
That's why you want an application server environment. Enterprise Services/COM+ and most J2EE application servers provide such an environment. These principles count on the server, but also on the desktop. Paying the price of cross-process marshaling is not something you are forced to do under torture, it's something which you'll do because you'll get something for that price. Power = Work /Time.
Today's favorite Enterprise Services attribute: [assembly:ApplicationActivation(ActivationOption.Server)]
Next installment: Part 3: Management of expensive and limited resources
Why you want to use Enterprise Services for your .NET application
Part 1: Introduction
Yesterday I did a 4.5 hour talk about the relevance and basics of Enterprise Services here at TornadoCamp.NET. In our audience we have about 90% developers who have been using mostly VB6/VB5 up until now and more than half are writing "classic" client/server applications with (very) fat clients and the only server-side actions happening inside SQL Server, Oracle, Sybase by way of stored procedures. What I've found here is consistent with what I find at our other workshops and very many other events where I speak: Only very few developers really ever used COM+ or MTS for anything but server-side transaction handling and the majority didn't even look at Enterprise Services/COM+/MTS ever, at all.
Why that is the case is easily explained and there are two primary reasons:
(a) Visual Basic 6 (and previous versions) is the most popular language for writing business applications on Windows, at least with our customers and the people I usually talk to at conferences and events. COM+ provides quite a few very useful features, which either can't be used from within the VB's "STA ghetto" due to its inability to produce thread-safe code (like ObjectPooling) or which are very difficult to deploy without rather complex installation scripts (like "loosely coupled events") .
(b) The main reason is a different one: COM+ provides the implementation of a lot of common architectural patterns and solutions to very typical functional challenges. If I either don't understand these patterns or, more often, don't see an obvious mapping of such a functional challenge that I find in my project to a feature provided by COM+, I simply won't use it. The dilemma: If you don't really know what's in COM+ feature bag, you won't be able to find out why you'd ever want to consider using it. If you have no interest in COM+, you will not buy a book on it. For most developers, all feature areas of COM+ beyond "Transaction.Required" therefore remain in the dark.
So, instead of blogging random Enterprise Services features out-of-context (such as CoRegisterSurrogateEx), I will try to illustrate the "why" and use-cases for several (best:all) Enterprise Services/COM+ services in a very compact, blog compatible form, which will hopefully create a context for the other obscure things I am typically writing about and will allow more people to see why this stuff is very relevant for their apps.
Next installment: Part 2: Basic Architectural Considerations and the benefits of Processes and Process Models
I start to wonder whether it may make sense to do a conference or tour really dedicated to Enterprise Services in XP/.NET Server.
CoRegisterSurrogateEx continued: Tomas comments here in the blog: "Let me see if I get one thing straight: You say we should _enable_ the application when the host starts up and _disable_ it when it shuts down, right?" --- Yes, exactly.
"While we're on it, I think in many cases it would just be enough (and easier, up to a point), to simply make the application components fail activation if they're not running inside your custom surrogate.... what do you think?" -- That would be a good safeguard to avoid failures causes by an incorrect "activation environment". Still, throwing exceptions at activation time is a good indicator that something is wrong, but it won't help you spinning up a working host. Also, once a "wong host" is running, you'll have problems getting the custom host to run properly. Therefore, disabling or "run as NT services" is the better idea
Summary: You can host COM+ server applications in a custom host process, with all features. In fact, the host process doesn't need to be fully dedicated to hosting the COM+ app. You just need to spin up one thread for CoRegisterSurrogateEx (which in turn will spawn all thread required for the COM+ app), while other threads can do other things.
|