Sample code from Norway

Here's the zipped up sample code that I've been showing during my Metadata talk at the VS.NET launch event in Oslo. It's demonstrating yet another use of the constraint attributes that I've written quite a while ago. The idea is simple. When you use restriction facets on types defined in XSD and map them into a programming model, most of the facets get crippled away. They get totally lost when XSD.EXE generates classes and only length constraints survive DataSet code generation. Now, if you use the code in this archive, you can retrofit your classes with the appropriate facets like this:

 [System.Xml.Serialization.XmlTypeAttribute(
    Namespace="urn:schemas-newtelligence-com:
    transactionsamples:customerdata:v1")]
 public class customerType 
 {
  [Match(@"\p{L}[\p{L}\p{P}0-9\s]+"),MaxLength(80)]
  public string FirstName;
        
  [Match(@"\p{L}[\p{L}\p{P}0-9\s]+"),MaxLength(80)]
  public string LastName;
        
  [Match(@"[0-9\+][0-9\-\(\)\s]*"),MaxLength(26)]
  public string HomePhone;
        
  [Match(@"[0-9\+][0-9\-\(\)\s]*"),MaxLength(26)]
  public string BusinessPhone;
        
  [Match(@"[0-9\+][0-9\-\(\)\s]*"),MaxLength(26)]
  public string MobilePhone;
 }

.. and then you simply run an entire object graph through the validation function provided by the ConstraintValidator class like this:

ConstraintValidator cv = new ConstraintValidator();
cv.Validate( customers );

If validation fails, you get an exception with a detailed description of the problem. The net result is that all validation happens on the object graph and not using a serialization step and a run through the XmlValidatingReader. That saves time and allows you to keep promises in regards to XSD based message contracts even if your data will never be serialized into angle brackets. XSD rocks. Consider queued components. Voilá.

Update: I got an email comment detailing an (obviously pretty stupid) bug that I have in the validation function. I'll take a look at it and will post an update shortly. Fixed.

Norwegian Visual Studio .NET 2003 launch event

I am on Norway today and I am sitting here blogging between two talks. "Say what you will say and listen to what you will listen to" was the first and now it's about Enterprise Services. It's going to take a day or two to wrap up the samples.

The final version of Steve's and my slides from the "Scalable Applications" tour is now available at www.thearchitectexchange.com. -- including all changes and tweaks that we put in during the tour. We're very happy with that content as it is now. Requests for re-runs at an event location near your house are accepted here ;)

Roy Osherove writes about typed DataSets and appropriate use. I would take it even a bit further.

If you look at them closely, DataSets are nothing more than a relational programming model for the XML Infoset. The DOM is a hierarchical programming model for the Infoset. XML Serialization is an object-typed programming model for the Infoset. Typed DataSets is a schema-bound programming model for Infoset and with that its really a relational programming model to PSVI. Still, it's a programming model for Infoset and I would argue that sharing programming models (or forcing programming models upon others) across boundaries isn't a good thing. Sharing serialized Infosets (one way to do so happens to be called "XML 1.0") is a Good Thing.

Update: Read the comments.

In the presence of significant latency or significant work to guard boundaries, throughput (read: appropriate architecture) matters, not performance (read: spending a week to shave off 50 CPU cycles).  

Don asks what I think about mixing code and data "at storage" inside a database. I simply call that a "data service". I'll give you the long answer:

The "layers and tiers" story that we presented at our tour is very simple. Everything you can talk to has layers: a public interface, an internal implementation and services and resource access hidden inside the internal implementation. It starts as simple as this:

public class MyService : IMyService
{
    // from IMyService 
    public int MyMethod(SomeData data) 
    { 
        return InternalMyMethod(); 
    }

    protected int InternalMyMethod(SomeData data)
    {
         MyBackend backend = new MyBackend();
         // do something with data
         backend.SomeMethod(data);
         return 0;
    }
}

I can add any number of variations of similar interfaces to that class without having to rewrite my internal implementation allowing for flexible binding of public interfaces to my implementation. Whenever I go outside the class I am using an external service or resource that I am often rather tightly bound to. I can go through a factory, but in the end I need to call a method or create a certain message. This basic principle remains true as you look at code from higher and higher up:

public class MyWebService : WebService
{
    [WebMethod]
    public int MyWebMethod(SomeData data) 
    { 
        return InternalMyWebMethod(data); 
    }
   
    [WebMethod]
    public int MyLiberalWebMethod(XmlElement any) 
    { 
        // try to map XML to SomeData here
        return InternalMyWebMethod(data); 
    }

    protected int InternalMyWebMethod(SomeData data)
    {
         // the "service access" layer is the proxy-portion of MyService().
         MyService backend = new MyService();
         // do something with data
         backend.MyMethod(data);
         return 0;
    }
}

Now we have a web service that uses the above class as its backend service. (When you leave the triplet of layers you are hitting a tier boundary, by the way). MyWebService, implementing an endpoint appropriate for "far" scenarios, as a whole is the public interface for our internal implementation MyService.

[Note: "Far" is not only strictly geographical, but also related to platform, trust, organization and other hurdles]

public class MyRPCService : ServicedComponent
{
    public int MyRPCMethod(SomeData data) 
    { 
        return InternalMyRPCMethod(data); 
    }
    
    protected int InternalMyRPCMethod(SomeData data)
    {
         MyService backend = new MyService();
         // do something with data
         backend.MyMethod(data);
         return 0;
    }
}

If can have any number of public interfaces for my internal implementation. Now I also have an RPC service with appropriate infrastructure for "near" scenarios (ServicedComponent) for the same internal implementation. Any service that deals with data can serve as an external service or resource manager for any other service. Layers are a recursive thing.

Any service provides appropriate functionality for its context. A web service front deals with all matters that have to do with mapping data from and to XML representations of data within the limits of its contract and leaves all other work to a backend service that serves as internal implementation. Another web service, possibly using the same backend is again scoped in functionality by its contract. A web form application deals with rendering data into HTML and handling interactive events and, again, leaves the actual work to its backend. There is very few appropriate code that might be useful in multiple contexts.

Now back to Don's question:

Sure it's appropriate to have data and code mixed in the database. The public interface implementation tools for a database are called "stored procedure" and "view". The internal implementation tools are called "table", "index" and "view". The service and resource access interface is called SQL. A database is just another layered service. When I have a proper public interface to submit my data to or initiate operations on and if the internal implementation is appropriately hidden from me, it doesn't matter how the database is implemented. The database gods can throw indexes and tables around as they wish and they can make it look for me as if nothing ever changed -- if it is an OODBMS that's just fine, too. The OODB "objects" in the database just can't "leave the database" and can't implement functionality that is not immediately appropriate for the context they're in. Storing instances of a class that has methods knowing how to render its data onto spinning cube using Direct3D isn't appropriate; storing an instance of a class that has methods that implement functions to transition between two consecutive consistent states are appropriate. Any use of these methods outside the database is inappropriate again.

Of course, what one wants to do in the bigger picture is to have a proper data service on the client side, whose internal implementation deals with client caching and techniques to eliminate/limit locking as appropriate the the character of data. With that, a "data service" consists of two sub-services. One service lives in the database, one service lives on the client and they have a tier boundary between them. Still, from the outside, the "data service" is a black box with a single public interface (the client's) that serves as an  implementation tool for the services and resource access for layers higher up.

Putting code and data into a single thing is a bad idea for distributed systems. Behavior is bound to the role that data plays in a functional scope. Presentation is a scope, calculation, combination, aggregation and validation are scopes, services and resource access is a scope. Having all code for all scopes on a single thing is bad. "Technology choice" causes inability to share implementation across organization boundaries. Mobile devices and web services are both like the "big bang": things get further and further away from each other (in terms of geography, trust and organizational association), causing more inability to share implementation and causing behavior that's bound to data and is located at central places to be the mother of all bottlenecks and plenty of other problems for scenarios where the boundary between tiers may ever have such "far" characteristics. I leave the conclusions to the reader, but here's a hint: MarshalByRefObject

First drop of samples from scalable apps tour.

Here are the samples that we ran (for the purpose of illustrating some points, not really to drill into these specific implementations) at the events in the past two weeks. These are not demos that illustrate scalability per-se. We couldn't carry enough laptops with us to illustrate scalability for real. These demos deal with transactions, layers and performance tuning. (Reminder: Performance has little to do with throughput -- think about all the time people spend in traffic jams with their Porsche 911 running in idle).

newtelligence SDK for Enterprise Services 2.0 RC1

The newtelligence SDK for Enterprise Services is a set of classes and infrastructure extensions for the System.EnterpriseServices programming model to Windows Enterprise Services (COM+) in the Microsoft Windows .NET Framework. The extensions are designed to work with Windows Server 2003, Windows XP and Windows 2000.

The tools contained in this kit provide the following enhancements for Enterprise Services.

• Custom interception extensibility for managed components (ServicedComponents) hosted in COM+ 1.0 and COM+ 1.5.
      (AspectServicedComponent class and utilities)
• Windows installer support
      (EnterpriseServicesApplicationInstaller class)
• Shared, context-scoped state using custom context properties
      (ContextUtilEx class)
• The ability to create serviced components in non-default application domains
      (ApplicationDomainAttribute class and ServicedComponentEx class)
• Inline transaction support using "services without components" also on Windows XP 
      (InlineTransaction class)
• Just-in-time activation proxy pooling
      (JITAPool class)
• Managed wrappers for the configuration catalog
     (CatalogManager and related classes)

Requirements:

• Microsoft Visual Studio .NET 2003
• .NET Framework 1.1
• Microsoft Windows Server 2003 (recommended) or
• Microsoft Windows XP or
• Microsoft Windows 2000 Server/Professional

The full reference documentation and samples are available inside Visual Studio .NET 2003 after installing the SDK.

We still do not plan to charge license fees for the binaries when we move them to "release" status, but we cannot and will not offer any reliable support on "free" terms, of course.

 

A simple in memory cache for "business data"

Here is a little project demonstrating a little in-memory data cache using DataSets and serviced components. The demo illustrates how to deal with near data in a smarter way than by just walking up to SQL Server all the time. My "Layers and Tiers" slide deck of the recent tour makes the point that it's a good idea to delegate data access to services which are aware of the temporal character of data (static, near static, dynamic and "hot") and optimize access to that data accordingly. The sample also illustrates a way of implementing proper layering for such a data service: The public interface is a Serviced Components (an alternate public interface could be a web service or whatever else), the internal implementation is a base class providing the caching logic and the resource access is realized by deriving one version for SQL Server access and one of OleDb access from this base class. Creating instances of the concrete implementations is done through a config-driven factory.  -- looser binding towards the "client edge" and tighter binding towards the "backend edge".

The concrete implementation assumes that the "Products" and "Categories" tables in the accessed Northwind database copy are in a read-only replica and assumes that the data will always conveniently fit into memory. The demo shows the perf. difference between the cached and the direct database access versions.

The time difference isn't only due to the caching, though. The sample also shows how to use "connection pooling" to JITA Enterprise Service components using a client side JITA pool. The code for that is included.

Thank you, Steve :) 

Yesterday, in Portugal, was the last stop on the "Scalable Applications Tour". And I consider this one more significant than any event I've done before. Why?

• This was the only event series I know of where an external, "regular guy" like me and senior architect form a Microsoft product group have been together on a speaking tour of this scale. 
• This was the only event series I know of where Enterprise Services and ASP.NET Web Services got an even-handed coverage in term of features and use-cases. The wwinning combination is to use both. 
• This was the only event series I know of where Windows product features introduced in 1997 got even-handed coverage with Windows product features introduced in 2003. The goal was to paint the big picture and not to give the usual, incremental "here's something new" talks.

It was a lot of fun. I learned a lot, Steve says that he learned a lot and I am sure that the 2200+ people we could do these talks for were also able to take some valuable information home.

Notice to attendees: All sample code and the final set of slides will be available within the next 24-72 hours for download. Watch this space for a pointer to the download location(s).

Scalable Applications Tour update

All the travel this week (Oslo, Copenhagen, Paris, Lisbon) has definitely taken its toll on Steve's and my ability to communicate with the outside world, including blogging and posting updates to the slides and the sample code. However, the whole tour is great fun and we've made some great improvements on our content and the "overall experience". We'll have the final slides up at www.thearchitectexchange.com by Saturday and I will spend some time tomorrow to wrap up all samples into a single zip archive that I'll post there as well.

"Bill Gates! Bill Gates!"

Steve and I went to the Kremlin and the Red Square here in Moscow today. We toured the Kremlin and its churches before we went to see Lenin (that's a warped experience and a "must see") and quite a few other sights around Red Square. Yesterday we had a great event with about 700 people, lots of very interesting questions, two most excellent guys doing simultaneous translation, our slides in a booklet all translated into Russian, and we were even asked to give autographs....

The funniest little story at the Kremlin went like this: We're walking by the congress center (of the former communist party) towards the Palace of the Patriarchs and a group of maybe 20 school kids from St. Petersburg walks in front of us. Suddenly one of the kids (who were all about 15) turns around after hearing us chat and says "Where you from?". Steve grins and says "Seattle ... Washington ... USA". All of a sudden he's got the crowd's attention, all the kids stop. And then he puts up his special kind of amused smile and says "I work for Microsoft". Awe. Disbelief. Happiness. One kid says: "Windows XP!". Steve, very visibly enjoying himself, says "Yes. Yes", pointing to himself, "I worked on that!". More happiness and the kids start chanting joyfully "Bill Gates! Bill Gates! Bill Gates!" and would not let us get further away than 10 meters while we were in the museum in that building. Finally, Steve made one of the kids the star of the day by trading 2 rubels (about 6 cents) for a 1 dollar note. The dollar from the "Amerikanskij" was surely one of the trophies of the day judging by the way they were all inspecting it. Since they're not from Moscow, it appears like they're not seeing Amerikanskijs that often, much less Microsofties.

I am not making this up. And for me as a German, non-Microsoft person it was sure a lot of fun to watch ;)

 

Aspects in Bucharest.

Tomorrow (today, strictly speaking), Steve and I will be doing the second round of talks on our tour speaking about scalable systems. Microsoft Romania expects 700 (!) people and on Friday we're going to have another 700 in Moscow. Add the 200 people we had on the first event yesterday in Warsaw (which went really well) and we'll have had 1600 people altogether in a single week. Fantastic! Great fun.

On of the things I'll be showing tomorrow (in the context of "programming model extensibility") and which you can build yourself using VS.NET 2003 and my new set of libraries is a little custom "aspect" (or "interceptor" for more old-fashioned thinking people):

using System;
using newtelligence.Aspects;
using newtelligence.EnterpriseServices;
namespace AspectComponentServer
{
 [Serializable]
 public class MonitorData
 {
  private string data;
  public MonitorData( string data )
  {
   this.data = data;
  }
  public string Data 
  {
   get
   {
    return data;
   }
  }
 }
  public class MonitorAspectAttribute : Attribute, IAspectServerContextLeave, IAspectServerContextEnter
 {
  public MonitorAspectAttribute()
  {
  }
  
  public void OnServerContextEnter(object transparentProxy, System.Runtime.Remoting.Messaging.IMessage message, 
                                   System.Reflection.MemberInfo member, System.Reflection.ParameterInfo paramInfo)
  {
   Console.WriteLine( "Enter context for method {0}", message.Properties["__MethodName"] );
   Console.WriteLine( "  -- Setting property into context" );
   ContextUtilEx.SetUserProperty( "AspectComponentServer.MonitorAspectAttribute.MonitorData", 
                                  new MonitorData("Test information") );
  }
  public void OnServerContextLeave(object transparentProxy, System.Runtime.Remoting.Messaging.IMessage message, 
                                  System.Reflection.MemberInfo member, System.Reflection.ParameterInfo paramInfo, bool ReturnValue)
  {
   Console.WriteLine( "Leave context for method {0}", message.Properties["__MethodName"] );
   MonitorData monitorData = 
          ContextUtilEx.GetUserProperty( "AspectComponentServer.MonitorAspectAttribute.MonitorData") as MonitorData;
   Console.WriteLine( "  -- Property value found: {0}", monitorData.Data );
  }
  
 }
}

and use it like this

 
using System;
using System.EnterpriseServices;
using newtelligence.EnterpriseServices;
using newtelligence.Aspects.Constraints;
using newtelligence.Aspects;
[assembly: ApplicationActivation( ActivationOption.Library)]
[assembly: ApplicationAccessControl( false )]
namespace AspectComponentServer
{
 [Transaction(TransactionOption.Required)]
 [EventTrackingEnabled]
 [MonitorAspect]
 public class FlightReservation : AspectServicedComponent 
 {
   [ContextState]
   int jitaSessionState;
   public string ReserveSeat( 
            [Match("[A-Z]+")]     
            string Airline, 
            [MinLength(1),MaxLength(4),Match("[0-9]+")] 
            string FlightNumber,
            [Between(1,100)] 
            int Row,
            [Match("[A-K]")] 
            string Seat,
            [MinLength(3),MaxLength(3)] 
            string DepartureAirport,
            [LaterThanToday]
            DateTime DepartureTime,
            [MinLength(3),MaxLength(3)] 
            string ArrivalAirport)
  {
   ...   
  }
  public bool ConfirmSeat( [MinLength(1),MaxLength(8),Match("[0-9]+")] string ReservationCode )
  {
    ...
  }
  public bool CancelSeat( string ReservationCode )
  {
    ....
  }
 }
}

Steve said yesterday that the capability to add your own interceptors and my extended "ContextUtilEx" class that allows you to flow arbitrary items in the context is enough to build your own services on top of the Enterprise Services infrastructure much in the same way as COM+ does it internally.

The [ContextState] attribute sits on top the ContextUtilEx and gives you custom "session state" for just-in-time activated components, which means that you can have fields that have context scope and whose values will survive the recycling of the actual object instance across activations. That brings Enterprise Services and ASMX one step closer together in terms of programming models.

If Microsoft wouldn't be grumpy with me if I did it (and they would), I would name my stuff not "newtelligence SDK for Enterprise Services" but rather "COM+ 1.75" ;)

"Resistance is futile". Someone said that to me at a dinner a few weeks ago, and it seems that happened to Chris, too. Seems like Chris gave in straight away and got assimilated ;)

newtelligence SDK for Enterprise Services.

Here's the first drop ("RC0") of the library that I am releasing along with our tour. The ZIP file contains a single MSI, which installs into Visual Studio .NET 2003. There will be more drops during the next two weeks, which will mostly add more documentation and more samples.

Greetings from Warsaw...
PS: Does Aspects, too.