Distributed EAI

Managed Windows NT Services, Application Domains & Principal Policy

Rick G. Garibay - The more I learn, the less I kno — Thu, 10 May 2007 17:34:58 GMT

Update to orginal post: After doing some more research, the actual problem turned out to be that the legacy Windows NT service never specified the WindowsPrincipal and had nothing to do with the Gateway assembly. In fact, it turns out that when you load a class using Activator.CreateInstanceFrom(), the class loads into the same parent application domain. I've updated the solution below appropriately.

Thanks to Rick Strahl for clarifying the Activator.CreateInstanceFrom(), app domain behavior. You can also find an excellent article by Rick in CoDe magazine that covers this extensively: http://www.code-magazine.com/article.aspx?quickid=0211081&page=1

- - -

I recently ran into a security problem in a somewhat typical scenario for which the specific details I imagine are fairly esoteric because I was unable to find anything relevant on the web or on MSDN forums. It is amazing how easy it is to forget (or at least suffer from delayed recall) fundamental aspects of the framework when you are in the thick of it and trying to ship a product!

So, I figured I’d post an overview of the scenario and the solution to the problem in hopes of helping anyone else who runs into this.

Scenario

A common approach for setting up any kind of automated process is to use a Windows NT service which is continually running, monitoring a queue or database for records/messages that represent events for wich some work should be performed. For example, I can have a Windows NT Service monitor a private queue every 5 seconds, and when a message is available, process it (whatever process means for the given service).

One option for the processing bit would be to raise and event and delegate the work out to a method that would maintain all of the event handling logic. This handler could reside in a single class within the same assembly as the service, but for obvious reasons this could quickly get unruly and introduce a maintanance nightmare. If every time you need to change the event handling business logic you had to recompile and redeploy the service executable, although you might be building job security, you'd find yourself getting frustrated at best and at worst, you would be tightly coupled to a single assembly making it difficult to leverage this pattern in a more abstract, loosely coupled manner.

One possible solution might be to apply a typical publish and subscribe pattern where the Windows NT service polls a backing store (or ideally was notified automagically) and when a record (or message) meets a specific criteria fire an event. The event would then be mapped to a delegate which implements the handling of the event. However, to keep the implementation of the handler from being static and hard coded, we could further delegate the implementation of the work to an external component, making the original event handling delegate only worry about the plumbing for dispatching the work to another component. Ideally, this would allow a plug and play approach where assemblies could be "dropped" in and just work. Of course, this would require that the message in a queue has some kind of metadata about the component to execute (such as assembly name, type and URI) and when the event is fired, the handler gets access to this URI and dispatches the call to it.

This is precisely what the System.Activator class allow you to. If you know the name of the assembly that contains the component you want to call, simply call the CreateInstanceFrom method on the Activator class. The CLR will then probe the local private directory for the assembly name, load it and instantiate an instance of the type. Figure 1 below summarizes what this might look like.

Figure 1: Mock Sequence Diagram

When the IGateway object is “unwrapped” and made available to the MyNTService Windows NT Service, the service can call a method (or methods) on the instance to do some work. To achieve polymorphism, since the Gateway component that is unwrapped implements the IGateway interface, the service is guaranteed to be able to call a particular method (Execute, in this case).

Figure 2 (below, right) provides a possible object model that supports this design.

The purpose of the Gateway class is to serve as wrapper around a component/service call while maintaining location transparency. This can thought of as a "super proxy" and is really just the classic Service Gateway/Service Agent pattern. The MyNTService Windows NT Service has no idea if the component is local to the machine or process or somewhere out in the cloud. With this flexibility comes significant power in being able to make decisions on distribution in a post-deployment manner.

As the saying goes “with great power comes great responsibility”. Looking again at the sequence diagram in Figure 1, two important things need to be considered.

First, this scenario demonstrates WS-I Profle interop between a WCF service and a managed Windows NT Service running on .NET 1.1 CLR (pretty cool huh?).

Second, between every call in the sequence is an authentication and authorization boundary. Though impractical, the Gateway component could ensure that only the identity of the Windows account running the MyNTService Windows NT Service can call it’s Execute method. More importantly, however, the Gateway should flow the identity of the Windows account running the MyNTService Windows NT Service process to the WCF service that is out in the cloud so that the WCF service can authenticate and authorize the call. Pretty simple,right?

Problem

Well, one of the things that may not be overly apparent is that the assembly in which the Gateway type that implements the IGateway interface resides will actually load in the same application domain that exists within the host NT service process. The MyNTService Windows NT Service knows NOTHING about the Gateway component (other than that it implements a standard interface). When an instance of the Gateway component is activated, we want to run in the security context of the parent process (of the MyNTService Windows NT Service ), however if the host application domain has not specifically set the the Principal Policy, the component will execute with no security context, or to be a bit more technical, will load using the default UnauthenticatedPrincipal enumeration flag. This basically means that the principal under which the MyNTService Windows NT Service is running will not be attached to the thread on which the Gateway component runs. In fact, as the MSDN documentation for the System.Security.Principal.PrincipalPolicy enumeration states “Specifies how principal and identity objects should be created for an application domain. The default is UnauthenticatedPrincipal.”.

Solution

Those keen to the default behavior regarding application domains and Windows Principals already know where this is going, but for those who aren’t sure, stay with me.

Although the MyNTService Windows NT Service was running as a fixed domain account, the WCF service requires transport level authentication and uses role-based authorization to ensure the caller (MyNTService) is authorized. However, calls from the MyNTService Windows NT Service via the Gateway were failing with "The request failed with HTTP status 401: Authorization Required."

I triple checked IIS security settings. My unit tests worked just fine. My test harness also worked and even browsing to the service’s metadata page was working. For some reason, the Gateway was calling the WCF service in a seamingly anonymous manner. This threw me for a loop for a couple of days.

Then, a few days ago, while in the shower (I get many of my “light-bulb on” moments in the shower or in my dreams for some odd reason) the solution hit me like a thunderbolt! The Gateway assembly was being loaded without an authenticated principal, and hence the solution was simple, sweet and elegant:

AppDomain.CurrentDomain.SetPrincipalPolicy(System.Security.Principal.PrincipalPolicy.WindowsPrincipal);

Now the Gateway took on the identity of the Windows AD account that was running the Windows NT Service process and was able to authenticate to and be authorized by the WCF service.

You can find more info on principal policy and application domain behavior here: http://msdn2.microsoft.com/en-us/library/90395801.aspx

Distributed SQL Server Integration Services

Rick G. Garibay — Fri, 22 Dec 2006 01:45:10 GMT

SSIS is the successor to Microsoft SQL Server DTS. To that end, it is very much an ETL (extract, transform and load) tool, however with the new product, there are several new possibilities that were either untenable or inpractical to implement before.

I am by no means a data warehouse or business intellegence expert and would defintely consider myself a generalist on related topics, but as a system developer/archtiect, addressing ETL is sometimes part of the job. Historically, I've pretty much glazed over in conversations about star schemas, normilization, facts and dimensions as my world is largely OLTP based. That said, I am quite impressed by SSIS and I'll explain why.

In previous versions of DTS, the engine itself was coupled to SQL Server. This was before the dawn of .NET and the idea of hosted environments and runtimes were still quite distant. To make a long story short, you developed DTS packages on SQL Server and deployed to SQL Server. Data sources and transformations were very tightly bound to the package itself and to the security context of the SQL Server on which the DTS package ran. Although you could drop down to a command line, remote execution was either ardous or impractical due to the severe lack of reliability and the reason was simple- classic DTS was meant to run in the context of the SQL Server that "hosted" it. As a result, packages were normally scheduled through SQL Server Agent jobs and life was good.

A fantastic way to introduce SSIS concepts would be to make analogies to BizTalk Server, because in many ways, it is quite obvious that the two teams talked during development. Everything from the fully integrated Visual Studio 2005 integration to the really slick design surface and constituent shapes seem very much influenced by BTS. Understanding that I may have already lost a large majority of readers in this analogy, I'll leave it at this: SSIS is very much to the database layer what BTS is the middle tier. Fair enough? Good.

Not so fast. There's no way I'm getting away with that, so let's go into some details.

I already mentioned the design and development experience. Whethere your using Visual Studio 2004 SSIS projects or BIDS (Business Intellegence Design Studio), the design time experience is pretty sweet. Scope/boundaries are represented as Sequence Containers and Tasks are the drivers for the overall logic of the package. For example, a Sequence Container will contain Tasks and each Task is linked to other Tasks via precedence constraints. While this concept existed in DTS, it was largely brain dead and confusing. In SSIS, you drag a Task constraint from the predecesor to the contingent task and a green arrow intuitevely reflects this relationship. Precedence constraints can also be established between Sequence Containers, and the result of this apprach is reflected in the following diagram which represents the Control Flow of the package logic:

A major distinguishing factor between SSIS and DTS is the explicit seperation of logic flow from data manipulation, or Data Flow. The Data Flow Task, which is a part of the Control Flow, seeds the Data Flow, which is the sequencing of ETL tasks at a finer grained level.

These Data Flow tasks are all about transformation, and with the introduction of ADO.NET-like DataSets, a new emphasis is placed on working with data from in-memory data streams. Like its predecessor, one of the big strengths here is the ability to define data sources on different machines, and this is now the role of the Connection Manager. What is fantastic about the Connection Manager model, however, is the ability to configure the connections and deployment time via a facility known as SSIS Package Configurations. This is really an excellent approach to addressing the deployment problem of targetting different servers and is a great pattern that could very well be applied to ASP.NET as well.

Back to Data Flow. In addition to Transformation Tasks and Data Sources (which reference Connection Managers), it should come as no surprise that SSIS handles XML input and SOAP web service interaction naturally. In addition, messaging via MSMQ is also natively supported. This is really where the overlap (at a purely conceptual level) with BTS comes in. SSIS Packages can be deployed to any SQL Server 2005 machine which really acts as a host. Once you deploy the package to its host, you can run the package literally from anywhere, either interactively or programatically. So in a sense, SSIS really becomes a service which waits to be invoked, only its concerns lie more on the database side, particularly in ETL.

It should be noted as well that as a CLR host, SQL Server facilitates the loading of managed assemblies from within SSIS packages. This is valuable, for instance, when you want to call into a managed .NET DAL that is perhaps exposed via a SOAP web service which returns a dataset or a datareader. Why not? SSIS natively understands ADO.NET data transfer objects. The potential for abuse here should be evident: maintaining a seperation of concerns between business tiers and data tiers is essential to avoid spaghetti architectures.

Let's wrap up by looking at a practical example. An ASP.NET application provides the front end for an adminstration console which is used to refresh a data mart with OLTP data on demand. The ASP.NET application connects to a WCF service which in turn invokes the SSIS API and executes the package remotely. In this scenario, the machines are truly distributed. The ASP.NET application is hosted on an IIS 6 Server and the WCF service may be hosted on a seperate IIS server although both on the same machine is possible as well. Where things get interesting is in the fact that the source OLTP and target Data Mart are very likely on different machines, and the OLTP data could very well come from several disperate sources on different machines.

In this scenario, there are a few things to consider.

First, the ASP.NET client is completely decoupled from the WCF service and underlying SSIS API calls. All the client knows of the service is the address, binding and contract.

Second, the WCF service wraps the call to the SSIS API, completely isolating all aspects of the transaction from the client.

Third, the WCF service interacts with the SSIS package as a service- the only known information is the package name and host server. The service doesn't necessarily need to know or care what SQL Servers are being targetted by the ETL process- this is all encapsulated by the SSIS package.

Optionally, of course, the service could pass specific parameters in from the ASP.NET client, instructing the SSIS package to manage its Connection Managers at runtime. This is a truly powerful concept where servers may vary from environment to environment (i.e. Dev, Test, Prod) or in large enterprises where servers are constantly upgraded, changed and moved around.

Based on this .NET paradigm, all of the existing rules for design and security apply. Specifically, from a security perspective, using a fixed identity to flow the call from the ASP.NET client to the WCF service, and from the WCF service to the SSIS host is required. Impersonation simply won't work unless you are delegating which is undesirable in most enterprise scenarios.

SSIS very much fits nicely into the .NET architecture stack, and like BizTalk Server can be wraped in descrete services to maximize its ability to perform serious heavy lifting while leaving a small footprint on the system-level design.

In closing, SSIS has very much come of age and for the most part, has been a pleasure to work with- even as a DW generalist ;-)

In a future post, I'll walk through how to accomplish the described scenario step by step and this may well turn in to future speaking topic- let me know what you think!