The Web Services Kernel

Don Box presented a set of (mildly hallucinogenic [1]) slides at the XML Developers Conference on the topic of WS-Why? The goal of the talk was to provide some background on the various WS-* specifications and contextualize them a bit to talk more about the intent of the specs as opposed to their mechanics. This essay is a remixed version of the first part of Don’s talk, interspersed with some of my own noodlings.

The Web Services Kernel

Open question: how much do you really need to know about all these WS-* specs? The answer of course depends on who you are and what you’re trying to do. I think a good way of exploring this question is by analogy to kernel architecture and how various types of developers relate to the core components of the OS.

In an operating system, the kernel implements the critical low-level plumbing that lets you do important stuff. More precisely, the kernel is often practically defined as “the important stuff you don’t need to know too much about.” Most developers can live long and happy lives with only a dim awareness of the NT kernel, because there are so many other higher-order abstractions that are tremendously easier to use. There are also the people who “know the kernel” in the Biblical sense because they’ve been writing device routines in C for n+10 years. Statistically, however, you are highly unlikely to be one of these people. The most interesting group in somewhere in the middle between blissful ignorance and uncomfortable intimacy – these people know a bit about what the kernel does and how it’s laid out on a conceptual level but aren’t using WinDbg to pore through crash dumps on a daily basis. These folks have simply found that having a little bit more knowledge of the inner workings helps them understand their world better.

I think the ultimate goal of the Indigo people is to make interacting with the WS-* protocols a lot like interacting with the NT Kernel – it just happens, you can take it on faith and get on to solving problems that are actually important to your business. This is a lofty and admirable goal and I really hope they succeed -- unfortunately, we’re not quite there yet. Until the tools catch up a little bit, being a good web service developer means that you need to have a little bit more low-level knowledge than you probably would like to have. Eventually, I think things will evolve to the point where this knowledge is solely academic (compare the number of people who use DCOM with the number of people who understand the DCOM wire protocol), but for now it’s helpful to know a little bit about the “kernel” of web services. To that end, here are the five specs that make up the “kernel” of the WS-* stack along with why they’re important and what problems they solve.

Kernel component #1: XML
Problems it addresses: Common data model and way of looking at the world


XML is the heart of the web services stack – it’s the single point that everyone agrees on. XML is the tool of choice because it’s more or less ubiquitously understood – there’s nothing inside it that’s especially magical. Had things gone slightly differently in the standardization process, it’s entirely possible that the entire web services movement might have been based around the exchange of LISP sexp’s [2]. However, the industry seems to prefer angle brackets to parentheses and we went with XML.

With XML, we agreed that hierarchical sequences of elements and attributes would be the metaphor we would all use to think about the data we exchange. There’s nothing intrinsically special about elements and attributes other than the fact that everyone agrees to think about the world this way. This agreement is tremendously beneficial, because it lets us move on to other problems (like what the messages look like).

Kernel component #2: SOAP
Problems it addresses: Orthogonal content and extensibility


Yes, it’s possible to do web services without SOAP. Ebay and Amazon do this quite successfully. Interacting with non-SOAP services is like saying “Here’s a blob of XML that describes what I need to do.” It doesn’t really matter what that blob of XML looks like, because it’s specific to each service and each service is free to define its own message forma. Things get more complicated when you need to say “Here’s a blob of XML that describes what I want to do, and by the way, embedded somewhere inside that blob is some additional information about who I am and what I’m allowed to do”. It would be nice to have some common ideas around where exactly one would look inside an arbitrary XML blob in order to find stuff that’s only tangentially related to the service being performed. Generally, this can be described as the problem of orthogonal content.

Orthogonal content is the Achilles heel of hierarchical data structures. Partitioning ideas into conceptual hierarchies works great so long as every idea can be cleanly mapped into exactly one hierarchy. Problem is, most of the information in the real world doesn’t work this way. Some ideas in real systems can be represented hierarchically, but others cut across many different hierarchies that are otherwise unrelated. A good example of this is color – trees are green, money is green, Kermit the Frog is green. It seems weird to have them all live in the same conceptual hierarchy, given that these three wildly unrelated concepts share only the tangential commonality of “greenness”. Any system that uses a set of hierarchical concepts to describe the world must eventually tackle the problem of orthogonal content – it’s what lead the OO people to start thinking of aspect-oriented programming and what drives the usage of SOAP in the WS-* stack.

SOAP addresses the orthogonal content problem by introducing a common frame around the blobs of arbitrary XML we send to services. The SOAP envelope model defines two parts of a message: stuff that’s specific to the service goes in the body, and orthogonal content goes in the headers. This gives a common solution to the problem of orthogonal content – which is something that the ad-hoc non-SOAP message formats would have had to invent anyway. By separating the messages into headers and bodies, SOAP offers both a general way to deal with orthogonal content and also provides a generic extensibility mechanism. Over time, we can change the behavior of the SOAP infrastructure to account for things like security by introducing new headers without affecting the body of the message.

Kernel component #3: WS-Addressing
Problems it addresses: Transport-agnostic view of a message’s destination and source


WS-Addressing is something that really should have been included in the original SOAP specification. However, when SOAP was written nobody was really thinking about transports other than HTTP. As a result, pure SOAP relies on the characteristics of the HTTP transport to convey addressing information. For example, a pure SOAP message does not contain any information about the address to which it was sent – that information is carried by the transport and is lost once the message is pulled off the wire.

It turns out that having a notion of “address” embedded in a SOAP message itself is incredibly useful. It means that you can stick a SOAP message in a database and queue it for later processing without having to store additional information about the message’s original destination. Addressing is also critically important to alternative message exchange patterns. Request/response doesn’t need an explicit notion of a reply-to address, because it’s always implicitly interpreted to be “the connection this message came in on”. For other message exchange patterns like one-way or duplex, it’s nice to be able to say “send your response to this guy over there”.

WS-Addressing introduces the notion of an EndpointReference, which describes a place in network space that can receive SOAP messages. WS-Addressing also introduces a couple of common SOAP headers that allow systems to know where to look inside of a SOAP message to pull out addressing information. All in all, it’s a pretty simple specification.

It’s true that some ideas in WS-Addressing make some people uncomfortable. Specifically, some people don’t like the fact that WS-Addressing favors a proliferation of verbs over a proliferation of endpoints. While it’s somewhat useful to know that this debate exists (if only to contextualize the term “RESTafarian”), it’s basically kernel-mode wonkery that average people should not have to expose themselves to.

Kernel component #4: WS-MetadataExchange
Problems it addresses: Self-description


WS-MetadataExchange is the social custom of the web services world. It lets two services exchange the customary pleasantries and small-talk upon first meeting each other – on the order of “Who are you? What do you do? My, aren’t these hors d’oeuvres delicious?”

WS-MetadataExchange is a SOAP-based formalization of the http://fooservice.asmx?wsdl idiom made popular by ASP.NET Web Services. Believe it or not, not everyone knows about the ‘?wsdl’ thing – other stacks have their own way of kicking a service and making WSDL ooze out. WS-MEX defines a standard set of messages that you can send to any service that supports the protocol, and means that you don’t have to know the implementation-specific message you need to send it in order to get back its description.

Kernel component #5 (begrudgingly): XSD/WSDL
Problems it addresses: Contract description languages


XSD and WSDL are the common languages services use to describe themselves to the outside world. They’re definitely not perfect, but they’re what we’ve got. There are arguably better ways to describe data structure and behavioral capabilities, but for better or for worse the industry seems to have settled on XSD and WSDL. The bottom line is that we need to have some standard vocabulary for describing service contracts and XSD/WSDL fit that bill for now. Maybe something better will come along eventually, but there’s a general preference for widespread adoption over perfection in the world right now.

Conclusion
Anybody who’s been following the WS-* space will notice that there are a number of specs that aren’t on this list. The reality is that the problems those specs solve (security, reliability, transactions, etc) are not precisely fundamental problems and are not ultimately critical to a service-oriented world. They certainly address problems that need to be solved, but they are not problems that will be encountered by everyone. The kernel of WS-* -- the stuff that’s really central to getting this whole thing of the ground – is a much smaller (and simpler) set of specs.

Footnotes
[1] Anybody who saw Don’s slides knows what I’m talking about. The color scheme was godawful – those slides, combined with the after-effects of the previous night’s socialization and several cups of coffee nearly caused my brain to seize up.

[2] Consider the following:
<Person name=”Steve” sign=”Leo”>
<Age>26</Age>
<City>Seattle</City>
</Person>

Vs.

‘(Person ((name Steve) (sign Leo))
(Age () 26)
(City () Seattle))

Arguably the same thing. It’s just a question of presentation.

 

#1 Aaron on 10.23.2004 at 12:33 PM

What's even scarier about Don's slides was that he got the format/color scheme from Chris Anderson... a UI guy.My original reaction here - www.dogcaught.com/.../index.php

#2 bliz on 10.23.2004 at 6:13 PM

excellent post!

#3 Christian Weyer on 10.24.2004 at 1:37 AM

I love you.(I hope this does not count as comment spam ;))After all the confusing buzz around WS-* and friends, this is actually all that has to be said (OK, I seem to miss policies, but it's OK for now).And yes, there will be voices that tell it is not the way you put it. Forget about them, please.I always have wondered why suddenly everbody wants to be a 'kernel level programmer' these days ... hm ...

#4 Szoke on 10.25.2004 at 9:59 PM

Great!

#5 Jake on 7.17.2006 at 12:27 PM

Good one