Yossi Dahan [BizTalk]


Monday, December 31, 2007

BizTalk's Pub/Sub

The publish/subscribe mechanism in BizTalk is one of the key features of the product and is very useful and powerful.

I guess there's some learning curve around it, and that most first implementations of any BizTalk developer do not make much use of it (as they often start with all ports being directly bound) and that it takes a while befor a BizTalk developer and the organization involved establish a good architecture and move more towards using true publish-and-subscribe, losely-coupled, approach to implementation.

However, the existing model is not perfrect; in my view (and I suspect it is shared by many) it has two main weak points -

  • The pub/sub is implemented on top of MS-SQL which introduces a significant performance overhead

  • The orchestration subscriptions are 'compiled' and cannot be configured withouth a build-and-deploy cycle

  • The first point is quite an obvious one - there would be a latency associated with any implementation of publish/ubscribribe mechanism;. in the BizTalk case it involved writing the message and it's meta data (context) to the message box (a SQL database) and having a separate process locate newly published messages, figuring out which subscribers need to receive a copy of the message and manage the activation/correlationthe of message-process interaction (as well as keeping a list of references for house keeping etc).

    Reading and writing to the database, the the polling interval of the subscription evaluation process, etc. all introduce latency, which, in certain scenarios, can be crucial.

    If to believe the fractions of information floating around regarding Oslo then we might see an in-memory pub/sub mechanism in future version of BizTalk (in addition, not as a replacment to the existing model I suspect) which, while will no-doubt come with a price (persistance, and therefore scalabiltiy and durability to some extent), will no-doubt make supporting low-latency scenarios much easier.

    As for the second point -

    At first look the pub/sub in BizTalk is very flexible; in all the BizTalk demonstrations I can remember from the past the presenter would create a recieve port and a couple of send ports and will edit the subscriptions of those ports in the administration console to show how easy it is to create content-based routing in BizTalk server and configure it at runtime.

    In BizTalk 2006 you even did not have to restart the host to speed things up (as you did in demos with 2004), it happens pretty much instantly.

    However, the case with orchestrations is not that simple...

    The subscription for orchestrations is specified as a a filter in the properties of the initalizing receive shape in the process; this gets compiled into your assembly together with the process, and will be used to create the subscription when you deploy the orchestration.

    As far as I know, short of manipulating the management database yourself (which would not be supported) there's no way to change those subscription at runtime.

    If you want to change the subscription you have to change the filter in the orchestration, build, undeploy the old version and deploy the new one (or version the process and perform the side-by-side deployment)

    This is, in my view, an un-necessary pain, in dynamic organizations (aren't they all?) that require changes often; and to that extend developers had to find a solution to the "I need to be able to change that subscription from outside the process" requirement.

    That solution is often adding some routing metadata to messages in the form of context properties ('nextProcess', 'Operation', etc.) which would be set by publishing processes and/or pipeline components and use these in the filters (rather than the actual content data).

    So you could often see a pipeline component, often driven by some external configuraion, that would check for certain bits in the message or it's meta-data and set these context properties based on the values it found; the premise is that pipeline components are easier to replace, but also - thesee components often use database or a rules engine in one form or another to decide what goes into the message context and by doing so introducing real flexibility as is advertised.

    What all of this means is that we, developers, end up developing a pub/sub mechanism on top of the existing pub/sub simply because we need flexibility the product does not provide.

    I don't like this apprach, but I end up doing this myself occasionally, simply because I have to.

    I could possibly understand why MS has decided to do so - there are benefits to editing the subscription expression within the orchestration (known types would be one thing), and also - one could argue that the process subscription is part of the process design and so changing it is likely to involve code changes as well which will require a re-build, but really - I think we would all have benefited from the ability to edit the orchestration subscription in the same way we can edit send port subscriptions - through the admin console.

    Labels: , ,


    • Hi Yossi,

      Like the idea of abstracting the subscription from the recieve shape. So you could define your orchestration and say it will use a subscription called XYZ and then define that subscription elsewhere or much later

      Im sure this would bring lots of benefits and opportunities

      By Anonymous Anonymous, at 01/01/2008, 13:50  

    Post a Comment

    << Home