Yossi Dahan [BizTalk]

Google
 

Thursday, August 13, 2009

BizTalk WCF adapter and multipart messages

I wrote myself notes for this post months ago, but haven’t taken the time to write it down properly; a passionate discussion today with Rupert Benbrook prompted me to get back to this.

The WCF adapter does not support multi-part messages, and some would argue that’s a suggestion of the feature possibly being dropped in some future version of the product.

This small fact seems to have been glossed over by most – I, for one, haven’t heard this discussed anywhere, to the point that I’m left wondering if I’m really just “stuck in my ways”, but I honestly think that’s a poor decision.

Trying to search for some detail about this fact I found the following, rather old, but still very valid and informative, post; here are some of the key points made -

The reason support for multi-part message in WCF adapter is not being recommended is because WCF does not support it and the model proposed is that whoever wants to use something equivalent, need to aggregate the messages into one xml and send it as one single-part message.

(Tapas Nayak, Principal SDE, BizTalk Adapter Pack)

Multipart messages are a marginalized feature in BizTalk, the pipeline components like the flat-file and xml don’t understand them, same story for the majority of the adapters. So we decided to not support them in the WCF Adapter too
(John Taylor, Senior Technical Lead, BizTalk Server)

 

With regards to the first statement – I think that is somewhat down to interpretation – WCF did not change how web services work; WSDL is the same WSDL, specifications have not really changed; so – WCF is just a new (and of course – much better) implementation; this is an important basis for any discussion because it indicates that the WCF adapter could have taken the same path the SOAP adapter did.

Consider the following Service definition in WCF:

        [OperationContract]
CompositeType GetDataUsingDataContract(CompositeType composite, CompositeType composite2, out CompositeType compositeOut);


or the equivalent web method in asmx:



        [WebMethod]
public CompositeType GetDataUsingDataContract(CompositeType composite, CompositeType composite2, out CompositeType compositeOut)
{
}



the WSDL for both methods is pretty much identical, the operation definition looks as follows -



<wsdl:operation name="GetDataUsingDataContract">

<wsdl:input wsaw:Action="http://tempuri.org/IService1/GetDataUsingDataContract" message="tns:IService1_GetDataUsingDataContract_InputMessage"/>


<wsdl:output wsaw:Action="http://tempuri.org/IService1/GetDataUsingDataContractResponse" message="tns:IService1_GetDataUsingDataContract_OutputMessage"/>


</wsdl:operation>



the messages are defined as so -



<wsdl:message name="IService1_GetDataUsingDataContract_InputMessage">

<wsdl:part name="parameters" element="tns:GetDataUsingDataContract"/>


</wsdl:message>


<wsdl:message name="IService1_GetDataUsingDataContract_OutputMessage">


<wsdl:part name="parameters" element="tns:GetDataUsingDataContractResponse"/>


</wsdl:message>





And the schemas, in both cases, define the aggregated messages – with all the in/out parameters as required.



So – what is the difference? the way I see it is it is in the way the different adapters handle the messages as they pass through the adapter; the SOAP adapter has a layer in which, when receiving messages from the outside,  breaks them to individual parts, delivering a multi-part message to the message box; similarly, when delivering messages from BizTalk to the service, it composes them, aggregating the various parts in the request message, to a single SOAP envelope; the WCF adapter does not include such functionality – it will take the message BizTalk provides and will wrap it with the SOAP envelope pretty much as is.



All my grievances about the SOAP adapter aside – I think its approach is a good one; it allowed the process to focus about the flow of the business process and avoided the need to define and construct artificial entities in the form of aggregated schemas, which often make very little sense (as the various parts can be largely unrelated from a process perspective, but still necessary for the service to perform its operation).



In my discussion with Rupert the same point was raised – the fact that SOAP supports one message in and one message out (i.e. no multipart messages), but in my view that’s a problematic argument – again – the whole point of the BizTalk architecture is that the send port can mask the transport mechanisms; ok – the SOAP adapter is not doing a particularly good job at that, but that’s hardly an excuse to make matters worse – I don’t want to have to know that I need an aggregated message to call this particular message because it uses the WCF adapter;



Moving on - I particularly disagree with the second statement mentioned - I think multipart messages are a very important feature in BizTalk, and would be surprised if any seasoned BizTalk developer would tell me they prefer not to use them (do let me know if you think I’m wrong); to start with – it allows one to define a type once and then use it multiple times; this makes maintenance infinitely better when types need to be changed (I thought it ought to be this schema, but found that this other schema is better); ok – so you have to be careful about how and where you re-use, to not introduce unwanted dependencies, but if you have large orchestrations which follow several steps,  this is very useful.



Secondly, it allows one to easily link pieces of information that are only ‘somewhat-related’, especially important, in my view, when using pub/sub; having to pass several pieces of information from the publisher to the subscribers; the alternative, not surprisingly, would be exactly what is suggested above – create a schema defining the various pieces of information required and use that instead, but I think that’s a much larger maintenance overhead than a multi-part message definition; it is much easier to change a message definition that it is to change a schema; especially in large implementation with many different assemblies;



This was the point most passionately discussed with Rupert today – he argued that if a message is important enough to be published (with all its parts) it is important enough to create a schema for it; the point I was making, possibly not good enough, is that – to me, specifically in this area, the two are the same – I can define my aggregated message in a schema, that would have, say – 2 elements, each using a type defined in my other, existing schemas, or I could use a multi-part message, with two parts, each pointing at existing schemas; there’s very little difference in principal – I’m defining something – strongly typed and well defined,  that is composed of two other, well known things, and I’m passing it around.



Obviously interoperability can only be achieved using standards like XSD, but that’s not a consideration at this point I’m publishing a message from BizTalk, which would be subscribed to by BizTalk, what’s wrong with describing it using BizTalk terms where conceptually there’s virtually no difference?



If there is no difference – why do I care? because multi-part messages are easier to create than aggregated schemas, and there’s less deployed artifact; I can simply put an assign shape and assign existing messages to the relevant parts; if I was to go down the aggregated schema route I would have to define the schema and create a map that would aggregate the messages; I could use code, or I could use xpath to assign the parts to the various elements, but I’d have to use a map or something to create the shell of the message first; seems like a lot of overhead for simply passing a couple of messages between processes.



I agree that many adapters don’t support multipart messages (the file and ftp adapters are the most prominent examples, outside the WCF adapter), and in many cases this makes perfect sense, but some do – SMTP and POP adapters, for example, use it very well; so does the SOAP adapter, of course.



There was one piece where we were in mostly in consensus, about the benefits of a single schema versus multi-part messages – when you wish to have  a map in the port, I believe BizTalk would only consider the body part of the message, but again – to me that’s a shortfall in the product, not a proof that the existing feature is not useful; for both this and the pub/sub scenario I would have lover to be able to treat a multi-part message type in the same way a single-part message is treated – with a specific type that you can subscribe on (I can always set the message type manually, I guess, but would have been great to get out of the box support)



Similarly I agree that the built in xml and FF disassemblers don’t support multipart messages ( only in the sense that they will not disassemble them), but the model does – you can easily access, and parse, multipart messages from within a pipeline component, even a custom disassembler, so I can’t help wondering if the answer is in the following statement made by John Taylor, as quoted in that blog post -




The feature set for the WCF Adapter was potentially very large indeed and we were under some pressure to make cuts where we could.




and I think it’s a big shame.



I, for one, would be very disappointed if multi-part messages stop being supported, and I’m slightly worried – WCF is obviously taking more and more front stage, generally and with BizTalk specifically (even more so with the LOB adapter SDK and the BizTalk adapter pack that makes use of it), and as it does not support multi-part messages, it may well set the tone for future BizTalk developments.

Labels:

2 Comments:

  • Hopefully, after almost two years, I am not breaking NDR too badly by reporting that the issue of WCF's lack of support for multipart messages, and the resulting impedance mismatch with BizTalk Server, was discussed at a private event hosted by Microsoft back in 2007. At the time, there was some discussion about the possibility of injecting BizTalk pipelines directly into WCF channels. Several of the BizTalk people there questioned how this could be achieved given that WCF and BTS do not share a common view of the how messages are structured. The issue of lack of support for multi-part messages was discussed in some detail, and I recall that there was a suggestion that some kind of framing technique might be adopted in WCF to level the two technologies. I frankly doubt that anything will ever come of these ideas (and I never really saw the point anyway), but it was an interesting discussion.

    I broadly agree with what you are saying here. Multi-part message support in BizTalk Server is not perfect, but is a really welcome feature. It is sometimes a life-saver, and, as you say, they can be used to make orchestrations far more maintainable. I don't believe it is likely they will be removed from the product because this would break a lot of existing code.

    I'm not convinced that the wider use of WCF in BizTalk Server poses too much of a problem. For example, imagine that in some future release, Microsoft decides to replace the SMTP adapter with a WCF equivalent. The WCF adapter would have to be written to deal with MIME encoding and so, just like today, we would presumably continue to use a MIME Encoder pipeline component to encode multipart messages (I must say, though, that today's MIME Encoder is not very impressive). In other words, lack of direct support for BizTalk multipart messages in adapters is not really a problem in itself. You generally need to control the framing of message content very specifically when passing messages between different systems, and there is no reason why multi-part framing cannot be done in specialised WCF adapters where appropriate or via more general-purpose pipeline components. I would say here is a reasonably strong case for equipping a future version of BizTalk Server with better and wider out-of-the-box support for framing. Maybe we should have frame-based disassembly features.

    On point where I would diverge slightly with you is the idea that the failure to handle multi-part messages in BizTalk port-bound maps represents a shortfall in the product. BizTalk Server has always tended to emphasise the use of XML, but can, of course, handle any content type (I once created a BizTalk application that passed raw binary data all the way through BizTalk Server, including through orchestrations). If BizTalk maps have a 'shortfall', it is simply that they use XSLT. XSLT is restricted in that it can only act on single XML inputs. Hence, there is no obvious or convincing way in which general-purpose XSLT-based mapping features could ever be constructed to handle multipart messages that may contain a mixture of XML and non-XML part content.

    I have one other observation about this issue. A weakness of BizTalk Server is that there are some serious internal mismatches between the orchestration and pipeline 'views' of multi-part messages, specially in relation to the way that part-specific metadata (context properties) are represented. This can lead to some subtle challenges when doing more advanced stuff in BizTalk Server. At this stage, redesign of these two divergent models would probably result in breaking changes, but it is a pity that a degree of incompatibility crept into the product at such an early stage.

    By Blogger Charles Young, at 14/08/2009, 02:38  

  • A little time has passed before i found this post. Excellent argumentation and eleboration on the topic.

    I myself teach BizTalk (R2006 R2 at this moment). I haven't been able to deep dive into the internals of the 2006 R3 (oops, sorry.. 2009) release of BizTalk yet. As I was researching changes this one hit me hard.

    The thing i'm thinking about is the propagation of this problem.

    Imaging a scenario like:

    WCF -> Orch -> WCF

    (which i think is one of the most 'interoperable' approaches).
    Within my Orchestration I always force the use of multipart messaging since it creates a level of indirection between the BizTalk items (like port-operations and message variables) and the underlying type used.

    One of the major reason for this is the behavior of the Orchestration editor (loosing any link between items using a type!!!) if a type has changed. Since messages are based on schemas, a change will now break my orchestration editor on 'next load' if the schema was changed outside of the Visual Studio environment.

    Since I cannot use multipart messages anymore, how will this affect my projects?

    My hope is that the receive port architecture will take care of this.

    WCF(single part?)->Port(makes multipart, fills in part 0 only)->Orch(works as usual)->Port(make single part?? what happens to part 2..* )->WCF(single part)

    Last i checked WCF, it could use MTOM, Attachments and the likes. I see no architectural contraints in supporting multipart BizTalk messages coming from WCF.

    Therefore, i suspect, Microsoft has a technilogical issue or strategic issue with implementing multipart message support in the WCF adapters.

    I agree with the author of this blog post about the serious miss in this Microsoft decision.

    By Anonymous Marvin Smit, at 19/08/2009, 17:35  

Post a Comment

<< Home