Yossi Dahan [BizTalk]


Thursday, September 27, 2007

Problem due to an overload with a generic type

Here's another interesting one -

Imagine a class that looks something like this -

public class MyClass
public static void test(string a, string b)
public static void test(int a, string b)

From an expression shape in BizTalk you can now use



and everything compiles just fine.

but add the following overload to your class -
        public static void test(string a, IDictionary c)
and now try to use the first overload from an expression shape (which worked just fine before) -


and you will get - "unknown system exception" with no file name or line number to help in sight.

I know BizTalk does not support generics, but I did not think having generics in the another method, one I'm not calling from BizTalk, would be a problem; and in fact - change the call in the expression shape to the overload that takes the int first -
and it would compile just fine.

From this I can only gather that this has something to do with how .net resolves the overload to to use - in the error case, it looks at the first parameter passed in and identifies it as a string, so both the [string,string] and the [string,IDictionary] overloads fit, which means it needs to evaluate the second parameter, and, presumably, this is where it "blows up";
By changing the first parameter passed in to be an int, the compiler does not need to look into the overload that uses IDictionary as all and so it compiles ok.

And indeed, as soon as I add one this overload to the class -

public static void test(int a, IDictionary c)
I get an "unknown system exception" when calling using int as the first parameter.

Labels: , ,

Thursday, September 20, 2007

Limitations of web services support in BizTalk

Are listed in MSDN under the somewhat confusing title "Considerations When Consuming Web Services"

We were hit by the last one this week -

The Add Web Reference do not support the Web Services Description Language (WSDL) import element

The Add Web Reference fails when you add Web references for the WSDL file with the import element.

It didn't quite fail, but it did not import all the schemas it should have (it did some)

Labels: ,

Recursive Orchestrations - follow up

A few weeks ago I've posted this entry about my short, unsuccessful attempt to call an orchestration from another instance of itself using the call orchestration shape and thus introducing recursion.

In comments to that post G Michels pointed out that it seems to be impossible by design according to a very specific error he'd seen.

Adam, however descirbed a very neat approach that is pretty close to recursion (although not quite there in my view) and so I spent some time looking into this. here is the outcome -

The scenario that Adam intorduced is calling an orchestration from another instance of itself. as such this is not exactly a recursive call as, unless more effort is made, the call is completely asynchronouos - the calling orchestration will not wait for the called orchestration to complete, nor will it receive any feedback from it's operation.

The idea is to use the publish-subscribe mechanism BizTalk provides to kickstart another instance of the process; however, Adam's idea of using the Listen shape when activating the orchestration and using partner ports for exchanging messages between instances makes the implemenetation much simpler than it would have been if we tried to do the same thing using basic direct ports and subscriptions and thus it is much more maintainable.

Here's a screenshot of the process I've created following Adam's description -

There are two key points to this idea -

The first one is apparent from the start - the orchestration is started using a couple of activating receive shapes inside a listen shape.

This allows the orchestration to be activated by one of two ports; without this we'd have to fiddle around with promoted properties and subscriptions to get this to work which would have been much more confusing.

In my example the left receive port is late bound to a file drop that would start the process initially.
The right branch is another activating receive shape, directly bound to the message box, but we'll get to that in a minute.

The message I drop has a count attribute initially set to 1, and is configured as a distinguished proeprty.

This allows me to check in the decide shape that whether count is lower than 3 (just so I have some stop condition).

If it is - I contruct a new message by copying the existing one and increasing the count by one.

the message is then sent out using a send port, and this is where the second key point comes into play - the send port is direct bound to a partner port - the second receive port in the same orchestration.

What that means is that, again, without worrying too much about subscription, we are able to publish a message to the message box and ensure it will be picked up by an orchestration of our choice - in this case the very same orchestration, into a particular port - the second receive port linked to the listen shape.

The second half of this - the partner receive port is configured, quite counter intuitively, as Kevin Lam explains in his fantastic post, to have itself as the partner port.

..and so, by sending a message from the partner send port, we are triggering a second instance of the same process in a recursive-like manner.

As I've mentioned earlier this is still somewhat lacking as the calling orchestration will simply carry on (and complete in my test case) and not wait for any called orchestration to complete and will not be able to receive any feedback from such processes.

To support that I had to add a few bits to the scenario, which did increase it's complexity beyond what I would have hoped - here's the modified example -

The first half remains largly unchanged -

As you can see I've added a boolean variable to indicate whether I'm "inside" a recursion or not and am setting it to True on the left branch of the listen shape (after making sure it's defaulted to False) - this would tell me later whether there's something waiting for a response from this instance of the orchestration or not.

Also, for convinience and nothing else, I'm now keeping the iteration number in a variable.

But in order to achieve my goal I had to make a few bigger changes - here's the second half of the process -

The first addition is a receive shape after publishing the recursion request - this would make the process sit and wait for a response from the initiated process before proceeding.

As you can imagine that involved initializing a correlation set in the previous send shape which uses the iteration number, and following on it in the receive shape, which in turn meant I had to introduce a property schema to define my context property (and changing the schema to promote it).

Another consequence of this is that I could no longer use the same messsage type all around - if I did I the receive shape will simply get the published message before any second process is initiated; I had to create a response schema and use that for the received message type.

The second change was - after the resposne is received - to check whether the current iteration is the first one. this is done by checking the boolean variable I've set in the listen shape - if I'm not within the first iteration, I need to publish a response to the message box initializing a similiar correlation set (with the iteration number, but using iterationNumber-1 to link it to the parent iteration) to ensure the calling process is resumed.

This message could be used to provide any feedback to the calling process if needed.

By now it should be obvious this is way more complicated than simply adding a call orchestration shape and selecting the same orchestration, which is all we had to do if BizTalk did support recusrions out-of-the-box. in fact - this grew to be so complicated I think I will have to consider 10 times before choosing to use this approach in any production environment.

Either way - I found this a very interesting exercise to go through. hope you have as well. and thanks to Adam for sharing the idea.

Labels: ,

Wednesday, September 19, 2007

Bug in Serializaion in .net 2.0 - hotfix available

We were hit by this last week - we published a web service with some optional elements; we've called the web service from a client passing all fields, sniffing the call on the wire we see the complete XML, but in BizTalk some elements were missing.

They were removed within the SOAP adapter.

Oleg then found this hotfix information - http://support.microsoft.com/?id=925272 which seems to be exactly our problem

Thursday, September 13, 2007

Answer to a question from U3

"U3" has asked a question in a comment on my previous post -

"Dear Yossi , I have a n Question not in this post , but about how to oublish an bizTalk 2006 Orchestration that takes an Empty Request , hope to get my question..!!!! "

I'll start by saying, not trying to avoid the question obviously (or I wouldn't post this here) - it is always better to post such questions in the newsgroups as more people (I humbly presume) will see them and get a chance to throw in their 2 cents. None-the-less here's my view, and I officially invite others to add theirs as comments to this post if they so wish (not that any official invitation is ever needed)

If I understand correctly - and I'm almost certain I'm not, for reasons that will be apparent soon - "U3" wants to publish a web service that would start an orchestration, but passing in an "empty" request - i.e. without a message.

This does not make sense to me - first of all if you're specifically thinking about BizTalk it is quite clearly a platform driven by messages; nothing happens in BizTalk without a message.

But even if you broaden your scope and look at web services in general - there has to be something that goes over the wire to tell the server "Hey! I'm here, and I need you to do something for me."; this definitely requires a message to go across.

Create an ASP.net web service with a web method that takes not parameters and returns nothing, and look at the WSDL - messages for both input and output will be clearly defined (unless you explicitly flag your web method as one way in which case you could avoid the return message).

For example take the HelloWorld example you get when you create a new ASP.net web service project in VS, change the return type to void and your WSDL representation will look something like -

A message is going in and a message is going out. No way around it.

How else would the server know it needs to do something?!

So - for U3 - If I misunderstood your question please send me a clarification (and I guess you could use a comment on this post now..), otherwise - I hope this makes sense.

Tuesday, September 11, 2007

Web Service and inheritance problem

I suspect WSDL.exe's attempt to be too clever has hit us again - here's a (very graphic) description of what we found -

We have a set of classes that look like this -

If we create a schema for ContainerClass using XSD.exe it looks like this -

All good so far, and indeed serialization and deserialization between the two work just fine.

We’ve then created a web method that returns ContainerClass, as such -

public ContainerClass ReturnContainer()
ContainerClass cc = new ContainerClass();
cc.Items = new System.Collections.Generic.List();
cc.Items.Add(new Item());
cc.Items.Add(new SubItem());
cc.Items.Add(new AnotherSubItem());
return cc;

Looking at the rendering of the WSDL in IE it looks promising -

You can notice the WSDL tells us we will get a ContainerClass with an Items element, and in that we would get Item, Sub Item and/or AnotherSubItem as expected.

What happens if we call the web methods then? (remember that the web methods adds one of each type to the list) -
We've created a windows app, added a web reference and called the web method - here's what we've received back -

You will immediately notice that the first member added to the Items list in the web method - Item - does not appear to reach the client.

Using HTTP analyser we've sniffed for the stream coming back from the web service, and it did contain all three members, the "Item" one was removed in the serialisation from the stream to the class.

As we've tested the serialisation before, we knew it should work, so we had to look at the proxy generated - and this is when the problem became apparent - here's the relevant extract from the generated proxy -

WSDL exe - when generating the proxy - has "kindly" decided to remove the XmlArrayItem option we've explicitly added for "Item" and only included "SubItem" and "AnotherSubItem".
This meant that the client did not know how to deserialise the Item member of the list and it got emmitted from the deserialised class.

By adding the missing XmlArrayItemAttribute following line to the proxy we've managed to get all 3 items without touching the web service.

As this is not something we would like to do in production code we've opened a call with Microsfot to try and see what we can do, but we're obviously open to suggestions, anyone?

Friday, September 07, 2007

Another dive into the SOAP adapter's behaviour

Being keen to fine tune our solution's performance we turned to look into the message exchanges between the SOAP adapter and one of the web services we're usign quite extensively.

Two things became obvious very quickly (well, to be fair - they may have not been without some very useful tips from our colleagues working for the 3rd party providing that service) -

  • All request messages we've sent specify the "Expect: 100-continue" HTTP header
  • Neither of the request set out use pre-authentication, although we explicitly set credentials on the send port

so we promptly turned to look into them -


I'm no network expert; I hold only the basic understanding of HTTP you'd expect from someone dealing with messaging and integration; the way I understand it then is that if a requester sets the "Expect:100-continue" HTTP header on a request, the client would sends the HTTP headers only to the server and then wait to receive a response with "100-continue" status.

It is only when the "100-continue" status is returned from the server that the client sends the actual request.

I also understand that, in order to support older implementations of HTTP (or those that do not support 100-continue), if the client does not receive 100-continue it will send the request anyway, but only after the delay introduced while waiting for the server to respond.

What all that means is about 200ms delay to every message sent from BizTalk.

I'm sure Microsoft have given a lot of consideration to this before deciding this would be the default behaviour of web services in .net but to us, communicating with a long-term partner over a good leased line, it would have been nice to be able to turn it off and skip the extra step.

Unfortunately, although spending some time on this we could not, as of now, find a way to do so.

Our best bet was (and still is) to set the relevant setting in the ServicePointManager in our custom web service proxy, but this does not seem to have the desired effect.

From a bit of reading we've done it seems there's some element of re-use here and that the ServicePointManager may use an existing service point, in which case our newly set property will not be used.

It could be that this is created somewhere in the adapter before our custom proxy or something like that, we truly don't know at this point.


Fortunately this is an area we’ve had better success in, and it's good because it has a significantly bigger impact on performance than the 100-continue problem.

The web service I've mentioned before uses basic authentication on the wire and then further elements of security in the message, so on the BizTalk port we've set the credentials the SOAP adapter should use when calling this web service, and, as you'd expect, this is working just fine.

However, analysing the traffic going out of BizTalk on the wire we've noticed that for each message we send to the 3rd party, the SOAP adapter would transmit the message out without the security HTTP headers first, receive an HTTP 401 (Unauthorised) error and only then send the request with the HTTP headers baring the credentials we've set in the send port.

Obviously, and especially when relatively large messages are involved, this has a significant impact on the performance of the server (and the farm as a whole as more bandwidth is used for the message exchange).

Again, doing some reading and experimenting, we've learnt that this is pretty much down to the choices made around the DEFAULT behaviour of .net, and again, we thought, we had no control over it.

But, while we could not do anything if we were using the standard configuration of the SOAP adapter, the fact that were using a custom web proxy to call the web service (for other reasons) meant we could try and tweak the way messages were transmitted, and tweaking we did.

It didn't help us in the 100-continue case, but it definitely helped with the pre-authentication requirement - all we had to do is add a line setting the PreAuthenticate property of the HttpWebRequest to true before using it and BizTalk (well, the .net framework web service stack it uses) would quite happily provide the credentials up front avoiding the need to duplicate the round trip to the 3rd party server.

Much better now!

Labels: , , , ,

Thursday, September 06, 2007

Unnecessary limitation when using custom proxy for web services in BizTalk 2006

We're using a third party web service quite extensively; sometimes the size of messages exchanged between us can be quite big, which is not ideal from a performance point of view.

Luckily - they are quire advanced and mature and have a good support for compressed messages exchange.

To use that we needed to compress the stream going out over the wire through the adapter and set the Content-Encoding HTTP header to "gzip" to indicate the stream is compressed, on their end, if the header is set that way, they would decompress the stream before processing the message , and will compress the response in a similar way.

They were even kind enough to provide us with a class that enhances System.Web.Services.Protocols.SoapHttpClientProtocol (through inheritance) to do all of that, so really - all we needed to do to use their web service in our playground windows application was to change the proxy class generated by Visual Studio when we added the web reference to inherit from their class rather than SoapHttpClientProtocol (their class would ultimately ingerit from SoapHttpClientProtocol).

We've played with it for a while and it worked great, which meant we were ready to move it from our playground application to our BizTalk implementation; As you can guess by the fact I'm bothering with this post - it didn't go smooth…

I've played with custom web service proxies before - generated a proxy from a WSDL and configured the port to use it - so I knew that in general principle that works quite nicely;

And so I went on and generated a proxy class for that web service and changed the inheritance to the class they've provided.

When I tried to select it in the Admin Console as the proxy class to use, I could browse to the assembly I've built but, once selected, I would get the following error:

This did not make sense to me - the proxy I generated inherits from their class which, as I have the source code, I'm sure inherits from SoapHttpClientProtocol. This error did not make sense to me.

Trying to figure this one out I went on to create my own class ("TestClass") that inherits from SoapHttpClientProtocol (and had not other implementation otherwise) and changed the generated proxy to inherit from that - so basically I created the same scenario - proxy inherits from class inherits from SoapHttpClientProtocol.

When I tried to select my updated assembly in the admin console I could not select the original proxy (now inheriting from TestClass), I could, however, select TestClass itself (which inherits directly from SoapHttpClientProtocol ) but only to get another error -

This error was quickly solved by adding a public web method to TestClass; now TestClass could be selected in the UI, my proxy class, however, the one I really needed, that does not inherit directly from SoapHttpClientProtocol, could not be selected.

It has also solved the mistery, or so I I believe - it seems the UI will only display classes whose base type is SoapHttpClientProtocol, when it should havem in my view, displayed all classes whose IsSubclassOf(typeof(SoapHttpClientProtocol)) is true.

For us the solution was simple - we took all the code we were kindly given from the 3rd party and merged it into our proxy class which could then be left to inherit directly from SoapHttpClientProtocol . The problem is that this is quite limiting - there's no real scope for re-use and for no good reason.

In fact - I suspect this is just a design time problem, of the kind we spotted quite often recently where the runtime would actually work with something, but you have no way of configuring BizTalk to use it.

Labels: , ,