Yossi Dahan [BizTalk]


Wednesday, June 28, 2006

Xml declaration gotcha

This is one of those things that you spend an hour finding out and another half a day banging your head againts the wall and wishing all sorts of things to people you've never actually met (and are probably very nice).

I've been running a simple biztalk test orchestration that, as with most test processes, starts through a message arriving at a folder.

My message was generated by an application written by another department.
As soon as a message is dropped in the file drop it's being picked up by Biztalk as expected, which reacts with an immediate failure in the receive pipeline (I've used standard XmlReceive) and with an error message in the event log -

There was a failure executing the receive pipeline:
Microsoft.BizTalk.DefaultPipelines, Version=, Culture=neutral,
PublicKeyToken=31bf3856ad364e35" Source: "XML disassembler" Receive Port:
"***********" URI: "**********" Reason: Length cannot be less than
Parameter name: length

It took me a while to figure this out, but the bottom line is -

The problem was in the document's xml declaration. the application used single quotes around the attributes -

<?xml version='1.0' encoding='utf-8' ?>

- which is perfectly legal as far as I understand according to the w3c definition here .

Apparently BizTalk does not like that.
Changing the xml declaration to use double quotes -

<?xml version="1.0" encoding="utf-8" ?>

- solved the problem, which was easy enough to fix but I wish we didn't need to...

If this is a valid xml declaration I wish BizTalk would support that, if it's not - I'd love to hear from someone.

Friday, June 23, 2006

Setting a custom SOAP header to null

Again - this is related to my two posts from today.

I've already mentioned the facts that you cannot have a decision inside a construct shape and you cannot set a context property to null.

Now I'll add the last layer on top of this -

Our context proeprty is actually a custom SOAP header implementation.
I will not discuss this in detail here as it is described quite nicely in the documentation (at least in 2006).

The bottom line is that to add a custom SOAP header to a response going out of BizTalk you need to set a context property value to an xml as it needs to appear in the SOAP header.

In our case we needed to do this conditionally. so in some cases we needed a header and in others we didn't

we could have had a decision with two branches, and a construct message shape in each one that would construct the response and then set the context proeprty in one branch and not set it on the other. but this would mean duplicating all the logic of the message creation which is not a good idea.

so in our implementation we used an external web method to construct to soap error header.
When a header was needed the method will return the xml as it should appear in the header.

if a header was not needed the method would return only the root node of the xml and add the xsi: nil attribute.

When the SOAP adapter serialises the message it will omit the header alltogher - which is exactly what we wanted.

bare in mind you will have to set the root node to nullable in the schema, and you will need to define the namespace declaration for xsi in the xml you return. so it would look something like:

<http://www.w3.org/2001/XMLSchema-instance xsi:nil="true" />

Wish list: setting context properties to null

I wish it was possible to set a context property to null.
If you try to do that now an exception is thrown (and the property is not set)

This is somewhat related to my other wish list post from today about being able to use decision shpae inside a construct message.

But basically - say you need to set a context property to a value only if a certain condition applies. if you could have a decision shape inside the construct container that would not be an issue, but since you can't the only way to do that is to pass your message together with whatever information you need to make the decision to a .net method in an external assembly that will make the decision and if needed use the message API to write to the message context.

Another approach I hoped would work (again - ignoring the decision shape option) is to call an external method to make the decision and have it return the value to set or null (if there's nothing to promote).

This would make perfect sense as even currently - if you try to read a context property that does not exist - you get a null reference. so it follows the same idea of a null proeprty value when not set. (which, btw - is a pain by its own)
This would also help us avoid needed all those nasty references

But evidently - you cannot assign a null value to a context property, so that does not help.

Wish list: Decision and loop in message construct

I wish it was possible to have more complex logic in a message construct and not simply an assign or transfor shape.

For instacne - a decision shape could be very useful.

For instance - imagine you construct a multi-part message where one part is always a result of a map and the other parts are assignments of other messages, only that the source message you need to use depends on something...

Another, even more common scenario is when you are constructing a message and then assign some context properties. but you need to assign different values, possibly to different properties. again - based on something. doesnt matter what.

There are easy enough ways around that, but not being able to have an if-then-else
logic in the construct shape makes the process quite big and ugly (and wasteful)

To push it a bit I even believe it would be quite useful to use a loop in a construct shape. for instance for a message aggregation. but that might be too much to ask for, I admit.

Tuesday, June 20, 2006

Lifetime of .net assemblies called from a map..

(or custom xslt)

I should have done this a long time ago, but I did not, until last week, test how are assemblies used from within a map (using the scripting functiod) or from custom xslt reused by Biztalk (if at all)

It seems that, as with any other assembly in the messaging side of BizTalk it loads them on first use and keeps them in memory for re-use.

This is excellent news if you are making sure they are absolutly stateless (as you should) and obviously not so good news if you counted on a fresh instance created for each translation.

Published web services and optional nodes in schema

This post is a third in a series of posts about the web services proxies generated by visual studio when you add a web reference.

While the previous posts were pretty general .net stuff, this one is much closer to the BizTalk realm.

When publishing a web service from BizTalk, the definition of the arguments and return type is derived from the schema (this is true also when publishing an orchestration as a web service, as the orchestration port has a schema associated with it).

This highlights a scenario not usually found when creating “standard” web services – optional arguments.

We’ve recently found out (the hard way) that when you are adding a web reference to a web service that has optional elements of types that are not nillable (such as dates, booleans or integers) in the schema defined in the WSDL some special behaviour is triggered in the code generator.

Here’s an example - say you have a scheme with a root node called MyMessage and and two elements – MyRequiredElement and MyOptionalElement.
Now consider that MyOptionalElement is both optional (as you must have guessed) and integer (which cannot be null)

When you expose such a schema as a web service and add a web reference to it from a client, the proxy class generated will have three properties – MyRequiredElement, MyOptionalElement and MyOptionalElementSpecified.

So – if you have a property that looks like this –

public int MyProperty
return this.myPropertyField;
this.myPropertyField = value;

This one would be generated as well –

public bool MyPropertySpecified
return this.myPropertyFieldSpecified;
this.myPropertyFieldSpecified = value;

The reason for the third property is to allow .net to distinct between a case where no value specified for MyOptionalElement and when a value was provided which is the default value for this type (for instance – 0 when dealing with integers)

To understand the problem consider a scenario where you're submitting a message that does nto contain an instance of an optional node, and that node is of type int.

When the .net deserialiser takes in that message and tries to deserialise it, it has to give a value to that int property , so it will use the default value for the type – 0 (again - this is because int cannot be null)

Now the server taking in the message has no way of knowing that a value was not really specified by the client, but that a default value was specified by the deserialisation logic.

That’s where the ___Specified property comes into play. It will indicate whether the value was specified in the message or used as a default value.

This is quite all right once you know it’s there, but if you don’t it’s a cause for a major headache.

We've actually encountered this from the other direction as we’re not dealing with the server side, but with the client side (BizTalk being the server side). Not being aware of the __specified logic we’ve created a windows client for our published orchestration and used it to call the web service.

We populated the request object’s properties with values and we're very confused when the message arriving to BizTalk was empty.

Apparently the .net serialisation logic has some black magic in it.

When the serialiser creates the message from the object, it checks, for each property its serialising, for the existence of an equivalent __Specified property.
If such property exists it uses it value (true/false) to decide whether to serialise this property or not. If a __Specified property does not exist it does not serialise the property.

Since we did not set the __Specified proerpties to true none of our values we’re actually serialised into the message and therefor it arrived to BizTalk empty.

Again – once we found out the cause the solution was quite simple – generally all we needed to do is make sure to set the __Specified properties for each property we used to "true".

Although this makes the code quite “dirty” so we’ve decided to alter the generated proxy code so that the Setter of each “real” property sets the equivalent __Specified property.
I really don’t know why this is not how the code is generated code by default.

So – this can be a major gotcha and another reason we ended up tweaking the generated web service proxies.

Thanks to Ben Gimblett, John Plummer and Xen Lategan for all their help on this one

Friday, June 16, 2006

Changing an underlying custom xslt does not cause a project to build

A side note:

When you have a map that uses custom xslt file and you make a change to the xsl the project is still being identified by Visual Studio as "up-to-date" and therefore when you will build it it will actually be skipped.

You have to select Rebuild from the Build menu (or the project context menu) to ensure your project is being built and the changes are reflected.

and I always forget!

Friday, June 09, 2006

Web services and generics

In my previous post I've talked about the somewhat pointless casting that usually occurs when you need to pass objects between your client and a web service.

The solution to this was to tweak the generated proxy class to use the shared definition of the classes rather then athe local, generated, definition of them.

Another requirement that sent us changing the proxy is the use of generics in our shared classes.

Generics have proven to be very convinient and useful to us and we use them quite frequently in our project.

However, when we started to expose web services that takes or returns our objects that use generics we've noticed that in all the generated proxies the generic memebrs have been converted to arrays.

For instance - a member defined as List(Of String) (we're using VB.net) has been changed to be an array of strings.

This meant that we cannot use the code we already have to use these objects as it assumes the members are defined as lists and not arrays.

the solution is to modify the generated proxy to use the generics form. (in our case, since we removed the local definition of the class anyway as described in my previous post, we "killed two birds with one stone" as the original class definition, now used by the proxy, uses the list types)

While you do have to be a bit careful with this approach when exposing such services to external paries, as you can't assume they can use generics, it's definitely useful for internal service.

Howerver, a lesson we learnt while hooking all of this to BizTalk (which needs to expose services that take these objects) is that on the wire, List (Of T) and Arrays look the same. which means they are quite interopable (and that you can use this form of generics with BizTalk as well)

Web Services that take custom classes as parameters

While I’m surprised it didn’t hit me before this subject hit us this week from 3 different angles, which I will describe in the next 3 posts -

One unnecessary casting

When consuming a web service that take an instance of a custom class as a parameter visual studio generates an equivalent class to the one defined by the web service. However – as this is not the class used by the web service but an equivalent class, it is not the same type, which means some sort of casting is required -

Say you have a class called MyClass in namespace MyNamespace defined in an assembly called assemblyA

You then create a web service, which references assembly A and has a single method - MyMethod(MyNamespace.MyClass)

You then move on and create a win form client. Again you reference assemblyA so you could created an instance of MyNamespace.MyClass to pass it to the web service

You go on and add the web reference; at this point Visual Studio generates the proxy to the web service for you, neat, only that it is not aware that you know the definition for MyNamespace.MyClass so it goes and creates a local definition of MyClass.

However, since it’s created as part of the generated web service’s proxy the generated class will be created in the proxy's namespace (say – localhost). So - the new class is localhost.MyClass while you have MyNamespace.MyClass

This means that you can’t just pass MyNamespace.MyClass to the web service. You have to create a localhost.MyClass instance. There are a few ways to convert one to the other, neither of them is very nice and all of them mean casting of some sort which is inefficient.

Of course, if all the logic happens in the winform app you could start with localhost.MyClass to begin with and then you don’t need any casting. But what if MyNamespace.MyClass is shared throughout your enterprise and you get it from somewhere else?

The solution, as introduced to me by Jon Simpson, is simple and elegant, but may raise some concerns from a maintenance perspective -

By intorducing the original class to the generated proxy we can avoid the need to use a localy defined class. to do this we add a “using” statment in the generated to introduce the original namespace (MyNamespace) and then remove the definition of the local class generated by the proxy.

This will make the proxy use our original definition of the class, which means we can now pass MyNamespace.MyClass to the web method. Since the two classes were identical to begin with (one was generated from the other) – they are identical on the wire and therefore no changes are required on the server which already coded to accept this class

As I’ve mentioned – the only problem with this approach is that you are modifying a generated code, which means you will have to modify it again if you update the web reference. A simple macro to do it for you might ease this small issue.

Wednesday, June 07, 2006

Strange Visual Studio behaviour

I hate it when a tool I use behaves in ways I don't understand, and even more so when it is so inconsistent, but I have had a strange error with VS.net 2005 recently which has all the indication of being a bug.

Then again - it might just be me doing something wrong...

I have an orchestration with a multi-part message.
One of the parts of that message is of type System.String. (The other is a schema)

Sometimes, when I try to build my project I get the following compile time error -

"'System.String' must be Xml serializable to be a message part type -- check event log for more information"

Despite what the error says - there's no further information in the event log as it suggests

Now - what stuck me as very strange is that last time I checked System.String was serialisable, and besides - I've definitely used System.String parts before and it worked just fine.

More then that - I also consume a third party's web service who's taking in a string as a parameter. adding a web reference to that Web Service generates the web messages for me with a System.String part that works just fine, so it's quite clear the problem is not with having a System.String part.

Anyway - After a couple of hours of chasing my tale around this I did the only sensible thing to do - I restarted visual studio.

The problem was gone. at least for a while, I've been able to compile it and run for a while before it came back again.

Restarting VS.net solved it everytime. Great!

BTW - This had happened for us on several machines, all of them built by hand, using RTM version of VS.net and BizTalk 2006.

Monday, June 05, 2006

Publish orchestration as web service woes

Hopefully I’ll save someone the time we had to waste today on this one –

We’ve created an orchestration, published it as a web service and created a test client to call the web service, all deployed and tested. Standard stuff.

Then we decided we need another part to our message, so we went back to the orchestration and changed the message type definition to include that second part. since I use multi part messages for every message anyway, adding a part was quite easy.

For the sake of being accurate I’ll mention that we have an empty orchestration in another, referenced, assembly in which we declare types that we share between orhcestrations. this makes things a little bit easier when many assemblies are involved.

After changing the message type we built both orchestrations and re-run the web services publishing wizard, but to our surprise the generated web service did not have the additional part. It was generated as if we did nothing.

After checking ourselves about a million times, and seconds before I seriously considered a career change, we figured out that it was all because the old assemblies, previously deployed to allow us to run the tests, were still in the GAC.

Since we are still in development mode, we did not increment a version of the assembly. With an assembly of the same version in the GAC I assume that when the wizard need to load the assembly it took the GAC-ed one rather then loading it from the path we’ve provided in the first step of the wizard.

This is very confusing and I’d wish it would load the assembly form the path provided to it as expected or at least warn if it’s going to take another one.

Anyway, once we’ve discovered the problem it was, of course, very easy to correct. We simply deleted the previous version from the GAC and the new one was picked up all right.

Sunday, June 04, 2006

Sync to Async conversion

The scenario explained

There's a scenario I bumped into for the second time last week and it always poses some dillema - that is a scenario in which you want to expose a synchronous interface to the external world but do the work asynchronously in the backend.

Here are two examples of when such need can be useful -

1. Pure messaging - you are exposing a synchronous web service, when a request comes in you need to forward it to another service, possibly of a third party, and return the response, maybe do a couple of transformations in the middle.
That other service is implemented for asynchronous communication.

2. Orchestraction - you are exposing a synchronus web service in the front end because your client cannot implement a call back for async.
As a request comes in a single process needs to be executed on it, but the decision which one is made at runtime and not design time (according to data in the message)
This means you cannot simply bind the orchestration two-way port to the two way SOAP receive port.

While the two scenarios are different, they share the same problem - how to route the message coming back (from the third party async callback or from the dynamically selected orcehstartion) - to the instance of the receive port that accepted the initial request.

The problem

When BizTalk publishes a message from a two way receive port into the message box it creates an instance subscription, so when the response message is published back to the message box it will get routed back to that instance of the receive port to be returned to the original sender.
This subscribtion is based on a correlation token which is generated and written to the EpmRRCorrelationToken system context property (as well as a "true" value in the RouteDirectToToTP)

In a classic scenario where a two way send port subscribesto a two way receive port, when the send port publishes the response tothe message box BizTalk will ensure that the EpmRRCorrelationToken has the correct value and is promoted.

Looking to implement the second scenario described above, I've created an orchestration that subscribes to the message box to pick up the request message using a directly bound two way port.

The process then does all that needs to be done and when it finished it published the message to the message box through the send side of the two way receive port it got the initial request from.

Following my understanding of the correlation logic described above I make sure to set and promote the values of both EpmRRCorrelationToken and RouteDirectToToTP (to be accurate the EpmRRCorrelationToken has already rather magically added to my newlycreated response message, but it was not promoted). I do this by initialising a correlation set with these values on the send shape.

To my surprise - the message published to the message box had the EpmRRCorrelationToken property in a not promoted state, regadless of my best efforts. It seems as if BizTalk is actually ensuring that it is not promoted

This, of course, causes a routing failure as the subscription cannot befilled without this property.

Error by design

It seems that Microsoft has decided to force such implementations to fail by design. Or so it was explained to me a while ago.

The reason being that (as far as I understand), since the orchestration in my scenario is not tightly bound to the two way receive port, it is possible, theoretically, that more the one subscribers (orchestrations or others) could get the message and return a response, in which case BizTalk will have no way of knowing which response should actually be returned to the sender.


So, taking into this into account, I need to create an orchestration that will be tightly bound to the receive port at design time.

This orchestration will then get every message from the receive port, and pretty much publish the message back to the message box using a "direct" send port, only that I will make sure I initialize a correlation set on the sand shape to help me get the response back into this instance of the orchestration.

This, of course, involves coming up with some correlation token, defining a correlation type that uses it, and then making sure it's in the message context before the message is sent.

The other orchestration (the one we really intended to execute, and selected dynamically based on the message context) will then kick in and pick up the message, and when it complets its work it will re-publish the message back to the message box ensuring the correlation token provided by the generic orchestration exists in the message and promoted to its context (this is most likely do be done, counter intuitively maybe, by initialising the correlation set on the send shape).

This will ensure the message will get routed back to the correct instance of the generic orchestration and through it back to the sender synchronously.

The main problem with this is that it creates the same sort of problem we had anyway. If more then one orchestration picks up the request a race condition is created. In this case, since it is a private implementation BizTalk will allow it and any responses after the first one will become zombies.

So - we got what we could have had anyway if Microsoft wouldn't decide to "protect" us from it, only that now we have another orchestration on the way which is of course a performance hit.

Personally I would prefer if BizTalk would behave the same - let us publish the response from whatever and return the first response coming in, but at least there is an easy way to get this to working and, from my experience, as long as you're careful with your subscription, it is working quite well.