The scenario explained
There's a scenario I bumped into for the second time last week and it always poses some dillema - that is a scenario in which you want to expose a synchronous interface to the external world but do the work asynchronously in the backend.
Here are two examples of when such need can be useful -
1. Pure messaging - you are exposing a synchronous web service, when a request comes in you need to forward it to another service, possibly of a third party, and return the response, maybe do a couple of transformations in the middle.
That other service is implemented for asynchronous communication.
2. Orchestraction - you are exposing a synchronus web service in the front end because your client cannot implement a call back for async.
As a request comes in a single process needs to be executed on it, but the decision which one is made at runtime and not design time (according to data in the message)
This means you cannot simply bind the orchestration two-way port to the two way SOAP receive port.
While the two scenarios are different, they share the same problem - how to route the message coming back (from the third party async callback or from the dynamically selected orcehstartion) - to the instance of the receive port that accepted the initial request.
When BizTalk publishes a message from a two way receive port into the message box it creates an instance subscription, so when the response message is published back to the message box it will get routed back to that instance of the receive port to be returned to the original sender.
This subscribtion is based on a correlation token which is generated and written to the EpmRRCorrelationToken system context property (as well as a "true" value in the RouteDirectToToTP)
In a classic scenario where a two way send port subscribesto a two way receive port, when the send port publishes the response tothe message box BizTalk will ensure that the EpmRRCorrelationToken has the correct value and is promoted.
Looking to implement the second scenario described above, I've created an orchestration that subscribes to the message box to pick up the request message using a directly bound two way port.
The process then does all that needs to be done and when it finished it published the message to the message box through the send side of the two way receive port it got the initial request from.
Following my understanding of the correlation logic described above I make sure to set and promote the values of both EpmRRCorrelationToken and RouteDirectToToTP (to be accurate the EpmRRCorrelationToken has already rather magically added to my newlycreated response message, but it was not promoted). I do this by initialising a correlation set with these values on the send shape.
To my surprise - the message published to the message box had the EpmRRCorrelationToken property in a not promoted state, regadless of my best efforts. It seems as if BizTalk is actually ensuring that it is not promoted
This, of course, causes a routing failure as the subscription cannot befilled without this property.
Error by design
It seems that Microsoft has decided to force such implementations to fail by design. Or so it was explained to me a while ago.
The reason being that (as far as I understand), since the orchestration in my scenario is not tightly bound to the two way receive port, it is possible, theoretically, that more the one subscribers (orchestrations or others) could get the message and return a response, in which case BizTalk will have no way of knowing which response should actually be returned to the sender.
So, taking into this into account, I need to create an orchestration that will be tightly bound to the receive port at design time.
This orchestration will then get every message from the receive port, and pretty much publish the message back to the message box using a "direct" send port, only that I will make sure I initialize a correlation set on the sand shape to help me get the response back into this instance of the orchestration.
This, of course, involves coming up with some correlation token, defining a correlation type that uses it, and then making sure it's in the message context before the message is sent.
The other orchestration (the one we really intended to execute, and selected dynamically based on the message context) will then kick in and pick up the message, and when it complets its work it will re-publish the message back to the message box ensuring the correlation token provided by the generic orchestration exists in the message and promoted to its context (this is most likely do be done, counter intuitively maybe, by initialising the correlation set on the send shape).
This will ensure the message will get routed back to the correct instance of the generic orchestration and through it back to the sender synchronously.
The main problem with this is that it creates the same sort of problem we had anyway. If more then one orchestration picks up the request a race condition is created. In this case, since it is a private implementation BizTalk will allow it and any responses after the first one will become zombies.
So - we got what we could have had anyway if Microsoft wouldn't decide to "protect" us from it, only that now we have another orchestration on the way which is of course a performance hit.
Personally I would prefer if BizTalk would behave the same - let us publish the response from whatever and return the first response coming in, but at least there is an easy way to get this to working and, from my experience, as long as you're careful with your subscription, it is working quite well.