Yossi Dahan [BizTalk]


Saturday, December 26, 2009

UK SOA/BPM User Group meetings in January

The next ‘in-person’ meeting for the UK SOA/BPM User Group will be held on the 27th of Jan in Paddington – London, check out the details here, and if you’re coming – be sure to come and say hi – I’ll probably be the loudest chap in the room. again.

And even before that, on January 12th, an online meeting will be held with the title ‘Developing Composited Solutions with Sharepoint and SAP’

Friday, December 18, 2009

This should not be a routing failure!

Important update: it looks like it’s not after all! and I’ve posted the details here; I suggest you check it out after reading this post.

Some routing failures are straight forward – you have a message, it has context, and no one subscribes to it. Routing Failure Report notifications are very useful on helping troubleshooting these – and most likely someone has mistyped a property’s value, a subscription value or forgot to ensure that the relevant properties are promoted and that relevant services are enlisted.

Some routing failures are a little bit more tricky to map out -

Consider a simple scenario where you have a two way receive port publishing a request, which is being picked up by a two way send port -


Now add a second two-way subscriber to the mix…


In this case BizTalk cannot know which of the two potential responses it may get (from the two different two way send ports) it should deliver to the [send side of the] receive port, and so it opts to not even publish the request and instead deliver a routing failure.

This is fair enough - race conditions are not a good thing and it’s pretty clever that BizTalk identifies this up front and prevents this.

However - the same routing failure will happen, it appears, even if the publisher is NOT a two way port, and that – I did not expect.

If the publisher of the original message (the receive port) was not a two way port, but one-way, it would not expect any response from anybody, which removes the race-condition situation, and therefore should have been supported in my view, but it does not appear that it does.

Imagine a scenario like this -

1st Process publishes message A, through a one way port
1st Send port (2-way) picks up message A and calls a service, publishing the service’s response - Message B.
2nd Send port (also 2-way) picks up message A as well and calls a different service, publishing this service’s response - Message C.


As far as I can tell, this is a valid scenario – call it message enrichment via external services.

Subscribers should be able to subscribe to message A, B or C as they wish, and all is nice and loosely coupled, if you only ignore the fact that this does not seem to work!

Building such a scenario I'm getting a routing failure for message A, which - unless I'm not seeing correctly - I can only explain according to the rule mentioned above, which I find really frustrating.


Sunday, December 13, 2009

The Microsoft Application Platform Event - February 9th 2010

Microsoft UK have published the details of this event taking place in the MS Campus in Reading, see the details and register here

The event has a very promising agenda -

I had the pleasure of meeting Robert Hogg several times through the MVP program, and he is probably one of the most suitable guys I know to be talking about pragmatic approach to projects with a track record of successful “rescue projects” under his belt.

I also have the pleasure (although I’m not sure it’s mutual) to be working closely with Paul Saggar and Jon Simpson, and if anybody can talk about implementing complex business processes in an ever changing, and highly competitive, environment it’s these two.

And of course, as many know, I work closely with PNMsoft, and so its only natural that I’d say it’s always very interesting to hear from their vast experience in the BPM field in which they’ve been working for many years.

See you there!

Friday, December 11, 2009

What not to do – avoid reading entire message’s to memory unnecessarily

This one is fairly common, I suspect, and I can certainly see why – the temptation is simply to big – but too many pipeline components start by reading the message into memory, when, with a little bit more effort this could have been avoided.

One pipeline component I’ve seen, for example, receives a flat file, and needs to remove records already processed (duplication elimination) – quite a good thing to do in a pipeline, and I also liked the approach of doing so before the disassembler, to make the xml produced smaller.

V1 of the component used a memory stream – the incoming stream was read line by line, each line was assessed, and – if was not a duplicate – it would get written to the memory stream.

When the component had finished going through the entire incoming stream, the memory stream would be assigned to the message, replacing the original stream, and the message would get returned to the pipeline.

There are two downsides to this approach – the first is memory consumption – the component will always consume at least as much memory as the size of the (outgoing) message; done properly BizTalk would then clean this memory, but only after completing the processing of the message; the second downside is potentially unnecessary delay in further processing of the message – one of the huge benefits of the pipeline, in my view, is its streaming fashion, where subsequent components, if developed in the correct manner, can start working on parts of the message before preceding components completed their processing; basically each component passes back to the pipeline the portion of the message it already processed, whilst working on the next portion.

It appears that the customer in question encounter memory issues as the component’s code was changed to use virtual stream instead of memory stream; a virtual stream is effectively a stream that uses disk for storage instead of memory.

This solves the memory consumption issue, but merely replaces it with IO operations which may have an even bigger impact on the server’s overall performance (and does not address the processing delay point at all).

What would have been the correct way to implement this in my view?

The component should have create a custom stream, wrapping the original stream from the message; It would then replace the message’s stream with the custom stream immediately returning the message back to the pipeline. Note that so far the component hadn’t touched the message stream – zero bytes have been read.

As BizTalk (and not the component!) would read the message (for instance when persisting it to the message box), the custom stream’s read function would be called which would contain that reads the underlying stream (the original stream received by the component), probably buffering reads until the end of a line for simplicity (although in many cases this is not necessary) and assessing whether the record is a duplicate or not; if it is a duplicate the function will simply read the next line and so on until a non-duplicate record is found, at which point the line would be returned as the output byte stream from the read function.

This effectively means that the next component, or the message box, will receive the message line by line, duplicate records removed, without having to wait for the component to process the entire message, and with only a maximum of one line ever loaded into memory.

Labels: ,

Friday, December 04, 2009

LOB Adapter SDK

I’ve been doing some work recently with the LOB adapter SDK.

The adapter I’ve been creating is effectively a wrapper around a web service we’re using extensively, to which we often need to make several consecutive calls.

Doing so from an orchestration, using a send port, is inefficient, as each request involves a pub/sub cycle; on the other hand – I’m not a big fan of ‘inline sends’, which was one of the options considered.

Building a custom adapter would give us the best of both worlds, if you like – the process would create a message containing all the requests to be sent and publish it; a send port, configured with the custom adapter, would pick this message and deliver it to the adapter which would, in turn, debatch the requests, send them to the service, aggregate the responses back and return the aggregated message.

This way we still have all the tracking and management BizTalk provides but only one pub/sub cycle.

The LOB adpater SDK does a very good job in lowering the barrier of creating adapters (I’ve created ‘native’ adapters before) which is a good thing.

Several points I’ve been left with from this, very initial, exercise -

Implementing the metadata browser, resolver and search handler was a bit tedious; I ended up creating an xml file with my metadata, embedded this in my project and wrote code to generate the metadata off that. I wish this was part of the SDK.

Once the metadata was set I needed to write the outbound handler (in my case), for which I needed to know how the incoming messages and outgoing responses should look like; with no generated samples this was a bit too painful to my liking – figuring out the request format is easy although tedious again – I’ve ended up consuming my adapter from a client, making a call, setting a breakpoint and going over the message at runtime.

This is, of course, not possible for the response – where I need to create the message in the adapter, so trial and error was the only way (ok – and reading the docs, and applying some common sense) – in my case I actually made a mistake in my metadata configuration which took me a while to figure out (and Rupert’s help, thanks!)

Bottom line is that if there was a tool to look at the metadata and generate request/response samples for each operation this work would have been much easier. I might be tempted to create one if I get the time.

The last thing that phased me was working with metadata from the client.

I needed the client to provide some information outside the message; I knew the mechanism in BizTalk would be to use a context property, for which I will need to deploy, with the adapter itself, the property schema, but I wasn’t sure if and how the adapter gets access to that or how I could consume this from a .net client.

Figuring out the adapter access was easy enough, turns out all the BizTalk message’s context properties are added as properties to the request WCF message received from the framework, so

token = message.Properties[PROPERTY].ToString();

where PROPERTY is the context property’s namespace#node would return the value of the property requested.

I was a bit stumbled with how to provide this property outside BizTalk, this time Manas Garg came to the rescue in this thread

Turns out you can simply do -

ConcurrentPrograms_ARClient client1 = new ConcurrentPrograms_ARClient(binding, address);

using (new OperationContextScope(client1.InnerChannel))


          OperationContext.Current.OutgoingMessageProperties.Add("Property Name", "Property Value");



Labels: ,

Tuesday, December 01, 2009

What not to do – ‘bundle’ unrelated components together’

So – here’s another real-world they shouldn’t have done this –really! item; somewhat related to my previous post about projects, but a different slant -

MyP had used pipeline components fairly extensively, a very good idea in my view, especially in this case.

But, as discussed in that previous post all pipeline components, regardless of where they were used, were bundled together in a single assembly.

To make matters worse, a single pipeline component often served more than one purpose, for more than one client.

So – for example – a pipeline initially created to remove an unwanted trailer from a message from a particular sender, ended up also converting the message to xml, and then extended to support another format, from another sender, only that the two don’t share any code – the execute method of the pipeline component has a switch statement on the sender name, and runs two separate functions.

Now – considering all the components are in the same assembly already, how can this make matters worse?

Well – single responsibility principle is one that I generally like – I’m a new developer working on this project, I see a component called TrailerRemover, used in a pipeline called <someSender>_Receive I assume this is processing messages from <someSender> and that it removes a trailer.

I eventually discover it does a lot more, and processes messages from another sender as well.

One of the side effects of this is time wasting – it is much more difficult, in my experience, to maintain systems that don’t follow the single responsibility principle.

This is aggrevated by the fact that this is usually a symptom, if not a cause, for bad architecture – I shouldn’t be able to mix logic for two different senders, not unless specified common logic is factored out and reused properly, from a shared assembly.

Now when I come to change some code I find it difficult to know what the impact may be – where exactly is this thing used?

Labels: ,