Yossi Dahan [BizTalk]

Google
 

Friday, December 17, 2010

More on BAM

In some places we’re using inline WCF calls to services for various reasons (a practice I’m not entirely comfortable with, but I know many people advocate, so I guess the jury is out….)

One of the problems with this approach, in our case, is that we’re bypassing the elaborate tracking infrastructure we have in place which relies, largely, on BizTalk pipelines that don’t exist when using inline sends.

The actual solution is a topic for its own blog post, hopefully in the near future, but when I conducted some load tests of it I saw more of the phantom records in BAM, which meant I had to look closely for the cause of these first.

Trying to isolate the issue, as one does, I started to cut more and more logic out of the test scenario, until I ended up with a very distilled piece of code; By now I was no longer running any of my new code either, just using the API we have, that wraps the BAM API, to begin and end an activity in a strongly typed fashion.

The test method ended up having only two lines in it, which looked like this -

Tracking.BeginActivity(<activity name>, <activity id>)
Trakcing.EndActivity(<activity id>)

The pseudo code for these two methods is as follows -

BeginActivity -

EventStream stream = new BufferedEventStream(<connection string to MsgBoxDB>,1);
stream.BeginActivity(<activitiy name>,<Activity Id>);
stream.UpdateActivity(<activity id>, <data array>);

End Activity =

EventStream stream = new BufferedEventStream(<connection string to MsgBoxDB>,1);
stream.UpdateActivity(<activity id>, <data array>); 
stream.EndActivity(<activity id>);

Running this code on its own works ok and no issues are observed, but under load this code causes more of these ‘phantom records’ where updates to the activity seem to happen after it has been closed.

As the code at this stage was very simple (I’ve removed the calls to what I actually wanted to test) the issue became apparent -

Foolishly I have left the FlushThreshold parameter of the BufferedEventStream constructor as a constant 1!
This parameter tells the stream when to flush the data to the database and exists on both the BufferedEventStream and DirectEventStream. A value of 0 tells the stream to never auto-flush and an explicit call to flush is required from the client code; any other figure is the number of records to wait for before flushing the data to the database.

Having the value of 1 for the flush threshold hardcoded is not only inefficient (in the case stated above, I always perform two actions on the record, with a value of 1 this will always cause two round-trips to the database, when it could have been batched up and done in 1), but also, due to the asynchronous nature of BAM, may cause events to be processed out of order (as they are persisted separately).

Ensuring the threshold is set to the correct value helps ensure that records, which are now persisted together, are processed in the right order, which sorts out the data inconsistency issue we’ve seen.

Labels: ,

3 Comments:

  • What did you set the threshold to?

    By Anonymous Erik M, at 18/02/2011, 14:25  

  • Well - we made that call in every case individually, which was exactly the point - everytime we created a connection we figured out the correct threshold based on the number of operations we were going to do over that connection

    By Blogger Yossi Dahan, at 18/02/2011, 14:44  

  • My understanding is that, when using a BufferedEventStream, the order that activity updates are committed is not guaranteed unless they are done on the same Stream ID. So, in order to address this kind of issue, I just make sure I set the BufferedEventStream.StreamID property to the same GUID for updates that need to be in order.

    By Anonymous Walter Michel, at 25/02/2013, 18:34  

Post a Comment

<< Home