Yossi Dahan [BizTalk]

Google
 

Sunday, September 27, 2009

Don’t be scared of custom disassemblers

During the last few weeks I’ve been asked to review two separate projects, for two separate companies, developed – naturally – by two separate teams.

The two things both projects had in common were that they both had to deal with legacy “flat files” and they both chose to process these files outside BizTalk using custom code.

In both these cases I completely agree with the decision to use custom code to parse the incoming files – as good as the out-of-the-box flat file support in BizTalk is (made significantly better with the introduction of the flat file wizard, once one gets the hang of using it) – there’s no avoiding from writing custom code to parse flat files every now and then – some file formats are pretty challenging with different records types, conditional records, interleaving segments etc.

I do not agree with, however, the decision to perform this custom parsing outside BizTalk.

I’m pretty sure I would not even bother posting this point had I not seen two of these in the same month, but the fact that I did suggests it may be worth posting a quite note.

One of the projects had the code in a console app, called from a windows scheduled task; the application would pick up files from a folder, parse them, and drop the xml representation in another folder, for BizTalk to consume.

The other had a windows service monitoring a folder and pick up any files, parse them to a different, simplified, flat file format (!), and drop them in another folder for BizTalk to consume.

Both of these introduce another component to the mix; such component needs it’s own error handling, it’s own monitoring, deployment strategy, operations manual etc. similarly it includes a fair bit of re-inventing the wheel – writing code to monitor folders, read files, and write files – stuff that BizTalk is doing out of the box.

What would have been the correct approach then? quite simple – a custom disassembler in the receive pipeline -

Writing a custom disassembler it quite simple - at the end of the day, it boils down to developing a class library, which implements a few simple interfaces, the main one – IDisassemblerComponent defines two methods - Disassmeble and GetNext (the other interface are even simpler, almost insignificant in terms of effort)

Disassemble gets the source message and potentially parses it up front, GetNext is called repeatedly by the pipeline to receive 0 or more parsed messages, until it returns null. simple.

In one of the projects I’ve since taken their existing code (console app), refactored it into a class library, and wrapped it in a custom disassembler class that calls it; converting the scenario to a BizTalk pipeline and performing the key “developer testing” took less than a day.

Why did they not do it to begin with? whilst sometimes there are valid reasons, technical or otherwise, I suspect that in this case it was just unfamiliarity with with BizTalk and some lack of confidence in the development team’s ability to learn and implement (or their belief in themselves); these are valid concerns to any project manager, but I would suggest that a better course of action would have been to spend some time looking at what it takes to implement a custom disassembler, seeing that’s its not at all that scary, and by doing so learning more about a product used in the solution (BizTalk) and achieving a better architecture, and more maintainable approach.

Labels:

8 Comments:

  • Yep,
    Even My experience falls in same way. However sometime the time is also key decision factor on cutom components development.
    I like the approach on custom pipeline component.
    I know the blogers Guide was very great initative.
    However I think we need to come up with one guide where we can take a look at all cutom componets avaliable in biztalk and most commonly used.

    Thanks,
    Abby

    By Blogger MyBTSExperience, at 29/09/2009, 19:38  

  • Hi Yossi,

    Feel your pain on this one...

    I think this is a common result of two of the most annoying things i think you come across in BizTalk development.

    1. Sometime you come across a solution where it has been made to work in development but no consideration has been made to how this solution will live in the real world. You raise the right concerns about this solution i think

    2. I think sometimes people come across a problem and have a poor problem solving approach. Its just a case of get something which does the job regardless of how much of a hack it is. Once the code is working there is no refactoring or revisiting of the problem to see what the right way to make this work is.

    I think on point 2 probably two of the best tips for people are if you get a tricky problem and need to experiment the first thing to do is solve the problem in a small scale POC first and not your actual code base so you dont end up with a mess of code. And secondly talk to your team mates as there is a chance some of them may have solved this before and if you can convince your team mates that this is the right way to solve the problem then you are more likely to have considered other options and got rid of bad ideas.

    By Anonymous mike stephenson, at 29/09/2009, 21:55  

  • Hi Yossi,

    From time to time I read your grat posts, but now I've got a little off-topic question:
    Please read here http://www.biztalkgurus.com/forums/p/13228/25713.aspx#25713

    - Can you answer my question there?
    Thank you,

    Yonathan.

    By Blogger Yonathan, at 30/09/2009, 12:14  

  • Yonathan - I've posted an answer on that thread

    By Blogger Yossi Dahan, at 30/09/2009, 13:36  

  • Hi Abby

    I agree that time can be a factor, as many other things, but isn't it often a very short sighted one? how much time are you going to lose later on on trying to maintain a more complex solution? what would the time-to-market of future enhancements or bug fixes be? I am a bit of a purist, but I maintain that more often than not doing the right thing is better than doing the quick thing.

    As for the guide....I sort of agree, but for the most part this is called "experience", isn't it? as its quite difficult to document all possible approaches to all possible solutions...

    By Blogger Yossi Dahan, at 30/09/2009, 13:38  

  • Yossi,

    Thank you!

    Yonathan.

    By Blogger Yonathan, at 30/09/2009, 14:12  

  • Hi Yossi,

    A good, thought provoking post. Traditionally pipeline and adapter development have not been particularly well documented and examples are often too simple.

    For instance, I tried to find a good example of creating a custom disassembler that extends the XmlDisassembler without much joy.

    Any pointers appreciated.

    Regards
    Mark

    By Blogger Mark, at 05/11/2009, 10:49  

  • Hi Mark

    I agree with you that samples have not always been good enough, but there is a lot more content these days than several years back and it is significantly better.

    As for your question - it does depend on what you mean by extending.

    You could easily wrap the xml disassembler in your own component and then apply further logic on top of its logic, I've done this a couple of times and its been useful there, but that's only one interpretation of "extend"

    If you have specific requirements you wish to discuss drop me a line, or take it to the msdn forums, where lots of other people (more than the 2 reading this blog) can contribute!

    By Blogger Yossi Dahan, at 05/11/2009, 11:24  

Post a Comment

<< Home