Yossi Dahan [BizTalk]

Google
 

Sunday, March 29, 2009

Oslo based solution for deploying BizTalk applications – the runtime

This is a third post in a series describing my Oslo based solution for deploying BizTalk applications; I’ve used this exercise to play around with ‘M’, but it was important for me to work on a real solution, with real benefits – something I could actually use…in Part I I discussed the concept and presented both the “source code” of my app and the output I was working toward; Part II was all about the MGrammar part of the solution.

In this, third, part I will discuss the last missing piece –the runtime.

Before I start, though, I would say that I did find getting into ‘M’ somewhat confusing at first; and while it’s more than just possible I’m still missing some things , I hope this series could help one or two people in their journey with Oslo – which is, without a doubt, an exciting one!

There are two things, I believe, that contributed to my confusion - the first is the fact the M is really many things, quite different things, actually; from what I hear Microsoft have identified the challenge some of us (me) are having getting a grasp on ‘M’ and are hard at work to bring things [closer] together; hopefully it won’t be long before we know how the converged language looks like, in the mean time one simply has to remember that -

There’s MSchema - which you could use to define models,  a bit like xml-schema, or declaring your classes in code or even tables in SQL; I haven’t really touched on MSchema in this series butI might come back to that later.

Then there’s MGraph, which is a way to define instances of things, possibly ones that have been modelled using MSchema, but, as is evident from my little project, not necessarily - MGraph can be very useful even if you don’t have a model- as long as you have your grammar – in comes MGrammar, the third spect of ‘M’, which can be used to define a syntax for your very own [domain-sepcific-]languge for describing things;
A ‘runtime’ could then be used to processes instances described as MGrammar as a result of inputs in your language.

And that is the second thing that really confused me – what is that ‘runtime’? in all the ‘M’ presentations I’ve seen, the ‘runtime’ was merely mentioned and has never received enough “floor space” and yet – an MGrammar without a runtime, in the majority of cases, is, quite useless; you have to have a runtime that would act on your source code; in fact – the runtime would act on the MGraph resulting from your language, which is what makes it all so brilliant, because in a sense, this is where everything comes together – you runtime can work on instances described in your language, on MGraph instances stored in the repository created using MSchema and possibly even ones defined using Quadrant.

The point is that there must be a runtime that understand the model behind your language , can parse its graph and then do whatever you need it to do; and it is your job to build that runtime.

So what have I done for my runtime? here’s a quick overview (reminder: the full source code will find its way shortly onto codeplex) -

My runtime is a console application, one that takes a source code file path as an argument and outputs MSBuild files (and dependencies) that can be used to deploy the application described in the source code onto BizTalk Server.

The first part of my runtime - which I will not bore you with -  is about validating the command line arguments; standard stuff.

The second part is about creating  the parser for my language, where, thankfully, the Oslo SDK does all of the heavy lifting – it includes a class called DynamicParser which, once created, you can use to parse your source code.

To create the DynamicParser you must first compile your language, and that’s easy enough to do – you start by creating a compiler

MGrammarCompiler compiler = new MGrammarCompiler();


and continue by supplying your grammar



compiler.SourceItems = new SourceItem[] {
new SourceItem {
Name="BTSDeploy",
ContentType = ContentType.Mg,
TextReader = new StreamReader(GetLanguageDefinition())
}
};


(GetLanguageDefinition() is a simple helper method I wrote to get the grammar file embedded as a resource in the exe)



Now you’re ready to compile your language, but to make things manageable you want to provide it with an error reported; the compiler would report any errors to the stream you would provide, I’ve naturally used the console



TextWriterReporter errorReporter = new TextWriterReporter(Console.Out);
if (compiler.Compile(errorReporter) != 0 || errorReporter.HasErrors)
{
Log("Failed to compile language definition\nSee above for details");
return null;
}



If the compilation succeeded you are ready to create your parser -



DynamicParser parser = new DynamicParser();
compiler.LoadDynamicParser(parser);


That’s part one of three done.



The next step is to use the dynamic parser to parse your source code, the output of which would be a graph representation of the source; luckily the SDK does virtually all the lifting here as well, and it comes down to one line -



object rootNode = parser.Parse<object>(sourceCodeFileName, null, errorReporter);



Note that the output type is object – which, as you will find out if you try this out, is quite painful– currently all the types used in the Graph are internal, which makes debugging quite difficult (you can’t quite look at any variables you hold in any meaningful way, you have to keep calling methods, as you’ll see next; hopefully this will change one of the next updates to the SDK.



In any case rootNode is now pointing at the root of a graph – a tree like structure you could ‘walk’ to extract the pieces of information you care about in the source code; here you’re expected to use methods like GetLabel, GetSequenceElements and GetSuccessors to reach nodes and their values in the graph and, of course, to do that you need to know exactly how your graph looks like; my first instinct was to look at the PreviewMode pane in intellipad (usually the right most pane when working with MGrammar) as it shows you a representation of the MGraph created for the source code and language used; this worked quite well, but, as I found out, wasn’t the most trivial thing – the two didn’t align completely and I ended up having to resort to trail-and-error to get the parsing logic right.



The reason is that M has a few shortcuts one could take, but the graph you would be working on is the very basic, more verbose format; some information on this is mentioned here.



Then, on a recent visit to Redmond, Dana Kaufman passed on a great tip – if you ‘compile’ your grammar using mg.exe to create the mgx file (basically a ZIP file containing XAML representation of language) and then use mgx.exe on your source file adding  a reference to the mgx file you just created, you end up with an ‘M’ file which is exactly the graph your runtime would be working on.end up with; so useful!



So – here are a few example of how I worked the graph – to start with I knew my root node should be a node with the label ‘Application’, so I checked it this way -



string label = graph.GetLabel(rootNode).ToString();
if (label != "Application")



I then knew that the application name would be a child element of the root node, so I extracted it like this



// extract the application's data - this should contain two nodes - the application name and the list of items in the application
List<object> appData = graph.GetSequenceElements(rootNode).ToList<object>();
//first line should be the application name, make sure it is not a node and extract the label
if (!graph.IsNode(appData[0]))
Contents.AppName = appData[0].ToString();



the second node in the appData collection is where the graph ‘continues’, so to get the list of things that compose my application I needed to walk down that path -



//the second element should be the list of lines
foreach (object section in graph.GetSuccessors(appData[i]))
{
//each successor would be a category (reference, importing binding, resource, etc), with a list of items
processSection(graph,section);
}


with processSection start with



string sectionName = graph.GetSequenceLabel(graph.GetSuccessors(section).First()).ToString();
List<object> items = graph.GetSuccessors(graph.GetSuccessors(section).First()).ToList<object>();

Log("Found section '{0}'", sectionName);
switch (sectionName)




I hope that from these few examples you can see what it takes to work the graph – the graphBuilder (which is a somewhat confusing name, as I’m using it to walk the graph, not build it) has all the methods you need to access the various nodes ( but there’s no xpath-like- support), but as all the types are (currently) internal to the MS assembly you’re always working with objects, which is less then ideal.



Again – my full source code is on its way to codeplex, I just want to make sure it’s commented well enough to be well understood, and am struggling with time, but the bottom line is that once you figure out how the graph builder works, learnt how to see your graph visually (using mg.exe and mgx.exe) and got used to the fact that you’re dealing with objects for now, parsing the source code is very easy.



Obviously it is completely down to you what you then do with all the information you’ve extracted from the source code; in my case  my runtime is using a plug-in model so the first part is all about using the Oslo SDK to get an instance of a BizTalkDeployment class populated based on the contents of the input file, this class looks like -



    public class BizTalkDeployment
{
public string AppName { get; set; }
public List<object> References { get; set; }
public List<object> Build { get; set; }
public List<BizTalkAssembly> BizTalkAssemblies { get; set; }
public List<object> ImportBindings { get; set; }
public List<Binding> AddBindings { get; set; }
public List<Assembly> Resources { get; set; }
}

I then use late binding and configuration to load a plug in that would take an instance of this class and do the work, be it generation of msbuild scripts, deploying to the local machine using BTSTask or anything else.

Labels: ,

Monday, March 23, 2009

On Atomic Scope and Message Publishing

A few weeks back I worked on a process that looked something like this -

It was triggered by the scheduled task adapter and then used a SQL send port to call SP to return list of ‘things’.
It needed to split the things in the list to individual records, and to start a new, different, process, through pub/sub (to avoid the binary dependency with the called process), for each ‘thing’.

Fairly simple.

A lot of have been said on the different ways to split messages, I won’t repeat this discussion here; I would just say that initially I used a different approach – I used the SQL adapter in the initial, triggering, receive port and then used a receive pipeline, with an XmlDisassembler component, to split the incoming  message so that each record was published individually thus avoiding the need to have a ‘master process’; that back fired though, in my case – I quickly realised I’ll be choking the server with the amount of messages published and needed a way to throttle the execution; I’ve played a bit with host throttling but then came to the conclusion the best approach for me would be to throttle in a process, which is what I’ve done.

And so - to make things interesting, and because I already had it all ready - I decided to use a call to a pipeline from my process to split the message.

The first thing I realised, trying to take that approach, was that I had to change type of the response message received from the SQL port to be XmlDocument (which is an approach I generally dislike – I’m a sucker for strongly-typed-everything) – but my schema was configured as an envelope so that when I call the pipeline from my process it knows how to split it correctly, but, when used in the SQL port BizTalk split the message too early for me – I needed to whole message in the process first, which was no good to me; if , however, I removed the envelope definition from the schema when I would call the pipeline directly from my process it won’t know how to split the message, which is no good either; nor could i have two schemas (BizTalk, as we all know, dones’t like that bit at all, not without even more configuration); XmlDocument it is.

It then came back to me (in the form of a compile time error :-)) that the pipeline variable has to exist in an atomic scope, and so I added one to contain my pipeline variable; I then added the necessary loop with the condition set to the GetNext() method of the pipeline and in each iteration constructed a message using the GetCurrent() method; all standard stuff.

I would then set some context properties to route my message correctly and allow me to correlate the responses (I used a scatter-gather pattern in my master process) and published it to the message box

What I noticed when testing my shiny new process was that all those sub-processes that were meant to start as a result the published messages in my loop were delayed by quite a few minutes (6-8), which seemed completely unreasonable, so I embarked on a troubleshooting exercise which resulting in that big “I should have thought of that!” moment.

While the send shape in my loop successfully completed its act of publishing the message in each iteration, moving my loop to the next message and so on, being in an atomic scope BizTalk would not commit the newly published messages to the message box database, allowing subscriptions to kick in, before the atomic scope would finish; that is to allow it to rollback should something in the atomic scope would fail.
What it meant for me though, was that all the messages were still effectively published at once, which brought me back to square one (or, minus one, actually, considering that the great delay caused my this approach means I’m even worse off from my first debatch-in-pipeline approach).

And so I went back to the old and familiar approach of splitting the messages using xpath in the process, which allowed me to carefully control the publishing rate of messages for my process and throttle them as needed.

Labels: ,

Wednesday, March 18, 2009

ASMX, WCF and enums woes

We’ve been slowly migrating our services from asmx to WCF, but as we’re still using BizTalk 2006 with no support for WCF we’ve been exposing endpoints configured for basicHttpBinding and consume them using the SOAP adapter.

Generally speaking things have been going well, although we completely gave up on the idea of moving the services to WCF and NOT have to change the client, until yesterday we’ve stumbled into a serialisation issue –

The SOAP adapter, as part of its work deserialises the request message arriving through the send port into t he web service proxy class it generated, before calling the web service (which would result in the class now being serialised back into xml, which is another story); that deserialisation failed.

The error message was clear enough and indicated it failed to deserialise an enum parameter the service was expecting, and that ran a bell – I posted on exactly that back in September, but after carefully checking and re-checking everything we could swear that our message (which was now suspended) matches perfectly the schema generated by the add web reference wizard; what’s going on then??

After chasing our tail for a short while we brought up reflector to the rescue and found out the cause of our woe is a combination of a difference in behaviour between WCF and ASMX and the use of BizTalk – here are the details –

Consider the following asmx web method  –

[WebMethod]
public string GetDataUsingDataContract(CompositeType.someEnum myEnum)
{
return "Hello World";
}

With CompositeType being

public class CompositeType
{
public enum someEnum
{
Value1,
Value2
}
}

(..and pretend CompositeType has many more things, but these are irrelevant to this topic)


The definition for myEnum in the WSDL looks like

<s:element minOccurs="1" maxOccurs="1" name="myEnum" type="tns:someEnum" />

Where the type tns:someEnum looks like

<s:simpleType name="someEnum">
<
s:restriction base="s:string">
<
s:enumeration value="Value1" />
<
s:enumeration value="Value2" />
</
s:restriction>
</
s:simpleType>

As a result the definition of the enum in a proxy generated via the add web reference VS 2005 option (which is what BizTalk would use) looks like –

[System.CodeDom.Compiler.GeneratedCodeAttribute("System.Xml", "2.0.50727.3053")]
[System.SerializableAttribute()]
[System.Xml.Serialization.XmlTypeAttribute(Namespace="http://tempuri.org/")]
public enum someEnum
 {
Value1,
Value2,
}

All makes sense.


Now, let’s look at what WCF does in the same case; consider the following service –

[ServiceContract]
public interface IService1
{

[OperationContract]
string GetDataUsingDataContract(CompositeType.someEnum myEnum);
}

[DataContract]
public class CompositeType
{

public enum someEnum
{
Value1,
Value2
}
}

The WSDL generated looks like

<xs:simpleType name="CompositeType.someEnum">
<
xs:restriction base="xs:string">
<
xs:enumeration value="Value1" />
<
xs:enumeration value="Value2" />
</
xs:restriction>
</
xs:simpleType>
<
xs:element name="CompositeType.someEnum" nillable="true" type="tns:CompositeType.someEnum" />

The key difference is that the name of the class containing the enum has made it into the type name for the enum, which never happened in the ASMX version.


As a result the proxy is generated as such -

[System.CodeDom.Compiler.GeneratedCodeAttribute("System.Runtime.Serialization", "3.0.0.0")]
[System.Runtime.Serialization.DataContractAttribute(Name="CompositeType.someEnum",
Namespace="http://schemas.datacontract.org/2004/07/WcfService1")]
public enum CompositeTypesomeEnum : int
{
[System.Runtime.Serialization.EnumMemberAttribute()]
Value1 = 0,
[System.Runtime.Serialization.EnumMemberAttribute()]
Value2 = 1,
}

Again – note the name given to the element now contains the class name and, crucially, a dot (‘.’).


On it’s own – nothing to malicious – although it’s another nail in the coffin for the idea that you can substitute web service with WCF service, configured them to use basicHttpBinding and all should be the same (ok – am I the only one still wishing this was possible?)


Enters BizTalk.


When you use the add web reference wizard to add a reference to the WCF service, BizTalk generates all the schemas and proxy for you, which is what you would use to create requests going to the service (and process responses).


Because the WSDL of the WCF service contains the longer name of the enum (with the class name, the dot and the enum name) the .net proxy generated is identical to the one created for the WCF service above; the schema, however, is generated incorrectly!


BizTalk “kindly” decides that having dots in the element name is not a good idea and removes it so the schema generated looks like this –

<xs:schema xmlns:tns="http://schemas.datacontract.org/2004/07/WcfService1" elementFormDefault="qualified" 
targetNamespace="http://schemas.datacontract.org/2004/07/WcfService1"
xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:element name="CompositeTypesomeEnum" type="tns:CompositeTypesomeEnum" />
<
xs:simpleType name="CompositeTypesomeEnum">
<
xs:restriction base="xs:string">
<
xs:enumeration value="Value1" />
<
xs:enumeration value="Value2" />
</
xs:restriction>
</
xs:simpleType>
</
xs:schema>

“CompositeTypesomeEnum”??????


Well, we’ve seen this, and created a message with exactly that element, which – of course – the SOAP adapter failed to deserialise into

[System.Runtime.Serialization.DataContractAttribute(Name="CompositeType.someEnum", 
Namespace="http://schemas.datacontract.org/2004/07/WcfService1")]
public enum CompositeTypesomeEnum : int
{
[System.Runtime.Serialization.EnumMemberAttribute()]
Value1 = 0,
[System.Runtime.Serialization.EnumMemberAttribute()]
Value2 = 1,
}


The solution was fairly simple – we’ve simple change our xsl to put the element name as the .net proxy requires it, and not as the schema describes it, and it all worked well.

Labels: , ,

Monday, March 16, 2009

Ok, got your message, now let us have xsl re-use!

From the Microsoft Knowledgebase  -

The <xsl:import> element is used to import an external XSLT file. The <xsl:include> element is used to include an external XSLT file. You cannot use these elements in custom XSLT files that are defined in the Custom XSL Path Grid Properties in a BizTalk project. You cannot do this because a Uniform Resource Identifier (URI) that is defined in an external XSLT file may be from a nonsecure source such as the Internet.

Am I (well, and Ben Gimblett here)  the only one who thinks this is a lame excuse? since when MS tries to protect developers from stupidity? and in any case, if they really wanted to do that – wouldn’t they have to prevent us from writing ANY code?

Labels: , ,

Saturday, March 14, 2009

Oslo based solution for deploying BizTalk applications – the grammar

A couple of weeks ago I published a post describing my Oslo based deployment framework for BizTalk.

Two parts were missing from that post – the actual MGrammar and the runtime that processes the source code files.

In this post I will go over the grammar I created for the framework; I will try to go over the complete grammar explaining the various steps, this is not intended to be a complete description of MGrammar (not that there’s a chance I could write one), but rather an overview by example; for more information on Oslo and ‘M’ visit the Oslo Dev Centre on MSDN

It was important for me to create a solution that is completely usable, and indeed I have started to use this to generate the build scripts for my application, the price of which is that it might not be the best example code out there, but I hope you will find this useful.

Below is the complete grammar, after which I walk though it step by step; it might be useful to have anther look at the example source code I included in my previous post to better understand what I’m trying to achieve with the syntax -

module Sabra.BizTalk.Deployment 
{

language BTSDeploy
{
//main syntax is the entry point for the grammar - the first syntax to be parsed
syntax Main = app:AppDef
"{"
items:ApplicationItems
"}" => Application[app,valuesof(items)];
//application definition at root of source code
syntax AppDef = applicationKW name:ApplicationName => name;

//application items supports including all possible items multiple times in any order
syntax ApplicationItems = items:(Add | Build | ImportBinding | Comment)* => {valuesof(items)};

//now define the syntax for each item type -
//'Build' deinfes a solution or project to build during execution
//(due to limitations in our msbuild framework, runtime currently supports solutions only, but language should support both)
syntax Build = "build" path:Path";" => Build[path];
//binding to import into application
syntax ImportBinding = importKW bindingKW path:Path";" => ImportBinding{Path = path};

//syntax of add further specified different add 'options'
syntax Add = addKW add:(Add_Reference | Add_Binding | Add_Assembly | Add_BTS_Assembly) => Add{valuesof(add)};
//each add option is defined next
//binding to add as resource to application. must specify environment name
syntax Add_Binding = bindingKW path:Path env:MultiWordName";" => Binding[path,env];
//defined a reference to another application, supports providing multiple applications in the same instruction
syntax Add_Reference = referenceKW ref1:ApplicationName refs:Add_AdditionalReferences*";" => Reference{ref1,valuesof(refs)};
syntax Add_AdditionalReferences = "," app:ApplicationName => app;
//add assembly defines an assembly to be added as a resource to the application
syntax Add_Assembly = assemblyKW path:Path details:AssemblyDetails";" => Resource[path,Details{details}];
//add biztalk assembly is similar to assembly, but allows specifiying any contained orchestrations
syntax Add_BTS_Assembly = "biztalk" assemblyKW path:Path orch:Orchestrations? details:AssemblyDetails";" => BizTalkAssembly[path,orch,Details{details}];
syntax Orchestrations = withKW orchestrationsKW "{" type1:ApplicationName types:AdditionalOrchestrations* "}" => Orchestrations{type1,valuesof(types)};
syntax AdditionalOrchestrations = "," type:ApplicationName => type;
//assembly details
syntax AssemblyDetails = ver:AssemblyVersion+ culture:Culture+ pkt:PublicKeyToken+=>{Version{valuesof(ver)},Culture{valuesof(culture)},PublicKeyToken{valuesof(pkt)}};
token AssemblyVersion = versionKW "=" version:(AnyDigit*"."AnyDigit*"."AnyDigit*"."AnyDigit*)=>version;
token Culture = cultureKW "=" culture:Word=>culture;
//TODO: token should be 16 chars exactly
token PublicKeyToken = publicKeyTokenKW "="pkt:(AnyChar|AnyDigit)*=>pkt;
//keywords
@{Classification["Keyword"]}
token applicationKW = "Application";
@{Classification["Keyword"]}
token addKW = "add";
@{Classification["Keyword"]}
token bindingKW = "binding";
@{Classification["Keyword"]}
token referenceKW = "reference";
@{Classification["Keyword"]}
token importKW = "import";
@{Classification["Keyword"]}
token buildKW = "build";
@{Classification["Keyword"]}
token assemblyKW = "assembly";
@{Classification["Keyword"]}
token biztalkKW = "biztalk";
@{Classification["Keyword"]}
token withKW = "with";
@{Classification["Keyword"]}
token orchestrationsKW = "orchestrations";
@{Classification["Keyword"]}
token versionKW = "version";
@{Classification["Keyword"]}
token cultureKW = "culture";
@{Classification["Keyword"]}
token publicKeyTokenKW = "publicKeyToken";
//definition of a comment, similar to c# syntax
@{Classification["Comment"]}
token Comment = "//" CommentLineContent*;
token CommentLineContent
= ^(
'\u000A' // New Line
| '\u000D' // Carriage Return
| '\u0085' // Next Line
| '\u2028' // Line Separator
| '\u2029' // Paragraph Separator
);
//application name must start with a character and then include any character, digit or '.'
@{Classification["String"]}
token ApplicationName = AnyChar+(AnyChar | AnyDigit | ".")*;
//tokens use for definition of a file path
token Path = "\""PathRoot?FileSystemName("\\"FileSystemName)*"\"";
token PathRoot = AnyChar":\\";
token FileSystemName = (AnyChar | AnyDigit | Space | "-" | "_" | ".")+;

//common token definitions
token AnyChar = "A".."Z" | "a".."z";
token AnyDigit = "0".."9";
token MultiWordName = "\""a:(Word | Space)"\"" => a;
token Word = (AnyChar | AnyDigit | "-" | "_")+;

//the interleave will ensure the language allows whitespace anywhere
interleave Whitespace = Tab | LF | CR | Space | Comment;
token LF = "\u000A";
token CR = "\u000D";
token Space = "\u0020";
token Tab = "\u0009";

}
}



I’ve build the grammar top down and this is how I will walk through it -



First (1) I define my module, in a namespace like manner; this is the logical container for my language; I then(4) declare my language and give it a name.



The main constructs in mgrammar are syntax and token; I often heard the guys at Redmond explain that when it comes to languages you can think of syntax as being the sentence and tokens as being the words; I think this is a very clear explanations; there are a few rules relating to them as you can imagine, important ones to remember at this point are that syntaxes can contain other syntaxes as well as tokens (and literals), tokens can only contain other tokens (and literals); also – interleave does not apply to tokens .



The main syntax, and the entry point for any language is Main and you can see mine defined on line 7, and it looks like this -



//main syntax is the entry point for the grammar - the first syntax to be parsed 
syntax Main = app:AppDef
"{"
items:ApplicationItems
"}" => Application[app,valuesof(items)];



‘//’ is used for comments, just like in c#, so the first line will be ignored.



syntax is one of the few keywords that exist in mgrammar, no explanation needed;



Main is the name of the syntax, which allows is to be referred to (used) by other syntaxes; in this case, as I’ve mentioned, ‘Main’ is also the entry point –the syntax the parser will start at; everything else should flow from here.



Now for - app:AppDef, but first – a note - in my mind there are two aspects to creating a language in mgrammar – there is the ‘parsing aspect’ – you define your language so that it describes the rules to parse your source code; and there is the ‘output aspect’ (or ‘production aspect’) – this is where you define the output of your language – in ‘M’ this is mgraph – so that it descries accurately the intent of any source code (and is easy-ish to work with at runtime)



Two are inevitably very mixed in any real-world work with ‘M’ which can be confusing, and today I want to focus on the parsing aspect – firstly because it is the more important one in my view (there’s nothing to work with before you’ve declared a good syntax for your language), and secondly – because I suspect we’re going to see some changes to the production aspect in the near future.



Mainly production aspect ‘stuff’ is defined after the ‘=>’ operator as you can see in my syntax above, so for the time being just try to ignore that; there will be a bit more to ignore as you will see shortly.



AppDef is a name of a syntax declared somewhere else in the language ( line 13); it could also be defined in any imported languages, but I don’t have any, we will look at that in a second; app: is an alias assigned to this syntax which allows for it to be referenced in the production on the right side of the arrow operator; again – for the time being feel free to ignore any aliases, they have no impact on the parsing aspect.



So – my syntax main basically says we’re expecting to have in our source code something that matches the AppDef syntax, then an opening curly bracket then something that matches the ApplicationItems syntax and then closing curly bracket. simple.



Of course next the parser would look at the definition of AppDef and ApplicationItems, and so will we.



AppDef is defined in line 13 as an ApplicationKW followed by ApplicationName; these are defined in lines 51 and 90 respectively; lets look at the ApplicationKW definition-



@{Classification["Keyword"]} 
token applicationKW = "Application";



ApplicationKW itself is a token with a fixed literal 'Application’ – this is a very simple rule to follow, and in fact I could have simply included this literal in the syntax definition and not use this token at all (which is what I have done previously).



The reason I have separated it out to its own token is related to the preceding line in the grammar - the classification attribute allows me to mark this token as a “keyword” for my language, this would tell intellipad (and, presumably, any other editor that would learn how to work with mgrammar), that this token is a keyword and should be displayed as such; in intellipad this means it would be bolded in the editor, as you can see in the image below of my language open in intellipad -



image



Back to the main syntax’ components - ApplicationName, defined in line 51 states that an application name is composed of AnyChar  followed by any number of AnyChar, AnyDigit or the literal ‘.’ with AnyChar and AnyDigit defined in lines 98 and 99.



The ‘+’ sign indicates the syntax or token it follows must exist at least once; the ‘*’ sign indicates the syntax or token it follows can exist 0 or more times.



So – we have the definition of our application name, now lets look at what ApplicationItems says -



syntax ApplicationItems = items:(Add | Build | ImportBinding | Comment)* => {valuesof(items)}; 



This syntax tells the parser that an application can have any number of Add, Build, ImportBinding or Comment in any order.



Moving on we’ll look briefly at how ImportBinding looks like -



syntax ImportBinding = importKW bindingKW path:Path";" => ImportBinding{Path = path}; 



The importKW (which is the literal  ‘import’, look it up!) followed by the bindingKW (‘binding’) and the syntax for Path.



I could have combined both literals import and binding to a single token and mark that as a keyword, but there are two benefits to splitting them up- firstly, by having two tokens I can have as many whitespaces as I want between them, which I think is what developers generally expect, and, secondly – the 'binding’ keyword is re-used for the add binding syntax I’ll describe shortly.



I’ll skip the Path definition, you can follow it yourself if you wish to; so next we can look at another item in the application items list – Add:



syntax Add = addKW add:(Add_Reference | Add_Binding | Add_Assembly | Add_BTS_Assembly) => Add{valuesof(add)}; 



The Add syntax starts with the addKW (‘add’) followed by one of the syntaxes for adding a reference, adding a binding, adding an assembly or adding a BizTalk assembly, but it only allows one; the add keyword (and therefore the entire add syntax) must be repeated as a whole to add multiple items to the application, as is suggested by the ApplicationItems syntax.



Lets look at a couple of these items; first – the syntax for add binding -



syntax Add_Binding = bindingKW path:Path env:MultiWordName";" => Binding[path,env]; 



Here you can see the binding keyword being reused, as does the Path syntax; I’m then allowing a multi-word-name (which is essentially a string contained in double quotes) as the environment name for the added binding.



Quite simple, right? that’s the thing I love about mgrammar. let’s look at one more syntax -



syntax Add_BTS_Assembly = "biztalk" assemblyKW path:Path orch:Orchestrations? details:AssemblyDetails";" => BizTalkAssembly[path,orch,Details{details}]; 
syntax Orchestrations = withKW orchestrationsKW "{" type1:ApplicationName types:AdditionalOrchestrations* "}" => Orchestrations{type1,valuesof(types)};
syntax AdditionalOrchestrations = "," type:ApplicationName => type;



The Add_BTS_Assembly syntax should be very clear, the only thing I haven’t mentioned so far is the ? sign which indicates 1 or 0 appearances of the syntax/token it follows, I use this to allow a BizTalk assembly to optionally describe the orchestrations it contains so that, potentially, any instances of these could be terminated when undeploying the application.



The Orchestrations syntax, if exists, requires at least one orchestration to be specified (I’m reusing the ApplicationName token as the orchestration name) but allows additional orchestratiosn to be specified as well; I’ve used the same approach for the add reference syntax.



I hope this makes sense, and that it gives you a glimpse into a practical use of mgrammar, I am certainly excited about this stuff.



Soon I hope to post about the last missing piece of the puzzle – the runtime that uses the language definition to parse, and then execute, any source code provided; after that the whole thing is likely to find a spot on CodePlex, bare with me a little bit longer….

Labels: ,

Sunday, March 08, 2009

MSBuild support in BizTalk 2009

In my notes on BizTalk 2009 post I have highlighted the fact that now, as part of the BizTalk installation, you can select the “Project Build Components” which allow you to build BizTalk projects without Visual Studio or BizTalk installed on the build machine.

This is very cool feature of the product, but I have to admit that I have missed something quite important; luckily - Mikael Hakansson hadn’t – the provided tasks do not support deployment of BizTalk projects, but only build (see Mikael’s article here).

What this means is that, while you can use the tasks deployed through the installation to verify that your BizTalk projects build correctly (as part of continuous integration setup, for example), you have no way of verifying that they can be deployed or, more importantly, run any automated tests;

The SDC tasks, with their support for BizTalk deployment, can come in very handy here, and you can certainly use them to augment the built in build support with deployment capabilities, but they add more dependencies to the plate, which has to be planned for.

Of course, attending the MVP summit in Redmond last week we had a great chance of raising this (and various other) subjects with the product group, and we certainly have not let that opportunity go missed, now we just need to wait and see if they “pick up the glove”.

Labels: ,

Wednesday, March 04, 2009

Oslo based solution for deploying BizTalk applications

Since PDC I’ve been working on and off on an “Oslo” based solution for deploying a BizTalk application; unfortunately I couldn’t get a good chunk of time to play with this, so it’s been dragging a bit, but I’m getting close, so here are some details -

I’m a big advocate of automated builds; it’s a topic that probably deserves a post of its own, so I won’t get started on this here, but the idea is that one must have a way to be confident that, when its time to [re-]deploy the app, it will get deployed successfully time after time, without a hitch;

From my experience, deploying a BizTalk application of a decent size is often not trivial, and often it does not go smoothly if done manually (things are being missed, done out of order, wrong versions being picked up, etc), which does not help to boost the confidence in the solution (or BizTalk as a whole) in the organisation; automating the process can save a lot of time (can be done unattended, while in a meeting, out for lunch or overnight), save a lot of head scratching, boost the confidence in the solution and set the ground for proper automated builds, auto testing etc.

To that extent, I have previously built an MSBuild based framework for deploying BizTalk applications.

While I’m sure it does not suit all purposes as it was developed to support those scenarios I had, I think it’s quite comprehensive and had served me well over the past couple of years.

It allows one to provide an MSBuild-formatted file with some parameters and lists and, using the pre-created framework, would do things like remove the existing application (after terminating current instances and un-enlisting services), build the new solution(s), deploy assemblies to BizTalk, add/import binding files, start ports etc.

This is working great for us, and we’ve been using it extensively for quite a while now, but there’s one major downside – it requires one to maintain those MSBuild scripts.

Now -  MSBuild is cool, I do like it very much, but as it’s generic – it does not speak in BizTalk terms, and as it’s XML – it’s quite verbose, and I wanted an easier format for all of us to work with.

So – inspired after seeing Oslo and even more driven after visiting PDC - I’ve decided to come up with an MGrammar based language for describing a BizTalk application to be deployed.

So far I’ve come up with “version 1.0” of my grammar that allows to describe the major artifacts in a BizTalk application and a basic runtime to process source files written in this language, the language allows you to describe things like -

  • References to other applications
  • Solutions and projects to build
  • BizTalk assemblies to deploy
  • Orchestrations contained in those (these can then be terminated before attempting to remove the application prior to deployment)
  • .net assemblies to deploy
  • Binding file to import
  • Binding files to add (and the name of the environment to attach to those)

As Oslo hasn’t RTM-ed yet, I can’t quite rely on it yet, and so I cannot use it in production in any shape or form, or can I?
I found a good middle ground for us which allows us to gain from the benefits I’m hoping to get by using my language, while not exposing ourselves too much to the risks of using early technology –

At the moment the runtime I’ve built for this is used to generate the MSBuild scripts we’re already using out of the source files; in this way - if Oslo disappears tomorrow, or significantly changes (not that I think that’s going to be the case), we’re safe – we still have the MSBuild scripts as “checked-in artifacts” and so we have lost nothing.

So – how does an instance of my language looks like? here’s an example:

Application MyApp
{
//add references to other applications
add reference SomeOtherApp;
add reference AndAnother,andAThird;

//build the solution
build "SomeFolder\SomeOtherFolder\SomeSolution.sln";
build "SomeFolder\SomeOtherFolder\SomeProject.btproj";

//add required biztalk assemblies
add biztalk assembly "SomeFolder\SomeOtherFolder\bin\deployment\Schemas.dll"
version=1.0.0.0
culture=neutral
publicKeyToken=314bd7037656ea65;

add biztalk assembly "SomeFolder\SomeOtherFolder\bin\deployment\Maps.dll"
version=1.0.0.0
culture=neutral
publicKeyToken=314bd7037656ea65;

add biztalk assembly "SomeFolder\SomeOtherFolder\bin\deployment\Orchestrations.dll"
with orchestrations
{
MyApp.SomeOrchestration,
MyApp.AnottherOrchestration,
MyApp.AndAnotherOrchestration
}
version=1.0.0.0
culture=neutral
publicKeyToken=314bd7037656ea65;

//and .net helpers assembly
add assembly "SomeFolder\SomeOtherFolder\bin\release\HRG.FareWatch.Hotel.Helpers.dll"
version=1.0.0.0
culture=neutral
publicKeyToken=314bd7037656ea65;
//import dev bindings
import binding "SomeFolder\SomeOtherFolder\Bindings.xml";

//add various environment's bindings.
add binding "SomeFolder\SomeOtherFolder\Bindings.Dev.xml" "development";
add binding "SomeFolder\SomeOtherFolder\Bindings.Stage.xml" "stage";
add binding "SomeFolder\SomeOtherFolder\Bindings.Prod.xml" "production";
}


as I’ve said – I already have a runtime that generates the MSBuild scripts required to deploy these - the runtime outputs several files to the temp folder - 




  • The “framework” MSBuild script that contains all the targets I’m using


  • The Microsoft.Sdc.Tasks.BizTalk.dll and the Microsoft.Sdc.Tasks.dll which contains many useful custom tasks.


  • The relevant import file for the SDC tasks


  • The generated MSBuild file that contains all the various artifacts based on the source code



That latter file, which is the equivalent of my source code, and what we had to maintain so far, looks like -



<?xml version="1.0" encoding="utf-8"?>
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003" DefaultTargets="DeployAll">
<Import Project="Deploy.Framework" />
<ItemGroup>
<MyApp Include="SomeOtherApp" />
<MyApp Include="AndAnother" />
<MyApp Include="andAThird" />
</ItemGroup>
<ItemGroup>
<Bindings Include="General">
<path>SomeFolder\SomeOtherFolder\Bindings.xml</path>
</Bindings>
</ItemGroup>
<ItemGroup>
<EnvironmentBindings Include="General">
<path>"SomeFolder\SomeOtherFolder\Bindings.Dev.xml"</path>
<environment>development</environment>
</EnvironmentBindings>
<EnvironmentBindings Include="General">
<path>"SomeFolder\SomeOtherFolder\Bindings.Stage.xml"</path>
<environment>stage</environment>
</EnvironmentBindings>
<EnvironmentBindings Include="General">
<path>"SomeFolder\SomeOtherFolder\Bindings.Prod.xml"</path>
<environment>production</environment>
</EnvironmentBindings>
</ItemGroup>
<ItemGroup>
<ExternNetAssembly Include="HRG.FareWatch.Hotel.Helpers">
<Version>1.0.0.0</Version>
<Culture>neutral</Culture>
<PublicKeyToken>314bd7037656ea65</PublicKeyToken>
<ProcessorArchitecture>MSIL</ProcessorArchitecture>
<path>SomeFolder\SomeOtherFolder\bin\release</path>
</ExternNetAssembly>
</ItemGroup>
<ItemGroup>
<Solution Include="SomeSolution">
<path>SomeFolder\SomeOtherFolder</path>
</Solution>
<Solution Include="SomeProject">
<path>SomeFolder\SomeOtherFolder</path>
</Solution>
</ItemGroup>
<ItemGroup>
<BTSProject Include="Schemas">
<Version>1.0.0.0</Version>
<Culture>neutral</Culture>
<PublicKeyToken>314bd7037656ea65</PublicKeyToken>
<ProcessorArchitecture>MSIL</ProcessorArchitecture>
<path>SomeFolder\SomeOtherFolder\bin\deployment</path>
</BTSProject>
<BTSProject Include="Maps">
<Version>1.0.0.0</Version>
<Culture>neutral</Culture>
<PublicKeyToken>314bd7037656ea65</PublicKeyToken>
<ProcessorArchitecture>MSIL</ProcessorArchitecture>
<path>SomeFolder\SomeOtherFolder\bin\deployment</path>
</BTSProject>
<BTSProject Include="Orchestrations">
<Version>1.0.0.0</Version>
<Culture>neutral</Culture>
<PublicKeyToken>314bd7037656ea65</PublicKeyToken>
<ProcessorArchitecture>MSIL</ProcessorArchitecture>
<path>SomeFolder\SomeOtherFolder\bin\deployment</path>
</BTSProject>
</ItemGroup>
<PropertyGroup>
<BTApplicationName>MyApp</BTApplicationName>
<!-- Set for a remote deployment -->
<!-- Deploying BizTalk Server name - leave blank if local-->
<!-- not currently supported by runtime-->
<BTServerName>
</BTServerName>
<!-- Deploying BizTalk Server database - leave blank if BizTalkMsgBoxDb-->
<BTServerDatabase>
</BTServerDatabase>
<!-- Deploying BizTalk Server SQL user name - leave blank if local-->
<BTServerUserName>
</BTServerUserName>
<!-- Deploying BizTalk Server SQL password - leave blank if local-->
<BTServerPassword>
</BTServerPassword>
</PropertyGroup>
</Project>


Would you agree that the former source code is easier to maintain?



 



note: one thing you would notice is that none of the paths contains a root; I assume that this would be used by different developers/IT pros which may have the code in a different path; however, as the assumption is that the code will come from source control, my framework script expects you to provide the root path to your source code and assumes the specified paths start there.





So – what’s next?



From a coding perspective - I plan to add support in my runtime to perform the actual deployment; this is I would like it to do once Oslo is in production so I’ll add it as an option through a command line switch.



This option would tell the runtime to deploy the artifacts to a specified BizTalk server using BTSTask or the Object model instead of generated the MSBuild script.



I also want to make some modification to my language definition to make the MGraph produced cleaner (and better geared towards using the repository and Quadrant at a later stage) as well as, obviously, add support for more features, after which I plan to publish both my language and the runtime somewhere (here, codeplex, DevCente..we’ll see)



I’ll post some more notes on all of these here in the near future hopefully…

Labels: , , , ,