Integrate xUnit tests into your daily Team Build

Doing proper Test Driven Development with the default visual studio test runner (mstest) is like playing Gears of War 2 on an old fashioned tube television, it’s just wrong ;).. There are several alternatives out there like mbUnit and nUnit, but my favorite is xUnit. xUnit is heavily extensible and there is a project called xunitbddextensions, which extends xunit making it possible to do specication based testing (BDD). Using resharper or testdriven.net this makes for an excellent experience when developing in visual studio. xUnit tests can also be integrated into your daily TFS (Microsoft Team Foundation Server) build by using the xUnit build task. However, the details of the testresults are not displayed in the build overview and the results are not used in the reporting functionality of TFS (Like the Quality Indicators report).

On the xUnit site there is some discussion about this functionality, but at the moment there is no implementation. For nUnit there is a project on codeplex called nUnit for Team Build. This consists of a sample build script that uses an XSLT file to translate the xml results of nUnit into a visual studio test result file and publishes this result to TFS. This way integrating the nUnit test results into TFS.

I figured I could modify the XSLT file to translate the xUnit tests results instead. However, having a better look at the xUnit build task I noticed a NunitXml property, which makes it possible to output the xUnit results into a nUnit compatible xml file.  So I could just use the XSLT file and integrate the xUnit results into our TFS build.

To make this work I only had to modify the build script to use the xUnit build task instead of  the nUnit build task. This resulted in the following script (This just needs to be added to the end of your build definition):

<!-- Xunit Build task--><PropertyGroup><XunitBuildTaskPath>$(ProgramFiles)\MSBuild\Xunit\</XunitBuildTaskPath></PropertyGroup><UsingTask AssemblyFile="$(XunitBuildTaskPath)\xunit.runner.msbuild.dll"
    TaskName="Xunit.Runner.MSBuild.xunit"/><Target Name="AfterCompile"><!-- Create a Custom Build Step --><BuildStep TeamFoundationServerUrl="$(TeamFoundationServerUrl)" BuildUri="$(BuildUri)" Name="XUnitTestStep" Message="Running Xunit Tests"><Output TaskParameter="Id" PropertyName="XUnitStepId"/></BuildStep><CreateItem Include="$(OutDir)\*.Test.dll"><Output TaskParameter="Include" ItemName="TestAssemblies"/></CreateItem><!-- Run the tests --><xunit ContinueOnError="true" Assembly="@(TestAssemblies)" NUnitXml="$(OutDir)nunit_results.xml"><Output TaskParameter="ExitCode" PropertyName="XUnitResult"/></xunit><!-- Set the status of the buildstep in the build overview --><BuildStep Condition="'$(XUnitResult)'=='0'" TeamFoundationServerUrl="$(TeamFoundationServerUrl)" BuildUri="$(BuildUri)" Id="$(XUnitStepId)" Status="Succeeded"/><BuildStep Condition="'$(XUnitResult)'!='0'" TeamFoundationServerUrl="$(TeamFoundationServerUrl)" BuildUri="$(BuildUri)" Id="$(XUnitStepId)" Status="Failed"/><!-- Regardless of NUnit success/failure merge results into the build --><Exec Command="&quot;$(XunitBuildTaskPath)\nxslt3.exe&quot; &quot;$(OutDir)nunit_results.xml&quot; &quot;$(XunitBuildTaskPath)\nunit transform.xslt&quot; -o &quot;$(OutDir)xunit_results.trx&quot;"/><Exec Command="&quot;$(ProgramFiles)\Microsoft Visual Studio 9.0\Common7\IDE\mstest.exe&quot; /publish:$(TeamFoundationServerUrl) /publishbuild:&quot;$(BuildNumber)&quot; /publishresultsfile:&quot;$(OutDir)xunit_results.trx&quot; /teamproject:&quot;$(TeamProject)&quot; /platform:&quot;%(ConfigurationToBuild.PlatformToBuild)&quot; /flavor:&quot;%(ConfigurationToBuild.FlavorToBuild)&quot;"/><Error Condition="'$(XUnitResult)'!='0'" Text="XUnit Tests Failed"/></Target>

This script assumes you have copied the xunit  build task, the XSLT file and the nxslt3.exe file to a \MSBuild\Xunit\ folder under the program files folder on your build server.

Whether the tests fail or not, the resuls should always be published. However, the xUnit build task does not have an ExitCode and I could not find another way to get the result from the xUnit task. So I have modified the xUnit task to provide an ExitCode. Open up the xUnit code from codeplex. Locate the xunit.runner.msbuild project and open up the xUnit class. Add the following property and modify the Excute method to modify the value:

        [Output]
        publicint ExitCode { get; privateset; }

        publicoverridebool Execute()
        {
            try
            {
                string assemblyFilename = Assembly.GetMetadata("FullPath");

                if (WorkingFolder !=null)
                    Directory.SetCurrentDirectory(WorkingFolder);

                using (ExecutorWrapper wrapper =new ExecutorWrapper(assemblyFilename, ConfigFile, ShadowCopy))
                {
                    Log.LogMessage(MessageImportance.High, "xUnit.net MSBuild runner (xunit.dll version {0})", wrapper.XunitVersion);
                    Log.LogMessage(MessageImportance.High, "Test assembly: {0}", assemblyFilename);

                    IRunnerLogger logger =
                        TeamCity ? (IRunnerLogger)new TeamCityLogger(Log) :
                        Verbose ?new VerboseLogger(Log) :
                        new StandardLogger(Log);

                    List<IResultXmlTransform> transforms =new List<IResultXmlTransform>();

                    using (Stream htmlStream = ResourceStream("HTML.xslt"))
                    using (Stream nunitStream = ResourceStream("NUnitXml.xslt"))
                    {
                        if (Xml !=null)
                            transforms.Add(new NullTransformer(Xml.GetMetadata("FullPath")));
                        if (Html !=null)
                            transforms.Add(new XslStreamTransformer(htmlStream, Html.GetMetadata("FullPath"), "HTML"));
                        if (NUnitXml !=null)
                            transforms.Add(new XslStreamTransformer(nunitStream, NUnitXml.GetMetadata("FullPath"), "NUnit XML"));

                        TestRunner runner =new TestRunner(wrapper, logger);
                        if(runner.RunAssembly(transforms) == TestRunnerResult.Failed)
                        {
                            ExitCode =-1;
                            returnfalse;   
                        }

                        ExitCode =0;
                        returntrue;
                    }
                }
            }
            catch (Exception ex)
            {
                Exception e = ex;

                while (e !=null)
                {
                    Log.LogError(e.GetType().FullName +": "+ e.Message);

                    foreach (string stackLine in e.StackTrace.Split(new[] { "\r\n" }, StringSplitOptions.RemoveEmptyEntries))
                        Log.LogError(stackLine);

                    e = e.InnerException;
                }

                ExitCode =-1;

                returnfalse;
            }
        }

Now build the project and deploy the xUnit build task to the server. To realize the xUnit build integration just follow the steps you find on the nunit4teambuild site and use the previous mentioned modifications. After this, you should be set to go.

BTW: In the nunit4teambuild they use nxslt2, I used nxslt3. So depending on which one you use, modify the script accordingly.

11 comments
  1. Hi Jonne,

    Nice work on the write-up

    I’m looking at doing the exact same thing.

    1) I’m wondering if there’s a better way of trapping the exit code built into msbuild. If not, I want to get your change into xUnit

    2) I’m trying to run tests in multiple assemblies, but the xunit task ponly permits one. I’m still sussing the batching syntax, but am wondering whether the CombineXunitXml Task used in xunit.tests.msbuild [in the xunit source] will also work for NUnit results (having not looked in the source). (And if I dont, whether running multiple mstest.exe /publish steps will overwrite eachother.

    Will publish results here, but interested if anyone else has already done the legwork on this.

  2. Another way we found of trapping the ExitCode is to:

    Thanks once again for posting this.

  3. Glad this stuff was useful to you. I’m interested in your findings. Hoe did you end up trapping the ExitCode? I cannot see it in your comment. Thanks.

    • Hmmm, he no like XML Mr Fawlty?

      Stripped the opening lt for tags:-

      UsingTask AssemblyFile=”$(XunitBuildTaskPath)\xunit.runner.msbuild.dll” TaskName=”Xunit.Runner.MSBuild.xunit”/>

      XmlQuery XmlFileName=”$(OutDir)nunit_results.xml” XPath = “/test-results”>
      Output TaskParameter=”Values” ItemName=”TestResultsElement” />

      CreateProperty Value=”%(TestResultsElement.failures)”>
      Output TaskParameter=”Value” PropertyName=”XUnitResult” />

  4. Atanas Korchev said:

    Great work! I’ve just used it to wire up xUnit and team build.

    Why not send a patch to the xUnit team with the error code handling?

  5. Hey Atanas, good to hear it was useful. I already supplied the path (nr: 2861), but it is still being evaluated. But maybe you can use the way Ruben describes in the comments above?

  6. Hi,

    Atanas replied to my post:- http://stackoverflow.com/questions/1063779/msbuild-task-for-combining-nunit-or-mstest-xml-results

    This would allow you to amend the script above to work with multiple test assemblies (right now you use an @ arount the test list and feed it into TestAssembly, but it only allows one. Hence you need to do a % and loop. (The “mstest /publish” step is expensive hence the desire to glue the results together on the agent box insteasd of submitting indivudually.

    Script isnt quite ready ofr public consumption or I’d post it…

  7. fabi23 said:

    I’m using VS2012 and cannot edit the build definition as XML. How do you do that? Thanks!

  8. fabi23 said:

    Just adding me to be notified of follow-up comments.

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: