There are numerous blogs about getting code coverage in your TFS build process and failing the build when it’s below certain percentage. Finding out code coverage in your build process is such a common action that there is a readymade activity (GetCodeCoverageTotal) for it in the TfsBuildExtensions (https://github.com/tfsbuildextensions/CustomActivities).
If you can, go ahead and use the TfsBuildExtensions’s GetCodeCoverateTotal activity as described by Colin Dembovsky here: https://tfsbuildextensions.codeplex.com/wikipage?title=Failing%20Builds%20based%20on%20Code%20Coverage. However, there are three scenarios where you will need a different approach:
- You do not have access to TFS build servers; and your admin doesn’t allow adding custom DLLs to build servers. You need to copy TfsBuildExtensions DLL to all the build servers to use the customer activities defined therein.
- You have all the access you need. But getting everything to work as mentioned in the blog above takes too much time. You need to download the source and compile locally, to get the build activities show up in your Visual Studio Toolbox and to be able to drag drop in workflow designre (For eg. it took my several hours to figure out why Visual Studio kept complaining about missing IonicZip.dll even though it was present in my GAC).
- You have access to TFS build servers, and tried all the steps mentioned in the blog above. But could not get it to work for some reason (in my case GetCodeCoverageTotal
kept throwing NullReferenceException without any explanation)
I needed an approach that doesn’t involve deploying any DLLs to build server. So I took a look at the code for GetCodeCoverageTotal activity in TfsBuildExtensions source code. It looked like this:
As you can see there isn’t much code here, which got me thinking if I can implement this logic in my build template workflow designer using primitive actions? I will just need Assign, ForEach and If activities.
Here’s how to do it:
1) Open your build template xaml in Visual Studio to bring up the workflow designer
2) Locate the If CompilationStatus = Unknown activity.
3) Add a Sequence activity after the If CompilationStatus = Unknown activity and name it “Sequence – Check Coverage”
4) Double click to expand the sequence. First thing we need to do is add a Delay activity. As Colin mentioned in his blog, the coverage results are uploaded to TFS asynchronously. So a delay will give enough time for that to complete. Set the Duration property to TimeSpan.FromSeconds(10)
(All the actions so far are similar to what Colin’s blog mentions)
5) Now we are ready to add actions that implements the code from GetCodeCoverageTotal.cs. The first two lines of code are as follows:
var buildDetail = this.BuildDetail.Get(this.ActivityContext);
var testService = buildDetail.BuildServer.TeamProjectCollection.GetService<ITestManagementService>();
We basically need to get an instance of ITestManagementService. First let’s add a variable to hold this instance. Click on Variables tab at the bottom of the screen and add a variable named MyTestService, for Variable type, click Browse for Types… and select ITestManagementService from the Browse and Select a .Net Type popup. Make sure the Scope is set to your sequence (in our case Sequence – Code Coverage)
Now that we have created a variable, it’s time to assign it. Add an Assign activity after the Delay activity. Set the To property to MyTestService and the Value property to BuildDetail.BuildServer.TeamProjectCollection.GetService(Of
If you see any exclamation or warnings on Assign activity, just same the xaml, close and reopen it. Sometimes it requires a reopen to recognize new variable types.
6) The next line on GetCodeCoverageTotal.cs is
var project = testService.GetTeamProject(buildDetail.TeamProject);
This will be similar to the step above. Add a variable called MyTestProject and choose the type ITestManagementTeamProject.
Then we’ll add an Assign activity and set MyTestProject to MyTestService.GetTeamProject(BuildDetail.TeamProject)
7) Next, let’s code this line: (we skipped a line about TestRuns, we’ll cover it in next step)
var covManager = project.CoverageAnalysisManager;
Create a variable named MyCoverageAnalyser of type ICoverageAnalysisManager. Then add an Assign activity to set MyCoverageAnalyser to MyTestProject.CoverageAnalysisManager
8) Now consider the following line that gets test runs:
var runs = project.TestRuns.ByBuild(buildDetail.Uri);
And the loop that goes through each test run and queries for coverage:
var totalBlocksCovered = 0;
var totalBlocksNotCovered = 0;
foreach (var run in runs)
var coverageInfo = covManager.QueryTestRunCoverage(run.Id, CoverageQueryFlags.Modules);
totalBlocksCovered += coverageInfo.Sum(c => c.Modules.Sum(m => m.Statistics.BlocksCovered));
totalBlocksNotCovered += coverageInfo.Sum(c => c.Modules.Sum(m => m.Statistics.BlocksNotCovered));
We can do this using a ForEach<T> activity. But first let’s create three Decimal variables called MyTotalBlocksCovered,
MyTotalBlocksNotCovered and MyTotalBlocks
Now add a ForEach<T> activity to your sequence. For TypeArgument property of ForEach, Browse and select ITestRun. And for the Values property type in MyTestProject.TestRuns.ByBuild(BuildDetail.Uri)
Now we have to implement the body of the foreach loop. Double click on ForEach<ITestRun> and then add a Sequence activity in the Body section.
Double-click the sequence. We need to create some variable in this sequence first. Create a variable called MyCoverageInfo of type Array of [T], then browse and select ITestRunCoverage. Also add two Decimal variables TempCovered and TempNotCovered
Now add an Assign activity to the Sequence. Assign MyCodeCoverageInfo with MyCoverageAnalyser.QueryTestRunCoverage(item.Id, CoverageQueryFlags.Modules)
Next add another Assign, TempCovered = MyCoverageInfo.Sum(Function(c) c.Modules.Sum(Function(m) m.Statistics.BlocksCovered))
Next add another Assign, TempNotCovered = MyCoverageInfo.Sum(Function(c) c.Modules.Sum(Function(m) m.Statistics.BlocksNotCovered))
Next add another Assign, MyTotalCovered = MyTotalCovered + TempCovered
Next add another Assign, MyTotalNotCovered = MyTotalNotCovered + TempNotCovered
That’s it for the ForEach loop. Now come back to the “Sequence – Check Coverage”
9) Add an Assign activity after the ForEach<ITestRun> activity. Set MyTotalBlocks to MyTotalCovered + MyTotalNotCovered
This basically represents the following line from GetCodeCoverageTotal.cs
var totalBlocks = totalBlocksCovered + totalBlocksNotCovered;
10) Now let’s do the last bit of code.
if(totalBlocks == 0)
return (int)(totalBlocksCovered * 100d / totalBlocks);
We will of course not return anything, instead we can check the value with the desired coverage we want. For example, let’s say we want to fail the build if coverage is less than 90%.
Add an If activity to Sequence – Check Coverage. Set the Condition property to MyTotalBlocks > 0
Then double click the If activity, and add a Sequence in the Then section
Then double click the Sequence. In the Sequence, create a Decimal variable called TotalCoverageActual
Add an Assign activity to set TotalCoverageActual to
MyTotalCovered * (100D / MyTotalBlocks)
Then add an If activity with condition TotalCoverageActual < 90
Double-click the If activity and add a Sequence in the Then section
Double-click the Sequence and add a WriteBuildWarning activity and set the Message property to “Coverage too low “ + TotalCoverageActual.ToString(“##.##”)
Then add a SetBuildProperties action and set the field PropertiesToSet to TestStatus and Status. Set Status property to BuildStatus.PartiallySucceeded (or to Failed if you like) and set the TestStatus property to BuildPhaseStatus.Failed
That’s it. Save the build template, and check it in.
To test it out, create a build definition that uses this build template. Or if you already have build definition, just Edit it and Refresh the template in the Process section.
That’s it folks. Your build should now fail if the coverage is less than 90%. Btw, instead of hard-coding 90, you can add an Argument to the build template so that you can set it from Build definition.
If you don’t get expected results, add WriteBuildWarning activities different places to see what’s going on.
If you have questions of face problems, drop me a line below.