Kivela

(Post found on Buck Hodges - How to run tests in a build without test metadata files and test lists (.vsmdi files) in TFS Blog)

Although I normally never copy paste complete posts of other people I found this article very interesting and it took me quite some time to figure the following (bottom of the post) out so here it goes…

Since the beginning, running tests in Team Build (or MSBuild in general) has meant having to use .vsmdi files to specify the tests to run. Tons of people have complained about it, as it’s a burden to create and edit the files, as either VSTS for Testers or the full suite is required in the 2005 release, and merging changes to the file is painful when multiple developers are updating the file. While mstest.exe has a /testcontainer option for simply specifying a DLL or load/web test file, the test tools task used with MSBuild does not expose an equivalent property.

Attached to this post is a zip archive containing the files needed to run tests without metadata. There are three files.

TestToolsTask.doc contains the instructions for installing the new task DLL and modifying the TfsBuild.proj files to use it.
Microsoft.TeamFoundation.Build.targets is a replacement for the v1 file by the same name that resides on the build machine.
Microsoft.TeamFoundation.PowerToy.QualityTools.dll contains the new TestToolsTask class that extends the TestToolsTask class that shipped in v1 and exposes a TestContainers property that is is the equivalent of mstest.exe’s /testcontainer option.
After you read the instructions (please, read the instructions), you’ll be able to run all of your unit tests by simply specifying the DLLs or even specifying a file name pattern in TfsBuild.proj. Here are a couple of examples. The TestToolsTask.doc file explains how they work, including what it all means.

This new task will be included in future releases of the Team Foundation Power Toys (soon to be called Power Tools). The TestToolsTask in the shipping product will support test containers and Team Build will support using it in the next release of the product.

I started with code and documentation that someone else wrote and finished it off. Thanks to Clark Sell for digging up an old email thread about the original work and Tom Marsh, development lead in VSTS Test, for pointing me to the code and doc. Thanks to Aaron Hallberg and Patrick Carnahan, Team Build developers, for their help. Thanks to Brian Harry for letting me post it.

Please post your feedback in the comments to this post. We’d like to know if this hits the mark, or if there’s something else we need to support.

New features: Test categories and test names

Pierre Greborio, a developer over in MSTV, has contributed a great new feature: test categories. Those of you who have used NUnit are probably familiar with the Category attribute. Test categories allow you to execute specific groups of unit tests. Unlike the test container feature, the test category feature will not be in Orcas.

While the details are discussed in the TestToolsTask.doc included in the zip file attached to this blog post, here’s the quick version.

Add the Category attribute to your unit test method.
Specify the category to run in your tfsbuild.proj file.
The other feature that’s new in this release is the support for test names. This is equivalent to the mstest.exe /test command line switch. The name that’s specified is implicitly a wildcard match. Specifying “Blue” as the test name, for example, will execute every test method that has Blue in the name, including DarkBlue and BlueLight.

Pierre did his testing on Vista. Because the dll is not signed, he ran into some trust issues. If you hit a similar problem, he recommends this forum post for how to get it to work.

Stefan’s update

One thing I stumbled on while trying to make this work with external file deployment items (f.e. xml files as the expected result) was that this type of testing doesn’t make it possible to include these folders automagically. Unless …

TestToolsTask on msdn

If you add the deployment thingies to a testrunconfig file and add following to your tfsbuild.proj file (where testrun.testrunconfig is the name of your testrunconfig file ofcourse)

$(SolutionRoot)\testrun.testrunconfig

You should add this line at the end of the first that you come across (also contains the setting normally)

Small example of (a piece of) the content of the testrunconfig file.
<deploymentItems type=”Microsoft.VisualStudio.TestTools.Common.DeploymentItemCollection”>
<m_container type=”System.Collections.Hashtable”>
<key type=”Microsoft.VisualStudio.TestTools.Common.DeploymentItem”>

<path type=”System.String”>TestProject.Test\TestFeeds</path>
<outputDirectory type=”System.String” />
</key>
<value />
</m_container>
</deploymentItems>

Do keep in mind however that the testrunconfig file needs some manual editing after you edit it through visual studio. By Default it will add an extra slash to the folder name, which will result in incorrect behaviour. (won’t work ). So open up the file with your favorite xml editor and remove the final slash. f.e.

After editing in visual studio:

<key type=”Microsoft.VisualStudio.TestTools.Common.DeploymentItem”>
<path type=”System.String”>TestProject.Test\TestFeeds\</path>
<outputDirectory type=”System.String” />
</key>

Should become:

<key type=”Microsoft.VisualStudio.TestTools.Common.DeploymentItem”>
<path type=”System.String”>TestProject.Test\TestFeeds</path>
<outputDirectory type=”System.String” />
</key>

Filed under: Tech on May 15th, 2007