I just upgraded the Gallio NCover integration to support NCover v3. It all looked okay except that I could not get the integration test to pass!
No matter what I did, NCover would exclude coverage of any Gallio code at all including my test assembly! However it would include coverage information about Castle assemblies.
On a whim, I ran grep Gallio * over the NCover program files folder, and here's what I found in a file called NCover.Console.exe.config:
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<configSections>
<section name="ncover.systemAssemblies" type="NCover.Framework.SystemAssemblySectionHandler, NCover.Framework"/>
</configSections>
<startup>
</startup>
<ncover.systemAssemblies>
<assembly>msvc[mrp].*</assembly>
<assembly>mscorcfg</assembly>
<assembly>mscorlib</assembly>
<assembly>sysglobl</assembly>
<assembly>CppCodeProvider</assembly>
<assembly>Microsoft.*</assembly>
<assembly>PresentationBuildTasks</assembly>
<assembly>PresentationCore</assembly>
<assembly>PresentationFramework.*</assembly>
<assembly>ReachFramework</assembly>
<assembly>System.*</assembly>
<assembly>VSLangProj.*</assembly>
<assembly>VSWebSite\.Interop</assembly>
<assembly>VSWebSite\.Interop90</assembly>
<assembly>SMDiagnostics</assembly>
<assembly>UIAutomationClient</assembly>
<assembly>UIAutomationClientsideProviders</assembly>
<assembly>UIAutomationProvider</assembly>
<assembly>UIAutomationTypes</assembly>
<assembly>WindowsBase</assembly>
<assembly>WindowsFormsIntegration</assembly><assembly>nunit\..*</assembly>
<assembly>nunit-.*</assembly>
<assembly>xunit.*</assembly>
<assembly>MbUnit.*</assembly>
<assembly>MSTest.*</assembly>
<assembly>log4net</assembly>
<assembly>QuickGraph.*</assembly>
<assembly>WiseOwl.Obfuscation</assembly>
<assembly>Rhino.Mocks</assembly>
<assembly>Gallio.*</assembly>
<assembly>mbunit.console</assembly>
<assembly>Gallio.Echo</assembly>
<assembly>Gallio.Icarus</assembly>
</ncover.systemAssemblies>
</configuration>
Aside: This list seems to contain some redundant or incorrect entries. For example, the old MbUnit v2 console application is called MbUnit.Cons, not MbUnit.Console.
Whoo!
Ok, so out of the box NCover will exclude from coverage analysis anything related to several popular testing packages. In principle, it's not a bad idea since excluding these assemblies is a common support request from users... but in practice this worries me because it can become overbroad and lead to user confusion (as it did for me). Gallio in particular is in a good position to take care of specifying its own necessary exclusions automatically (while taking plug-ins into account also).
To work around this issue, it suffices to edit the file. The list of system assemblies seems to override anything specified in the list of assembly inclusion arguments.
I will send the author a proposal to add a new switch to disable some of this behavior...
4 comments:
We have found that the majority of users are confused as to why they are obtaining coverage data on the test framework. The only work around at this time is to remove those items from the config file.
@joefeser
Yup, pretty much what I figured.
I have received many requests from users for information about how to prune test framework assemblies from the output, so I'm familiar with the problem! :-)
I will add it to the documentation to cut back on the head scratching. NCover 3 will instrument any dll now where v2 only instrumented items that had pdbs. If there is a typo or additional items need to be added, email us at support.
I guess I am to blame :-) I requested the team add that exclusion (as well as several others)...
Post a Comment