In short - projects created using Visual Studio 2019 by default are compiled in a deterministic fashion, meaning if you compile the file multiple times without any changes, you will get identical output each time. Previous versions of Visual Studio used to include the compilation timestamp as part of the file header, causing the output to be different each time.
I'm not sure of the reasons why, but some of the tools provided with older versions of Visual Studio are unable to process assemblies compiled with this flag - they report them as not being valid image files and won't continue.
The solution is to either compile the assemblies in the
traditional way, by deleting the flag from the
Or, use a newer version of the tool that is specifically
designed for use with these assemblies.
Last week I was performing an automated test run of a product suite. The tests are written using MSTest and are shelled by a custom application that takes care of instrumenting assemblies for code coverage, running the code coverage daemon, running the tests, analysing the results and then producing a nice report for QA. Originally written in 2009 using Visual Studio 2008 it was incrementally updated to Visual Studio 2013 and has been untouched for a few years whilst successive versions of Visual Studio roll by.
The last change I made to this product suite involved the addition of several new libraries, all of which are .NET Framework libraries using the old style project format. As I recently switched permanently to using Visual Studio 2109 (from 2015), the new libraries and their associated test libraries were all created using Visual Studio 2019. Two of these new libraries were flagged as requiring code coverage, which is where this saga started.
When the test run commenced I noticed it had reported warnings stating that instrumentation of the new assemblies had failed and after the test run had completed, no coverage was present for these files.
Naturally the first thing I tried to do was manually instrument
the files using the Visual Studio 2013 version of
as used by our testing tool.
> "C:\Program Files (x86)\Microsoft Visual Studio 12.0\Team Tools\Performance Tools\vsinstr.exe" "<Company>.<Product>.<Tech>.IntegrationTests.dll" Microsoft (R) VSInstr Post-Link Instrumentation 12.0.31101 x86 Copyright (C) Microsoft Corp. All rights reserved.
File to Process: <Company>.<Product>.<Tech>.IntegrationTests.dll --> <Company>.<Product>.<Tech>.IntegrationTests.dll Original file backed up to <Company>.<Product>.<Tech>.IntegrationTests.dll.orig
Error VSP1048: Internal instrumentation error.
Next, I tried Visual Studio 2015.
> "C:\Program Files (x86)\Microsoft Visual Studio 14.0\Team Tools\Performance Tools\vsinstr.exe" "<Company>.<Product>.<Tech>.IntegrationTests.dll" Microsoft (R) VSInstr Post-Link Instrumentation 14.0.25420 x86 Copyright (C) Microsoft Corp. All rights reserved.
Error VSP1033: The file '<Company>.<Product>.<Tech>.IntegrationTests.dll' does not contain a recognized executable image.
Finally, the last version of Visual Studio I had available on my machine was 2019.
> "C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Team Tools\Performance Tools\vsinstr.exe" "<Company>.<Product>.<Tech>.IntegrationTests.dll" Microsoft (R) VSInstr Post-Link Instrumentation 16.1.28929.3 x86 Analyzing component... Instrumenting component... ################### (100.00 %%) Successfully instrumented file <Company>.<Product>.<Tech>.IntegrationTests.dll.
While this sounds obvious, in my defence I somehow mucked up my initial testing of manual instrumentation using the VS2019 tooling so that I received the exact same error message as from VS2015. This threw me off thinking that all Visual Studio versions were affected and that there was another issue at play.
Eventually I opened one of the affected
.csproj files in
Notepad++ and started looking for anomalies. Right away in the
first group, I found this innocuous line
<PropertyGroup> <!-- snip --> <Deterministic>true</Deterministic> <!-- snip --> </PropertyGroup>
Long story short, I deleted those lines from the affected
.csproj files (I left the test projects untouched, MSTest
wasn't bothered about those), reran my tests with full code
coverage metrics. Problem solved!
This is merely an informative article on the off-chance there are other shops with older build processes such as described above. At some point we need to transition this product suit to use NUnit for tests as our other, newer, products do, but at this point I'll probably just update our venerable testing application to use the Visual Studio 2019 versions of the tools. As the years go by I find myself more and more on the side of the "if it's not broken, don't fix it camp"!
As part of this exercise I also discovered that this method of
code coverage is long deprecated and I should be using the
dynamic code coverage tool instead. As long as it generates a
.coverage file that is compatible with the old version then I
shouldn't have too much trouble updating to this this tool
- 2019-08-03 - First published
- 2020-11-22 - Updated formatting