A place for spare thoughts

17/04/2013

WeakEvents catch with tests

Filed under: c#, Unit testing — Ivan Danilov @ 19:01

Some time ago I wrote about weak events including non-standard ones. While this post is not totally dependent on weak events, I will assume you’re acquainted with MakeWeak method.

So, suppose you have some service that publishes an event, like that:

        public class SomeService
        {
            public event EventHandler TestEvent;

            internal void RaiseTestEvent()
            {
                var handler = TestEvent;
                if (handler != null) handler(this, EventArgs.Empty);
            }
        }

And you’re writing new class that will subscribe to this event weakly and re-raise it (possibly under some specific conditions). Extremely simplified it could look like this:

        public class Sut
        {
            public Sut(SomeService helper)
            {
                helper.TestEvent += new EventHandler(OnEvent).MakeWeak(eh => helper.TestEvent -= eh);
            }

            public event EventHandler TestEventReraised = delegate { };

            private void OnEvent(object s, EventArgs args)
            {
                TestEventReraised(this, EventArgs.Empty);
            }
        }

So far, so good. Now naturally you want to test if your Sut does what it should:

        [Test]
        public void SutReRaisesEventCorrectly()
        {
            var helper = new SomeService();
            var sut = new Sut(helper);
            bool raised = false;
            sut.TestEventReraised += (s, e) => raised = true;
            helper.RaiseTestEvent();
            Assert.IsTrue(raised);
        }

Seems logical, isn’t it? You may be surprised, but if you run this test in Release mode, it will fail randomly. Most of the time it will pass, but if you use build server and have many tests in your project – this one will fail eventually. If you have been reading my blog for some time, you may recognize the symptoms as they are very similar to this case. Really, if you change test like below it will fail consistently in Release mode (in Debug it will still succeed always):

        [Test]
        public void SutReRaisesEventCorrectly()
        {
            var helper = new SomeService();
            var sut = new Sut(helper);
            bool raised = false;
            sut.TestEventReraised += (s, e) => raised = true;

            GC.Collect();

            helper.RaiseTestEvent();
            Assert.IsTrue(raised);
        }

Really, sut variable’s value is eligible for collection at highlighted line, so it is collected (our event has weak subscription, remember?), and event is never raised.

As it takes much attention to not make such errors, I’d suggest to make sut variable not local for test method but rather class variable. Then it will never be eligible for collection in such cases and you will not write randomly failing tests. Like that:

    [TestFixture]
    public class WeakEventsCheck
    {
        private Sut _sut;

        [Test]
        public void SutReRaisesEventCorrectly()
        {
            var helper = new SomeService();
            _sut = new Sut(helper);
            bool raised = false;
            _sut.TestEventReraised += (s, e) => raised = true;

            GC.Collect();

            helper.RaiseTestEvent();
            Assert.IsTrue(raised);
        }
    }

12/04/2013

MSTest broke compatibility. Again

Filed under: tools, Unit testing, VisualStudio — Ivan Danilov @ 20:20

Visual Studio 2012 Update 2 has changed something in MSTest so that it broke compatibility with ReSharper. From the description it seems it forced JetBrains to release next version on ReSharper, so users of R# had problems only for one week. Great work, actually: I bet it is not a trivial task to establish development process that allow something of R# caliber to be released in one week.

Though it is another nail to MSTest coffin, IMO. It is not only stuck in place and not evolving (gosh, they can’t make asserts for exceptions or even fix bug that ExpectedException attribute don’t check message even if asked to for years!), but actually disrupt work process from time to time with breaking changes, strange SEH exceptions or incompatibilities.

I’m extremely happy we were able to abandon MSTest and adopt NUnit.

If you have new project and deciding which unit-test framework to use – do yourself a favor and avoid MSTest as much as possible.

21/02/2012

MSTest 2010 without Visual Studio at build server – again

Filed under: MSBuild, Unit testing — Ivan Danilov @ 02:53

A while back I wrote on that topic. Today I found that that solution was not working when MSTest private accessors were used in the solution.

In general I think that these ability to access private members of the class encourages bad design and nullifies one of the most useful features of testing: necessity to think about how your class can be used and how convenient its interface is. If you have ability to work with privates – you don’t have to think. You could just write code as it goes. How are consumers of the class supposed to use it when even its author can’t?!
Thus, IMO, ability to generate private accessor to a class is one of the most abominable things in testing and I’ll never ever use it.

But unfortunately others do sometimes. If you have such a project and try to build it on a machine that was setup using my previous advice – you’ll see something about compiler can’t find out meaning of name YourClass_Accessor. Of course it can’t – it wasn’t generated yet! To overcome this issue you have to copy in addition to previous files also this folder:

c:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v10.0\TeamTest\

It contains *.targets file that was foresightfully imported even in default Microsoft.Common.Targets (even if you will never have MSTest – you still have line to import it if it is found), one assembly with custom MSBuild Task implementation and some exe-module. For some reason, MSTest developers can’t manage to write a product without such hacks at every corner. Whatever.

So, you just copied these files and msbuild will find them. Do not even hope to have such a luxury for your products 🙂
But when I rerun the build – I found such error:

C:\Program Files\MSBuild\Microsoft\VisualStudio\v10.0\TeamTest\Microsoft.TeamTest.targets(14,5): error : Could not load file or assembly ‘file:///C:\work\TestSolution\TestProject\Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll’ or one of its dependencies. The system cannot find the file specified.

What?! Why the hell MSTest tries to load an assembly from current directory and not with normal binding mechanism? I have this assembly on my build machine and it actually can be loaded when referenced from project in normal way. I didn’t found answer to that question as well as option to alter this probe somehow. But with fusionlog I found that just after probe to current dir it tries to load same assembly from its default location at c:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\PublicAssemblies\Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll. Ok. Let’s stick to it. And I’ve copied all of my assemblies not to c:\vs2010stub as I suggested in previous article, but to default c:\Program Files (x86)\Microsoft Visual Studio 10.0\ folder. And it worked – private accessors got generated and publicized correctly.

So I made related changes to my archive and re-uploaded it to old location. Just unpack somewhere on build server and run install.bat. Should work afterwards.

25/07/2011

Checking solutions and project files against some set of rules at build-time

Filed under: MSBuild, Unit testing, VisualStudio — Ivan Danilov @ 04:23

Currently I’m busy with multi-solution project. There’re several teams on the project and each team works on one or several solutions. Sometimes together, sometimes not. There are network of dependencies between solutions (e.g. some shared library that is produced as build artifact of one solution and used by two other).

Clearly that we have some set of rules how things should be organized. And this set at some stage became complex enough to be almost unmaintainable manually.

For example, each project should contain property named OutputRoot defined relative to $(SolutionDir) and OutputPath should be relative to it. Thus we achieve that every artifact is placed to some well-known place under OutputRoot which could be redefined from msbuild command line.

Or the rule that requires HintPaths properties for references should be relative to that $(OutputRoot) or $(SolutionDir). So that no one could place absolute path there that will work only in his environment. Well, it could work on others machines as well by some lucky accident. But build robustness is not that sort of things we want to have by accident.

Also there’s some general rules:

  • every C# project (*.csproj file) should belong to the solution and only one;
  • solutions can’t be nested so if some folder has *.sln file – entire tree from this point should have to other *.slns;
  • ProjectReference couldn’t point outside of solution directory or have absolute path;
  • every project should have property TreatWarningsAsErrors with value true;
  • if Deployment project exists in solution – every other project should have reference to it;
  • test projects (we are using convenience that test projects should be named like *.Tests) are marked with corresponding guids so that test runner can run them;
  • etc…

And so the idea of having some kind of source checker was born. In fact, *.sln and *.csproj are not very complex formats to parse. And we’re not going to support every possible thing in them. We want only to read and check some set of rules.

This source checker now runs on each commit in the source control from the build server and tests every rule we decided to check. If some rule was violated – it reports that, fails build and suggests what should you do to fix the thing.

It is able also to ignore some directories (for example it ignores itself because it shouldn’t be built into $(OutputRoot)). Other useful thing is that it could be integrated with TeamCity so that build log looks pretty there (you should specify /teamcity key in the command line).

Also rules could be easily added or removed.

It proved to be very useful in our day-to-day work and so I’d like to share them with everybody.

It could be not production quality sometimes and several hard-coded things could be inside but nevertheless it is still usable.

Sources could be found here. It is targeting .NET 4.0, but currently it is almost trivial to retarget it against .NET 3.5, so if you’re still there – it is not a problem.

12/07/2011

MSTest 2010 on the build server without VS2010 installed

Filed under: Unit testing, VisualStudio — Ivan Danilov @ 23:24

Some time ago I found description how to get MSTest 2008 running without Visual Studio. This time I faced the same question with VS2010. Quick googling hadn’t result in any decent answer, so I did some digging.

I won’t enumerate countless exceptions, silent hangs, unclear error messages and happy MSTest’s reports about ‘Not executed’ tests without any reason. I’m pissed as hell with MSTest and its problems but forced to use it. So I will just describe how to have it working.

What you’ll need is some machine with VS2010 SP1 installed to get assemblies and MSTest.exe from. SP1 is very desirable if you want to test .NET 4.0 project and absolutely mandatory in case of earlier platform versions because without SP1 it is impossible AFAIK. Go to that box and take:

  1. c:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\PrivateAssemblies\Microsoft.VisualStudio.QualityTools.*.dll
  2. c:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\PublicAssemblies\Microsoft.VisualStudio.QualityTools.*.dll
  3. c:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\MSTest.exe
    c:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\MSTest.exe.config
  4. c:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\QTAgent.exe
    c:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\QTAgent.exe.config
    c:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\QTAgent32.exe
    c:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\QTAgent32.exe.config
    c:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\QTAgent32_35.exe
    c:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\QTAgent32_35.exe.config
    c:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\QTAgent_35.exe
    c:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\QTAgent_35.exe.config

Make on target machine folder C:\VS2010Stub\ and copy these assemblies/configs there preserving the structure (so you’ll have C:\VS2010Stub\Common7\IDE\ and everything mentioned there and in PrivateAssemblies/PublicAssemblies).

The first step is done. The next is getting dependencies from GAC. Take gacutil.exe (check that you have one from .NET 4, i.e. it should have version like 4.0.30319.1 – older versions will result in an error), take gacutlrc.dll (without it gacutil.exe will exit silently – at least it was so on my build server) and copy them to target machine. We will need them to register extracted dependencies to GAC.

Now you need to take Microsoft.VisualStudio.QualityTools.*.dll from source machine’s GAC. Notice that some of these assemblies have two versions if you have VS2010 SP1 installed: 10.0.0.0 and 10.1.0.0. You need them both. Do not rename assembly files because gacutil.exe won’t be able to register renamed assemblies afterwards. Just put different versions with same name in different folders.

In my dev box these files was placed in C:\Windows\assembly\GAC_MSIL\Microsoft.VisualStudio.QualityTools.*\Microsoft.VisualStudio.QualityTools.*.dll and in c:\Windows\Microsoft.NET\assembly\GAC_MSIL\Microsoft.VisualStudio.QualityTools.*\…\Microsoft.VisualStudio.QualityTools.*.dll. Most probably you will find them there as well, but as I don’t know actual GAC structure (I’m not sure official documentation on GAC internals exists actually) – I can’t guarantee that.

UPD: if you will use database unit testing, you will need three more assemblies from the GAC – Microsoft.Data.Schema.dll, Microsoft.Data.Schema.Utilities.dll and Microsoft.Data.Schema.UnitTesting.dll. Note that archive at the end of this post doesn’t include them.

In any case take each of these assemblies to the target box and execute ‘gacutil.exe /i Microsoft.etc-etc-etc.dll’. You should see ‘Assembly successfully added to the cache’ for each one.

The second step is done. Now goes the last one. MSTest has some settings in the registry about file extensions that could be tested. Actually now you have MSTest functional, but if you try to run it on any assembly with tests it will say something about ‘dll is not correct test container’. Well, lets fix that.

Go to registry on the source box, find key HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\VisualStudio\10.0\EnterpriseTools\QualityTools\TestTypes (*)  and export it in some *.reg. And import on the target box. Be careful though to run reg file from 64-bit program. Explorer will go well. I have Total Commander which is 32-bit and sometimes accidentally run reg files from it. Which results in doubling Wow6432Node part.

That’s  it. Now you should be able to run MSTest.exe and test something with it successfully.

UPD: If you’re using MSTest private accessors – you’ll also need this as I discovered much later. Zip-archive linked below is already updated.

In case you’re lazy, there‘s zip-archive with everything already made for you. Just extract contents of the MSTest2010Install.zip to the target box and run install.bat.

 

(*) I’m assuming both source and target boxes have 64-bit OS. Otherwise you should cope with Wow6432Node part. It is easy but somewhat annoying. Feel free to ask me if you don’t know how to do this.

11/07/2011

Troubleshooting: MSTest assembly doesn’t get discovered by VS native test runner

Filed under: Unit testing, VisualStudio — Ivan Danilov @ 16:06

Well, I already faced similar problem here so I know where to look. And it was really the cause.

To have test runner see tests in the assembly its csproj file should contain this line:

<ProjectTypeGuids>{3AC096D0-A1C2-E12C-1390-A8335801FDAB};{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}</ProjectTypeGuids>

Troubleshooting: Resharper, Visual Studio 2010, MSTest and .NET 3.5 in single bottle

Filed under: Unit testing, VisualStudio — Ivan Danilov @ 15:59

We are in process of moving .NET 3.5 project to .NET 4.0 and VS2010. We’re still not ready to move our app to .NET 4.0 but decided to adopt VS2010 first. It is somewhat unusual situation so it caused some troubles.

First of all, unit testing projects under MSTest just can’t target framework v3.5 prior to VS2010 SP1.

VS2008 for unit testing was using Microsoft.VisualStudio.QualityTools.UnitTestFramework assembly, version 9.0.0.0. It was for v3.5 so everything is cool. Except the fact that if VS2010 sees version 9.0.0.0 in any reference to this assembly – it starts Project Conversion Dialog and prompts you to convert your solution. Thus in VS2010 you just can’t have reference to 9.0.0.0 (nevertheless MSBuild works well with it, so it seems it is VS limitation).

VS2010 (without SP1) switched to 10.0.0.0 with the same name. Well, to be precise file version is 10.0.30319.1. If your unit testing project references 10.0.0.0 – it can’t have TargetFrameworkVersion=v3.5. VS just sets it back to v4.0. So if you need to test v3.5 assemblies with it – well, you’re in trouble.

Fortunately, MS recognized the problem before I faced it and with SP1 there’s new version of the assembly – 10.1.0.0 or 10.0.40219.1. It allows you to target v3.5 framework and test in the VS2010. After I realized it and changed my references to 10.1.0.0 – native VS test runner get test correctly. Everything was green and bright.

Just until I ran it in the ReSharper 5.1 which I had installed. Well, it had test count correctly but don’t run anything. After some stack overflow searching I ensured that I’m not only one with the problem and that it is fixed in ReSharper 6.0. So I installed 6.0, run tests and… half of them were broken! Something is not right. Native runner got them well.

After some debugging I find out in Debug -> Windows -> Modules that resharper’s runner had 10.0.30319.1 version loaded. Maybe it is the problem was my first thought. Well, how could I force program to load other version of the assembly in .NET? With configuration, namely with assembly binding redirection. I had just happened to see MS guide to troubleshoot VS2010 unit testing. So my problem seemed very close…

So, I had opened my c:\Windows\Microsoft.NET\Framework\v4.0.30319\Config\machine.config and added there such lines:

<runtime>
    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
        <dependentAssembly>
            <assemblyIdentity name="Microsoft.VisualStudio.QualityTools.UnitTestFramework" publicKeyToken="b03f5f7f11d50a3a" culture="neutral"/>
            <bindingRedirect oldVersion="10.1.0.0" newVersion="10.0.0.0"/>
          </dependentAssembly>
        <dependentAssembly>    
            <assemblyIdentity name="Microsoft.VisualStudio.QualityTools.Tips.UnitTest.Adapter" publicKeyToken="b03f5f7f11d50a3a" culture="neutral"/>
            <bindingRedirect oldVersion="10.1.0.0" newVersion="10.0.0.0"/>
        </dependentAssembly>
        <dependentAssembly>
            <assemblyIdentity name="Microsoft.VisualStudio.QualityTools.Tips.UnitTest.ObjectModel" publicKeyToken="b03f5f7f11d50a3a" culture="neutral"/>
            <bindingRedirect oldVersion="10.1.0.0" newVersion="10.0.0.0"/>
        </dependentAssembly>
        <dependentAssembly>
            <assemblyIdentity name="Microsoft.VisualStudio.QualityTools.Tips.UnitTest.Tip" publicKeyToken="b03f5f7f11d50a3a" culture="neutral"/>
            <bindingRedirect oldVersion="10.1.0.0" newVersion="10.0.0.0"/>
        </dependentAssembly>
    </assemblyBinding>
</runtime>

After that ReSharper gets my tests correctly.

To be honest I don’t understand why this redirection actually solved the problem. We are telling runtime to load 10.0.0.0 instead of 10.1.0.0 in case it is requested, not vice-versa. But it is analogous to config file in many MS applications like devenv.exe.config, mstest.exe.config etc, so probably they know what to do with their own libraries better than I do.

17/05/2011

MSTest and SEHException

Filed under: Unit testing — Ivan Danilov @ 18:10

Corporate standards sometimes are evil thing, you know. And so on one of my work projects I’m forced to use MSTest.

So today I’ll describe the history of one problem I faced with Microsoft testing framework.

So, I’ve been writing some unit tests happily when strange message from next test run made me crazy:
‘Test method Test threw exception:  System.Runtime.InteropServices.SEHException: External component has thrown an exception..’

Without any clues what is the cause. Googling on the problem gave only not very helpful complaint about the same problem. After some time of trials I was able to narrow problematic case to the following:

[TestClass]
public class FailingTest
{
    [TestMethod]
    public void Test()
    {
        var mocks = new MockRepository();
        var someService = mocks.Stub<ISomeService>();
        someService.Expect(s => s.Method()).IgnoreArguments().Return(null);
    }
}

ISomeService is some interface from third-party vendor placed in it’s own assembly (let it be ‘Interface.dll’ for convenience) referenced from test project. MockRepository is part of Rhino Mocks (I’m just too lazy to rewrite code again without it, the problem is not specific to Rhino Mocks in any way).

But could it be some my error? I decided to check — referenced nunit.famework.dll, replaced [TestClass] with [TestFixture], [TestMethod] with [Test] and run. Success was the result. Well, something wrong with MSTest not my test code.

Maybe problem is within VS test runner? But ReSharper runner and VS integrated runner shows the same… Nevertheless I tried mstest.exe command line also. Same failure. Hmm…

It turned out that MSTest saw that Interface.dll has reference to some other assembly, Other.dll and tried to load it when testing. Why does it do such thing is mystery, but it does. If you put Other.dll to references of our test project – test will succeed. If you have no reference and no Other.dll accessible to the test – you’ll have adequate error message about MSTest can’t load required assembly named Other.dll. But (the most frustrating case!) if you have no reference to Other.dll but have it accessible (for example placed in the bin\Debug of test project) — you’ll have that terrible SEHException without any information.

The scenario is not as theoretic as you could think. Imagine TFS build server (God save us from it). With default settings it will put every resulting artifact (compiled assemblies, debug symbols, referenced assemblies that have Copy local=true etc) to the same Binaries folder. And if you have complex solution where Other.dll is actually used somewhere – it gets copied to Binaries. And if test project doesn’t have reference – you’ll have SEHException. Ghrrrr!

Blog at WordPress.com.