This might be the first question that pops to your mind when some of the unit tests in your solution fail. The second thing will probably be: it can’t have been me, I didn’t change anything that should make these tests fail! Sounds familiar? How can you check this and know for sure it wasn’t you?
New feature coming
Well, in the near future (when you’re switching to Visual Studio Team System 2010) you’ll be able to verify if it might have been you after all… You might still be unsure – or have difficulties with the fact it was you – but VSTS might just help you get off the hook saying your code changes didn’t affect those tests that fail. Sure, the tests still need to get fixed, but you just might feel better knowing you didn’t mess things up 
You probably already know I’m referring to the Test Impact Analysis Pieter talked about yesterday. Well, Pieter’s post reminded me of something we discussed some months ago regarding this subject. Actually, I was talking to him about a feature request around this matter, but what I had in mind still was something different. Yet, I don’t really know if a lot of people would actually like (or use) such a feature (a lot). I guess it would be nice to have, but not more than that either.
The situation
One day, I was found myself in the situation that I made quite some changes to the code base (and didn’t follow the best practice of “check in often”). After merging the latest version of the source code with my changes I was ready to perform my check-in. Just a quick test run so I could feel assured nothing was broken and I’m off, I thought. You know the rest of the story? Right! A lot of the unit tests failed preventing me to check in my changes. Obviously, I didn’t feel comfortable seeing all these red lights, but still, I needed to get the code checked in as soon as possible (isn’t it always the case that all goes wrong when time isn’t on your side?). I was pretty sure I didn’t cause all the issues, but still, you can’t just assume it’s not your problem and still check-in. In my opinion red lights are everyone’s problem, no matter who or what caused them!
There I was in the middle of ‘the guessing game’ looking for the cause of the failing unit test. And you all know: finding the problem is the first step to the solution. That’s why at that point I thought it would be nice if Visual Studio would help me find out if my changes caused the failing tests, or if I had to look somewhere else for the real cause.
Current options
You’ll say: “you could do this manually” Yes, I know, you could shelve your own changes, undo them and check if the unit tests work with the latest version in source control. After you’ve checked you could unshelve your changes and start fixing the problems with a better picture of the problem at hand. But you know, time wasn’t on my side on this one… so it would be nice if Visual Studio offered some automated way of doing this. If you’re lucky, you might just need to check the CI build if that runs the unit tests after a succesfull build. Which would be the fastest way to know. But since you can never have enough information, I feel there could be more to the feature I’m requesting than just checking the latest version under source control.
Feature request
The feature to determine if you caused the tests to fail could do the shelving, undoing, testing, unshelving for you. Hell, it could even check the situation when you started making the changes, going back to your last ‘get latest’ or even your first file check-out since your last check-in. So there’s some metrics that could give you the bigger picture in failing the unit tests.
- The current test results (failure)
- The test results of the situation you started off with (some scenario’s possible)
- The latest version without your changes (this is the automated shelving/unshelving part)
Depending on the results of n° 2 and 3, you could determine where to look for the problem. It could give you different results per tests, helping you determine where some issues found their way into your code base. When and where it started going wrong. Another option could be that – per individual failing test in n° 1, you could go back the trail until the test still succeeded. This way determining what changeset you should examine first. It’s not feasible doing this by hand, but if VSTS could offer this out-of-the-box in some automated way and offer you some nifty report, this could help, don’t you think?
What do you feel about this? It sure needs some more thinking and reality-checks. But there sure is room for improvement in making unit tests more loved than feared, don’t you think. Are these ‘metrics’ and reports I described interesting things to know? Would you like/use such a feature if it’s robust and well performing? It doesn’t have to get as complicated as my ramblings about the feature. I’m sure a clean solution for this must be possible. I also feel it has to be feasible if you look at some features as historical debugging and the test impact analysis the team is adding in VSTS 2010.
Let me know!
