Tuesday, May 12, 2009

I failed!

I failed! Yes I did and I'm not ashamed about it either. I learned an important lesson and I thought I’d share it.

As I have mentioned before on this blog I am involved in both testing and in developing a PDM system with a large number of users and it was in this system I failed to find a pretty nasty bug.

The bug had been in production for some time and no one had reported any problems. Now we were putting in new functionality that opened this Pandora’s Box. The bug was particularly nasty because as a user you would get no feedback from the system at all even if you fell victim to it. This is an extremely nasty class of bugs. The bugs that will mess up data and you have no idea.

So what was it? What did this nasty bug do? It changed data in the background, underneath an edit form, causing you to believe you were changing metadata for one object when in fact you could be changing the same metadata for a different object. The metadata not being displayed in the standard presentation of the objects you could do this without seeing it. The only way you’d know something was wrong was if you looked for the change you thought you made and discovered it wasn’t there. You might guess that you closed the editing form without committing the change or that possibly the system failed to commit the data. I don’t think that a lot of people, even testers, would think that perhaps they changed the data for some other object. If your testing a brand new system you would look for things of that nature but in a system that has been live in production for years and has been evolving over those years you would, at least I would and did, assume that there’s no way a bug of that kind could have slipped through. That was one of my mistakes! Don’t make assumptions without testing them . I know this, and I’m sure you do to, but the human mind being what it is, we assume things. We take things for granted and we are biased. Might as well admit it. It’s true.

We found it, or a colleague of mine found it, and there was much rejoicing. Well, that’s not the counting the poor guy who had to fix it quickly since we were in the middle of putting a new release in place in the production environment. He did pull of a good enough fix.

All is well that ends well and the good enough fix will be made better and eventually launched into the production environment in June. I learned a few things and that is always good. Don’t make assumptions without testing them, test all around and question yourself if you don’t find any bugs. Test all around is my key to remember to test clicking on stuff outside the form/dialog/div that I’m currently in and also to looking for interactions between the surrounding context of a form/dialog/div that might not be obvious or even readily visible.

Think for yourselves. I try to!

Sunday, April 19, 2009

Tester Tested!: Checkmate heuristic :: A security testing attack

I just read Pradeep's blog Tester Tested!: Checkmate heuristic :: A security testing attack and I just loved it. Thinking is the key here, as it always should be. If you want to test something you can't do it without thinking. Following a classic scripted test case means you'll have to shut down your natural curiosity and your natural thinking. You need to be in machine-mode and just perfom. I'm sorry, but you'll never find anything that way. You'll only find what the developers already should have found and fixed.

In one of the projects I'm working on currently we do have something that we can call a script, if we want to and if we, for political reasons, need to. It's really just a map of the system written out in the english language instead of as a drawing. It shows the whole system on two levels. First it's the five, or six, major functionallity areas of the system and secondly it's the major functions within each area.

This is really all we need and since the people who do test the system, and the new releases, are thinking human beings, it works great. We tend to find issues and bugs in every release and the most interesting are the ones that have been in the system probably for years without us finding them or anyone reporting them. They may or may not be critical. Some are and some are not. Anyway we are all excited when we find a bug! If it's an old one we tend to get even more excited. Finding bugs in the new, or changed, functionallity is easy. It's finding the old ones that is hard, and therefore more fun.

I'll keep posting new stuff as often as I can, and if I find something worth sharing.

Wednesday, February 18, 2009

I found a bug!

Yes, I found a bug and it made me so happy! Last Saturday we deployed a new release of a PDM system at a large company and everything was going fine. We had all new components in place and were testing to make sure we hadn't missed anything when I decided to test something we hadn't changed in this release. Why? Well, it was a case of "Why not?" combined with "The wrong place". I need to come up with a good name for this combination of heuristics/triggers that I find useful. "Why not?" is, in my book, simply applying a punk attitude and question the system profoundly blended with a bit of reckless abandon, also very much punk. In this case it led me to "The wrong place". The wrong place is any part that is not affected by the changes deployed. Or any place of the system where I believe that the changes I test have any impact at all. So I decided to test input that produced a problem in one part of the system, where we had made changes, in a completely different place. This place also being one that handles input in a totally different way for different purposes and in this case a place that was not affected by the changes we deployed.

I did find a bug! The tester in me was excited and happy and knowing the system well, having been a developer on this system for several years now, I also realized that this is an easy bug to exterminate. Since we were deploying a new release in the production environment we didn’t do anything about the bug other than make a note of it.

Had I not been thinking for myself and used my testing tools the bug would still be there. Undetected, unreported and unsolved. The bug wasn’t critical but it still was a bug that caused loss of information, loss of meta data in this case, unreported and difficult to detect since you would have to know that there was supposed to be more information than you see. It’s never easy to spot what isn’t there! "What’s missing?"is a trigger I use when testing. It demands creativity and an open mind and is something I constantly have to remind myself of using. Is something missing here? Could there be something missing here? In most cases I find that if there is a time you ever needed an oracle this is it! Go talk to the developer, if available, or RTFM. Anything, or anyone, that can provide you with more information.

I'm still learning and I get excited when the tools I've learnt to use works so well.