Why do we test? - A bit more on the subject

I'm very happy with the response I got on my last post about the book project. I thought I'd stay on that subject and write some more about it.
My starting point for this project was the "Why" that seems to missing or at least very strange when you encounter some PM's and others that thinks that brush test off with: "Oh, that's getting to be too much. We don't need you. We don't need that much, better do it ourselves", and other similar comments. I'm trying not to get stunned and speechless when I encounter such ignorance, and I do believe that's exactly what it is, about test. This is of course an educational issue and sometimes you need a clever approach to be able to get the point across and teach these people about test. It is probably not easy and it is challenging. This is a good thing. When it is difficult and challenging it forces you to think about your craft and it forces you to work on your arguments and teaching tactics. Even if it's difficult to embrace the chance to learn and become better at our craft in a situation where you most probably is pretty upset and down right steaming mad, try to cool off and use it to your advantage if you can.
This "Why" depends on the context just like the other "Why"'s that I'll try to cover.

These are some of the different "Why"s that come to (my) mind:

• Why do we test? - on a general and perhaps abstract level that's related to non-testers attitude towards test, as a craft and as a service.
•  Why do we test? - on a personal level. Perhaps I test because I love to explore and enjoy discovering things, or I test because it's a nice job and it helps me pay my bills, or I'm not a tester I just do this because the PM told me I had to.
•  Why do we test? - in this particular project. It may be regulatory demands that needs to be fulfilled or it may be a business critical production system that has high availability or high security demands. The reasons on this level are not always as clear as they perhaps should be. Being clear about why we test on this level will help when decisions need to be made about how much effort and work is needed when designing tests or when designing automated checks. Communicating the reasons is also
•  Why do we test? - integrations. This is not as obvious when you think about it as it might look. Think about it.
• Why do we test? - performance, in this particular way or at all. This is also not as obvious as it first may seem.
• Why do we test at all? - Why not do automated checks and run those and be done with it? Why bother with testing? Do we need real human beings with brains and minds of their own testing this? 

I am far from done with this, I have only started to write and I will try to make use of the community to gather stories and experiences. There will be more on the blog as I write and I guess there will be one or two posts about writers block and distractions. It's only human, I hope!

Why do we test?

I recently started writing on what will be my first book. The title was the thing that came to me first together with the idea of subject. Why do we test? That's the title, at this point, and my intention is to gather up my own experiences and the experiences that are shared with me by colleagues and other testers around me. My hope is that it will be of some value to someone and most of all I view it as a great opportunity for myself to become a better writer. Writing is a large part of what I do as a tester and test manager. Strategies, plans, reports of different kinds are sometime too large a part of my job. I enjoy it so I shouldn't complain and I take every chance I can to make these thing more useful and usable and as short as possible. I'm sure I'm not the only one who have experienced that large documents never get used, never get read. They tend to collect dust on whatever disk space they occupy.
Back to the main question: Why do we test? The obvious answer would be: To learn. That's a correct answer but it's hard to convey why we test and what the value of test is with that short reply. It needs to be followed up, and backed up, with the reasons and arguments behind it. It is also different things to different people and that is one of the major challenges when communicating the Why. More than once have I seen test have its allocated time cut down and even completely scrapped in projects. Many times it has been obvious that the decision maker(s) have a very different view of what test really is and what test really does. More than once it has been too late to change the decision when I've gotten a chance to try to enlighten them about what test is and what test brings to the table but now, collecting and writing this book, I will build a nice collection, for myself and others, with experiences and arguments concerning why we do test.
I'll get back to this topic more as my work on the book continues.

A special thanks you to Mike Sutton for helping get off my lazy behind and write a blog post! He is writing to, here.

The certification experience

I have always taken pride in the fact that I have no degrees and no certifications since leaving high school many years ago. I must confess that it hurt my self image and my rebel heart when I agreed to get a certification. My manager, who ranks at the very top as far as managers I've had goes, wanted me to get the ISTQB Foundation certificate. I agreed to do it. Why? There are several reasons for this decision.

• I felt that it was OK to agree to it since I only started at this job a few months ago. But I also let my manager know that she'll owe me for this.
• I was curious about how this would be taught and what exactly was in the test.
• I wanted first hand experience so that I could criticize it and know what I'm talking about when I do.
• I'd get a chance to meet some more people working in test. Networking is always a priority for me.
• Probably the most important reason was this: I saw an opportunity to question the stuff being taught and hopefully that would get the other participants thinking and questioning as well.

There are many reasons for doing this but as you may have noticed I didn't expect any amazing revelations and I didn't expect to learn some new amazing skill I could use in testing. That would have been to naive I think.

The first day of the three day course started interestingly since the teacher agreed with me that there are no best practices. I liked hearing that but throughout the three days there were many occasions when I could see and hear how he tried hard to stay away from the best practice traps I tried to lay. I questioned a lot of process stuff and asked if one step was deemed better and more valuable than others. The material implied that the more "formal" a technique was, according to the syllabus, the more valuable it was and the better it was. I questioned this and pointed out this implication and the fact that this could easily overturn your test effort. I didn't get an answer about whether more "formal" was better or not, or considered better. I could see the teacher, or ISTQB-dude as I referred to him while tweeting about it, biting his lip and try hard not to fall in that "best practice trap".

It was obvious that what ISTQB called more formal meant better, to ISTQB. The problem here is that when ISTQB says more formal it includes heaps of documentation and meetings and in my world that means less time to actually test. It also, in most cases, contradict my belief that in order to discover things we need to explore and we cannot predict what we will learn. We cannot identify risks without first considering the context, or rather contexts. Without identifying the risks, and contexts, we cannot make any form of decision about what to test first or where to put in the biggest test effort. In order to succeed I must start with context in order to determine the proper course of actions.

I did manage to get the teacher to admit that we were learning this stuff for the exam. He did say exactly that. Sure, we do learn this for the exam and that means that since there is very little substantial stuff that can be useful in practice that statement in it self meant that three days doing this was truly a waste of out time. Personally I learn more useful stuff any three days of any year than I did doing this.

Models? Oh yes, there were many models presented to us. No new revolutionary models, I didn't expect that anyway, but some well known ones like state transition models etc. Now personally I don't have any problems with models and I think models can be useful. In the right context. With the right information to the people the models are presented to. The primary piece of information, for me, is the reminder that a model is just a model. It's not the whole picture and it never can be. It's just a model. When using a model it's imperative that all that are presented with the model are aware that it is just that. A model. It can be useful to spark new test ideas. Any state transition model would likely give you many new test ideas. The same goes for equivalence partitioning. It can show you a small part of what needs to be tested in an area of the system but don't fool yourself that it will provide you with all data you need to test that area. It's just a model. It can give you ideas about what to test and how to test. In many cases, in my experience, creating a model may take more time to do than simply consulting my own experience and thinking. Many times that will produce the same test ideas that a model will and it will do it faster. Sometimes starting on a model may be all that is needed to get the test ideas. If that helps, do it! To me there is no value in completing a model unless I intend to use it for communication or I believe it will help me get more test ideas. Simply completing it for its own sake has no value to me.

The main thing I will remember about this experience is when the teacher, the ISTQB-dude, admitted that we learn this for the exam and thereby implying that it's not learning to be better testers that is the point. I knew that going in but it's nice to hear a "believer" admit that it is so.

SWET2 - A great way to spend a weekend

Back home again, tired and inspired, after a weekend of test talk outside Göteborg. The second installment of SWET, SWET2, with 15 testers. This was my first peer conference and I hope there'll be more to come. The format was great and we had some very inspired, and inspiring, discussions. I come back with a bunch of new thoughts and ideas that I will try to mold into something I can use, when the opportunity arises.

Thanks to Rikard, Martin and Henrik for arranging a great gathering.

The delegates were: Christin Wiedemann, Torbjörn Ryber, Azin Bergman, Fredrik Scheja, Henrik Andersson, Johan Jonasson, Ola Hyltén, Sigge Birgisson, Simon Morley, Rikard Edgren, Henrik Emilsson, Martin Jansson, Steve Öberg, Robert Bergqvist, Saam Koororian. All contributed to a great weekend!

The tweets from SWET2 might tell you something about the discussions that went on

EuroSTAR 2010 - a great experience

I attended my first EuroSTAR conference this week and it was very inspirational and a great learning experience. I was reminded of a lot of stuff I had more or less forgotten and I picked up many new ideas that I will try out. Amongst the many talks and workshops there are a few that sticks out. Michael Bolton on Test Framing was very good. I picked up some very useful stuff attending that. Lynn McKee on inspiring passion in your team was excellent. She made a lot of good points and I will remember them and use them. There were some very good keynotes and Rob Sabourin, Anthony Marcano, Dino Petti, Bob Galen were all really good. They all put ideas into my head and I came back with some new ideas to write about. There were others that were inspiring as well. Henrik Andersson och Fredrik Rydberg made great presentations that gave me some new ideas to test. Ajay Balamurugadas was amazing when he presented Weekend Testers.

Perhaps the best thing was the Test Lab. A good place to do the fun stuff, test, and a great place to learn, listen, discuss with some very good testers. I came back from the conference with some good ideas for new posts on the blog and I will start writing this weekend. I might as well walk the walk and do my best to keep my own passion going and write while I still remember why I made those notes and thought of those topics.

Hanging out with some great people was inspiring and a lot of fun! Testing is fun and testers are fun! At least the good ones :)

Bug magnet?

Do you ever feel like a bug magnet? Well I do, occasionally. I experienced a bug, well it there were several wrongs involved, last Friday in Gothenburg. Here's the story.

I was in Gothenburg to see ZZ Top with my brother and we as we walked from out hotel to the venue I stopped by an ATM to get some cash to spend at the show. The ATM was located outside a bank, Nordea to be exact, and the ATM was a really old one. It has a small black and white screen and I know this model has been in use in Sweden since the 80's. Now I do guess the software has been upgraded several times since, but that is pure assumption on my part. Anyway, I inserted my VISA card, entered my PIN, as instructed. I then got to choose if I wanted a receipt printed or not, I chose not to. Then I entered 600, that's what I wanted to withdraw. Now is when the funny stuff happened. I get a printed receipt withe the text "Your bank does not allow the withdrawal", the standard message when you haven't enough money on your account. Then I get a message on the screen saying "Take your money". OK? What did just happen here? The machine prints, when I said I didn't want it to do that and it told me I couldn't withdraw that amount. Then it still gives me the cash! Just another Friday night for the tester. I'll send a link to this post to the bank and see if they react. I'll post the reaction if I get one.

I guess most people wouldn't think more of it when they get there cash but the tester in me noted the strange behavior and hopefully I'll make use of it when testing in the future. I did check my bank account when I got home and the withdrawal was logged as it should and there was more funds in the account, which I expected.

Learning experiences

So, the man who claims to be thinking for himself woke up and remembered he had a blog. I haven't written anything here in a long time but I will start over and try to establish some kind of flow.

I got an idea for a series of posts today. It was my own idea but I didn't see it until Pradeep Soundararajan kicked it back to me. He suggested I write a series of posts about what makes my "eyes and brains tiered". That was a challenge back at me based on a comment I made on his blog. So i will!

As the title suggests I will keep going along this track for a while. It will be about what I use to learn new stuff and maybe even how I use my new knowledge in my work. That may be related or relatable to testing. That is entirely up to you. Read it. Comment it. Use it. Don't use it. If you do find something useful please use it and remember where you found it. I give no guarantees that anything here will be useful for anyone other than me.

Experience #1

The other day my brother asked me if I wanted to join him look at a house. He and his wife are moving down here from Stockholm and I thought it might be fun and maybe even helpful to him. It was one of those evenings when it was just me and my 3 and a half year old daughter home so I put her in her seat next to mine in the car and we were off. On the way I we talked and I pointed out that soon we'll pass the garage where they fixed my car about a month ago. She started talking about that air comes out, at two places on my car. I was completely lost there for about 30 seconds until she said it was at the back of my car and that that was what they fixed. Then it clicked. She was talking about the exhaust, of course. It was air to her, exhaust to me. Two places, was really one. At least in my world.

The problem with the car was that after sliding, not very fast, of the road I got stuck in the snow and had to get pulled out of the snow by a very big and friendly tractor. The catalytic converter got beat up and my car got a bad case of asthma and the exhaust fumes couldn't get through so it leaked out and ended up in my air intake when I hit the accelerator. So it came out in two places!

I was so happy when I understood what she meant. She got all the parts right, using her own words. I had to use my tired brain to decode her language, she speaks very well and has a large vocabulary for her age, so it's usually easy to understand what she's talking about. In this case it took some thinking. I quickly realized, and actually thought this as we were driving along and I was praising her for remembering the details, that this is what I encounter everyday when dealing with clients and, not always to a lesser extent, my colleagues.

Today this helped me understand a clients problem late in the afternoon. His words were not the ones I would use but I remembered that drive and the talk I had with my daughter and I do believe I has a pretty good idea about how to find and fix the problem tomorrow when I go to see him.

Sometimes this is obvious. The fact that we call things by different names that hold different meaning and has different values for us. In those cases it's easy to see through it and understand the person we're talking to. A few questions and we're usually on the right track. The tricky part is when we assume that the person we're talking to is coming from the same place as we are. That's when we think that we understand each other but later we find out that not only were we not on the same page but not even in the same book! That's when it can get very expensive!

In testing we may be fooled by something that looks familiar and we think we understand what we're supposed to do or think we understand what is going on. If we don't question familiar looking stuff we can miss a lot. It's about communication with the system and testing it! I might use my "Why not?" heuristic in that case. It's about communication with oracles and here we really need to think in order to make sure we understand the oracle. It's not a very useful oracle if we don't understand it. To me it is all about remembering and recognizing the current context. I'm sure there are more ways were remembering this is important. I just can't think of them right now.

The point to me was that I learnt something in a context that was not one where I was expecting to learn. I didn't dismiss my daughter because I didn't understand her at first, or because she's only 3 and a half years old, and that fact alone made me proud of myself as a father, and I did learn an important lesson. Listen and think and keep listening while you think. It will help!

I failed!

I failed! Yes I did and I'm not ashamed about it either. I learned an important lesson and I thought I’d share it.

As I have mentioned before on this blog I am involved in both testing and in developing a PDM system with a large number of users and it was in this system I failed to find a pretty nasty bug.

The bug had been in production for some time and no one had reported any problems. Now we were putting in new functionality that opened this Pandora’s Box. The bug was particularly nasty because as a user you would get no feedback from the system at all even if you fell victim to it. This is an extremely nasty class of bugs. The bugs that will mess up data and you have no idea.

So what was it? What did this nasty bug do? It changed data in the background, underneath an edit form, causing you to believe you were changing metadata for one object when in fact you could be changing the same metadata for a different object. The metadata not being displayed in the standard presentation of the objects you could do this without seeing it. The only way you’d know something was wrong was if you looked for the change you thought you made and discovered it wasn’t there. You might guess that you closed the editing form without committing the change or that possibly the system failed to commit the data. I don’t think that a lot of people, even testers, would think that perhaps they changed the data for some other object. If your testing a brand new system you would look for things of that nature but in a system that has been live in production for years and has been evolving over those years you would, at least I would and did, assume that there’s no way a bug of that kind could have slipped through. That was one of my mistakes! Don’t make assumptions without testing them . I know this, and I’m sure you do to, but the human mind being what it is, we assume things. We take things for granted and we are biased. Might as well admit it. It’s true.

We found it, or a colleague of mine found it, and there was much rejoicing. Well, that’s not the counting the poor guy who had to fix it quickly since we were in the middle of putting a new release in place in the production environment. He did pull of a good enough fix.

All is well that ends well and the good enough fix will be made better and eventually launched into the production environment in June. I learned a few things and that is always good. Don’t make assumptions without testing them, test all around and question yourself if you don’t find any bugs. Test all around is my key to remember to test clicking on stuff outside the form/dialog/div that I’m currently in and also to looking for interactions between the surrounding context of a form/dialog/div that might not be obvious or even readily visible.

Think for yourselves. I try to!

I found a bug!

Yes, I found a bug and it made me so happy! Last Saturday we deployed a new release of a PDM system at a large company and everything was going fine. We had all new components in place and were testing to make sure we hadn't missed anything when I decided to test something we hadn't changed in this release. Why? Well, it was a case of "Why not?" combined with "The wrong place". I need to come up with a good name for this combination of heuristics/triggers that I find useful. "Why not?" is, in my book, simply applying a punk attitude and question the system profoundly blended with a bit of reckless abandon, also very much punk. In this case it led me to "The wrong place". The wrong place is any part that is not affected by the changes deployed. Or any place of the system where I believe that the changes I test have any impact at all. So I decided to test input that produced a problem in one part of the system, where we had made changes, in a completely different place. This place also being one that handles input in a totally different way for different purposes and in this case a place that was not affected by the changes we deployed.

I did find a bug! The tester in me was excited and happy and knowing the system well, having been a developer on this system for several years now, I also realized that this is an easy bug to exterminate. Since we were deploying a new release in the production environment we didn’t do anything about the bug other than make a note of it.

Had I not been thinking for myself and used my testing tools the bug would still be there. Undetected, unreported and unsolved. The bug wasn’t critical but it still was a bug that caused loss of information, loss of meta data in this case, unreported and difficult to detect since you would have to know that there was supposed to be more information than you see. It’s never easy to spot what isn’t there! "What’s missing?"is a trigger I use when testing. It demands creativity and an open mind and is something I constantly have to remind myself of using. Is something missing here? Could there be something missing here? In most cases I find that if there is a time you ever needed an oracle this is it! Go talk to the developer, if available, or RTFM. Anything, or anyone, that can provide you with more information.

I'm still learning and I get excited when the tools I've learnt to use works so well.

In the trenches and slowly moving ahead

I heard a rumour while I was attending Öredev in Malmö in november about a teamleader at my company. It was an inspirational rumour. The main thing was that he wanted to scipt more testing of the product he is working with!
Is this inspirational, you may ask, and I have to say YES. It should be inspirational to anyone who believes in the possibillities of an exploratory testing approach. There's a good debate in the making when someone wants to go in that direction.

We planned a meeting to talk about the impressions from the conference with a focus on testing. I went into the meeting with a guerilla attitude but no such tactics were needed. It was a good meeting. We quickly discovered that the they do exploratory testing, but in a very freestyling way and only rarely. The test lead on that team does it when completely new services and new major features are getting ready for testing and have reached a level where it can be done.

I should tell you that the product we're dealing with here does not come equipped with a UI. That part is provided to our customers by other companies. This makes testing a bit more challenging, and interesting. All ideas about testing under these circumstances are welcome. I want to learn more and I try to listen and learn from all sorts of sources.

Anyway, we didn't really arrive at any decisions about ET but we decided to keep talking about integrating it in the processes. The development team is working hard on implementing development processes that are intended to help raise the quality of their output and instead of trying to change to much at the same time testing, and test approaches, where put on hold. Have you seen this before? It's not all that bad, there is alot of positive vibes and an open mindedness for change that will lead to improvement in the delivery and development so I have hope for the future. I just got to remind them that there are ways to get better at testing to!