Testing dojo session 8 - confront with the right tone

Imagine you see another tester testing a text field. He inputs valid&invalid values, special chars, then moves on to other elements. Wait, you think, he missed empty value and the max length of the field. You want to tell him. What will you start with?

"Hey, you missed empty value, and the max length". No, that's weak

"You made a mistake." Better. But not serious enough.

Eventually you come up with this one. "You are a bad tester. I can't believe you are testing so carelessly. Do you understand how important your job is? You are the last defense of quality! If you miss 1 case, it will affect the whole product, and of course customer satisfaction." Here you go. 

You start by explaining how important this product is, how important his role is, how a simple mistake would affect the whole team, blah, blah, blah. Because you want your voice sounds serious, authoritative, being heard once and remembered forever. But let's be real, how many times did it work?

Imagine you are testing a product on a tight schedule, hard work for several weeks. Right at this freaking busy and tired moment, someone stops by and says. "You are a bad tester." What's gonna happen next? Of course you start to defend whatever you are doing.

"Oh, empty value? I covered it in a precious text field. I'm on a tight schedule, couldn't cover every type of variable in every text field"

or "I'm using pairwise."

or  "Of course we can cover all types for all elements, as soon as our automation is ready."

You might not cover this "empty value" at all, but you would still say so. Because you work hard, you can't just let him judge your job like this. How dare he calling you a bad tester?

I find myself a hard time confronting people in both situations. I seldom succeed, even that I do, it costs too much time and energy, and I piss off someone. 

I look for answers from others, and find them divide into 2 groups. Group A loves to confront, it makes them feel more powerful and authoritative, they would do as much as they could; while group B feel it is boring and outputs nothing, they simply shut up. Neither group solves the problem.

Then we started testing dojo. Confronting others becomes a must for every session. The performer is always challenged. It offers me a close look to see how people confront each other, how they tell others their opinions, and how they receive them. I find that the more emotional, personal, judgmental an opinion sounds, the harder the defend will be. And even though everyone knows that we should take nothing personal, it is quite easy to make a conversation personal. See if you say or hear these a lot:

"You are making a mistake."

"Clearly you didn't understand how important this is to us, if you do, you wouldn't do such a bad job."

"You should improve your testing skills."

"Why you didn't test that?"

"How could you miss such a simple case?"

In dojo, we challenge to help people understand what they could do better. But "you made a mistake" doesn't give them a clue. They still don't know how to make things better. On the other hand, a precise challenge, like "we still don't know the boundary of this variable, let's find out", is quite helpful. They are easier to be received. People who get these would not feel offended, they would consider right away whether it makes sense, and give a try.

After realizing this, I start to pay attention to the language I use in dojo. When I'm an auditor, I say, "maybe we could try with empty value, see how the system handles it?". Less "you should...", "I would...", more "what if we..", "can we...". 

And when I'm an performer, and getting judgmental and blaming challenges, I would reply with, "Tell me what do we do next?" or "what exactly to improve here?"

Even though I'm trying hard, it is still a little hard for those who attend dojo for the first time. 2 weeks earlier, we organized a testing dojo with developers, trying to improve their testing skills. 3 developers were involved, 2 of them refused to take the performer role, the last one took the job unwillingly. 

The performer was quite defensive from the beginning, he asked a lot questions before started testing. He said, "if I don't get it clearly, you will blame me for not doing it later." No matter how many times I told him I wasn't there for judging nor blaming, but helping, he still showed this "I'm angry, be nice" face all the time.

In the book "the fifth Discipline", the author suggests an exercise to master your confronting skills. Draw a vertical line in a white paper to record a conversation. In the left section, write down what kind of tone you need; in the right section, write down the real conversation. See if you are who you think you are. I tried once in dojo, and I was surprised I talked so much just to convince someone was wrong.

Testing dojo session 7 - Scenario testing part 2 - boring

In this session, we continued with BBST video lecture / scenario testing part2.

Doctor Cem Kaner started by introducing how he met Hans, a weird but good tester, and how Hans considered Soap Operas as "the boundaries of life". I guess "Soap Opera" is something like "Friends" or "Growing pains", where several major characters get together, break up, and together again. In Friends, Ross has already devoiced 2 times, and yet, he devoiced, again, the 3rd time. You never know what might happen next.

Back to the lecture, Doctor Kaner demoed several ways to create a "soap opera" story:

• set extreme value to variable in a story
• repeat steps many more times
• create a hard to survice environment

It reminded me 1 old bug. I was testing a component which collected a list of files, I asked developers, "what's the limit here? how many files do we support?" "As many as users want, we do not set limit for that.", they replied. I created 200 files, and it failed. "No body in real environment would do with that many files." Developers explained. A few months later, a customer reported the same bug. Error log showed that he was trying to input 902 files.

Then Doctor Kaner listed 16 ways to create a scenario.

• create a mock business, treat it real and process its data
• try converting a real life data from competing application
• list possible users, analysis their interests and objectives - I develop this one by myself without watching the video :-)

I love these 3 because after domain testing and function testing, they could still reveal something new about the system and customers.

In the end, Doctor Kaner reminded us the limitation and risks of scenario testing.

• hard to maintain cases
• not good at reveal coverage
• should not be executed before domain and function testing

After the lecture, we moved on applying scenario testing with an in-class-activity assignment - "select 1 way from 16 and apply it to create 3 scenarios". The number was overlooked, which was a big mistake, and we started creating one scenario after another. The more we created, the more boring it became. We couldn't stop because we were not sure if we finished it, and we couldn't continue because we didn't know where this was going.

To kill this boring part, I decided to switch this dojo to a rounded dojo, each tester contributed an example on how the 16 ways could help to test their current project, we passed the ball around the room and every one was contributing. After several rounds, we shut this session down.

One good reason for the rounded dojo is to keep everyone involved. And figuring out how to apply a technique, coming up with an example is one of the best way to make sure people understand something.

If you organize your own dojo some day and suddenly boring shows up, I suggest you try this.

Testing no-dojo session 6 - Agile testing part 2 - tons of bugs

After I posted Agile testing part 1, one developer commented it on twitter. "Nice article, your next challenge is how to sell your idea to developers." He said. The idea he meant was unit testing. Among all components I tested, there was only 1 component with unit testing. "Because it is easy to write unit test in this one." Developers said. That sounded like right. Do the easy part! And hey, the results didn't look too bad, on the contrary, it looked better. By avoiding unit testing, developers released earlier. A couple more bugs found by testers, so what?

It seemed like right for a while. Until some day, a developer needed to commit a piece of core code that might affect some other code, he was not sure about the consequence. So he came to me and asked,

"Is there a way to detect that my code doesn't break other code?"

"Yeah, unit testing." I said.

"What about automated regression?"

"Sure, but that doesn't detect every thing, and you have to wait 4 hours."

Developers always considered this automated function level regression could detect everything. Because it was built by testers. And if it failed to detect a change, blame testers! But what they didn't know was that functions in one component always involved other components, machines, network, and some other environment issues.

Brady as a developer with testing experience explained it this way, "let's assume 90% chance of each factor might work, and the chance to successfully execute a regression is 0.9*0.9*0.9*0.9*...*0.9." And the worst part was when a regression failed, tester needed to check up all factors to find the cause.

Pretty soon, developers found that it wasn't perfectly safe to commit code, even with automated function level regression. So they started to commit less, as little as possible.

Elisabeth mentioned the exact scene in the next half of the lecture - "Technical Debt", when people chose short term pleasure and ended in long term pain.

She also introduced what Agile testing iteration was:

* Automated acceptance testing => Automated unit testing => Manual exploratory testing

And after the code passed unit testing and acceptance testing, testers should still cover 3 kind of cases in exploratory testing:

* end-to-end - usability

* sequence - scenarios involve complex steps

* data - historical data

She explained why it was hard for management team to accept Agile:

* customers, testers, developers work together on acceptance cases

* testers and developers work on unit test

* all involve in exploratory testing

In agile, the whole team involved in all steps, therefore everyone was responsible for the potential risks. Traditional managers always were looking for someone to blame when things went wrong. That wouldn't work in Agile Team.

In the end, Q&A started. I got to say I seldom enjoyed Q&A session, but I loved this one. Elisabeth mentioned that Microsoft released a package with 63,000 bugs fixed. " I wander what kind of bug tracking system could handle this many bugs. Instead of providing more features in bug tracking system, I think they should reduce bug count in the first place." She said.

This one really touched me. Without the power of unit testing, our team had over 3000 bugs, and 1000 waiting to be fixed. Each release, testers would find over 200 bugs, and verify almost the same mount from last release. To make testers' lives easier, I developed a series of tools to help us, including batch creating, resolving, verifying and searching. But no matter how many tools I provided, I just didn't feel right. Now I knew, I wasn't focusing in source of the problem. Reducing bug in the first place, then I could just throw away all these tools.

Isn't it weird? When I finally get to the right answer, part of me feel happy for knowing it, the other part feel sad that I didn't know it earlier.

Testing no-dojo session 5 - agile testing part 1

Last Friday we had a discussion with developers about whether our team was doing Agile. Developers claimed that according to a 10-question quiz which was supposed to help people check their Agility, our team matched "yes" to over 6 of these 10 questions. Therefore our team was an Agile Team, especially with the 0 document part. That was true, they did not provide any document to testers in the past 4 years.

But for what I knew about Agile, it was about practices and principles.
So I asked:"what was the practices and principles we had in our team?"
"0 document", developers answered.
"Any other one?"
"We release fast."
"Any other one?"
"Oh, and we do not follow process. That's quite enough. We match the major spirit of Agile."

I didn't continue my challenge, because I knew Agile only from several infoq videos. I chose not to discuss something I didn't quite understand. So the discussion ended.

This week while we were scheduling our dojo, Qia suggested 1 video lecture on agile testing. She got confused with the discussion, and wanted to know what Agile was, and how Agile we were doing in our team.

So we started our session with the lecture - Agile Testing by Elisabeth. In the beginning, Elisabeth introduced how she met XP, and got herself involved in her first Agile/XP project. With her former experience in a tradition team, she made a comparison between the 2 teams.

Then she started introducing Agile including 4 major methodologies: XP/Scrum/Lean Development/Crystal. She claimed that most people considered Agile was all about chaos, but a real XP team was quite disciplined.
* developer work with customer to come up automated acceptance testing - as a replacement of specification
* developer write unit test before they start coding - as design document
* developer commit code only after they pass their unit testing

Then she moved on to testing in Agile Team, how testing diffed between traditional team and agile team.
* plan for iteration
* instead of being the last line of defense, become team support

We ended up in the middle of the lecture and started discussion. We focused on 2 major questions
* are we agile
* how could we apply it

We all knew that developers refused to do unit-testing. "We tried, but it really slowed us down." They said. The schedule they referred here was called "development cycle", it counted from the day developers started coding till the day they released the package to testers. Without unit testing, developer could release the package in shorter time.

But hey, without unit testing, testers would find more bugs, need more time to investigate, more time to file it, more back and forth for developers to fix bug and release again and again. Developers knew that but they simply didn't care. Because after the day they released package, "testing cycle" started. Why developers should give a damn about how long it would take. The longer testing cycle took, the worse testers looked. Everyone loved developers, because they always beat the schedule. Everyone hated testers, because they always got beaten!

After the complain, I still found something I could do - become team support. So I went back my seat, found out the bug list for the latest release, and went through all the bugs 1 by 1. Among all 16 bugs, I found 5 bugs could be avoided if I could give comments to developers before code commit. So I sent out mail with the evidence. 30 minutes later I got a confirm from developers that from now on I would be invited in the code review. That was not exactly what I hoped for. But at least I would not need to file those not-so-smart bugs any more.

BTW, Elisabeth mentioned Mary in the section of lean development. Mary is one of my favorite speakers among all infoq speakers. The ideas she has on Lean development and Deliberate Practice lights my way in darkness. I would encourage everyone who considers software engineering as craftsmanship to watch her series on infoq.

Testing dojo session 4 - scenario testing part 1

We almost missed this week's session because it was too hard to have everyone available at the same time. But Yan insisted, she said:"once we let 1 session pass, there will be more sessions get away. After that, we might just never get this weekly activity on track." So thanks to her, our dojo continued. And this week, we learned scenario testing.

Scenario testing was my favourite. In my testing report, there was a section called user tory - an alias of scenario testing. There, I listed all stakeholders with their interests, and tried to figure out what they wanted to see and to do in the story.

In the past 2 years, this user story section helped me to find a lot of interesting bugs. For example: 2 years ago, I was testing a web application. Before I started, developers did a short demo on what to click, how to check logs and database records. I tested it for 2 weeks. Then one day I decided to write a user story like this:
I am 40 year-old manager, my daughter starts dating some wild kid, my wife just spent 1000 dollars on some stupid sallon VIP card.
Then I login the web app as him, I found a search field on the top of the main page. It was so weird that I never saw it before. Developers didn't mention it, and I was busy working on variables, boundries, domains. But now after I login as a manager with zero knowledge about the app, I noticed it. So I did what a new comer would do, I searched my name, then my department name. None of these search actually worked. Later I proved that only very few queries could work in this search field.
I reported this bug with the scenario and how it confused this poor manager. The bug then got a priority more than several functional issues, and got itself fixed a week later.

Just like my story, the lecture started with a live example. At the end of the example a bug finally got fixed because testers told the right story. Then lecture moved on to " what is ideal scenario testing" and "why we need scenario testing":
* ideal scenario testing
** based on a story, motivating, credible, complex use and complex environment, easy to evaluate results
* why scenario testing
** explore expert use, make a bug report more motivating

It took us 30 minutes to watch the first part of the lecture, and 10 minutes for discussion.
Jing: The story part took too long, even that I know scenario testing could be important, I still do not know how to apply it.
Other testers were all lost while following examples mentioned in the lecture, so we had to start over again, just so to have a better understanding.
I actually didn't like this. For most of the lectures, there would always be something that we didn't understand for the first time. But we didn't have to understand everything in the lecture to learn and apply it. All we needed was to catch what his point was, and how he was proving it.

Then we moved on applying it. This time our SUT was a code checking system. [Admin] defined [several] [rules] for the code, like no logging password in code, always add comments for class, etc. The system did the checking before [user] committed code. If the code was including some thing like "system.log $password", the system would warn user and stop commit.

Though we didn't get to the part on "16 ways to do scenario testing", I applied some of my user story tricks, and we made a sheet like this:
| stakeholders | interests | background knowledge | tasks | desire |
| admin | garantee good code | code structure, invalid code format | define rules | define 1 time, work forever |
| user | easy commit code | coding, commit tools | write good code, commit | pass commit check |

1 scenario we designed was that:
admin added 1 new rule, with interest - wanted this rule to prevent bad code from committing, but also with desire - wanted this rule work forever. So he/she must create some kind of way to verify the new rule. But the system currently did not provide such function. It was exactly one of the reasons why we need scenario testing - bring requirement related issues to the surface.

Testing dojo session 3 - domain testing part2

In this session, we continued to study domain testing with BBST domain testing video lecture 2.

At the beginning of the lecture, Cem Kaner did a short review on lecture 1, he claimed that examples mentioned in lecture 1 were all about numbers, therefore linear and with clear boundaries. But in real life, most of the variables were not linear. So how do we classify variables with values in non-ordered sets?

Cem Kaner gave us a generalized concept for "equivalence" including 4 views:

* intuitive similarity

* specified as equivalence - specification

* equivalent paths - code path

* risk based - same type of possible error

With each of these 4 types, testers should be able to classify several domains of a variable. But which one to choose in 1 domain as the best representitive? Cem Kaner suggested using risk driven method.

We spent 30 minutes on the lecture and 10 minutes on discussion.

* Jing: Lecture 2 is not as good as lecture 1. In lecture 1, it introduces a sheet so that we know how to apply domain testing, there's no sheet in this one.

* Mei: if there is a perfect tester, he/she eventually will figure out the same set of test cases with any type of the 4 views.

I agreed with Mei. If we had a perfect specification, then it would include all possible variables and how the system would handle them. If we had a perfect developer, then he/she would handle all exceptions in the code. As a tester, we could follow either the perfect specification or the perfect code path to come up with the same set of test cases - the complete test cases. But as we knew there was no perfect, we could only rely on principles and practices, that was all we have, and we could try to get closer to perfection by mastering those techniques.

Then we moved on to apply domain testing. This time our SUT was a dir scanning system.

User started by writing [several] [dir paths] in 1 file, then started scanning. The system first parsed file to get the list of paths, then scan each dir 1 by 1. After finished scanning, the system outputed the result including total number of dirs and files inside these paths, and the total size occupied on disk.

We updated the domain testing sheet mentioned in lecture 1, added 3 new sheets. Each sheet represented for 1 view. And a small part of our sheet looked like:

| variables | similiarity |

| several | 0 , 20 |

| path | short path, long path, relative path, absolute path, path including glob expression |

| variables | risks |

| several | 128 - might break unsigned int for line numbers|

| path | empty line, empty file |

| variables | code path |

| several | 128, 1024, 65536|

| path | path name including space |

Since we didn't have a specification, we skipped the view of specified.

Jing: each view has something overlapped with each other, is it right?

I thought it was not the overlap part we cared. What we cared was that every view lead us to something new, something we missed in other views. That was the reason why we needed different views, different methodologies, to review more, and to hope for a better result.

Testing dojo session 2 - domain testing Part 1

After the first dojo, we realized that we had been working on one project for too long:

* test cases had been there for 3 years

* automation took care of most of them

* new features were somehow related with the old ones, need not much attention

Therefore we started ignoring testing methodologies, we executed test cases like it was some kind of routine. When we tested something new, we followed our instinct. To stop becoming black box software tester, we decided to re-learn all testing methodologies, started from BBST.

BBST(Black Box software Testing) was a series of free online testing courses including both video lectures and exams. In every lecture, Professor Cem Kaner mentioned not only testing techniques, but also real examples from his experience. That gave the audience a brief understanding about how these techniques could be applied.

At the beginning, we spent 30 minutes watching the video lecture "Domain Testing" part 1. Some notes that I took:

* test broadly before deeply

* boundary table - | boundaries | valid cases | invalid cases | special cases |

* invalid combination - each variable itself is valid, but the combine of 2 make an invalid input

* consider output domain

After a short discussion about the lecture, we found our SUT-vocation submitting system to apply domain testing.

Vocation submitting system was a tool provided inside every company. [User] login, input the [date] when he/she decided to leave and the [date] when to be back. The system calculated the [time] in duration, if it was bigger than the [time] that user had left, it rejected the request; otherwise, it sent out a email to user's approver which usually was his/her [manager]. After the manager approved, the request took effect.

While testing, we connected 1 laptop to the projector. Everyone contributed ideas and one recorder kept recording. We used a tool mentioned in the lecture to guide our ideas. It looked like this:

| variable | risks | classes should not trigger failure | classes should trigger failure | best representatives | notes |

Now we needed to find variables and analysis domains for each one. I created a very handy tool to build variable list. Finding every noun in the user story and labeling them with [ ]. I applied this to the above user story, then we had a variable list: starting date/ending date/time duration/time user had/user name/manager name

The sheet went too big to fill in this blog. Among them several test cases I'd like to highlight though:

* combination

** both 2010 as year, 2 as month, and 29 as day were valid values, but a combination of 2010/2/29 was not.

** both 2010/2/1 and 2010/1/2 were valid dates, but a combination of 2010/2/1 as starting date and 2010/1/2 as ending date was not

* output domain

** output domain was usually missing because user story did not go that far.

** e.g. user requested a leave date, and the system saved this request into database. Here database was an output target. The system needed to insert this request as a sql statement. So here sql statement was an output domain. A special char or sql keyword in the request might cause some trouble, a sql statement in the comments of a request might even leave hacker a hole to hack into database.

Similiar cases could be:

* If output variable would be saved as a filename, then "/" might cause some trouble. And for linux, 1024 was the longest filename, another boundry to check.

* If output variable would be saved in mysql database, then "AA" and "Aa" and "aa" might cause some trouble because sometimes sql databases default configured as ignore-case.

* If output variable would be a list divided by space, then "a " might cause some trouble.etc.

Testing dojo session1 - black box software tester

It was a Monday afternoon, our testers finally got together. So we decided to start our testing dojo.

Testing dojo was an idea I stole from Coding Dojo. A dojo took a projector, a pc/laptop,

and 5 - 10 testers. 1 or 2 testers played the role as performer who tested on the pc which was connected to projector. While others played the role as auditor who watched the process on the projector. After testing finished, auditors could challenge on the testing process, tools selected, testing methodologies applied, etc.

In this session, we decided to do pair testing:

* one tester testing and

* one helper recording issues and contributing ideas.

Jing as tester, and Yan as recorder, made the performer team. The SUT(system under testing) was a http proxy server with white list control.

The performer team spent 10 minutes discussing the functionality, then started testing.

* Jing started Firefox, and configured proxy server to the system.

* She defined several allowed URL in the white list, then tried to access the URL in Firefox.

* She tried to access several out of scope URL.

* Here Yan reminded the risks with special char in URL, Jing followed her suggestion with ? and # in the URL.

* then a series of allow/deny with different types of URL. (ftp, http, https, file://)

Testing part took about 50 minutes. That left us 30 minutes for dicussion. All 3 of our auditors were lost while following Jing's testing process. So we asked a lot of questions to clear our minds.

Q: Are you doing function testing or scenario testing?

A: I'm not sure. I just did what I think is necessary.

Q: If you are not certain about what kind of testing you are doing, how could you focus?

A: I think both function testing and scenario testing would end up into almost the same test cases eventually.

Q: while you are testing special chars, why those 2 - # and ? - but not the others?

A: I think it is the most commonly used special char in URL.

Q: Most commonly used by user? So you are doing scenario testing?

A: Yes, I guess. I might be doing user scenario testing, I think that's what users probably do.

Q: What would you call it - your methodology today?

A: I think it is kind of like exploratory testing, I do not know the software before, and I'm trying to get familiar with it while testing it.

Our session ended in discussing whether a tester should always be aware and follow some kind of testing methodology or tester sometimes could follow one's instinct. Some of us believed that testing a new product was OK to follow one's experience or say nature, just test whatever occured.

I disagreed. Tester should always be aware of testing methodologies. Tester without testing methodology became black box software tester. It was not the one who did black box software testing. It was a type of tester who took requirements or sometimes no requirements, pre-designed test cases or sometimes no test cases, and the SUT, went back into a black box. Several days later, the tester came back with a list of bugs. No one would ever know how he/she did it.

That including

* how he/she analyzed the requirement,

* how he/she intepretted the test cases,

* how he/she interacted with the SUT.

We could only believe in black box software tester's experience or instinct. That didn't sound good. But the truth was everyone was a black box software tester for some time. When you were tired of a product, when your schedule was too tight to fit in, when you were unfamiliar with the SUT, when you had test cases too hard to understand, you became

black box software tester, you knew you were testing, you were finding bugs, but you didn't know what exactly you were doing and why you were doing so.