Testing dojo session 3 - domain testing part2

In this session, we continued to study domain testing with BBST domain testing video lecture 2.

At the beginning of the lecture, Cem Kaner did a short review on lecture 1, he claimed that examples mentioned in lecture 1 were all about numbers, therefore linear and with clear boundaries. But in real life, most of the variables were not linear. So how do we classify variables with values in non-ordered sets?

Cem Kaner gave us a generalized concept for "equivalence" including 4 views:

* intuitive similarity

* specified as equivalence - specification

* equivalent paths - code path

* risk based - same type of possible error

With each of these 4 types, testers should be able to classify several domains of a variable. But which one to choose in 1 domain as the best representitive? Cem Kaner suggested using risk driven method.

We spent 30 minutes on the lecture and 10 minutes on discussion.

* Jing: Lecture 2 is not as good as lecture 1. In lecture 1, it introduces a sheet so that we know how to apply domain testing, there's no sheet in this one.

* Mei: if there is a perfect tester, he/she eventually will figure out the same set of test cases with any type of the 4 views.

I agreed with Mei. If we had a perfect specification, then it would include all possible variables and how the system would handle them. If we had a perfect developer, then he/she would handle all exceptions in the code. As a tester, we could follow either the perfect specification or the perfect code path to come up with the same set of test cases - the complete test cases. But as we knew there was no perfect, we could only rely on principles and practices, that was all we have, and we could try to get closer to perfection by mastering those techniques.

Then we moved on to apply domain testing. This time our SUT was a dir scanning system.

User started by writing [several] [dir paths] in 1 file, then started scanning. The system first parsed file to get the list of paths, then scan each dir 1 by 1. After finished scanning, the system outputed the result including total number of dirs and files inside these paths, and the total size occupied on disk.

We updated the domain testing sheet mentioned in lecture 1, added 3 new sheets. Each sheet represented for 1 view. And a small part of our sheet looked like:

| variables | similiarity |

| several | 0 , 20 |

| path | short path, long path, relative path, absolute path, path including glob expression |

| variables | risks |

| several | 128 - might break unsigned int for line numbers|

| path | empty line, empty file |

| variables | code path |

| several | 128, 1024, 65536|

| path | path name including space |

Since we didn't have a specification, we skipped the view of specified.

Jing: each view has something overlapped with each other, is it right?

I thought it was not the overlap part we cared. What we cared was that every view lead us to something new, something we missed in other views. That was the reason why we needed different views, different methodologies, to review more, and to hope for a better result.

Testing dojo session 2 - domain testing Part 1

After the first dojo, we realized that we had been working on one project for too long:

* test cases had been there for 3 years

* automation took care of most of them

* new features were somehow related with the old ones, need not much attention

Therefore we started ignoring testing methodologies, we executed test cases like it was some kind of routine. When we tested something new, we followed our instinct. To stop becoming black box software tester, we decided to re-learn all testing methodologies, started from BBST.

BBST(Black Box software Testing) was a series of free online testing courses including both video lectures and exams. In every lecture, Professor Cem Kaner mentioned not only testing techniques, but also real examples from his experience. That gave the audience a brief understanding about how these techniques could be applied.

At the beginning, we spent 30 minutes watching the video lecture "Domain Testing" part 1. Some notes that I took:

* test broadly before deeply

* boundary table - | boundaries | valid cases | invalid cases | special cases |

* invalid combination - each variable itself is valid, but the combine of 2 make an invalid input

* consider output domain

After a short discussion about the lecture, we found our SUT-vocation submitting system to apply domain testing.

Vocation submitting system was a tool provided inside every company. [User] login, input the [date] when he/she decided to leave and the [date] when to be back. The system calculated the [time] in duration, if it was bigger than the [time] that user had left, it rejected the request; otherwise, it sent out a email to user's approver which usually was his/her [manager]. After the manager approved, the request took effect.

While testing, we connected 1 laptop to the projector. Everyone contributed ideas and one recorder kept recording. We used a tool mentioned in the lecture to guide our ideas. It looked like this:

| variable | risks | classes should not trigger failure | classes should trigger failure | best representatives | notes |

Now we needed to find variables and analysis domains for each one. I created a very handy tool to build variable list. Finding every noun in the user story and labeling them with [ ]. I applied this to the above user story, then we had a variable list: starting date/ending date/time duration/time user had/user name/manager name

The sheet went too big to fill in this blog. Among them several test cases I'd like to highlight though:

* combination

** both 2010 as year, 2 as month, and 29 as day were valid values, but a combination of 2010/2/29 was not.

** both 2010/2/1 and 2010/1/2 were valid dates, but a combination of 2010/2/1 as starting date and 2010/1/2 as ending date was not

* output domain

** output domain was usually missing because user story did not go that far.

** e.g. user requested a leave date, and the system saved this request into database. Here database was an output target. The system needed to insert this request as a sql statement. So here sql statement was an output domain. A special char or sql keyword in the request might cause some trouble, a sql statement in the comments of a request might even leave hacker a hole to hack into database.

Similiar cases could be:

* If output variable would be saved as a filename, then "/" might cause some trouble. And for linux, 1024 was the longest filename, another boundry to check.

* If output variable would be saved in mysql database, then "AA" and "Aa" and "aa" might cause some trouble because sometimes sql databases default configured as ignore-case.

* If output variable would be a list divided by space, then "a " might cause some trouble.etc.