Black Box Software Testing

Academic Version 2004

Copyright (c) Cem Kaner & James Bach

Faculty Notes for the

Study Guide

In my course of software testing, I hand students a study guide at the start of the term. The study guide includes a long list a questions. All questions on my tests come from this list. Some other teachers add one or two surprise questions.

Because students have plenty of time to work with these questions, I can expect well-organized, well-focused, thoughtful answers. Along with the list of questions, I hand students suggestions for answering questions (see also <Kaner>, <McClennan>, and <Indiana University>). Some grading suggestions appear in my paper on assessment in the testing course.

Notes on the Study Guide Questions

• The questions in the study guide often address browser or word processor issues. This is arbitrary. It reflects the applications we've tested recently in class. If you're class is testing some other type of application, I suggest that you modify these questions so that most of them reflect that type.
• Many of the questions in this list go beyond the detail that is available in the slides. In the course that I teach, students get several supplementary readings, and I provide additional information in lecture.
• Be thoughtful about the size of the list of study questions that you give students. If the list is too long, students will not try to answer the questions before the exam; they'll just read the answers. The quality of their performance on the final will suffer substantially. I currently use a set of about 50 definitions, 25 short answers, and 25 long answers in a 1-semester course that has a large population of students who write English as a second language. Your students might find a different pool more appropriate.
• You'll notice that several of the questions are related. This is intentional. Here's an example:
  • 5.L.4. You are testing the group of functions that let you create and format a table in a word processor (your choice of MS Word or Open Office). List 5 ways that these functions could fail. For each potential type of failure, describe a good test for it, and explain why that is a good test for that type of failure.
  • 5.L.5. You are testing the group of functions that let you create and format a table in a word processor (your choice of MS Word or Open Office).Think in terms of persistent data. What persistent data is (or could be) associated with tables? List three types. For each type, list 2 types of failures that could involve that data. For each type of failure, describe a good test for it and explain why that is a good test for that type of failure. (There are 6 failures, and 6 tests, in total).
  • 5.L.6. You are testing the group of functions that let you create and format a table in a word processor (your choice of MS Word or Open Office). Think in terms of data that you enter into the table . What data is (or could be) associated with tables? List five types of failures that could involve that data. For each type of failure, describe a good test for it and explain why that is a good test for that type of failure..
  • 5.L.7. You are testing the group of functions that let you create and format a table in a word processor (your choice of MS Word or Open Office). Think in terms of user interface controls . What user interface controls are (or could be) associated with tables? List three types. For each type, list 2 types of failures that could involve that data. For each type of failure, describe a good test for it and explain why that is a good test for that type of failure. (There are 6 failures, and 6 tests, in total).
• 5.L.8. You are testing the group of functions that let you create and format a table in a word processor (your choice of MS Word or Open Office). Think in terms of compatibility with other software. What compatibility features or issues are (or could be) associated with tables? List three types. For each type, list 2 types of failures that could involve compatibility. For each type of failure, describe a good test for it and explain why that is a good test for that type of failure. (There are 6 failures, and 6 tests, in total).
• 5.L.9. You are testing the group of functions that let you create and format a table in a word processor (your choice of MS Word or Open Office). Suppose that a critical requirement for this release is scalability of the product. What scalability issues might be present in the table? List three. For each issue, list 2 types of failures that could involve scalability. For each type of failure, describe a good test for it and explain why that is a good test for that type of failure. (There are 6 failures, and 6 tests, in total).

These are all about testing the word processor's table functions, but the sequence tells a lesson.

• The first one simply identifies the function and asks the student to imagine potential failures. When students work only on this question, their answers are often superficial and defocused.
• The next three are taken from the Product Elements list of Bach's Heuristic Test Strategy Model.
• The last two are from the Quality Criteria list of Bach's model.
• The six together give the student six different angles on the same problem. The student will come up with different tests, after thinking about different risks. Collectively, they make a point--the more angles that you come at the product from, the more types of failures you can imagine (and the more bugs you can find). These questions thus provide an experience basis for talking about the Strategy model. In Bach's and my experience, students only come to understand the value of this model after a lot of practice.
• You might ask a full sequence of questions like this on an assignment, but you would not (well, I would not) ask more than one or two of these questions on an exam. The questions have more value than use on an exam--they guide study, and so you can lead students to insights with a sequence of questions, followed up by comments in lecture after the students have had occasion (e.g. the test) to study.

• It helps to actively encourage students to work together. I host two study sessions before each test or exam. I hope a typical session on a weekend day at the local cafe, and pay for breakfast for anyone who arrives before 1030 a.m. I don't answer the questions for the students. I sit in the cafe, but several tables away, doing my own work. I do help them get back on track when a few of them disagree with each other and can make no progress. (The main benefit of my presence is that it makes clear to the students that I think this is important, and they should come.)
• Grade the first midterm aggressively. You might fail a lot of students. (Things change gradually, as your course develops a reputation.) Don't worry about it. They can rise to reasonable standards. Give comments on the answers, and conduct a review class (or long after-hours review session) in which you walk through questions, point out common mistakes, and give examples of better answers. If you use a detailed outline or checklist for grading, share it with them. And hand out some sample answers--I use good answers written by the students, rather than my own. The samples are imperfect, but they show what can actually be done in the exam situation.
• I often host a makeup midterm, in a 1-semester course.
  • This is a voluntary makeup. Students who did well on the first midterm won't take it. I schedule the exam outside of class hours, and hold reviews (before and after) out of class hours.
  • The makeup replaces the original midterm. If the student's grade goes up, good. If the grade goes down, it goes down. This limits the number of exams to grade. You don't get frivolous exams submitted just in case they're OK (which you do get if you use the best of both exams) and you don't get the compulsive A student hoping to raise her grade by 2% . Keeping the number of submissions down is important--this course involves a lot of grading. You need to protect your time.
  • I do not always give a makeup midterm. I don't promise it in the syllabus and refuse to talk with students about it before they have finished the midterm (except to warn students who ask that there might not be one). This is important--if students know that there is a makeup midterm, some will prepare inadequately for the first midterm with the idea that they can do well on the makeup.
• These exams are just part of the assessment strategy for the Testing course. They give students a focus for interacting with the lecture and reading materials, but students also need to develop experience with the testing techniques, troubleshooting, and bug reporting. For those, you have to give students assignments and homework, not exams.

These notes are partially based on research that was supported by NSF Grant EIA-0113539 ITR/SY+PE: "Improving the Education of Software Testers." Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

Kaner & Bach grant permission to make digital or hard copies of this work for personal or classroom use, including use in commercial courses, provided that (a) Copies are not made or distributed outside of classroom use for profit or commercial advantage, (b) Copies bear this notice and full citation on the front page, and if you distribute the work in portions, the notice and citation must appear on the first page of each portion. Abstracting with credit is permitted.

The proper citation for this work is "Black Box Software Testing (Course Notes, Academic Version, 2004) www.testingeducation.org", (c) Each page that you use from this work must bear the notice "Copyright (c) Cem Kaner and James Bach, kaner@kaner.com", or if you modify the page, "Modified slide, originally from Cem Kaner and James Bach", and (d) If a substantial portion of a course that you teach is derived from these notes, advertisements of that course should include the statement, "Partially based on materials provided by Cem Kaner and James Bach." To copy otherwise, to republish or post on servers, or to distribute to lists requires prior specific permission and a fee. Request permission to republish from Cem Kaner, kaner@kaner.com.