Black Box Software Testing

Academic Version 2004

Copyright (c) Cem Kaner & James Bach

Student Notes for the

Study Guide

In my course of software testing, I hand students a study guide at the start of the term. The study guide includes a long list a questions -- a subset of the list you see here. All of the questions on my tests and exams come from this study guide. Some other teachers who use this list give exams based 80% (or so) on this list, but with one or two additional questions.

Because students have plenty of time to work with these questions, I can expect well-organized, well-focused, thoughtful answers.

I hand these notes to students with the study guide, to help them prepare for the exams. For additional guidance, I suggest my paper on assessment in the testing course, or these shorter discussions on answering essay questions: <Kaner>, <McClennan>, and <Indiana University>.

Notes on Studying & Answering Test Questions

• Your tests and exams will be based on these questions and a later, smaller, update to this set of questions.
• You may not use any reference materials during the test (closed book test).
• I recommend that you study with one or more partners. 3-5 people is a good sized group. 8 is too many.
• I strongly recommend that you read the questions carefully before the test, and that you attempt an answer to each one.
• AFTER you have tried your own answers, compare notes with your friends.
• Working with others will help you discover and work through ambiguities before you take the test. If a question is unclear, send me a note before the test. If you tell me early enough, I can fix it. If a question takes too long to answer, send me a note about that too.
• When you write the test, keep in mind that I read your answer with the goal of finding reasons to give you points:
  • If the question contains multiple parts, be sure to give me a separate answer for each part.
  • If a question asks about “some”, that means at least two. I normally expect three items in response to a “some.” Similarly if the question asks for a list, I am expecting a list of at least three.
  • Be aware that different words in questions have different meanings. For example, each of the following words calls for a different answer: identify, list, define, describe, explain, compare, contrast. If I ask you to list and describe some things, give me a brief identification (such as a name) of each and then a description for each one.
  • If you find it hard to define or describe something, try writing your answer around an example.
  • If you are asked to describe the relationship among things, you might find it easiest to work from a chart or a picture. You are not required to use a diagram or chart (unless I ask for one), but feel free to use one if it will help you get across your answer.
  • If I ask you to describe or define something that is primarily visual (such as a table or a graph), your answer will probably be easier to write and understand if you draw an example of what you are defining or describing.
  • If I ask you for the result of a calculation, such as the number of paths through a loop, show your calculations or explain them. Let me understand how you arrived at the answer.
  • If I ask you to analyze something according to the method described in a particular paper or by a particular person, I expect you to do it their way. If I ask you to describe their way, do so. If I ask you to apply their way, you don’t have to describe it in detail, but you have to do the things they would do in the order they would do them, and you have to use their vocabulary to describe what you are doing.
• The test is time-limited—75 minutes for a test, 2 hours for an exam. Plan to spend no more than 4 minutes on any definition, no more than 10 minutes on any short answer, and no more than 15 minutes on any long answer. Spend less on most answers. Suppose the test has 4 definitions (20 points), 2 short answers (20 points), and 3 long answers (60 points). You should plan to spend, on average, about 3 minutes per definition, about 8 minutes per short answer, and about 12 minutes per long answer (total = 64 minutes). Use the remaining 11 minutes to check your work.
• Many students find it extremely difficult to finish the test on time if they don't prepare answers in advance. If you have to spend time during the exam figuring out what the questions mean, you're probably in trouble.
• Pick the order of your answers. If you spend too long on definitions, start writing your long answers first. If you are nervous, start with the questions you find easiest to answer.
• Be aware of some factors that, in general, bias markers. These are generalizations, based on research results. I try, of course, to be unbiased, but it’s a good idea to keep these in mind with ANY grader for ANY exam:
  • Exams that are hard to read tend to get lower grades. Suggestions:
    • Write in high contrast ink (such as black, medium).
    • Write in fairly large letters.
    • Skip every second line.
    • Don’t write on the back of the page.
    • If your writing is illegible, print.
    • Start each new question on a new page.
  • Answers that are well-organized tend to be more credible. Suggestions:
    • If the question has multiple parts, start each part on a new line, and identify each part at its start.
    • In a list, start each list item on a new line—maybe bullet the list.
  • Don’t answer what has not been asked. For example, if I ask you to define one thing, don’t define that and then give me the definition of something related to it. If you do, (a) I won’t give you extra credit, (b) I’ll think that you don’t know the difference between the two things, and (c) if you make a mistake, I’ll take off points.
• If I can’t read something you wrote, I reserve the right to ignore it.
• Break a question down into component answers. For example, consider the question: "What is the difference between black box and white box testing?" Describe the advantages and disadvantages of each.” This should have five headings:
  • Difference between black and white
  • Advantages of black box
  • Disadvantages of black box
  • Advantages of white box
  • Disadvantages of white box
• Beware of simply memorizing some points off a slide. If I see these as a memorized list w/o understanding, I will ruthlessly mark you down for memorization errors. In general, if you are repeating a set of bullet points, write enough detail for them that I can tell that you understand them.
• Use a good pen. Lawyers and others who do lots of handwriting buy expensive fountain pens for a reason. The pen glides across the page, requiring minimal pressure to leave ink. If you use a fountain pen, I suggest a medium point (write large) to avoid clogging. Also try gel pens or rollerballs. Get one that you can write with easily, to avoid writer’s cramp. Basic ballpoints are very hard on you, look at how tightly you hold it and how hard you press on the page.

These notes are partially based on research that was supported by NSF Grant EIA-0113539 ITR/SY+PE: "Improving the Education of Software Testers." Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

Kaner & Bach grant permission to make digital or hard copies of this work for personal or classroom use, including use in commercial courses, provided that (a) Copies are not made or distributed outside of classroom use for profit or commercial advantage, (b) Copies bear this notice and full citation on the front page, and if you distribute the work in portions, the notice and citation must appear on the first page of each portion. Abstracting with credit is permitted.

The proper citation for this work is "Black Box Software Testing (Course Notes, Academic Version, 2004) www.testingeducation.org", (c) Each page that you use from this work must bear the notice "Copyright (c) Cem Kaner and James Bach, kaner@kaner.com", or if you modify the page, "Modified slide, originally from Cem Kaner and James Bach", and (d) If a substantial portion of a course that you teach is derived from these notes, advertisements of that course should include the statement, "Partially based on materials provided by Cem Kaner and James Bach." To copy otherwise, to republish or post on servers, or to distribute to lists requires prior specific permission and a fee. Request permission to republish from Cem Kaner, kaner@kaner.com.