Black Box Software Testing

Fall 2006 (September 1, 2006)

Study Guide for CSE 3411

Copyright (c) Cem Kaner

Here is your study guide. All questions on my tests and exams come from this study guide.

I invite you to submit candidate questions for the study guide.

The typical midterm includes questions that total between 90 and 110 points, and the typical final exam includes questions that total between 140 and 160 points where

• short answers are worth 10 points
• long answers are worth 20 points

Notes on Studying & Answering Test Questions

Because you have plenty of time to work with these questions, I can expect well-organized, well-focused, thoughtful answers. For additional guidance, I suggest my paper on assessment in the testing course http://www.testingeducation.org/articles/assessment_in_the_software_testing_course_wtst_2003_paper.pdf, or these shorter discussions on answering essay questions:

• <Kaner> http://www.kaner.com/pdfs/EssayQuestions.pdf,
• <McClennan> http://lilt.ilstu.edu/smexpos/website/essay_writing_guide_.htm, and
• <Indiana University> http://www.indiana.edu/~wts/wts/essayexam.html.

Here are some additional suggestions:

• Your tests and exams will be based on these questions and a later, smaller, update to this set of questions.
• You may not use any reference materials during the test (closed book test).
• I recommend that you study with one or more partners. 3-5 people is a good sized group. 8 is too many.
• I strongly recommend that you read the questions carefully before the test, and that you attempt an answer to each one as part of your studying.
• Many students write their first draft answers while watching the relevant course videos.
• Write your draft answers open book. Use the course videos and slides, the required readings, and any other readings that you find useful.
• AFTER you have tried your own answers, compare notes with your friends.
• Working with others will help you discover and work through ambiguities before you take the test. If a question is unclear, send me a note before the test. If you tell me early enough, I can fix it. If a question takes too long to answer, send me a note about that too.
• When you write the test, keep in mind that I read your answer with the goal of finding reasons to give you points:
  • An answer that is "complete" but is clearly memorized from the course notes without showing insight of yours will probably earn a C-level grade.
  • Also, if you simply memorize some points off a slide and I interpret an answer as a memorized list without understanding, I will ruthlessly mark the answer down for memorization errors. In general, if you are repeating a set of bullet points, write enough detail for them that I can tell that you understand them.
  • If the question contains multiple parts, be sure to give me a separate answer for each part. Break a question down into component answers. For example, consider the question: "What is the difference between black box and white box testing?" Describe the advantages and disadvantages of each.” This should have five headings:
    • Difference between black and white
    • Advantages of black box
    • Disadvantages of black box
    • Advantages of white box
    • Disadvantages of white box
  • If a question asks about “some”, that means at least two. I normally expect three items in response to a “some.” Similarly if the question asks for a list, I am expecting a list of at least three.
  • Be aware that different words in questions have different meanings (see http://www.indiana.edu/~wts/wts/essayexam.html). For example, each of the following words calls for a different answer: identify, list, define, describe, explain, compare, contrast. If I ask you to list and describe some things, give me a brief identification (such as a name) of each and then a description for each one.
  • If you find it hard to define or describe something, try writing your answer around an example.
  • If you are asked to describe the relationship among things, you might find it easiest to work from a chart or a picture. You are not required to use a diagram or chart (unless I ask for one), but feel free to use one if it will help you get across your answer.
  • If I ask you to describe or define something that is primarily visual (such as a table or a graph), your answer will probably be easier to write and understand if you draw an example of what you are defining or describing.
  • If I ask you for the result of a calculation, such as the number of paths through a loop, show your calculations or explain them. Let me understand how you arrived at the answer.
  • If I ask you to analyze something according to the method described in a particular paper or by a particular person, I expect you to do it their way. If I ask you to describe their way, do so. If I ask you to apply their way, you don’t have to describe it in detail, but you have to do the things they would do in the order they would do them, and you have to use their vocabulary to describe what you are doing.
• The test is time-limited—75 minutes for a test, 2 hours for an exam. Plan to spend no more than 10 minutes on any short answer, and no more than 15 minutes on any long answer. Spend less on most answers. Suppose the test has 4 short answers (20 points) and 3 long answers (60 points). You should plan to spend, on average, about 8 minutes per short answer, and about 12 minutes per long answer (total = 68 minutes). Use the remaining 7 minutes to check your work.
• Many students find it extremely difficult to finish the test on time if they don't prepare answers in advance. If you have to spend time during the exam figuring out what the questions mean, you're probably in trouble.
• Pick the order of your answers. If you spend too long on definitions, start writing your long answers first. If you are nervous, start with the questions you find easiest to answer.
• Be aware of some factors that, in general, bias markers. These are generalizations, based on research results. I try, of course, to be unbiased, but it’s a good idea to keep these in mind with ANY grader for ANY exam:
  • Exams that are hard to read tend to get lower grades. Suggestions:
    • Write in high contrast ink (such as black, medium).
    • Write in fairly large letters.
    • Skip every second line.
    • Don’t write on the back of the page.
    • If your writing is illegible, print.
    • Start each new question on a new page.
    • If I can’t read something you wrote, I will ignore it.
  • Answers that are well-organized tend to be more credible. Suggestions:
    • If the question has multiple parts, start each part on a new line, and identify each part at its start.
    • In a list, start each list item on a new line—maybe bullet the list.
  • Don’t answer what has not been asked. For example, if I ask you to define one thing, don’t define that and then give me the definition of something related to it. If you do, (a) I won’t give you extra credit, (b) I’ll think that you don’t know the difference between the two things, and (c) if you make a mistake, I’ll take off points.
• Use a good pen. Lawyers and others who do lots of handwriting buy expensive fountain pens for a reason. The pen glides across the page, requiring minimal pressure to leave ink. If you use a fountain pen, I suggest a medium point (write large) to avoid clogging. Also try gel pens or rollerballs. Get one that you can write with easily, to avoid writer’s cramp. Basic ballpoints are very hard on you--look at how tightly you hold it and how hard you press on the page.

Short Answers

S.1. What is the primary difference between black box and glass box testing? What kinds of bugs are you more likely to find with black box testing? With glass box?

S.2. Suppose that your testing goal is to demonstrate nonconformance with customer expectations. If you were designing test documentation, how would you design it to support that goal? How do those design decisions support those goals?

S.3. In lecture, I asserted that all oracles are heuristic. What is the basis for that assertion? What do you think of that assertion? Why? Bonus points: describe a counter-example to this assertion.

S.4 What does it mean to specify a test by describing the precondition state of the program (and the system it runs on), the steps you take during the test and the resulting postcondition state? (Define the terms.) Would this be a complete specification of the test? Why or why not? Is t practical to do this? Why or why not?

S.5. How is it that you can achieve very high coverage from your tests but still miss lots of bugs?

S.6. Why is it usually impossible to achieve complete path coverage? Use examples to clarify your answer.

S.7. Consider a program with two loops, controlled by index variables. The first variable increments (by 1 each iteration) from -3 to 20. The second variable increments (by 2 each iteration) from 10 to 20. The program can exit from either loop normally at any value of the loop index. (Ignore the possibility of invalid values of the loop index.)

• If these were the only control structures in the program, how many paths are there through the program?
  • If the loops are nested
  • If the loops are in series, one after the other
• If you could control the values of the index variables, what test cases would you run if you were using a domain testing approach?
• Please explain your answers with enough detail that I can understand how you arrived at the numbers.
• Note: a question on the test might use different constants but be identical to this question in all other respects.

S.8. A program asks you to enter a password, and then asks you to enter it again. The program compares the two entries and either accepts the password (if they match) or rejects it (if they don’t). An entry is "valid" if it contains only letters and/or digits and is neither too short nor too long.

How many valid entries could you test? (Please show and/or explain your calculations.)

S.9. A program is structured as follows:

• It starts with a loop, the index variable can run from 0 to 20. The program can exit the loop normally at any value of the index.
• Coming out of the loop, there is a case/switch statement that will branch to one of 10 places depending on the value of X. X is a positive, non-zero integer. It has a value from 1 to 10.
• In 9 of the 10 cases, the program executes X statements and then goes into another loop. If X is even, the program can exit the loop normally at any value of its index, from 1 to X. If X is odd, the program goes through the loop 666 times and then exits. In the 10th case (I am explicitly NOT specifying which of the 10 values of X corresponds to the 10th case), the program exits.

Ignore the possibility of invalid values of the index variable or X. How many paths are there through this program? Please show and/or explain your calculations.

Note: a test question might use different constants but would be identical to this question in all other respects.

S.10. Consider the program described by Myers to illustrate calculating number of paths through the program. Change the program as follows: (a) from E, the program can go to H or I or J, not just H or I and (b) Within each iteration of the loop from A to X, the program can loop back from H to A, at most 10 times.

• Draw a chart to show the new program
• Calculate the number of paths through the program.

Note: a test question might use different constants but would be identical to this question in all other respects.

S.11. Distinguish between using code coverage to highlight what has not been tested from using code coverage to measure what has been tested. Use an example to make your contrast clearer.

S.12. Use Weinberg's definition of quality. Suppose that the software behaves in a way that you don't consider appropriate. Does it matter whether the behavior conflicts with the specification? Why? Why not?

S.13. Why are late changes to a product often more expensive than early changes?

S.14. What are some advantages and problems with using "conforms to specifications" as an indicator of quality?

S.15. One of the reasons often given for fully scripting test cases is that the tester who follows a script will know what she was doing when the program failed, and so she will be able to reproduce the bug. What do you think of this assertion? Why?

S.16. Compare, contrast, and give some examples of internal failure costs and external failure costs. What is the most important difference between these two types of failure cost?

S.17. In the Page Setup dialog in Firefox, you can choose for printing:

• Portrait or landscape orientation
• Whether or not to print background colors and images

(a) Would you do a domain analysis on these (Yes/No) variables? Why or why not?

(b) What benefit(s) (if any) would you gain from such an analysis?

S.18. Late in a project, after you have been testing for a long time, you find a serious bug that will require expensive revisions to an internal data structure. The project manager is furious because this type of change would have been much cheaper if the bug had been found much earlier. You both check, and determine that the bug was in the code before you ever started testing--you've missed it all these months. How would you decide whether missing this bug up to now was the result of inadequate testing, an erroneous testing strategy, or sensible conformance to a reasonable test strategy?

S.19. What is a quick test? Why do we use them? Give two examples of quick tests.

S.20. Why would you use scenario testing instead of domain testing? Why would you use domain testing instead of scenario testing?

S.21. Compare and contrast scenario testing and specification-based testing.

S.22. Advocates of GUI-level regression test automation often recommend creating a large set of function tests. What are they actually advocating and why? What are some benefits and risks of this?

S.23. What is a function list and how would you build one for Firefox?

S.24. List and describe four different dimensions (different "goodnesses") of "goodness of tests."

S.25 What is the power of a test? Credibility of a test? Contrast them with an example of a good test that has high power/low credibility and another that has low power/high credibility.

S.26. What is opportunity cost and why is it such an important issue in testing?

S.27. What is strong combination testing? What is the primary strength of this type of testing? What are two of the main problems with doing this type of testing? What would you do to improve it?

S.28. What is weak combination testing? What is the primary strength of this type of testing? What are two of the main problems with doing this type of testing? What would you do to improve it?

S.29. What kinds of errors are you likely to miss with specification-based testing?

S.30. Describe two benefits and two risks associated with using test matrices to drive your more repetitive tests.

S.31. What risks are we trying to mitigate with black box regression testing?

S.32. What risks are we trying to mitigate with unit-level regression testing?

S.33. What are the differences between risk-oriented and procedural regression testing?

S.34. Describe three risks of exploratory testing.

S.35. What is a configuration test matrix? Draw one and explain its elements.

S.36. What do you think is a reasonable ratio of time spent documenting tests to time spent executing tests? Why?

S.37. What factors drive up the cost of maintenance of test documentation?

S.38. Does detailed test documentation discourage exploratory testing? If so, how? Why?

S.39. How can test documentation support delegation of work to new testers? What would help experienced testers who are new to the project? What would help novice testers?

S.40. How could a test suite support prevention of defects?

S.41. What does it mean to do maintenance on test documentation? What types of things are needed and why?

S.42. Distinguish between a test script and a task checklist.

S.43. What do we mean by "diverse half-measures"? Give some examples.

S.44. What are the reporters' questions? Why do we call them context-free?

S.45. What is a failure mode and effects analysis and how would you do/use one to support exploratory testing?

S.46. Distinguish between project-level and product-level risks.

S.47. What is a model? Why are models important in measurement?

S.48. What is "active reading" and why is it important for specification analysis?

S.49. How can it be that you don't increase coverage when using extended random regression testing but you still find bugs? Apply your explanation to an example.

S.50. Why do we argue that even capture-replay automation is software engineering?

Long Answer

L.1. State, describe, compare and contrast three different definitions of software testing. Which do you prefer? Why?

L.2. State, describe, compare and contrast three different definitions of software error. Which do you prefer? Why?

L.3. Suppose that a test group's mission is to achieve its primary information objective. Consider (and list) three different objectives. For each one, how would you focus your testing? How would your testing differ from objective to objective?

L.4. While testing a browser, you find a formatting bug. The browser renders single paragraph blockquotes correctly—it indents them and uses the correct typeface. However, if you include two paragraphs inside the <blockquote>…</blockquote> commands, it leaves both of them formatted as normal paragraphs. You have to mark each paragraph individually as blockquote.

Consider the consistency heuristics that we discussed in class. Which three of these look the most promising for building an argument that this is a defect that should be fixed?

For each of the three that you choose:

• Name the heuristic
• Describe research that this heuristic suggests to you
• Describe an argument that you can make on the basis of that research

L.5. The oracle problem is the problem of finding a method that lets you determine whether a program passed or failed a test.

Suppose that you were doing automated testing of page layout (how the document will look like when printed) of an HTML editor. Describe three different oracles that you could use or create to determine whether layout-related features were working. For each of these oracles,

• identify a bug that would be easy to detect using the oracle. Why would this bug be easy to detect with this oracle? and
• identify another bug that your oracle would be more likely to miss. Why would this bug be harder to detect with this oracle?

L.6. Consider testing a browser. Describe 5 types of coverage that you could measure, and explain a benefit and a potential problem with each. Which one(s) would you actually use and why?

L.7. Some theorists model the defect arrival rate using a Weibull probability distribution. Suppose that a company measures its project progress using such a curve. Describe and explain the impact of two of the pressures testers are likely to face early in the testing of the product and two of the pressures they are likely to face near the end of the project.

L.8. Explain two advantages and three disadvantages of using bug counts to evaluate testers' work.

L.9. Vendor sells custom software with a development contract that promises the applications will be "completely tested." In fact, the testing done by Vendor includes complete statement and branch coverage. Vendor delivers a product to Customer, the product corrupts its data without indicating any problem, and Customer loses $2 million. Customer sues, arguing that no product that has a serious bug (you should assume in your answer that this is, in fact, a serious bug) could have been completely tested.

Vendor responds with two arguments: First, that complete statement and branch coverage is complete testing. Second, that no reasonable customer could believe that a software contract would promise such extensive testing that they could guarantee bug-free software.

Your tasks:

• Describe Vendor's arguments in the most convincing (but accurate and honest) way that you can
• Evaluate the arguments.

L.10. SoftCo publishes software. Their president hates Easter Eggs and has instructed the test group to find every one (if there are any) in the product it is testing. As lead tester, it is your to figure out how to test for Easter Eggs and when to declare the job done. How will you decide when you have finished this task? Present your ideas, their strengths and weaknesses.

L.11. Suppose that you found a reproducible failure, reported it, and the bug was deferred. Other than further testing, what types of evidence could you use to support an argument that this bug should be fixed, and where would you look for each of those types of evidence?

L.12. What is the strongest ethical argument in favor of doing a cost-of-quality analysis that evaluates only cost to the vendor and does not estimate cost to the customer? What is the strongest ethical criticism? Evaluate both.

L. 13. Imagine testing a date field. The field is of the form MM/DD/YYYY (two digit month, two digit day, 4 digit year). Do an equivalence class analysis and identify the boundary tests that you would run in order to test the field. (Don’t bother with non-numeric values for these fields.)

L.14. Ostrand & Balcer described the category-partition method for designing tests. Their first three steps are:

1. Analyze
2. Partition, and
3. Determine constraints

Apply their method to this function:

I, J, and K are unsigned integers. The program calculates K = I *J. For this question, consider only cases in which you enter integer values into I and J.

Do an equivalence class analysis on the variable K from the point of view of the effects of I and J (jointly) on K. Identify the boundary tests that you would run (the values you would enter into I and J) in your tests of K.

Note: In the exam, I might use K = I / J or K = I + J or
K = IntegerPartOf (SquareRoot (I*J))

L.15. In EndNote, you can create a database of bibliographic references, which is very useful for writing essays. Here are some notes from the manual:

• Each EndNote reference stores the information required to cite it in a bibliography. Keywords, notes, abstracts, URLs and other information can be stored in a reference as well.
• Each reference added to a library is automatically assigned a unique record number that never changes for that reference in that particular library. . . .
• Each library can reach a record limit of 32,767 or a size of 32MB (whichever comes first). Once a record number is assigned, it cannot be used again in that library. So, if you import 30,000 records, then delete all but 1000 of them, you cannot enter more than another 2767 records into that particular library.

List the variables of interest and do a domain analysis on them.

L.16. Imagine testing a file name field. For example, go to a File Open dialog, you can enter something into the field.

Do a domain testing analysis: List a risk, equivalence classes appropriate to that risk, and best representatives of the equivalence classes.

For each test case (use a best representative), briefly explain why this is a best representative. Keep doing this until you have listed 10 best-representative test cases.

L.17. The course notes describe a test technique as a recipe for performing the following tasks:

• Analyze the situation
• Model the test space
• Select what to cover
• Determine test oracles
• Configure the test system
• Operate the test system
• Observe the test system
• Evaluate the test results

How does scenario testing guide us in performing each of these tasks?

L.18. The course notes describe a test technique as a recipe for performing the following tasks:

• Analyze the situation
• Model the test space
• Select what to cover
• Determine test oracles
• Configure the test system
• Operate the test system
• Observe the test system
• Evaluate the test results

Compare and contrast how scenario testing and risk-based testing guide us in performing each of these tasks.

L.19. In Firefox, you can download Extensions that add new capabilities or change the browser's existing capabilities.

• How would you develop a list of risks for this capability? (If you are talking to people, who would you ask and what would you ask them?) (If you are consulting books or records or databases, what are you consulting and what information are you looking for in it?)
• Why is this a good approach for building a list of risks?
• List 7 risks associated with this function.
• For each risk, briefly (very briefly) describe a test that could determine whether there was an actual defect.

L.20. You are testing the group of functions that let you organize bookmarks in Firefox.

List 5 ways that these functions could fail. For each potential type of failure, describe a good test for it, and explain why that is a good test for that type of failure. (NOTE: When you explain why a test is a good test, make reference to some attribute(s) of good tests, and explain why you think it has those attributes. For example, if you think the test is powerful, say so. But don't stop there, explain what about the test justifies your assertion that the test is powerful.)

L.21. You are testing the group of functions that let you manage tabs in Firefox.

Think in terms of compatibility with external software. What compatibility features or issues are (or could be) associated with tabs? List three types. For each type, list 2 types of failures that could involve compatibility. For each type of failure, describe a good test for it and explain why that is a good test for that type of failure. (There are 6 failures, and 6 tests, in total). (NOTE: When you explain why a test is a good test, make reference to some attribute(s) of good tests, and explain why you think it has those attributes. For example, if you think the test is powerful, say so. But don't stop there, explain what about the test justifies your assertion that the test is powerful.)

L.22. You are testing the group of functions that let you manage tabs in Firefox.

Suppose that a critical requirement for this release is scalability of the product. What scalability issues might be present with tabs? List three. For each issue, list 2 types of failures that could involve scalability. For each type of failure, describe a good test for it and explain why that is a good test for that type of failure. (There are 6 failures, and 6 tests, in total). (NOTE: When you explain why a test is a good test, make reference to some attribute(s) of good tests, and explain why you think it has those attributes. For example, if you think the test is powerful, say so. But don't stop there, explain what about the test justifies your assertion that the test is powerful.)

L.23. Define a scenario test and describe the characteristics of a good scenario test.

Imagine developing a set of scenario tests around tabs in Firefox.

• What research would you do in order to develop a series of scenario tests?
• Describe two scenario tests that you would use and explain how these would relate to your research and why each is a good test.

L.24. Imagine that you were testing how Firefox manages bookmarks.

• Explain how you would develop a set of scenario tests that focus on this feature.
• Describe a scenario test that you would use while testing this feature.
• Explain why this is a particularly good scenario test.

L.25. Suppose that scenario testing is your primary approach to testing. What controls would you put into place to ensure good coverage? Describe at least three and explain why each is useful.

L.26. We are going to do some configuration testing on the Windows version of Firefox. We want to test it on

• Windows 98, 2000, XP home, and XP Pro (the latest service pack level of each)
• Printing to an HP inkjet, a LexMark inkjet, an HP laser printer and a Xerox laser printer
• Connected to the web with a dial-up modem (28k), a DSL modem, a cable modem, and a wireless card (802.11b)
• With a 640x480 display, an 800 x 600 display, a 1024x768 display and a an 1152 x 720 display
  • How many combinations are there of these variables?
  • Explain what an all-pairs combinations table is
  • Create an all-pairs combinations table. (Show at least some of your work.)
  • Explain why you think this table is correct.
  Note: In the exam, I might change the number of operating systems, printers, modem types, or display

L.27. Compare and contrast all-pairs testing and scenario testing. Why would you use one over the other?

L.28. You are testing the group of functions that let you manage passwords in Firefox. Think about the different types of users of browsers. Why would they want to manage passwords? Describe up to four different types of users, and up to two ways that each one might choose to use Firefox to manage their passwords. (In total, describe 4 different ways for people to manage passwords with Firefox, not including "don't use it"). Describe a scenario test for one of these and explain why it is a good scenario test.

L.29. Consider domain testing and specification-based testing. What kinds of bugs are you more likely to find with domain testing than with specification-based testing? What kinds of bugs are you more likely to find with specification-based testing than with domain testing?

L.30. Consider scenario testing and function testing. What kinds of bugs are you more likely to find with scenario testing than with function testing? What kinds of bugs are you more likely to find with function testing than with scenario testing?

L.31. Compare and contrast the mechanical, risk-based, and scenario-based approaches to developing combination tests. Discuss their strengths and weaknesses relative to each other. Use examples to clarify your points.

L.32. Describe a traceability matrix.

• How would you build a traceability matrix for the Bookmarks feature in Firefox?
• What specification(s) are you tracing to and where would you get them?
• What is the traceability matrix used for?
• What are the advantages and risks associated with driving your testing using a traceability matrix?
• Give examples of advantages and risks that you would expect to deal with if you used a traceability matrix for Bookmarks. Answer this in terms of two of the Bookmarks features that you can use in Firefox. You can choose which two features.

L.33. What is regression testing? What are some benefits and some risks associated with regression testing? Under what circumstances would you use regression tests?

L.34. Compare exploratory and scripted testing. What advantages (name three) does exploration have over creating and following scripts? What advantages (name three) does creating and following scripts have over exploration?

L.35. Your company decides to outsource test execution. Your senior engineers will write detailed test scripts and the outside test lab's staff will follow the instructions. How well do you expect this to work? Why?

L.36. Imagine that you are an external test lab, and Mozilla.org comes to you with Firefox. They are considering paying for some testing, but before making a commitment, they want to understand your approach and get a sense of what they'll get and how much it will cost. So, they ask you what test documentation they (or you) should create. You ask what they want, and they say that they want to rely on your expertise.

How will you decide what test documentation to give them?

To decide what to give them, what questions would you ask (up to 7 questions) and for each answer, how would the answer to that question guide you?

L.37. Suppose that your company decides to write test scripts in order to foster repeatability of the test across testers. Is repeatability worth investing in? Why or why not?

L.38. Consider a browser and its ability to display pages that contain an embedded video (and call an appropriate player to play the video).

• List two stakeholders for whom this type of feature would be important. (Identify them by function, not by personal name. You don't have to know the exact name or job title of the actual people. This is a hypothetical example.)
• For each stakeholder, list and briefly describe three interests that you think this stakeholder probably has. Are these favored or disfavored interests?
• For each interest (there are a total of six), how does this feature relate to that interest?

L.39. Suppose that Boeing developed a type of fighter jet and a simulator to train pilots to fly it. Suppose that Electronic Arts is developing a simulator game that lets players "fly" this jet. Compare and contrast the test documentation requirements you would consider appropriate for developers of the two different simulators.

L.40. Why is it important to design maintainability into automated regression tests? Describe some design (of the test code) choices that will usually make automated regression tests more maintainable.

L.41. Contrast developing a GUI-level regression strategy for a computer game that will ship in one release (there won't be a 2.0 version) versus an in-house financial application that is expected to be enhanced many times over a ten-year period.

L.42. We considered all-pairs as an approach for strictly independent variables. Why is this restriction important? Suppose you were testing four variables (A, B, C, D) that had 3 values each (A1, A2, A3 and similarly for B, C and D) but A1 was always an invalid combination with B2. How would you modify your all-pairs test set to handle this one dependence? Now, suppose that A1 is ALSO invalid with C3 or D4 and A2 is always invalid with B3, C2 or D4, and C1 is invalid with D2. How would you design your combination tests in this case?

L. 43. You are using a high-volume random testing strategy for the Mozilla Firefox program. You will evaluate results by using an oracle.

• Consider entering a field into a form, using the entry to look up a record in a database, and then displaying the record’s data in the form. How would you create an oracle (or group of oracles)? What would the oracle(s) do?
• Now consider displaying a table whose column widths depend on the width of the column headers. Test this across different fonts (vary typeface and size). How would you create an oracle (or group of oracles) for this? What would the oracle(s) do?
• Which oracle would be more challenging to create or use, and why?

L. 44. Suppose that you were managing a test group whose primary approach to testing was exploratory. How would you check the coverage of your group's work, how would you improve coverage, and how would you report testing status? Will this be sufficient? Why or why not?

L.45. Schroeder and Bach argue that there is little practical difference between all-pairs coverage and an equivalent random sample of combination tests. What do you think? From the list of attributes of good tests, list three that might distinguish between these approaches and explain why.

L. 46. Think about the personnel / staffing issues associated with high-volume test automation. Characterize three of the high-volume techniques in terms of the skills required of the staff. If you were managing a typical testing group, which has few programmers, which technique would you start with and why?

L. 47. Doug Hoffman's description of the square root bug in the MASPAR computer provides a classic example of function equivalence testing. What did he do in this testing, why did he do it, and what strengths and challenges does it highlight about function equivalence testing?

L. 48. Suppose that you had access to the Mozilla Firefox source code and the time / opportunity to revise it. Suppose that you decided to use a diagnostics-based high volume automated test strategy to test Firefox’s treatment of links to different types of files.

• What diagnostics would you add to the code, and why?
• Describe 3 potential defects, defects that you could reasonably imagine would be in the software that handles downloading or display of linked files, that would be easier to find using a diagnostics-based strategy than by using a lower-volume strategy such as exploratory testing, spec-based testing, or domain testing.

L. 49. Compare and contrast the mechanical, risk-based, and scenario-based approaches to developing combination tests. Discuss their strengths and weaknesses relative to each other. Use examples to clarify your points.

L. 50. What's a testing project plan? Describe some of the elements of a testing project plan. What are some of the costs and benefits of creating such a document?

Copyright (c) Cem Kaner 2004-2006

This work is licensed under the Creative Commons Attribution-ShareAlike License. To view a copy of this license, visit http://creativecommons.org/licenses/by-sa/2.0/ or send a letter to Creative Commons, 559 Nathan Abbott Way, Stanford, California 94305, USA.

These notes are partially based on research that was supported by NSF Grant EIA-0113539 ITR/SY+PE: "Improving the Education of Software Testers." Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.