Black box software testing: By Cem Kaner & James Bach
Combination testing--testing multiple variables together
- Video lectures
- Part 1 Introduction to combination testing [16:31]
- Mechanical/procedural, random, risk-based and scenario-based approaches to combination testing. Domain analysis of underlying variables. Jorgensen's characterizations of weak versus strong and normal versus robust multivariable tests.
- Part 2 Introduction to all-pairs. [10:33]
- The find dialog all pairs example.
- All singles, all pairs, all triples, all n-tuples
- Part 3 More on all pairs [15:55]
- The Netscape preferences example.
- Part 4 References and a few complications [9:48]
- Pairwise.org
- Contrast with orthogonal arrays
- Variables that have many values of interest
- Variables that interact
- Part 5 Complex interactions among variables [10:39]
- The OpenOffice Format Page dialog
- Concluding notes on mechanical approaches to combination testing
- Part 6 Exploring complex relationships [15:58]
- The TimeSlips example
- Relationships among variables
- Closing notes
- Part 1 Introduction to combination testing [16:31]
- Lecture slides
- Activity (notes) Combination test of Open Office formatting
- Examples
- Essay test Questions
- Readings and tools: See www.allpairs.org
In a combination test (multivariable test), the tester sets the values of several variables in an individual test. There are four common approaches to multivariate testing
- In mechanical or procedural testing, we create tests by applying a procedure or using a tool (that applies a procedure). Combinatorial testing, including "all pairs" are combinatorial tests.
- In random testing, we create tests by assigning randomly selected values to (what may or may not be randomly) selected variables
- In risk-driven testing, we start with a theory of error, how the program could be broken, and select values to optimize the chance of exposing that error.
- In scenario testing, we start with ideas of how people or other systems or devices will use or interact with the product and we assign values to variables as needed to enable the scenario.
We have solid, easy to apply techniques for handling combinations of variables that are (or should be) independent. Combination testing is much more challenging when one variable under test constrains another's values or changes how the program will respond to the second variable's values.
We are setting up a mailing list for announcements about this course and, perhaps, a tightly focused and moderated discussion of how to teach it or self-study with it. (This won't be a general, high-traffic, intro-to-testing discussion.) If you're interested in the course, please sign up by sending us an email. We will NOT share your email address with third parties or send commercial advertising to you.
We are publishing this course under a Creative Commons license that allows you to freely reuse and distribute the materials and to modify the slides and associated printable materials (but not the videos). We would be appreciate a few mirror sites, to reduce the growing burden on our servers. If you can help in this way, or any other way, please send a note to Cem Kaner.
