Software testing is arguably the least well understood phase of the software development lifecycle [1]. This makes the important task teaching software testing a challenging exercise. Many open questions remain on exactly what and how to teach software testing.

In our work, we teach testing using the learn by doing approach. This means we directly involve students in writing and testing software at the unit level, and directly involve students in system-level testing of ongoing software development projects. To some extent, we operate this way because it is our bias. We learned to test software by doing it and so we teach it that way. However, learn by doing is also often touted as an effective teaching technique by researchers. The assumption is that "students learn better in situations which more closely approximate the situations in which they will use their knowledge" [2]. These situations allow students to learn from their mistakes, as well as, develop their own individualized constructions of the knowledge obtained. Clearly, the learn by doing approach is best suited to students that prefer an active learning style and may not be effective for others. Our goal in presenting this material in not to sell the learn by doing approach, but rather to gain insights into teaching software testing that can be applied regardless of the teaching technique used.

Our software testing course, CS790 Software Testing and Quality Assurance, was taught in the Fall of 2003 at the University of Wisconsin - Milwaukee (UWM). The course is a graduate-level course and has a prerequisite of CS536 Software Engineering. UWM now also offers CS657 Introduction to Software Testing. At some point, the introductory course will be the prerequisite for graduate-level testing course, but it is not at the current moment.

Our course is intended to be a "Software Test Engineering" Course. The focus of the course is on applying testing strategies, tools and techniques to solve complex testing problems. In the Fall of 2003, the students in the course had varying backgrounds. Some student filled the prerequisite by taking a software engineering course at other universities and may have had little exposure to software testing concepts. On the other hand, 4 of 22 students in the course had industrial software testing experience. While not intended to be a course for students with no exposure to testing concepts, the course did cover terminology and a many introductory topics. The level of coverage in many areas was insufficient, especially for students with little experience with software systems. An outline of the topics covered in the course can be found in Appendix A.

In this paper, we will discuss the two learn by doing testing projects that were assigned during the course. The assignments were designed to fit the "Developer Tester vs. Independent Tester" model [3]. As a software engineer, one may be called upon to develop/write/create software. In these situations, an engineer must take on the role of "developer tester." In this role, the engineer contributes to the quality of their own software by thoroughly testing it at the unit-level and/or integration-level. The first project given to the students was a unit testing project that incorporated the Test -Driven Development (TDD) process [4].

As a software engineer, one may also be called upon to test software developed by others. In these situations, an engineer must take on the role of "independent tester," that is, independent (organizationally or otherwise) of the process that created the software. In this role, the engineer contributes to the quality of the delivered product by testing it thoroughly at the upper levels of testing (e.g., feature, system, or acceptance level of testing). The second project given to the students was a system testing exercise in which they were assigned the role of independent tester for an ongoing academic software development project.

In section 2 of the paper, we discuss the unit testing exercise using TDD. In section 3 of the paper, we discuss the system testing exercise. Section 4 presents our conclusion and future work.

2 Unit Testing Exercise

There exists a long history and strong theoretical basis for unit testing. However, in many instances, theory has contributed little to the practice of unit testing. In our experience, and in the experience of others [5], regardless of what the software development process dictates, many developers test their own code by "poking around," meaning that they try a few things and then release the code to the build. Craig and Jaskiel state that few companies train developers to do testing, and few companies invest in code coverage tools essential in the theoretical approach to unit testing [6].

With the relatively recent development of Agile Methods [7, 8], another approach to testing at the unit-level, or class-level was created. TDD is a software development process with and integral testing strategy. Under TDD, test cases are written before code is produced and all test cases must be automated. A tool referred to as a unit testing framework is used to automate the test cases [9].

While it is still too early to know the impact of TDD on software development and testing, many reports indicate that software developers will use the process (traditional unit test strategies, for various reasons, are not used). The testing community has pointed out that TDD is not "real" testing; however, as often is the case, the best tool is the tool that gets used.

TDD was incorporated into the course largely because we think it has merit. While it is likely that the TDD process will evolve over time, automating unit test cases using a unit testing framework seems appealing to developers and appears to be a significant "next step" in improving the practice of unit testing. Including TDD in the course appears to be a good way to expose students to a popular current practice, as well as, highlight the difference between TDD and traditional unit testing.

After covering material on TDD and unit testing frameworks, the students were given an assignment to implement an object-oriented (OO) class in C++. The OO class assigned is an important part of the software system to be used later in the semester in the system testing exercise. The teaching objectives for the unit testing exercise are listed in table 1. A description of the unit testing assignment can be found in Appendix B.

Table 1. Teaching Objective for the Unit Testing Assignment

#	Teaching Objectives
1	Students shall be able to develop an object-oriented class to specification using the TDD process.
2	Students shall be able use a unit testing framework (CppUnit) to create and execute automated test cases required by the TDD process.
3	Students shall be able to describe white-box and black-box unit testing techniques.
4	Students shall be able to explain the difference between "write all tests first", TDD, and "test later" unit testing approaches.
5	Students shall be able to explain why TDD is not "testing" in the traditional sense of the word.
6	Students shall be able to explain the process of "structured unit testing" [10].

Immediately after completing the exercise, the students were given a survey on their experiences with TDD. Some of the results of this survey can be found in Appendix E. In general, most students believed that compared to their "usual" development process, TDD produced higher quality code, but that it took longer to complete. It is also interesting to note that 58% of the student did not believe their code to be adequately tested using the TDD process.

As part of the assessment of the student's unit testing assignment, we measured the statement coverage of the OO class when executing their own test suites. We also developed our own test suite of 30 test cases based on the functional requirements of the OO class and executed it on each of the student's OO classes. The results are listed in Table 2. The statement coverage achieved by most students is in the correct range for industrial software. One might not expect to exceed 85% statement coverage for a variety of reasons, including internal error conditions that are difficult to produce. The coverage tools used were not capable of producing other coverage measures (e.g., branch, decision, path, dataflow). The more interesting number is the test pass rate on the functional test suite. At an average of 70%, this is well below what we would have expected to find, given that the code had already been tested under TDD. Contributing factors to this low pass rate are the complexity of the functional requirements of the OO class, the inexperience of some graduate students with OO programming, and the inexperience of the students in using TDD. Clearly, more study is needed to quantify the difference in fault detection capability of TDD compared to that of traditional unit testing.

Table 2. Unit Testing Assignment Statistics

	Statement coverage achieved by student's test suite	Pass rate on professor's test suite
Minimum	75.40%	43.33%
Average	84.47%	70.63%
Max	94.40%	90.00%

Following the unit testing assignment, information on control flow graphs and traditional unit testing adequacy criteria were presented in lecture. We feel that this ordering and approach to presenting the unit testing material was effective in emphasizing the differences between TDD and traditional unit testing.

3 System Testing Exercise

3.1. Description

In the system testing exercise, teams of students from the CS790 Software Testing and Quality Assurance are paired with teams of students from the CS536 Software Engineering (SE) course.

The students in the software engineering course are assigned to an SE team. The SE team is 3-5 students working on a semester long software project to design and implement a relatively complex industrial software application. All SE teams implement the same software project.

The students in the software testing course are assigned to an Alpha team, so named because they perform "alpha" testing. Alpha teams consist of 3-4 students that perform "friendly," functional, system-level testing in the SE development environment.

Each SE team is then paired with an Alpha team. The SE teams use an incremental development process model. They deliver 3-4 software increments to the Alpha teams for testing. In the Fall of 2003 there were 7 pairs of teams. The schedule of software deliveries can be found in Appendix D. After the final increment, we conducted a final customer acceptance test on each project to evaluate the effectiveness of the SE and Alpha teams.

3.2. History

A year earlier, in the Fall of 2002, a system testing exercise using pairs of SE and Alpha teams was conducted at UWM. The exercise was the topic of a presentation at WTST 3. The results of the system test exercise in Fall of 2002 were not encouraging. After spending their time in the software testing course the students did a substandard job of testing the software engineering projects. The final CA test of the SE projects easily found numerous, serious software defects. This result was puzzling, and somewhat embarrassing at the same time. During the presentation of results at WTST 3, many suggestions were made on how to improve the course and improve the education of software testers. Many of these suggestions were implemented in the system test exercise in the Fall 2003. The major changes included:

Reducing the complexity of the software application being developed. In the previous system testing assignment, the application was extremely complex, resulting in largely unstable code from almost all SE teams. Unstable code is very difficult to test and contributed to the poor performance of the Alpha teams.
Improved mentoring/management. To improve performance, the SE and Alpha teams must be more closely managed and mentored. Teams may make bad decisions early in the project and then have a difficult time recovering within the time constraints of the semester. To improve this situation, status reports were more closely monitored and feedback was given when necessary. Additionally, fewer lectures were given allowing more open class sessions spent answering questions and working examples.
Reducing expectations. How well students can test software, even after an advanced software testing course, remains an open question. Some students simply do not have the temperament or desire to perform testing activities. Some students simply do not have time outside of class to contribute to the heavy work load the system testing exercise imposes.
Applying the system testing exercise in an advanced testing course. The software engineering curriculum at UWM is in a state of transition. The goal is to establish an introductory testing course at the undergraduate level and an advanced testing course at the graduate level. The system testing exercise will be used at the graduate level. When students have the correct prerequisites for the advanced course, the course contents will also be upgraded to an advanced level, and less introductory material on testing will be presented.

3.3. Teaching Objectives

The teaching objectives for the system testing exercise are found in table 3.

Table 3. Teaching Objectives for the System Testing Exercise

#	Teaching Objectives
1	Students shall be able to create and document a light-weight test plan (based on IEEE 829) from a relatively complex set of software specifications.
2	Students shall be able to use test objectives and inventories in the process of creating functional test cases from software specifications [6]
3	Students shall be able to conduct risk analysis to set testing priorities and constrain the number of test cases created.
4	Students shall be able to create an inventory tracking matrix to map functional test cases to test objectives and inventories.
5	Students shall be able to document functional test cases using a light-weight spreadsheet technique.
6	Student shall be able to discuss the context of the testing project [11].
7	Student shall be able to list the testing team's mission [11].
8	Students shall be able to conduct exploratory, scripted and regression testing.
9	Students shall be able to create clear, accurate, and effective bug reports using a defect tracking tool.
10	Students shall be able to report the results of testing sessions using test logs.
11	Students shall be able to demonstrate the use of test automation using scripting tools (Perl, Ruby) at the UI and API level.
12	Students shall be able to interact with the teammates and software developers in a productive and professional manner.

3.4. Software Application

The software application selected for the system testing exercise was the Data Management and Analysis System (DMAS). DMAS is used to manage workflow, data and perform analysis in the analytical chemistry laboratories. This application was originally written in FORTRAN in the late 1980s when one of the authors was a programmer in the pharmaceutical industry. The goals of the DMAS project are to:

1. Implement a digital work list feature that allows the Lab Managers to assign tests to Lab Technicians, monitor status of tests, and review results of tests.

2. Give Lab Technicians the capability to automatically produce a Product Sample Concentration Report (PSCR) from the data collected during product testing using High Pressure Liquid Chromatography (HPLC).

3. Allow the Lab Manager to conduct statistical process control analysis using all, or a portion of, the historical data collected for any product under test.

4. Provide data management to support the previous goals.

The DMAS requirements specification consists of 23 functional requirements, 3 non-functional requirements, and 14 use cases.

3.5. Project Execution

The overall mission given to the Alpha test teams was to improve the quality of the DMAS product through defect detection. Other important aspects of the mission included: