Testing

Introduction

Testing is about making sure the software matches the requirements. There are a variety of testing methods but all of them revolve around having a set of "Test Scripts" that are executed on the software. There is also a "Defect Tracking System" that makes sure each defect is dealt with appropriately.

Software is challenging to test because it is impossible to test all possible combinations of situations even for the simplest programs. This is especially true of GIS data where we may not have control of the data running with our tools.

Test Scripts

Test scripts can be on paper or in MS-Word files, or in a complex database. The key is to capture the critical features that a user will be executing and then test them regularly. Test scripts fall into the following categories:

• Basic User Tests - the most common and critical steps most users will take
• Sparse Matrix Tests - testing the full capabilities of the software but not with every possible combination of features and data
• Robustness Test - typically automated tests to check if the software will work after thousands of uses
• Random Tests - randomly looking for problems with the software

Remember that we not only need to test the features of the software but also test it with different operating systems, versions of required libraries, and different datasets.

Defect Tracking System

A defect tracking system keeps track of the defects that have been found in the software and what their status is. A defect is fundamentally something that either does not match the requirements or a problem from the user's perspective. A defect tracking system can be paper based, an Excel spreadsheet, or a website. Even if there is one person on the project it's good to keep track of the defects in some way.

The status of a defect can be:

• New - the defect has been added but
• Confirmed - a developer has recreated the defect
• Fixed - the developer has fixed the defect, it's time for the user to make sure it is fixed.
• Duplicate - the developer believes this is a duplicate, time for the user to make sure
• Not a Defect - developer does not see this as a defect, time for the user to make sure
• Resolved - the defect has been resolved in some way

While running a software quality organization, I found that developers had a habit of resolving defects without taking to the tester who found it. This led to a variety of problem, none of which resulted in the defect being fixed. The solution was for the developer to propose a resolution for the defect and then the tester would review it to make sure the developer had understood and fixed the intended defect. This led to better communication between testers and developers and defects were fixed.