Some Basic Testing Concepts

Tests are Tools

A test is simply a tool that is used to measure something. To narrow that definition a little - after all, we do measure stuff all the time without having any interest in testing it - a test is usually formal, in the sense that is it created and applied with a purpose and intentionally. I may measure a television set because I have an idle curiosity about its size, but if I'm in the market for a new television, and I have a specific space to put the set, then I'm measuring that TV for a very definite reason.and I'm therefore testing that television for its ability to meet my space restrictions.

The "something" that a test is measuring can often be summarized with a question:

What are the subject's characteristics or properties? This kind of measurement looks at the test subject itself.
Does the test subject pass or fail the test? This kind of measurement compares the subject, or the subjects performance or behavior, against a concrete definition of what success means. The test evaluates the subject based on that definition; if that requirement is met, the subject passes the test.
How does the subject respond to the test? This kind of measurement evaluates some arena of performance or behavior, and is usually intended to improve understanding of the test subject.
How do multiple test subjects compare in characteristics, performance or behavior? This kind of measurement creates a matrix of compared elements, allowing comparisons across various axes and data points.

A test requires more than just asking one of these questions, however. Tests must be planned and thought out a head of time; you have to decide such things as what exactly you are testing and testing for, the way the test is going to be run and applied, what steps are required, etc. A test is usually based on some kind of understanding of what a good result would be, or a specific definition of what "good" means. Using the example above, say I find a television that will fit, so I based on the fact that it passed my measurement test. I get home and set it up and then realize - oops, it doesn't come with a remote control - that I hadn't specified all of my requirements, and as a result hadn't tested for the all of the correct things that I needed.

A misunderstood or inadequately planned test can waste time and provide bad information, because the interpretation of the test results will be flawed and misleading. Oops again, I brought a metric tape measure, and I don't know how to convert. Darn, was I supposed to measure with or without the antenna? Did my wife tell me to look for a projection TV, or a Web TV?

Before running any test, you should be able to answer the following rough questions:

What are you testing? Define the test subject, whether it is a thing, a process, a behavior, a threshold, etc. Define the scope of the test subject. For example, if you are testing a web site's links, will you test every link, or only links to static pages, or only links to other pages as opposed to internal links, etc?
From what point-of-view are you testing? If your test is supposed to mimic the interaction of a specific agent or user, then you must have a strong understanding of that agent or user. [Read more about understanding users.]
What are you testing for? Be as specific as possible. If you are going to test one aspect of the test subject, make that limitation clear.
How are you going to test? Define the test plan, the specific test(s), test methodologies, etc.
What are the limits to the test? Set expectations carefully, because if the test can only measure a part of the test subject or its behavior, the results must be interpreted with this limitation in mind.

Testing, Quality Control and Quality Assurance

Testing is often confused with the processes of quality control and quality assurance. Testing is the process of creating, implementing and evaluating tests. If you are shopping for a new television, you can call that process "testing for the best TV for you"... it's kind of pretentious, but that is what you're doing as you compare prices and features to find what will work best for you. Testing usually has a limited scope and duration - you're just looking at TVs, and only in your town, you're not going to spend a year shopping, are you?

Testing is predicated on the use of a standard of quality: you have a specification of what's allowable (no broken links? ALT tags have values? maximum page weight is 10K?) and you compare the site to the standard, noting deviations and shortcomings. This seems simple, but your testing is only valuable if your standard of quality is comprehensive, well thought-out, and reasonable. If your standard has holes, then your testing process has blind spots.

Quality control is a refinement of testing, involving the formal and systematic use of testing and a precise definition of what quality means for the purposes of the test. You aren't just testing, you are testing and then doing something with the results. Quality control is used for testing a product or output of a process, with the test measuring the subject's ability to meet a certain benchmark or threshold of quality. The tests usually take the form of "does this product meet requirement X?", and are often pass-fail.

Effective quality control testing requires some basic goals and understanding:

You must understand what you are testing; if you're testing a specific functionality, you must know how it's supposed to work, how the protocols behave, etc.
You should have a definition of what success and failure are. In other words, is close enough good enough?
You should have a good idea of a methodology for the test, the more formal a plan the better; you should design testcases.
You must understand the limits inherent in the tests themselves.
You must have a consistent schedule for testing; performing a specific set of tests at appropriate points in the process is more important than running the tests at a specific time.

Any true attempt at quality control requires a great deal of planning before any tests are ever applied, and extensive documentation of quality standards, test plans, test scenarios, test cases, test results -- anything that goes into the testing must be carefully tracked and written down. In fact, for companies that manufacture products, as well as for software companies, a series of formal accreditation programs exist to measure and certify the company's adherence to some very strict standards, for example the ISO 9000 series of rules. No such certification systems exist for web sites, perhaps because sites are more experiences and resources than products to buy.

The distinctions between testing and quality control are important for an understanding of the roles and purposes of testing, but they are especially important to anyone involved in testing or creating large a web site. Based on my own experiences, I strongly recommend that testing for site quality be a priority for anyone who

works as part of a team that is building and/or maintaining a big web site, and whose responsibility is for testing, quality control, or quality assurance
delivers a site to a customer
receives site code from a contractor, agency, or technology partner

Testing -- and by extension quality control -- is reactive; that is, you test to find deviations from a standard. If you systematically employ a formal battery of tests on a consistent schedule, you will be able to pass a product with fairly stable quality. The shortcoming here is that this kind of testing does nothing to improve the quality of output; as far as user-experience is concerned, you're just running in place. Testing and quality control do nothing to raise the level of quality beyond perhaps tweaking the standard to "raise the bar". Quality assurance goes beyond quality control to examine the processes that create and shape the product: quality assurance looks at the quality of output, as well as at the quality of the inputs.