Building a Test Suite

A web site is designed and built for an audience, and much attention should be paid to the process of understanding that audience. Any testing of a web site should proceed from one obvious point: the people on the web site teams - whether they are designers, programmers, marketers, or testers - are not typical users.

Just as these folks aren't typical users, any computer they use for their daily tasks is unlikely to represent the computers used by true end-users. Realistic testing of a web site requires test environments that match - as closely as reasonably possible - computers used by the site's audience(s).

The Need for a Test Suite

A test suite is a set of machines configured as platforms for testing a web site; these machines should represent the client-side environments of the majority of the site's audience. A test lab on the other hand would be a specially equipped and designed facility or space used for testing, specifically usability testing. Test labs allow for the monitoring of users acting as test subjects. Of course, with that distinction made, any computers set aside for testing are going to be used as an ad hoc test lab.

Testing a web site requires an understanding of what and how to test. You must know what you are going to test: what functionality, what logic, what scenario, what user conditions, etc. You must also know how you are going to test: which tests, which tools, what methodology. And just as important, you must know where you are going to run your tests: what machines, what configurations, what connectivity, etc. This last point is what the test suite addresses.

Any computer that is not dedicated solely to testing -- like a designer's Mac or a tester's personal desktop -- is going to be compromised as a test environment, unless your site's targeted audience includes people exactly like you. This is a difficult path to follow, believing that a personal desktop is adequate for testing, because this path takes you away from what you can qualitatively show to be your audience's typical system and configuration.

The Goals for a Test Suite

A test suite can yield several benefits to the production and testing process, stated here as a set of goals:

1. Provide "clean" environments for testing platform/browser compatibility for pages in development and in production, allowing a more objective view of what standard configurations would encounter.
2. Provide an increase in productivity by providing a means to rapidly test and review prototypes on all common browsers and platforms.
3. Provide environments for testing network connectivity and performance to the production servers over the open net (as opposed to testing over a "back door" LAN connection). This would duplicate the connections experienced by end-users.
4. Provide a "lab" for usability testing. This assumes that the test suite will be located within a space that allows for most of the machines to be in use at the same time, and in a way that allows for some level of observations of the users.

Designing the Test Suite

A well-designed test suite should address the following points:

• The browsers most likely to be used by your audience. This information comes from research on commonly available browsers combined with analysis of your site's logs.
• The platforms most likely to be used by your audience. This information comes from research on commonly available platforms combined with analysis of your site's logs.
• The ways in which different browsers and platforms interact. Some browsers and versions of browsers don't play well together; for example, different levels of Microsoft's Internet Explorer won't install correctly on the same Windows 95 environment. Browsers also render HTML differently and handle plug-ins differently depending on the operating system. Design a test suite that correctly handles these behaviors and interactions.
• The relative importance of certain user profiles. Some sets of users may have platform/browsers needs that are especially important to your site's business goals, so it makes sense to prioritize the accommodation of these users.
• The budget for testing. Less money available to spend on test machines translates to less specialization of the test environments. It is possible to use a few machines with multiple, bootable operating system environments, but managing these environments requires time and attention resources. In addition, the more complex the maintenance of a test machine, the harder it is to create realistic usability test environments.

Basic Test Suite For a Mainstream Commerce Site

Most commerce sites will have approximately the same requirements for test suite environments, mostly because the range of client-side environments is dominated by a rather small set of machine types, operating systems, and browsers. As soon as a site starts drawing increased proportions of its user base from people using palm-top browsers as platforms (such as Palm Pilots), WebTVs, or browsing-enabled kitchen appliances, then this sample suite would need to be revised.

As the test suite machines get older, and as newer configurations and generations of hardware and software become commonly available, these boxes should shift downward and newer boxes should be added.

These machine configurations were chosen as being the best general match for a particular set of users; the identification of the set is based on a range of characteristics, extending from ubiquity of the machine to the sociological tendency for adopting new technology.

It is certainly possible to create dual-boot machines, allowing for multiple operating systems on one box and requiring less space for these machines. Using one box for more than one operating system means that in the event that any one machine dies, multiple test environments are lost. The added convenience of a space or hardware cost savings must be balanced against any potential chance of decreased stability on a test machine; in my experience, the test phase of any project often becomes fixed, so there is no time to repair a downed machine during critical test phases. Along these lines, every machine should have a CD-ROM drive and a SCSI card for facilitating restores from a CD or from some other SCSI peripheral.

1. Archaic PC

This PC reflects a 16-bit environment. As a test environment, this machine will only be useful for a few more years, as Windows 3.1 is steadily dropping in usage; I would expect this box to be re-born as a Linux box pretty soon.

• Intel PC, can be 386 or 486 (486 would remain useful longer, but 386 is probably cheaper)
• Windows 3.1 OS
• Internet Explorer 16 bit engine
• AOL 16 bit engine
• Netscape Navigator 16 bit engine
• Lynx 16 bit engine
• Opera 16 bit engine
• 28K modem
• analog line
• 14" monitor (or, this box could share a monitor with the "Older Mainstream PC" by means of a switchbox)

2. Older Mainstream PC

• Intel PC, can be a 486 or P166 Pentium
• Windows 95 OS
• Internet Explorer 3.* engine
• AOL 3.*
• Netscape Navigator 3.04
• Lynx
• Opera
• 56K modem
• analog line
• sound card + speakers
• 15" monitor

3. Early Adopter Platform

The "Early Adopter Platform" and the "Corporate Desktop" environments can be based on the same hardware; the more machines based on the same model computer, the easier the maintenance tasks.

• Intel PC Pentium II, at least 64 Megs RAM
• Windows 98
• Internet Explorer 4.* engine
• AOL 4.*
• Netscape Navigator 4.*
• Opera
• special use browsers (screenreader, TBD)
• 56k modem
• analog line
• sound card + speakers
• 17" monitor

4. Corporate Desktop Platform

The "Early Adopter Platform" and the "Corporate Desktop" environments can be based on the same hardware; the more machines based on the same model computer, the easier the maintenance tasks.

• Intel PC Pentium II
• Windows NT 4.* Workstation
• Internet Explorer 4.* engine
• Netscape Navigator 4.*
• 56k modem
• analog line
• 17" monitor

5. Mainstream Mac Platform

• Mac PowerPC 2-3 years old
• Internet Explorer 3.* engine
• AOL 3.*
• Netscape Navigator 3.*
• 56k modem
• analog line
• sound card + speakers

6. New Mac User

• iMac
• Internet Explorer 4.* engine
• AOL 4.*
• Netscape Navigator 4.*
• 56K modem (standard)
• analog line
• sound card + speakers (standard)

7. New UNIX User

I'm still working out the specification for this environment, but it is basically supposed to represent the new converts to UNIX, those people who load Linux onto a PC.

• Intel PC
• Internet Explorer 4.* engine
• AOL 4.*
• Netscape Navigator 4.*
• 56K modem (standard)
• analog line

8. Experienced UNIX User

In contrast to the new UNIX user, the experienced user is running some flavor of UNIX on a non-PC box, say a SunSparc or an SGI workstation. I'm still trying to figure this one out.

9. WebTV

• television
• cable connection (analog line?)
• WebTV receiver