Performance Testing Is Not Just a Stopwatch
Stress Testing vs Load Testing
by History of Stress and Load Testing
My first exposure to performance testing was some time ago working on a government contract developing a Command-and-Control System. This was both hardware and software, and testing included “Preliminary” testing for message transmission.
Don’t laugh, but we actually used a stopwatch to time the transmission. The military was more interested not in the computer times that we could have extracted from logs, but the time the messages would actually display on the screens.
Mainframe computers were developed in the 50’s and 60’s. It was not until the 60’s that ways of capturing data, internal to the computer was developed. In 1966 System Management Facilities was developed to collect data as part of the OS/360 release. In the 1970’s, further developments led to the release of Resource Management Facility (RMF) which was part of the MVS and provided real time monitoring. in the 1980s, many other performance types of tools then were produced for the mainframe computers and software testing tools.
Mercury Interactive was founded in 1989 and shortly thereafter they released the first version of LoadRunner. Prior to this release there were still ways of measuring software performance, but the collection of data was manual and tedious. Logs that contained the data were exported and manipulation of the data was done to provide understandable results. Now the tools provide those results without much manipulation.
Performance Testing Terminology
Load
Testing an application using a predefined amount of processing to determine how that application responds. This usually ensures that the application will function properly under normal conditions.
Stress
Tests using above normal number of processing/requests to find out how an application responds to the increased workload. This will at times “stress” the application or the hardware.
Endurance
This test is executed over a long period of time to validate the application can maintain performance over time.
Spike
A test where a very high amount of processing is done at one time. This can be many users logging onto an application at once, or other such processing.
Volume
A large amount of data is used to perform volume testing. This usually consists of increasing the database size or the amounts of transactions for the database.
Scalability
Testing an application/system to determine if network, memory or CPU affects the performance. It is important to note that the scale of the increments is manageable to accurately determine when the changes occur.
Are Stress and Load Testing the same?
These two terms are probably the most misused terms in performance testing. People interchange the two quite often. The definitions have been previously given, and by that the two types of tests are not the same. Most people just use the terms interchangeably because it is easier to remember than performance testing. The actions and goals of the two types are different.
Requirements
Performance testing really needs good requirements so that the results can be of any use.
Like all software requirements they should be detailed and explicit. Another key item for performance requirements is that it should be quantifiable.
Here is an example of a requirement to display the home screen:
- The home screen should be displayed as fast as possible.
That requirement as written is totally subjective. It does not have a quantifiable amount. Lastly it is not detailed.
Without well written requirements a performance test would not be able to meet the goals of measuring the attributes of the system like responsiveness, speed, and stability.
A revised example of the requirement above might be:
- When a user logs on, then the home screen will be displayed in 1.2 seconds.
This requirement now is detailed and quantifiable, without a possible subjective interpretation.
Summary
Performance testing is more than timing an application or just a part of the application. A full understanding of the types of performance testing would guide the effort. It also involves the systems that are being used for the testing, the requirements that are being verified and the tools that are used during the testing.
So put that stopwatch away and get your shovel out and dig for more knowledge about performance testing.
Check out some of our other Tech Talks!