1 Objective and Scope
The primary aim of
this document is to highlight key considerations in Performance Testing
and to provide an insight into the rigor and depth of performance
testing
2 Performance Testing
Performance Testing can be viewed as the systematic
process of collecting and monitoring the results of system usage and
analyzing them to aid system improvement towards desired results.
As part of the performance testing process, one needs to gather
statistical information, examine logs of system state histories,
determine system performance under natural and artificial conditions and
alter system modes of operation.
Performance testing complements functional testing. Functional
testing can validate proper functionality under correct usage and proper
error handling under incorrect usage. It cannot, however, tell how much
load an application can handle before it breaks or performs improperly.
Finding the breaking points and performance bottlenecks, as well as
identifying functional errors that only occur under stress requires
performance testing.
The purpose of Performance testing is to demonstrate that
• The application processes required transaction volumes within
specified response times in a real-time production database (Speed).
• The application can handle various user load scenarios (stresses),
ranging from a sudden load “spike” to a persistent load “soak”
(Scalability).
• The application is consistent in availability and functional integrity (Stability).
• Determination of minimum configuration that will allow the system
to meet the formal stated performance expectations of stakeholders
Basis for inclusion in Load Test
High frequency transactions: The most frequently used transactions
have the potential to impact the performance of all of the other
transactions if they are not efficient.
Mission Critical transactions The more important transactions that
facilitate the core objectives of the system should be included, as
failure under load of these transactions has, by definition, the
greatest impact.
Read Transactions: At least one READ ONLY transaction should be
included, so that performance of such transactions can be differentiated
from other more complex transactions.
Update Transactions At least one update transaction should be
included so that performance of such transactions can be differentiated
from other transactions.
2.1 Types of Performance Testing
Benchmark Testing:
The
objective of Benchmark tests is to determine end-to-end timing of
various critical business processes and transactions while the system is
under low load with a production sized database.
The best time to execute benchmark tests is at the earliest
opportunity. Developing performance test scripts at such an early stage
provides opportunity to identify and remedy serious performance problems
and expectations before load testing commences.
A key indicator of the quality of a benchmark test is its
repeatability. That is, the re-execution of a performance test should
give the same set of results. If the results are not the same each time,
differences in results cannot be attributed to changes in the
application, configuration or environment being tested.
Stress Tests:
Stress tests have one primary
objective, and that is to determine the maximum load under which a
system fails, and how it fails.
It is important to know in advance if a ‘stress’ situation will
result in catastrophic system failure or if all components of the system
simply ‘just go really slow’. Catastrophic failures often require the
restarting of various infrastructures and contribute to downtime,
stressful work environments for support staff and management, as well as
possible financial loss and breaching of SLAs.
Targeted Infrastructure Tests:
The objective of Targeted Infrastructure tests is to individually
test isolated areas of an end-to-end system configuration. This type of
testing would include communications infrastructure such as:
-Load balancers;
-Web servers;
-Applications servers;
-Databases.
Targeted Infrastructure testing allows for the identification of
any performance issues that would fundamentally limit the overall
ability of a system to deliver at a given performance level. Targeted
Infrastructure testing separately generates load on each component of an
end-to-end system, measuring the response of each component under load.
Each test can be simple, focusing specifically upon the individual
component being tested. It is often wise to execute Targeted
Infrastructure tests upon isolated components prior to Load or Stress
testing as it is much easier to identify (and quicker to rectify)
performance issues in this situation rather than in a full end-to-end
test.
Soak Tests (Endurance Testing):
Objective of soak
testing is to identify any performance problems that may appear after a
system has been running at a high level for an extended period of time.
It is possible that a system may ‘stop’ working after a certain number
of transactions have been processed, maybe due to:
-Serious memory leaks that would eventually result in a memory crisis;
-Failure to close connections between tiers of a multi-tiered system which could halt some or all modules of a system;
-Failure to close database cursors under some conditions which could eventually result in the entire systems stalling;
-Gradual degradation in response time of some function as internal
data structures become less efficient during a long high intensity test.
Volume Tests:
Volume tests are tests directly
relating to throughput, and are usually associated with the testing of
‘messaging’, ‘batch’ or ‘conversion’ type processing situations.
The objectives of Volume tests are:
- To determine throughput associated with a specific process or transaction;
- To determine the ‘capacity drivers’ associated with a specific process or transaction.
Volume testing a system involves the focusing of throughput through a
system function (say, in bytes) rather than response time of a system
function (say, in seconds).
It is important when designing Volume tests that the capacity
drivers are identified prior to the execution of the Volume testing to
ensure meaningful results are recorded. Capacity drivers in a batch
processing function could be:
Record Types: The record types contained within one specific
batch job run may require significant CPU processing while other record
types may invoke substantial database and disk activity. Some batch
processing function can also contain aggregation processing, and the mix
of data contained within a batch job can significantly impact the
processing requirements of the aggregation phase.
Database Size: The total amount of processing effort for a batch
processing function may also depend upon the size and make-up of the
database the batch job is interacting with.
Failover Tests:
Objective of Failover Test is to
get the system under test into steady state and start failing components
(servers, routers, etc) and observe how response times are effected
during and after the failover and how long the system takes to
transition back to steady state.
Failover testing determines what will occur if multiple
web-servers are being used under peak anticipated load, and one of them
dies. Does the load balancer used in this architecture react quickly
enough? Can the other web-servers handle the sudden dumping of extra
load?
Network Sensitivity Tests:
Network sensitivity
tests specifically focus on Wide Area Network (WAN) limitations and
network activity (traffic, latency, error rates, etc) and then measure
the impact of that traffic on an application that is bandwidth
dependant. The primary objectives of Network Sensitivity Tests are:
• Determine impact on system response time over a WAN;
• Determine the capacity of a system based on a given WAN;
• Determine the impact on a system under test that is under ‘dirty’ communications load.
Response time is the primary metric of measure for Network
Sensitivity testing, and is recorded as part of scenario test execution.
Response time can be estimated as –
Response Time = Transmission Time + Delays + Client Processing Time + Server Processing Time
Where:
Transmission Time = Data to be transferred divided by bandwidth
Delays = Number of turns multiplied by ‘Round Trip’ response time
Client Processing Time = Time taken on users software to fulfill request
Server Processing Time = Time taken on server computer to fulfill request
2.2 When to Start and Stop Performance Testing?
When to Start Performance Testing
A common practice is to start performance testing only after
functional, integration, and system testing are complete; that way, it
is understood that the target application is “sufficiently sound and
stable” to ensure valid performance test results.
However, the problem with the above approach is that it delays
performance testing until the latter part of the development lifecycle.
Then, if the tests uncover performance-related problems, one has to
resolve problems with potentially serious design implications at a time
when the corrections made might invalidate earlier test results. In
addition, the changes might destabilize the code just when one wants to
freeze it, prior to beta testing or the final release.
A better approach is to begin performance testing as early as
possible, just as soon as any of the application components can support
the tests. This will enable users to establish some early benchmarks
against which performance measurement can be conducted as the components
are developed.
When to Stop Performance Testing
The conventional approach is to stop testing once all planned
tests are executed and there is consistent and reliable pattern of
performance improvement. This approach gives users accurate performance
information at that instance. However, one can quickly fall behind by
just standing still. The environment in which clients will run the
application will always be changing, so it’s a good idea to run ongoing
performance tests.
Another alternative is to set up a continual performance test and
periodically examine the results. One can “overload” these tests by
making use of real world conditions. Regardless of how well it is
designed, one will never be able to reproduce all the conditions that
application will have to contend with in the real-world environment.
2.3 Pre-Requisites for Performance Testing
Following are the prerequisite which should be in place before performance testing is commenced –
• Quantitative, relevant, measurable, realistic, achievable requirements
As a foundation to all tests, performance requirements should be
agreed prior to the test. This helps in determining whether or not the
system meets the stated requirements. The following attributes will help
to have a meaningful performance comparison.
• Stable system
A test team attempting to construct a performance test of a system
whose software is of poor quality is unlikely to be successful. If the
software crashes regularly, it will probably not withstand the
relatively minor stress of repeated use. Testers will not be able to
record scripts in the first instance, or may not be able to execute a
test for a reasonable length of time.
• Realistic test environment
The test environment should ideally be the production environment or
a close simulation and be dedicated to the performance test team for
the duration of the test. A test environment that bears no similarity to
the actual production environment may be useful for finding obscure
errors in the code, but is, however, useless for a performance test.
• Controlled test environment
Performance testers require stability not only in the hardware and
software in terms of its reliability and resilience, but also need
changes in the environment or software under test to be minimized.
Automated scripts are extremely sensitive to changes in the behavior of
the software under test. Test scripts designed to drive client software
GUIs are prone to fail immediately, if the interface is changed even
slightly. Changes in the operating system environment or database are
equally likely to disrupt test preparation as well as execution and
should be strictly controlled.
• Performance testing toolkit
The execution of a performance test must be, by its nature,
completely automated. However, there are requirements for tools
throughout the test process. Main tool requirements for Performance
Testing Toolkit are as following -
Test database creation/maintenance
Load generation tools
Resource monitoring
Reporting Tools
3 Performance Testing Methodology
The
typical performance testing methodology includes four phases:
preparation, development, execution and results summary as shown in the
diagram below.
Performance Test Preparation: The first phase starts prior to
commencing the performance testing. Project Manager / QA Manager
perform preparation tasks such as planning, designing, configuring the
environment setup etc.
Script Development: The second phase involves creating the
performance test scenarios and relevant test scripts that will be used
to test the system.
Test Execution/Analysis: The third phase includes running the
scenario. The data gathered during the run is then used to analyze
system performance, develop suggestions for system improvement and
implement those improvements. The scenarios may be iteratively rerun to
achieve load test goals.
Test Results Reporting: The purpose of the last phase is to report the outcome of the work performed for the load test.
3.1 Performance Test Preparation
The
first step in a successful implementation is to perform preparation
tasks which include planning, analysis/design, defining “white box”
measurement, configuring the environment setup, completing product
training and making any customization, if needed.
Planning Purpose of planning is to define the implementation
goals, objectives, and project timeline. Project managers and/or
technical leads typically perform the planning phase in conjunction with
the implementation teams –
• Project goals broadly define the problems that will be addressed and the desired outcome for testing.
• The project objectives are measurable tasks that, once completed, will help meet the goals.
• The project timeline will outline the sequence, duration and staff responsibility for each task.
Analysis/Design
In this context, the analysis/design should first identify a set
of scenarios that model periods of critical system activity. This
analysis is especially important in global operations where one
continent’s batch processing is running concurrently with another
continent’s online processing.
High volume business processes/transactions should be built into the
test. Choosing too few transactions might leave gaps in the test while
choosing too many will expand the script creation time. It is effective
to model the most common 80% of the transaction throughput; trying to
achieve greater accuracy is difficult and expensive. This is typically
represented by 20% of the business processes—roughly five to 10 key
business processes for each system module.
Margin for Error
Since load testing is not an exact science, there should be
accommodations made to provide a margin for error in the test results.
This can compensate for poor design and help avoid false positives or
negatives. A load test should include at least one stress test or a peak
utilization scenario. A stress test will overdrive the system for a
period of time by multiplying the load by some factor—120% or greater.
Peak utilization will address the testing of peak system conditions.
White-Box Measurement
The white-box measurement section defines the tools and metrics used to measure internal system
Under-test (SUT) performance. This information helps to pinpoint the
cause of external performance issues. It also leads to recommendations
for resolving those issues.
Environment Setup
The purpose of the environment setup phase is to install and
configure the system under test. Preparation includes setting up
hardware, software, data, performance test tool and white-box tools.
Since the sole purpose of this test environment is to conduct
performance tests, it must accurately represent the production
environment. It is crucial to know the specifications of the Web server,
databases, or any other external dependencies the application might
have.
• Software In addition to the hardware required for a load test,
the test bed must also have fully installed and functioning software.
Since Performance Test Tool functions, “just like a user,” the system
would need to successfully support all user actions.
• Network it is probably impossible to accurately model each and
every network access (FTP, print, Web browse, e-mail download, etc.),
it is judicious to examine the current network utilization and
understand the impact of incremental network traffic.
• Geography Often the application under test will support a
global enterprise. In this environment tests may often need to be run at
remote sites across the WAN. WAN connectivity needs to be emulated in
the lab, or assumptions must be made.
• Interfaces Large systems seldom service a company’s entire
information needs without interfacing to existing legacy systems. The
interfaces to these external data sources need to be emulated during the
test, or excluded with supporting analysis and justification.
3.2 Script Development
During the
script development phase, the test team builds the tests specified in
the design phase. This depends on the number of tests, test complexity
and the quality of the test design.
Initial Script Development
It is desirable to have a
high degree of transparency between virtual users and real human users.
In other words, the virtual users should perform exactly the same tasks
as the human users. At the most basic level, any performance test tool
offers script capture by recording test scripts as the users navigate
the application. This recording simplifies test development by
translating user activities into test code. Scripts can be replayed to
perform exactly the same actions on the system. These scripts are
specified in the design and should be self-explanatory. Any of these
following issues can easily increase script development time. –
Lack of Functional Support
One of the most important factors in script creation productivity is
the amount of functional support provided—access to individuals who
understand application functionality. This manifests itself when a test
team member encounters a functional error while scripting—the business
process won’t function properly. The team member typically has to stop
since he or she is not equipped with the skills to solve the issue. At
that point, script creation is temporarily halted until a functional
team member helps resolve the issue.
Poor Quality of Test Design
The second script development factor is the quality of the test
design. Ideally the test design should specify enough information for an
individual with little or no application experience to build tests.
System test documentation is often an excellent source of this
information. Often designs are incorrect or incomplete. As a result, any
omission will require functional support to complete script
development.
Low Process Stability
To load/stress test a large system, the system’s business processes
first need to function properly. It is typically not effective to
attempt to load test a system that won’t even work for one user. This
typically means that the system needs to be nearly completed.
System Changes
A key factor in script development is the frequency of system
changes. For each system revision, test scripts need to be evaluated.
Tests may require simple rework or complete reconstruction. While
testing tools are engineered to minimize the effect of system change,
limiting the system changes will reduce scripting time.
Availability of Test Data
The system will need to be loaded with development test data. This
data often comes from a legacy-system conversion and will be a
predecessor to the volume data for the test.
Parameterization Script
Replaying the same user
actions is not a load test. This is especially true for large multi-user
systems where all the users perform different actions. Virtual user
development should create a more sophisticated emulation—users should
iteratively perform each business process with varying data. Script
development next extends the tests to run reliably with parameterized
data. This process reflects the randomness of the user population
activity.
Build Volume Data Parallel to script development
Volume
data should be constructed to support the execution of the load test.
Typically business processes consume data—each data value may be used
only once. As a result, there needs to be sufficient data to support
large numbers of users running for a number of iterations— often 10,000
items or more.
3.3 Test Execution/Analysis
The
execution/analysis phase is an iterative process that runs scenarios,
analyzes results and debugs system issues. Test runs are performed on a
system that is representative of the production environment. Performance
test tool is installed on driver hardware that will create traffic
against the application under test.
Data Seeding
The system should be “pre-seeded” with data consumed by the testing process. To keep testing
productivity high, there should be enough data to support several iterations before requiring a system refresh.
System Support
The purpose of system support is to help
interpret performance results and white-box data. While the Performance
tool will describe what occurred, the system support could help to
describe why and suggest how to remedy the problems. These suggestions
can be implemented and the tests rerun. This iterative process is a
natural part of the development process, just like debugging.
Light Load
The first step is to run the scenario’s
test scripts with a small numbers of users. Since the scripts functioned
properly in the development environment, the emphasis should be to
recreate this functional environment for execution. Any new script
execution errors will typically indicate system configuration
differences. It is advisable to avoid script modifications at this stage
and concentrate on system-under-test installation.
Heavy Load
Finally the last step is to run a
full-scale load test. This typically consumes 50% of the total
execution/analysis time. Once the entire scenario is running, the effort
shifts to analyzing the transaction response times and white-box
measurements. The goal here is to determine if the system performed
properly.
3.4 Test Results Reporting
Finally,
the results summary describes the testing, analysis, discoveries and
system improvements, as well as the status of the objectives and goals.
This typically occurs after the completion of testing and during any
final “go live” preparation that is outside the scope of testing.
The Performance Test deliverables could be:
• Performance Test Strategy
• Load Scenarios
• Virtual User Scripts
• Status/Analysis Reports
• Performance Test Summary Document
Following measurements can be illustrated in performance test report –
Attempted Connections: The total number of times the virtual clients attempted to connect to the AUT (Application under Test).
Connect Time: The time it takes for a virtual client to
connect to the application being tested (the ABT), in seconds. In other
words, the time it takes from the beginning of the HTTP request to the
TCP/IP connection.
Hit Time: The time it takes to complete a successful HTTP
request, in seconds. (Each request for each database transaction,
business logic execution, etc. is a single hit). The time of a hit is
the sum of the Connect Time, Send Time, Response Time, and Process Time.
Load Size: The number of virtual clients running concurrently.
Receive Time: The elapsed time between receiving the first byte and the last byte. (Network traffic)
Response Time: The time it takes the ABT to send the object
of an HTTP request back to a Virtual Client, in seconds. In other words,
the time from the end of the HTTP request until the Virtual Client has
received the complete item it requested (Wait Time + Receive Time).
Successful Hits: The total number of times the virtual
clients made an HTTP request and received the correct HTTP response from
the ABT. (Each request for each database transaction, business logic
execution, etc. is a single hit).
Transactions/sec (passed): The number of completed, successful transactions performed per second.
Transactions/sec (failed): The number of incomplete failed transactions per second.
Bandwidth Utilization: Assesses the network health during performance testing.
Memory Utilization: Comparison of memory usage on the server before and during the load test.
%CPU Utilization: Comparison of % CPU utilization on the server before and during the load test.
DB connections: Variation in the number of open db connections
