organizations have a method to test services for scalability and
performance. One fortune 50 company asked a summer intern to run a few
performance tests when he had time between other assignments to check
and identify scalability problems in an SOA-based information system!
That was their entire approach to scalability and performance testing.
I'm an advocate for testing XML-based Service-Oriented Architecture
(SOA), Enterprise Service Bus (ESB), Web services and Business
Integration (BI) based services. Even risky ad-hoc testing is good.
(It's risky because most of the time results from ad-hoc tests are
misleading). In my experience the business value of running
scalability and performance tests comes once an organization
formalizes a test method that includes the following:
1. The organization chooses an appropriate set of test cases. For
instance, testing a multiple-interface and high volume service is
different than testing a service that handles periodic requests with
huge message sizes. The test needs to be oriented to address the
end-user goals in using the service and deliver actionable knowledge.
2. The organization runs enough tests and accurately. Understanding
the scalability and performance of a service requires dozens to
hundreds of test case runs. Ad-hoc recording of few test results is
unsatisfactory.
3. The organization makes the right conclusions when analyzing the
results. Understanding the scalability and performance of a service
requires understanding how the throughput - measured as Transactions
Per Second (TPS) at the service consumer - changes with increased
message size and complexity and increasing concurrent requests.
All of this requires much more than an ad-hoc approach to reach useful
and actionable knowledge. I usually begin a test project with the
following question: What do I need to learn from a performance test
and how will it benefit my company, business, or institution?
This question gets directly to the heart of the reason why we test
services built with XML, Web Service, ESB, and BI tools. It may be
more difficult to answer than it first appears. In some test projects
the answer took longer to find than the time it took to run the actual
test.
The question challenges us to understand what parts of a service will
be optimized to contribute a business benefit for the testing effort.
For instance, most services are built on tools that provide the
following components:
* Application and Service Components: Workflow, Business Process
Execution, Orchestration
* Service Buses: Connectors, Caching, Policies, Scripting, Messaging
* Infrastructure Components: SOAP bindings, XML parsers, Message
service bindings, clustering, replication and high availability
The components build on each other in three tiers: At the bottom are
the fundamental components for SOAP bindings, XML parsing, Java
inter-application messaging services and clustering. Building on these
are service bus components for services to interoperate at a message
level. The top tier provides interoperability at an application level
exchanging messages at a business process level. Given this threefold
layer, where would you start testing to understand the scalability and
performance of an SOA implementation?
For instance, would it make sense to test only the connectors and
caching objects at the service bus level exclusively? If you did so
you may miss key performance bottlenecks in the SOAP bindings and
message service. Each component impacts the scalability and
performance of the resulting service.
Service architectures today depend on a consumer, service and broker
architecture. I find most performance tests normally check, at most,
two of the three. For instance, one performance test may check a
consumer and service, and in another example, test a consumer and a
broker. The following should help you identify which test will most
benefit your system for scalability and performance.
* XML Parsing - The benefit is to decrease the time it takes to
route a message to a service-handling object to lower network
bandwidth and server hardware costs. This is a stateless test. The
parameters to understand the test results include schema complexity
(depth and element count), document size, concurrent request levels.
* Persistence - The benefit is to decrease the time it takes to
store and retrieve messages to lower network bandwidth, server
hardware, and disk costs. This is a stateful test. The parameters to
understand the test results include schema complexity (depth and
element count,) document size, and concurrent request levels.
* Transformation - The benefit is to decrease the time it takes to
transform a message into a given XML schema to lower network bandwidth
and server hardware costs. This is a stateless test. The parameters to
understand the test results include source and destination schema
complexity (depth and element count,) request and output document
size, and concurrent request levels.
* Aggregation and Federation - The benefit is to decrease the time
it takes to respond to service requests that require inputs from
several up-stream data sources and to reduce network bandwidth and
server hardware costs. This is a stateful test. The parameters to
understand the test results include schema complexity (depth and
element count) for each up-stream service, data persistence
quantities, and time-to-live (TTL) values for each message.
* Data Mitigation - The benefit is to improve service availability
and user satisfaction by reducing the times when a service is
unavailable at times of peak usage. This is a stateful test. The
parameters to understand the test results includes schema complexity
(depth and element count) for each request.
The above list covers the service test goals I most often encounter.
However, there are many more possibilities and the pace of innovation
within SOA building tools is very high. The above list describes
stateful, stateless, and test parameters. The following explains these.
* Stateless testing checks the impact of concurrent requests and
message payload size on a service. The service responds to each
request independently from all the other requests. It is therefore
stateless.
* Stateful testing is similar to stateless testing; however, the
service provides data persistence, workflow transaction processing,
message queuing, sessions, or data indexing. These required shared
resources that are often interdependent. Sometime even with extra CPU,
network and memory resources available the throughput does not increase.
* The parameters to understand the scalability index define the
inputs to a test that will show us the scalability index - the usage
pattern that shows us what we can expect of our service's performance
in production.
By taking these steps, considering these parameters, and adopting a
test methodology appropriate to your service your business benefits by
developing actionable knowledge. Improved performance and scalability
come as a result.>>
You can find this at:
http://www.theserverside.com/tt/articles/content/ReadytoTest/article.html
Gervas
SPONSORED LINKS
| Computer software | Computer aided design software | Computer job |
| Soa | Service-oriented architecture |
YAHOO! GROUPS LINKS
- Visit your group "service-orientated-architecture" on the web.
- To unsubscribe from this group, send an email to:
[EMAIL PROTECTED]
- Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service.
