Enterprise

Despite the increasingly widespread adoption of J2EE for enterprise applications, measuring their performance in production continues to be a black art. Without knowing what to look for, many people measure anything that seems useful, which soon results in an overloaded system and reams of meaningless metrics data. It's tempting to just throw xup your hands and start making system changes based mainly on hunches.

This article provides a strategy for measuring performance in production. Starting with an explanation of some basic metrics that express the performance of a J2EE application, we'll apply these to a model of a J2EE system, showing how these metrics interact in different areas of the system. From here we will drill-down to discuss the nuts and bolts of gathering metrics from each type of J2EE component, from the client Web browser through to the back-end database. Finally, we'll uncover the quality and cost of different metrics and how to handle that tradeoff so that minimal overhead is imposed on the system being measured.

Basic Metrics
Some of the most basic metrics for a J2EE system (or any system really) are:

• Response Time (R)
• Throughput (X)
• Resource Utilization (U)
• Service Demand (D)

Response Time is the best overall indication of your end-users' experience. It has a much higher variation than the other metrics so it is essential to understand the distribution of response time. If most users experience two-second response times but 10% are getting 10 second response times, for example, you need to know this in order to assess and fix the problem.

Throughput, the number of transactions executed by the system over a period of time, is a good indication of the system's ability to handle load.

Resource Utilization, or how heavily a particular system element is being used, is the easiest metric to understand. It's not necessarily the most useful for determining system performance because only the utilization of contended resources has a significant impact on performance. We have found it more useful to define a new metric called Service Demand, calculated by the following formula:

Service Demand = Utilization of the resource / Throughput

Service Demand looks at resources in terms of the demands being put on them. This gives us a clear idea of the utilization of a resource as the users' demand for it increases.

Interrelating Metrics
These metrics are related, as shown by Figure 1. Understanding this relationship is central to building a model of J2EE performance.

The first thing to notice is that throughput and response time are often at odds. For an interactive application in production, we typically want to maximize throughput, as long as response time is at or below some threshold. For example, we may find we can achieve a maximum throughput of 100 transactions per second while keeping response time at or below two seconds.

Figure 1 also shows that resource utilization typically controls system behavior. It is resource contention that causes the dramatic drop in throughput and the commensurate rise in response time that marks the Buckle Zone. The effect of the Buckle Zone is a severe drop in application performance, due to the system spending most of its time managing resource contention, rather than servicing requests.

It's important to see how your application behaves at each of these three zones and the specific metric values that cause the system to shift from light to high load to buckle zone. This will be particularly useful for setting alerts in your production monitoring tool.

A Model for J2EE Systems
Let's fit these metrics into a model of a J2EE system that has four main processes, illustrated by Figure 2. Client Handling includes the originating requests and the sessions those requests may be tied to. Execution Management is where requests are queued before being assigned to execution threads. Applications includes both your code and all the standard J2EE components. Services include anything that connects your application to the outside, such as JDBC and JMS. Of course, all these elements run in a Java Virtual Machine (JVM) which, in turn, is using the resources of the operating system such as its processor, memory, disks, and network connectivity.

Figure 2 also shows how the metrics discussed earlier can be measured at each level. We'll need to measure and understand response time dispersion at the client-handling level, the application code level, as well as for services like JDBC. Throughput is most important at the client handling level - we have to understand how much user load our system can handle. Resource utilization is measured at many points in the system (OS, execute threads, services, etc) so that we can correlate the information and see how different elements in the system are affecting each other.

Points of Measurement and Overhead
There are no free metrics. Measuring something in real time always adds some overhead; the key is to make intelligent choices about what information to collect while keeping the overhead we're willing to incur in mind. The best way to determine the overhead of a measurement is to use the Service Demand calculation defined earlier. It is through that lens that we will now look at how we can most effectively and efficiently gather measurements from the J2EE system. Figure 3 illustrates the many possible measurement points in a J2EE system.

There are two methods for measuring client response time: browser scripting and injection of synthetic transactions. Browser scripting is usually implemented through JavaScript in the HTML pages returned to users. While the best measure of the user's experience, it does present several significant difficulties: it's very hard to measure all the clients reliably; deployment and maintenance of the scripting code for those pages can become difficult and tedious.

Synthetic transactions address most of these shortcomings and have become more commonly used. The idea is to inject synthetic or scripted user transactions into the system with some tool. These transactions can be easily measured and give a good approximation of what the users are experiencing. It's important to realize their limitations - unless the injector is placed near the end users, rather than just outside or inside the firewall, it cannot provide data on the wider network effects; also, creating realistic synthetic transactions does require a fairly detailed knowledge of user/site interaction and the patience to accurately model this interaction. However, the control that synthetic transactions provide overshadows these limitations.

Operating system (OS) metrics, familiar to most developers, will be gathered from machines throughout the J2EE system. Seeing the shifting patterns of CPU, memory, and disk usage on the various tiers of the system greatly aids understanding. But to build that data into a useful J2EE system model, you have to be able to accurately associate system metric information with application container and application code data at specific intervals in time. Only by doing this can you draw a picture of the complex interactions between the application, its application server container, and the underlying system.