This is exactly what happened when using JMS protocol in HPPC ( Loadrunner).  There is a memory leak in HPPC when using JMS. The mmdrv.exe process invokes JVM (in-process) during use of one of the jms functions (jms_send_message_queue).

During one of our tests we started getting out of memory error. Of course if you increase the JVM heap ( using -Xmx ) the error will be delayed but the fact remains it is leaking memory. It was very embarrassing to tell the application team that our tool is breaking during load testing.

We have been trying desperately to get HP to admit and fix it but they are making us go around in circle. We  used JConsole to monitor the JVM and prove to them that there is a leak but they still do not get it. The only thing that is left is for us to now fix the code for them.

In order to use JConsole with loadrunner JMS, you need to add the following in the -Dcom.sun.management.jmxremote in the “Additional VM Parameters” in order for JConsole to connect to the JVM.  Once you do that you can see the JVM memory keep on climbing inspite of you manually invoking GC.

Are you listening HP?

Now that Oracle is in talks to buy Sun what is in store for open source community. What will happen to all the open source projects initiated by Sun? The most interesting of them for Oracle is MySQL. MySQL was getting positioned to provide stiff competition to Oracle in near future. What will be its fate under Larry Ellison’s leadership?

It would be actually in best interest of Oracle to expand the reach of MySQL. If MySQL gets killed there will be other open source database taking birth and it will be a big headache for Oracle to keep fending off these open source initiatives. Oracle is in a great position now that it is acquiring the leading open source database and it can really use this opportunity to be at the center of the Open Source action. Of course Java  brand  has tremendous value and Larry Ellison might find ways to monetize it, which Sun had miserably failed to do so. What happens to the rest of the open source initiatives? It is anybody’s guess. It will be interesting to see what happens to OpenSolaris, Glassfish, Netbeans etc.

The only thing that does not fit well in this acquisition is the hardware aspect of Sun. Oracle has always been a software company but with this acquisition it suddenly has to have a hardware strategy. One possible scenario is that Oracle will just absorb the software products of Sun and sell off the hardware business. It is difficult to envision Oracle putting itself into the center of an already overcrowded server market.

Is this a acquisition a big blow to Open Source Community? I doubt so. It might just be an hiccup and probably the open source community will keep chugging along and getting people on board. What ever happens it will surely be very interesting to see how this acquisition plays out.

Measurement is the most important aspect of Performance Engineering. There is no place for guess work in performance engineering. Measuring tools are the most important tools in a Performance Engineer’s tool set.

In one of the projects I was involved in, the second day of my job I was validating the solution for a high throughput. Everything was going fine till the performance test started throwing lot of errors. The solution involved around 15 servers. I had set up monitoring of resources on all the 15 servers. First thing I did was look up the resource utilization of all the servers. There was one server where the memory utilization kept on growing. I narrowed down to the process which was growing in memory. The graph of the memory utilization for the process showed a linear growth, it went up as high as 1GB and then the process terminated. This graph was proof enough to show the developer that the process had a big memory leak. This was fixed within a day and solution was ready for further testing.

Performance engineering requires lot of discipline and a methodical approach. Lot of times when we come across problems we tend to start giving expert opinions, start guessing where the problem might be or start looking at the code. One needs to take a scientific approach. One has to look at the facts, start with resource utilization and then narrow down to looking at logs, timestamps, database metrics, application server metrics, compare with historical benchmarks etc.

Check this out. The very first computer bug.

Are you going to add 20 more servers since the load has doubled?

Are you sure that the application will scale up to 100 tps with added hardware?

Lets assume we have a perfectly scalable application. The first step is finding out which server is the bottleneck. Lets say the authentication server is at 90% CPU busy during peak load and is the only bottleneck in the system. Probably all you need is to add just one more authentication server and you could support 100 tps. You would be wasting time, money and resources by buying 20 more servers instead of just a single server which would have supported the 100 tps load. That is where Performance Engineering comes into picture.  Performance engineer is responsible for determining the scalability of the system and determine the bottlenecks both in the hardware resources and the software application.

There are two types of scalability, Vertical and Horizontal. Vertical scalability involves verifying that the software scales up on adding more resources ( cpu, memory, io, network) within a single server machine. Horizontal scalability involves verifying that the software scales up by adding more physical servers machines ( probably balanced via a load balancer ).  You determine the vertical scalability of the software by benchmarking on a particular hardware ( cpu, memory, io and network) for the maximum throughput and increasing the relevant resources and run another benchmark for the maximum throughput. If the software scales proportionately then it is scaling vertically.  Most of the times in real world you do not have time to test for vertical scalability and you just find out what is the maximum throughput on a single server and then see if it scales horizontally. Testing for horizontal scalability requires adding additional servers. Try to get three points in a graph to better understand the scalability. For e.g  on a 6 server architecture you could get benchmarking results on 1 server, 3 server and 6 servers and plot the throughput graph and understand if the software scales. If the software is not scaling proportionally or not scaling up at all then there is  probably a software bottleneck. If none of the hardware resources are found to be a bottleneck then we have a tough task at hand to determine the software bottleneck.

If you have a bottleneck in your software it will not scale up in spite of throwing more hardware at it. The most difficult and crucial part is finding and removing bottlenecks in the software. In one of the critical applications I had been working on, during my scalability tests I saw that with increased load the throughput was not increasing and the response time was growing linearly. This seemed to be a classic software bottleneck and I used all the tools at disposal, resource utilization tools, profiling tools, timestamp analysis tools etc. to narrow down where the problem is. The problem was in a method called “processMessage” within a class. This method was the entry point for multiple threads and there was one big lock sitting in there. This class had a member variable as a DOM object and in order to make the method “processMessage” as threadsafe there was a mutex surrounding the processing of this DOM object. This would serialize all the requests instead of processing them simultaneously in different threads. There was no need to have the DOM object as a member variable. All that was required was to instantiate the DOM object within the method “processMessage” , remove it from being a member variable and remove the mutex. This was a huge bottleneck and was due to careless programming and it would have been a disaster if the product went into production without a fix. A simple fix resolved the problem. Lot of times this is not so easy and needs much more time and repeated tests to narrow down the bottleneck. Just imagine if you have to find a software bottleneck in production environment. The customer impact could be huge.

The bottom line is that adding hardware is not always the right solution. Yes if your solution is scalable then adding additional hardware will increase the throughput. But you still need to know which hardware to add. Sometimes adding hardware also does not improve performance so one needs to know what are the bottlenecks within the software. This is one of the most difficult and challenging aspects of performance engineering. A good performance engineer will be able to determine by monitoring the resources if there are hardware bottlenecks and after resolving hardware bottlenecks if the software does not scale up he will be able to determine the software bottlenecks and resolve it for improved scalability. Bottlenecks sometimes keeps on moving from one layer to another. You might resolve a bottleneck in application layer and then the bottleneck might move to the database layer. That is what makes performance engineering so challenging and interesting.

For example a particular query might be doing a full table scan and this issue might not be noticed when testing with small amounts of data. When it goes into production and there is substantial amount of data, this single query can bring down the performance of the whole solution to its knees. This single query could eat up the CPU of the database machine thereby adversely affecting all the other queries. The time to find out the root cause of this issue while the product is in production could be substantial and the customer impact could be huge. There might be need to optimize your database configuration ( SGA size, tablespace settings, redo log settings, archival settings etc.)  in order to support huge amounts of data. Some of your queries would be using complex joins and dealing with large number of records and could impact performance. All these issues could be found and resolved prior to going in production by running performance tests with production like data.

In a recent consulting work I was involved in a project where not much thought was put into data retention. There was actually no data retention strategy. The application was dealing with large amounts of data and was retaining the data in the live database and would grow to 15 TB in just a few months. There is no way we could test with this kind of data and also there was no storage immediately available to handle this kind of data. Actually there was no need of storing all this data in the live database. All that was required was an archiving strategy. The data (documents) would only be looked into if there was some issue so there was no need to keep it in live database, also there was a downstream system which would do most of the processing and it would be safe to archive the data if the processing was successful.  In this case it was an architecture issue and just required an archival strategy.

Also another challenge is to upload huge amounts of data for performance testing  especially if it is the first release and there is not production data available. There is a need to generate huge amounts of data. Should you use the performance load tool to drive the load and create the data? I would advise against that. It might take whole lot of time just to populate the database especially if you drive from the front end and go through the whole solution. If possible one should use some kind of script to directly load data in the database and bypass the whole solution and speed up the process.

One area where open source has not yet made a big dent is in test automation and performance testing.
This area is still dominated by commercial tools. Most of the commercial tools available for automation and performance testing are extremely expensive. The leading tools from HP (QTP, Loadrunner) and Borland ( Silk Performer, Silk Test) come with a huge price tag.

I used to be a developer before I jumped into Performance Testing back in 1998. That time I was still not able to let go of my development skills. For the first Performance Testing assignment I had built a load testing tool using MS VC++ ( are we not talking of open source here?), the GUI for it is shown below. This was perfect for me. It suited all my needs for driving the load on the server. I had other scripts to automate monitoring, logging and analyzing the results ( see the tool image below).  Well right around then the company I used to work for, decided on standardizing the tool set and we settled on Silk Performer ( then a product of Segue). Using the tool was good for my resume value and it did have lots more bells and whistles.

But I can not say for sure that using this tool instead of my own nifty tool added more value to quality of the product I was testing. Load generation tool contributes to probably 20-30% of the whole performance engineering aspect. There is more to performance engineering then just using a load tesing tool, like benchmarking, tuning, reliability, availability and capacity planning. Also there is a human factor involved. A good Performance Engineer has to have the good breadth and depth of knowledge about the whole architecture and components involved. Performance engineering touches all aspects of the solution involving both the software and the hardware. There can be so many moving parts to a solution. A typical solution could comprise of application servers, frameworks, Operating systems, databases, user interfaces, Virtual Servers. Then there is the hardware factor CPU, Memory, Network and IO. Detecting performance bottlenecks in these complex solutions is an art and needs lot of experience. Is it worth it to spend enormous amounts of money for a load testing tool when it helps with just a 20-30% of the performance engineering aspect. Frankly I do not think it is worth it.

If you have a server gateway which is queue based ( JMS/MQSeries/MS MQ) why do you need to invest in LoadRunner or Silk Performer. Just a small script to pump the messages in the Queue to drive the load will suffice. The good thing about these expensive commercial tools is that they promise to satisfy all your needs (different  protocols, frameworks , OS etc.) and the bad thing is also that they promise to satisfy all your needs. You do not go buy a Ferrari when all you need is a bike!

There is a huge opportunity for open source tools to provide a subsitute for these expensive commercial tools. At present there is no single tool which can act as a substitute for the comercial counterparts, although open source provides you the opportunity to select and use tools as per your needs. One tool is not going to satisfy all your needs.

Here are some of the tools which can be used for Performance Testing and Automation

Performance Testing:

Apache JMeter
Grinder
OpenSTA
TestMaker

Automation:

Selenium
Software Testing Automation Framework (STAF) 
Web App Testing in Ruby (Watir)
JWebUnit

The following is the GUI for the tool I had built for my first assignment.

Nifty Load generating tool

Here is a nice youtube video on OpenSource Performance Testing..

Let us take a real world example. You go to a restaurant and place an order for lunch. What is the total time to execute this order? Let’s look at the sequence of events. The waiter spends some time taking your order and then places it as the last item in a queue of orders. When this order reaches the top of the queue one of the cooks takes this order and cooks the dish and when it is ready the waiter brings it to your table. So the total time for your order is the sum of processing time ( time to take order + cooking time) and the wait time ( time the order was in the queue).

Response time can be defined as the total time taken to perform an action. This total time could include time processing the action ( in the app server, database, client etc.) and the time spent waiting ( network, IO, memory..). In Software Performance Engineering, Response Time can be surmised as sum of processing time and wait time.

Lets take an example from software engineering. In unix there is a utility “time” or “timex” for measuring the elapsed time for a particular process. We will be looking at a granular level, analyzing a particular process within a system and not the response time for a particular action through the whole solution. Let’s say your application process is “myapp”, you can run the time command on that application as below.

>time myapp
4.6 real 0.5 user 0.8 sys

The output shows that the elapsed time ( real time or the response time perceived by the user) for running the “myapp” process is 4.6 seconds. The total CPU time for running this process is 1.3 CPU seconds, out of which 0.5 cpu seconds is spent in user mode and 0.8 cpu second in system mode (kernel). Where is the rest of the time? How do you account for 3.3 seconds.

Time could have been spent performnig I/O operation some of which is attributed to system time, but time spent by disk drives, network interfaces, terminal controller, or other hardware is not accounted for by the “time” command. Time could have been also spent running jobs on behalf of other users ( context switching) or waiting for memory. Many different components contribute to a program’s elapsed time.

Some of the notable elements for the breakup of elapsed time is as follows

1. User CPU Time : The amount of time the CPU spends running user’s program in user state ( executing library function but excluding time spent in kernel).
2. System CPU Time: The amount of time the CPU spends in the system state ( amount of time executing kernel code ).
3. I/O time: The amount of time the I/O Subsystem spends servicing I/O requests by the process.
4. Network time: The amount of time the I/O Subsystem spends servicing the netwrok requests that the job issues.
5. Time spent running other programs: As the load increases, the CPU will spent less time on a particular program and will be switching between programs to give them some slice of time.
6. Virtual Memory: Ideally all the programs remain in the system’s memory, but when the memory is overloaded and not all program’s can exist in the memory, the system starts paging memory to and from the disk therby increasing the time spent servicing memory request.

The bottom line is  Response time = Processing Time + Wait Time.

Oracle has also changed the way they collect and report performance. The impetus is on wait time. Oracle introduced Oracle Wait Interface ( OWI) in 7.0.12 but it has really matured 10g onwards. This interface provides wait time metrics for numerous events. Mostly the response time degrades significantly due to high wait time and if you can reduce the wait time you can really improve the response times. I will cover OWI in a different post. Also there are system tools ( sar, iostat etc.) which give good indication of the wait time at the system level, those will also be covered in a different post.

Refernces: System Performance Tuniong (Mike Loukides, O’Reilly)

I have recently been building a web site based on ROR and I was looking for a plugin to accomplish this. The plugin I used is queso/sitemap from http://github.com/queso/sitemap/tree/master. This link has instructions and also the readme with the plugin gives enough information to install this plugin. The problem I faced was with setting up widgets and the static link in the sitemap_settings link.

Since I am new to Ruby on Rails it took me a while to figure out how to setup the widgets and static link in the sitemap_settings url.  To explain how I had setup the sitemap I am going to use  the model “Term” as an example. Basically I needed Sitemap to display the url of list of terms http://www.mydomain.com/terms and all the individual terms like http://www.mydomain.com/terms/term1, http://www.mydomain.com/terms/term2 etc.

In the Widgets setting screen I added a widget named Term and entered the information as shown in the image below. The model is going to be “Term”. The named route is “terms_path” which resolves to http://www.mydomain.com/terms. Correction: The named route is “terms_url” which resolves to http://www.mydomain.com/terms. Make sure to look up your routes to see if the name route you are providing is relevant and available ( use  >> rake routes command). If you leave the finder method blank, the sitemap plugin will use find(:all) method of the model to get the list of all items in the model. Sitemap plugin is going to generate an XML comprising of the url for list of terms ( http://www.mydomain.com/terms ) and the url of individual terms (http://www.mydomain.com/terms/term1, http://www.mydomain.com/terms/term2 , etc.).

You can create a custom finder method. We have created a find_sitemap class method in the Term model as shown in the code below:

 def self.find_sitemap(*args)
    find(:all,:conditions=>['state=?','published'])
  end

 This method is going to find all terms that are in published state. One issue I ran into was the url being generated by the sitemap for the list was coming out as “/terms” instead of “http://www.mydomain.com/terms”. I had to hack the sitemap code to get this right. I had to change the code in show.xml.builder of the sitemap plugin. The code block is shown below. The following was added in line 3 in the url_for method: “root_url.chop + “. I am not sure if this is the most graceful way but I was kind of in hurry. Pardon my ignorance. There was nothing wrong with the sitemap plugin it expects a named url ( contains the entire url) instead of the path ( contains relative path). The named route in the following image should be “terms_url” instead of “terms_path”.

Sitemap Widget

Designers.find(:all).each(&:save)

This would give an error: Iconv::InvalidEncoding: invalid encoding (”ascii//translit//IGNORE”, “utf-8″)

Turns out code needs to be changed in permalink_fu.rb at line 90.
The order is not correct. Commented out portion is what comes with the plugin. The subsequent statement is the correction.

# PermalinkFu.translation_to = ‘ascii//translit//IGNORE’
PermalinkFu.translation_to = ‘ascii//ignore//translit’

Please refer to this tutorial on using permalink_fu.

Also use this Reference

To show the leak the following command is used:

(gdb) info leaks

To view a particular leak from a list of leaks detected use the following:

(gdb) info leak  <leak number> ( leak number is the relevant number from the leak)

It is very important that program be linked with librt.sl shared library to use heap profiling.

The following example is using xscAppAdapter as a C++ program to demonstrate memory leak detection.

1) Set Heap Options

>>gdb xscAppAdapter.gdb
(gdb) set heap-check leaks on
(gdb) set heap-check free on
(gdb) set heap-check bounds on
(gdb) set heap-check scramble on
(gdb) b xscAppAdapter::processMessage

2) Set Breakpoint and Run Application and continue once past the breakpoint to process a message and then detect a leak

Breakpoint 1 at 0x202e4: file xscAppAdapter.C, line 383.
(gdb) run -s PFAppStation -c xsc_Config.8.0 -t xsc_StateManager -r dflt > /dev/null 2>&1 0</dev/null
Starting program: /perf/app/bin/xscAppAdapter.gdb -s PFAppStation -c xsc_Config.8.0 -t xsc_StateManager -r dflt > /dev/null 2>&1 0</dev/null
Breakpoint 1, xscAppAdapter::processMessage (this=0x4081edf8, msg=@0x407b19b0) at xscAppAdapter.C:383
(gdb) c 1
Will stop next time breakpoint 1 is reached.  Continuing.
Breakpoint 1, xscAppAdapter::processMessage (this=0x4081edf8, msg=@0x407b1a50) at xscAppAdapter.C:383
383     in xscOpedsAdapter.C

3) Find leak using info leaks.

(gdb) info leaks
Scanning for memory leaks...
338 bytes leaked in 13 blocks

No. Total bytes Blocks Address Function
0 88 1 0x40510768 operator new(unsigned long)()
1 64 1 0x405321b8 operator new(unsigned long)()
2 37 1 0x4053f1c8 operator new[](unsigned long)()
3 37 1 0x4053f208 operator new[](unsigned long)()
4 31 1 0x406790f0 strdup()
5 22 3 0x4074d548 operator new[](unsigned long)()
6 20 1 0x405f3078 pthread_atfork()
7 15 1 0x40733810 operator new[](unsigned long)()
8 12 1 0x40536510 operator new(unsigned long)()
9 8 1 0x4040e268 operator new(unsigned long)()
10 4 1 0x405efca8 operator new[](unsigned long)()

4) Lookup Individual leak info.

(gdb) info leak 4