Posts Tagged ‘memory leak’

What if the load test tool itself can not handle load?

Tuesday, September 22nd, 2009

You use load test tools to break application software you are testing. What if the tool itself breaks and the applications under test do not even have a dent?

This is exactly what happened when using JMS protocol in HPPC ( Loadrunner).  There is a memory leak in HPPC when using JMS. The mmdrv.exe process invokes JVM (in-process) during use of one of the jms functions (jms_send_message_queue).

During one of our tests we started getting out of memory error. Of course if you increase the JVM heap ( using -Xmx ) the error will be delayed but the fact remains it is leaking memory. It was very embarrassing to tell the application team that our tool is breaking during load testing.

We have been trying desperately to get HP to admit and fix it but they are making us go around in circle. We  used JConsole to monitor the JVM and prove to them that there is a leak but they still do not get it. The only thing that is left is for us to now fix the code for them.

In order to use JConsole with loadrunner JMS, you need to add the following in the -Dcom.sun.management.jmxremote in the “Additional VM Parameters” in order for JConsole to connect to the JVM.  Once you do that you can see the JVM memory keep on climbing inspite of you manually invoking GC.

Are you listening HP?

 JMS Advanced Settings

One accurate measurement is worth a thousand expert opinions

Monday, April 6th, 2009

“One accurate measurement is worth a thousand expert opinions” – Adm Grace Murray Hopper ( Dec 9 1906 to Jan 1 1992)

Measurement is the most important aspect of Performance Engineering. There is no place for guess work in performance engineering. Measuring tools are the most important tools in a Performance Engineer’s tool set.

In one of the projects I was involved in, the second day of my job I was validating the solution for a high throughput. Everything was going fine till the performance test started throwing lot of errors. The solution involved around 15 servers. I had set up monitoring of resources on all the 15 servers. First thing I did was look up the resource utilization of all the servers. There was one server where the memory utilization kept on growing. I narrowed down to the process which was growing in memory. The graph of the memory utilization for the process showed a linear growth, it went up as high as 1GB and then the process terminated. This graph was proof enough to show the developer that the process had a big memory leak. This was fixed within a day and solution was ready for further testing.

Performance engineering requires lot of discipline and a methodical approach. Lot of times when we come across problems we tend to start giving expert opinions, start guessing where the problem might be or start looking at the code. One needs to take a scientific approach. One has to look at the facts, start with resource utilization and then narrow down to looking at logs, timestamps, database metrics, application server metrics, compare with historical benchmarks etc.

Check this out. The very first computer bug.

Using gdb to find memory leaks in HP Unix

Sunday, March 1st, 2009

The following gdb commands are used to setup memory leak detection in C++ programs:
set heap-check leaks on
set heap-check free on
set heap-check bounds on
set heap-check scramble on

To show the leak the following command is used:

(gdb) info leaks

To view a particular leak from a list of leaks detected use the following:

(gdb) info leak  <leak number> ( leak number is the relevant number from the leak)

It is very important that program be linked with librt.sl shared library to use heap profiling.

The following example is using xscAppAdapter as a C++ program to demonstrate memory leak detection.

(more…)