Open Source has taken a center stage in the development world. It is a community effort which can not stopped. Either you ride it or get run over by it.

One area where open source has not yet made a big dent is in test automation and performance testing.
This area is still dominated by commercial tools. Most of the commercial tools available for automation and performance testing are extremely expensive. The leading tools from HP (QTP, Loadrunner) and Borland ( Silk Performer, Silk Test) come with a huge price tag.

I used to be a developer before I jumped into Performance Testing back in 1998. That time I was still not able to let go of my development skills. For the first Performance Testing assignment I had built a load testing tool using MS VC++ ( are we not talking of open source here?), the GUI for it is shown below. This was perfect for me. It suited all my needs for driving the load on the server. I had other scripts to automate monitoring, logging and analyzing the results ( see the tool image below).  Well right around then the company I used to work for, decided on standardizing the tool set and we settled on Silk Performer ( then a product of Segue). Using the tool was good for my resume value and it did have lots more bells and whistles.

But I can not say for sure that using this tool instead of my own nifty tool added more value to quality of the product I was testing. Load generation tool contributes to probably 20-30% of the whole performance engineering aspect. There is more to performance engineering then just using a load tesing tool, like benchmarking, tuning, reliability, availability and capacity planning. Also there is a human factor involved. A good Performance Engineer has to have the good breadth and depth of knowledge about the whole architecture and components involved. Performance engineering touches all aspects of the solution involving both the software and the hardware. There can be so many moving parts to a solution. A typical solution could comprise of application servers, frameworks, Operating systems, databases, user interfaces, Virtual Servers. Then there is the hardware factor CPU, Memory, Network and IO. Detecting performance bottlenecks in these complex solutions is an art and needs lot of experience. Is it worth it to spend enormous amounts of money for a load testing tool when it helps with just a 20-30% of the performance engineering aspect. Frankly I do not think it is worth it.

If you have a server gateway which is queue based ( JMS/MQSeries/MS MQ) why do you need to invest in LoadRunner or Silk Performer. Just a small script to pump the messages in the Queue to drive the load will suffice. The good thing about these expensive commercial tools is that they promise to satisfy all your needs (different  protocols, frameworks , OS etc.) and the bad thing is also that they promise to satisfy all your needs. You do not go buy a Ferrari when all you need is a bike!

There is a huge opportunity for open source tools to provide a subsitute for these expensive commercial tools. At present there is no single tool which can act as a substitute for the comercial counterparts, although open source provides you the opportunity to select and use tools as per your needs. One tool is not going to satisfy all your needs.

Here are some of the tools which can be used for Performance Testing and Automation

Performance Testing:

Apache JMeter
Grinder
OpenSTA
TestMaker

Automation:

Selenium
Software Testing Automation Framework (STAF) 
Web App Testing in Ruby (Watir)
JWebUnit

The following is the GUI for the tool I had built for my first assignment.

Here is a nice youtube video on OpenSource Performance Testing..

Tags: automation, grinder, jmeter, junit, jWebUnit, linkedin, loadrunner, open source, opensta, performance, pushtotest, qtp, selenium, silk performer, silk test, STAF, testing, testmaker, watir

This entry was posted on Friday, March 20th, 2009 at 2:57 pm and is filed under Open Source technologies, Perfromance Engineering. You can follow any responses to this entry through the RSS 2.0 feed. You can skip to the end and leave a response. Pinging is currently not allowed.