Load Testing Tools: JMeter vs. The Grinder

This is an attempt at reviewing load testing tools in an evaluation format (JMeter vs. The Grinder). At my organization we have the following use requirements for a load testing tool.

1. Load test an application (e.g. HTTP) with approx. 2000 concurrent users with approx. 500 transactions/sec.
   
2. Parameterize URL for session id (e.g. Modifying URL parameters in the request based on previous test data)
3. Measure performance of logical page transactions/sets of requests. This could count an entire page load as a single transaction. The entire page load may have html, javascript, css, and images but the page should be regarded as a single unit.
   

For the two tools I've researched and used I will focus on the following:

1. Usability -- How usable the tool is (e.g. Functional GUI, fulfills goals specified above). It is important to reflect upon the target audience when looking at usability. Is it functional for a tester, QA lead, or developer?
   
2. Extensibility -- How extensible the tool is (e.g. Does it offer scripting support?).
   
3. Experience -- My experience with the tool and any comments while trying to offer an unbiased point of view.
   

For this article I am using JMeter 2.3.1 and The Grinder 3.0.1.

Note: By any means I am not an expert in either tool. This is simply a limited review of both tools. Please correct me if I am wrong in my statements.

JMeter
JMeter is a load testing tool for functional behavior and measuring performance. A user interface is provided to record, create, and run test scenarios.

Usability
The interface tool is very easy to use. However, there are issues with the included components for creating a test to be complicated. The tool was initially easy to figure out. I believe this is a product of it having a decent tutorial. Setting up a proxy is easy enough, but isn't enabled out of the box. It is very usable from a non technical perspective. I could trust that if the tool was given to a QA team the documentation would be okay to start performance/load testing.

Extensibility
There are many provided components when creating your test to do many things. It provides components for standard tasks. Many of the components aren't easily understood to work as you would expect them to.

Experience
In my limited experience with JMeter it is not viable for complex solutions to load testing. The documentation is plentiful, but not very helpful. It is extensible up to a point where the underlying use of the tool reaches a point. Given the above requirements JMeter isn't extensible enough. If your needs for a load testing tool involve less customizing then it should work just fine.

The Grinder
The Grinder is a tool designed to work in a distributed agent environment for load testing an application from many machines. It allows for easy scripting and customization of test scripts using Jython. It is very easy to load test and define logical transactions by redefining the way your data is recorded so it can then be analyzed.

Usability
The Grinder is seemingly aimed at the developer. It provides a usable interface that provides various metrics for collecting samples and looking at result data. The metrics that are given aren't too advanced, but it provides facilities for data to be exported into a higher quality analysis tool.

Extensibility
This is an area where The Grinder excels. It is very easy to customize the test scripts in Jython. This even includes importing other Jython scripts for other types of addons (e.g. for randomizing logins for multiple users. See below.)

Experience
With limited experience The Grinder has been an easy tool to work with. A strength is that it is very extensible and scripts can be written free form as said above. There are scripts out there for various functionality. When implementing login for multiple users it was very easy to adapt to the existing recorded/modified script that was written. If using Jython is a problem then the tool is probably not the best choice.

Conclusion
After looking at both applications I choose The Grinder as the tool to go with given the requirements for a load testing tool for my team. If your requirements are different and don't require customization then JMeter would be just as good.

Note: The Grinder has been in use and has so far been working great.

References

1. http://blackanvil.blogspot.com/2006/06/shootout-load-runner-vs-grinder-vs.html


2. http://blackanvil.blogspot.com/2006/11/grinder-addressing-warts.html