With nearly two decades of experience working in the IT industry in various roles as Enterprise technology consultant, Java Architect, project leader, Engineer, Designer and Developer, Mr. Knutson has gained a wide variety of experience in disciplines including JavaEE, Web Services, Mobile Computing and Enterprise Integration Solutions. Over the course of his career, Mr. Knutson has enjoyed long lasting partnerships with many of the most recognizable names in the Health Care, Financial, Banking, Insurance, Manufacturing, Telecommunications, Utilities, Product Distribution, Industrial and Electronics industries employing industry standard full software life cycle methodologies including the Rational Unified Process (RUP), Agile, SCRUM, and Extreme Programming (XP). Mr. Knutson has also undertaken speaking engagements, training seminars, white paper and book publishing engagements world-wide. As an active Blogger and tweeter, Mr. Knutson has also been inducted in the prestigious DZone.com Most Valuable Blogger (MVB) group and can be followed at http://www.dzone.com/page/mvbs and twitter at http://twitter.com/mickknutson. Able to quickly and effectively adapt to any phase of an existing project from Business Modeling to Production Support, Mr. Knutson’s wide array of technical expertise in OOA / OOD / OOP, Java, J2EE, Internet, Oracle, Enterprise Integration and Message Oriented Middleware allow him to seamlessly integrate into your Information Systems project at any phase and provide immediate productivity. Mr. Knutson is exceptional at team building and motivating either at a peer-to-peer level, or in a leadership role with excellent communications skills and the ability to adapt to all environments and cultures with ease. Mick is a DZone MVB and is not an employee of DZone and has posted 18 posts at DZone. You can read more from them at their website. View Full User Profile

JMeter load testing against Apache Webserver: Errors and Resolutions

01.05.2012
| 7460 views |
  • submit to reddit

have been working on a fairly simple JMeter load script that I can run a series of 4 sequential pages against an Apache server, but the goal was to have the server support 2,000 concurrent requests for 5 minutes without error.

Most of my issues in this exercise have been with JMeter and the client machine used to test the Apache server.

To begin, I must state I was originally configuring Apache with a prefork MPM:

StartServers        100
MinSpareServers      75
MaxSpareServers     100
ServerLimit        2000
MaxClients         2000
MaxRequestsPerChild   0

At approximately line 72 of Jmeter.bat, there are several entries the manage the JVM for running Jmeter.

set HEAP=-Xms512m -Xmx512m
set NEW=-XX:NewSize=128m -XX:MaxNewSize=128m
set SURVIVOR=-XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=50%
set TENURING=-XX:MaxTenuringThreshold=2
set RMIGC=-Dsun.rmi.dgc.client.gcInterval=600000 -Dsun.rmi.dgc.server.gcInterval=600000
set PERM=-XX:PermSize=64m -XX:MaxPermSize=64m

I decided to start with 1,000 concurrent requests for 5 minutes just to see how the test would fair.

With the above settings I started getting OOM errors almost immediately so I decided to increase the HEAP and NEW memory to eliminate the issue and wanted to add more GC settings to increase the JVM’s ability to clean up:

set HEAP=-Xms1024m -Xmx1024m -Xss128k
set NEW=-XX:NewSize=256m -XX:MaxNewSize=256m
set SURVIVOR=-XX:SurvivorRatio=14 -XX:TargetSurvivorRatio=50%
set "TENURING=-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -XX:+DisableExplicitGC -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxTenuringThreshold=4"
set "EVACUATION=-XX:+AggressiveOpts -XX:+UseFastAccessorMethods -XX:+UseCompressedStrings -XX:+OptimizeStringConcat"
set RMIGC=-Dsun.rmi.dgc.client.gcInterval=600000 -Dsun.rmi.dgc.server.gcInterval=600000
set PERM=-XX:PermSize=64m -XX:MaxPermSize=64m

This did resolve the JMeter OOM issues, but now started getting Apache errors. During the ramp-up phase, I started getting connection refused errors:

Response code: Non HTTP response code: org.apache.http.conn.HttpHostConnectException
Response message: Non HTTP response message: Connection to http://pasundtastgprt2:8001 refused

I started looking at the Apache server and noticed that the number of httpd threads was at 1,000 and it appeared that JMeter was running out of memory because the requests where starting to back up.

This is why we load test right!

So I decided to run a worker MQM and recompiled Apache to support the new MPM

ServerLimit           80
StartServers          25
MaxClients          2000
MinSpareThreads       75
MaxSpareThreads      125
ThreadsPerChild        5
MaxRequestsPerChild    0

I started testing this configuration and while monitoring the server running 1,000 concurrent requests and the server looked like Apache was handling 1,000 requests just fine. I was running a simple command to output the sockets and httpd processes on the server during the load test:

while true
do
echo -----`date '+%r'` -----:
netstat -ant | awk '{print $6}' | sort | uniq -c | sort -n
echo httpd processes: [`ps aux | grep httpd | wc -l`]
echo .
sleep 30
done

Then when I was monitoring the load test, I was concerned about seeing 82 httpd processes running which was the ServerLimit I had set.

-----08:02:37 AM -----:
      1 established)
      1 Foreign
      4 CLOSE_WAIT
     17 LISTEN
     32 FIN_WAIT2
     41 ESTABLISHED
     69 FIN_WAIT1
    630 SYN_RECV
  45386 TIME_WAIT
[82] httpd processes
.

I now increased the load to my target of 2,000 concurrent requests and restarted the JMeter test and was able to get to around 1,800 concurrent request and started getting connection refused errors again.

I suspected that my Servers where maxed out and was not able to create anymore threads for those servers where having:

80 server * 5 threads each server == 400 requests processed concurrently

So I increased the number of threads to 25

80 server * 25 threads each server == 2,000 requests processed concurrently

To end up with this worker setting:

ServerLimit           80
StartServers          25
MaxClients          2000
MinSpareThreads       75
MaxSpareThreads      125
ThreadsPerChild       25
MaxRequestsPerChild    0

I was then able to turn the load up to 2,000 concurrent requests.

-----08:53:27 AM -----:
      1 established)
      1 FIN_WAIT2
      1 Foreign
      4 CLOSE_WAIT
     12 CLOSING
     17 LISTEN
    129 ESTABLISHED
    621 FIN_WAIT1
   1203 SYN_RECV
  55556 TIME_WAIT
[53] httpd processes

At this point we are only using 53 Servers and 2,000 clients. The tests sustained zero errors for 5 minutes during the test.

As a test I increased the ThreadsPerChild to 40 and run the load test against 3,000 concurrent requests.

I was able to get to around 2,800 concurrent requests then I started getting connection refused errors:

Error Count: 1
Response code: Non HTTP response code: org.apache.http.conn.HttpHostConnectException
Response message: Non HTTP response message: Connection to http://pasundtastgprt2:8001 refused

and I also started getting JMeter errors:

Response code: Non HTTP response code: java.net.BindException
Response message: Non HTTP response message: Address already in use: connect

So in the furure I would like to see how much further I can push the server and I think I can get to 3,000 concurrent and most likely far more than that on my current installation.

 

From http://www.baselogic.com/blog/development/adding-memory-jmeter/

Published at DZone with permission of Mick Knutson, author and DZone MVB.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Tags:

Comments

Dzmitry Kashlach replied on Fri, 2013/02/08 - 7:01am

I know some methods how to deal with OOM issues of JMeter.

1. Running tests without GUI;

2. Not using heavy Listeners, just Simple Data Writer to save results for further analyzing.

3. Increasing number of load-generators.

Also there is commercial solution, <a href="www.blazemeter.com">Jmeter testing cloud</a>, which has  <a href="http://community.blazemeter.com/knowledgebase/articles/83191-blazemeter-plugin-to-jmeter">good reporting facilities</a>. So, in this case we do not need even to save results locally, we'll be able to view them in real-time mode.  


Sorabh Kalra replied on Sun, 2013/10/13 - 3:30pm

Hi - 

I am facing connection refused issue from Jmeter.

we have 4 Apache FE boxes.

httpd.conf worker settings are below.

<IfModule worker.c>ServerLimit         40StartServers        10MaxClients        1000MinSpareThreads     30MaxSpareThreads    100ThreadsPerChild     25MaxRequestsPerChild  0</IfModule>

However, we started getting connection refused at 250 users around.

We need to support 3000 concurrent users.

Not sure, if this is an issue with Jmeter conf or with Apache conf.

Any help will be highly appreciated.


Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.