Testing

To test

I recommend the ApacheBench (ab) program that is included with all recent versions of Apache. To test, try: 

        ab -k -n 10000 -c 10 http://localhost:8080/index.html

or for bandwidth testing: 

        ab -k -n 10000 -c 10 http://localhost:8080/medium.bin

Note that as the tests are on static files the system can saturate most network connections. I had to test on the local loopback which severely influences the results.

Example results

Using ab Version 2.0.40-dev on my iBook/800MHz with a pre-release of the jdk (build 1.4.1-beta-20) the basic results for the bandwidth test is: 

      
Time per request: 1.102 [ms] (mean, across all concurrent requests)
Transfer rate:    29127.05 [Kbytes/sec] received

The output of top while during the test is similar to: 

      
 Processes:  63 total, 3 running, 60 sleeping... 147 threads
 CPU usage:  35.0% user, 65.0% sys, 0.0% idle
 PID COMMAND      %CPU   TIME   #TH #PRTS #MREGS RPRVT  RSHRD  RSIZE
 6681 java        48.0%  0:35.72   9   236   109  26.7M   712K  14.7M
 6696 ab          29.4%  0:12.90   1    11    28  1.58M+  612K  2.00M+
    0 kernel_tas  13.4% 15:23.10  27     0     -      -      -  33.4M+

Perfcap results

Perfcap (juju.reattore.perfcap.Perfcap) is an automated performance tester that uses ApacheBench for the actual testing.

The row index is the number of concurrent connections. The Tomcat 3.3.1 test is unfair as ApacheBench and Tomcat don't support the same type of keepalive connections.

Transfer rate (kb/s)

Higher numbers are better

Reattore Reattore w/out cache Boa 0.94.13 Tomcat 3.3.1
1 3982.14 1878.27 2946.82 562.92
2 4237.72 2073.16 3420.25 637.07
3 4234.14 2107.32 3515.15 646.68
4 4183.6 2178.06 3453.75 640.09
5 4371.17 2140.58 3550.64 641.78
10 4022.33 2147.6 3686.71 659.1
15 4037.03 2093.94 3665.51 650.01
20 3972.97 2063.57 3619.83 668.32
25 3781.21 1982.24 3586.7 639.32
30 3804.17 1919.5 3472.88 679.09
40 3465.17 1781.18 3406.35 684.99
50 3234.28 1742.22 3278.71 682.46
60 2984.39 1673.75 3098.76 679.53
70 2742.83 1630.17 2954.97 688.37
80 2668.72 1578.76 2832.52 674.25

Requests per second

Higher numbers are better

Reattore Reattore w/out cache Boa 0.94.13 Tomcat 3.3.1
1 2256.89 1064.53 1557.41 307.43
2 2401.76 1175.0 1807.62 347.94
3 2399.74 1194.34 1857.77 353.19
4 2371.07 1234.43 1825.32 349.6
5 2477.4 1213.19 1876.52 350.31
10 2279.69 1217.17 1948.43 359.96
15 2288.01 1186.76 1937.23 354.32
20 2251.7 1169.54 1913.09 364.2
25 2143.03 1123.46 1895.57 348.26
30 2156.03 1087.89 1835.44 370.9
40 1963.88 1009.49 1800.23 372.2
50 1833.04 987.35 1732.77 372.73
60 1691.39 948.62 1637.68 368.64
70 1554.51 923.88 1561.69 373.34
80 1512.49 894.78 1496.93 368.19

Time per request (ms)

Lower numbers are better

Reattore Reattore w/out cache Boa 0.94.13 Tomcat 3.3.1
1 0.443 0.939 0.642 3.253
2 0.416 0.851 0.553 2.874
3 0.417 0.837 0.538 2.831
4 0.422 0.81 0.548 2.86
5 0.404 0.824 0.533 2.855
10 0.439 0.822 0.513 2.778
15 0.437 0.843 0.516 2.822
20 0.444 0.855 0.523 2.746
25 0.467 0.89 0.528 2.871
30 0.464 0.919 0.545 2.696
40 0.509 0.991 0.555 2.687
50 0.546 1.013 0.577 2.683
60 0.591 1.054 0.611 2.713
70 0.643 1.082 0.64 2.678
80 0.661 1.118 0.668 2.716