Stress-testing httperf, siege, apache benchmark, and pronk

[lionet]

Pun intended. Today I am doing totally unscientific (but quite useful) comparison between a few well-known HTTP performance tools.

System: FreeBSD 8.2, x86_84 (L3426, no HT, no TB). Both client and servers were on the same host.

Load generators tested:

• httperf (0.9.0)
  
• siege (2.70)
  
• Apache Benchmark, ab (2.3)
  
• Pronk (git master; ghc 7.0.3)

Server

Comments

[levgem]

It is very strange that Node.js ignores Keep-Alive header.

It uses http_parser.c that properly tells whether to use or not to use keepalive.

[lionet]

See for yourself:

[vlm@nala:~]>telnet localhost 8081
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
GET / HTTP/1.0
Connection: keep-alive
Host: localhost:8081

HTTP/1.1 200 OK
Connection: close

SuccessConnection closed by foreign host.
[vlm@nala:~]>telnet localhost 8081
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
GET / HTTP/1.1
Host: localhost:8081

HTTP/1.1 200 OK
Connection: keep-alive
Transfer-Encoding: chunked

7
Success
0

^]
telnet> Connection closed.
[vlm@nala:~]>

[levgem]

Add, please, command line for launching httperf

[lionet]

Many connections, non-keepalive:
httperf --uri / --server localhost --port 8081 --num-conns=500000 --rate 7200
Single connection, keep-alive:
httperf --uri / --server localhost --port 8081 --num-calls=500000

[_slw]

к сожалению все потроганные мной генераторы запросов довольно неэффективны (по CPU) и малофункциональны.
ну или переусложнены настолько, что даже трогать не хочется.

[lionet]

Для начала расскажи, что тебе от них нужно.

[_slw]

на мало CPU много запросов/трафика.
т.е. хочется И много запросов (мелкого файла) И получить много гигабит.

т.е. мне надо из писюка с несколькими гигабитами сделать генератор трафика (для нагрузки внешней сетевой железки).

как заставить FreeBSD пакеты с одной сетевой отправлять на другую -- я знаю (а вот линух нормально это не заставить делать, похоже), nginx достаточно роизводителен, а вот httperf похоже существенно больше ресурсов хочет. а ядра не резиновые. а хочется гигабит 6-8 в результате получить.

[lionet]

Ну вот, наконец, появилось: https://github.com/machinezone/tcpkali

[thesz]

Where's Haskell in here?

[lionet]

Pronk

[synergy_ek]

What about Tsung?
And why server and client was on the same machine?

[lionet]

> What about Tsung?

I did not test Tsung in this experiment, but I know that speed is not its core strength. Being distributed and having more elaborate testing scenarios is. This is a different kind of tool, useful in its own right.

> And why server and client was on the same machine?

Because the tests were conducted on a single machine. Despite this not being the best practice, I think in the first approximation the results of this experiment are going to be useful. Please don't pay attention to the absolute numbers, they are irrelevant. More relevant pieces of data revealed by this test are:

httperf's outdated reliance on select() backfires in non-keepalive setting when operating against GC-based runtime;

ab and node.js interoperability problem noticed;

discovered pronk performance problem when concurrency setting is small;

Node.JS is flaky [on FreeBSD

[alxzhr]

You may want to check Siege (http://www.joedog.org/index/siege-manual).

[lionet]

Checked, added to the table, thanks!