10GbE Tests

Introduction

This test presents performance result of ØMQ/0.3.1 on 10Gb Ethernet.

Environment

Box 1:

8-core AMD Opteron 8356, 2.3GHz
Mellanox ConnectX MT25408 in 10GbE mode
Linux/Debian 4.0 (kernel version 2.6.24.7)
ØMQ version 0.3.1

Box 2:

8-core Intel Xeon E5440, 2.83GHz
Mellanox ConnectX MT25408 in 10GbE mode
Linux/Debian 4.0 (kernel version 2.6.24.7)
ØMQ version 0.3.1

Boxes were connected by non-switched 10GbE network.

Results

All the tests were run for message sizes of 1, 2, 4, 8, 16, 32, …, 32768 and 65536 bytes.

Latency

End-to-end latency is 28.45 us for 1 byte messages and remains stable up to messages 512B long (33.08 us). Latency is quite obviously higher for larger messages:

10Glatency.png

Throughput

Throughput gets to the maximum of 2.8 million messages per second for messages 8 bytes long:

10Gthr_mms.png

As for bandwidth usage we are able to get to 1Gb/sec boundary for messages 128 bytes long. 2Gb/sec boundary is passed for messages 1024 bytes long and bandwidth of 2.5Gb/s is reached for messages 32kB long:

10Gthr_gb.png

Conclusion

Latency for small messages is as low as 28.45 us which we consider to be quite decent. For single stream of messages of up to 32kB we've been able to use bandwidth ~2.5Gb/s.