Effect of copying on latency

Test

The goal of this test was to show, how copying the message data (on the application level) affects the latency. One test was standard ØMQ latency test (local_lat and remote_lat) the other one was modified version of the test that copied the message data once on each peer.

The test results were provided by Michael Santy.

Environment

  • ØMQ/2.0 beta 2
  • CPU: Intel Core i7 940 @ 2.93 GHz
  • Memory: 12GB DDR3 @ 1333 MHz
  • IB HCA: Mellanox MT25204 — InfiniHost III Lx HCA
  • IB Switch: Mellanox MT47396 Infiniscale III
  • OS: Centos 5.4 x86_64
  • OFED: 1.4.1
  • libSDP: 1.1.99

Results

message size latency with copying latency w/o copying
4 kB 45.856 us 45.471 us
8 kB 55.690 us 57.462 us
16 kB 81.309 us 79.635 us
32 kB 99.423 us 97.009 us
64 kB 164.521 us 152.227 us
128 kB 227.577 us 192.544 us
256 kB 371.325 us 307.026 us
512 kB 682.949 us 528.641 us
1 MB 1464.738 us 1125.205 us
2 MB 2697.648 us 2085.429 us
4 MB 5045.856 us 3714.495 us
8 MB 9662.857 us 7199.121 us
16 MB 20898.646 us 13861.691 us
32 MB 44297.914 us 27565.414 us
64 MB 87754.242 us 54715.375 us
128 MB 167838.859 us 108787.906 us
256 MB 327360.406 us 216642.000 us

Here are the results of the test in the form of graph. Red line represents latency with copying, while the black line represents latency without copying:

copying1.png

The same results displayed in log/log graph to make the smaller message sizes more readable:

copying2.png