WARNING: This text is deprecated and refers to an old version of ØMQ. It remains here for historical interest. DO NOT USE THIS TO LEARN ØMQ.

HISTORICAL WHITEPAPER

Introduction

ØMQ/0.4 doesn't add much to ØMQ/0.3 in terms of design. The most visible thing in 0.4 release is support for additional OS platforms. Most importantly, Windows port allows you to use ØMQ to power your client GUI applications running on Windows PCs. Aside of Win32, QNX Neutrino, AIX, HP-UX and OpenBSD are now supported.

The other change worth of notice is solution for C10K problem - how to efficiently handle thousands of concurrent connections.

We have already discussed some of the concepts and algorithms in design documents for previous versions of ØMQ:

• The ØMQ message wire format;
• The analysis of how to pass messages efficiently between threads;
• The analysis of how to achieve high message throughput over the network;
• How to get flexible handling of many concurrent message flows with possibly distinct wire formats and polling mechanisms while preserving optimal scaling on multicore systems is discussed;
• How AMQP-style exchange/queue semantics are implemented in ØMQ;
• Discussion of ØMQ directory service - how objects are located on the network;
• Discussion of how congestion control is handled in ØMQ.

Platform independence

ØMQ supported several platforms in version 0.3 already (Linux, Solaris, FreeBSD, Mac OS X), however, as Windows is not a POSIX platform, more work was required to make ØMQ work on top of it. Currently, following components have different implementations on different platforms:

• Thread management - either pthreads or Win32 threads;
• Mutexes - pthread mutexes vs. Win32 critical sections;
• Simple semaphores - pthread semaphore, optimised semaphore on the basis of pthread mutex or Win32 event;
• Sockets - BSD sockets in all cases, however, individual platforms tend to require little tweaks to make the code to work correctly;
• Error handling - POSIX errno vs. Win32 GetLastError;
• Resolving network interface names - ioctl vs. getifaddrs.

However, the most complex topic is the build system. System of building and distribution is completely different on Unix-based and Windows platforms. Currently we are providing GNU autotools package for Unix-based platforms (configure/make) and binaries for Windows. For developers (as opposed to users) the source can be downloaded from Subversion and built using autotools (Unix-based platforms) or Microsoft Visual C++ Express 2008 on Windows.

Still, there are two problems with the approach:

• The build on Windows is tied to particular version of development GUI;
• There are two build systems in the project which may result in maintenance problems.

At the moment we are investigating two different ways to handle this problem:

• Generic build system based on CMake;
• Building on Windows using MinGW and MSYS.

C10K

Handling large amount (thousands or tens of thousands) of concurrent connections is a performance problem. Different operating systems handle the issue in different ways. Say, Linux introduces epoll mechanism, Solaris uses /dev/poll mechanism, BSD-based systems have kqueue mechanisms whereas Windows provides I/O completion events mechanism. Moreover, single operating system may provide several of these mechanisms. So say Linux provides POSIX select and poll functions plus Linux-specific epoll mechanism:

To handle this variety, ØMQ defines separate plug-in for each of these polling mechanisms (select_thread_t, poll_thread_t, epoll_thread_t, devpoll_thread_t, kqueue_thread_t). That way user can create several I/O threads at the same time each using different polling mechanism. However, as user would in most cases want to use the best polling mechanism available on the given platform rather than to explicitly specify it, ØMQ defines a typedef io_thread_t that resolves to the best available mechanism on the given OS (actual implementation of io_thread_t on a given platform is marked by asterisk (^*) in the table above.

There's one more reason for making polling strategy a plug-in into ØMQ. In the future we want to allow for different non-standard networking stacks, often with kernel by-pass and specific mechanisms to poll for network events. This way we are able to implement new polling strategy as a plug-in component without breaking ØMQ core functionality.

Conclusion

ØMQ/0.4 delivers enlarged set of supported operating systems (Linux, Windows, Solaris, QNX Neutrino, FreeBSD, OpenBSD, AIX and HP-UX) and thus an efficient high-performance low-latency mechanism for communication between these platforms in a heterogenous environment.

Comments: 0