The Long and Winding Road Behind

The following is a point-by-point account of what we have achieved so far during the development of the ØMQ project. The most interesting aspect is how the project gradually evolved from "just another messaging solution" to "layer of the Internet stack".

Getting the messaging stack into order

Messaging systems have their origin in the mid-80’s corporate environment. At that time the existence of a standard and fully functional network stack wasn’t assumed. The design of corporate messaging took this fact into account and implemented almost all of the functionality that is available in the standard Internet stack today.

The relative isolation of corporate environments, combined with the long life cycles of enterprise software, resulted in a legacy that persists to this day. Functionality remains on the messaging level, virtually duplicating almost every feature of the Internet stack. Moreover, backward compatibility issues with legacy enterprise applications require any new messaging system to duplicate the design of the system being replaced.

In our design we took a completely opposite approach and instead of duplicating a traditional heavyweight messaging system we decoupled individual features from the messaging layer, delegating them to the appropriate layer in the Internet stack, leaving ØMQ to implement only the core messaging functionality — so called "messaging patterns".

For example: Many legacy systems provide their own implementation of multiplexed virtual circuits. We delegate multiplexing to the IP layer, where it belongs. This approach eliminates the feared "head of line blocking" problem and it is network-friendly, meaning that existing networking components (routers, switches) can transparently apply sensible congestion control, traffic shaping, and so on.

Different example: We deliberately refrain from structured content. ØMQ messages are opaque binary data, to be contrasted with the rich system of data types provided by legacy messaging systems such as CORBA. Serialisation and de-serialisation of data can be easily implemented on the application layer using many existing libraries.

Yet another example: Flow control is delegated to the TCP layer, where it belongs. In this way we avoid broken models of flow control at the messaging layer as well as duplicate transmit and receive windows.

Fixing broken functionality

Delegating functionality to it’s natural place in the Internet stack led us to compare the algorithms used by messaging systems with the algorithms used by the Internet stack.

We found that the heavily isolated nature of messaging system development prevents the flow of innovation between the Internet community and the world of messaging. For example, such a widely accepted algorithm as fair queueing (first proposed in RFC970 in 1985) has as of 2010 not yet made it across the border.

Simplifying the wire protocol

Enterprise messaging is a world of proprietary protocols, products with closed protocols make up the majority of the market. Several attempts to introduce open protocols to the messaging sphere have been made over the years however, none of these have yet succeeded.

We have participated in the standardisation of the AMQP protocol almost from it’s inception in 2004, and over time we came to realise that the major obstacle to adoption of AMQP is it’s inherent complexity. A complex specification makes implementation a costly prospect and compromises interoperability by the sheer amount of detail that may be misinterpreted by implementers.

To counter this problem, we designed a minimalist wire protocol which can be described in a couple of paragraphs. Moreover, the protocol is designed as any well-behaved Internet protocol should be: It can be layered on top of an arbitrary underlying transport protocol.

Simplifying the API

The complexity of tasks performed by traditional messaging systems is naturally reflected by overgrown and convoluted APIs. Most of them are closed, the only widely accepted open API is called JMS, which is Java-specific and comes with a specification that is 138 pages long.

During the design of ØMQ we initially mimicked widely used messaging APIs. After delegating functionality to the appropriate place in the Internet stack as described above the original API no longer matched the functionality.
To solve this problem we designed a new API inspired by Berkeley sockets. This API is a simple extension of the concepts used by Berkeley sockets, mapping almost 1:1 to BSD socket API, flattening the learning curve almost to zero and leaving the way open towards the possible inclusion of ØMQ into the large family of protocols available via POSIX sockets.

Making it ubiquitous

For ØMQ as a solution which is steadily evolving towards being a layer in the Internet stack, it is important to run everywhere. Our implementation is designed to be portable and currently runs on POSIX-compliant platforms, Microsoft Windows and OpenVMS.

For exactly the same reason it’s important that the implementation runs – and runs well – on any available hardware, and scales across a range of options. On the top end of the range, it runs on large multi-core servers (we have tested scaling on machines with up to 16 CPU cores). On the low end it runs on embedded systems and vintage computers. Further, ØMQ has been designed and tested to scale from unreliable slow networks such as GPRS all the way to high performance data centre infrastructure such as InfiniBand and 10Gb Ethernet.

For ØMQ to be universally useful a key requirement is that it be language-agnostic. Core implementation thus provides the lowest common denominator, a C library. Thanks to the help from community the number of languages that can natively access ØMQ increases at astounding pace.

Providing industrial strength

The ØMQ networking engine is extremely fast. By eliminating unnecessary calls to the underlying network stack, message throughput can exceed the network stack’s performance by a factor of 3-4. For message latency, the engine adds only a couple of microseconds on top of the network stack.

The performance thus achieved is a prerequisite for ØMQ to be suitable for operation on a Internet scale, for example on the core backbone.