zeromq

The primary use case for HTTP is to go through firewalls, proxies, etc without needing to have special configuration. Note that this is generally unidirectional - the machine behind the firewall has to reach out to a machine which is out in the wild.

Because its unidirectional, the subscriber would issue the http session to the publisher, and that session would be relatively long lived. The subscriber would issue one http request to kick things off, and from then on the messages coming from the publisher might be encoded using chunked transfer encoding so that there is no need for any roundtrip between subsequent messages. ie, each message might be encoded as a chunk, that way a whole stream of messages can be sent without the subscriber needing to request them individually. Chunk overhead is small, just a hex number reflecting chunk length and a newline. Chunks are a good unit for messages so that any device in the path will forward the chunk along as it is complete, which is may or may not do with incomplete chunks.

if anything needs to be sent in a backchannel (i dont know 0mq internals), the subscriber could just send as many new request headers as it wanted, completely interspersed with the incoming message "chunks". This is ok because HTTP 1.1 allows pipelining of multiple requests. All network devices which decode HTTP that I know of support this, even though it is virtually never used in practice because many web _servers_ dont support this correctly, so any client that uses it might trip up the server. No problem in this case, you'd have zeromq on both ends.

If the publisher is behind the firewall, you may need a slightly different setup than the current API to allow the publisher to designate the subscriber. Of course, my understanding of the API is only cursory so this may not be the case.

I think this sort of implementation could allow use of HTTP as a transport with virtually no performance impact and I would love to hear any feedback on this idea. And I may be willing to develop or help develop this funcationality.

Another benefit of HTTP is that it supports redirects, but if I explain why thats useful it may start a philosophical argument about why a message-based system doesnt need redirects, and I'd prefer to have that debate live because I think there are some distributed systems cases where it may be useful.

I think mapping 0MQ to HTTP would be useful, but I think long-polling is relatively cumbersome. On the other hand, I think the whole WebSockets thing is a very good fit for 0MQ, and we've actually started using this at work. Our use case was to have a browser/JavaScript subscriber for a 0MQ publisher. I wrote a simple threaded Python proxy that allows web page clients to subscribe to 0MQ publishers. Each stream any client subscribes to gets set up with a thread in the proxy that will distribute incoming messages from that publisher to a queue for all interested clients (each with its own thread; either managed using a Python threading.Queue or inproc 0MQ), the client thread will then send it over the WebSocket. Because 0MQ allows working with arbitrary binary data and it seemed easiest to only deal with JSON on the client side (AFAICT parsing arbitrary binary data in JavaScript would be cumbersome at best), we recode the incoming data to JSON at the proxy. This works very well for our use case, and it could probably cleaned up to a fairly general 0MQ-to-WebSockets proxy.

In any case, I think it would be much more useful to map to WebSockets than trying to force the 0MQ paradigm on plain HTTP.

What about a possibility of tunneling via proxy on 433 port by HTTP CONNECT?
We could get a BSD-socket descriptor (i know that zApi reference forbids it) and use it for issuing HTTP CONNECT request to proxy. After establishing of the tunnel we could use zmq_socket as usual to reach desired server beyond the proxy.