The previous release of RabbitMQ (2.7.0) brought with it a better way of managing plugins, one-stop URI connecting by clients, thread-safe consumers in the Java client, and a number of performance improvements and bug-fixes. The latest release (2.7.1) is essentially a bug-fix release; though it also makes RabbitMQ compatible with Erlang R15B and enhances some of the management interface. The previous release didn’t get a blog post, so I’ve combined both releases in this one.  (These are my own personal remarks and are NOT binding; errors of commission or omission are entirely my own – Steve Powell.)

We were wondering how to present SockJS and its possibilities to a wider audience. Having a working demo is worth much more than explaining dry theory, but what can you present if you are just a boring technologist, with no design skills whatsoever?

With questions like that it’s always good to open a history book and review previous generation of computer geeks with no artistic skills. What were they doing? On consoles with green letters they were playing geeky computer games, MUDs (Multi User Dungeons) were especially popular.

Hey, we can do that!

Since the new persister arrived in RabbitMQ 2.0.0 (yes, it’s not so new anymore), Rabbit has had a relatively good story to tell about coping with queues that grow and grow and grow and reach sizes that preclude them from being able to be held in RAM. Rabbit starts writing out messages to disk fairly early on, and continues to do so at a gentle rate so that by the time RAM gets really tight, we’ve done most of the hard work already and thus avoid sudden bursts of writes. Provided your message rates aren’t too high or too bursty, this should all happen without any real impact on any connected clients.

Some recent discussion with a client made us return to what we’d thought was a fairly solved problem and has prompted us to make some changes.

In RabbitMQ 2.6.0 we introduced Highly Available queues. These necessitated a new extension to AMQP, and a fair amount of documentation, but to date, little has been written on how they work.

There’s a lot of hot stuff happening in the web technology lately. JavaScript seems to be bearing the torch, both browser-side and server-side. At the RabbitMQ HQ we’re interested in developments in the wide world of messaging, and we’re particularly excited about the JavaScript angle on messaging - namely WebSockets and related technologies.

I was asked to do a short presentation during the PubSubHuddle meetup. The talk was about current development of WebSockets, its issues and building web applications using them.

One of the problems we face at the RabbitMQ HQ is that whilst we may know lots about how the broker works, we don’t tend to have a large pool of experience of designing applications that use RabbitMQ and which need to work reliably, unattended, for long periods of time. We spend a lot of time answering questions on the mailing list, and we do consultancy work here and there, but in some cases it’s as a result of being contacted by users building applications that we’re really made to think about long-term behaviour of RabbitMQ. Recently, we’ve been prompted to think long and hard about the basic performance of queues, and this has lead to some realisations about provisioning Rabbits.

All of a sudden there’s just one week to go until the PubSub huddle. It’s a one day conference, in London, about messaging. Not just RabbitMQ, but ZeroMQ, MQTT, XMPP and PuSH.

WebSocket technology is catching up, but it will take a while before all browsers support it. In the meantime there are loads of projects that aim to substitute for WebSockets and enable ‘realtime’ capabilities for web apps. But all attempts solve only a part of the general problem, and there isn’t any single solution that works, is scalable and doesn’t require special deployment tricks.

While the firehose is quite a cool feature, I always thought that it was a shame we didn’t have a simple GUI to go on top and make it accessible to system administrators. So I wrote one. You can download it here.

We are very pleased to announce the availability in beta of RabbitMQ as a Heroku add-on. With our RabbitMQ service on CloudFoundry, this extends our commitment to supporting the community of cloud application developers.

We believe that cloud messaging is fundamental in two senses. First as a core capability to build applications that scale to cloud use cases as explained in our blog post launching RabbitMQ on CloudFoundry. And second, because messaging can be extended to solve common problems like integration and data push. For example: to connect traditional on-premise applications with virtualized and cloud deployments.

The idea of ‘realtime web’ or messaging using web browsers has been around for quite some time. First it was called ’long-polling’, then ‘Comet’, the latest incarnation is named ‘WebSockets’. Without doubt it’s going in a good direction, WebSockets is a neat technology.

But during the fight for realtime capabilities we’ve lost focus on what is really important how to actually use messaging. In the web context everything is request-response driven and marrying a typical web stack to asynchronous messaging isn’t easy.

Recently we launched a RabbitMQ service for Cloud Foundry, making it simple to spin up a message broker to use with your apps on Cloud Foundry. There are tutorials online for using it with Ruby on Rails and with Java apps using Spring. Here we are going to look at using the RabbitMQ service with Node.JS apps.

I fundamentally disagree with the APIs exposed by our current AMQP client libraries.

There is a reason why they’re imperfect: we intentionally avoided innovation in APIs since the beginning. The purpose of our client libraries is to expose generic AMQP, not any one view of messaging. But, in my opinion, trying to map AMQP directly to client libraries APIs is just wrong and results in over-complication and abstractions hard to use.

There is no common ground: the client libraries blindly following AMQP model will be complex; easy to use client libraries must to be opinionated.

Recently I saw a tweet saying “ZeroMQ Erlangizes everything!” or some such. While I realise that not everything posted on the web is meant seriously, it does seem there is a stream of similar claims lately that ought to be dammed.

In the article Multi-threading Magic¹, Pieter Hintjens and Martin Sustrik persuasively explain why concurrency is better served by message-passing than by locks and shared memory. And they are fair, I think, in their analysis – except for the insinuation that using ZeroMQ transforms your chosen programming language into a domestic Erlang.

Note: this blog post talks about the federation plugin preview that was released for RabbitMQ 2.5.0. If you’re using 2.6.0 or later, federation is part of the main release; get it the same way you would any other plugin.

Another day, another new plugin release 😃 Today it’s federation. If you want to skip this post and just download the plugin, go here. The detailed instructions are here.

The high level goal of federation is to scale out publish / subscribe messaging across WANs and administrative domains.

To do this we introduce the concept of the federation exchange. A federation exchange acts like a normal exchange of a given type (it can emulate the routing logic of any installed exchange type), but also knows how to connect to upstream exchanges (which might in turn themselves be federation exchanges).

The RabbitMQ team is delighted to announce the release of RabbitMQ 2.5.0.

Most of us at RabbitMQ HQ have spend time working in a number of functional languages in addition to Erlang, such as Haskell, Scheme, Lisp, OCaml or others. Whilst there is lots to like about Erlang, such as its VM/Emulator, there are inevitably features that we all miss from other languages. In my case, having spent a couple of years working in Haskell before returning to the RabbitMQ fold, all sorts of features are “missing”, such as laziness, type classes, additional infix operators, the ability to specify precedence of functions, fewer parenthesis, partial application, more consistent standard libraries and do-notation. That’s a fair list, and it’ll take me a while to get around to implementing them all in Erlang, but here are two for starters.

In our previous blog post we talked about a few approaches to topic routing optimization and described the two more important of these in brief. In this post, we will talk about a few things we tried when implementing the DFA, as well as some performance benchmarking we have done on the trie and the DFA.

RabbitMQ 2.4.0 introduced an extension that allows publishers to specify multiple routing keys in the CC and BCC message headers. The BCC header is removed from the message prior to delivery. Direct and topic exchanges are the only standard exchange types that make use of routing keys, therefore the routing logic of this feature only works with these exchange types.

I decided to run some benchmarks of my AMQP encoder/decoder (AMQ Protocol gem) against the old one in the AMQP gem to see whether it performs better or not. So far I did only the most basic optimisations like storing reusable values in constants, nothing special (yet).

I did two sets of benchmarks: CPU time benchmarking using my fork of RBench with support for custom formatters (like writing results into a YAML file) and memory benchmarking using Object.count_objects (Ruby 1.9).

In many messaging scenarios, you must not lose messages.  Since AMQP gives few guarantees regarding message persistence/handling, the traditional way to do this is with transactions, which can be unacceptably slow.  To remedy this problem, we introduce an extension to AMQP in the form of Lightweight Publisher Confirms.

RabbitMQ 2.3.1 introduces a couple of new plugin mechanisms, allowing you much more control over how users authenticate themselves against Rabbit, and how we determine what they are authorised to do. There are three questions of concern here:

1. How does the client prove its identity over the wire?
2. Where do users and authentication information (e.g. password hashes) live?
3. Where does permission information live?

Question 1 is answered in the case of AMQP by SASL - a simple protocol for pluggable authentication mechanisms that is embedded within AMQP (and various other protocols). SASL lets a client and a server negotiate and use an authentication mechanism, without the “outer” protocol having to know any of the details about how authentication works.

SASL offers a number of “mechanisms”. Since the beginning, RabbitMQ has supported the PLAIN mechanism, which basically consists of sending a username and password over the wire in plaintext (of course possibly the whole connection might be protected by SSL). It’s also supported the variant AMQPLAIN mechanism (which is conceptually identical to PLAIN but slightly easier to implement if you have an AMQP codec lying around). RabbitMQ 2.3.1 adds a plugin system allowing you to add or configure more mechanisms, and we’ve written an example plugin which implements the SASL EXTERNAL mechanism.

From time to time, on our mailing list and elsewhere, the idea comes up of using a different backing store within RabbitMQ. The backing store is the bit that’s responsible for writing messages to disk (a message can be written to disk for a number of reasons) and it’s a fairly frequent suggestion to see what RabbitMQ would look like if its own backing store was replaced with another storage system.

Such a change would permit functionality that is not currently possible, for example out-of-band queue browsing, or distributed storage, but there is a fundamental difference in the nature of data storage and access patterns between a message broker such as RabbitMQ and a generic database. Indeed RabbitMQ deliberately does not store messages in such a database.

I’m happy to announce that the AMQP 0.7 is released, as I promised in the previous blog post. So what are the changes?

In the past year development of the AMQP gem was practicaly stagnating, as its original author Aman Gupta (@tmm1) was busy. A lot of bugs stayed unresolved, the code was getting old and out-dated and no new features or documentation were made.

At this point I started to talk with the RabbitMQ guys about possible collaboration on this. Actually originally I contacted VMware when I saw Ezra Zygmuntowicz looking for people to his cloud team, but when I found that VMware recently acquired the RabbitMQ project in London, I got interested. I signed the contract, switched from script/console to Wireshark and the RabbitMQ Tracer and since November I’ve been happily hacking on the AMQP and AMQ-Protocol gems.