Archive for year 2011December 20, 2011
The previous release of RabbitMQ (2.7.0) brought with it a better way of managing plugins, one-stop URI connecting by clients, thread-safe consumers in the Java client, and a number of performance improvements and bug-fixes. The latest release (2.7.1) is essentially a bug-fix release; though it also makes RabbitMQ compatible with Erlang R15B and enhances some of the management interface. The previous release didn’t get a blog post, so I’ve combined both releases in this one. (These are my own personal remarks and are NOT binding; errors of commission or omission are entirely my own – Steve Powell.)
November 30, 2011
We were wondering how to present SockJS and its possibilities to a
wider audience. Having a working demo is worth much more than
explaining dry theory,
but what can you present if you are just a boring technologist, with
no design skills whatsoever?
With questions like that it’s always good to open a history book
and review previous generation of computer geeks with no artistic
skills. What were they doing? On consoles with green letters
they were playing geeky computer games,
MUDs (Multi User Dungeons) were
Hey, we can do that!
October 27, 2011
Since the new persister arrived in RabbitMQ 2.0.0 (yes, it’s not so
new anymore), Rabbit has had a relatively good story to tell about
coping with queues that grow and grow and grow and reach sizes that
preclude them from being able to be held in RAM. Rabbit starts writing
out messages to disk fairly early on, and continues to do so at a
gentle rate so that by the time RAM gets really tight, we’ve done most
of the hard work already and thus avoid sudden bursts of
writes. Provided your message rates aren’t too high or too bursty,
this should all happen without any real impact on any connected
Some recent discussion with a client made us return to what we’d
thought was a fairly solved problem and has prompted us to make some
October 25, 2011
we introduced Highly Available
queues. These necessitated a
to AMQP, and a fair amount of
documentation, but to date, little has
been written on how they work.
October 19, 2011
There’s a lot of hot stuff happening in the web technology lately.
browser-side and server-side.
At the RabbitMQ HQ we’re interested in developments in the wide world of
messaging - namely WebSockets and related technologies.
September 26, 2011
I was asked to do a short presentation during the
PubSubHuddle meetup. The talk was
about current development of WebSockets, its issues and building web
applications using them.
September 24, 2011
One of the problems we face at the RabbitMQ HQ is that whilst we may
know lots about how the broker works, we don’t tend to have a large
pool of experience of designing applications that use RabbitMQ and
which need to work reliably, unattended, for long periods of time. We
spend a lot of time answering questions on the mailing list, and we do
consultancy work here and there, but in some cases it’s as a result of
being contacted by users building applications that we’re really made
to think about long-term behaviour of RabbitMQ. Recently, we’ve been
prompted to think long and hard about the basic performance of queues,
and this has lead to some realisations about provisioning Rabbits.
September 16, 2011
All of a sudden there’s just one week to go until the PubSub huddle. It’s a one day conference, in London, about messaging. Not just RabbitMQ, but ZeroMQ, MQTT, XMPP and PuSH.
September 13, 2011
WebSocket technology is catching up, but it will take a while before
all browsers support it.
In the meantime there are loads of projects that aim to substitute for
WebSockets and enable ‘realtime’ capabilities for web apps. But all
attempts solve only a part of the general problem, and there isn’t any
single solution that works, is scalable and doesn’t require special
September 9, 2011
While the firehose is quite a cool feature, I always thought that it was a shame we didn’t have a simple GUI to go on top and make it accessible to system administrators. So I wrote one. You can download it here.
September 1, 2011
We are very pleased to announce the availability in beta
of RabbitMQ as a Heroku
service on CloudFoundry, this extends our commitment to
supporting the community of cloud application developers.
We believe that cloud messaging is fundamental in two senses. First
as a core capability to build applications that scale to cloud use
cases as explained in
post launching RabbitMQ on CloudFoundry. And second, because
messaging can be extended to solve common problems like integration
and data push. For example: to connect traditional on-premise
applications with virtualized and cloud deployments.
August 22, 2011
The idea of ‘realtime web’ or messaging using web browsers has been
around for quite some time. First it was called ’long-polling’, then
‘Comet’, the latest incarnation is named ‘WebSockets’.
Without doubt it’s going in a good direction, WebSockets is a neat
But during the fight for realtime capabilities we’ve lost focus on
what is really important how to actually use messaging. In the web
context everything is request-response driven and marrying a typical
web stack to asynchronous messaging isn’t easy.
August 16, 2011
Recently we launched a RabbitMQ service for Cloud Foundry, making it simple to spin up a message broker to use with your apps on Cloud Foundry. There are tutorials online for using it with Ruby on Rails and with Java apps using Spring. Here we are going to look at using the RabbitMQ service with Node.JS apps.
August 10, 2011
Today we launched a RabbitMQ service on CloudFoundry.com. This service brings the messaging functionality of RabbitMQ to developers building applications on Cloud Foundry. You can read the main announcement over on the Cloud Foundry blog. There’s also an FAQ with more details on the Cloud Foundry knowledge base. CloudFoundry.com is a free beta service. So please register there (if you haven’t already), then take a look at the RabbitMQ service, try out the sample apps, and write your own.
July 8, 2011
I fundamentally disagree with the APIs exposed by our current AMQP client libraries.
There is a reason why they’re imperfect: we intentionally avoided innovation in APIs since the beginning. The purpose of our client libraries is to expose generic AMQP, not any one view of messaging. But, in my opinion, trying to map AMQP directly to client libraries APIs is just wrong and results in over-complication and abstractions hard to use.
There is no common ground: the client libraries blindly following AMQP model will be complex; easy to use client libraries must to be opinionated.
June 30, 2011
Recently I saw a tweet saying “ZeroMQ Erlangizes everything!” or some such. While I realise that not everything posted on the web is meant seriously, it does seem there is a stream of similar claims lately that ought to be dammed.
In the article Multi-threading Magic, Pieter Hintjens and Martin Sustrik persuasively explain why concurrency is better served by message-passing than by locks and shared memory. And they are fair, I think, in their analysis – except for the insinuation that using ZeroMQ transforms your chosen programming language into a domestic Erlang.
June 22, 2011
Note: this blog post talks about the federation plugin preview that was released for RabbitMQ 2.5.0. If you’re using 2.6.0 or later, federation is part of the main release; get it the same way you would any other plugin.
Another day, another new plugin release 😃 Today it’s federation. If you want to skip this post and just download the plugin, go here. The detailed instructions are here.
The high level goal of federation is to scale out publish / subscribe messaging across WANs and administrative domains.
To do this we introduce the concept of the federation exchange. A federation exchange acts like a normal exchange of a given type (it can emulate the routing logic of any installed exchange type), but also knows how to connect to upstream exchanges (which might in turn themselves be federation exchanges).
June 16, 2011
The RabbitMQ team is delighted to announce the release of RabbitMQ 2.5.0.
May 17, 2011
Most of us at RabbitMQ HQ have spend time working in a number of functional languages in addition to Erlang, such as Haskell, Scheme, Lisp, OCaml or others. Whilst there is lots to like about Erlang, such as its VM/Emulator, there are inevitably features that we all miss from other languages. In my case, having spent a couple of years working in Haskell before returning to the RabbitMQ fold, all sorts of features are “missing”, such as laziness, type classes, additional infix operators, the ability to specify precedence of functions, fewer parenthesis, partial application, more consistent standard libraries and do-notation. That’s a fair list, and it’ll take me a while to get around to implementing them all in Erlang, but here are two for starters.
March 28, 2011
Vlad Alexandru Ionescu
In our previous blog post we talked about a few approaches to topic routing optimization and described the two more important of these in brief. In this post, we will talk about a few things we tried when implementing the DFA, as well as some performance benchmarking we have done on the trie and the DFA.
March 23, 2011
RabbitMQ 2.4.0 introduced an extension that allows publishers to specify multiple routing keys in the CC and BCC message headers. The BCC header is removed from the message prior to delivery. Direct and topic exchanges are the only standard exchange types that make use of routing keys, therefore the routing logic of this feature only works with these exchange types.
March 1, 2011
I decided to run some benchmarks of my AMQP encoder/decoder (AMQ Protocol gem) against the old one in the AMQP gem to see whether it performs better or not. So far I did only the most basic optimisations like storing reusable values in constants, nothing special (yet).
I did two sets of benchmarks: CPU time benchmarking using my fork of RBench with support for custom formatters (like writing results into a YAML file) and memory benchmarking using
Object.count_objects (Ruby 1.9).
February 10, 2011
In many messaging scenarios, you must not lose messages. Since AMQP gives few guarantees regarding message persistence/handling, the traditional way to do this is with transactions, which can be unacceptably slow. To remedy this problem, we introduce an extension to AMQP in the form of Lightweight Publisher Confirms.
February 7, 2011
RabbitMQ 2.3.1 introduces a couple of new plugin mechanisms, allowing you much more control over how users authenticate themselves against Rabbit, and how we determine what they are authorised to do. There are three questions of concern here:
- How does the client prove its identity over the wire?
- Where do users and authentication information (e.g. password hashes) live?
- Where does permission information live?
Question 1 is answered in the case of AMQP by SASL - a simple protocol for pluggable authentication mechanisms that is embedded within AMQP (and various other protocols). SASL lets a client and a server negotiate and use an authentication mechanism, without the “outer” protocol having to know any of the details about how authentication works.
SASL offers a number of “mechanisms”. Since the beginning, RabbitMQ has supported the PLAIN mechanism, which basically consists of sending a username and password over the wire in plaintext (of course possibly the whole connection might be protected by SSL). It’s also supported the variant AMQPLAIN mechanism (which is conceptually identical to PLAIN but slightly easier to implement if you have an AMQP codec lying around). RabbitMQ 2.3.1 adds a plugin system allowing you to add or configure more mechanisms, and we’ve written an example plugin which implements the SASL EXTERNAL mechanism.
January 20, 2011
From time to time, on
list and elsewhere, the idea comes up of using a
different backing store within RabbitMQ. The backing store is
the bit that’s responsible for writing messages to disk (a message can
be written to disk for a number of reasons) and it’s a fairly frequent
suggestion to see what RabbitMQ would look like if its own backing
store was replaced with another storage system.
Such a change would permit functionality that is not currently
possible, for example out-of-band queue browsing, or distributed
storage, but there is a fundamental difference in the nature of data
storage and access patterns between a message broker such as RabbitMQ
and a generic database. Indeed RabbitMQ deliberately does not store
messages in such a database.
January 19, 2011
I’m happy to announce that the AMQP 0.7 is released, as I promised in the previous blog post. So what are the changes?
January 12, 2011
In the past year development of the AMQP gem was practicaly stagnating, as its original author Aman Gupta (@tmm1) was busy. A lot of bugs stayed unresolved, the code was getting old and out-dated and no new features or documentation were made.
At this point I started to talk with the RabbitMQ guys about possible collaboration on this. Actually originally I contacted VMware when I saw Ezra Zygmuntowicz looking for people to his cloud team, but when I found that VMware recently acquired the RabbitMQ project in London, I got interested. I signed the contract, switched from
script/console to Wireshark and the RabbitMQ Tracer and since November I’ve been happily hacking on the AMQP and AMQ-Protocol gems.