RabbitMQ Streams Overview

July 13, 2021

RabbitMQ 3.9 introduces a new type of data structure: streams. Streams unlock a set of use cases that could have been tedious to implement with “traditional” queues. Let’s discover in this post how streams expand the capabilities of RabbitMQ.

What are RabbitMQ Streams

A RabbitMQ stream models an append-only log with non-destructive consuming semantics. This means that – contrary to traditional queues in RabbitMQ – consuming from a stream does not remove messages.

Streams in RabbitMQ are persisted and replicated. This translates to data safety and availability (in case of the loss of a node), as well as scaling (reading the same stream from different nodes.)

Streams can look a bit opinionated compared to the very versatile queues, but they come in handy for a set of use cases. They expand the capabilities of RabbitMQ in a very nice way.

What are streams good for

RabbitMQ Streams shine for the following use cases:

And as streams ship as a core plugin in RabbitMQ 3.9, you can use them along all the already existing RabbitMQ features.

RabbitMQ Streams in a nutshell

Let’s get more specific about streams:

You can have a look at the streams overview presentation from RabbitMQ Summit 2021 below if you want to learn more. If you are in a hurry, you can skip it and go directly to the quick start with Docker in the next section.

Without further ado, let’s make this thing run.

Quick start with Docker

Exercising a stream is very easy with Docker. Let’s make sure you don’t already have the Docker images we are about to use locally:

docker rmi rabbitmq:3.9-rc pivotalrabbitmq/stream-perf-test

You’ll get an error message if the images are not on the computer, but this is fine.

Let’s create now a network for our server and performance tool containers to communicate:

docker network create rabbitmq-streams

It is time to start the broker:

docker run -it --rm --network rabbitmq-streams --name rabbitmq \
    -e RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS='-rabbit loopback_users "none"' \

The broker should start in a few seconds. When it’s ready, enable the stream plugin:

docker exec rabbitmq rabbitmq-plugins enable rabbitmq_stream

Now launch the performance tool. It will create a stream, and publish and consume as fast as possible:

docker run -it --rm --network rabbitmq-streams pivotalrabbitmq/stream-perf-test \
    --uris rabbitmq-stream://rabbitmq:5552

You can let the performance tool run for a while and then stop it with Ctrl+C:

19, published 1180489 msg/s, confirmed 1180145 msg/s, consumed 1180648 msg/s, \
    latency min/median/75th/95th/99th 1537/7819/9631/12136/14425 µs, chunk size 2639
20, published 1181929 msg/s, confirmed 1181597 msg/s, consumed 1182074 msg/s, \
    latency min/median/75th/95th/99th 1537/7838/9562/11967/14355 µs, chunk size 2657
Summary: published 1205835 msg/s, confirmed 1205435 msg/s, consumed 1205477 msg/s, latency 95th 12158 µs, chunk size 2654

These are numbers on a regular Linux workstation, what you’ll get depends on your own setup. Note numbers can be significantly lower on macOS and Windows, as Docker runs in a virtualized environment on those operating systems.

You can then stop the broker container with Ctrl+C and delete the network:

docker network rm rabbitmq-streams

If you want to go further and start building applications, the stream Java client documentation is a good starting point.

This concludes our overview of RabbitMQ Streams, a new append-only log data structure with awesome capabilities and tooling. Stay tuned to discover more about streams in subsequent posts!

Written by: Arnaud Cogoluègnes

Categories: New Features Streams