RabbitMQ 3.9 introduces a new type of data structure: streams. Streams unlock a set of use cases that could have been tedious to implement with “traditional” queues. Let’s discover in this post how streams expand the capabilities of RabbitMQ.
A RabbitMQ stream models an append-only log with non-destructive consuming semantics. This means that – contrary to traditional queues in RabbitMQ – consuming from a stream does not remove messages.
Streams in RabbitMQ are persisted and replicated. This translates to data safety and availability (in case of the loss of a node), as well as scaling (reading the same stream from different nodes.)
Streams can look a bit opinionated compared to the very versatile queues, but they come in handy for a set of use cases. They expand the capabilities of RabbitMQ in a very nice way.
RabbitMQ Streams shine for the following use cases:
And as streams ship as a core plugin in RabbitMQ 3.9, you can use them along all the already existing RabbitMQ features.
Let’s get more specific about streams:
You can have a look at the streams overview presentation from RabbitMQ Summit 2021 below if you want to learn more. If you are in a hurry, you can skip it and go directly to the quick start with Docker in the next section.
Without further ado, let’s make this thing run.
Exercising a stream is very easy with Docker. Let’s make sure you don’t already have the Docker images we are about to use locally:
docker rmi rabbitmq:3.9-rc pivotalrabbitmq/stream-perf-test
You’ll get an error message if the images are not on the computer, but this is fine.
Let’s create now a network for our server and performance tool containers to communicate:
docker network create rabbitmq-streams
It is time to start the broker:
docker run -it --rm --network rabbitmq-streams --name rabbitmq \ -e RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS='-rabbit loopback_users "none"' \ rabbitmq:3.9-rc
The broker should start in a few seconds. When it’s ready, enable the stream plugin:
docker exec rabbitmq rabbitmq-plugins enable rabbitmq_stream
Now launch the performance tool. It will create a stream, and publish and consume as fast as possible:
docker run -it --rm --network rabbitmq-streams pivotalrabbitmq/stream-perf-test \ --uris rabbitmq-stream://rabbitmq:5552
You can let the performance tool run for a while and then stop it with
19, published 1180489 msg/s, confirmed 1180145 msg/s, consumed 1180648 msg/s, \ latency min/median/75th/95th/99th 1537/7819/9631/12136/14425 µs, chunk size 2639 20, published 1181929 msg/s, confirmed 1181597 msg/s, consumed 1182074 msg/s, \ latency min/median/75th/95th/99th 1537/7838/9562/11967/14355 µs, chunk size 2657 ^C Summary: published 1205835 msg/s, confirmed 1205435 msg/s, consumed 1205477 msg/s, latency 95th 12158 µs, chunk size 2654
These are numbers on a regular Linux workstation, what you’ll get depends on your own setup. Note numbers can be significantly lower on macOS and Windows, as Docker runs in a virtualized environment on those operating systems.
You can then stop the broker container with
Ctrl+C and delete the network:
docker network rm rabbitmq-streams
If you want to go further and start building applications, the stream Java client documentation is a good starting point.
This concludes our overview of RabbitMQ Streams, a new append-only log data structure with awesome capabilities and tooling. Stay tuned to discover more about streams in subsequent posts!
Written by: Arnaud Cogoluègnes