Menu

Serving Millions of Clients with Native MQTT

March 21, 2023 by David Ansari

RabbitMQ’s core protocol has been AMQP 0.9.1. To support MQTT, STOMP, and AMQP 1.0, the broker transparently proxies via its core protocol. While this is a simple way to extend RabbitMQ with support for more messaging protocols, it degrades scalability and performance.

In the last 9 months, we re-wrote the MQTT plugin to not proxy via AMQP 0.9.1 anymore. Instead, the MQTT plugin parses MQTT messages and sends them directly to queues. This is what we call Native MQTT.

The results are spectacular:

  1. Memory usage drops by up to 95% and hundreds of GBs with many connections.
  2. For the first time ever, RabbitMQ is able to handle millions of connections.
  3. End-to-end latency drops by 50% - 70%.
  4. Throughput increases by 30% - 40%.

Native MQTT turns RabbitMQ into an MQTT broker opening the door for a broader set of IoT use cases.

Native MQTT ships in RabbitMQ 3.12.

Read More...

Migrating from Mirrored Classic Queues to Quorum Queues

March 2, 2023 by Alexey Lebedeff

Quorum Queues are a superior replacement for Classic Mirrored Queues that were introduced in RabbitMQ version 3.8. And there are two complementary reasons why you would need to migrate.

First of all, Classic Mirrored Queues were deprecated in 3.9, with a formal announcement posted on August 21, 2021. They will be removed entirely in 4.0

But also they are more reliable and predictable, faster for most workloads and require less maintenance - so you shouldn’t feel that your hand is being forced without no apparent reason.

Quorum Queues are better in all regards, but they are not 100%-compatible feature-wise with Mirrored Queues. Thus the migration can look like a daunting task.

After a sneak peek into the future performance improvements, this post will outline a few possible migration strategies, including guidance on how to deal with all the incompatible features. Hopefully making migration process not so scary.

Read More...

High Initial Memory Consumption of RabbitMQ Nodes on Centos Stream 9

August 30, 2022 by Michael Klishin

Team RabbitMQ and community members have recently identified a curious scenario where a freshly started node could consume a surprisingly high amount of memory, say, 1.5 GiB or so. We’d like to share our findings with the community and explain what short term and longer term workarounds are available.

Read More...

RabbitMQ 3.11.0 release calendar

August 1, 2022 by Mirah Gary

We intend to release RabbitMQ 3.11.0 on 5 September 2022. While we have been testing it internally for some time, with production-like workloads, we need your help to check that it is as stable and reliable as we believe it is.

Read More...

OIDC Integration

July 22, 2022 by Marcial Rosales
Today when we use the rabbitmq-management with the rabbitmq_auth_backend_oauth2 plugin, the only supported Authorization server is UAA, making it difficult to connect to other OAuth 2.0 servers. Additionally, rabbitmq-management plugin uses the OAuth 2.0 implicit flow which is no longer recommended for security reasons. RabbitMQ 3.11 will support practically any Authorization server compliant with OpenID Connect and OAuth 2.0 protocols. Furthermore, OAuth 2.0 authorization code grant becomes the default grant and implicit grant is no longer supported.
Read More...

Required feature flags in RabbitMQ 3.11.0

July 20, 2022 by Jean-Sébastien Pédron
RabbitMQ 3.11.0 will make all feature flags introduced during the life of RabbitMQ 3.8.x required. People who initially created clusters using RabbitMQ 3.8.9 or older should enable all feature flags before upgrading to RabbitMQ 3.11! If the feature flags are not enabled, RabbitMQ 3.11.0+ will refuse to start. Feature flags are a mechanism to make breaking changes to RabbitMQ while still maintaining the compatibility between several versions of RabbitMQ. They usually come with code to migrate data structures in the internal schema database or on disk for instance.
Read More...

RabbitMQ 3.11 Feature Preview: Super Streams

July 13, 2022 by Arnaud Cogoluègnes

RabbitMQ 3.11 will bring a feature with one of the coolest names in its history: super streams. Super streams are a way to scale out by partitioning a large stream into smaller streams. They integrate with single active consumer to preserve message order within a partition.

This blog post gives an overview of super streams and the use cases they unlock. Read on to learn more, we value your feedback to make this feature the best it can be.

Read More...

RabbitMQ 3.11 Feature Preview: Single Active Consumer for Streams

July 5, 2022 by Arnaud Cogoluègnes

RabbitMQ 3.11 will bring a noteworthy feature to streams: single active consumer. Single active consumer provides exclusive consumption and consumption continuity on a stream. It is also critical to get the most out of super streams, our solution for partitioning, that provide scalability for streams.

Read on to find out more about single active consumer for streams and don’t hesitate to experiment with what is already available: try it, break it, tell us what you like and don’t like, what’s missing. Your feedback is essential to make this feature the best it can be.

Read More...

Improving RabbitMQ Performance with Flame Graphs

May 31, 2022 by David Ansari

Recent Erlang/OTP versions ship with Linux perf support. This blog post provides step by step instructions on how you can create CPU and memory flame graphs in RabbitMQ to quickly and accurately detect performance bottlenecks. We also provide examples of how flame graphs have helped us to increase message throughput in RabbitMQ.

Read More...

RabbitMQ 3.10 Performance Improvements

May 16, 2022 by Michał Kuratczyk

RabbitMQ 3.10 was released on the 3rd of May 2022, with many new features and improvements. This blog post gives an overview of the performance improvements in that release. Long story short, you can expect higher throughput, lower latency and faster node startups, especially with large definitions files imported on startup.

Read More...