AMQP 1.0 Modified Outcome
This blog post explores use cases of the AMQP 1.0 modified outcome.
The modified outcome is a feature exclusive to AMQP 1.0 and not available in AMQP 0.9.1 It is supported in quorum queues, but not in classic queues.
This feature enables consumers to add or update message annotations before requeueing or dead lettering a message.
Requeue
Including additional metadata when requeuing a message can be valuable for improving traceability and debugging during message processing.
For example, an application using the RabbitMQ AMQP 1.0 Java Client can set specific message annotations before requeuing the message at the head of a quorum queue, as shown below:
Consumer consumer = connection.consumerBuilder()
.queue(ordersQueue)
.messageHandler((context, message) -> {
Map<String, Object> annotations = new HashMap<>();
annotations.put("x-opt-requeue-reason", "external_service_unavailable");
annotations.put("x-opt-requeue-time", System.currentTimeMillis());
annotations.put("x-opt-requeued-by", "consumer_1");
context.requeue(annotations);
}).build();
These annotations could use different types including map, list, or array. This flexibility allows not only setting details like the last requeue reason, time, and consumer, but also tracking a history of requeue events. Maintaining such a history can reveal patterns, such as identifying consumers that requeue messages more frequently or discovering common requeue reasons across the system. However, keep in mind that quorum queues retain modified message annotations in memory, which increases the memory overhead per requeued message.
Setting custom headers before requeueing a message at the head of the queue is not supported in AMQP 0.9.1.
Whether requeuing a message to a quorum queue via AMQP 1.0 or AMQP 0.9.1, the x-delivery-count annotation will always be incremented.
Dead Letter
When dead lettering a message, the consumer can include a custom reason for the dead lettering in the message annotations:
Consumer consumer = connection.consumerBuilder()
.queue(ordersQueue)
.messageHandler((context, message) -> {
Map<String, Object> annotations = new HashMap<>();
annotations.put("x-opt-dead-letter-reason", "Incompatible Message Format");
context.discard(annotations);
}).build();
When dead lettering to a headers exchange, the consumer can even decide which target queue the message will be routed to:
In this example, two dead letter quorum queues are bound to the dead letter headers exchange:
transient-failures-dlq
business-logic-failures-dlq
Different dead letter queues can be processed by different apps or teams, with varying actions taken depending on the nature of the failure.
For instance, all messages in the transient-failures-dlq
could be re-published to the original orders
queue, while messages in the business-logic-failures-dlq
might require human intervention.
More dead letter queues could be added, such as:
data-integrity-dlq
for messages with unknown schemaresource-limit-dlq
for cases where rate limits were exceededcritical-errors-dlq
for situations that require administrator attention.
It’s crucial that all messages dead lettered from the orders
queue are routable.
The alternate exchange in the above diagram provides "or else" routing semantics, ensuring messages end up in the uncategorised-dlq
if no x-opt-dead-letter-category
annotation is set.
This might occur, for example, if the publisher sets a ttl
header but no consumer grants link credit, causing the message to expire and be dead lettered.
The scenario depicted above is demonstrated in the modified-outcome sample application.
modified-outcome sample application
The sample app uses the RabbitMQ AMQP 1.0 Java Client.
You can run this sample application as follows:
- Start RabbitMQ server via
docker run -it --rm --name rabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:4.0-management
- In the root directory of the sample app, start the client via
mvn clean compile exec:java
.
After publishing a message to the orders
queue, the client app consumes the message and outputs the following on the console:
publisher: received ACCEPTED outcome
consumer: setting annotations {x-opt-dead-letter-reason=Customer Not Eligible for Discount, x-opt-dead-letter-category=business-logic} and dead lettering...
The message will be dead lettered to the business-logic-failures-dlq
.
To prevent message loss during dead lettering, the sample app uses at-least-once dead lettering.
Dead Letter vs. Re-publish
An AMQP 0.9.1 consumer cannot set custom headers before dead lettering a message.
However, instead of using basic.nack
or basic.reject
with requeue=false
to dead letter a message, an AMQP 0.9.1 client could follow this approach:
- Re-publish the message directly to a specific "dead letter" queue with new custom headers.
- Wait for RabbitMQ to confirm the re-published message.
- Acknowledge the original message via
basic.ack
.
An AMQP 1.0 client can choose between dead lettering with custom message annotations or re-publishing the message. Both approaches have their advantages and trade-offs:
Criteria | Dead Letter with Custom Reason | Re-publish with Custom Reason |
---|---|---|
Simplicity | Easier for consumers. | More complex, as the consumer must handle the republishing process. |
Overhead | Low overhead. | Higher overhead for the client: the message payload must be re-published from the client to RabbitMQ, with additional latency due to the extra publish and confirm steps. |
Network Failure between client and RabbitMQ before settling the consumed message. | Message gets requeued. | The message might have been both re-published and requeued, resulting in one copy ending up in the "dead letter" queue and another in the original queue. |
Flexibility | Can modify only message annotations and route based on dead letter headers exchange. | Allows modification of any part of the message and re-publishing to any exchange. |
Wrapping Up
AMQP 1.0's modified outcome feature allows consumers to modify message annotations before requeueing or dead lettering.
Rather than relying solely on RabbitMQ's built-in dead lettering tracking via x-opt-deaths, consumers can customise dead lettering event tracking and even choose which dead letter queue a message is sent to.