Skip to content

RabbitMQ disaster recovery#

Ametnes runs RabbitMQ as a normal clustered broker: quorum queues replicate a queue across the nodes of one cluster (in-cluster HA). Cross-cluster disaster recovery is a federation relationship between two clusters — a primary (serving traffic) and a secondary (holding a replica) — and it is configured with the broker's own management API, not through the resource config.

What the platform provides#

Every RabbitMQ cluster ships with the federation and shovel plugins enabled, so either cluster can be cast as either side of a link without a chart change. There is no DR field on the resource: Ametnes does not create, promote, or demote a replication link for you. The link is a small amount of broker state:

  • a federation-upstream runtime parameter on the secondary pointing at the primary's AMQPS endpoint, and
  • a policy selecting which exchanges to federate.

DR uses exchange federation: the unit of replication is the publish flow through a named exchange, not queue contents.

  • Federation is configured on the secondary cluster only; the primary needs no DR configuration.
  • The policy is a regular-expression pattern over exchange names, matched continuously, so exchanges created later are federated automatically.
  • The link buffers in an internally declared upstream queue on the primary, so nothing is lost while the secondary reconnects, and the link is self-healing: if the primary restarts, the secondary re-establishes it automatically.

Two preconditions

1. Publish through a named exchange. The default (blank) exchange cannot be federated, so applications that publish with exchange="" are not replicated. Publish to a named direct/topic/fanout exchange.

2. The secondary must hold the topology. Federation forwards a message to the secondary only if its exchange can route it to a queue there, so declare the same exchanges/queues/bindings on both clusters.

Set the admin credentials and the primary's AMQPS endpoint, then create the upstream and the policy over the management API:

SEC=https://<secondary-host>:15671
PRIMARY_AMQPS=<primary-host>:5671
U=<admin-user>; P=<admin-password>

# 1. upstream: the secondary pulls from the primary over AMQPS
curl -fsS -u "$U:$P" -X PUT "$SEC/api/parameters/federation-upstream/%2f/primary-upstream" \
  -H 'content-type: application/json' \
  -d "{\"value\":{\"uri\":\"amqps://$U:$P@$PRIMARY_AMQPS\"}}"

# 2. policy: federate every exchange whose name matches the pattern
curl -fsS -u "$U:$P" -X PUT "$SEC/api/policies/%2f/federate" \
  -H 'content-type: application/json' \
  -d '{"pattern":"^(?!amq\\.).*","definition":{"federation-upstream":"primary-upstream"},"apply-to":"exchanges"}'

%2f is the URL-encoded default vhost /. Verify the link:

curl -fsS -u "$U:$P" "$SEC/api/federation-links"

A running link appears once a federated exchange exists on the secondary.

Fail over#

  1. Detect the primary is unavailable.
  2. Repoint applications (producers and consumers) at the secondary's endpoint. Failover is application-driven — no DNS failover is performed.
  3. The secondary already holds everything published to the federated exchanges while the link was up; new traffic goes straight to it.

Fail back#

  1. Restore the primary. The federation link re-establishes itself; nothing to reconfigure.
  2. Repoint applications back to the primary.
  3. Drain the secondary: messages produced directly to it during the outage do not flow back automatically. Consume that backlog, or run a one-off reverse shovel for the affected queues, then remove it.

Properties#

Property Behavior
RPO Near-zero while the link is up (publishes during a link outage are not replicated)
Failover trigger Manual (detect + repoint applications)
Failback trigger Manual (repoint applications after the primary recovers)
Dynamic exchanges Handled automatically by the exchange policy
Queue backlog catch-up None — exchange federation replicates the live publish flow only
Default-exchange traffic Not replicated — publish to named exchanges
Replication direction One-way (primary → secondary); reverse needs a temporary shovel

Quorum queues are in-cluster HA, not cross-cluster DR

Queues default to the quorum type, which Raft-replicates them across the nodes of a single cluster. Quorum queues cannot span clusters, so cross-cluster replication uses federation (or a shovel).