RabbitMQ Production Operations: Monitoring, Alerting, and Incident Response

The Operational Reality of Enterprise RabbitMQ

RabbitMQ is a mature, stable message broker — and like all mature systems, its failure modes are well-understood but subtle. The most common RabbitMQ production incidents we respond to at OSSeva are not caused by bugs in RabbitMQ itself; they are caused by the interaction between RabbitMQ's memory management model, consumer throughput assumptions, and the operational practices (or lack thereof) that teams put in place when the cluster was first deployed.

This is a practical guide to the monitoring, alerting, and incident response practices that keep enterprise RabbitMQ clusters running reliably.

Critical Metrics to Monitor

Queue Depth and Consumer Lag

Queue depth — the number of messages in a queue — is the primary health signal for a RabbitMQ deployment. Healthy queues process messages faster than they arrive; unhealthy queues accumulate. Monitoring queue depth per queue with a threshold alert (e.g., alert when queue depth exceeds 10,000 messages for more than 5 minutes) catches consumer degradation before it becomes a full queue backup. The Prometheus rabbitmq_queue_messages metric is the right source.

Memory Usage and Memory Alarms

RabbitMQ has a built-in memory watermark at 40% of available system RAM by default. When this watermark is reached, RabbitMQ enters a "flow control" state and stops accepting new publishes from all producers until memory usage drops below the watermark. This is a necessary safety mechanism but can cause cascading failures in systems that don't handle publish blocks gracefully. Monitor rabbitmq_process_resident_memory_bytes and alert at 70% to give operators time to act before the watermark is reached.

Disk Space and Disk Alarms

Similar to memory, RabbitMQ enters a disk alarm state when free disk space drops below 50MB (configurable). In this state it stops accepting publishes. Durable queues and persistent messages accumulate disk usage that grows with queue depth; a backed-up queue in a cluster with limited disk space can trigger both queue backup and disk alarm simultaneously. Monitor disk usage on the RabbitMQ data directory separately from the system disk.

Connection and Channel Count

Each AMQP connection to RabbitMQ consumes a file descriptor and memory. Applications that don't properly close connections (connection leaks in error paths, connection pools that grow without bounds) drive connection counts up over time. Alert when total connections exceed your expected steady-state count by more than 20%; alert at a hard ceiling that you define based on your ulimit configuration.

Network Partition Detection

RabbitMQ clusters use Erlang's distributed Erlang mechanism for inter-node communication. Network partitions (temporary loss of connectivity between nodes) can cause RabbitMQ to enter a "partition" state where each sub-cluster believes the other is dead and continues operating independently, leading to split-brain scenarios. The rabbitmq_node_partitions metric is 0 in healthy clusters; any non-zero value requires immediate intervention.

Alerting Configuration

A minimal but effective alerting configuration for enterprise RabbitMQ covers:

Queue depth: warn at 5,000, critical at 50,000 (adjust per queue expected throughput)
Memory usage: warn at 70% of watermark, critical at 90%
Disk usage: warn at 5GB free, critical at 1GB free
Node count: alert if any node drops from the cluster
Network partitions: alert immediately (severity critical)
Consumer count per critical queue: alert if consumers drop to zero

Incident Response Runbook Basics

Queue Backup (Consumer Lag Growing)

Identify which queues are backing up using the management API or Grafana dashboard.
Check consumer count — if zero, consumers have stopped. Check consumer application logs and restart if necessary.
If consumers are running but throughput is insufficient, check for slow consumer patterns: blocking calls in the consumer, database connection pool exhaustion, downstream service latency.
If the queue backup is transient (spike in publish rate), increase consumer count temporarily by scaling the consumer application.

Memory Alarm Active

Identify which queues hold the most messages — these are the primary memory consumers.
If consumers are backed up, the fix is processing messages faster, not adding memory. Identify and resolve the consumer bottleneck.
If memory usage is being driven by connection count or channel count, identify the leaking application component and restart it.
As a temporary measure, increase the memory watermark (via rabbitmqctl) to give more headroom — but this only buys time; the root cause must be resolved.

Network Partition

Identify which nodes are in the partition using rabbitmqctl cluster_status.
Determine the partition handling policy: automatic (if configured) may have already chosen a winner; manual intervention is required for the ignore and pause-minority policies.
Resolve network connectivity between nodes first.
After connectivity is restored, restart RabbitMQ on the minority partition nodes to force them to rejoin the cluster rather than continuing as an independent cluster.
Verify cluster health using rabbitmqctl cluster_status and confirm all queues are accessible from all nodes.

OSSeva Operate for RabbitMQ

OSSeva Operate covers 24/7 monitoring of RabbitMQ clusters with the alerting configuration above pre-deployed, named engineers who know your specific topology, and a practiced incident response playbook tailored to your environment. When a memory alarm fires at 2am, the right person is already watching and already knows the runbook.

Conclusion

RabbitMQ operational reliability is achievable without heroics. The monitoring metrics are clear, the failure modes are well-documented, and the incident response procedures are deterministic. What requires investment is the operational discipline to deploy monitoring before incidents happen, to practice runbooks before they're needed under pressure, and to review incident post-mortems to prevent recurrence.