Skip to main content
Version: Next

Memory Alarm Threshold

Overview​

This guide covers RabbitMQ memory threshold (watermark) settings. It is accompanied by a few closely related guides:

Memory Threshold: What it is and How it Works​

RabbitMQ nodes can be provided with a memory footprint limit hint. If the node's memory footprint goes above the value, a resource alarm will be triggered on this and eventually all other cluster nodes to block publishers.

The limit can be configured as an absolute or relative value. In the latter case, RabbitMQ will try to detect the total amount of RAM available to it on startup and when rabbitmqctl set_vm_memory_high_watermark value is executed.

By default, including if the limit hint is not configured, a RabbitMQ node will use about 60% of the available RAM, it raises a memory alarm and will block all connections that are publishing messages. Once the memory alarm has cleared (e.g. due to delivering some messages to clients that consume and acknowledge the deliveries) normal service resumes.

warning

The limit does not prevent RabbitMQ nodes from using more than the computed limit, it is merely the point at which publishers are throttled

Configuring the Memory Limit (or Threshold)​

Absolute Memory Limit​

tip

Using an absolute memory threshold is highly recommended in containerized environments such as Kubernetes. RabbitMQ nodes won't always be able to detect the effective cgroups limit

The memory threshold can be adjusted by setting an absolute limit of RAM used by the node. The example below sets the threshold to 1073741824 bytes (1024 MiB):

vm_memory_high_watermark.absolute = 1073741824

Same example, but using memory units:

vm_memory_high_watermark.absolute = 1024MiB
vm_memory_high_watermark.absolute = 4Gi
vm_memory_high_watermark.absolute = 1Ti

The supported decimal (power-of-ten) memory information units are:

  • GB for gigabytes (1000^3 or 10^9 bytes)
  • MB for megabytes (1000^2)
  • TB for terabytes (1000^4)
  • PB for petabytes

The supported binary (power-of-two) memory information units are:

  • Gi for gibibytes (1024^3 or 2^30 bytes)
  • Mi for mebibytes (1024^2)
  • Ti for tebibytes (1024^4)
  • Pi for pebibytes

Kubernetes-style information units are also supported:

  • Gi for gibibytes (1024^3 or 2^30 bytes)
  • Mi for mebibytes
  • Ti for tebibytes
  • Pi for pebibytes

If the absolute limit is larger than the installed RAM or available virtual address space, the threshold is set to whichever limit is smaller.

The memory limit is appended to the log file when the RabbitMQ node starts:

2023-06-10 23:17:05.976 [info] <0.308.0> Memory high watermark set to 1024 MiB (1073741824 bytes) of 8192 MiB (8589934592 bytes) total

The memory limit may also be queried using the rabbitmq-diagnostics memory_breakdown and rabbitmq-diagnostics status commands.

Relative Memory Threshold​

warning

Using a relative memory threshold is not recommended in containerized environments such as Kubernetes. Prefer the absolute threshold instead.

The memory threshold at which the flow control is triggered can be adjusted by editing the configuration file.

The example below sets the threshold to the default value of 0.6:

# new style config format, recommended
vm_memory_high_watermark.relative = 0.6

The default value of 0.6 stands for 60% of available (detected) RAM.

Updating Memory Threshold on a Running Node​

The threshold can be changed while the broker is running using the

rabbitmqctl set_vm_memory_high_watermark <fraction>

command or

rabbitmqctl set_vm_memory_high_watermark absolute <em><memory_limit></em>

For example:

rabbitmqctl set_vm_memory_high_watermark 0.7

and

rabbitmqctl set_vm_memory_high_watermark absolute "4G"

For the memory information units supported, see Absolute Threshold

Both commands will have an effect until the node stops. To make the setting survive node restart, use the configuration setting instead.

The memory limit may change on systems with hot-swappable RAM when this command is executed without altering the threshold, due to the fact that the total amount of system RAM is queried.

Running RabbitMQ in Containers and on Kubernetes​

tip

RabbiMQ Cluster Operator for Kubernetes automatically configures the memory limit. Using the Operator is the recommended way to run RabbitMQ on Kubernetes.

When a RabbitMQ node is running in a container, its ability to detect the amount of available memory will depend on external factors: the version of the runtime used, the OS version and settings used by the image, the version of cgroups used, and ultimately the version of Kubernetes.

This means that in containerized environments, the optimal option is to configure an absolute memory limit.

Another Kubernetes-specific memory footprint aspect is how the OS-managed kernel page cache, in particular in clusters where streams and super streams are used.

How to Temporarily Stop All Publishing​

When the threshold or absolute limit is set to 0, it makes the memory alarm go off immediately and thus eventually blocks all publishing connections. This may be useful if you wish to deactivate publishing globally:

rabbitmqctl set_vm_memory_high_watermark 0

Limited Address Space​

::: danger RabbitMQ only targets 64 bit operating systems and a 64-bit Erlang runtime :::

RabbitMQ only targets 64 bit operating systems and a 64-bit Erlang runtime.

When running RabbitMQ inside a 32 bit Erlang VM in a 64 bit OS (or a 32 bit OS with PAE), the addressable memory is limited. The server will detect this and log a message like:

2018-11-22 10:44:33.654 [warning] Only 2048MB of 12037MB memory usable due to limited address space.

Unrecognised Platforms​

If the RabbitMQ server is unable to detect the operating system it is running on, it will append a warning to the log file. It then assumes than 1GB of RAM is installed:

2018-11-22 10:44:33.654 [warning] Unknown total memory size for your OS {unix,magic_homegrown_os}. Assuming memory size is 1024MB.

In this case, set total_memory_available_override_value to the amount of memory actually available on your system. The vm_memory_high_watermark.relative will then be calculated relative to the value set in total_memory_available_override_value.

For guidelines on recommended RAM watermark settings, see Deployment Guidelines.