Feature Flags
Overview
In a mixed version cluster (e.g. some versions are 3.11.x and some are 3.12.x) during an upgrade, some nodes will support a different set of features, behave differently in certain scenarios, and otherwise not act exactly the same: they are different versions after all.
Feature flags are a mechanism that controls what features are considered to be enabled or available on all cluster nodes. If a feature flag is enabled, so is its associated feature (or behavior). If not then all nodes in the cluster will disable the feature (behavior).
The feature flag subsystem allows RabbitMQ nodes with different versions to determine if they are compatible and then communicate together, despite having different versions and thus potentially having different feature sets or implementation details.
This subsystem was introduced to allow for rolling upgrades of cluster members without shutting down the entire cluster.
Feature flags are not meant to be used as a form of cluster configuration. After a successful rolling upgrade, users should enable all feature flags.
All feature flags become mandatory (graduate) at some point. For example, RabbitMQ 3.12 requires feature flags introduced in the 3.11 series to be enabled prior to the upgrade, RabbitMQ 3.11 graduates all 3.8 flags, and so on.
Quick summary (TL;DR)
Feature Flag Ground Rules
- A feature flag can be enabled only if all nodes in the cluster support it
- A node can join or re-join a cluster only if:
- it supports all the feature flags enabled in the cluster and
- if every other cluster member supports all the feature flags enabled on that node
- Once enabled, a feature flag cannot be disabled
For example, RabbitMQ 3.13.x and 3.12.x nodes are compatible as long as no 3.13.x-specific feature flags are enabled.
This subsystem does not guarantee that all future changes in RabbitMQ can be implemented as feature flags and entirely backwards compatible with older release series. Therefore, a future version of RabbitMQ might still require a cluster-wide shutdown for upgrading.
Please always read release notes to see if a rolling upgrade to the next minor or major RabbitMQ version is possible.
Key CLI Tool Commands
- To list feature flags:
rabbitmqctl list_feature_flags
- To enable a feature flag (or all currently disabled flags):
rabbitmqctl enable_feature_flag <all | name>
It is also possible to list and enable feature flags from the Management plugin UI, in "Admin > Feature flags".
Examples
Example 1: Compatible Nodes
- If nodes A and B are not clustered, they can be clustered.
- If nodes A and B are clustered:
- "Coffee maker" can be enabled.
- "Juicer machine" cannot be enabled because it is unsupported by node B.
Example 2: Incompatible Nodes
- If nodes A and B are not clustered, they cannot be clustered because "Juicer machine" is unsupported on node B.
- If nodes A and B are clustered and "Juicer machine" was enabled while node B was stopped, node B cannot re-join the cluster on restart.
Feature Flags and RabbitMQ Versions
As covered earlier, the feature flags subsystem's primary goal is to allow upgrades regardless of the version of cluster members, to the extent possible.
Feature flags make it possible to safely perform a rolling upgrade to the next patch or minor release, except if it is stated otherwise in the release notes. Indeed, there are some changes which cannot be implemented as feature flags.
However, note that only upgrading from one minor to the next minor or major is supported. To upgrade from e.g. 3.9.16 to 3.12.3, it is necessary to upgrade to 3.9.29 first, then to the latest 3.10 patch release, then the latest 3.11 release, then 3.12.3. After certain steps in the upgrade process it will also be necessary to enable all stable feature flags available in that version. For example, 3.12.0 is a release that requires all feature flags to be enabled before a node can be upgraded to it.
Likewise if there is one or more minor release branches between the minor version used and the next major release. That might work (i.e. there could be no incompatible changes between major releases), but this scenario is unsupported by design for the following reasons:
- Skipping minor versions is not tested in CI.
- Non-sequential releases may or may not support the same set of feature flags. Feature flags present for several minor branches can be marked as required and their associated feature/behavior is now implicitly enabled by default. The compatibility code is removed in the process, preventing clustering with older nodes. Remember their purpose is to allow upgrades, they are not a configuration mechanism.
Their is no policy defining the life cycle of a feature flag in general. E.g. there is no guaranty that a feature flag will go from "stable" to "required" after N minor releasees. Because new code builds on top of existing code, feature flags are marked as required and the compatibility code is removed whenever it is needed.
How to List Supported Feature Flags
When a node starts for the first time, all stable feature flags are enabled by default. When a node is upgraded to a newer version of RabbitMQ, new feature flags are left disabled.
To list the feature flags, use rabbitmqctl list_feature_flags
:
rabbitmqctl list_feature_flags
# => Listing feature flags ...
# => name state
# => empty_basic_get_metric enabled
# => implicit_default_bindings enabled
# => quorum_queue enabled
For improved table readability, switch to the pretty_table
formatter:
rabbitmqctl -q --formatter pretty_table list_feature_flags \
name state provided_by desc doc_url
which would produce a table that looks like this:
┌───────────────────────────┬─────────┬───────────────────────────┬───────┬────────────┐
│ name │ state │ provided_by │ desc │ doc_url │
├───────────────────────────┼─────────┼───────────────────────────┼───────┼────────────┤
│ empty_basic_get_metric │ enabled │ rabbitmq_management_agent │ (...) │ │
├───────────────────────────┼─────────┼───────────────────────────┼───────┼────────────┤
│ implicit_default_bindings │ enabled │ rabbit │ (...) │ │
├───────────────────────────┼─────────┼───────────────────────────┼───────┼────────────┤
│ quorum_queue │ enabled │ rabbit │ (...) │ http://... │
└───────────────────────────┴─────────┴─────────────────────────── ┴───────┴────────────┘
As shown in the example above, the list_feature_flags
command accepts
a list of columns to display. The available columns are:
name
: the name of the feature flag.state
: enabled or disabled if the feature flag is enabled or disabled, unsupported if one or more nodes in the cluster do not know this feature flag (and therefore it cannot be enabled).provided_by
: the RabbitMQ component or plugin which provides the feature flag.desc
: the description of the feature flag.doc_url
: the URL to a webpage to learn more about the feature flag.stability
: indicates if the feature flag is required, stable or experimental.
How to Enable Feature Flags
After upgrading one node or the entire cluster, it will be possible to enable new feature flags. Note that it will be impossible to roll back the version or add a cluster member using the old version once new feature flags are enabled.
To enable a feature flag, use:
rabbitmqctl enable_feature_flag <name>
To enable all stable feature flags, use:
rabbitmqctl enable_feature_flag all
The rabbitmqctl enable_feature_flag all
command enables stable feature flags
only and not experimental ones.
The list_feature_flags
command can be used again to verify the feature
flags' states. Assuming all feature flags were disabled initially, here
is the state after enabling the quorum_queue
feature flag:
rabbitmqctl -q --formatter pretty_table list_feature_flags