Hi all,
For anyone contributing changes to the service configs, I'd like to highlight a
couple of important cases that affect Rolling/Express Upgrade. In the past,
there have been several bugs during RU/EU because we forgot to make changes to
the Config Packs whenever editing stack configs, so I'd like to cover some
developer tips.
A little background: RU/EU only changes configs when moving across a major
stack version, e.g., HDP 2.3 to 2.4 (or even multiple versions such as HDP 2.2
to 2.4).
It tries to auto-merge newly added configs and persist configs that the user
has changed from their default values in their current stack.
In order to track explicit config changes, we use Config Packs that can add,
set, delete, rename, transform configs. E.g.
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml
Today, the Config Pack inside the HDP 2.3 directory is used whenever upgrading
to HDP 2.3. However, there's a code review currently in-progress to change the
file locations so a config pack inside folder X means upgrading from version X.
Let's assume that the cluster is on HDP 2.3 and the user wishes to RU/EU to HDP
2.4.
As a developer, consider what happens when you make the following changes to
configs in HDP 2.4
* New config property in higher stack: then the new config will be
auto-merged into the RU/EU from HDP 2.3 to 2.4, so you don't need to modify the
Config Pack.
* Edit existing config:
* Preferred: Assume HDP 2.3 has some config with value X and HDP 2.4
decides to use value Y. Then if the cluster is still on value X then RU/EU will
automatically change the config to value Y. But if the user has modified it to
Z, then we will persist Z during the upgrade. This is because the user's change
was intentional so we keep them on their value.
* Forced: Assume HDP 2.3 has some config with value X and HDP 2.4
decides to use value Y. If you want to force the cluster to use value Y even if
the cluster has modified the value, then you need to include a "set" directive
in the Config Pack to forcibly change the config.
<set key="my_config_name" value="value1"/>
* Transform/Rename: Assume HDP 2.3 uses value X, and HDP 2.4 applies a
transformation to value X, e.g., "foo, bar" => "['foo', 'bar']", then we need
to explicitly apply a transformation during the RU/EU.
The Config Pack supports directives for transforming a property value and
renaming a property.
* Delete: Deprecated configs are never deleted explicitly due to stack
inheritance. In order to delete a config, you must do it explicitly.
<transfer operation="delete" delete-key="my_config_name" />
This means we need to be extremely careful when making config changes because
it also impacts RU/EU.
Further, if we change the default value of a config in an already-released
stack, it has implications because any clusters with that stack will treat the
current config as having been modified from the default, and will persist the
user's value even if a more recent stack changes the value.
If you have any questions related to stack configs, feel free to ping me.
Thanks,
Alejandro
P.S., happy coding
