Paul Rogers created DRILL-7613:
----------------------------------
Summary: Revise, harden the preliminary storage plugin upgrade
facility
Key: DRILL-7613
URL: https://issues.apache.org/jira/browse/DRILL-7613
Project: Apache Drill
Issue Type: Improvement
Affects Versions: 1.17.0
Reporter: Paul Rogers
Drill provides a way to upgrade storage plugins after installing a new Drill
version. However the mechanism is very crude and error prone. It is based on
overwriting the current contents of the persistent store with a new set of
configs. However, doing so is likely to discard essential user config.
For example, we cannot upgrade format plugins individually. So, to add a new
format plugin, we must overwrite, say, the {{dfs}} config. In so doing we may
throw away the user's S3 or HDFS config.
Further, we don't want to reapply the upgrades on every restart. So, the
mechanism has the ability to delete a file to mark the system as upgraded.
There are several problems with this idea. First, any such file is likely to be
in a jar as a resource, so is not deletable except in an IDE (when we would
really *not* want to delete it.)
Suppose the user does an upgrade, suffers a ZK loss, and restores from backup.
Drill will not know to re-upgrade ZK because the file is gone (assuming that
the delete actually worked.) This shows that using a file to indicate ZK state
is a poor implementation choice.
The code does not handle race conditions. If we bring up a cluster of 10
Drillbits, all 10 will race to perform upgrades.
The code has partial code to upgrade format plugins, but that code does not
actually work as there is no no good way to do that upgrade. (Each DFS storage
plugin has its own set of format plugins, unfortunately, and there is no code
to find all such DFS storage plugins and apply format plugin updates.)
A better solution would be to:
* Store a version in the plugin registry. Upgrade only if the version in the
registry is lower than the current Drillbit version.
* Better, provide a SQL command to force the upgrade. This allows users to do
rolling v-1/v upgrades without the v-1 Drillbits seeing plugins that they
cannot handle.
* Implement a race-condition-proof upgrade: select one Drillbit to do the
upgrade and let the others wait. (Leader election.)
* Separate format plugins from the DFS storage plugin. Allow the same formats
to be used across all configured DFS plugins. (There is no harm in offering a
format for non-existent files.)
* Complete the work to upgrade the (now separate) format plugins so we can
automatically roll out new formats without users having to do the upgrade
manually.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
