[
https://issues.apache.org/jira/browse/KUDU-2202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16242753#comment-16242753
]
Andrew Wong commented on KUDU-2202:
-----------------------------------
(2) is fixed in 47b81c452194e75da7fd966f07766de4bdcdeab0.
Leaving this open to track (3).
> Removing a data directory is unsafe
> -----------------------------------
>
> Key: KUDU-2202
> URL: https://issues.apache.org/jira/browse/KUDU-2202
> Project: Kudu
> Issue Type: Bug
> Components: fs
> Affects Versions: 1.6.0
> Reporter: Adar Dembo
>
> I wrote a [patch|https://gerrit.cloudera.org/c/8352] that modifies the Kudu
> CLI to allow for data directory addition and removal. It turns out that
> implementing removal safely is quite complicated. Below I've outlined the
> various issues and their potential solutions or workarounds:
> # The data dir to be removed may be the first data dir, used for tablet and
> consensus metadata. Until these can be striped to other data dirs, we'll work
> around this by prohibiting the removal of the first data dir outright.
> # Tablets may have data blocks on the removed directory. No problem, just
> consider those tablets to be failed. Except, this could lead to the failed
> tablets' block IDs being reused in the creation of new blocks, which can lead
> to all sorts of issues. For example, deletion of a failed tablet whose block
> IDs were reused means deleting another tablet's data. Some band-aid solutions
> here include rewriting the superblocks on removal to also strip out all block
> IDs that were on the removed directory or persisting the maximum block ID on
> every disk (or in every superblock) to prevent block ID reuse. More solutions
> are discussed in the aforementioned patch as well.
> # Even if the removed data dir is empty of data, existing tablets may still
> be configured to stripe to it, either explicitly (their data dir group
> includes this data dir) or implicitly (they're from an older version of Kudu
> and don't have a data dir group). We could work around this by rewriting
> these tablets' superblocks to prune the removed data dir from the tablets'
> data dir groups.
> The patch works around issue #1 via prohibition, but is vulnerable to issues
> #2 and #3. Given that, and given that data dir addition is of limited value
> (we lack intra-node data rebalancing and existing tablets' superblocks aren't
> rewritten to take advantage of a new data dir), I'm shelving the patch. Note
> that it depends on [this other patch|https://gerrit.cloudera.org/c/8376],
> which allows tablets with removed data directories to start up.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
