[
https://issues.apache.org/jira/browse/HDFS-5223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13779511#comment-13779511
]
Todd Lipcon commented on HDFS-5223:
-----------------------------------
bq. I love the flexibility of feature bits however I'm very nervous about the
complexity they tend to bring. As long as there is incredibly tight controls it
can work but more often than not I've seen this sort of approach lead to some
incredibly unmaintainable code. The code can get very complex dealing with
multiple combinations and the testing/QA can get also be very difficult to
manage. Things can get overwhelmingly complex quite quickly.
I agree that it's a bit more complex, but I'm not sure it's quite as bad in our
context as it might be in others. Most of our edit log changes to date have
been fairly simple. Looking through the Feature enum, they tend to fall into
the following categories:
- Entirely new opcodes (eg CONCAT) - these are easy to do on the writer side by
just throwing an exception in logEdit() if the feature isn't supported.
Sometimes these also involve a new set of data written to the FSImage (eg in
the case of delegation token persistence) but again it should be pretty
orthogonal to other features.
- New "container" format features (eg fsimage compression, or checksums on edit
entries). These are new features which are off by default and orthogonal to any
other features.
- Single additional fields in existing opcodes. We'd need to be somewhat
careful not to make use of any of these fields if the feature isn't enabled,
but I think there's usually pretty clear semantics.
Certainly it's more complex than option 1, but I think the ability to downgrade
without data loss is pretty key. A lot of Hadoop operators are already hesitant
to upgrade between minor versions already, and losing the ability to roll back
would make it a non-starter for a lot of shops. If that's the case, then I
think it would be really tough to add new opcodes or other format changes even
between minor releases (eg 2.3 to 2.4) and convince an operator to do the
upgrade.
Am I being overly conservative in what operators will put up with, instead of
overly conservative in the complexity we introduce?
(btw, I agree completely about the "no-delete" mode -- I think a "TTL delete
mode" is also a nice feature we could build in at the same time, where block
deletions are always delayed for a day, to mitigate potential for data loss
even with bugs present)
> Allow edit log/fsimage format changes without changing layout version
> ---------------------------------------------------------------------
>
> Key: HDFS-5223
> URL: https://issues.apache.org/jira/browse/HDFS-5223
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
> Affects Versions: 2.1.1-beta
> Reporter: Aaron T. Myers
>
> Currently all HDFS on-disk formats are version by the single layout version.
> This means that even for changes which might be backward compatible, like the
> addition of a new edit log op code, we must go through the full `namenode
> -upgrade' process which requires coordination with DNs, etc. HDFS should
> support a lighter weight alternative.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
