[
https://issues.apache.org/jira/browse/HDFS-12615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16476295#comment-16476295
]
Anu Engineer commented on HDFS-12615:
-------------------------------------
[~chris.douglas] Thanks for the response. Please forgive me for the length of
this response, I wanted to make sure that I have carefully considered all your
arguments and respond to them.
I am arguing that the core issue of this JIRA is lack of communication, and I
am perhaps overcompensating for it :)
{quote}"All future work" is too broad, but developing complex features in
branches is a fair ask.
{quote}
I agree that is a reasonable statement. I hope that by the end of this message
I would have articulated my concerns and my position reasonably well.
{quote}Would you mind looking through the JIRAs implementing quotas and
multi-subcluster mounts, as an exercise?
{quote}
Will do, but I also want to make sure that we are not missing the forest for
the trees. Please read on to understand my concerns.
{quote}It is important to distinguish between routine maintenance and
significant features.
{quote}
Completely Agree.
{quote}For the latter, design docs should be written, experts' opinion
solicited, and implementation should be in a branch if it requires more than a
handful of patches.
{quote}
You nailed it on the head. I don't quite agree with the patch count logic, for
I have seen patches with are more than 200 KB at times. Let us just say "we
know a big feature when we see it"
([https://en.wikipedia.org/wiki/I_know_it_when_I_see_it])
Given that, I feel that a JIRA that has these following features.
# Security
# Quotas
# Multi-subcluster mounts
# Dynamic Cluster Build-outs
# Change of Configuration Management
# DN Protocol changes
# Inter-Cluster Rebalancing
Sounds like a major undertaking in my mind and feels like these do need a
branch and these are significant features.
{quote}In that particular case, I assume the patch was shared for discussion,
not commit.
{quote}
Perhaps there is a communication problem here. I am not sure where your
assumption comes from; reading the comments on the security patch, I am not
able to come to that conclusion. Please take a look at HDFS-12284.
I have copied some sample comments from the JIRA(to make it easy for you to
read), do these comments sound to you as if it was a proposal?
_"I've tested it on the command line. Adding directory/deleting
directory/copying file from the local file system, etc. I am working on the
unit test. Please review the patch in the meantime. "_
_"Thanks, Sherwood Zheng, nice work! Could you click "Submit Patch" to trigger
Jenkins testing?"_
_"Can you open a new JIRA for the delegation tokens part?"_
If we both are reading the same comments, I am at a loss on how you came to the
conclusion that it was a proposal and not a patch.
And that brings us to the most critical issues I have.
While building all these features; there are no design docs, and from where I
am standing there is not even a consistency what is a real patch, a proposal or
a real JIRA. Let me show you an example:
# Let us look at HDFS-13098. This is supposed to be a discussion JIRA
(Something that I learn after my comments)
# At some point, the discussion went to let us commit this other patch since
that will help this work (HDFS-13312).
# I come and comment that it all looks great, but please commit to your
branch, so that people understand what you are proposing (See the earlier point
about lack of design document in this important JIRA.)
# I specifically write "I don't object to patches, just move it to a branch"
Maybe this is research idea, maybe it is not well-formed whatever it is, please
keep it in a branch.
# Then, and only then I get a comment which says, "This just an idea", we are
not planning to commit it.
# If you have 97 patches, How am I supposed to know that committing HDFS-13312
is not to further this cause and this is just an idea? If this is just an idea,
why would you want to commit that other patch with the reasoning that it helps
this patch?
# And now we have decided to abandon HDFS-13312? I am confused about what
state this work is in.
Overall, this JIRA does a very poor job of communicating what they want to do.
*That is the core issue*.
As I said, the people involved seem to understand (or at least I hope) what
they are doing, but since they are not very keen on communicating details, I am
proposing that you move all this work to a branch and bring it back when the
whole idea is baked. I am really trying to be helpful here.
{quote}you should look at the code/JIRAs if you're going to make sweeping
statements like this. Many features did have design documents.
{quote}
I hope I have given you examples of what triggered this statement. Other than
Quota work, there is no design documents. If you have them please do feel free
to share. Perhaps I am mistaken.
We have lots of ideas sprinkled in these 97 JIRAs. I have not read thru all of
them, but I hope you agree that these JIRAs
collectively constitute a forest and not few trees, And may I also point out
that it was not my decision to club them together.
{quote}The only data point you've cited is the number of subtasks in this JIRA,
which is a clerical problem, not a stop-the-world emergency.
{quote}
Disagree, Go thru the list of ideas discussed in various JIRAs. I am not
concerned with the number of JIRAs ONLY. That is just a tip of the Ice-berg.
{quote}Let's wrap up this issue as whatever was released, then transition to
(a) significant features in branches with subtasks and (b) normal maintenance
in individual JIRAs.
{quote}
Here is my take on it, +We should have opened a branch, long time ago+. We as a
community failed to catch it. I also know that all contributors on this branch
are working hard. I am suggesting a path which would minimize pain for the
contributors. I am keen on moving all unreleased JIRAs to a branch and
continuing work there.
I am fine (or let us say I am making peace with it) with whatever happened, but
let us not use that as an excuse to continue down this path.
Thanks for being patient and reading through this long response.
> Router-based HDFS federation phase 2
> ------------------------------------
>
> Key: HDFS-12615
> URL: https://issues.apache.org/jira/browse/HDFS-12615
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Íñigo Goiri
> Assignee: Íñigo Goiri
> Priority: Major
> Labels: RBF
>
> This umbrella JIRA tracks set of improvements over the Router-based HDFS
> federation (HDFS-10467).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
