[jira] [Commented] (HDFS-2983) Relax the build version check to permit rolling upgrades within a release
[ https://issues.apache.org/jira/browse/HDFS-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13249687#comment-13249687 ] Hadoop QA commented on HDFS-2983: - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521907/HDFS-2983.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2227//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2227//console This message is automatically generated. > Relax the build version check to permit rolling upgrades within a release > - > > Key: HDFS-2983 > URL: https://issues.apache.org/jira/browse/HDFS-2983 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Eli Collins >Assignee: Aaron T. Myers > Attachments: HDFS-2983.patch, HDFS-2983.patch, HDFS-2983.patch, > HDFS-2983.patch > > > Currently the version check for DN/NN communication is strict (it checks the > exact svn revision or git hash, Storage#getBuildVersion calls > VersionInfo#getRevision), which prevents rolling upgrades across any > releases. Once we have the PB-base RPC in place (coming soon to branch-23) > we'll have the necessary pieces in place to loosen this restriction, though > perhaps it takes another 23 minor release or so before we're ready to commit > to making the minor versions compatible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2983) Relax the build version check to permit rolling upgrades within a release
[ https://issues.apache.org/jira/browse/HDFS-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-2983: - Attachment: HDFS-2983.patch Thanks a lot for the review, Todd. Here's an updated patch which addresses all of your feedback. > Relax the build version check to permit rolling upgrades within a release > - > > Key: HDFS-2983 > URL: https://issues.apache.org/jira/browse/HDFS-2983 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Eli Collins >Assignee: Aaron T. Myers > Attachments: HDFS-2983.patch, HDFS-2983.patch, HDFS-2983.patch, > HDFS-2983.patch > > > Currently the version check for DN/NN communication is strict (it checks the > exact svn revision or git hash, Storage#getBuildVersion calls > VersionInfo#getRevision), which prevents rolling upgrades across any > releases. Once we have the PB-base RPC in place (coming soon to branch-23) > we'll have the necessary pieces in place to loosen this restriction, though > perhaps it takes another 23 minor release or so before we're ready to commit > to making the minor versions compatible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3214) InterDatanodeProtocolServerSideTranslatorPB doesn't handle null response from initReplicaRecovery
[ https://issues.apache.org/jira/browse/HDFS-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13249669#comment-13249669 ] Hudson commented on HDFS-3214: -- Integrated in Hadoop-Mapreduce-trunk-Commit #2038 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2038/]) HDFS-3214. InterDatanodeProtocolServerSideTranslatorPB doesn't handle null response from initReplicaRecovery. Contributed by Todd Lipcon. (Revision 1311125) Result = ABORTED todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311125 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/InterDatanodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/InterDatanodeProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/InterDatanodeProtocol.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestInterDatanodeProtocol.java > InterDatanodeProtocolServerSideTranslatorPB doesn't handle null response from > initReplicaRecovery > - > > Key: HDFS-3214 > URL: https://issues.apache.org/jira/browse/HDFS-3214 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node >Affects Versions: 2.0.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Blocker > Fix For: 2.0.0 > > Attachments: hdfs-3214.txt, hdfs-3214.txt > > > The initReplicaRecovery function may return null to indicate that the block > doesn't exist on the local node. However, the translator doesn't handle this > case, which results in NPEs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3214) InterDatanodeProtocolServerSideTranslatorPB doesn't handle null response from initReplicaRecovery
[ https://issues.apache.org/jira/browse/HDFS-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13249660#comment-13249660 ] Hudson commented on HDFS-3214: -- Integrated in Hadoop-Common-trunk-Commit #2027 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2027/]) HDFS-3214. InterDatanodeProtocolServerSideTranslatorPB doesn't handle null response from initReplicaRecovery. Contributed by Todd Lipcon. (Revision 1311125) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311125 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/InterDatanodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/InterDatanodeProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/InterDatanodeProtocol.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestInterDatanodeProtocol.java > InterDatanodeProtocolServerSideTranslatorPB doesn't handle null response from > initReplicaRecovery > - > > Key: HDFS-3214 > URL: https://issues.apache.org/jira/browse/HDFS-3214 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node >Affects Versions: 2.0.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Blocker > Fix For: 2.0.0 > > Attachments: hdfs-3214.txt, hdfs-3214.txt > > > The initReplicaRecovery function may return null to indicate that the block > doesn't exist on the local node. However, the translator doesn't handle this > case, which results in NPEs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3214) InterDatanodeProtocolServerSideTranslatorPB doesn't handle null response from initReplicaRecovery
[ https://issues.apache.org/jira/browse/HDFS-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13249659#comment-13249659 ] Hudson commented on HDFS-3214: -- Integrated in Hadoop-Hdfs-trunk-Commit #2102 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2102/]) HDFS-3214. InterDatanodeProtocolServerSideTranslatorPB doesn't handle null response from initReplicaRecovery. Contributed by Todd Lipcon. (Revision 1311125) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311125 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/InterDatanodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/InterDatanodeProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/InterDatanodeProtocol.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestInterDatanodeProtocol.java > InterDatanodeProtocolServerSideTranslatorPB doesn't handle null response from > initReplicaRecovery > - > > Key: HDFS-3214 > URL: https://issues.apache.org/jira/browse/HDFS-3214 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node >Affects Versions: 2.0.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Blocker > Fix For: 2.0.0 > > Attachments: hdfs-3214.txt, hdfs-3214.txt > > > The initReplicaRecovery function may return null to indicate that the block > doesn't exist on the local node. However, the translator doesn't handle this > case, which results in NPEs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3214) InterDatanodeProtocolServerSideTranslatorPB doesn't handle null response from initReplicaRecovery
[ https://issues.apache.org/jira/browse/HDFS-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-3214: -- Resolution: Fixed Fix Version/s: 2.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed new version to trunk and branch-2. Thanks for reviewing. > InterDatanodeProtocolServerSideTranslatorPB doesn't handle null response from > initReplicaRecovery > - > > Key: HDFS-3214 > URL: https://issues.apache.org/jira/browse/HDFS-3214 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node >Affects Versions: 2.0.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Blocker > Fix For: 2.0.0 > > Attachments: hdfs-3214.txt, hdfs-3214.txt > > > The initReplicaRecovery function may return null to indicate that the block > doesn't exist on the local node. However, the translator doesn't handle this > case, which results in NPEs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3214) InterDatanodeProtocolServerSideTranslatorPB doesn't handle null response from initReplicaRecovery
[ https://issues.apache.org/jira/browse/HDFS-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13249649#comment-13249649 ] Hadoop QA commented on HDFS-3214: - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521901/hdfs-3214.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2226//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2226//console This message is automatically generated. > InterDatanodeProtocolServerSideTranslatorPB doesn't handle null response from > initReplicaRecovery > - > > Key: HDFS-3214 > URL: https://issues.apache.org/jira/browse/HDFS-3214 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node >Affects Versions: 2.0.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Blocker > Attachments: hdfs-3214.txt, hdfs-3214.txt > > > The initReplicaRecovery function may return null to indicate that the block > doesn't exist on the local node. However, the translator doesn't handle this > case, which results in NPEs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2983) Relax the build version check to permit rolling upgrades within a release
[
https://issues.apache.org/jira/browse/HDFS-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13249646#comment-13249646
]
Todd Lipcon commented on HDFS-2983:
---
bq. Technically the quote characters inside the Javadoc should be " - or you
could just use single-quotes instead to avoid the hassle.
erg, JIRA went and formatted my explanation :) The quote characters should be
{{& quot;}} without the space.
> Relax the build version check to permit rolling upgrades within a release
> -
>
> Key: HDFS-2983
> URL: https://issues.apache.org/jira/browse/HDFS-2983
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Eli Collins
>Assignee: Aaron T. Myers
> Attachments: HDFS-2983.patch, HDFS-2983.patch, HDFS-2983.patch
>
>
> Currently the version check for DN/NN communication is strict (it checks the
> exact svn revision or git hash, Storage#getBuildVersion calls
> VersionInfo#getRevision), which prevents rolling upgrades across any
> releases. Once we have the PB-base RPC in place (coming soon to branch-23)
> we'll have the necessary pieces in place to loosen this restriction, though
> perhaps it takes another 23 minor release or so before we're ready to commit
> to making the minor versions compatible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2983) Relax the build version check to permit rolling upgrades within a release
[
https://issues.apache.org/jira/browse/HDFS-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13249645#comment-13249645
]
Todd Lipcon commented on HDFS-2983:
---
- Can VersionUtil be made abstract, since it only has static methods?
{code}
+ * This function splits the two versions on "." and performs a lexical
+ * comparison of the resulting components.
{code}
Technically the quote characters inside the Javadoc should be " - or you
could just use single-quotes instead to avoid the hassle.
VersionUtil should be doing numeric comparison rather than straight string
comparison. For example "10.0.0" should be considered greater than "2.0", but I
think the current implementation doesn't implement this correctly.
Please add a test for this case to TestVersionUtil as well.
{code}
+ private static void assertExpectedValues(String lower, String higher) {
+assertTrue(0 > VersionUtil.compareVersions(lower, higher));
+assertTrue(0 < VersionUtil.compareVersions(higher, lower));
+ }
{code}
These comparisons read backwards to me. ie should be:
{code}
+ private static void assertExpectedValues(String lower, String higher) {
+assertTrue(VersionUtil.compareVersions(lower, higher) < 0);
+assertTrue(VersionUtil.compareVersions(higher, lower) > 0);
+ }
{code}
don't you think?
{code}
+if (VersionUtil.compareVersions(dnVersion, minimumDataNodeVersion) < 0) {
+ IncorrectVersionException ive = new IncorrectVersionException(
+ minimumDataNodeVersion, dnVersion, "DataNode", "NameNode");
+ LOG.warn(ive.getMessage());
+ throw ive;
+}
{code}
Here, does the log message end up including the remote IP address somehow? If
not, I think we should improve it to include that (and maybe the stringified
DatanodeRegistration object)
> Relax the build version check to permit rolling upgrades within a release
> -
>
> Key: HDFS-2983
> URL: https://issues.apache.org/jira/browse/HDFS-2983
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Eli Collins
>Assignee: Aaron T. Myers
> Attachments: HDFS-2983.patch, HDFS-2983.patch, HDFS-2983.patch
>
>
> Currently the version check for DN/NN communication is strict (it checks the
> exact svn revision or git hash, Storage#getBuildVersion calls
> VersionInfo#getRevision), which prevents rolling upgrades across any
> releases. Once we have the PB-base RPC in place (coming soon to branch-23)
> we'll have the necessary pieces in place to loosen this restriction, though
> perhaps it takes another 23 minor release or so before we're ready to commit
> to making the minor versions compatible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3218) Use multiple remote DN interfaces for block transfer
[
https://issues.apache.org/jira/browse/HDFS-3218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13249641#comment-13249641
]
Todd Lipcon commented on HDFS-3218:
---
- I think it would make sense to add a utility function like
{{DFSUtil.getRandomXferAddress(DatanodeID)}}, since you have a lot of
repetition fo the {{DFSUtil.getRandom().nextInt}} stuff. Or even make it a
member function of the DatanodeID?
Otherwise looks good.
> Use multiple remote DN interfaces for block transfer
>
>
> Key: HDFS-3218
> URL: https://issues.apache.org/jira/browse/HDFS-3218
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs client
>Reporter: Eli Collins
>Assignee: Eli Collins
> Attachments: hdfs-3218.txt
>
>
> HDFS-3146 and HDFS-3216 expose multiple DN interfaces to the client. In order
> for clients, in aggregate, to use multiple DN interfaces clients should pick
> different interfaces when transferring blocks. Given that we cache client <->
> DN connections the policy of picking a remote interface at random for each
> new connection seems best (vs round robin for example). In the future we
> could make the client congestion aware. We could also establish multiple
> connections between the client and DN and therefore use multiple interfaces
> for a single block transfer. Both of those are out of scope for this jira.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3216) DatanodeID should support multiple IP addresses
[
https://issues.apache.org/jira/browse/HDFS-3216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13249640#comment-13249640
]
Todd Lipcon commented on HDFS-3216:
---
Should we deprecate this function? Or do we need some concept of the
"canonical"/"main" IP address? If the latter, we should explain this in the
javadoc of this function.
{code}
public String getIpAddr() {
-return ipAddr;
+return ipAddrs[0];
+ }
{code}
- is it ever valid to construct a DatanodeID with no IP addresses? If not we
should add a Preconditions check or at least an assert on the length of the
ipAddrs array in the constructor and the setter
{code}
+return new DatanodeID(ipAddrs.toArray(new String[ipAddrs.size()]) ,
dn.getHostName(), dn.getStorageID(),
dn.getXferPort(), dn.getInfoPort(), dn.getIpcPort());
{code}
Can you re-wrap this to 80chars?
- Is the code change in JsonUtil covered by TestJsonUtil? (are you sure that
the cast to String[] is right?)
- in some of the tests, it's filling in hostnames instead of IPs for the
ipAddrs field. Is that right, or do we expect that it will always be resolved
IPs? The dual nature makes me nervous.
> DatanodeID should support multiple IP addresses
> ---
>
> Key: HDFS-3216
> URL: https://issues.apache.org/jira/browse/HDFS-3216
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Eli Collins
>Assignee: Eli Collins
> Attachments: hdfs-3216.txt
>
>
> The DatanodeID has a single field for the IP address, for HDFS-3146 we need
> to extend it to support multiple addresses.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3214) InterDatanodeProtocolServerSideTranslatorPB doesn't handle null response from initReplicaRecovery
[ https://issues.apache.org/jira/browse/HDFS-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-3214: -- Attachment: hdfs-3214.txt Good point. New patch switches the ids to be sequential by their declaration order > InterDatanodeProtocolServerSideTranslatorPB doesn't handle null response from > initReplicaRecovery > - > > Key: HDFS-3214 > URL: https://issues.apache.org/jira/browse/HDFS-3214 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node >Affects Versions: 2.0.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Blocker > Attachments: hdfs-3214.txt, hdfs-3214.txt > > > The initReplicaRecovery function may return null to indicate that the block > doesn't exist on the local node. However, the translator doesn't handle this > case, which results in NPEs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3146) Datanode should be able to register multiple network interfaces
[
https://issues.apache.org/jira/browse/HDFS-3146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13249638#comment-13249638
]
Todd Lipcon commented on HDFS-3146:
---
{code}
+ public static InetSocketAddress[] getInterfaceAddrs(
+ String interfaceNames[], int port) throws UnknownHostException {
{code}
Sorry I missed this in the earlier review of this function, but I think it
would be better to call the parameter something like {{interfaceSpecs}} --
because each one may specify an interface name, an IP address, or a subnet.
In the same function, for the subnet case, you're using port 0 instead of the
specified port. Looks like a mistake?
{code}
+ LOG.warn("Invalid address given " + addrString);
{code}
Nit: add a ':' to the log message
{code}
+// If the datanode registered with an address we can't use
+// then use the address the IPC came in on instead
+if (NetUtils.isWildcardOrLoopback(nodeReg.getIpAddr())) {
{code}
I found this comment a little unclear. Under what circumstance would the DN
pass a loopback or wildcard IP? Aren't they filtered on the DN side? I think
this should be at least a WARN, or maybe even throw an exception to disallow
the registration.
Edit: I got to the part later in the patch where the DN potentially sends a
wildcard to the NN. I think it might be simplier to have the DN send an empty
list to the NN if it's bound to wildcard -- and adjust the comment here to
explain why it would be registering with no addresses.
{code}
+ // TODO: haven't determined the port yet, using default
{code}
Are you planning another patch to fix this on the branch before merging? What's
the backward-compatibility path with the existing configurations for bind
address, etc, where the port's specified? We should be clear about which takes
precedence, and throw errors on startup if both are configured, I think?
Maybe it makes sense to change these to just be InetAddress instead of
InetSocketAddress, and never fill in a port there?
This patch should add the new config to hdfs-default, and edit the existing
config's documentation to explain how the two interact.
{code}
+ if (0 != interfaceStrs.length) {
+LOG.info("Using interfaces [" +
+Joiner.on(',').join(interfaceStrs)+ "] with addresses [" +
+Joiner.on(',').join(interfaceAddrs) + "]");
+ }
{code}
- need indentation for the joiner lines
- add comment explaining how this eventually gets filled in, if it's empty?
{code}
+ * @param addrs socket addresses to convert
+ * @return an array of strings of IPs for the given addresses
+ */
+ public static String[] toIpAddrStrings(InetSocketAddress[] addrs) {
{code}
javadoc should specify that ports aren't included in the stringification of
addresses
> Datanode should be able to register multiple network interfaces
> ---
>
> Key: HDFS-3146
> URL: https://issues.apache.org/jira/browse/HDFS-3146
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
>Reporter: Eli Collins
>Assignee: Eli Collins
> Attachments: hdfs-3146.txt
>
>
> The Datanode should register multiple interfaces with the Namenode (who then
> forwards them to clients). We can do this by extending the DatanodeID, which
> currently just contains a single interface, to contain a list of interfaces.
> For compatibility, the DatanodeID method to get the DN address for data
> transfer should remain unchanged (multiple interfaces are only used where the
> client explicitly takes advantage of them).
> By default, if the Datanode binds on all interfaces (via using the wildcard
> in the dfs*address configuration) all interfaces are exposed, modulo ones
> like the loopback that should never be exposed. Alternatively, a new
> configuration parameter ({{dfs.datanode.available.interfaces}}) allows the
> set of interfaces can be specified explicitly in case the user only wants to
> expose a subset. If the new default behavior is too disruptive we could
> default dfs.datanode.available.interfaces to be the IP of the IPC interface
> which is the only interface exposed today (per HADOOP-6867, only the port
> from dfs.datanode.address is used today).
> The interfaces can be specified by name (eg "eth0"), subinterface name (eg
> "eth0:0"), or IP address. The IP address can be specified by range using CIDR
> notation so the configuration values are portable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2983) Relax the build version check to permit rolling upgrades within a release
[
https://issues.apache.org/jira/browse/HDFS-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated HDFS-2983:
-
Attachment: HDFS-2983.patch
Here's an updated patch which should fix TestNNThroughputBenchmark. The only
difference between this patch and the last is the following:
{noformat}
diff --git
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroug
index 4b8f225..ec5b8a7 100644
---
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java
+++
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java
@@ -58,6 +58,7 @@ import org.apache.hadoop.net.DNS;
import org.apache.hadoop.net.NetworkTopology;
import org.apache.hadoop.security.Groups;
import org.apache.hadoop.util.StringUtils;
+import org.apache.hadoop.util.VersionInfo;
import org.apache.log4j.Level;
import org.apache.log4j.LogManager;
@@ -783,6 +784,7 @@ public class NNThroughputBenchmark {
String hostName = DNS.getDefaultHost("default", "default");
dnRegistration = new DatanodeRegistration(ipAddr, getNodePort(dnIdx));
dnRegistration.setHostName(hostName);
+ dnRegistration.setSoftwareVersion(VersionInfo.getVersion());
this.blocks = new ArrayList(blockCapacity);
this.nrBlocks = 0;
}
{noformat}
> Relax the build version check to permit rolling upgrades within a release
> -
>
> Key: HDFS-2983
> URL: https://issues.apache.org/jira/browse/HDFS-2983
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Eli Collins
>Assignee: Aaron T. Myers
> Attachments: HDFS-2983.patch, HDFS-2983.patch, HDFS-2983.patch
>
>
> Currently the version check for DN/NN communication is strict (it checks the
> exact svn revision or git hash, Storage#getBuildVersion calls
> VersionInfo#getRevision), which prevents rolling upgrades across any
> releases. Once we have the PB-base RPC in place (coming soon to branch-23)
> we'll have the necessary pieces in place to loosen this restriction, though
> perhaps it takes another 23 minor release or so before we're ready to commit
> to making the minor versions compatible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2983) Relax the build version check to permit rolling upgrades within a release
[ https://issues.apache.org/jira/browse/HDFS-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13249531#comment-13249531 ] Hadoop QA commented on HDFS-2983: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521882/HDFS-2983.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.server.namenode.TestNNThroughputBenchmark +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2224//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2224//console This message is automatically generated. > Relax the build version check to permit rolling upgrades within a release > - > > Key: HDFS-2983 > URL: https://issues.apache.org/jira/browse/HDFS-2983 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Eli Collins >Assignee: Aaron T. Myers > Attachments: HDFS-2983.patch, HDFS-2983.patch > > > Currently the version check for DN/NN communication is strict (it checks the > exact svn revision or git hash, Storage#getBuildVersion calls > VersionInfo#getRevision), which prevents rolling upgrades across any > releases. Once we have the PB-base RPC in place (coming soon to branch-23) > we'll have the necessary pieces in place to loosen this restriction, though > perhaps it takes another 23 minor release or so before we're ready to commit > to making the minor versions compatible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2983) Relax the build version check to permit rolling upgrades within a release
[ https://issues.apache.org/jira/browse/HDFS-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-2983: - Attachment: HDFS-2983.patch Here's an updated patch which implements Todd's proposal. > Relax the build version check to permit rolling upgrades within a release > - > > Key: HDFS-2983 > URL: https://issues.apache.org/jira/browse/HDFS-2983 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Eli Collins >Assignee: Aaron T. Myers > Attachments: HDFS-2983.patch, HDFS-2983.patch > > > Currently the version check for DN/NN communication is strict (it checks the > exact svn revision or git hash, Storage#getBuildVersion calls > VersionInfo#getRevision), which prevents rolling upgrades across any > releases. Once we have the PB-base RPC in place (coming soon to branch-23) > we'll have the necessary pieces in place to loosen this restriction, though > perhaps it takes another 23 minor release or so before we're ready to commit > to making the minor versions compatible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3214) InterDatanodeProtocolServerSideTranslatorPB doesn't handle null response from initReplicaRecovery
[ https://issues.apache.org/jira/browse/HDFS-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13249511#comment-13249511 ] Aaron T. Myers commented on HDFS-3214: -- Patch looks pretty good to me, Todd. One small comment: How about changing the numeric tags to be numerically increasing in InitReplicaRecoveryResponseProto in InterDatanodeProtocol.proto? Considering there's so far been no release of HDFS with PB support, there doesn't seem to be any reason to concern ourselves with maintaining PB backward compatibility. +1 once this is addressed, either by accepting the feedback or explaining your reasoning not to. > InterDatanodeProtocolServerSideTranslatorPB doesn't handle null response from > initReplicaRecovery > - > > Key: HDFS-3214 > URL: https://issues.apache.org/jira/browse/HDFS-3214 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node >Affects Versions: 2.0.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Blocker > Attachments: hdfs-3214.txt > > > The initReplicaRecovery function may return null to indicate that the block > doesn't exist on the local node. However, the translator doesn't handle this > case, which results in NPEs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
