[jira] [Commented] (HDFS-3015) NamenodeFsck and JspHelper duplicate DFSInputStream#copyBlock and bestNode
[
https://issues.apache.org/jira/browse/HDFS-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13237798#comment-13237798
]
Muddy Dixon commented on HDFS-3015:
---
I surveys NamenodeFsck, JspHelper, DFSInputStream and DFSClient (which referred
in NamenodeFsck#copyBlock comment).
copyBlock is declared in
* NamenodeFsck
bestNode is declared in
* NamenodeFsck
* DFSInputStream
* JspHelper (3 arg patterns)
At first, copyBlock is not declared duplicate.
And then, all implementations of bestNode have different arg patterns and
implementation.
So these are not duplicated.
NamenodeFsck#bestNode and DFSInputStream#bestNode are similar implementation.
NamenodeFsck
{code}
private DatanodeInfo bestNode(DFSClient dfs, DatanodeInfo[] nodes,
TreeSetDatanodeInfo deadNodes) throws
IOException {
if ((nodes == null) ||
(nodes.length - deadNodes.size() 1)) {
throw new IOException(No live nodes contain current block);
}
DatanodeInfo chosenNode;
do {
chosenNode = nodes[DFSUtil.getRandom().nextInt(nodes.length)];
} while (deadNodes.contains(chosenNode));
return chosenNode;
}
{code}
DFSInputStream#bestNode
{code}
static DatanodeInfo bestNode(DatanodeInfo nodes[],
AbstractMapDatanodeInfo, DatanodeInfo
deadNodes)
throws IOException {
if (nodes != null) {
for (int i = 0; i nodes.length; i++) {
if (!deadNodes.containsKey(nodes[i])) {
return nodes[i];
}
}
}
throw new IOException(No live nodes contain current block);
}
{code}
NamenodeFsck and JspHelper duplicate DFSInputStream#copyBlock and bestNode
--
Key: HDFS-3015
URL: https://issues.apache.org/jira/browse/HDFS-3015
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Eli Collins
Priority: Minor
Labels: newbie
Both NamenodeFsck and JspHelper duplicate DFSInputStream#copyBlock and
bestNode. There should be one shared implementation.
{code}
/*
* XXX (ab) Bulk of this method is copied verbatim from {@link DFSClient},
which is
* bad. Both places should be refactored to provide a method to copy blocks
* around.
*/
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-342) Reduce time taken to run TestUnderReplicatedBlocks
[
https://issues.apache.org/jira/browse/HDFS-342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13237799#comment-13237799
]
Sho Shimauchi commented on HDFS-342:
I ran this test on trunk but it finished within 2 minutes.
{quote}
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 1:27.990s
[INFO] Finished at: Sun Mar 25 17:15:00 JST 2012
[INFO] Final Memory: 43M/123M
[INFO]
{quote}
Is this still an issue?
Reduce time taken to run TestUnderReplicatedBlocks
---
Key: HDFS-342
URL: https://issues.apache.org/jira/browse/HDFS-342
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Tom White
Labels: newbie
TestUnderReplicatedBlocks (introduced in HADOOP-4598) takes around 10 minutes
to run, which seems excessive. Can we write a test for this behaviour which
runs in significantly less time (e.g. less than a minute)?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3116) Typo in fetchdt error message
[
https://issues.apache.org/jira/browse/HDFS-3116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13237802#comment-13237802
]
Aaron T. Myers commented on HDFS-3116:
--
+1, the patch looks good to me. I'm quite confident that the test failures are
unrelated. No test required as it's just changing a log message.
I'm going to commit this momentarily.
Typo in fetchdt error message
-
Key: HDFS-3116
URL: https://issues.apache.org/jira/browse/HDFS-3116
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs client
Affects Versions: 0.24.0
Reporter: Aaron T. Myers
Assignee: AOE Takashi
Priority: Trivial
Labels: newbie
Attachments: HDFS-3116.txt
In {{DelegationTokenFetcher.java}} there's the following typo of the word
exactly:
{code}
System.err.println(ERROR: Must specify exacltly one token file);
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3116) Typo in fetchdt error message
[
https://issues.apache.org/jira/browse/HDFS-3116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated HDFS-3116:
-
Resolution: Fixed
Fix Version/s: 0.24.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I've just committed this to trunk. Thanks a lot for the contribution, AOE!
Typo in fetchdt error message
-
Key: HDFS-3116
URL: https://issues.apache.org/jira/browse/HDFS-3116
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs client
Affects Versions: 0.24.0
Reporter: Aaron T. Myers
Assignee: AOE Takashi
Priority: Trivial
Labels: newbie
Fix For: 0.24.0
Attachments: HDFS-3116.txt
In {{DelegationTokenFetcher.java}} there's the following typo of the word
exactly:
{code}
System.err.println(ERROR: Must specify exacltly one token file);
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3116) Typo in fetchdt error message
[
https://issues.apache.org/jira/browse/HDFS-3116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13237805#comment-13237805
]
Hudson commented on HDFS-3116:
--
Integrated in Hadoop-Common-trunk-Commit #1925 (See
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1925/])
HDFS-3116. Typo in fetchdt error message. Contributed by AOE Takashi.
(Revision 1304996)
Result = SUCCESS
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1304996
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DelegationTokenFetcher.java
Typo in fetchdt error message
-
Key: HDFS-3116
URL: https://issues.apache.org/jira/browse/HDFS-3116
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs client
Affects Versions: 0.24.0
Reporter: Aaron T. Myers
Assignee: AOE Takashi
Priority: Trivial
Labels: newbie
Fix For: 0.24.0
Attachments: HDFS-3116.txt
In {{DelegationTokenFetcher.java}} there's the following typo of the word
exactly:
{code}
System.err.println(ERROR: Must specify exacltly one token file);
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3116) Typo in fetchdt error message
[
https://issues.apache.org/jira/browse/HDFS-3116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13237804#comment-13237804
]
Hudson commented on HDFS-3116:
--
Integrated in Hadoop-Hdfs-trunk-Commit #1999 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1999/])
HDFS-3116. Typo in fetchdt error message. Contributed by AOE Takashi.
(Revision 1304996)
Result = SUCCESS
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1304996
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DelegationTokenFetcher.java
Typo in fetchdt error message
-
Key: HDFS-3116
URL: https://issues.apache.org/jira/browse/HDFS-3116
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs client
Affects Versions: 0.24.0
Reporter: Aaron T. Myers
Assignee: AOE Takashi
Priority: Trivial
Labels: newbie
Fix For: 0.24.0
Attachments: HDFS-3116.txt
In {{DelegationTokenFetcher.java}} there's the following typo of the word
exactly:
{code}
System.err.println(ERROR: Must specify exacltly one token file);
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3116) Typo in fetchdt error message
[
https://issues.apache.org/jira/browse/HDFS-3116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13237810#comment-13237810
]
Hudson commented on HDFS-3116:
--
Integrated in Hadoop-Mapreduce-trunk-Commit #1935 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1935/])
HDFS-3116. Typo in fetchdt error message. Contributed by AOE Takashi.
(Revision 1304996)
Result = ABORTED
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1304996
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DelegationTokenFetcher.java
Typo in fetchdt error message
-
Key: HDFS-3116
URL: https://issues.apache.org/jira/browse/HDFS-3116
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs client
Affects Versions: 0.24.0
Reporter: Aaron T. Myers
Assignee: AOE Takashi
Priority: Trivial
Labels: newbie
Fix For: 0.24.0
Attachments: HDFS-3116.txt
In {{DelegationTokenFetcher.java}} there's the following typo of the word
exactly:
{code}
System.err.println(ERROR: Must specify exacltly one token file);
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3116) Typo in fetchdt error message
[
https://issues.apache.org/jira/browse/HDFS-3116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13237851#comment-13237851
]
Hudson commented on HDFS-3116:
--
Integrated in Hadoop-Hdfs-trunk #995 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/995/])
HDFS-3116. Typo in fetchdt error message. Contributed by AOE Takashi.
(Revision 1304996)
Result = FAILURE
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1304996
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DelegationTokenFetcher.java
Typo in fetchdt error message
-
Key: HDFS-3116
URL: https://issues.apache.org/jira/browse/HDFS-3116
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs client
Affects Versions: 0.24.0
Reporter: Aaron T. Myers
Assignee: AOE Takashi
Priority: Trivial
Labels: newbie
Fix For: 0.24.0
Attachments: HDFS-3116.txt
In {{DelegationTokenFetcher.java}} there's the following typo of the word
exactly:
{code}
System.err.println(ERROR: Must specify exacltly one token file);
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3116) Typo in fetchdt error message
[
https://issues.apache.org/jira/browse/HDFS-3116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13237867#comment-13237867
]
Hudson commented on HDFS-3116:
--
Integrated in Hadoop-Mapreduce-trunk #1030 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1030/])
HDFS-3116. Typo in fetchdt error message. Contributed by AOE Takashi.
(Revision 1304996)
Result = SUCCESS
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1304996
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DelegationTokenFetcher.java
Typo in fetchdt error message
-
Key: HDFS-3116
URL: https://issues.apache.org/jira/browse/HDFS-3116
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs client
Affects Versions: 0.24.0
Reporter: Aaron T. Myers
Assignee: AOE Takashi
Priority: Trivial
Labels: newbie
Fix For: 0.24.0
Attachments: HDFS-3116.txt
In {{DelegationTokenFetcher.java}} there's the following typo of the word
exactly:
{code}
System.err.println(ERROR: Must specify exacltly one token file);
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2413) Add public APIs for safemode
[ https://issues.apache.org/jira/browse/HDFS-2413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated HDFS-2413: -- Attachment: HDFS-2413.patch Sounds good Nicholas. I updated the patch to add that in. Add public APIs for safemode Key: HDFS-2413 URL: https://issues.apache.org/jira/browse/HDFS-2413 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs client Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Harsh J Attachments: HDFS-2413.patch, HDFS-2413.patch, HDFS-2413.patch Currently the APIs for safe-mode are part of DistributedFileSystem, which is supposed to be a private interface. However, dependent software often wants to wait until the NN is out of safemode. Though it could poll trying to create a file and catching SafeModeException, we should consider making some of these APIs public. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2941) Add an administrative command to download a copy of the fsimage from the NN
[ https://issues.apache.org/jira/browse/HDFS-2941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13237942#comment-13237942 ] Aaron T. Myers commented on HDFS-2941: -- I'm confident that the failed tests are unrelated to this patch. Add an administrative command to download a copy of the fsimage from the NN --- Key: HDFS-2941 URL: https://issues.apache.org/jira/browse/HDFS-2941 Project: Hadoop HDFS Issue Type: New Feature Components: hdfs client, name-node Affects Versions: 0.24.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Attachments: HDFS-2941.patch, HDFS-2941.patch It would be nice to be able to download a copy of the fsimage from the NN, e.g. for backup purposes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3145) Disallow self failover
Disallow self failover
--
Key: HDFS-3145
URL: https://issues.apache.org/jira/browse/HDFS-3145
Project: Hadoop HDFS
Issue Type: Bug
Components: ha
Reporter: Eli Collins
Assignee: Eli Collins
It is currently possible for users to make a standby NameNode failover to
itself and become active. We shouldn't allow this to happen in case operators
mistype and miss the fact that there are now two active NNs.
{noformat}
bash-4.1$ hdfs haadmin -ns ha-nn-uri -failover nn2 nn2
Failover from nn2 to nn2 successful
{noformat}
After the failover above, nn2 will be active.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3145) Disallow self failover
[
https://issues.apache.org/jira/browse/HDFS-3145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Collins updated HDFS-3145:
--
Attachment: hdfs-3145.txt
Patch attached.
Disallow self failover
--
Key: HDFS-3145
URL: https://issues.apache.org/jira/browse/HDFS-3145
Project: Hadoop HDFS
Issue Type: Bug
Components: ha
Reporter: Eli Collins
Assignee: Eli Collins
Attachments: hdfs-3145.txt
It is currently possible for users to make a standby NameNode failover to
itself and become active. We shouldn't allow this to happen in case operators
mistype and miss the fact that there are now two active NNs.
{noformat}
bash-4.1$ hdfs haadmin -ns ha-nn-uri -failover nn2 nn2
Failover from nn2 to nn2 successful
{noformat}
After the failover above, nn2 will be active.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3145) Disallow self failover
[
https://issues.apache.org/jira/browse/HDFS-3145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated HDFS-3145:
-
Affects Version/s: 0.23.2
Status: Patch Available (was: Open)
Disallow self failover
--
Key: HDFS-3145
URL: https://issues.apache.org/jira/browse/HDFS-3145
Project: Hadoop HDFS
Issue Type: Bug
Components: ha
Affects Versions: 0.23.2
Reporter: Eli Collins
Assignee: Eli Collins
Attachments: hdfs-3145.txt
It is currently possible for users to make a standby NameNode failover to
itself and become active. We shouldn't allow this to happen in case operators
mistype and miss the fact that there are now two active NNs.
{noformat}
bash-4.1$ hdfs haadmin -ns ha-nn-uri -failover nn2 nn2
Failover from nn2 to nn2 successful
{noformat}
After the failover above, nn2 will be active.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2413) Add public APIs for safemode
[ https://issues.apache.org/jira/browse/HDFS-2413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13237956#comment-13237956 ] Hadoop QA commented on HDFS-2413: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12519869/HDFS-2413.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.tools.TestDFSHAAdmin org.apache.hadoop.hdfs.TestGetBlocks org.apache.hadoop.hdfs.tools.TestDFSHAAdminMiniCluster org.apache.hadoop.cli.TestHDFSCLI +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2095//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2095//console This message is automatically generated. Add public APIs for safemode Key: HDFS-2413 URL: https://issues.apache.org/jira/browse/HDFS-2413 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs client Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Harsh J Attachments: HDFS-2413.patch, HDFS-2413.patch, HDFS-2413.patch Currently the APIs for safe-mode are part of DistributedFileSystem, which is supposed to be a private interface. However, dependent software often wants to wait until the NN is out of safemode. Though it could poll trying to create a file and catching SafeModeException, we should consider making some of these APIs public. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2413) Add public APIs for safemode
[ https://issues.apache.org/jira/browse/HDFS-2413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13237962#comment-13237962 ] Harsh J commented on HDFS-2413: --- Per https://builds.apache.org/job/PreCommit-HDFS-Build/2095//testReport/, all reported failures appear unrelated. The HA tests have been failing for a while now (possible misconfig?) and the other ones have failed since 3 runs before (And are unrelated to this safe mode wrapper change, as noticeable in org.apache.hadoop.hdfs.TestGetBlocks.getBlocksWithException). Add public APIs for safemode Key: HDFS-2413 URL: https://issues.apache.org/jira/browse/HDFS-2413 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs client Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Harsh J Attachments: HDFS-2413.patch, HDFS-2413.patch, HDFS-2413.patch Currently the APIs for safe-mode are part of DistributedFileSystem, which is supposed to be a private interface. However, dependent software often wants to wait until the NN is out of safemode. Though it could poll trying to create a file and catching SafeModeException, we should consider making some of these APIs public. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3146) Datanode should be able to register multiple network interfaces
Datanode should be able to register multiple network interfaces
---
Key: HDFS-3146
URL: https://issues.apache.org/jira/browse/HDFS-3146
Project: Hadoop HDFS
Issue Type: Sub-task
Components: data-node
Reporter: Eli Collins
Assignee: Eli Collins
The Datanode should register multiple interfaces with the Namenode (who then
forwards them to clients). We can do this by extending the DatanodeID, which
currently just contains a single interface, to contain a list of interfaces.
For compatibility, the DatanodeID method to get the DN address for data
transfer should remain unchanged (multiple interfaces are only used where the
client explicitly takes advantage of them).
By default, if the Datanode binds on all interfaces (via using the wildcard in
the dfs*address configuration) all interfaces are exposed, modulo ones like the
loopback that should never be exposed. Alternatively, a new configuration
parameter ({{dfs.datanode.available.interfaces}}) allows the set of interfaces
can be specified explicitly in case the user only wants to expose a subset. If
the new default behavior is too disruptive we could default
dfs.datanode.available.interfaces to be the IP of the IPC interface which is
the only interface exposed today (per HADOOP-6867, only the port from
dfs.datanode.address is used today).
The interfaces can be specified by name (eg eth0), subinterface name (eg
eth0:0), or IP address. The IP address can be specified by range using CIDR
notation so the configuration values are portable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3147) The client should be able to specify which network interfaces to use
The client should be able to specify which network interfaces to use
Key: HDFS-3147
URL: https://issues.apache.org/jira/browse/HDFS-3147
Project: Hadoop HDFS
Issue Type: Sub-task
Components: data-node
Reporter: Eli Collins
Assignee: Eli Collins
HDFS-3146 exposes multiple interfaces to the client. However, not all
interfaces exposed to clients should be used, eg because not all addresses
given to clients may be routable by the client, or a user may want to restrict
off-cluster clients from using cluster-private interfaces. Therefore the user
should be able to configure clients to use a subset of the addresses they are
given. This can be accomplished by a new configuration option
({{dfs.client.available.interfaces}}) that takes a list of interfaces to use,
interfaces that don't match the configuration are ignored. Acceptable
configuration values are the same as the {{dfs.datanode.available.interfaces}}
parameter. In addition, we could also add an option where clients automatically
check if they can connect to each interface that's given them, and filter those
out by default.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3147) The client should be able to specify which network interfaces to use
[
https://issues.apache.org/jira/browse/HDFS-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Collins updated HDFS-3147:
--
Component/s: (was: data-node)
hdfs client
Description: HDFS-3146 exposes multiple Datanode interfaces to the client.
However, not all interfaces exposed to clients should be used, eg because not
all addresses given to clients may be routable by the client, or a user may
want to restrict off-cluster clients from using cluster-private interfaces.
Therefore the user should be able to configure clients to use a subset of the
addresses they are given. This can be accomplished by a new configuration
option ({{dfs.client.available.interfaces}}) that takes a list of interfaces to
use, interfaces that don't match the configuration are ignored. Acceptable
configuration values are the same as the {{dfs.datanode.available.interfaces}}
parameter. In addition, we could also add an option where clients automatically
check if they can connect to each interface that's given them, and filter those
out by default. (was: HDFS-3146 exposes multiple interfaces to the client.
However, not all interfaces exposed to clients should be used, eg because not
all addresses given to clients may be routable by the client, or a user may
want to restrict off-cluster clients from using cluster-private interfaces.
Therefore the user should be able to configure clients to use a subset of the
addresses they are given. This can be accomplished by a new configuration
option ({{dfs.client.available.interfaces}}) that takes a list of interfaces to
use, interfaces that don't match the configuration are ignored. Acceptable
configuration values are the same as the {{dfs.datanode.available.interfaces}}
parameter. In addition, we could also add an option where clients automatically
check if they can connect to each interface that's given them, and filter those
out by default.)
The client should be able to specify which network interfaces to use
Key: HDFS-3147
URL: https://issues.apache.org/jira/browse/HDFS-3147
Project: Hadoop HDFS
Issue Type: Sub-task
Components: hdfs client
Reporter: Eli Collins
Assignee: Eli Collins
HDFS-3146 exposes multiple Datanode interfaces to the client. However, not
all interfaces exposed to clients should be used, eg because not all
addresses given to clients may be routable by the client, or a user may want
to restrict off-cluster clients from using cluster-private interfaces.
Therefore the user should be able to configure clients to use a subset of the
addresses they are given. This can be accomplished by a new configuration
option ({{dfs.client.available.interfaces}}) that takes a list of interfaces
to use, interfaces that don't match the configuration are ignored. Acceptable
configuration values are the same as the
{{dfs.datanode.available.interfaces}} parameter. In addition, we could also
add an option where clients automatically check if they can connect to each
interface that's given them, and filter those out by default.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3148) The client should be able to use multiple local interfaces for data transfer
The client should be able to use multiple local interfaces for data transfer
Key: HDFS-3148
URL: https://issues.apache.org/jira/browse/HDFS-3148
Project: Hadoop HDFS
Issue Type: Sub-task
Components: hdfs client
Reporter: Eli Collins
Assignee: Eli Collins
HDFS-3147 covers using multiple interfaces on the server (Datanode) side.
Clients should also be able to utilize multiple *local* interfaces for outbound
connections instead of always using the interface for the local hostname. This
can be accomplished with a new configuration parameter
({{dfs.client.local.interfaces}}) that accepts a list of interfaces the client
should use. Acceptable configuration values are the same as the
{{dfs.datanode.available.interfaces}} parameter. The client binds its socket to
a specific interface, which enables outbound traffic to use that interface.
Binding the client socket to a specific address is not sufficient to ensure
egress traffic uses that interface. Eg if multiple interfaces are on the same
subnet the host requires IP rules that use the source address (which bind sets)
to select the destination interface. The SO_BINDTODEVICE socket option could be
used to select a specific interface for the connection instead, however it
requires JNI (is not in Java's SocketOptions) and root access, which we don't
want to require clients have.
Like HDFS-3147, the client can use multiple local interfaces for data transfer.
Since the client already cache their connections to DNs choosing a local
interface at random seems like a good policy. Users can also pin a specific
client to a specific interface by specifying just that interface in
dfs.client.local.interfaces.
This change was discussed in HADOOP-6210 a while back, and is actually
useful/independent of the other HDFS-3140 changes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3149) The client should blacklist failed local/remote network interface pairs
The client should blacklist failed local/remote network interface pairs --- Key: HDFS-3149 URL: https://issues.apache.org/jira/browse/HDFS-3149 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs client Reporter: Eli Collins Assignee: Eli Collins If a client or worker can not connect to a given remote address, eg due to network or interface failure, then it should blacklist the local/remote interface pair. Only the pair is blacklisted in case the remote interface is routable via another local interface. The pair is black listed for a configurable period of time and another local/remote interface pair is tried. For full fault tolerance, the host interfaces need to be connected to different switches. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3150) Add option for clients to contact DNs via hostname in branch-1
Add option for clients to contact DNs via hostname in branch-1
--
Key: HDFS-3150
URL: https://issues.apache.org/jira/browse/HDFS-3150
Project: Hadoop HDFS
Issue Type: Sub-task
Components: data-node, hdfs client
Reporter: Eli Collins
Assignee: Eli Collins
Per the document attached to HADOOP-8198, this is just for branch-1, and
unbreaks DN multihoming. The datanode can be configured to listen on a bond, or
all interfaces by specifying the wildcard in the dfs.datanode.*.address
configuration options, however per HADOOP-6867 only the source address of the
registration is exposed to clients. HADOOP-985 made clients access datanodes by
IP primarily to avoid the latency of a DNS lookup, this had the side effect of
breaking DN multihoming. In order to fix it let's add back the option for
Datanodes to be accessed by hostname. This can be done by:
# Modifying the primary field of the Datanode descriptor to be the hostname, or
# Modifying Client/Datanode - Datanode access use the hostname field instead
of the IP
I'd like to go with approach #2 as it does not require making an incompatible
change to the client protocol, and is much less invasive. It minimizes the
scope of modification to just places where clients and Datanodes connect, vs
changing all uses of Datanode identifiers.
New client and Datanode configuration options are introduced:
- {{dfs.client.use.datanode.hostname}} indicates all client to datanode
connections should use the datanode hostname (as clients outside cluster may
not be able to route the IP)
- {{dfs.datanode.use.datanode.hostname}} indicates whether Datanodes should use
hostnames when connecting to other Datanodes for data transfer
If the configuration options are not used, there is no change in the current
behavior.
I'm doing something similar to #1 btw in trunk in HDFS-3144 - refactoring the
use of DatanodeID to use the right field (IP, IP:xferPort, hostname, etc) based
on the context the ID is being used in, vs always using the IP:xferPort as the
Datanode's name, and using the name everywhere.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3148) The client should be able to use multiple local interfaces for data transfer
[
https://issues.apache.org/jira/browse/HDFS-3148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Collins updated HDFS-3148:
--
Attachment: hdfs-3148.txt
Patch attached (for trunk). Depends on HADOOP-8210.
The client should be able to use multiple local interfaces for data transfer
Key: HDFS-3148
URL: https://issues.apache.org/jira/browse/HDFS-3148
Project: Hadoop HDFS
Issue Type: Sub-task
Components: hdfs client
Reporter: Eli Collins
Assignee: Eli Collins
Attachments: hdfs-3148.txt
HDFS-3147 covers using multiple interfaces on the server (Datanode) side.
Clients should also be able to utilize multiple *local* interfaces for
outbound connections instead of always using the interface for the local
hostname. This can be accomplished with a new configuration parameter
({{dfs.client.local.interfaces}}) that accepts a list of interfaces the
client should use. Acceptable configuration values are the same as the
{{dfs.datanode.available.interfaces}} parameter. The client binds its socket
to a specific interface, which enables outbound traffic to use that
interface. Binding the client socket to a specific address is not sufficient
to ensure egress traffic uses that interface. Eg if multiple interfaces are
on the same subnet the host requires IP rules that use the source address
(which bind sets) to select the destination interface. The SO_BINDTODEVICE
socket option could be used to select a specific interface for the connection
instead, however it requires JNI (is not in Java's SocketOptions) and root
access, which we don't want to require clients have.
Like HDFS-3147, the client can use multiple local interfaces for data
transfer. Since the client already cache their connections to DNs choosing a
local interface at random seems like a good policy. Users can also pin a
specific client to a specific interface by specifying just that interface in
dfs.client.local.interfaces.
This change was discussed in HADOOP-6210 a while back, and is actually
useful/independent of the other HDFS-3140 changes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2941) Add an administrative command to download a copy of the fsimage from the NN
[
https://issues.apache.org/jira/browse/HDFS-2941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238054#comment-13238054
]
Todd Lipcon commented on HDFS-2941:
---
Looks pretty good. A few more small comments:
- I'd rather see the HADOOP_FSIMAGE_HEADER header be added in all cases -- not
just when it's the latest -- just for consistency.
- Given that, I think the code in TransferFsImage will make a little more
sense. Right now, the localPaths variable is either a file (when not getting
latest) or a directory (when getting latest) which seems inconsistent. If you
send the header in all cases, then the code can change to something like:
{code}
String fsImageName = connection.getHeaderField(HADOOP_FSIMAGE_HEADER);
// If the local paths refer to directories, use the server-provided header
// as the filename within that directory
ListFile newLocalPaths = new ArrayListFile();
for (File localPath : localPaths) {
if (localPath.isDirectory()) {
if (fsImageName == null) {
throw new IOException(No filename header provided by server);
}
newLocalPaths.add(new File(localPath, fsImageName));
} else {
newLocalPaths.add(localPath);
}
}
localPaths = newLocalPaths;
{code}
Add an administrative command to download a copy of the fsimage from the NN
---
Key: HDFS-2941
URL: https://issues.apache.org/jira/browse/HDFS-2941
Project: Hadoop HDFS
Issue Type: New Feature
Components: hdfs client, name-node
Affects Versions: 0.24.0
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
Attachments: HDFS-2941.patch, HDFS-2941.patch
It would be nice to be able to download a copy of the fsimage from the NN,
e.g. for backup purposes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3151) Improve ActiveStandbyElector's behavior when session expires
Improve ActiveStandbyElector's behavior when session expires Key: HDFS-3151 URL: https://issues.apache.org/jira/browse/HDFS-3151 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 0.24.0, 0.23.3 Reporter: Todd Lipcon Assignee: Todd Lipcon Currently when the ZK session expires, it results in a fatal error being sent to the application callback. This is not the best behavior -- for example, in the case of HA, if ZK goes down, we would like the current state to be maintained, rather than causing either NN to abort. When the ZK clients are able to reconnect, they should sort out the correct leader based on the normal locking schemes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2185) HA: HDFS portion of ZK-based FailoverController
[ https://issues.apache.org/jira/browse/HDFS-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2185: -- Attachment: hdfs-2185.txt Attached patch is for the HDFS side after splitting out the common components. It includes a simple unit test which makes sure failover occurs when the NNs shut themselves down. I'll continue to add test cases and also run some cluster tests while this (and its prereqs HADOOP-8206 and HADOOP-8212) are under review. HA: HDFS portion of ZK-based FailoverController --- Key: HDFS-2185 URL: https://issues.apache.org/jira/browse/HDFS-2185 Project: Hadoop HDFS Issue Type: Sub-task Components: ha Affects Versions: HA branch (HDFS-1623) Reporter: Eli Collins Assignee: Todd Lipcon Attachments: Failover_Controller.jpg, hdfs-2185.txt, hdfs-2185.txt This jira is for a ZK-based FailoverController daemon. The FailoverController is a separate daemon from the NN that does the following: * Initiates leader election (via ZK) when necessary * Performs health monitoring (aka failure detection) * Performs fail-over (standby to active and active to standby transitions) * Heartbeats to ensure the liveness It should have the same/similar interface as the Linux HA RM to aid pluggability. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2941) Add an administrative command to download a copy of the fsimage from the NN
[ https://issues.apache.org/jira/browse/HDFS-2941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-2941: - Attachment: HDFS-2941.patch Thanks a lot for the review, Todd. Here's an updated patch which should address your comments. Add an administrative command to download a copy of the fsimage from the NN --- Key: HDFS-2941 URL: https://issues.apache.org/jira/browse/HDFS-2941 Project: Hadoop HDFS Issue Type: New Feature Components: hdfs client, name-node Affects Versions: 0.24.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Attachments: HDFS-2941.patch, HDFS-2941.patch, HDFS-2941.patch It would be nice to be able to download a copy of the fsimage from the NN, e.g. for backup purposes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2413) Add public APIs for safemode
[ https://issues.apache.org/jira/browse/HDFS-2413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238119#comment-13238119 ] Aaron T. Myers commented on HDFS-2413: -- Hey Harsh, the patch looks like it will work to me, and I agree that the test failures are unrelated: the HAAdmin ones should now be fixed, the other two are HDFS-3142 and HDFS-3143. But, per Nicholas, I thought we were going to change the InterfaceAudience of DistributedFileSystem to LimitedPrivate from Private? Add public APIs for safemode Key: HDFS-2413 URL: https://issues.apache.org/jira/browse/HDFS-2413 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs client Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Harsh J Attachments: HDFS-2413.patch, HDFS-2413.patch, HDFS-2413.patch Currently the APIs for safe-mode are part of DistributedFileSystem, which is supposed to be a private interface. However, dependent software often wants to wait until the NN is out of safemode. Though it could poll trying to create a file and catching SafeModeException, we should consider making some of these APIs public. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
