[jira] [Commented] (HDFS-5228) The RemoteIterator returned by DistributedFileSystem.listFiles(..) may throw NPE
[ https://issues.apache.org/jira/browse/HDFS-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13774354#comment-13774354 ] Tsz Wo (Nicholas), SZE commented on HDFS-5228: -- > We have extensive symlink tests checked in as part of HADOOP-8040, ... Which part of HADOOP-8040? Where are the new tests? > The RemoteIterator returned by DistributedFileSystem.listFiles(..) may throw > NPE > > > Key: HDFS-5228 > URL: https://issues.apache.org/jira/browse/HDFS-5228 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.1.0-beta >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE >Priority: Blocker > Attachments: h5228_20130919.patch, h5228_20130919_test.patch, > HDFS-5228.2.patch > > > Get a RemoteIterator from DistributedFileSystem.listFiles(..) with a relative > path. Then, it will result a NullPointerException when calling hasNext() > from the RemoteIterator. > This bug was discovered by Arnaud: > http://hortonworks.com/community/forums/topic/new-bug-in-hdfs-listfiles-method/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5228) The RemoteIterator returned by DistributedFileSystem.listFiles(..) may throw NPE
[ https://issues.apache.org/jira/browse/HDFS-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13774366#comment-13774366 ] Tsz Wo (Nicholas), SZE commented on HDFS-5228: -- Chris, thanks for posting a patch. I do think that the best solution is to revert HADOOP-9418, which seemed not well tested. HADOOP-9418 first broke HADOOP-9761 and then this. As Andrew said, "... I think some number of bugs like this are unavoidable as symlinks was a pretty big change that required touching basically all the client API code. ..." We should not let HADOOP-9418 destabilize 2.1.1. > The RemoteIterator returned by DistributedFileSystem.listFiles(..) may throw > NPE > > > Key: HDFS-5228 > URL: https://issues.apache.org/jira/browse/HDFS-5228 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.1.0-beta >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE >Priority: Blocker > Attachments: h5228_20130919.patch, h5228_20130919_test.patch, > HDFS-5228.2.patch > > > Get a RemoteIterator from DistributedFileSystem.listFiles(..) with a relative > path. Then, it will result a NullPointerException when calling hasNext() > from the RemoteIterator. > This bug was discovered by Arnaud: > http://hortonworks.com/community/forums/topic/new-bug-in-hdfs-listfiles-method/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5228) The RemoteIterator returned by DistributedFileSystem.listFiles(..) may throw NPE
[ https://issues.apache.org/jira/browse/HDFS-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13774368#comment-13774368 ] Tsz Wo (Nicholas), SZE commented on HDFS-5228: -- Andrew, you have not answered my following question. I hope that you are not avoiding it. bq. How exactly to audit DFS do you suggest so that it could avoid further bugs? > The RemoteIterator returned by DistributedFileSystem.listFiles(..) may throw > NPE > > > Key: HDFS-5228 > URL: https://issues.apache.org/jira/browse/HDFS-5228 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.1.0-beta >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE >Priority: Blocker > Attachments: h5228_20130919.patch, h5228_20130919_test.patch, > HDFS-5228.2.patch > > > Get a RemoteIterator from DistributedFileSystem.listFiles(..) with a relative > path. Then, it will result a NullPointerException when calling hasNext() > from the RemoteIterator. > This bug was discovered by Arnaud: > http://hortonworks.com/community/forums/topic/new-bug-in-hdfs-listfiles-method/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5216) NumDecomDeadDataNodes not returning correct number of dead decommissioned nodes
[ https://issues.apache.org/jira/browse/HDFS-5216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Trevor Lorimer updated HDFS-5216: - Status: Patch Available (was: Open) > NumDecomDeadDataNodes not returning correct number of dead decommissioned > nodes > > > Key: HDFS-5216 > URL: https://issues.apache.org/jira/browse/HDFS-5216 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.1.0-beta >Reporter: Trevor Lorimer >Assignee: Trevor Lorimer > Attachments: HDFS-5216.diff, HDFS-5216.diff > > > For HDFS-4860 I essentially copied the process in > NamenodejspHelper.generateHealthReport(), so it would be in sync with the > original dfsHealth.jsp. > However looking at this now there may be a bug? in > getNumDecomDeadDataNodes(), where: > getBlockManager().getDatanodeManager().fetchDatanodes(dead, null, true); > > Where the parameter true indicates that decommissioned nodes should be > removed from the list. > If the flag is true fetchDatanodes calls removeDecomNodeFromList, which will > remove a node if an existing datanode does not appear in both include or > exclude lists and it has been decommissioned. > If I am looking to return the Number of Dead Decommissioned Nodes, should I > change the remove decommissioned nodes flag to False? i.e.: > getBlockManager().getDatanodeManager().fetchDatanodes(null, dead, false); -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5216) NumDecomDeadDataNodes not returning correct number of dead decommissioned nodes
[ https://issues.apache.org/jira/browse/HDFS-5216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Trevor Lorimer updated HDFS-5216: - Attachment: HDFS-5216.diff > NumDecomDeadDataNodes not returning correct number of dead decommissioned > nodes > > > Key: HDFS-5216 > URL: https://issues.apache.org/jira/browse/HDFS-5216 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.1.0-beta >Reporter: Trevor Lorimer >Assignee: Trevor Lorimer > Attachments: HDFS-5216.diff, HDFS-5216.diff > > > For HDFS-4860 I essentially copied the process in > NamenodejspHelper.generateHealthReport(), so it would be in sync with the > original dfsHealth.jsp. > However looking at this now there may be a bug? in > getNumDecomDeadDataNodes(), where: > getBlockManager().getDatanodeManager().fetchDatanodes(dead, null, true); > > Where the parameter true indicates that decommissioned nodes should be > removed from the list. > If the flag is true fetchDatanodes calls removeDecomNodeFromList, which will > remove a node if an existing datanode does not appear in both include or > exclude lists and it has been decommissioned. > If I am looking to return the Number of Dead Decommissioned Nodes, should I > change the remove decommissioned nodes flag to False? i.e.: > getBlockManager().getDatanodeManager().fetchDatanodes(null, dead, false); -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5216) NumDecomDeadDataNodes not returning correct number of dead decommissioned nodes
[ https://issues.apache.org/jira/browse/HDFS-5216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Trevor Lorimer updated HDFS-5216: - Assignee: Trevor Lorimer Status: Open (was: Patch Available) > NumDecomDeadDataNodes not returning correct number of dead decommissioned > nodes > > > Key: HDFS-5216 > URL: https://issues.apache.org/jira/browse/HDFS-5216 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.1.0-beta >Reporter: Trevor Lorimer >Assignee: Trevor Lorimer > Attachments: HDFS-5216.diff, HDFS-5216.diff > > > For HDFS-4860 I essentially copied the process in > NamenodejspHelper.generateHealthReport(), so it would be in sync with the > original dfsHealth.jsp. > However looking at this now there may be a bug? in > getNumDecomDeadDataNodes(), where: > getBlockManager().getDatanodeManager().fetchDatanodes(dead, null, true); > > Where the parameter true indicates that decommissioned nodes should be > removed from the list. > If the flag is true fetchDatanodes calls removeDecomNodeFromList, which will > remove a node if an existing datanode does not appear in both include or > exclude lists and it has been decommissioned. > If I am looking to return the Number of Dead Decommissioned Nodes, should I > change the remove decommissioned nodes flag to False? i.e.: > getBlockManager().getDatanodeManager().fetchDatanodes(null, dead, false); -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1213) Implement an Apache Commons VFS Driver for HDFS
[ https://issues.apache.org/jira/browse/HDFS-1213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13774520#comment-13774520 ] Jakub Narloch commented on HDFS-1213: - Hi, What is the status of this issue? Is there any change I would be released someday? Actually we could use such level of abstraction and I think the general idea is great. > Implement an Apache Commons VFS Driver for HDFS > --- > > Key: HDFS-1213 > URL: https://issues.apache.org/jira/browse/HDFS-1213 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs-client >Reporter: Michael D'Amour > Attachments: HADOOP-HDFS-Apache-VFS.patch, > pentaho-hdfs-vfs-TRUNK-SNAPSHOT.jar, > pentaho-hdfs-vfs-TRUNK-SNAPSHOT-sources.tar.gz > > > We have an open source ETL tool (Kettle) which uses VFS for many input/output > steps/jobs. We would like to be able to read/write HDFS from Kettle using > VFS. > > I haven't been able to find anything out there other than "it would be nice." > > I had some time a few weeks ago to begin writing a VFS driver for HDFS and we > (Pentaho) would like to be able to contribute this driver. I believe it > supports all the major file/folder operations and I have written unit tests > for all of these operations. The code is currently checked into an open > Pentaho SVN repository under the Apache 2.0 license. There are some current > limitations, such as a lack of authentication (kerberos), which appears to be > coming in 0.22.0, however, the driver supports username/password, but I just > can't use them yet. > I will be attaching the code for the driver once the case is created. The > project does not modify existing hadoop/hdfs source. > Our JIRA case can be found at http://jira.pentaho.com/browse/PDI-4146 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1213) Implement an Apache Commons VFS Driver for HDFS
[ https://issues.apache.org/jira/browse/HDFS-1213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13774553#comment-13774553 ] Matt Casters commented on HDFS-1213: Jakub, until the VFS driver can be incorporated into the HDFS or VFS Apache projects, Pentaho maintains a separate source tree. It's over here: http://source.pentaho.org/viewvc/svnroot/pentaho-commons/pentaho-hdfs-vfs/trunk/ > Implement an Apache Commons VFS Driver for HDFS > --- > > Key: HDFS-1213 > URL: https://issues.apache.org/jira/browse/HDFS-1213 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs-client >Reporter: Michael D'Amour > Attachments: HADOOP-HDFS-Apache-VFS.patch, > pentaho-hdfs-vfs-TRUNK-SNAPSHOT.jar, > pentaho-hdfs-vfs-TRUNK-SNAPSHOT-sources.tar.gz > > > We have an open source ETL tool (Kettle) which uses VFS for many input/output > steps/jobs. We would like to be able to read/write HDFS from Kettle using > VFS. > > I haven't been able to find anything out there other than "it would be nice." > > I had some time a few weeks ago to begin writing a VFS driver for HDFS and we > (Pentaho) would like to be able to contribute this driver. I believe it > supports all the major file/folder operations and I have written unit tests > for all of these operations. The code is currently checked into an open > Pentaho SVN repository under the Apache 2.0 license. There are some current > limitations, such as a lack of authentication (kerberos), which appears to be > coming in 0.22.0, however, the driver supports username/password, but I just > can't use them yet. > I will be attaching the code for the driver once the case is created. The > project does not modify existing hadoop/hdfs source. > Our JIRA case can be found at http://jira.pentaho.com/browse/PDI-4146 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5239) Allow FSNamesystem lock fairness to be configurable
[
https://issues.apache.org/jira/browse/HDFS-5239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daryn Sharp updated HDFS-5239:
--
Resolution: Fixed
Fix Version/s: 2.3.0
3.0.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks Todd! I've committed to trunk and branch-2.
> Allow FSNamesystem lock fairness to be configurable
> ---
>
> Key: HDFS-5239
> URL: https://issues.apache.org/jira/browse/HDFS-5239
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: 2.0.0-alpha, 3.0.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Fix For: 3.0.0, 2.3.0
>
> Attachments: HDFS-5239.patch
>
>
> The fairness of the {{FSNamesystem#fsLock}} is hardcoded to {{true}}. Using
> an unfair lock provides a negligible increase to throughput. However this is
> due to bottlenecks elsewhere in the system. In combination with other
> changes, such as RPC layer and audit logging, preliminary tests show up to a
> 5X improvement for a read heavy workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5240) Separate formatting from logging in the audit logger API
[ https://issues.apache.org/jira/browse/HDFS-5240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated HDFS-5240: -- Resolution: Fixed Fix Version/s: 2.3.0 3.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks Todd! I've committed to trunk and branch-2. > Separate formatting from logging in the audit logger API > > > Key: HDFS-5240 > URL: https://issues.apache.org/jira/browse/HDFS-5240 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Fix For: 3.0.0, 2.3.0 > > Attachments: HDFS-5240.patch > > > The audit logger API should be extended (in a compatible manner) to separate > the formatting of the log message from the actual logging of the message. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5240) Separate formatting from logging in the audit logger API
[ https://issues.apache.org/jira/browse/HDFS-5240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13774654#comment-13774654 ] Hudson commented on HDFS-5240: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4453 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4453/]) HDFS-5240. Separate formatting from logging in the audit logger API (daryn) (daryn: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1525626) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java > Separate formatting from logging in the audit logger API > > > Key: HDFS-5240 > URL: https://issues.apache.org/jira/browse/HDFS-5240 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Fix For: 3.0.0, 2.3.0 > > Attachments: HDFS-5240.patch > > > The audit logger API should be extended (in a compatible manner) to separate > the formatting of the log message from the actual logging of the message. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5242) Reduce contention on DatanodeInfo instances
[
https://issues.apache.org/jira/browse/HDFS-5242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daryn Sharp updated HDFS-5242:
--
Status: Patch Available (was: Open)
No test because there's no way to squeeze in a barrier.
> Reduce contention on DatanodeInfo instances
> ---
>
> Key: HDFS-5242
> URL: https://issues.apache.org/jira/browse/HDFS-5242
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: 2.0.0-alpha, 3.0.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: HDFS-5242.patch
>
>
> Synchronization in {{DatanodeInfo}} instances causes unnecessary contention
> between call handlers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5242) Reduce contention on DatanodeInfo instances
[
https://issues.apache.org/jira/browse/HDFS-5242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daryn Sharp updated HDFS-5242:
--
Attachment: HDFS-5242.patch
Remove synchronized keyword from {{get/setNetworkLocation}}. HDFS-1985 added
the synch with no clear reason and it's the only synch'ed mutable field in the
class. A profiler identified this unnecessary synch after peeling away other
issues. Unnecessary synchs cause thread contention and which harms performance
due to context switches.
The synch doesn't really protect against any race conditions. Further, the
network location is only set when registering a datanode so it's not changing
during normal operation.
> Reduce contention on DatanodeInfo instances
> ---
>
> Key: HDFS-5242
> URL: https://issues.apache.org/jira/browse/HDFS-5242
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: 2.0.0-alpha, 3.0.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: HDFS-5242.patch
>
>
> Synchronization in {{DatanodeInfo}} instances causes unnecessary contention
> between call handlers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5139) Remove redundant -R option from setrep
[ https://issues.apache.org/jira/browse/HDFS-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13774799#comment-13774799 ] Hudson commented on HDFS-5139: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4455 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4455/]) HDFS-5139. Remove redundant -R option from setrep. (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1525659) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/SetReplication.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/site/apt/FileSystemShell.apt.vm * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/testConf.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml > Remove redundant -R option from setrep > -- > > Key: HDFS-5139 > URL: https://issues.apache.org/jira/browse/HDFS-5139 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Affects Versions: 3.0.0, 1.3.0 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-5139.01.patch, HDFS-5139.02.patch, > HDFS-5139.03.patch, HDFS-5139.04.patch > > > The -R option to setrep is redundant because it is required for directory > targets and ignored for file targets. > We can just remove the option and make -R the default for directories. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5139) Remove redundant -R option from setrep
[ https://issues.apache.org/jira/browse/HDFS-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13774775#comment-13774775 ] Suresh Srinivas commented on HDFS-5139: --- +1 for the change. > Remove redundant -R option from setrep > -- > > Key: HDFS-5139 > URL: https://issues.apache.org/jira/browse/HDFS-5139 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Affects Versions: 3.0.0, 1.3.0 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-5139.01.patch, HDFS-5139.02.patch, > HDFS-5139.03.patch, HDFS-5139.04.patch > > > The -R option to setrep is redundant because it is required for directory > targets and ignored for file targets. > We can just remove the option and make -R the default for directories. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-5248) Checkpointing fails in uploading the Image to the Namenode if the merge requires more than 10 hours in secure cluster
Benoy Antony created HDFS-5248: -- Summary: Checkpointing fails in uploading the Image to the Namenode if the merge requires more than 10 hours in secure cluster Key: HDFS-5248 URL: https://issues.apache.org/jira/browse/HDFS-5248 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 1.2.1, 2.1.0-beta Reporter: Benoy Antony Assignee: Benoy Antony If the merge takes up more than 10 hours, the TGT of the SNN expires and SNN will fail to upload the image to the Namenode -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5248) Checkpointing fails in uploading the Image to the Namenode if the merge requires more than 10 hours in secure cluster
[ https://issues.apache.org/jira/browse/HDFS-5248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HDFS-5248: --- Attachment: HDFS-5248-1.patch Attaching a patch for branch-1.2 Will upload the patch for trunk soon. > Checkpointing fails in uploading the Image to the Namenode if the merge > requires more than 10 hours in secure cluster > - > > Key: HDFS-5248 > URL: https://issues.apache.org/jira/browse/HDFS-5248 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.1.0-beta, 1.2.1 >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HDFS-5248-1.patch > > > If the merge takes up more than 10 hours, the TGT of the SNN expires and SNN > will fail to upload the image to the Namenode -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5248) Checkpointing fails in uploading the Image to the Namenode if the merge requires more than 10 hours in secure cluster
[ https://issues.apache.org/jira/browse/HDFS-5248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HDFS-5248: --- Attachment: (was: HDFS-5248-1.patch) > Checkpointing fails in uploading the Image to the Namenode if the merge > requires more than 10 hours in secure cluster > - > > Key: HDFS-5248 > URL: https://issues.apache.org/jira/browse/HDFS-5248 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.1.0-beta, 1.2.1 >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HDFS-5248-1.patch > > > If the merge takes up more than 10 hours, the TGT of the SNN expires and SNN > will fail to upload the image to the Namenode -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5139) Remove redundant -R option from setrep
[ https://issues.apache.org/jira/browse/HDFS-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-5139: Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Thanks Suresh. I have committed this to trunk, branch-2 and branch-2.1-beta. Fix version has been set to 2.2.0. > Remove redundant -R option from setrep > -- > > Key: HDFS-5139 > URL: https://issues.apache.org/jira/browse/HDFS-5139 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Affects Versions: 3.0.0, 1.3.0 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Fix For: 2.2.0 > > Attachments: HDFS-5139.01.patch, HDFS-5139.02.patch, > HDFS-5139.03.patch, HDFS-5139.04.patch > > > The -R option to setrep is redundant because it is required for directory > targets and ignored for file targets. > We can just remove the option and make -R the default for directories. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5248) Checkpointing fails in uploading the Image to the Namenode if the merge requires more than 10 hours in secure cluster
[ https://issues.apache.org/jira/browse/HDFS-5248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13774813#comment-13774813 ] Kai Zheng commented on HDFS-5248: - I guess there might be a place calling ugi.doAs or like that in the callers of putFSImage(). If so, perhaps there adding the checkTGTAndReloginFromKeytab() call might work with minimized change. HADOOP-9797 is working on something that does such job automatically. For ugi logined from keytab or ticket cache, it has choice to be automatically renewed. > Checkpointing fails in uploading the Image to the Namenode if the merge > requires more than 10 hours in secure cluster > - > > Key: HDFS-5248 > URL: https://issues.apache.org/jira/browse/HDFS-5248 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.1.0-beta, 1.2.1 >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HDFS-5248-1.patch > > > If the merge takes up more than 10 hours, the TGT of the SNN expires and SNN > will fail to upload the image to the Namenode -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5248) Checkpointing fails in uploading the Image to the Namenode if the merge requires more than 10 hours in secure cluster
[ https://issues.apache.org/jira/browse/HDFS-5248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HDFS-5248: --- Attachment: HDFS-5248-1.patch > Checkpointing fails in uploading the Image to the Namenode if the merge > requires more than 10 hours in secure cluster > - > > Key: HDFS-5248 > URL: https://issues.apache.org/jira/browse/HDFS-5248 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.1.0-beta, 1.2.1 >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HDFS-5248-1.patch > > > If the merge takes up more than 10 hours, the TGT of the SNN expires and SNN > will fail to upload the image to the Namenode -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5216) NumDecomDeadDataNodes not returning correct number of dead decommissioned nodes
[
https://issues.apache.org/jira/browse/HDFS-5216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13775551#comment-13775551
]
Hadoop QA commented on HDFS-5216:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12604543/HDFS-5216.diff
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5018//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5018//console
This message is automatically generated.
> NumDecomDeadDataNodes not returning correct number of dead decommissioned
> nodes
>
>
> Key: HDFS-5216
> URL: https://issues.apache.org/jira/browse/HDFS-5216
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.1.0-beta
>Reporter: Trevor Lorimer
>Assignee: Trevor Lorimer
> Attachments: HDFS-5216.diff, HDFS-5216.diff
>
>
> For HDFS-4860 I essentially copied the process in
> NamenodejspHelper.generateHealthReport(), so it would be in sync with the
> original dfsHealth.jsp.
> However looking at this now there may be a bug? in
> getNumDecomDeadDataNodes(), where:
> getBlockManager().getDatanodeManager().fetchDatanodes(dead, null, true);
>
> Where the parameter true indicates that decommissioned nodes should be
> removed from the list.
> If the flag is true fetchDatanodes calls removeDecomNodeFromList, which will
> remove a node if an existing datanode does not appear in both include or
> exclude lists and it has been decommissioned.
> If I am looking to return the Number of Dead Decommissioned Nodes, should I
> change the remove decommissioned nodes flag to False? i.e.:
> getBlockManager().getDatanodeManager().fetchDatanodes(null, dead, false);
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5246) Hadoop nfs server binds to port 2049 which is the same as Linux nfs server
[ https://issues.apache.org/jira/browse/HDFS-5246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-5246: - Target Version/s: 2.1.0-beta, 3.0.0, 2.1.1-beta (was: 3.0.0, 2.1.0-beta, 2.1.1-beta) Status: Patch Available (was: Open) > Hadoop nfs server binds to port 2049 which is the same as Linux nfs server > -- > > Key: HDFS-5246 > URL: https://issues.apache.org/jira/browse/HDFS-5246 > Project: Hadoop HDFS > Issue Type: Improvement > Components: nfs >Affects Versions: 2.1.0-beta > Environment: Red Hat Enterprise 6 with Sun Java 1.7 and IBM Java 1.6 >Reporter: Jinghui Wang > Fix For: 3.0.0, 2.1.1-beta > > Attachments: HDFS-5246.patch > > > Hadoop nfs binds the nfs server to port 2049, > which is also the default port that Linux nfs uses. If Linux nfs is already > running on the machine then Hadoop nfs will not be albe to start. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5246) Hadoop nfs server binds to port 2049 which is the same as Linux nfs server
[
https://issues.apache.org/jira/browse/HDFS-5246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13774792#comment-13774792
]
Brandon Li commented on HDFS-5246:
--
{quote}"...2049, which is also the default port that Linux nfs uses. If Linux
nfs is already running on the machine then Hadoop nfs will not be albe to
start."{quote}
There can be only one NFSv3 server running on the same host no matter which
port is configured. right?
Regardless, this patch is a good idea. In the patch, you set the default port
to 2079 for tests, so the NFS test can run when an NFS server is using the
default port 2049 on the same host. This makes the testing more convenient.
> Hadoop nfs server binds to port 2049 which is the same as Linux nfs server
> --
>
> Key: HDFS-5246
> URL: https://issues.apache.org/jira/browse/HDFS-5246
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: nfs
>Affects Versions: 2.1.0-beta
> Environment: Red Hat Enterprise 6 with Sun Java 1.7 and IBM Java 1.6
>Reporter: Jinghui Wang
> Fix For: 3.0.0, 2.1.1-beta
>
> Attachments: HDFS-5246.patch
>
>
> Hadoop nfs binds the nfs server to port 2049,
> which is also the default port that Linux nfs uses. If Linux nfs is already
> running on the machine then Hadoop nfs will not be albe to start.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4971) Move IO operations out of locking in OpenFileCtx
[ https://issues.apache.org/jira/browse/HDFS-4971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13775568#comment-13775568 ] Jing Zhao commented on HDFS-4971: - Thanks again Brandon! I will commit the patch soon. > Move IO operations out of locking in OpenFileCtx > > > Key: HDFS-4971 > URL: https://issues.apache.org/jira/browse/HDFS-4971 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-4971.000.patch, HDFS-4971.001.patch, > HDFS-4971.002.patch, HDFS-4971.003.patch, HDFS-4971.004.patch, > HDFS-4971.005.patch, HDFS-4971.006.patch, HDFS-4971.007.patch > > > Currently some IO operations (such as writing data to HDFS and dumping to > local disk) in OpenFileCtx may hold a lock which can block processing > incoming writing requests. This jira aims to optimize OpenFileCtx and move > the IO operations out of the locking. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5246) Hadoop nfs server binds to port 2049 which is the same as Linux nfs server
[ https://issues.apache.org/jira/browse/HDFS-5246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13775574#comment-13775574 ] Brandon Li commented on HDFS-5246: -- The patch looks good to me. Some minor comments: 1. "nfsPort" can be final in Nfs3Base 2. nfs3.server.port could be a better name than nfs.server.port > Hadoop nfs server binds to port 2049 which is the same as Linux nfs server > -- > > Key: HDFS-5246 > URL: https://issues.apache.org/jira/browse/HDFS-5246 > Project: Hadoop HDFS > Issue Type: Improvement > Components: nfs >Affects Versions: 2.1.0-beta > Environment: Red Hat Enterprise 6 with Sun Java 1.7 and IBM Java 1.6 >Reporter: Jinghui Wang > Fix For: 3.0.0, 2.1.1-beta > > Attachments: HDFS-5246.patch > > > Hadoop nfs binds the nfs server to port 2049, > which is also the default port that Linux nfs uses. If Linux nfs is already > running on the machine then Hadoop nfs will not be albe to start. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5191) revisit zero-copy API in FSDataInputStream to make it more intuitive
[
https://issues.apache.org/jira/browse/HDFS-5191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13775611#comment-13775611
]
Chris Nauroth commented on HDFS-5191:
-
It looks like there are still some import-only changes in {{DFSUtil}}.
{code}
if (buffer == null) {
throw new UnsupportedOperationException("zero-copy reads " +
"were not available, and the ByteBufferPool did not provide " +
"us with a " + (useDirect ? " direct" : "indirect") +
"buffer.");
}
{code}
Minor nit: this exception message would have an extraneous space in the direct
case (i.e. "did not provide us with a direct"), and no space between the last
two words (i.e. "did not provide us with a indirectbuffer").
{code}
while (true) {
countingVisitor.reset();
mmapManager.visitEvictable(countingVisitor);
if (0 == countingVisitor.count) {
break;
}
}
{code}
This test would cause an infinite loop if a bug was introduced that left mmaps
lingering in evictable state, which could be hard to diagnose. Should we use
{{GenericTestUtils#waitFor}}, so that there is a timeout?
Lastly, can you help clarify something about the data structure behind
{{IdentityHashStore}} for me? In particular, I'm wondering about the math in
{{realloc}}:
{code}
private void realloc(int newCapacity) {
Preconditions.checkArgument(newCapacity > 0);
Object prevBuffer[] = buffer;
this.capacity = newCapacity;
int numElements = 1 + (2 * newCapacity);
this.buffer = new Object[2 * numElements];
this.numInserted = 0;
if (prevBuffer != null) {
for (int i = 0; i < prevBuffer.length; i += 2) {
if (prevBuffer[i] != null) {
putInternal(prevBuffer[i], prevBuffer[i + 1]);
}
}
}
}
{code}
{{put}} will call {{realloc}} to double capacity when needed. {{numElements}}
is double of the new capacity plus one extra. Then, the size is doubled again
for allocating the new array. Therefore the growth pattern looks like:
capacity == 2, buffer.length == 10
capacity == 4, buffer.length == 18
capacity == 8, buffer.length == 34
It looks like this causes a fair amount of extra unused slack in the array,
because only 2 * {{numInserted}} elements are used at a time. Is the slack
required, or should {{realloc}} only double the size once instead of twice?
Also, I wasn't sure why we needed 1 extra in the calculation of
{{numElements}}. Is that a defensive thing so that the loop in {{putInternal}}
always terminates? I would expect the check of capacity in {{put}} to be
sufficient to prevent that without needing an extra sentinel element.
On to the libhdfs changes...
> revisit zero-copy API in FSDataInputStream to make it more intuitive
>
>
> Key: HDFS-5191
> URL: https://issues.apache.org/jira/browse/HDFS-5191
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client, libhdfs
>Affects Versions: HDFS-4949
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-5191-caching.001.patch,
> HDFS-5191-caching.003.patch, HDFS-5191-caching.006.patch,
> HDFS-5191-caching.007.patch, HDFS-5191-caching.008.patch
>
>
> As per the discussion on HDFS-4953, we should revisit the zero-copy API to
> make it more intuitive for new users.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5230) Introduce RpcInfo to decouple XDR classes from the RPC API
[
https://issues.apache.org/jira/browse/HDFS-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13775576#comment-13775576
]
Jing Zhao commented on HDFS-5230:
-
The patch looks great to me. Some comments and thoughts:
# RpcInfo's constructor, we do not need to call super() explicitly.
# It will be helpful to add more javadoc for RpcInfo and RpcResponse.
# Similarly, we may want more javadoc for RpcxxxStage classes in RpcUtil to
clearly state the next stage for each stage (i.e., the whole pipeline).
# Creating Verifier#VERIFIER_RPC_GSS may be unnecessary here since we need to
create different GSS Verifier objects each time. This part we can leave it as
it is and update it in HDFS-5086.
# Currently we may hide ChannelBuffer creation into the RpcResponse. I.e., by
introducing a new constructor or static method in RpcResponse which takes
XDR/(Rpc Header + Rpc Response Body) as argument(s), we can avoid calling
{code}
ChannelBuffer buf =
ChannelBuffers.wrappedBuffer(reply.asReadOnlyWrap().buffer());
{code}
outside of RpcResponse.
# Currently we call Nfs3Response#writeHeaderAndResponse to generate an XDR
containing complete RPC response. We should put this logic into the new
RpcResponse class so that NFS3Response and its subclasses only need to handle
response information returned from NFS handlers.
> Introduce RpcInfo to decouple XDR classes from the RPC API
> --
>
> Key: HDFS-5230
> URL: https://issues.apache.org/jira/browse/HDFS-5230
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: nfs
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-5230.002.patch
>
>
> The XDR class is one fundamental aspect in the current implementation of NFS
> server. While the client might potentially have a higher level APIs, it also
> requires redundant copying since the upstream clients have insufficient
> information.
> This JIRA introduces a new class, RpcInfo, which (1) decouples XDR from the
> APIs, turning it into a utility class, and (2) exposes ChannelBuffer directly
> to the client in order to open the opportunity for avoid copying.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5139) Remove redundant -R option from setrep
[ https://issues.apache.org/jira/browse/HDFS-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13774820#comment-13774820 ] Hudson commented on HDFS-5139: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4456 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4456/]) HDFS-5139. Remove redundant -R option from setrep (update CHANGES.txt). (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1525665) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > Remove redundant -R option from setrep > -- > > Key: HDFS-5139 > URL: https://issues.apache.org/jira/browse/HDFS-5139 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Affects Versions: 3.0.0, 1.3.0 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Fix For: 2.2.0 > > Attachments: HDFS-5139.01.patch, HDFS-5139.02.patch, > HDFS-5139.03.patch, HDFS-5139.04.patch > > > The -R option to setrep is redundant because it is required for directory > targets and ignored for file targets. > We can just remove the option and make -R the default for directories. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4971) Move IO operations out of locking in OpenFileCtx
[ https://issues.apache.org/jira/browse/HDFS-4971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13775617#comment-13775617 ] Hudson commented on HDFS-4971: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4458 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4458/]) HDFS-4971. Move IO operations out of locking in OpenFileCtx. Contributed by Jing Zhao and Brandon Li. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1525681) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/AsyncDataService.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OffsetRange.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OpenFileCtx.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/WriteCtx.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/WriteManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/test/java/org/apache/hadoop/hdfs/nfs/nfs3/TestOffsetRange.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > Move IO operations out of locking in OpenFileCtx > > > Key: HDFS-4971 > URL: https://issues.apache.org/jira/browse/HDFS-4971 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-4971.000.patch, HDFS-4971.001.patch, > HDFS-4971.002.patch, HDFS-4971.003.patch, HDFS-4971.004.patch, > HDFS-4971.005.patch, HDFS-4971.006.patch, HDFS-4971.007.patch > > > Currently some IO operations (such as writing data to HDFS and dumping to > local disk) in OpenFileCtx may hold a lock which can block processing > incoming writing requests. This jira aims to optimize OpenFileCtx and move > the IO operations out of the locking. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4971) Move IO operations out of locking in OpenFileCtx
[
https://issues.apache.org/jira/browse/HDFS-4971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13774750#comment-13774750
]
Hadoop QA commented on HDFS-4971:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12604472/HDFS-4971.007.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs-nfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5019//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5019//console
This message is automatically generated.
> Move IO operations out of locking in OpenFileCtx
>
>
> Key: HDFS-4971
> URL: https://issues.apache.org/jira/browse/HDFS-4971
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: nfs
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: HDFS-4971.000.patch, HDFS-4971.001.patch,
> HDFS-4971.002.patch, HDFS-4971.003.patch, HDFS-4971.004.patch,
> HDFS-4971.005.patch, HDFS-4971.006.patch, HDFS-4971.007.patch
>
>
> Currently some IO operations (such as writing data to HDFS and dumping to
> local disk) in OpenFileCtx may hold a lock which can block processing
> incoming writing requests. This jira aims to optimize OpenFileCtx and move
> the IO operations out of the locking.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5191) revisit zero-copy API in FSDataInputStream to make it more intuitive
[
https://issues.apache.org/jira/browse/HDFS-5191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13774757#comment-13774757
]
Colin Patrick McCabe commented on HDFS-5191:
I decided to test {{TestByteBufferUtil#fallbackRead}} inside
{{TestEnhancedByteBufferAccess}}. The other file is not needed-- I will remove.
Removed extra imports in DFSUtil.
> revisit zero-copy API in FSDataInputStream to make it more intuitive
>
>
> Key: HDFS-5191
> URL: https://issues.apache.org/jira/browse/HDFS-5191
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client, libhdfs
>Affects Versions: HDFS-4949
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-5191-caching.001.patch,
> HDFS-5191-caching.003.patch, HDFS-5191-caching.006.patch,
> HDFS-5191-caching.007.patch, HDFS-5191-caching.008.patch
>
>
> As per the discussion on HDFS-4953, we should revisit the zero-copy API to
> make it more intuitive for new users.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5191) revisit zero-copy API in FSDataInputStream to make it more intuitive
[ https://issues.apache.org/jira/browse/HDFS-5191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-5191: --- Attachment: HDFS-5191-caching.008.patch > revisit zero-copy API in FSDataInputStream to make it more intuitive > > > Key: HDFS-5191 > URL: https://issues.apache.org/jira/browse/HDFS-5191 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client, libhdfs >Affects Versions: HDFS-4949 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HDFS-5191-caching.001.patch, > HDFS-5191-caching.003.patch, HDFS-5191-caching.006.patch, > HDFS-5191-caching.007.patch, HDFS-5191-caching.008.patch > > > As per the discussion on HDFS-4953, we should revisit the zero-copy API to > make it more intuitive for new users. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4971) Move IO operations out of locking in OpenFileCtx
[ https://issues.apache.org/jira/browse/HDFS-4971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-4971: Resolution: Fixed Fix Version/s: 2.1.1-beta Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I've committed this to trunk, branch-2 and branch-2.1-beta. > Move IO operations out of locking in OpenFileCtx > > > Key: HDFS-4971 > URL: https://issues.apache.org/jira/browse/HDFS-4971 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Reporter: Jing Zhao >Assignee: Jing Zhao > Fix For: 2.1.1-beta > > Attachments: HDFS-4971.000.patch, HDFS-4971.001.patch, > HDFS-4971.002.patch, HDFS-4971.003.patch, HDFS-4971.004.patch, > HDFS-4971.005.patch, HDFS-4971.006.patch, HDFS-4971.007.patch > > > Currently some IO operations (such as writing data to HDFS and dumping to > local disk) in OpenFileCtx may hold a lock which can block processing > incoming writing requests. This jira aims to optimize OpenFileCtx and move > the IO operations out of the locking. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5245) Retry policies should return false rather than rethrow an exception in branch-1
[ https://issues.apache.org/jira/browse/HDFS-5245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13775559#comment-13775559 ] Suresh Srinivas commented on HDFS-5245: --- [~wheat9], Jenkins only runs the submitted patch against trunk. There is no need to submit a patch, if it is for non-trunk. > Retry policies should return false rather than rethrow an exception in > branch-1 > --- > > Key: HDFS-5245 > URL: https://issues.apache.org/jira/browse/HDFS-5245 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 1.2.0, 1.2.1 >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-5245.000.patch > > > A RetryPolicy inspects the exception generated when performing work and > decides to try the work again. > In branch-1 a RetryPolicy either returns false or rethrow the exception in > the shouldRetry() function to indicate it decides not to retry. However, > rethrowing an exception confuses the upstream code and causes it to print out > excessive warning messages. > This bug is fixed in trunk and branch-2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5228) The RemoteIterator returned by DistributedFileSystem.listFiles(..) may throw NPE
[ https://issues.apache.org/jira/browse/HDFS-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13775561#comment-13775561 ] Andrew Wang commented on HDFS-5228: --- Chris, I'm +1 for your v2 patch, thanks for the help. Hey Nicholas: bq. Which part of HADOOP-8040? Where are the new tests? HADOOP-9370 and HADOOP-9355 extended the FileContext symlink tests to FileSystem. These are both linked on HADOOP-8040. You can also see this if you look for tests in the HADOOP-9418 patch. bq. Andrew, you have not answered my following question. I hope that you are not avoiding it. I'm going to quote myself to answer the question: bq. Basically all DFS methods are supposed to pass Paths through fixRelativePart and then getPathName, it just looks like we missed this one. It might be worth auditing DFS to make sure we didn't miss anything else. Chris already did this audit, perhaps you can ask him if you need additional in-person clarification. > The RemoteIterator returned by DistributedFileSystem.listFiles(..) may throw > NPE > > > Key: HDFS-5228 > URL: https://issues.apache.org/jira/browse/HDFS-5228 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.1.0-beta >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE >Priority: Blocker > Attachments: h5228_20130919.patch, h5228_20130919_test.patch, > HDFS-5228.2.patch > > > Get a RemoteIterator from DistributedFileSystem.listFiles(..) with a relative > path. Then, it will result a NullPointerException when calling hasNext() > from the RemoteIterator. > This bug was discovered by Arnaud: > http://hortonworks.com/community/forums/topic/new-bug-in-hdfs-listfiles-method/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5239) Allow FSNamesystem lock fairness to be configurable
[
https://issues.apache.org/jira/browse/HDFS-5239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13774645#comment-13774645
]
Hudson commented on HDFS-5239:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #4452 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/4452/])
HDFS-5239. Allow FSNamesystem lock fairness to be configurable (daryn) (daryn:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1525624)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java
> Allow FSNamesystem lock fairness to be configurable
> ---
>
> Key: HDFS-5239
> URL: https://issues.apache.org/jira/browse/HDFS-5239
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: 2.0.0-alpha, 3.0.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Fix For: 3.0.0, 2.3.0
>
> Attachments: HDFS-5239.patch
>
>
> The fairness of the {{FSNamesystem#fsLock}} is hardcoded to {{true}}. Using
> an unfair lock provides a negligible increase to throughput. However this is
> due to bottlenecks elsewhere in the system. In combination with other
> changes, such as RPC layer and audit logging, preliminary tests show up to a
> 5X improvement for a read heavy workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5246) Hadoop nfs server binds to port 2049 which is the same as Linux nfs server
[ https://issues.apache.org/jira/browse/HDFS-5246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13775693#comment-13775693 ] Jinghui Wang commented on HDFS-5246: "default port to 2079 for tests, so the NFS test can run when an NFS server is using the default port 2049 on the same host. This makes the testing more convenient" That's exactly the reason why we want to make the change as we encountered the failure during a UT job when NFS was running in the testing env. Attach new patch HDFS-5246-2.patch here with changes for the comments. > Hadoop nfs server binds to port 2049 which is the same as Linux nfs server > -- > > Key: HDFS-5246 > URL: https://issues.apache.org/jira/browse/HDFS-5246 > Project: Hadoop HDFS > Issue Type: Improvement > Components: nfs >Affects Versions: 2.1.0-beta > Environment: Red Hat Enterprise 6 with Sun Java 1.7 and IBM Java 1.6 >Reporter: Jinghui Wang > Fix For: 3.0.0, 2.1.1-beta > > Attachments: HDFS-5246-2.patch, HDFS-5246.patch > > > Hadoop nfs binds the nfs server to port 2049, > which is also the default port that Linux nfs uses. If Linux nfs is already > running on the machine then Hadoop nfs will not be albe to start. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5246) Hadoop nfs server binds to port 2049 which is the same as Linux nfs server
[ https://issues.apache.org/jira/browse/HDFS-5246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jinghui Wang updated HDFS-5246: --- Attachment: HDFS-5246-2.patch > Hadoop nfs server binds to port 2049 which is the same as Linux nfs server > -- > > Key: HDFS-5246 > URL: https://issues.apache.org/jira/browse/HDFS-5246 > Project: Hadoop HDFS > Issue Type: Improvement > Components: nfs >Affects Versions: 2.1.0-beta > Environment: Red Hat Enterprise 6 with Sun Java 1.7 and IBM Java 1.6 >Reporter: Jinghui Wang > Fix For: 3.0.0, 2.1.1-beta > > Attachments: HDFS-5246-2.patch, HDFS-5246.patch > > > Hadoop nfs binds the nfs server to port 2049, > which is also the default port that Linux nfs uses. If Linux nfs is already > running on the machine then Hadoop nfs will not be albe to start. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5137) Improve WriteManager for processing stable write requests and commit requests
[ https://issues.apache.org/jira/browse/HDFS-5137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13775701#comment-13775701 ] Brandon Li commented on HDFS-5137: -- Hi Jing, you may want to rebase the patch since HDFS-4971 has been resolved. > Improve WriteManager for processing stable write requests and commit requests > - > > Key: HDFS-5137 > URL: https://issues.apache.org/jira/browse/HDFS-5137 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: Jing Zhao >Priority: Minor > Attachments: HDFS-5137.patch > > > WriteManager#handleCommit needs to block until all the data before the > commitOffset has been received. This jira plans to improve the current > concurrency implementation for this blocking call. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5246) Hadoop nfs server binds to port 2049 which is the same as Linux nfs server
[
https://issues.apache.org/jira/browse/HDFS-5246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13775708#comment-13775708
]
Brandon Li commented on HDFS-5246:
--
{quote}...we encountered the failure during a UT job when NFS was running in
the testing env.{quote}
I suspect you might have same port problem with Mount daemon.
> Hadoop nfs server binds to port 2049 which is the same as Linux nfs server
> --
>
> Key: HDFS-5246
> URL: https://issues.apache.org/jira/browse/HDFS-5246
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: nfs
>Affects Versions: 2.1.0-beta
> Environment: Red Hat Enterprise 6 with Sun Java 1.7 and IBM Java 1.6
>Reporter: Jinghui Wang
> Fix For: 3.0.0, 2.1.1-beta
>
> Attachments: HDFS-5246-2.patch, HDFS-5246.patch
>
>
> Hadoop nfs binds the nfs server to port 2049,
> which is also the default port that Linux nfs uses. If Linux nfs is already
> running on the machine then Hadoop nfs will not be albe to start.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5230) Introduce RpcInfo to decouple XDR classes from the RPC API
[ https://issues.apache.org/jira/browse/HDFS-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5230: - Attachment: HDFS-5230.003.patch Address Jing's comments. Add JavaDoc to document how the pipeline works. I agree with Jing that more refactoring can be addressed in HDFS-5086. > Introduce RpcInfo to decouple XDR classes from the RPC API > -- > > Key: HDFS-5230 > URL: https://issues.apache.org/jira/browse/HDFS-5230 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-5230.002.patch, HDFS-5230.003.patch > > > The XDR class is one fundamental aspect in the current implementation of NFS > server. While the client might potentially have a higher level APIs, it also > requires redundant copying since the upstream clients have insufficient > information. > This JIRA introduces a new class, RpcInfo, which (1) decouples XDR from the > APIs, turning it into a utility class, and (2) exposes ChannelBuffer directly > to the client in order to open the opportunity for avoid copying. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5246) Hadoop nfs server binds to port 2049 which is the same as Linux nfs server
[ https://issues.apache.org/jira/browse/HDFS-5246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13775716#comment-13775716 ] Jinghui Wang commented on HDFS-5246: Right. The test was not able to start the Mount daemon since NFS was using port 2049. > Hadoop nfs server binds to port 2049 which is the same as Linux nfs server > -- > > Key: HDFS-5246 > URL: https://issues.apache.org/jira/browse/HDFS-5246 > Project: Hadoop HDFS > Issue Type: Improvement > Components: nfs >Affects Versions: 2.1.0-beta > Environment: Red Hat Enterprise 6 with Sun Java 1.7 and IBM Java 1.6 >Reporter: Jinghui Wang > Fix For: 3.0.0, 2.1.1-beta > > Attachments: HDFS-5246-2.patch, HDFS-5246.patch > > > Hadoop nfs binds the nfs server to port 2049, > which is also the default port that Linux nfs uses. If Linux nfs is already > running on the machine then Hadoop nfs will not be albe to start. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-5249) Fix dumper thread which may die silently
Brandon Li created HDFS-5249: Summary: Fix dumper thread which may die silently Key: HDFS-5249 URL: https://issues.apache.org/jira/browse/HDFS-5249 Project: Hadoop HDFS Issue Type: Sub-task Components: nfs Reporter: Brandon Li Assignee: Brandon Li Dumper thread can get an NPE when the WriteCtx it's about to work on is just deleted by write back thread. A dead dumper thread could cause out-of-memory error when too many pending writes accumulated for one opened file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5249) Fix dumper thread which may die silently
[ https://issues.apache.org/jira/browse/HDFS-5249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-5249: - Status: Patch Available (was: Open) > Fix dumper thread which may die silently > > > Key: HDFS-5249 > URL: https://issues.apache.org/jira/browse/HDFS-5249 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Reporter: Brandon Li >Assignee: Brandon Li > Attachments: HDFS-5249.patch > > > Dumper thread can get an NPE when the WriteCtx it's about to work on is just > deleted by write back thread. > A dead dumper thread could cause out-of-memory error when too many pending > writes accumulated for one opened file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5249) Fix dumper thread which may die silently
[ https://issues.apache.org/jira/browse/HDFS-5249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-5249: - Attachment: HDFS-5249.patch > Fix dumper thread which may die silently > > > Key: HDFS-5249 > URL: https://issues.apache.org/jira/browse/HDFS-5249 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Reporter: Brandon Li >Assignee: Brandon Li > Attachments: HDFS-5249.patch > > > Dumper thread can get an NPE when the WriteCtx it's about to work on is just > deleted by write back thread. > A dead dumper thread could cause out-of-memory error when too many pending > writes accumulated for one opened file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5246) Hadoop nfs server binds to port 2049 which is the same as Linux nfs server
[ https://issues.apache.org/jira/browse/HDFS-5246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13775736#comment-13775736 ] Brandon Li commented on HDFS-5246: -- Mount daemon(implemented inside HDFS-NFS server) uses a different port, default as 4242, as in class RpcProgramMountd. That port is also need to be made configurable as you did for NFS server, so that the Mount daemon(in the HDFS-NFS server) in tests will not use the same default port which is already used by the native Mountd on Linux. I think either way is ok: make mount port configurable here or in a different JIRA. > Hadoop nfs server binds to port 2049 which is the same as Linux nfs server > -- > > Key: HDFS-5246 > URL: https://issues.apache.org/jira/browse/HDFS-5246 > Project: Hadoop HDFS > Issue Type: Improvement > Components: nfs >Affects Versions: 2.1.0-beta > Environment: Red Hat Enterprise 6 with Sun Java 1.7 and IBM Java 1.6 >Reporter: Jinghui Wang > Fix For: 3.0.0, 2.1.1-beta > > Attachments: HDFS-5246-2.patch, HDFS-5246.patch > > > Hadoop nfs binds the nfs server to port 2049, > which is also the default port that Linux nfs uses. If Linux nfs is already > running on the machine then Hadoop nfs will not be albe to start. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-5250) Enable one url to browse the entire federated hdfs file system on dfsclusterhealth summary page
Vrushali C created HDFS-5250: Summary: Enable one url to browse the entire federated hdfs file system on dfsclusterhealth summary page Key: HDFS-5250 URL: https://issues.apache.org/jira/browse/HDFS-5250 Project: Hadoop HDFS Issue Type: New Feature Reporter: Vrushali C Ideally, we should have one url to browse the entire federated file system on the main dfsclusterhealth summary page along with the list of namenodes in the cluster -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5246) Hadoop nfs server binds to port 2049 which is the same as Linux nfs server
[ https://issues.apache.org/jira/browse/HDFS-5246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13775751#comment-13775751 ] Haohui Mai commented on HDFS-5246: -- In my opinion, Portmap is designed to take care of the exact issue -- any servers that register themselves in portmap server should use port 0, and portmap should take care of the rest. > Hadoop nfs server binds to port 2049 which is the same as Linux nfs server > -- > > Key: HDFS-5246 > URL: https://issues.apache.org/jira/browse/HDFS-5246 > Project: Hadoop HDFS > Issue Type: Improvement > Components: nfs >Affects Versions: 2.1.0-beta > Environment: Red Hat Enterprise 6 with Sun Java 1.7 and IBM Java 1.6 >Reporter: Jinghui Wang > Fix For: 3.0.0, 2.1.1-beta > > Attachments: HDFS-5246-2.patch, HDFS-5246.patch > > > Hadoop nfs binds the nfs server to port 2049, > which is also the default port that Linux nfs uses. If Linux nfs is already > running on the machine then Hadoop nfs will not be albe to start. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5191) revisit zero-copy API in FSDataInputStream to make it more intuitive
[ https://issues.apache.org/jira/browse/HDFS-5191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13775753#comment-13775753 ] Colin Patrick McCabe commented on HDFS-5191: re: test timouts, I like to add them at the test level using \@Test(timeout=...), rather than looking at every part of the test. re: hash table sizes. Any time you insert something into IdentityHashStore, you use two array slots... one for the key, and another for the value. The other 2x factor is because the hash table should stay half empty, to avoid the risk of collisions. In other words, we have a load factor of 0.50. This is especially important since every element takes up space (we don't use separate chaining). After thinking about it, I don't think we need the extra +1 element. It seems that System#identityHashCode provides a well-distributed enough hash that dividing by a power of two size works well. IdentityHashStore was necessary because ByteBuffer#hash and ByteBuffer#equals were just very unsuitable for what I was trying to do. And there's no way to parameterize HashTable to use different functions. Another nice advantage of IdentityHashStore is that it does zero allocations, unless you are growing the hash table. This might seem like a trivial point, but given how frequently read is called, it's nice to avoid generating garbage. We learned that when dealing with the edit log code... > revisit zero-copy API in FSDataInputStream to make it more intuitive > > > Key: HDFS-5191 > URL: https://issues.apache.org/jira/browse/HDFS-5191 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client, libhdfs >Affects Versions: HDFS-4949 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HDFS-5191-caching.001.patch, > HDFS-5191-caching.003.patch, HDFS-5191-caching.006.patch, > HDFS-5191-caching.007.patch, HDFS-5191-caching.008.patch > > > As per the discussion on HDFS-4953, we should revisit the zero-copy API to > make it more intuitive for new users. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5249) Fix dumper thread which may die silently
[
https://issues.apache.org/jira/browse/HDFS-5249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13775755#comment-13775755
]
Jing Zhao commented on HDFS-5249:
-
Patch looks pretty good. One comment is that for the dumper thread, we can put
the try-catch under the while loop:
{code}
while (activeState && enabledDump) {
try {
if (nonSequentialWriteInMemory.get() >= DUMP_WRITE_WATER_MARK) {
dump();
}
synchronized (OpenFileCtx.this) {
if (nonSequentialWriteInMemory.get() < DUMP_WRITE_WATER_MARK) {
try {
OpenFileCtx.this.wait();
} catch (InterruptedException e) {
}
}
}
} catch (Throwable t) {
// print and process
}
}
{code}
> Fix dumper thread which may die silently
>
>
> Key: HDFS-5249
> URL: https://issues.apache.org/jira/browse/HDFS-5249
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: nfs
>Reporter: Brandon Li
>Assignee: Brandon Li
> Attachments: HDFS-5249.patch
>
>
> Dumper thread can get an NPE when the WriteCtx it's about to work on is just
> deleted by write back thread.
> A dead dumper thread could cause out-of-memory error when too many pending
> writes accumulated for one opened file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-5251) Race between the initialization of NameNode and the http server
Haohui Mai created HDFS-5251: Summary: Race between the initialization of NameNode and the http server Key: HDFS-5251 URL: https://issues.apache.org/jira/browse/HDFS-5251 Project: Hadoop HDFS Issue Type: Bug Reporter: Haohui Mai Assignee: Haohui Mai The constructor of NameNode starts a HTTP server before the FSNameSystem is initialized. Currently there is a race where the HTTP server can access the uninitialized namesystem variable, throwing a NullPointerException. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5191) revisit zero-copy API in FSDataInputStream to make it more intuitive
[ https://issues.apache.org/jira/browse/HDFS-5191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13775765#comment-13775765 ] Chris Nauroth commented on HDFS-5191: - bq. In other words, we have a load factor of 0.50. That clears it right up for me. Thanks! Do you mind dropping a quick comment on the second 2x multiplication? > revisit zero-copy API in FSDataInputStream to make it more intuitive > > > Key: HDFS-5191 > URL: https://issues.apache.org/jira/browse/HDFS-5191 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client, libhdfs >Affects Versions: HDFS-4949 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HDFS-5191-caching.001.patch, > HDFS-5191-caching.003.patch, HDFS-5191-caching.006.patch, > HDFS-5191-caching.007.patch, HDFS-5191-caching.008.patch > > > As per the discussion on HDFS-4953, we should revisit the zero-copy API to > make it more intuitive for new users. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5249) Fix dumper thread which may die silently
[
https://issues.apache.org/jira/browse/HDFS-5249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13775815#comment-13775815
]
Hadoop QA commented on HDFS-5249:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12604683/HDFS-5249.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs-nfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5020//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5020//console
This message is automatically generated.
> Fix dumper thread which may die silently
>
>
> Key: HDFS-5249
> URL: https://issues.apache.org/jira/browse/HDFS-5249
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: nfs
>Reporter: Brandon Li
>Assignee: Brandon Li
> Attachments: HDFS-5249.patch
>
>
> Dumper thread can get an NPE when the WriteCtx it's about to work on is just
> deleted by write back thread.
> A dead dumper thread could cause out-of-memory error when too many pending
> writes accumulated for one opened file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5137) Improve WriteManager for processing stable write requests and commit requests
[ https://issues.apache.org/jira/browse/HDFS-5137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-5137: Attachment: HDFS-5137.001.patch Rebase the patch. > Improve WriteManager for processing stable write requests and commit requests > - > > Key: HDFS-5137 > URL: https://issues.apache.org/jira/browse/HDFS-5137 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: Jing Zhao >Priority: Minor > Attachments: HDFS-5137.001.patch, HDFS-5137.patch > > > WriteManager#handleCommit needs to block until all the data before the > commitOffset has been received. This jira plans to improve the current > concurrency implementation for this blocking call. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5137) Improve WriteManager for processing stable write requests and commit requests
[ https://issues.apache.org/jira/browse/HDFS-5137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13775825#comment-13775825 ] Jing Zhao commented on HDFS-5137: - Currently the commit request will block the processing thread. In general we should convert it to a non-blocking implementation. Maybe we can do it later in a separate jira. > Improve WriteManager for processing stable write requests and commit requests > - > > Key: HDFS-5137 > URL: https://issues.apache.org/jira/browse/HDFS-5137 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: Jing Zhao >Priority: Minor > Attachments: HDFS-5137.001.patch, HDFS-5137.patch > > > WriteManager#handleCommit needs to block until all the data before the > commitOffset has been received. This jira plans to improve the current > concurrency implementation for this blocking call. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5230) Introduce RpcInfo to decouple XDR classes from the RPC API
[
https://issues.apache.org/jira/browse/HDFS-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13775830#comment-13775830
]
Hadoop QA commented on HDFS-5230:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12604679/HDFS-5230.003.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-nfs hadoop-hdfs-project/hadoop-hdfs-nfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5022//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5022//console
This message is automatically generated.
> Introduce RpcInfo to decouple XDR classes from the RPC API
> --
>
> Key: HDFS-5230
> URL: https://issues.apache.org/jira/browse/HDFS-5230
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: nfs
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-5230.002.patch, HDFS-5230.003.patch
>
>
> The XDR class is one fundamental aspect in the current implementation of NFS
> server. While the client might potentially have a higher level APIs, it also
> requires redundant copying since the upstream clients have insufficient
> information.
> This JIRA introduces a new class, RpcInfo, which (1) decouples XDR from the
> APIs, turning it into a utility class, and (2) exposes ChannelBuffer directly
> to the client in order to open the opportunity for avoid copying.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5222) Move block schedule information from DatanodeDescriptor to DatanodeStorageInfo
[
https://issues.apache.org/jira/browse/HDFS-5222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13775837#comment-13775837
]
Arpit Agarwal commented on HDFS-5222:
-
Hi Nicholas, couple of spurious newlines. +1 otherwise. Please feel free to
commit after removing these.
{code}
private State state;
+
private long capacity;
private long dfsUsed;
private long remaining;
+
private volatile BlockInfo blockList = null;
{code}
> Move block schedule information from DatanodeDescriptor to DatanodeStorageInfo
> --
>
> Key: HDFS-5222
> URL: https://issues.apache.org/jira/browse/HDFS-5222
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h5222_20130819.patch
>
>
> In HDFS-4990, the block placement target type was changed from
> DatanodeDescriptor to DatanodeStorageInfo. The block schedule information,
> such as the number of blocks scheduled for replication (i.e.
> getBlocksScheduled()), should be moved from DatanodeDescriptor to
> DatanodeStorageInfo.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5246) Hadoop nfs server binds to port 2049 which is the same as Linux nfs server
[ https://issues.apache.org/jira/browse/HDFS-5246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13775841#comment-13775841 ] Brandon Li commented on HDFS-5246: -- As Haohui and I discussed offline, in theory using port 0 should work, but more people are used to the default port 2049 for NFS. Therefore, let's go with the approach in the current patch: make the port configurable, but use 2049 as default port for NFS. > Hadoop nfs server binds to port 2049 which is the same as Linux nfs server > -- > > Key: HDFS-5246 > URL: https://issues.apache.org/jira/browse/HDFS-5246 > Project: Hadoop HDFS > Issue Type: Improvement > Components: nfs >Affects Versions: 2.1.0-beta > Environment: Red Hat Enterprise 6 with Sun Java 1.7 and IBM Java 1.6 >Reporter: Jinghui Wang > Fix For: 3.0.0, 2.1.1-beta > > Attachments: HDFS-5246-2.patch, HDFS-5246.patch > > > Hadoop nfs binds the nfs server to port 2049, > which is also the default port that Linux nfs uses. If Linux nfs is already > running on the machine then Hadoop nfs will not be albe to start. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5246) Hadoop nfs server binds to port 2049 which is the same as Linux nfs server
[
https://issues.apache.org/jira/browse/HDFS-5246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13775840#comment-13775840
]
Hadoop QA commented on HDFS-5246:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12604673/HDFS-5246-2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-common-project/hadoop-nfs
hadoop-hdfs-project/hadoop-hdfs-nfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5021//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5021//console
This message is automatically generated.
> Hadoop nfs server binds to port 2049 which is the same as Linux nfs server
> --
>
> Key: HDFS-5246
> URL: https://issues.apache.org/jira/browse/HDFS-5246
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: nfs
>Affects Versions: 2.1.0-beta
> Environment: Red Hat Enterprise 6 with Sun Java 1.7 and IBM Java 1.6
>Reporter: Jinghui Wang
> Fix For: 3.0.0, 2.1.1-beta
>
> Attachments: HDFS-5246-2.patch, HDFS-5246.patch
>
>
> Hadoop nfs binds the nfs server to port 2049,
> which is also the default port that Linux nfs uses. If Linux nfs is already
> running on the machine then Hadoop nfs will not be albe to start.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5246) Hadoop nfs server binds to port 2049 which is the same as Linux nfs server
[ https://issues.apache.org/jira/browse/HDFS-5246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13775842#comment-13775842 ] Brandon Li commented on HDFS-5246: -- [~jwang302] Please let us know if you want to address the same issue for RpcProgramMountd in this JIRA. > Hadoop nfs server binds to port 2049 which is the same as Linux nfs server > -- > > Key: HDFS-5246 > URL: https://issues.apache.org/jira/browse/HDFS-5246 > Project: Hadoop HDFS > Issue Type: Improvement > Components: nfs >Affects Versions: 2.1.0-beta > Environment: Red Hat Enterprise 6 with Sun Java 1.7 and IBM Java 1.6 >Reporter: Jinghui Wang > Fix For: 3.0.0, 2.1.1-beta > > Attachments: HDFS-5246-2.patch, HDFS-5246.patch > > > Hadoop nfs binds the nfs server to port 2049, > which is also the default port that Linux nfs uses. If Linux nfs is already > running on the machine then Hadoop nfs will not be albe to start. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5230) Introduce RpcInfo to decouple XDR classes from the RPC API
[ https://issues.apache.org/jira/browse/HDFS-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13775851#comment-13775851 ] Jing Zhao commented on HDFS-5230: - [~wheat9], I guess you want to move 5&6 to HDFS-5086? Looks fine to me. Could you post how you test the patch and the result of your test? > Introduce RpcInfo to decouple XDR classes from the RPC API > -- > > Key: HDFS-5230 > URL: https://issues.apache.org/jira/browse/HDFS-5230 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-5230.002.patch, HDFS-5230.003.patch > > > The XDR class is one fundamental aspect in the current implementation of NFS > server. While the client might potentially have a higher level APIs, it also > requires redundant copying since the upstream clients have insufficient > information. > This JIRA introduces a new class, RpcInfo, which (1) decouples XDR from the > APIs, turning it into a utility class, and (2) exposes ChannelBuffer directly > to the client in order to open the opportunity for avoid copying. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5230) Introduce RpcInfo to decouple XDR classes from the RPC API
[ https://issues.apache.org/jira/browse/HDFS-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13775854#comment-13775854 ] Brandon Li commented on HDFS-5230: -- Please give me a few hours. I will also take a look of the patch tomorrow if I can't get to it tonight. > Introduce RpcInfo to decouple XDR classes from the RPC API > -- > > Key: HDFS-5230 > URL: https://issues.apache.org/jira/browse/HDFS-5230 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-5230.002.patch, HDFS-5230.003.patch > > > The XDR class is one fundamental aspect in the current implementation of NFS > server. While the client might potentially have a higher level APIs, it also > requires redundant copying since the upstream clients have insufficient > information. > This JIRA introduces a new class, RpcInfo, which (1) decouples XDR from the > APIs, turning it into a utility class, and (2) exposes ChannelBuffer directly > to the client in order to open the opportunity for avoid copying. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5249) Fix dumper thread which may die silently
[ https://issues.apache.org/jira/browse/HDFS-5249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-5249: - Attachment: HDFS-5249.02.patch Upload a new patch to address Jing's comment. > Fix dumper thread which may die silently > > > Key: HDFS-5249 > URL: https://issues.apache.org/jira/browse/HDFS-5249 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Reporter: Brandon Li >Assignee: Brandon Li > Attachments: HDFS-5249.02.patch, HDFS-5249.patch > > > Dumper thread can get an NPE when the WriteCtx it's about to work on is just > deleted by write back thread. > A dead dumper thread could cause out-of-memory error when too many pending > writes accumulated for one opened file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5249) Fix dumper thread which may die silently
[ https://issues.apache.org/jira/browse/HDFS-5249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13775859#comment-13775859 ] Brandon Li commented on HDFS-5249: -- No unit test included since it's difficult to time the race condition in a unit test. I manually tested file loading on a Linux cluster. > Fix dumper thread which may die silently > > > Key: HDFS-5249 > URL: https://issues.apache.org/jira/browse/HDFS-5249 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Reporter: Brandon Li >Assignee: Brandon Li > Attachments: HDFS-5249.02.patch, HDFS-5249.patch > > > Dumper thread can get an NPE when the WriteCtx it's about to work on is just > deleted by write back thread. > A dead dumper thread could cause out-of-memory error when too many pending > writes accumulated for one opened file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5249) Fix dumper thread which may die silently
[ https://issues.apache.org/jira/browse/HDFS-5249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13775860#comment-13775860 ] Jing Zhao commented on HDFS-5249: - +1 for the patch. > Fix dumper thread which may die silently > > > Key: HDFS-5249 > URL: https://issues.apache.org/jira/browse/HDFS-5249 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Reporter: Brandon Li >Assignee: Brandon Li > Attachments: HDFS-5249.02.patch, HDFS-5249.patch > > > Dumper thread can get an NPE when the WriteCtx it's about to work on is just > deleted by write back thread. > A dead dumper thread could cause out-of-memory error when too many pending > writes accumulated for one opened file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5250) Enable one url to browse the entire federated hdfs file system on dfsclusterhealth summary page
[ https://issues.apache.org/jira/browse/HDFS-5250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13775861#comment-13775861 ] Andrew Wang commented on HDFS-5250: --- Hi Vrushali, ViewFileSystem is a purely client-side abstraction, each client can have their own viewfs mount points, and the federated namenodes are unaware of where they are mounted as well as the list of other namenodes/mount points. I think something like this is only really possible with a single sharded namespace, not the client-side mount table that ViewFileSystem provides. Maybe I missed something, but we should probably close this JIRA as wontfix. > Enable one url to browse the entire federated hdfs file system on > dfsclusterhealth summary page > --- > > Key: HDFS-5250 > URL: https://issues.apache.org/jira/browse/HDFS-5250 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Vrushali C > > Ideally, we should have one url to browse the entire federated file system on > the main dfsclusterhealth summary page along with the list of namenodes in > the cluster -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-5252) Do unstable write only when stable write can't be honored
Brandon Li created HDFS-5252: Summary: Do unstable write only when stable write can't be honored Key: HDFS-5252 URL: https://issues.apache.org/jira/browse/HDFS-5252 Project: Hadoop HDFS Issue Type: Sub-task Components: nfs Reporter: Brandon Li When the client asks for a stable write but the prerequisite writes are not transferred to NFS gateway, the stableness can't be honored. NFS gateway has to treat the write as unstable write. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5249) Fix dumper thread which may die silently
[
https://issues.apache.org/jira/browse/HDFS-5249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13775890#comment-13775890
]
Hadoop QA commented on HDFS-5249:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12604706/HDFS-5249.02.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs-nfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5025//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5025//console
This message is automatically generated.
> Fix dumper thread which may die silently
>
>
> Key: HDFS-5249
> URL: https://issues.apache.org/jira/browse/HDFS-5249
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: nfs
>Reporter: Brandon Li
>Assignee: Brandon Li
> Attachments: HDFS-5249.02.patch, HDFS-5249.patch
>
>
> Dumper thread can get an NPE when the WriteCtx it's about to work on is just
> deleted by write back thread.
> A dead dumper thread could cause out-of-memory error when too many pending
> writes accumulated for one opened file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5251) Race between the initialization of NameNode and the http server
[ https://issues.apache.org/jira/browse/HDFS-5251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5251: - Attachment: HDFS-5251.000.patch > Race between the initialization of NameNode and the http server > --- > > Key: HDFS-5251 > URL: https://issues.apache.org/jira/browse/HDFS-5251 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-5251.000.patch > > > The constructor of NameNode starts a HTTP server before the FSNameSystem is > initialized. Currently there is a race where the HTTP server can access the > uninitialized namesystem variable, throwing a NullPointerException. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5251) Race between the initialization of NameNode and the http server
[ https://issues.apache.org/jira/browse/HDFS-5251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5251: - Status: Patch Available (was: Open) > Race between the initialization of NameNode and the http server > --- > > Key: HDFS-5251 > URL: https://issues.apache.org/jira/browse/HDFS-5251 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-5251.000.patch > > > The constructor of NameNode starts a HTTP server before the FSNameSystem is > initialized. Currently there is a race where the HTTP server can access the > uninitialized namesystem variable, throwing a NullPointerException. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5237) Get rid of nodes' registration names
[ https://issues.apache.org/jira/browse/HDFS-5237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13775898#comment-13775898 ] Eli Collins commented on HDFS-5237: --- Agree it's reasonable to validate dfs.datanode.hostname, but Junping's patch removes this config option which will break some configurations that use HDFS-3150 and need to set the reported hostname to something other than the default (for example if getDefaultHost returns a hostname on network A and the clients want to talk to the DN via network B). Shall we re-purpose this to add dfs.datanode.hostname validation? > Get rid of nodes' registration names > > > Key: HDFS-5237 > URL: https://issues.apache.org/jira/browse/HDFS-5237 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Junping Du > Fix For: 3.0.0 > > Attachments: HDFS-5237.patch > > > Per discussion in HDFS-5208 and may be some other discussions before, Node's > registration name is pretty confusing and shouldn't be used in production > environment as topology resolving issues. So we remove related configuration > dfs.datanode.hostname or its old one "slave.host.name". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5225) datanode keeps logging the same 'is no longer in the dataset' message over and over again
[
https://issues.apache.org/jira/browse/HDFS-5225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsuyoshi OZAWA updated HDFS-5225:
-
Attachment: HDFS-5225-reproduce.1.txt
A patch for reproducing this problem.
The analysis of the code is as follows:
1. DataBlockScanner#run() tries to scan blocks via
BlockPoolSliceScanner#scanBlockPoolSlice in the main loop.
2. At the same time, the other theads(BlockReceiver's constructor) issue
blockScanner.deleteBlock.
3. BlockPoolSliceScanner#deleteBlock() tries to remove BlockScanInfo from
blockInfoSet. However, this is not visible from
BlockPoolSliceScanner#scanBlockPoolSlice's thread, because there is no memory
barrier. This may be the critical section we've faced.
4. Other threads issue FsDatasetImpl#unfinalizeBlock, FsDataImpl,
checkAndUpdate, and updates BlockPoolSliceScanner#dataset, and remove the block
entry from dataset. This is because the log 'is no longer in the dataset' is
dumped repeatedly.
Note taht this may be not complete: I've faced the log 'is no longer in the
dataset' dumped repeatedly, but sometimes with my code.
> datanode keeps logging the same 'is no longer in the dataset' message over
> and over again
> -
>
> Key: HDFS-5225
> URL: https://issues.apache.org/jira/browse/HDFS-5225
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.1.1-beta
>Reporter: Roman Shaposhnik
> Attachments: HDFS-5225.1.patch, HDFS-5225-reproduce.1.txt
>
>
> I was running the usual Bigtop testing on 2.1.1-beta RC1 with the following
> configuration: 4 nodes fully distributed cluster with security on.
> All of a sudden my DN ate up all of the space in /var/log logging the
> following message repeatedly:
> {noformat}
> 2013-09-18 20:51:12,046 INFO
> org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner:
> BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1369 is no longer
> in the dataset
> {noformat}
> It wouldn't answer to a jstack and jstack -F ended up being useless.
> Here's what I was able to find in the NameNode logs regarding this block ID:
> {noformat}
> fgrep -rI 'blk_1073742189' hadoop-hdfs-namenode-ip-10-224-158-152.log
> 2013-09-18 18:03:16,972 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> allocateBlock:
> /user/jenkins/testAppendInputWedSep18180222UTC2013/test4.fileWedSep18180222UTC2013._COPYING_.
> BP-1884637155-10.224.158.152-1379524544853
> blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
> ReplicaUnderConstruction[10.34.74.206:1004|RBW],
> ReplicaUnderConstruction[10.224.158.152:1004|RBW]]}
> 2013-09-18 18:03:17,222 INFO BlockStateChange: BLOCK* addStoredBlock:
> blockMap updated: 10.224.158.152:1004 is added to
> blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
> ReplicaUnderConstruction[10.34.74.206:1004|RBW],
> ReplicaUnderConstruction[10.224.158.152:1004|RBW]]} size 0
> 2013-09-18 18:03:17,222 INFO BlockStateChange: BLOCK* addStoredBlock:
> blockMap updated: 10.34.74.206:1004 is added to
> blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
> ReplicaUnderConstruction[10.34.74.206:1004|RBW],
> ReplicaUnderConstruction[10.224.158.152:1004|RBW]]} size 0
> 2013-09-18 18:03:17,224 INFO BlockStateChange: BLOCK* addStoredBlock:
> blockMap updated: 10.83.107.80:1004 is added to
> blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
> ReplicaUnderConstruction[10.34.74.206:1004|RBW],
> ReplicaUnderConstruction[10.224.158.152:1004|RBW]]} size 0
> 2013-09-18 18:03:17,899 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> updatePipeline(block=BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1369,
> newGenerationStamp=1370, newLength=1048576, newNodes=[10.83.107.80:1004,
> 10.34.74.206:1004, 10.224.158.152:1004],
> clientName=DFSClient_NONMAPREDUCE_-450304083_1)
> 2013-09-18 18:03:17,904 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> updatePipeline(BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1369)
> successfully to
> BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1370
> 2013-09-18 18:03:18,540 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> updatePipeline(block=BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1370,
> newGenerationStamp=1371, newLength=2097152, newNodes=[10.83.107.80:1004,
> 10.34.74.206:1004, 10.224.158.152:1004],
> clientName=DFSClient_NONMAPREDUCE_-450304083_1)
> 2013-09-18 18:03
[jira] [Created] (HDFS-5253) Add requesting user's name to PathBasedCacheEntry
Andrew Wang created HDFS-5253:
-
Summary: Add requesting user's name to PathBasedCacheEntry
Key: HDFS-5253
URL: https://issues.apache.org/jira/browse/HDFS-5253
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-4949
Reporter: Andrew Wang
Assignee: Andrew Wang
It'll be useful to have the requesting user's name in {{PathBasedCacheEntry}}
for tracking per-user statistics (e.g. amount of data cached by a user).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5225) datanode keeps logging the same 'is no longer in the dataset' message over and over again
[
https://issues.apache.org/jira/browse/HDFS-5225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13775924#comment-13775924
]
Tsuyoshi OZAWA commented on HDFS-5225:
--
{quote}
Note that this may be not complete: I've faced the log 'is no longer in the
dataset' dumped repeatedly, but sometimes with my code.
{quote}
I mean the log messages itself are dumped every time, but the "logging the same
'is no longer in the dataset' message over and over again" can happen sometimes.
> datanode keeps logging the same 'is no longer in the dataset' message over
> and over again
> -
>
> Key: HDFS-5225
> URL: https://issues.apache.org/jira/browse/HDFS-5225
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.1.1-beta
>Reporter: Roman Shaposhnik
> Attachments: HDFS-5225.1.patch, HDFS-5225-reproduce.1.txt
>
>
> I was running the usual Bigtop testing on 2.1.1-beta RC1 with the following
> configuration: 4 nodes fully distributed cluster with security on.
> All of a sudden my DN ate up all of the space in /var/log logging the
> following message repeatedly:
> {noformat}
> 2013-09-18 20:51:12,046 INFO
> org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner:
> BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1369 is no longer
> in the dataset
> {noformat}
> It wouldn't answer to a jstack and jstack -F ended up being useless.
> Here's what I was able to find in the NameNode logs regarding this block ID:
> {noformat}
> fgrep -rI 'blk_1073742189' hadoop-hdfs-namenode-ip-10-224-158-152.log
> 2013-09-18 18:03:16,972 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> allocateBlock:
> /user/jenkins/testAppendInputWedSep18180222UTC2013/test4.fileWedSep18180222UTC2013._COPYING_.
> BP-1884637155-10.224.158.152-1379524544853
> blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
> ReplicaUnderConstruction[10.34.74.206:1004|RBW],
> ReplicaUnderConstruction[10.224.158.152:1004|RBW]]}
> 2013-09-18 18:03:17,222 INFO BlockStateChange: BLOCK* addStoredBlock:
> blockMap updated: 10.224.158.152:1004 is added to
> blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
> ReplicaUnderConstruction[10.34.74.206:1004|RBW],
> ReplicaUnderConstruction[10.224.158.152:1004|RBW]]} size 0
> 2013-09-18 18:03:17,222 INFO BlockStateChange: BLOCK* addStoredBlock:
> blockMap updated: 10.34.74.206:1004 is added to
> blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
> ReplicaUnderConstruction[10.34.74.206:1004|RBW],
> ReplicaUnderConstruction[10.224.158.152:1004|RBW]]} size 0
> 2013-09-18 18:03:17,224 INFO BlockStateChange: BLOCK* addStoredBlock:
> blockMap updated: 10.83.107.80:1004 is added to
> blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
> ReplicaUnderConstruction[10.34.74.206:1004|RBW],
> ReplicaUnderConstruction[10.224.158.152:1004|RBW]]} size 0
> 2013-09-18 18:03:17,899 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> updatePipeline(block=BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1369,
> newGenerationStamp=1370, newLength=1048576, newNodes=[10.83.107.80:1004,
> 10.34.74.206:1004, 10.224.158.152:1004],
> clientName=DFSClient_NONMAPREDUCE_-450304083_1)
> 2013-09-18 18:03:17,904 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> updatePipeline(BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1369)
> successfully to
> BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1370
> 2013-09-18 18:03:18,540 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> updatePipeline(block=BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1370,
> newGenerationStamp=1371, newLength=2097152, newNodes=[10.83.107.80:1004,
> 10.34.74.206:1004, 10.224.158.152:1004],
> clientName=DFSClient_NONMAPREDUCE_-450304083_1)
> 2013-09-18 18:03:18,548 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> updatePipeline(BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1370)
> successfully to
> BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1371
> 2013-09-18 18:03:26,150 INFO BlockStateChange: BLOCK* addToInvalidates:
> blk_1073742189_1371 10.83.107.80:1004 10.34.74.206:1004 10.224.158.152:1004
> 2013-09-18 18:03:26,847 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> InvalidateBlocks: ask 10.34.74.206:1004 to delete [blk_1073742178_1359,
> blk_1073742183_1362, blk_1073742184_1363, blk_1073742186_136
[jira] [Commented] (HDFS-5251) Race between the initialization of NameNode and the http server
[ https://issues.apache.org/jira/browse/HDFS-5251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13775926#comment-13775926 ] Suresh Srinivas commented on HDFS-5251: --- +1 for the change. I will commit it shortly. > Race between the initialization of NameNode and the http server > --- > > Key: HDFS-5251 > URL: https://issues.apache.org/jira/browse/HDFS-5251 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-5251.000.patch > > > The constructor of NameNode starts a HTTP server before the FSNameSystem is > initialized. Currently there is a race where the HTTP server can access the > uninitialized namesystem variable, throwing a NullPointerException. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5242) Reduce contention on DatanodeInfo instances
[
https://issues.apache.org/jira/browse/HDFS-5242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13775927#comment-13775927
]
Hadoop QA commented on HDFS-5242:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12604625/HDFS-5242.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5024//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5024//console
This message is automatically generated.
> Reduce contention on DatanodeInfo instances
> ---
>
> Key: HDFS-5242
> URL: https://issues.apache.org/jira/browse/HDFS-5242
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: 2.0.0-alpha, 3.0.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: HDFS-5242.patch
>
>
> Synchronization in {{DatanodeInfo}} instances causes unnecessary contention
> between call handlers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5216) NumDecomDeadDataNodes not returning correct number of dead decommissioned nodes
[
https://issues.apache.org/jira/browse/HDFS-5216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13775928#comment-13775928
]
Hadoop QA commented on HDFS-5216:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12604543/HDFS-5216.diff
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5023//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5023//console
This message is automatically generated.
> NumDecomDeadDataNodes not returning correct number of dead decommissioned
> nodes
>
>
> Key: HDFS-5216
> URL: https://issues.apache.org/jira/browse/HDFS-5216
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.1.0-beta
>Reporter: Trevor Lorimer
>Assignee: Trevor Lorimer
> Attachments: HDFS-5216.diff, HDFS-5216.diff
>
>
> For HDFS-4860 I essentially copied the process in
> NamenodejspHelper.generateHealthReport(), so it would be in sync with the
> original dfsHealth.jsp.
> However looking at this now there may be a bug? in
> getNumDecomDeadDataNodes(), where:
> getBlockManager().getDatanodeManager().fetchDatanodes(dead, null, true);
>
> Where the parameter true indicates that decommissioned nodes should be
> removed from the list.
> If the flag is true fetchDatanodes calls removeDecomNodeFromList, which will
> remove a node if an existing datanode does not appear in both include or
> exclude lists and it has been decommissioned.
> If I am looking to return the Number of Dead Decommissioned Nodes, should I
> change the remove decommissioned nodes flag to False? i.e.:
> getBlockManager().getDatanodeManager().fetchDatanodes(null, dead, false);
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5250) Enable one url to browse the entire federated hdfs file system on dfsclusterhealth summary page
[ https://issues.apache.org/jira/browse/HDFS-5250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13775932#comment-13775932 ] Suresh Srinivas commented on HDFS-5250: --- [~vrushalic], dfscluster health page was a way to show summary of all the namenodes in the federated cluster. Currently you can go to each namenode from that page and browse the file system supported on each of them. I agree, this should be closed as wontfix. > Enable one url to browse the entire federated hdfs file system on > dfsclusterhealth summary page > --- > > Key: HDFS-5250 > URL: https://issues.apache.org/jira/browse/HDFS-5250 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Vrushali C > > Ideally, we should have one url to browse the entire federated file system on > the main dfsclusterhealth summary page along with the list of namenodes in > the cluster -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5249) Fix dumper thread which may die silently
[ https://issues.apache.org/jira/browse/HDFS-5249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13775977#comment-13775977 ] Brandon Li commented on HDFS-5249: -- Thank you Jing for the review. I've committed the patch. > Fix dumper thread which may die silently > > > Key: HDFS-5249 > URL: https://issues.apache.org/jira/browse/HDFS-5249 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Reporter: Brandon Li >Assignee: Brandon Li > Attachments: HDFS-5249.02.patch, HDFS-5249.patch > > > Dumper thread can get an NPE when the WriteCtx it's about to work on is just > deleted by write back thread. > A dead dumper thread could cause out-of-memory error when too many pending > writes accumulated for one opened file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5249) Fix dumper thread which may die silently
[ https://issues.apache.org/jira/browse/HDFS-5249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-5249: - Fix Version/s: 2.1.1-beta > Fix dumper thread which may die silently > > > Key: HDFS-5249 > URL: https://issues.apache.org/jira/browse/HDFS-5249 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Reporter: Brandon Li >Assignee: Brandon Li > Fix For: 2.1.1-beta > > Attachments: HDFS-5249.02.patch, HDFS-5249.patch > > > Dumper thread can get an NPE when the WriteCtx it's about to work on is just > deleted by write back thread. > A dead dumper thread could cause out-of-memory error when too many pending > writes accumulated for one opened file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5249) Fix dumper thread which may die silently
[ https://issues.apache.org/jira/browse/HDFS-5249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-5249: - Resolution: Fixed Status: Resolved (was: Patch Available) > Fix dumper thread which may die silently > > > Key: HDFS-5249 > URL: https://issues.apache.org/jira/browse/HDFS-5249 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Reporter: Brandon Li >Assignee: Brandon Li > Fix For: 2.1.1-beta > > Attachments: HDFS-5249.02.patch, HDFS-5249.patch > > > Dumper thread can get an NPE when the WriteCtx it's about to work on is just > deleted by write back thread. > A dead dumper thread could cause out-of-memory error when too many pending > writes accumulated for one opened file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5251) Race between the initialization of NameNode and the http server
[
https://issues.apache.org/jira/browse/HDFS-5251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13775980#comment-13775980
]
Hadoop QA commented on HDFS-5251:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12604720/HDFS-5251.000.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5026//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5026//console
This message is automatically generated.
> Race between the initialization of NameNode and the http server
> ---
>
> Key: HDFS-5251
> URL: https://issues.apache.org/jira/browse/HDFS-5251
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-5251.000.patch
>
>
> The constructor of NameNode starts a HTTP server before the FSNameSystem is
> initialized. Currently there is a race where the HTTP server can access the
> uninitialized namesystem variable, throwing a NullPointerException.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5249) Fix dumper thread which may die silently
[ https://issues.apache.org/jira/browse/HDFS-5249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13775985#comment-13775985 ] Hudson commented on HDFS-5249: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4459 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4459/]) HDFS-5249. Fix dumper thread which may die silently. Contributed by Brandon Li (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1525770) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OpenFileCtx.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > Fix dumper thread which may die silently > > > Key: HDFS-5249 > URL: https://issues.apache.org/jira/browse/HDFS-5249 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Reporter: Brandon Li >Assignee: Brandon Li > Fix For: 2.1.1-beta > > Attachments: HDFS-5249.02.patch, HDFS-5249.patch > > > Dumper thread can get an NPE when the WriteCtx it's about to work on is just > deleted by write back thread. > A dead dumper thread could cause out-of-memory error when too many pending > writes accumulated for one opened file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5251) Race between the initialization of NameNode and the http server
[ https://issues.apache.org/jira/browse/HDFS-5251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-5251: -- Resolution: Fixed Fix Version/s: 2.2.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I committed the patch to trunk, branch-2 and branch-2.1-beta. Thank you Haohui! > Race between the initialization of NameNode and the http server > --- > > Key: HDFS-5251 > URL: https://issues.apache.org/jira/browse/HDFS-5251 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: 2.2.0 > > Attachments: HDFS-5251.000.patch > > > The constructor of NameNode starts a HTTP server before the FSNameSystem is > initialized. Currently there is a race where the HTTP server can access the > uninitialized namesystem variable, throwing a NullPointerException. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5251) Race between the initialization of NameNode and the http server
[ https://issues.apache.org/jira/browse/HDFS-5251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13776015#comment-13776015 ] Hudson commented on HDFS-5251: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4460 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4460/]) HDFS-5251. Race between the initialization of NameNode and the http server. Contributed by Haohui Mai. (suresh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1525787) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeJspHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeJspHelper.java > Race between the initialization of NameNode and the http server > --- > > Key: HDFS-5251 > URL: https://issues.apache.org/jira/browse/HDFS-5251 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: 2.2.0 > > Attachments: HDFS-5251.000.patch > > > The constructor of NameNode starts a HTTP server before the FSNameSystem is > initialized. Currently there is a race where the HTTP server can access the > uninitialized namesystem variable, throwing a NullPointerException. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
