[jira] [Updated] (HDFS-1568) Improve DataXceiver error logging
[ https://issues.apache.org/jira/browse/HDFS-1568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Echeverria updated HDFS-1568: -- Status: Open (was: Patch Available) Canceling patch to upload one rebased to trunk. > Improve DataXceiver error logging > - > > Key: HDFS-1568 > URL: https://issues.apache.org/jira/browse/HDFS-1568 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Joey Echeverria >Priority: Minor > Labels: newbie > Attachments: HDFS-1568-1.patch, HDFS-1568-3.patch, HDFS-1568-4.patch, > HDFS-1568-5.patch, HDFS-1568-output-changes.patch > > > In supporting customers we often see things like SocketTimeoutExceptions or > EOFExceptions coming from DataXceiver, but the logging isn't very good. For > example, if we get an IOE while setting up a connection to the downstream > mirror in writeBlock, the IP of the downstream mirror isn't logged on the DN > side. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1568) Improve DataXceiver error logging
[ https://issues.apache.org/jira/browse/HDFS-1568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Echeverria updated HDFS-1568: -- Attachment: HDFS-1568-5.patch Rebased patch to trunk. > Improve DataXceiver error logging > - > > Key: HDFS-1568 > URL: https://issues.apache.org/jira/browse/HDFS-1568 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Joey Echeverria >Priority: Minor > Labels: newbie > Attachments: HDFS-1568-1.patch, HDFS-1568-3.patch, HDFS-1568-4.patch, > HDFS-1568-5.patch, HDFS-1568-output-changes.patch > > > In supporting customers we often see things like SocketTimeoutExceptions or > EOFExceptions coming from DataXceiver, but the logging isn't very good. For > example, if we get an IOE while setting up a connection to the downstream > mirror in writeBlock, the IP of the downstream mirror isn't logged on the DN > side. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1568) Improve DataXceiver error logging
[ https://issues.apache.org/jira/browse/HDFS-1568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Echeverria updated HDFS-1568: -- Status: Patch Available (was: Open) > Improve DataXceiver error logging > - > > Key: HDFS-1568 > URL: https://issues.apache.org/jira/browse/HDFS-1568 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Joey Echeverria >Priority: Minor > Labels: newbie > Attachments: HDFS-1568-1.patch, HDFS-1568-3.patch, HDFS-1568-4.patch, > HDFS-1568-5.patch, HDFS-1568-6.patch, HDFS-1568-output-changes.patch > > > In supporting customers we often see things like SocketTimeoutExceptions or > EOFExceptions coming from DataXceiver, but the logging isn't very good. For > example, if we get an IOE while setting up a connection to the downstream > mirror in writeBlock, the IP of the downstream mirror isn't logged on the DN > side. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1568) Improve DataXceiver error logging
[ https://issues.apache.org/jira/browse/HDFS-1568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Echeverria updated HDFS-1568: -- Attachment: HDFS-1568-6.patch Oops, jumped the gun on that last upload. This patch should be good. > Improve DataXceiver error logging > - > > Key: HDFS-1568 > URL: https://issues.apache.org/jira/browse/HDFS-1568 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Joey Echeverria >Priority: Minor > Labels: newbie > Attachments: HDFS-1568-1.patch, HDFS-1568-3.patch, HDFS-1568-4.patch, > HDFS-1568-5.patch, HDFS-1568-6.patch, HDFS-1568-output-changes.patch > > > In supporting customers we often see things like SocketTimeoutExceptions or > EOFExceptions coming from DataXceiver, but the logging isn't very good. For > example, if we get an IOE while setting up a connection to the downstream > mirror in writeBlock, the IP of the downstream mirror isn't logged on the DN > side. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1568) Improve DataXceiver error logging
[ https://issues.apache.org/jira/browse/HDFS-1568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051550#comment-13051550 ] Hadoop QA commented on HDFS-1568: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12483041/HDFS-1568-6.patch against trunk revision 1136975. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/796//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/796//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/796//console This message is automatically generated. > Improve DataXceiver error logging > - > > Key: HDFS-1568 > URL: https://issues.apache.org/jira/browse/HDFS-1568 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Joey Echeverria >Priority: Minor > Labels: newbie > Attachments: HDFS-1568-1.patch, HDFS-1568-3.patch, HDFS-1568-4.patch, > HDFS-1568-5.patch, HDFS-1568-6.patch, HDFS-1568-output-changes.patch > > > In supporting customers we often see things like SocketTimeoutExceptions or > EOFExceptions coming from DataXceiver, but the logging isn't very good. For > example, if we get an IOE while setting up a connection to the downstream > mirror in writeBlock, the IP of the downstream mirror isn't logged on the DN > side. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2082) SecondayNameNode web interface doesn't show the right info
[ https://issues.apache.org/jira/browse/HDFS-2082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051549#comment-13051549 ] Hadoop QA commented on HDFS-2082: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12483014/hdfs-2082.3.patch against trunk revision 1136975. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: -1 contrib tests. The patch failed contrib unit tests. -1 system test framework. The patch failed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/795//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/795//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/795//console This message is automatically generated. > SecondayNameNode web interface doesn't show the right info > -- > > Key: HDFS-2082 > URL: https://issues.apache.org/jira/browse/HDFS-2082 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.22.0, 0.23.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Fix For: 0.23.0 > > Attachments: hdfs-2082.0.patch, hdfs-2082.1.patch, hdfs-2082.2.patch, > hdfs-2082.3.patch > > > HADOOP-3741 introduced some useful info to the 2NN web UI. This broke when > security was added. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2034) length in getBlockRange becomes -ve when reading only from currently being written blk
[
https://issues.apache.org/jira/browse/HDFS-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John George updated HDFS-2034:
--
Attachment: HDFS-2034-4.patch
New patch with Tod's and Daryn's comments included.
> length in getBlockRange becomes -ve when reading only from currently being
> written blk
> --
>
> Key: HDFS-2034
> URL: https://issues.apache.org/jira/browse/HDFS-2034
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: John George
>Assignee: John George
>Priority: Minor
> Attachments: HDFS-2034-1.patch, HDFS-2034-1.patch, HDFS-2034-2.patch,
> HDFS-2034-3.patch, HDFS-2034-4.patch, HDFS-2034.patch
>
>
> This came up during HDFS-1907. Posting an example that Todd posted in
> HDFS-1907 that brought out this issue.
> {quote}
> Here's an example sequence to describe what I mean:
> 1. open file, write one and a half blocks
> 2. call hflush
> 3. another reader asks for the first byte of the second block
> {quote}
> In this case since offset is greater than the completed block length, the
> math in getBlockRange() of DFSInputStreamer.java will set "length" to
> negative.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2081) trying to read configuration in hdfs write path might result in deadlock
[ https://issues.apache.org/jira/browse/HDFS-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John George updated HDFS-2081: -- Description: While trying to run yarn with 1024 block size, it looks as though the dfsclients are hung waiting to write to DN. The reason is because yarn holds the configuration file lock and tries to write to hdfs which in turn tries to read from the configuration which requires the same lock. The temporary fix is to make sure that dfsclient threads dont read the config in its write path. There might be better long term solutions like read/write locks and this needs to be investigated in another JIRA. was: While trying to run yarn with 1024 block size, it looks as though the dfsclients are hung waiting to write to DN. The reason is because yarn golds the configuration file lock and tries to write to hdfs which in turn tries to read from the configuration which requires the same lock. The temporary fix is to make sure that dfsclient threads dont read the config in its write path. There might be better long term solutions like read/write locks and this needs to be investigated in another JIRA. Affects Version/s: 0.23.0 Fix Version/s: 0.23.0 > trying to read configuration in hdfs write path might result in deadlock > > > Key: HDFS-2081 > URL: https://issues.apache.org/jira/browse/HDFS-2081 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.23.0 > Environment: MR-279 branch. This is specific to yarn since yarn tries > to write the configuration file to hdfs while holding the lock of the > configuration file. >Reporter: John George >Assignee: John George > Fix For: 0.23.0 > > Attachments: HDFS-2081.patch > > > While trying to run yarn with 1024 block size, it looks as though the > dfsclients are hung waiting to write to DN. The reason is because yarn holds > the configuration file lock and tries to write to hdfs which in turn tries to > read from the configuration which requires the same lock. > The temporary fix is to make sure that dfsclient threads dont read the config > in its write path. There might be better long term solutions like read/write > locks and this needs to be investigated in another JIRA. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1788) FsShell ls: Show symlinks properties
[ https://issues.apache.org/jira/browse/HDFS-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051629#comment-13051629 ] John George commented on HDFS-1788: --- Bochun, It is good that you submitted a patch for this ticket, because the important thing is for this issue to be resolved. I had a patch ready but it needed some cleanup since some of the junit tests were failing. So, feel free to assign this bug to yourself and submit patches :) > FsShell ls: Show symlinks properties > > > Key: HDFS-1788 > URL: https://issues.apache.org/jira/browse/HDFS-1788 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Jonathan Eagles >Assignee: John George >Priority: Minor > Attachments: HDFS-1788.patch > > > ls FsShell command implementation has been consistent with the linux > implementations of ls \-l. With the addition of symlinks, I would expect the > ability to show file type 'd' for directory, '\-' for file, and 'l' for > symlink. In addition, following the linkname entry for symlinks, I would > expect the ability to show "\-> ". In linux, the default is to > the the properties of the link and not of the link target. In linux, '-L' > option allows for the dereferencing of symlinks to show link target > properties, but it is not the default. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2081) trying to read configuration in hdfs write path might result in deadlock
[ https://issues.apache.org/jira/browse/HDFS-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051633#comment-13051633 ] John George commented on HDFS-2081: --- I overlooked the changes in the patch in HADOOP-7082. That patch is exactly what is required! Thanks Tod. Duplicating this JIRA to HADOOP-7082. > trying to read configuration in hdfs write path might result in deadlock > > > Key: HDFS-2081 > URL: https://issues.apache.org/jira/browse/HDFS-2081 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.23.0 > Environment: MR-279 branch. This is specific to yarn since yarn tries > to write the configuration file to hdfs while holding the lock of the > configuration file. >Reporter: John George >Assignee: John George > Fix For: 0.23.0 > > Attachments: HDFS-2081.patch > > > While trying to run yarn with 1024 block size, it looks as though the > dfsclients are hung waiting to write to DN. The reason is because yarn holds > the configuration file lock and tries to write to hdfs which in turn tries to > read from the configuration which requires the same lock. > The temporary fix is to make sure that dfsclient threads dont read the config > in its write path. There might be better long term solutions like read/write > locks and this needs to be investigated in another JIRA. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-2081) trying to read configuration in hdfs write path might result in deadlock
[ https://issues.apache.org/jira/browse/HDFS-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John George resolved HDFS-2081. --- Resolution: Duplicate Dupe of HADOOP-7082. > trying to read configuration in hdfs write path might result in deadlock > > > Key: HDFS-2081 > URL: https://issues.apache.org/jira/browse/HDFS-2081 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.23.0 > Environment: MR-279 branch. This is specific to yarn since yarn tries > to write the configuration file to hdfs while holding the lock of the > configuration file. >Reporter: John George >Assignee: John George > Fix For: 0.23.0 > > Attachments: HDFS-2081.patch > > > While trying to run yarn with 1024 block size, it looks as though the > dfsclients are hung waiting to write to DN. The reason is because yarn holds > the configuration file lock and tries to write to hdfs which in turn tries to > read from the configuration which requires the same lock. > The temporary fix is to make sure that dfsclient threads dont read the config > in its write path. There might be better long term solutions like read/write > locks and this needs to be investigated in another JIRA. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
