[jira] Commented: (HDFS-1496) TestStorageRestore is failing after HDFS-903 fix
[ https://issues.apache.org/jira/browse/HDFS-1496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989099#comment-12989099 ] Hairong Kuang commented on HDFS-1496: - Here is the inconsistency that is introduced by HADOOP-4885. After failed directories are added back, each failed directory contains new image + zero edit while each old good directory contains old image + old edit. If secondary namenode happens to fetch the image from a failed directory but fetch the edit from an old good directory, how would checkpointing work. > TestStorageRestore is failing after HDFS-903 fix > > > Key: HDFS-1496 > URL: https://issues.apache.org/jira/browse/HDFS-1496 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 0.22.0, 0.23.0 >Reporter: Konstantin Boudnik >Assignee: Hairong Kuang >Priority: Blocker > Fix For: 0.22.0 > > Attachments: HDFS-1496.sh, HDFS-1496.sh, HDFS-1496.sh > > > TestStorageRestore seems to be failing after HDFS-903 commit. Running git > bisect confirms it. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1496) TestStorageRestore is failing after HDFS-903 fix
[ https://issues.apache.org/jira/browse/HDFS-1496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989103#comment-12989103 ] Konstantin Shvachko commented on HDFS-1496: --- How do we fix it? > TestStorageRestore is failing after HDFS-903 fix > > > Key: HDFS-1496 > URL: https://issues.apache.org/jira/browse/HDFS-1496 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 0.22.0, 0.23.0 >Reporter: Konstantin Boudnik >Assignee: Hairong Kuang >Priority: Blocker > Fix For: 0.22.0 > > Attachments: HDFS-1496.sh, HDFS-1496.sh, HDFS-1496.sh > > > TestStorageRestore seems to be failing after HDFS-903 commit. Running git > bisect confirms it. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1496) TestStorageRestore is failing after HDFS-903 fix
[ https://issues.apache.org/jira/browse/HDFS-1496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989104#comment-12989104 ] Hairong Kuang commented on HDFS-1496: - As I said before, I could not figure out a good way to fix it. All I could do is to disable the feature introduced by HADOOP-4885. > TestStorageRestore is failing after HDFS-903 fix > > > Key: HDFS-1496 > URL: https://issues.apache.org/jira/browse/HDFS-1496 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 0.22.0, 0.23.0 >Reporter: Konstantin Boudnik >Assignee: Hairong Kuang >Priority: Blocker > Fix For: 0.22.0 > > Attachments: HDFS-1496.sh, HDFS-1496.sh, HDFS-1496.sh > > > TestStorageRestore seems to be failing after HDFS-903 commit. Running git > bisect confirms it. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1591) Fix javac, javadoc, findbugs warnings
[ https://issues.apache.org/jira/browse/HDFS-1591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-1591: -- Resolution: Fixed Status: Resolved (was: Patch Available) I just committed this. Thank you Po and Chris. > Fix javac, javadoc, findbugs warnings > - > > Key: HDFS-1591 > URL: https://issues.apache.org/jira/browse/HDFS-1591 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.22.0 >Reporter: Po Cheung >Assignee: Po Cheung > Fix For: 0.22.0 > > Attachments: hdfs-1591-trunk.patch > > > Split from HADOOP-6642 -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1591) Fix javac, javadoc, findbugs warnings
[ https://issues.apache.org/jira/browse/HDFS-1591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989138#comment-12989138 ] Konstantin Shvachko commented on HDFS-1591: --- Hudson fails because of the dependency on HADOOP-6642. I tested it locally. > Fix javac, javadoc, findbugs warnings > - > > Key: HDFS-1591 > URL: https://issues.apache.org/jira/browse/HDFS-1591 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.22.0 >Reporter: Po Cheung >Assignee: Po Cheung > Fix For: 0.22.0 > > Attachments: hdfs-1591-trunk.patch > > > Split from HADOOP-6642 -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1557) Separate Storage from FSImage
[ https://issues.apache.org/jira/browse/HDFS-1557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989260#comment-12989260 ] Todd Lipcon commented on HDFS-1557: --- JCarder looks pretty good - one potential deadlock but it's an old one I filed months back. So, +1 from me after addressing the nits Jakob and I mentioned above. > Separate Storage from FSImage > - > > Key: HDFS-1557 > URL: https://issues.apache.org/jira/browse/HDFS-1557 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: 0.21.0 >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: 0.23.0 > > Attachments: HDFS-1557-branch-0.22.diff, HDFS-1557-branch-0.22.diff, > HDFS-1557-trunk.diff, HDFS-1557-trunk.diff, HDFS-1557-trunk.diff, > HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, > HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff > > > FSImage currently derives from Storage and FSEditLog has to call methods > directly on FSImage to access the filesystem. This JIRA is to separate the > Storage class out into NNStorage so that FSEditLog is less dependent on > FSImage. From this point, the other parts of the circular dependency should > be easy to fix. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Kelly updated HDFS-1580: - Attachment: generic_wal_iface.pdf Addressed some of the comments. Have to discuss the rolling of logs as I'm unsure how 1073 changes it. > Add interface for generic Write Ahead Logging mechanisms > > > Key: HDFS-1580 > URL: https://issues.apache.org/jira/browse/HDFS-1580 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ivan Kelly > Attachments: generic_wal_iface.pdf, generic_wal_iface.txt > > -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1557) Separate Storage from FSImage
[ https://issues.apache.org/jira/browse/HDFS-1557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-1557: -- Attachment: 1557-suggestions.txt Hi Ivan. Here are a few more suggested changes (minor cosmetic fixes) and also a couple of notes in the code where some things looked strange - I figured it was easier to write these notes in patch form. > Separate Storage from FSImage > - > > Key: HDFS-1557 > URL: https://issues.apache.org/jira/browse/HDFS-1557 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: 0.21.0 >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: 0.23.0 > > Attachments: 1557-suggestions.txt, HDFS-1557-branch-0.22.diff, > HDFS-1557-branch-0.22.diff, HDFS-1557-trunk.diff, HDFS-1557-trunk.diff, > HDFS-1557-trunk.diff, HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, > HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff > > > FSImage currently derives from Storage and FSEditLog has to call methods > directly on FSImage to access the filesystem. This JIRA is to separate the > Storage class out into NNStorage so that FSEditLog is less dependent on > FSImage. From this point, the other parts of the circular dependency should > be easy to fix. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1557) Separate Storage from FSImage
[ https://issues.apache.org/jira/browse/HDFS-1557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989303#comment-12989303 ] Hadoop QA commented on HDFS-1557: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12469947/1557-suggestions.txt against trunk revision 1065960. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/142//console This message is automatically generated. > Separate Storage from FSImage > - > > Key: HDFS-1557 > URL: https://issues.apache.org/jira/browse/HDFS-1557 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: 0.21.0 >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: 0.23.0 > > Attachments: 1557-suggestions.txt, HDFS-1557-branch-0.22.diff, > HDFS-1557-branch-0.22.diff, HDFS-1557-trunk.diff, HDFS-1557-trunk.diff, > HDFS-1557-trunk.diff, HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, > HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff > > > FSImage currently derives from Storage and FSEditLog has to call methods > directly on FSImage to access the filesystem. This JIRA is to separate the > Storage class out into NNStorage so that FSEditLog is less dependent on > FSImage. From this point, the other parts of the circular dependency should > be easy to fix. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1557) Separate Storage from FSImage
[ https://issues.apache.org/jira/browse/HDFS-1557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Kelly updated HDFS-1557: - Status: Patch Available (was: Open) > Separate Storage from FSImage > - > > Key: HDFS-1557 > URL: https://issues.apache.org/jira/browse/HDFS-1557 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: 0.21.0 >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: 0.23.0 > > Attachments: 1557-suggestions.txt, HDFS-1557-branch-0.22.diff, > HDFS-1557-branch-0.22.diff, HDFS-1557-trunk.diff, HDFS-1557-trunk.diff, > HDFS-1557-trunk.diff, HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, > HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, > HDFS-1557.diff, HDFS-1557.diff > > > FSImage currently derives from Storage and FSEditLog has to call methods > directly on FSImage to access the filesystem. This JIRA is to separate the > Storage class out into NNStorage so that FSEditLog is less dependent on > FSImage. From this point, the other parts of the circular dependency should > be easy to fix. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1557) Separate Storage from FSImage
[ https://issues.apache.org/jira/browse/HDFS-1557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Kelly updated HDFS-1557: - Status: Open (was: Patch Available) > Separate Storage from FSImage > - > > Key: HDFS-1557 > URL: https://issues.apache.org/jira/browse/HDFS-1557 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: 0.21.0 >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: 0.23.0 > > Attachments: 1557-suggestions.txt, HDFS-1557-branch-0.22.diff, > HDFS-1557-branch-0.22.diff, HDFS-1557-trunk.diff, HDFS-1557-trunk.diff, > HDFS-1557-trunk.diff, HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, > HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, > HDFS-1557.diff, HDFS-1557.diff > > > FSImage currently derives from Storage and FSEditLog has to call methods > directly on FSImage to access the filesystem. This JIRA is to separate the > Storage class out into NNStorage so that FSEditLog is less dependent on > FSImage. From this point, the other parts of the circular dependency should > be easy to fix. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1557) Separate Storage from FSImage
[ https://issues.apache.org/jira/browse/HDFS-1557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Kelly updated HDFS-1557: - Attachment: HDFS-1557.diff > Separate Storage from FSImage > - > > Key: HDFS-1557 > URL: https://issues.apache.org/jira/browse/HDFS-1557 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: 0.21.0 >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: 0.23.0 > > Attachments: 1557-suggestions.txt, HDFS-1557-branch-0.22.diff, > HDFS-1557-branch-0.22.diff, HDFS-1557-trunk.diff, HDFS-1557-trunk.diff, > HDFS-1557-trunk.diff, HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, > HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, > HDFS-1557.diff, HDFS-1557.diff > > > FSImage currently derives from Storage and FSEditLog has to call methods > directly on FSImage to access the filesystem. This JIRA is to separate the > Storage class out into NNStorage so that FSEditLog is less dependent on > FSImage. From this point, the other parts of the circular dependency should > be easy to fix. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1557) Separate Storage from FSImage
[ https://issues.apache.org/jira/browse/HDFS-1557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Kelly updated HDFS-1557: - Attachment: HDFS-1557.diff > Separate Storage from FSImage > - > > Key: HDFS-1557 > URL: https://issues.apache.org/jira/browse/HDFS-1557 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: 0.21.0 >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: 0.23.0 > > Attachments: 1557-suggestions.txt, HDFS-1557-branch-0.22.diff, > HDFS-1557-branch-0.22.diff, HDFS-1557-trunk.diff, HDFS-1557-trunk.diff, > HDFS-1557-trunk.diff, HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, > HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, > HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff > > > FSImage currently derives from Storage and FSEditLog has to call methods > directly on FSImage to access the filesystem. This JIRA is to separate the > Storage class out into NNStorage so that FSEditLog is less dependent on > FSImage. From this point, the other parts of the circular dependency should > be easy to fix. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1557) Separate Storage from FSImage
[ https://issues.apache.org/jira/browse/HDFS-1557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Kelly updated HDFS-1557: - Status: Open (was: Patch Available) > Separate Storage from FSImage > - > > Key: HDFS-1557 > URL: https://issues.apache.org/jira/browse/HDFS-1557 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: 0.21.0 >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: 0.23.0 > > Attachments: 1557-suggestions.txt, HDFS-1557-branch-0.22.diff, > HDFS-1557-branch-0.22.diff, HDFS-1557-trunk.diff, HDFS-1557-trunk.diff, > HDFS-1557-trunk.diff, HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, > HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, > HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff > > > FSImage currently derives from Storage and FSEditLog has to call methods > directly on FSImage to access the filesystem. This JIRA is to separate the > Storage class out into NNStorage so that FSEditLog is less dependent on > FSImage. From this point, the other parts of the circular dependency should > be easy to fix. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1557) Separate Storage from FSImage
[ https://issues.apache.org/jira/browse/HDFS-1557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Kelly updated HDFS-1557: - Status: Patch Available (was: Open) > Separate Storage from FSImage > - > > Key: HDFS-1557 > URL: https://issues.apache.org/jira/browse/HDFS-1557 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: 0.21.0 >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: 0.23.0 > > Attachments: 1557-suggestions.txt, HDFS-1557-branch-0.22.diff, > HDFS-1557-branch-0.22.diff, HDFS-1557-trunk.diff, HDFS-1557-trunk.diff, > HDFS-1557-trunk.diff, HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, > HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, > HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff > > > FSImage currently derives from Storage and FSEditLog has to call methods > directly on FSImage to access the filesystem. This JIRA is to separate the > Storage class out into NNStorage so that FSEditLog is less dependent on > FSImage. From this point, the other parts of the circular dependency should > be easy to fix. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1605) Convert DFSInputStream synchronized sections to a ReadWrite lock
[ https://issues.apache.org/jira/browse/HDFS-1605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HDFS-1605: --- Attachment: DFSClientRWlock.3.txt Merged patch with latest branch. > Convert DFSInputStream synchronized sections to a ReadWrite lock > > > Key: HDFS-1605 > URL: https://issues.apache.org/jira/browse/HDFS-1605 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs client >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Attachments: DFSClientRWlock.1.txt, DFSClientRWlock.3.txt > > > Hbase does concurrent preads from multiple threads to different blocks of the > same hdfs file. Each of these pread calls invoke > DFSInputStream.getFileLength() and DFSInputStream.getBlockAt(). These methods > are "synchronized", thus causing all the concurrent threads to serialize. It > would help performance to convert this to a Read/Write lock -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989338#comment-12989338 ] Benjamin Reed commented on HDFS-1580: - looks good ivan. i like section 1. i see a couple of small problems in section 2: 1) you have a constructor in the interface. you should either make that a comment (meaning that is how you will construct those objects using reflection), or you should call it init(...) 2) why is transfer() in WriteAheadLogStreamFactory? shouldn't the transfer happen outside of the logging? other logic should be reading from the local WAL and transferring to the secondary. 3) can you comment a bit more on upgrade(), rollback(), and finalize()? > Add interface for generic Write Ahead Logging mechanisms > > > Key: HDFS-1580 > URL: https://issues.apache.org/jira/browse/HDFS-1580 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ivan Kelly > Attachments: generic_wal_iface.pdf, generic_wal_iface.txt > > -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (HDFS-1376) TestFileCreation times out
[
https://issues.apache.org/jira/browse/HDFS-1376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko resolved HDFS-1376.
---
Resolution: Cannot Reproduce
This does not seem to be a problem any more. Please reopen if you see it.
> TestFileCreation times out
> --
>
> Key: HDFS-1376
> URL: https://issues.apache.org/jira/browse/HDFS-1376
> Project: Hadoop HDFS
> Issue Type: Test
>Affects Versions: 0.22.0
>Reporter: Eli Collins
>Assignee: Po Cheung
>Priority: Blocker
> Fix For: 0.22.0
>
>
> I see TestFileCreation periodically time out on a host when ant test runs but
> w/o anything thing else running.
> {noformat}
> Testsuite: org.apache.hadoop.hdfs.TestFileCreation
> Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
> Testcase: testFileCreationNonRecursive took 0.002 sec
> Caused an ERROR
> Timeout occurred. Please note the time in the report does not reflect the
> time until the timeout.
> junit.framework.AssertionFailedError: Timeout occurred. Please note the time
> in the report does not reflect the time until the timeout.
> {noformat}
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1557) Separate Storage from FSImage
[ https://issues.apache.org/jira/browse/HDFS-1557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDFS-1557: --- Status: Patch Available (was: Open) > Separate Storage from FSImage > - > > Key: HDFS-1557 > URL: https://issues.apache.org/jira/browse/HDFS-1557 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: 0.21.0 >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: 0.23.0 > > Attachments: 1557-suggestions.txt, HDFS-1557-branch-0.22.diff, > HDFS-1557-branch-0.22.diff, HDFS-1557-trunk.diff, HDFS-1557-trunk.diff, > HDFS-1557-trunk.diff, HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, > HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, > HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff > > > FSImage currently derives from Storage and FSEditLog has to call methods > directly on FSImage to access the filesystem. This JIRA is to separate the > Storage class out into NNStorage so that FSEditLog is less dependent on > FSImage. From this point, the other parts of the circular dependency should > be easy to fix. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1557) Separate Storage from FSImage
[ https://issues.apache.org/jira/browse/HDFS-1557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDFS-1557: --- Status: Open (was: Patch Available) > Separate Storage from FSImage > - > > Key: HDFS-1557 > URL: https://issues.apache.org/jira/browse/HDFS-1557 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: 0.21.0 >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: 0.23.0 > > Attachments: 1557-suggestions.txt, HDFS-1557-branch-0.22.diff, > HDFS-1557-branch-0.22.diff, HDFS-1557-trunk.diff, HDFS-1557-trunk.diff, > HDFS-1557-trunk.diff, HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, > HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, > HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff > > > FSImage currently derives from Storage and FSEditLog has to call methods > directly on FSImage to access the filesystem. This JIRA is to separate the > Storage class out into NNStorage so that FSEditLog is less dependent on > FSImage. From this point, the other parts of the circular dependency should > be easy to fix. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1557) Separate Storage from FSImage
[ https://issues.apache.org/jira/browse/HDFS-1557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989402#comment-12989402 ] Todd Lipcon commented on HDFS-1557: --- +1 from me, looks great. > Separate Storage from FSImage > - > > Key: HDFS-1557 > URL: https://issues.apache.org/jira/browse/HDFS-1557 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: 0.21.0 >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: 0.23.0 > > Attachments: 1557-suggestions.txt, HDFS-1557-branch-0.22.diff, > HDFS-1557-branch-0.22.diff, HDFS-1557-trunk.diff, HDFS-1557-trunk.diff, > HDFS-1557-trunk.diff, HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, > HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, > HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff > > > FSImage currently derives from Storage and FSEditLog has to call methods > directly on FSImage to access the filesystem. This JIRA is to separate the > Storage class out into NNStorage so that FSEditLog is less dependent on > FSImage. From this point, the other parts of the circular dependency should > be easy to fix. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Created: (HDFS-1607) Fix references to misspelled method name getProtocolSigature
Fix references to misspelled method name getProtocolSigature Key: HDFS-1607 URL: https://issues.apache.org/jira/browse/HDFS-1607 Project: Hadoop HDFS Issue Type: Bug Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Trivial -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1607) Fix references to misspelled method name getProtocolSigature
[ https://issues.apache.org/jira/browse/HDFS-1607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-1607: -- Attachment: hdfs-1607.txt Patch to fix reference to typo name. Need to wait for next Common publish before this will compile. > Fix references to misspelled method name getProtocolSigature > > > Key: HDFS-1607 > URL: https://issues.apache.org/jira/browse/HDFS-1607 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Trivial > Attachments: hdfs-1607.txt > > -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1605) Convert DFSInputStream synchronized sections to a ReadWrite lock
[ https://issues.apache.org/jira/browse/HDFS-1605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989438#comment-12989438 ] Todd Lipcon commented on HDFS-1605: --- Hey Dhruba. I've never seen this lock actually be a problem in practice given current performance issues in DFSInputStream. I understand from Jonathan Gray that you're also testing some other patches to improve DFSInputStream performance. Maybe it would make sense to put those upstream first? (or do you have some benchmarks that shows that the rwlock helps things even with an otherwise "stock" DFSInputStream)? > Convert DFSInputStream synchronized sections to a ReadWrite lock > > > Key: HDFS-1605 > URL: https://issues.apache.org/jira/browse/HDFS-1605 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs client >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Attachments: DFSClientRWlock.1.txt, DFSClientRWlock.3.txt > > > Hbase does concurrent preads from multiple threads to different blocks of the > same hdfs file. Each of these pread calls invoke > DFSInputStream.getFileLength() and DFSInputStream.getBlockAt(). These methods > are "synchronized", thus causing all the concurrent threads to serialize. It > would help performance to convert this to a Read/Write lock -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1529) Incorrect handling of interrupts in waitForAckedSeqno can cause deadlock
[ https://issues.apache.org/jira/browse/HDFS-1529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989470#comment-12989470 ] Hairong Kuang commented on HDFS-1529: - +1. Let's commit this fix for now. > Incorrect handling of interrupts in waitForAckedSeqno can cause deadlock > > > Key: HDFS-1529 > URL: https://issues.apache.org/jira/browse/HDFS-1529 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs client >Affects Versions: 0.22.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Blocker > Fix For: 0.22.0 > > Attachments: Test.java, hdfs-1529.txt, hdfs-1529.txt, hdfs-1529.txt > > > In HDFS-895 the handling of interrupts during hflush/close was changed to > preserve interrupt status. This ends up creating an infinite loop in > waitForAckedSeqno if the waiting thread gets interrupted, since Object.wait() > has a strange semantic that it doesn't give up the lock even momentarily if > the thread is already in interrupted state at the beginning of the call. > We should decide what the correct behavior is here - if a thread is > interrupted while it's calling hflush() or close() should we (a) throw an > exception, perhaps InterruptedIOException (b) ignore, or (c) wait for the > flush to finish but preserve interrupt status on exit? -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1557) Separate Storage from FSImage
[ https://issues.apache.org/jira/browse/HDFS-1557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989492#comment-12989492 ] Hadoop QA commented on HDFS-1557: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12469966/HDFS-1557.diff against trunk revision 1065960. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 42 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. -1 release audit. The applied patch generated 2 release audit warnings (more than the trunk's current 0 warnings). -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.security.TestClientProtocolWithDelegationToken org.apache.hadoop.hdfs.security.token.block.TestBlockToken org.apache.hadoop.hdfs.server.namenode.TestStorageRestore -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/145//testReport/ Release audit warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/145//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/145//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/145//console This message is automatically generated. > Separate Storage from FSImage > - > > Key: HDFS-1557 > URL: https://issues.apache.org/jira/browse/HDFS-1557 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: 0.21.0 >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: 0.23.0 > > Attachments: 1557-suggestions.txt, HDFS-1557-branch-0.22.diff, > HDFS-1557-branch-0.22.diff, HDFS-1557-trunk.diff, HDFS-1557-trunk.diff, > HDFS-1557-trunk.diff, HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, > HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, > HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff > > > FSImage currently derives from Storage and FSEditLog has to call methods > directly on FSImage to access the filesystem. This JIRA is to separate the > Storage class out into NNStorage so that FSEditLog is less dependent on > FSImage. From this point, the other parts of the circular dependency should > be easy to fix. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1557) Separate Storage from FSImage
[ https://issues.apache.org/jira/browse/HDFS-1557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989495#comment-12989495 ] Jitendra Nath Pandey commented on HDFS-1557: The three failed tests and test-contrib also fail without the patch. Release audit warnings are due to OfflineImageViewer and are unrelated to this patch. > Separate Storage from FSImage > - > > Key: HDFS-1557 > URL: https://issues.apache.org/jira/browse/HDFS-1557 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: 0.21.0 >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: 0.23.0 > > Attachments: 1557-suggestions.txt, HDFS-1557-branch-0.22.diff, > HDFS-1557-branch-0.22.diff, HDFS-1557-trunk.diff, HDFS-1557-trunk.diff, > HDFS-1557-trunk.diff, HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, > HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, > HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff > > > FSImage currently derives from Storage and FSEditLog has to call methods > directly on FSImage to access the filesystem. This JIRA is to separate the > Storage class out into NNStorage so that FSEditLog is less dependent on > FSImage. From this point, the other parts of the circular dependency should > be easy to fix. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1557) Separate Storage from FSImage
[ https://issues.apache.org/jira/browse/HDFS-1557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989496#comment-12989496 ] Jitendra Nath Pandey commented on HDFS-1557: Correction in previous comment: Release audit warnings are in OfflineEditsViewer. > Separate Storage from FSImage > - > > Key: HDFS-1557 > URL: https://issues.apache.org/jira/browse/HDFS-1557 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: 0.21.0 >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: 0.23.0 > > Attachments: 1557-suggestions.txt, HDFS-1557-branch-0.22.diff, > HDFS-1557-branch-0.22.diff, HDFS-1557-trunk.diff, HDFS-1557-trunk.diff, > HDFS-1557-trunk.diff, HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, > HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, > HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff > > > FSImage currently derives from Storage and FSEditLog has to call methods > directly on FSImage to access the filesystem. This JIRA is to separate the > Storage class out into NNStorage so that FSEditLog is less dependent on > FSImage. From this point, the other parts of the circular dependency should > be easy to fix. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1557) Separate Storage from FSImage
[ https://issues.apache.org/jira/browse/HDFS-1557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDFS-1557: --- Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) > Separate Storage from FSImage > - > > Key: HDFS-1557 > URL: https://issues.apache.org/jira/browse/HDFS-1557 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: 0.21.0 >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: 0.23.0 > > Attachments: 1557-suggestions.txt, HDFS-1557-branch-0.22.diff, > HDFS-1557-branch-0.22.diff, HDFS-1557-trunk.diff, HDFS-1557-trunk.diff, > HDFS-1557-trunk.diff, HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, > HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, > HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff > > > FSImage currently derives from Storage and FSEditLog has to call methods > directly on FSImage to access the filesystem. This JIRA is to separate the > Storage class out into NNStorage so that FSEditLog is less dependent on > FSImage. From this point, the other parts of the circular dependency should > be easy to fix. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1557) Separate Storage from FSImage
[ https://issues.apache.org/jira/browse/HDFS-1557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989497#comment-12989497 ] Jitendra Nath Pandey commented on HDFS-1557: I just committed this. Thanks to Ivan! > Separate Storage from FSImage > - > > Key: HDFS-1557 > URL: https://issues.apache.org/jira/browse/HDFS-1557 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: 0.21.0 >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: 0.23.0 > > Attachments: 1557-suggestions.txt, HDFS-1557-branch-0.22.diff, > HDFS-1557-branch-0.22.diff, HDFS-1557-trunk.diff, HDFS-1557-trunk.diff, > HDFS-1557-trunk.diff, HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, > HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, > HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff > > > FSImage currently derives from Storage and FSEditLog has to call methods > directly on FSImage to access the filesystem. This JIRA is to separate the > Storage class out into NNStorage so that FSEditLog is less dependent on > FSImage. From this point, the other parts of the circular dependency should > be easy to fix. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
