[jira] Commented: (HDFS-270) DFS Upgrade should process dfs.data.dirs in parallel
[ https://issues.apache.org/jira/browse/HDFS-270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989815#comment-12989815 ] Hairong Kuang commented on HDFS-270: Matt, are you still working on this? If not, do you mind if I take the task? > DFS Upgrade should process dfs.data.dirs in parallel > > > Key: HDFS-270 > URL: https://issues.apache.org/jira/browse/HDFS-270 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: data-node >Affects Versions: 0.20.2 >Reporter: Stu Hood >Assignee: Matt Foley > > I just upgraded from 0.14.2 to 0.15.0, and things went very smoothly, if a > little slowly. > The main reason the upgrade took so long was the block upgrades on the > datanodes. Each of our datanodes has 3 drives listed for the dfs.data.dir > parameter. From looking at the logs, it is fairly clear that the upgrade > procedure does not attempt to upgrade all listed dfs.data.dir's in parallel. > I think even if all of your dfs.data.dir's are on the same physical device, > there would still be an advantage to performing the upgrade process in > parallel. The less downtime, the better: especially if it is potentially 20 > minutes versus 60 minutes. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1605) Convert DFSInputStream synchronized sections to a ReadWrite lock
[ https://issues.apache.org/jira/browse/HDFS-1605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989739#comment-12989739 ] dhruba borthakur commented on HDFS-1605: Hi todd, it will take me quite a while to get performance numbers for you. But it is fine if you do not want this in the 0.20-append branch, no problems with me. > Convert DFSInputStream synchronized sections to a ReadWrite lock > > > Key: HDFS-1605 > URL: https://issues.apache.org/jira/browse/HDFS-1605 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs client >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Attachments: DFSClientRWlock.1.txt, DFSClientRWlock.3.txt > > > Hbase does concurrent preads from multiple threads to different blocks of the > same hdfs file. Each of these pread calls invoke > DFSInputStream.getFileLength() and DFSInputStream.getBlockAt(). These methods > are "synchronized", thus causing all the concurrent threads to serialize. It > would help performance to convert this to a Read/Write lock -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Kelly updated HDFS-1580: - Attachment: generic_wal_iface.pdf Another update to the interfaces. I've changed it quite a bit. There's still a few open issues, but different ones to yesterday. > Add interface for generic Write Ahead Logging mechanisms > > > Key: HDFS-1580 > URL: https://issues.apache.org/jira/browse/HDFS-1580 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ivan Kelly > Attachments: generic_wal_iface.pdf, generic_wal_iface.pdf, > generic_wal_iface.txt > > -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1557) Separate Storage from FSImage
[ https://issues.apache.org/jira/browse/HDFS-1557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989569#comment-12989569 ] Ivan Kelly commented on HDFS-1557: -- That's great :) Thanks for your help on this and congrats on your first commit! > Separate Storage from FSImage > - > > Key: HDFS-1557 > URL: https://issues.apache.org/jira/browse/HDFS-1557 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: 0.21.0 >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: 0.23.0 > > Attachments: 1557-suggestions.txt, HDFS-1557-branch-0.22.diff, > HDFS-1557-branch-0.22.diff, HDFS-1557-trunk.diff, HDFS-1557-trunk.diff, > HDFS-1557-trunk.diff, HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, > HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff, > HDFS-1557.diff, HDFS-1557.diff, HDFS-1557.diff > > > FSImage currently derives from Storage and FSEditLog has to call methods > directly on FSImage to access the filesystem. This JIRA is to separate the > Storage class out into NNStorage so that FSEditLog is less dependent on > FSImage. From this point, the other parts of the circular dependency should > be easy to fix. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1605) Convert DFSInputStream synchronized sections to a ReadWrite lock
[ https://issues.apache.org/jira/browse/HDFS-1605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989563#comment-12989563 ] Hairong Kuang commented on HDFS-1605: - +1. The latest patch looks good to me. > Convert DFSInputStream synchronized sections to a ReadWrite lock > > > Key: HDFS-1605 > URL: https://issues.apache.org/jira/browse/HDFS-1605 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs client >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Attachments: DFSClientRWlock.1.txt, DFSClientRWlock.3.txt > > > Hbase does concurrent preads from multiple threads to different blocks of the > same hdfs file. Each of these pread calls invoke > DFSInputStream.getFileLength() and DFSInputStream.getBlockAt(). These methods > are "synchronized", thus causing all the concurrent threads to serialize. It > would help performance to convert this to a Read/Write lock -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1607) Fix references to misspelled method name getProtocolSigature
[ https://issues.apache.org/jira/browse/HDFS-1607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989555#comment-12989555 ] Eli Collins commented on HDFS-1607: --- +1 > Fix references to misspelled method name getProtocolSigature > > > Key: HDFS-1607 > URL: https://issues.apache.org/jira/browse/HDFS-1607 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Trivial > Attachments: hdfs-1607.txt > > -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
