[jira] [Updated] (HDFS-664) Add a way to efficiently replace a disk in a live datanode
[ https://issues.apache.org/jira/browse/HDFS-664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated HDFS-664: - Resolution: Duplicate Status: Resolved (was: Patch Available) Dupe of HDFS-1362, per comments above. Add a way to efficiently replace a disk in a live datanode -- Key: HDFS-664 URL: https://issues.apache.org/jira/browse/HDFS-664 Project: Hadoop HDFS Issue Type: New Feature Components: data-node Affects Versions: 0.22.0 Reporter: Steve Loughran Attachments: HDFS-664.0-20-3-rc2.patch.1, HDFS-664.patch In clusters where the datanode disks are hot swappable, you need to be able to swap out a disk on a live datanode without taking down the datanode. You don't want to decommission the whole node as that is overkill. on a system with 4 1TB HDDs, giving 3 TB of datanode storage, a decommissioning and restart will consume up to 6 TB of bandwidth. If a single disk were swapped in then there would only be 1TB of data to recover over the network. More importantly, if that data could be moved to free space on the same machine, the recommissioning could take place at disk rates, not network speeds. # Maybe have a way of decommissioning a single disk on the DN; the files could be moved to space on the other disks or the other machines in the rack. # There may not be time to use that option, in which case pulling out the disk would be done with no warning, a new disk inserted. # The DN needs to see that a disk has been replaced (or react to some ops request telling it this), and start using the new disk again -pushing back data, rebuilding the balance. To complicate the process, assume there is a live TT on the system, running jobs against the data. The TT would probably need to be paused while the work takes place, any ongoing work handled somehow. Halting the TT and then restarting it after the replacement disk went in is probably simplest. The more disks you add to a node, the more this scenario becomes a need. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-664) Add a way to efficiently replace a disk in a live datanode
[ https://issues.apache.org/jira/browse/HDFS-664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tony Valderrama updated HDFS-664: - Attachment: HDFS-664.0-20-3-rc2.patch.1 Steve, thanks for the review! One general note, before answering your questions - this patch was developed simply by extracting the relevant sections from the Datanode startup procedure. I chose 0.20.3-rc2 as the base because it's closest to what we use at Tuenti. If there is actual interest in the patch and no technical barriers (like the threading issues), I'll gladly forward-port it to trunk. 1. Not to my knowledge - a review from someone experienced would be much appreciated. 2. As mentioned above, much of this patch is identical to code elsewhere in 0.20.3-rc2, so I'll refrain from updating this particular section. For a trunk forward-port, these sections would be replaced with the more recent versions. 3. Thanks for the suggestion, it's included in the new patch. 4. Sorry, it's now included. 5. Unfortunately, we use Mercurial internally, and the export process is terrible. I've rebased this patch from our internal branch onto 0.20.3-rc2, so it applies cleanly there. 6. I did some basic testing to simulate swapping a volume by moving the data directory out of the way, touching a file in place of the directory (which causes the Datanode to register volume failure when it tries to write or read), and then using the shell to add the out-of-place volume. You can also add a second data directory on a disk that's already being used to test swapping in a clean disk. 7. Hmm... that's troubling. If you find the JIRA, please link it here. For now, we've tested this under low load, and when we have time to revisit this, we'll try it out under heavy load. Add a way to efficiently replace a disk in a live datanode -- Key: HDFS-664 URL: https://issues.apache.org/jira/browse/HDFS-664 Project: Hadoop HDFS Issue Type: New Feature Components: data-node Affects Versions: 0.22.0 Reporter: Steve Loughran Attachments: HDFS-664.0-20-3-rc2.patch.1, HDFS-664.patch In clusters where the datanode disks are hot swappable, you need to be able to swap out a disk on a live datanode without taking down the datanode. You don't want to decommission the whole node as that is overkill. on a system with 4 1TB HDDs, giving 3 TB of datanode storage, a decommissioning and restart will consume up to 6 TB of bandwidth. If a single disk were swapped in then there would only be 1TB of data to recover over the network. More importantly, if that data could be moved to free space on the same machine, the recommissioning could take place at disk rates, not network speeds. # Maybe have a way of decommissioning a single disk on the DN; the files could be moved to space on the other disks or the other machines in the rack. # There may not be time to use that option, in which case pulling out the disk would be done with no warning, a new disk inserted. # The DN needs to see that a disk has been replaced (or react to some ops request telling it this), and start using the new disk again -pushing back data, rebuilding the balance. To complicate the process, assume there is a live TT on the system, running jobs against the data. The TT would probably need to be paused while the work takes place, any ongoing work handled somehow. Halting the TT and then restarting it after the replacement disk went in is probably simplest. The more disks you add to a node, the more this scenario becomes a need. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-664) Add a way to efficiently replace a disk in a live datanode
[ https://issues.apache.org/jira/browse/HDFS-664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HDFS-664: Status: Patch Available (was: Open) Add a way to efficiently replace a disk in a live datanode -- Key: HDFS-664 URL: https://issues.apache.org/jira/browse/HDFS-664 Project: Hadoop HDFS Issue Type: New Feature Components: data-node Affects Versions: 0.22.0 Reporter: Steve Loughran Attachments: HDFS-664.patch In clusters where the datanode disks are hot swappable, you need to be able to swap out a disk on a live datanode without taking down the datanode. You don't want to decommission the whole node as that is overkill. on a system with 4 1TB HDDs, giving 3 TB of datanode storage, a decommissioning and restart will consume up to 6 TB of bandwidth. If a single disk were swapped in then there would only be 1TB of data to recover over the network. More importantly, if that data could be moved to free space on the same machine, the recommissioning could take place at disk rates, not network speeds. # Maybe have a way of decommissioning a single disk on the DN; the files could be moved to space on the other disks or the other machines in the rack. # There may not be time to use that option, in which case pulling out the disk would be done with no warning, a new disk inserted. # The DN needs to see that a disk has been replaced (or react to some ops request telling it this), and start using the new disk again -pushing back data, rebuilding the balance. To complicate the process, assume there is a live TT on the system, running jobs against the data. The TT would probably need to be paused while the work takes place, any ongoing work handled somehow. Halting the TT and then restarting it after the replacement disk went in is probably simplest. The more disks you add to a node, the more this scenario becomes a need. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-664) Add a way to efficiently replace a disk in a live datanode
[ https://issues.apache.org/jira/browse/HDFS-664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tony Valderrama updated HDFS-664: - Attachment: HDFS-664.patch This is a proof-of-concept patch against 0.20.3-rc2. You can add a volume to a live datanode by running the command hadoop dnshell -registerVolume /path/to/volume on the machine hosting the datanode. Add a way to efficiently replace a disk in a live datanode -- Key: HDFS-664 URL: https://issues.apache.org/jira/browse/HDFS-664 Project: Hadoop HDFS Issue Type: New Feature Components: data-node Affects Versions: 0.22.0 Reporter: Steve Loughran Attachments: HDFS-664.patch In clusters where the datanode disks are hot swappable, you need to be able to swap out a disk on a live datanode without taking down the datanode. You don't want to decommission the whole node as that is overkill. on a system with 4 1TB HDDs, giving 3 TB of datanode storage, a decommissioning and restart will consume up to 6 TB of bandwidth. If a single disk were swapped in then there would only be 1TB of data to recover over the network. More importantly, if that data could be moved to free space on the same machine, the recommissioning could take place at disk rates, not network speeds. # Maybe have a way of decommissioning a single disk on the DN; the files could be moved to space on the other disks or the other machines in the rack. # There may not be time to use that option, in which case pulling out the disk would be done with no warning, a new disk inserted. # The DN needs to see that a disk has been replaced (or react to some ops request telling it this), and start using the new disk again -pushing back data, rebuilding the balance. To complicate the process, assume there is a live TT on the system, running jobs against the data. The TT would probably need to be paused while the work takes place, any ongoing work handled somehow. Halting the TT and then restarting it after the replacement disk went in is probably simplest. The more disks you add to a node, the more this scenario becomes a need. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
