[ https://issues.apache.org/jira/browse/HDFS-1312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088688#comment-15088688 ]
Anu Engineer commented on HDFS-1312: ------------------------------------ Hi [~andrew.wang] Thanks for the quick response. I was thinking about what you said and I think the the disconnect we are having is because you are assuming that HDFS-1804 is always available on the HDFS clusters, but for many customers that is not true. bq. Have these customers tried out HDFS-1804? We've had users complain about imbalance before too, and after enabling HDFS-1804, no further issues. Generally administrators are wary of enabling a feature like HDFS-1804 in a production cluster. For new clusters it is more easier but for existing production clusters assuming the existence of HDFS-1804 is not realistic. bq. If you're interested, feel free to take HDFS-8538 from me. I really think it'll fix the majority of imbalance issues outside of hotswap. Thank you for the offer. I can certainly work on HDFS-8538 after HDFS-1312. I think of HDFS-1804, HDFS-8538 and HDFS-1312 as part of the solution to the same problem. Just attacking it from different angles, without all three some part of HDFS users will always be left out. bq. When I mentioned removing the discover phase, I meant the NN communication. Here, the DN just probes its own volume information. Does it need to talk to the NN for anything else? # I am not able to see any particular advantage is doing planning inside the datanode. However with that approach we do lose one the critical feature of the tool, that is ability to report what we did to the machine, capturing the "before state" is much more complex. I agree that users can manually capture this info before and after, but that is an extra administrative burden. With the current approach, we record the information of a datanode before and we make it easy to compare the state once we are done. # Also We wanted to merge mover into this engine later, with the current approach what we build inside datanode is a simple block mover, or to be precise an RPC interface to the existing mover's block interface. You can feed it any move commands you like, which provides better composability. With what you are suggesting we will lose that flexibility. # Less complexity inside datanode, planner code never needs to run inside datanode, it is a piece of code that plans a set of moves, why would we want to run it inside datanode ?. # Since none of our tools currently report the disk level data distribution, without talking to Namenode it is not possible to find which nodes are imbalanced. I know that you are arguing that it will never happen, if all customers use HDFS-1804. Two issues with that, one there are lots of customers without HDFS-1804, and HDFS-1804 is just an option that user can choose. Since it is configurable, it is arguable that we will always have customers without HDFS-1804. The current architecture addresses the needs of both group of users. In that sense current architecture is a better or more encompassing architecture. bq. Cluster-wide disk information is already handled by monitoring tools, no? The admin gets the Ganglia alert saying some node is imbalanced, admin triggers intranode balancer, admin keeps looking at Ganglia to see if it's fixed. I don't think adding our own monitoring of the same information helps, when Ganglia etc. are already available, in-use, and understood by admins. Please correct me if I am missing something here, Getting an alert due to low space on disk from datanode is very reactive. With the current disk balancer design we are assuming that disk balancer tool can be run to address and fix any issue you have in the cluster. You could argue that admins can write some scripts to monitor this issue using ganglia command line, but it is common enough problem that I think it should be solved at HDFS level. Here are two use cases that disk balancer addresses. First one is discovering nodes with potential issues and the second one is auto fixing those issues. This is very similar to current balancer. 1. Scenario 1: Admin can run {noformat} hdfs diskbalancer -top 100 {noformat} , and viola! we print out the top 100 hundred nodes that is having a problem. Let us say that admin now wants to look closely at the node and find out distribution on individual disks, he can now do that via. disk balancer, ssh or ganglia. 2. Scenario 2 : Admin wants not to be bothered with this balancing act at all, in reality he is thinking why doesn't HDFS just take care of this(I know HDFS-1804 is addressing that, but again we are talking about cluster which does not have it enabled.) and in that case we will let the admin run {noformat} hdfs diskbalancer -top 10 -balance {noformat}, this allows the admin to run disk balancer just like current balancer, without having to worry about what is happening or measuring each node. With gangalia a bunch of nodes will fire alerts, admin needs to copy the address of each datanode and give it to disk balancer. I think the current flow of disk balancer makes it easier to use. bq. I don't think this conflicts with the debuggability goal. The DN can dump the Info object (and even the Plan) object) if requested, to the log or somewhere in a data dir. Well, it is debuggable, but assuming that I am the one who will be called on to debug this, I prefer to debug by looking at my local directory instead of ssh-ing into a datanode. I think of writing to local directory as a gift I am making to my future self :). Plus as mentioned earlier the other use case where we want to report to user what our tool did, fetching this data out of datanode's log directory is hard(may be another RPC to fetch it ??). bq. Adding a note that says "we use the existing moveBlockAcrossStorage method" is a great answer. I will update the design doc with this info. Thanks for your suggestions. > Re-balance disks within a Datanode > ---------------------------------- > > Key: HDFS-1312 > URL: https://issues.apache.org/jira/browse/HDFS-1312 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode > Reporter: Travis Crawford > Assignee: Anu Engineer > Attachments: Architecture_and_testplan.pdf, disk-balancer-proposal.pdf > > > Filing this issue in response to ``full disk woes`` on hdfs-user. > Datanodes fill their storage directories unevenly, leading to situations > where certain disks are full while others are significantly less used. Users > at many different sites have experienced this issue, and HDFS administrators > are taking steps like: > - Manually rebalancing blocks in storage directories > - Decomissioning nodes & later readding them > There's a tradeoff between making use of all available spindles, and filling > disks at the sameish rate. Possible solutions include: > - Weighting less-used disks heavier when placing new blocks on the datanode. > In write-heavy environments this will still make use of all spindles, > equalizing disk use over time. > - Rebalancing blocks locally. This would help equalize disk use as disks are > added/replaced in older cluster nodes. > Datanodes should actively manage their local disk so operator intervention is > not needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)