Is there a way to see the rate at which under replicated blocks are being propagated to other data nodes besides running fsck command, meaning more in real time?
Artem Ervits Data Analyst New York Presbyterian Hospital From: Suresh Srinivas [mailto:sur...@hortonworks.com] Sent: Wednesday, September 19, 2012 02:38 PM To: hdfs-user@hadoop.apache.org <hdfs-user@hadoop.apache.org> Subject: Re: How does decommissioning work Bryan, I am going to assume that you know about replication factor per file, block, replicas etc. Nodes are marked for decommissioned by adding it to excludes file (btw you mean dfs.hosts.exclude right?). HDFS marks these datanodes as decommissioning. HDFS no longer counts those replicas towards the replication factor. This results in increased number of under replicated blocks. HDFS starts replicating this under replicated blocks, preferring decommissioning node as the source as much as possible. At this time, the decommissioning nodes are used reads only. Decommissioning completes when replication completes and replicas from those nodes are no more needed. The node is then marked decommissioned. Not sure I answered your questions. Regards, Suresh On Tue, Sep 18, 2012 at 4:01 PM, Bryan Beaudreault <bbeaudrea...@hubspot.com<mailto:bbeaudrea...@hubspot.com>> wrote: Hello, I'm using cdh3u2, if it matters. I'm using the dfs.exclude.hosts to decommission a good percentage of my cluster as I scale it down for a period of time. I'm just trying to understand how hdfs goes about this, because I haven't found anything more than a "how to use" documentation for the feature. When I look at the name node UI, I see under replicated blocks count go up when I decommission. Also, when I look at the dfsnodelist with whatNodes=decommissioning, there are stats there like Blocks with no live replicas, etc. When I decommission a node does it immediately make that node unavailable, thus these stats? Or does it move them off safely and these counts are just to say what would happen if the node was shut down without decommission? Something else? Thanks for any insight. -- http://hortonworks.com/download/ -------------------- This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you. -------------------- This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you.
