Re: DataNode not recognising available disk space
Hi Yusaku, Thank you for your reply, unfortunately that's not the problem - the new slave node has a single drive, so is using the default data directory path. I'll post this to the Ambari list. Regards, Charles On 16 September 2014 23:12, Yusaku Sako wrote: > Charles, > > If the newly added slave node has an extra disk that the other nodes > don't have, then you will have to do the following on the Ambari Web > UI so that "dfs.data.dir" is reflected to include this extra drive for > that node: > * Go to Services > HDFS > Configs > * Click on "Manage Config Groups" link > * Create a new HDFS Configuration Group (name it appropriately), add > the slave host in question to that group, and Save. > * In the list of configs for HDFS, you will see "DataNode directories" > with the current directories set up for DataNodes for all other hosts. > * Click on the "+" next to "DataNode directories" and list out all the > drives for this new DataNode. > * Save the configuration changes. > * Restart the DataNode. > * Ambari (and HDFS) should start reflecting the added disk space. > > I hope this solves your problem. > > FYI, there's an Ambari-specific mailing list at u...@ambari.apache.org > that you can subscribe by sending an email to > user-subscr...@ambari.apache.org. > > Thanks, > Yusaku > > > On Tue, Sep 16, 2014 at 5:28 AM, Charles Robertson > wrote: > > Hi all, > > > > I've added a new slave node to my cluster with a (single) larger disk > size > > (100Gb) than on the other nodes. However, Amabri is reporting a total of > 8.6 > > Gb disk space. lsblk correctly reports the disk size. > > > > Does anyone why this might be? As I understand things you need to tell > HDFS > > how much space *not* to use, and it will use what it needs from the rest. > > > > (BTW this is not the same data node in my other question on this list, > > "Cannot start DataNode after adding new volume.") > > > > Thanks, > > Charles > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. >
Re: DataNode not recognising available disk space
Charles, If the newly added slave node has an extra disk that the other nodes don't have, then you will have to do the following on the Ambari Web UI so that "dfs.data.dir" is reflected to include this extra drive for that node: * Go to Services > HDFS > Configs * Click on "Manage Config Groups" link * Create a new HDFS Configuration Group (name it appropriately), add the slave host in question to that group, and Save. * In the list of configs for HDFS, you will see "DataNode directories" with the current directories set up for DataNodes for all other hosts. * Click on the "+" next to "DataNode directories" and list out all the drives for this new DataNode. * Save the configuration changes. * Restart the DataNode. * Ambari (and HDFS) should start reflecting the added disk space. I hope this solves your problem. FYI, there's an Ambari-specific mailing list at u...@ambari.apache.org that you can subscribe by sending an email to user-subscr...@ambari.apache.org. Thanks, Yusaku On Tue, Sep 16, 2014 at 5:28 AM, Charles Robertson wrote: > Hi all, > > I've added a new slave node to my cluster with a (single) larger disk size > (100Gb) than on the other nodes. However, Amabri is reporting a total of 8.6 > Gb disk space. lsblk correctly reports the disk size. > > Does anyone why this might be? As I understand things you need to tell HDFS > how much space *not* to use, and it will use what it needs from the rest. > > (BTW this is not the same data node in my other question on this list, > "Cannot start DataNode after adding new volume.") > > Thanks, > Charles -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
DataNode not recognising available disk space
Hi all, I've added a new slave node to my cluster with a (single) larger disk size (100Gb) than on the other nodes. However, Amabri is reporting a total of 8.6 Gb disk space. lsblk correctly reports the disk size. Does anyone why this might be? As I understand things you need to tell HDFS how much space *not* to use, and it will use what it needs from the rest. (BTW this is not the same data node in my other question on this list, "Cannot start DataNode after adding new volume.") Thanks, Charles