What does your dfs.datanode.data.dir point to ?
On Wed, Nov 4, 2015 at 4:14 PM, Adaryl "Bob" Wakefield, MBA < adaryl.wakefi...@hotmail.com> wrote: > Filesystem Size Used Avail Use% Mounted on /dev/mapper/centos-root 50G 12G > 39G 23% / devtmpfs 16G 0 16G 0% /dev tmpfs 16G 0 16G 0% /dev/shm tmpfs 16G > 1.4G 15G 9% /run tmpfs 16G 0 16G 0% /sys/fs/cgroup /dev/sda2 494M 123M > 372M 25% /boot /dev/mapper/centos-home 2.7T 33M 2.7T 1% /home > > That’s from one datanode. The second one is nearly identical. I discovered > that 50GB is actually a default. That seems really weird. Disk space is > cheap. Why would you not just use most of the disk and why is it so hard to > reset the default? > > Adaryl "Bob" Wakefield, MBA > Principal > Mass Street Analytics, LLC > 913.938.6685 > www.linkedin.com/in/bobwakefieldmba > Twitter: @BobLovesData > > *From:* Chris Nauroth <cnaur...@hortonworks.com> > *Sent:* Wednesday, November 04, 2015 12:16 PM > *To:* user@hadoop.apache.org > *Subject:* Re: hadoop not using whole disk for HDFS > > How are those drives partitioned? Is it possible that the directories > pointed to by the dfs.datanode.data.dir property in hdfs-site.xml reside on > partitions that are sized to only 100 GB? Running commands like df would > be a good way to check this at the OS level, independently of Hadoop. > > --Chris Nauroth > > From: MBA <adaryl.wakefi...@hotmail.com> > Reply-To: "user@hadoop.apache.org" <user@hadoop.apache.org> > Date: Tuesday, November 3, 2015 at 11:16 AM > To: "user@hadoop.apache.org" <user@hadoop.apache.org> > Subject: Re: hadoop not using whole disk for HDFS > > Yeah. It has the current value of 1073741824 which is like 1.07 gig. > > B. > *From:* Chris Nauroth <cnaur...@hortonworks.com> > *Sent:* Tuesday, November 03, 2015 11:57 AM > *To:* user@hadoop.apache.org > *Subject:* Re: hadoop not using whole disk for HDFS > > Hi Bob, > > Does the hdfs-site.xml configuration file contain the property > dfs.datanode.du.reserved? If this is defined, then the DataNode > intentionally will not use this space for storage of replicas. > > <property> > <name>dfs.datanode.du.reserved</name> > <value>0</value> > <description>Reserved space in bytes per volume. Always leave this much > space free for non dfs use. > </description> > </property> > > --Chris Nauroth > > From: MBA <adaryl.wakefi...@hotmail.com> > Reply-To: "user@hadoop.apache.org" <user@hadoop.apache.org> > Date: Tuesday, November 3, 2015 at 10:51 AM > To: "user@hadoop.apache.org" <user@hadoop.apache.org> > Subject: hadoop not using whole disk for HDFS > > I’ve got the Hortonworks distro running on a three node cluster. For some > reason the disk available for HDFS is MUCH less than the total disk space. > Both of my data nodes have 3TB hard drives. Only 100GB of that is being > used for HDFS. Is it possible that I have a setting wrong somewhere? > > B. >
