RE: HDFS multiple dfs_data_dir disbalance

[Brahma Reddy Battula]

does /data1 is having non-hadoop data..? Please check for same..
check admin report(hdfs dfsadmin -report) and fsck report (hdfs fsck /)

Thinking that following might not 
a) /data/2 is not having the permission to write data where volumes tolerated 
configured as 1
b) /data/2 is not added after some time



Thanks  Regards
Brahma Reddy Battula

From: Georgi Ivanov [iva...@vesseltracker.com]
Sent: Wednesday, October 22, 2014 5:17 PM
To: user@hadoop.apache.org
Subject: HDFS multiple dfs_data_dir disbalance

Hi,
My cluster is configured with 2 data dirs.
/data/1
/data/2

Usually hadoop is balancing the utilization of these dirs.
Now i have one node where /data/1 is 100% full and /data/2 is not.

Is there anything i can do about this, as this results in failed
mapppers/rdecers ?

Georgi

RE: HDFS multiple dfs_data_dir disbalance

[Rakesh R]

Yes, there is a VolumeChoosingPolicy configuration 
dfs.datanode.fsdataset.volume.choosing.policy in hdfs 
and by default it is configured to choose volumes in round-robin order. Hope 
you are using the default policy?


Did you see any warnings or errors about the '/data/2' in datanode logs, 
datanode will do sanity checks 
and if there is any exception it will skip this directory. Probably you can see 
this chance.

Regards,
Rakesh

-Original Message-
From: Georgi Ivanov [mailto:iva...@vesseltracker.com] 
Sent: 22 October 2014 17:17
To: user@hadoop.apache.org
Subject: HDFS multiple dfs_data_dir disbalance

Hi,
My cluster is configured with 2 data dirs.
/data/1
/data/2

Usually hadoop is balancing the utilization of these dirs.
Now i have one node where /data/1 is 100% full and /data/2 is not.

Is there anything i can do about this, as this results in failed 
mapppers/rdecers ?

Georgi