I can see that too on one of our clusters, and the thing which is really weird is that another one of ours has the exact same configuration (as it is the pre-production cluster) and we don't see the problem there. I also did a lot of googling, but as we couldn't find a solution we simply made a cron to restart periodically the RegionServers (to avoid a full on Hadoop data partitions).
Regards, Loïc Loïc CHANEL System & virtualization engineer TO - XaaS Ind - Worldline (Villeurbanne, France) 2015-10-28 23:20 GMT+01:00 Yahoo <mike.dow...@gmail.com>: > I see exactly the same thing on one of our clusters, also running HBase > 0.98 (not sure of the rest of the version number since I'm not in the > office right now). The non-hdfs disk space slowly fills up and I failed to > locate the actual files using 'du'. I did a lot of googling but couldn't > find any other mentions of the problem at the time. > > Mike. > > On 24/10/2015 04:08, Otis Gospodnetić wrote: > >> Hi Ted, >> >> 0.98.6-cdh5.3.0 >> >> I did actually try to use lsof, but I didn't see anything unusual there. >> Is there something specific I should look for? Things owned by hbase user >> or hdfs or yarn? Hm, here, I don't really see anything interesting >> > <snip> > > >> Thanks, >> Otis >> -- >> Monitoring - Log Management - Alerting - Anomaly Detection >> Solr & Elasticsearch Consulting Support Training - http://sematext.com/ >> >> >> On Fri, Oct 23, 2015 at 10:26 PM, Ted Yu <yuzhih...@gmail.com> wrote: >> >> Which specific release of 0.98 are you using ? >>> >>> Have you used lsof to see which files were being held onto ? >>> >>> Thanks >>> >>> On Fri, Oct 23, 2015 at 7:21 PM, Otis Gospodnetić < >>> otis.gospodne...@gmail.com> wrote: >>> >>> Hello, >>>> >>>> Is/was there a known issue with HBase 0.98 "holding onto" files? >>>> >>>> We noticed the used disk space metric going up, up and up and we could >>>> >>> not >>> >>>> stop it with major compaction. >>>> But we noticed that if we restart a RegionServer 2 things happen: >>>> 1) its disk usage immediately drops a lot >>>> 2) the disk usage of other RegionServers drops some as well >>>> >>>> Have a look at this chart: >>>> https://apps.sematext.com/spm-reports/s/Ssy4ViFGHq >>>> >>>> At 1:54 we restarted the first RS (blue line) >>>> At 2:03 we restarted the second RS (dark green line) >>>> >>>> Is/was this a known HBase 0.98 issue? >>>> >>>> Thanks, >>>> Otis >>>> -- >>>> Monitoring - Log Management - Alerting - Anomaly Detection >>>> Solr & Elasticsearch Consulting Support Training - http://sematext.com/ >>>> >>>> >
