Interesting By same config I guess you mean same hbase config. Can you find out what was different between the two clusters ?
Thanks > On Oct 29, 2015, at 1:26 AM, Loïc Chanel <[email protected]> wrote: > > I can see that too on one of our clusters, and the thing which is really > weird is that another one of ours has the exact same configuration (as it > is the pre-production cluster) and we don't see the problem there. > I also did a lot of googling, but as we couldn't find a solution we simply > made a cron to restart periodically the RegionServers (to avoid a full on > Hadoop data partitions). > > Regards, > > > Loïc > > Loïc CHANEL > System & virtualization engineer > TO - XaaS Ind - Worldline (Villeurbanne, France) > > 2015-10-28 23:20 GMT+01:00 Yahoo <[email protected]>: > >> I see exactly the same thing on one of our clusters, also running HBase >> 0.98 (not sure of the rest of the version number since I'm not in the >> office right now). The non-hdfs disk space slowly fills up and I failed to >> locate the actual files using 'du'. I did a lot of googling but couldn't >> find any other mentions of the problem at the time. >> >> Mike. >> >>> On 24/10/2015 04:08, Otis Gospodnetić wrote: >>> >>> Hi Ted, >>> >>> 0.98.6-cdh5.3.0 >>> >>> I did actually try to use lsof, but I didn't see anything unusual there. >>> Is there something specific I should look for? Things owned by hbase user >>> or hdfs or yarn? Hm, here, I don't really see anything interesting >> <snip> >> >> >>> Thanks, >>> Otis >>> -- >>> Monitoring - Log Management - Alerting - Anomaly Detection >>> Solr & Elasticsearch Consulting Support Training - http://sematext.com/ >>> >>> >>> On Fri, Oct 23, 2015 at 10:26 PM, Ted Yu <[email protected]> wrote: >>> >>> Which specific release of 0.98 are you using ? >>>> >>>> Have you used lsof to see which files were being held onto ? >>>> >>>> Thanks >>>> >>>> On Fri, Oct 23, 2015 at 7:21 PM, Otis Gospodnetić < >>>> [email protected]> wrote: >>>> >>>> Hello, >>>>> >>>>> Is/was there a known issue with HBase 0.98 "holding onto" files? >>>>> >>>>> We noticed the used disk space metric going up, up and up and we could >>>> not >>>> >>>>> stop it with major compaction. >>>>> But we noticed that if we restart a RegionServer 2 things happen: >>>>> 1) its disk usage immediately drops a lot >>>>> 2) the disk usage of other RegionServers drops some as well >>>>> >>>>> Have a look at this chart: >>>>> https://apps.sematext.com/spm-reports/s/Ssy4ViFGHq >>>>> >>>>> At 1:54 we restarted the first RS (blue line) >>>>> At 2:03 we restarted the second RS (dark green line) >>>>> >>>>> Is/was this a known HBase 0.98 issue? >>>>> >>>>> Thanks, >>>>> Otis >>>>> -- >>>>> Monitoring - Log Management - Alerting - Anomaly Detection >>>>> Solr & Elasticsearch Consulting Support Training - http://sematext.com/ >>
