Here is the problem : the config is the exact same for HBase and anything Hadoop or system-related into the cluster. There is no apparent difference.
Loïc Loïc CHANEL System & virtualization engineer TO - XaaS Ind - Worldline (Villeurbanne, France) 2015-10-29 9:48 GMT+01:00 Ted Yu <yuzhih...@gmail.com>: > Interesting > > By same config I guess you mean same hbase config. > Can you find out what was different between the two clusters ? > > Thanks > > > On Oct 29, 2015, at 1:26 AM, Loïc Chanel <loic.cha...@telecomnancy.net> > wrote: > > > > I can see that too on one of our clusters, and the thing which is really > > weird is that another one of ours has the exact same configuration (as it > > is the pre-production cluster) and we don't see the problem there. > > I also did a lot of googling, but as we couldn't find a solution we > simply > > made a cron to restart periodically the RegionServers (to avoid a full on > > Hadoop data partitions). > > > > Regards, > > > > > > Loïc > > > > Loïc CHANEL > > System & virtualization engineer > > TO - XaaS Ind - Worldline (Villeurbanne, France) > > > > 2015-10-28 23:20 GMT+01:00 Yahoo <mike.dow...@gmail.com>: > > > >> I see exactly the same thing on one of our clusters, also running HBase > >> 0.98 (not sure of the rest of the version number since I'm not in the > >> office right now). The non-hdfs disk space slowly fills up and I failed > to > >> locate the actual files using 'du'. I did a lot of googling but couldn't > >> find any other mentions of the problem at the time. > >> > >> Mike. > >> > >>> On 24/10/2015 04:08, Otis Gospodnetić wrote: > >>> > >>> Hi Ted, > >>> > >>> 0.98.6-cdh5.3.0 > >>> > >>> I did actually try to use lsof, but I didn't see anything unusual > there. > >>> Is there something specific I should look for? Things owned by hbase > user > >>> or hdfs or yarn? Hm, here, I don't really see anything interesting > >> <snip> > >> > >> > >>> Thanks, > >>> Otis > >>> -- > >>> Monitoring - Log Management - Alerting - Anomaly Detection > >>> Solr & Elasticsearch Consulting Support Training - > http://sematext.com/ > >>> > >>> > >>> On Fri, Oct 23, 2015 at 10:26 PM, Ted Yu <yuzhih...@gmail.com> wrote: > >>> > >>> Which specific release of 0.98 are you using ? > >>>> > >>>> Have you used lsof to see which files were being held onto ? > >>>> > >>>> Thanks > >>>> > >>>> On Fri, Oct 23, 2015 at 7:21 PM, Otis Gospodnetić < > >>>> otis.gospodne...@gmail.com> wrote: > >>>> > >>>> Hello, > >>>>> > >>>>> Is/was there a known issue with HBase 0.98 "holding onto" files? > >>>>> > >>>>> We noticed the used disk space metric going up, up and up and we > could > >>>> not > >>>> > >>>>> stop it with major compaction. > >>>>> But we noticed that if we restart a RegionServer 2 things happen: > >>>>> 1) its disk usage immediately drops a lot > >>>>> 2) the disk usage of other RegionServers drops some as well > >>>>> > >>>>> Have a look at this chart: > >>>>> https://apps.sematext.com/spm-reports/s/Ssy4ViFGHq > >>>>> > >>>>> At 1:54 we restarted the first RS (blue line) > >>>>> At 2:03 we restarted the second RS (dark green line) > >>>>> > >>>>> Is/was this a known HBase 0.98 issue? > >>>>> > >>>>> Thanks, > >>>>> Otis > >>>>> -- > >>>>> Monitoring - Log Management - Alerting - Anomaly Detection > >>>>> Solr & Elasticsearch Consulting Support Training - > http://sematext.com/ > >> >
