Glad you were able to track down the issue, Liam. -n
On Wed, Mar 11, 2015 at 1:58 PM, Liam Slusser <lslus...@gmail.com> wrote: > I just wanted to update this thread as after more investigation I've > figured out why my oldWALs folder wasn't being cleaned up. I had a look at > the code to ReplicationLogCleaner and it makes this call: > > if (!config.getBoolean(HConstants.REPLICATION_ENABLE_KEY, > HConstants.REPLICATION_ENABLE_DEFAULT)) { > LOG.warn("Not configured - allowing all wals to be deleted"); > return; > } > > I searched through my logs and was never able to find that line of text. > So wrote a quick program to run that piece of code, and sure enough it came > back as True. getBoolean returns the value if it's been defined and if not > returns the default. And after reading HBASE-3489 replication is enabled > by default these days, which I also verified by looking at > HConstants.REPLICATION_ENABLE_DEFAULT. I run cloudera CDH5.3 and in the > user interface even with hbase replication set to false, wasn't putting > "hbase.replication" false in the configuration file. I manually added the > hbase.replication to false in the advance configuration for hbase-site.xml > and restarted hbase and sure enough it deleted all the logs! > > So this is probably a bug in CDH, at least in the version that I ran. I'm > running cloudera manger 5.2.1 with CDH5.3.0-1.cdh5.3.0.p0.30. > > thanks, > liam > > > On Wed, Mar 4, 2015 at 5:18 PM, Liam Slusser <lslus...@gmail.com> wrote: > > > So after removing all the replication peers hbase still doesn't want to > > clean up the oldWALs folder. In the master logs I don't see any errors > > from ReplicationLogCleaner or LogCleaner. I have my logging set to INFO > so > > I'd think I would see something. > > > > Is there anyway to run the ReplicationLogCleaner manually and see the > > output? Can I write something that calls the right API functions? > > > > thanks, > > liam > > > > > > On Fri, Feb 27, 2015 at 1:50 PM, Nick Dimiduk <ndimi...@gmail.com> > wrote: > > > >> I would let the cleaner chore handle the cleanup for you. You don't know > >> the state of all entries in that folder. To that extent, I'd avoid > making > >> any direct changes to the content of HBase's working directory, > especially > >> while HBase is running... > >> > >> On Fri, Feb 27, 2015 at 1:29 PM, Liam Slusser <lslus...@gmail.com> > wrote: > >> > >> > Once I disable/remove the replication, can I just blow away the > oldWALs > >> > folder safely? > >> > > >> > On Fri, Feb 27, 2015 at 3:10 AM, Madeleine Piffaretti < > >> > mpiffare...@powerspace.com> wrote: > >> > > >> > > Thanks a lot! > >> > > > >> > > Indeed, we had a replication enable in the past because we used the > >> > > hbase-indexer from NgData (use to replicate data from Hbase to > Solr). > >> > > The replication was disable from a long time but the hbase-indexer > >> peer > >> > was > >> > > still activated and so, as you mentioned, the data was keept to > >> > guarantee > >> > > to not lose data between disable and enable. > >> > > > >> > > I have removed the peer and empty the oldWALs folder. > >> > > > >> > > > >> > > > >> > > 2015-02-27 1:42 GMT+01:00 Liam Slusser <lslus...@gmail.com>: > >> > > > >> > > > Huge thanks, Enis, that was the information I was looking for. > >> > > > > >> > > > Cheers! > >> > > > liam > >> > > > > >> > > > > >> > > > On Thu, Feb 26, 2015 at 3:48 PM, Enis Söztutar < > enis....@gmail.com> > >> > > wrote: > >> > > > > >> > > > > @Madeleine, > >> > > > > > >> > > > > The folder gets cleaned regularly by a chore in master. When a > WAL > >> > file > >> > > > is > >> > > > > not needed any more for recovery purposes (when HBase can > guaratee > >> > > HBase > >> > > > > has flushed all the data in the WAL file), it is moved to the > >> oldWALs > >> > > > > folder for archival. The log stays there until all other > >> references > >> > to > >> > > > the > >> > > > > WAL file are finished. There is currently two services which may > >> keep > >> > > the > >> > > > > files in the archive dir. First is a TTL process, which ensures > >> that > >> > > the > >> > > > > WAL files are kept at least for 10 min. This is mainly for > >> debugging. > >> > > You > >> > > > > can reduce this time by setting hbase.master.logcleaner.ttl > >> > > configuration > >> > > > > property in master. It is by default 600000. The other one is > >> > > > replication. > >> > > > > If you have replication setup, the replication processes will > >> hang on > >> > > to > >> > > > > the WAL files until they are replicated. Even if you disabled > the > >> > > > > replication, the files are still referenced. > >> > > > > > >> > > > > You can look at the logs from master from classes (LogCleaner, > >> > > > > TimeToLiveLogCleaner, ReplicationLogCleaner) to see whether the > >> > master > >> > > is > >> > > > > actually running this chore and whether it is getting any > >> exceptions. > >> > > > > > >> > > > > @Liam, > >> > > > > Disabled replication will still hold on to the WAL files > because, > >> > > because > >> > > > > it has a guarantee to not lose data between disable and enable. > >> You > >> > can > >> > > > > remove_peer, which frees up the WAL files to be eligible for > >> > deletion. > >> > > > When > >> > > > > you re-add replication peer again, the replication will start > from > >> > the > >> > > > > current status, versus if you re-enable a peer, it will continue > >> from > >> > > > where > >> > > > > it left. > >> > > > > > >> > > > > > >> > > > > > >> > > > > On Thu, Feb 26, 2015 at 12:56 AM, Madeleine Piffaretti < > >> > > > > mpiffare...@powerspace.com> wrote: > >> > > > > > >> > > > > > Hi, > >> > > > > > > >> > > > > > The replication is not turned on HBase... > >> > > > > > Does this folder should be clean regularly? Because I have > data > >> > from > >> > > > > > december 2014... > >> > > > > > > >> > > > > > > >> > > > > > 2015-02-26 1:40 GMT+01:00 Liam Slusser <lslus...@gmail.com>: > >> > > > > > > >> > > > > > > I'm having this same problem. I had replication enabled but > >> have > >> > > > since > >> > > > > > > been disabled. However oldWALs still grows. There are so > >> many > >> > > files > >> > > > > in > >> > > > > > > there that running "hadoop fs -ls /hbase/oldWALs" runs out > of > >> > > memory. > >> > > > > > > > >> > > > > > > On Wed, Feb 25, 2015 at 9:27 AM, Nishanth S < > >> > > nishanth.2...@gmail.com > >> > > > > > >> > > > > > > wrote: > >> > > > > > > > >> > > > > > > > Do you have replication turned on in hbase and if so is > >> your > >> > > slave > >> > > > > > > > consuming the replicated data?. > >> > > > > > > > > >> > > > > > > > -Nishanth > >> > > > > > > > > >> > > > > > > > On Wed, Feb 25, 2015 at 10:19 AM, Madeleine Piffaretti < > >> > > > > > > > mpiffare...@powerspace.com> wrote: > >> > > > > > > > > >> > > > > > > > > Hi all, > >> > > > > > > > > > >> > > > > > > > > We are running out of space in our small hadoop cluster > >> so I > >> > > was > >> > > > > > > checking > >> > > > > > > > > disk usage on HDFS and I saw that most of the space was > >> > > occupied > >> > > > by > >> > > > > > > the* > >> > > > > > > > > /hbase/oldWALs* folder. > >> > > > > > > > > > >> > > > > > > > > I have checked in the "HBase Definitive Book" and others > >> > books, > >> > > > > > > web-site > >> > > > > > > > > and I have also search my issue on google but I didn't > >> find a > >> > > > > proper > >> > > > > > > > > response... > >> > > > > > > > > > >> > > > > > > > > So I would like to know what does this folder, what is > use > >> > for > >> > > > and > >> > > > > > also > >> > > > > > > > how > >> > > > > > > > > can I free space from this folder without breaking > >> > > everything... > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > If it's related to a specific version... our cluster is > >> under > >> > > > > > > > > 5.3.0-1.cdh5.3.0.p0.30 from cloudera (hbase 0.98.6). > >> > > > > > > > > > >> > > > > > > > > Thx for your help! > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > > > >