Hi Reid You suggest me to directly delete OldWals by using Hdfs command "hdfs -rm" Question is : It's really safe and I will not lose any data and Indexing from the system.
Thanks Manjeet Singh On Thu, Jul 12, 2018 at 12:09 PM, Manjeet Singh <manjeet.chand...@gmail.com> wrote: > Hi > > I have created HBASE-20877 > <https://issues.apache.org/jira/browse/HBASE-20877> for the same, request > you to please move it into active sprint. > > Thanks > Manjeet Singh > > On Thu, Jul 12, 2018 at 7:42 AM, Reid Chan <reidddc...@outlook.com> wrote: > >> oldWals are supposed to be cleaned in master background chore, I also >> doubt they are needed. >> >> HBASE-20352(for 1.x version) is to speed up cleaning oldWals, it may >> address your concern "OldWals is quite huge" >> >> >> R.C >> >> >> >> ________________________________________ >> From: Manjeet Singh <manjeet.chand...@gmail.com> >> Sent: 12 July 2018 08:19:21 >> To: user@hbase.apache.org >> Subject: Re: Query for OldWals and use of WAl for Hbase indexer >> >> I have one more question >> >> If solr is having its own data mean its maintaining data in their shards >> and hbase is maintaining in data folder... Why still oldWals need? >> >> Thanks >> Manjeet singh >> >> On Wed, 11 Jul 2018, 23:19 Manjeet Singh, <manjeet.chand...@gmail.com> >> wrote: >> >> > Thanks Sean for your reply >> > >> > I still have some question un answered like >> > Q1: How Hbase syncronized with Hbase indexer. >> > Q2 What optimization I can apply. >> > Q3 As it's clear from my stats, data in OldWals is quite huge so it's >> not >> > getting clear my HMaster., how can I improve my HDFS space issue due to >> > this? >> > >> > Thanks >> > Manjeet Singh >> > >> > On Wed, Jul 11, 2018 at 9:33 PM, Sean Busbey <bus...@apache.org> wrote: >> > >> >> Presuming you're using the Lily indexer[1], yes it relies on hbase's >> >> built in cross-cluster replication. >> >> >> >> The replication system stores WALs until it can successfully send them >> >> for replication. If you look in ZK you should be able to see which >> >> regionserver(s) are waiting to send those WALs over. The easiest way >> >> to do this is probably to look at the "zk dump" web page on the >> >> Master's web ui[2]. >> >> >> >> Once you have the particular region server(s), take a look at their >> >> logs for messages about difficulty sending edits to the replication >> >> peer you have set up for the destination solr collection. >> >> >> >> If you remove the WALs then the solr collection will have a hole in >> >> it. Depending on how far behind you are, it might be quicker to 1) >> >> remove the replication peer, 2) wait for old wals to clear, 3) >> >> reenable replication, 4) use a batch indexing tool to index data >> >> already in the table. >> >> >> >> [1]: >> >> >> >> http://ngdata.github.io/hbase-indexer/ >> >> >> >> [2]: >> >> >> >> The specifics will vary depending on your installation, but the page >> >> is essentially at a URL like >> >> https://active-master-host.example.com:22002/zk.jsp >> >> >> >> the link is on the master UI landing page, near the bottom, in the >> >> description of the "ZooKeeper Quorum" row. it's the end of "Addresses >> >> of all registered ZK servers. For more, see zk dump." >> >> >> >> On Wed, Jul 11, 2018 at 10:16 AM, Manjeet Singh >> >> <manjeet.chand...@gmail.com> wrote: >> >> > Hi All >> >> > >> >> > I have a query regarding Hbase replication and OldWals >> >> > >> >> > Hbase version 1.2.1 >> >> > >> >> > To enable Hbase indexing we use below command on table >> >> > >> >> > alter '<NameOfTable>', {NAME => 'CF1', REPLICATION_SCOPE => 1} >> >> > >> >> > By Doing this actually replication get enabled as hbase-indexer >> required >> >> > it, as per my understanding indexer use hbase WAL (Please correct me >> if >> >> I >> >> > am wrong). >> >> > >> >> > so question is How Hbase syncronize with Solr Indexer? What is the >> role >> >> of >> >> > replication? what optimization we can apply in order to reduce data >> >> size? >> >> > >> >> > >> >> > I can see that our OldWals are getting filled , if Hmaster it self >> >> taking >> >> > care why it's reached to 7.2 TB? what if I delete it, does it impact >> >> solr >> >> > indexing? >> >> > >> >> > 7.2 K 21.5 K /hbase/.hbase-snapshot >> >> > 0 0 /hbase/.tmp >> >> > 0 0 /hbase/MasterProcWALs >> >> > 18.3 G 60.2 G /hbase/WALs >> >> > 28.7 G 86.1 G /hbase/archive >> >> > 0 0 /hbase/corrupt >> >> > 1.7 T 5.2 T /hbase/data >> >> > 42 126 /hbase/hbase.id >> >> > 7 21 /hbase/hbase.version >> >> > 7.2 T 21.6 T /hbase/oldWALs >> >> > >> >> > >> >> > >> >> > >> >> > Thanks >> >> > Manjeet Singh >> >> >> > >> > >> > >> > -- >> > luv all >> > >> > > > > -- > luv all > -- luv all
