Fwd: Migrating from Apache Cassandra to Hbase

[onmstester onmstester]

Any idea? Sent using Zoho Mail  Forwarded message  From 
: onmstester onmstester  To : 
"user" Date : Sat, 08 Sep 2018 10:46:25 +0430 Subject : 
Migrating from Apache Cassandra to Hbase  Forwarded message 
 Hi, Currently I'm using Apache Cassandra as backend for my 
restfull application. Having a cluster of 30 nodes (each having 12 cores, 64gb 
ram and 6 TB disk which 50% of the disk been used) write and read throughput is 
more than satisfactory for us. The input is a fixed set of long and int columns 
which we need to query it based on every column, so having 8 columns there 
should be 8 tables based on Cassandra query plan recommendation. The cassandra 
keyspace schema would be someting like this: Table 1 (timebucket,col1, 
...,col8, primary key(timebuecket,col1)) to handle select * from input where 
timebucket = X and col1 = Y  Table 8 (timebucket,col1, ...,col8, primary 
key(timebuecket,col8)) So for each input row, there would be 8X insert in 
Cassandra (not considering RF) and using TTL of 12 months, production cluster 
should keep about 2 Peta Bytes of data With recommended node density for 
Cassandra cluster (2 TB per node), i need a cluster with more than 1000 nodes 
(which i can not afford) So long story short: I'm looking for an alternative to 
Apache Cassandra for this application. How HBase would solve these problem: 1. 
8X data redundancy due to needed queries 2. nodes with large data density (30 
TB data on each node if No.1 could not be solved in HBase), how HBase would 
handle compaction and node join-remove problems while there is only 5 * 6 TB 
7200 SATA Disk available on each node? How much Hbase needs as empty space for 
template files of compaction? 3. Also i read in some documents (including 
datastax's) that HBase is more of a offline & data-lake backend that better not 
to be used as web application backendd which needs less than some seconds QoS 
in response time. Thanks in advance Sent using Zoho Mail

Re: HBase 2.0.1 INCONSISTENT issues

[Oleg Galitskiy]

Upgrade to "2.0.2" version didn't help.
"Fixed" by re-creating tables, which is not good.
Hope in nearest future we will have tool for fix that.

On Mon, Sep 10, 2018 at 2:05 PM Oleg Galitskiy 
wrote:

> Yes, bow main issue in "hole".
>
> Could you give me example how I can fetch or scan section of the table?
>
> Thanks.
>
>
> On Mon, Sep 10, 2018 at 1:53 PM Stack  wrote:
>
>> On Mon, Sep 10, 2018 at 1:32 PM Oleg Galitskiy 
>> wrote:
>>
>> > Hello,
>> >
>> > Faced with inconsistent issues on HBase 2.0.1:
>> > --
>> >
>> > ERROR: Region \{ meta => null, hdfs =>
>> >
>> >
>> hdfs://master:50001/hbase/data/default/some_table/0646d0bee757d0fb0de1529475b5426f,
>> > deployed =>
>> >
>> hbase-region,16020,1536493017073;some_table,,1534195327532.0646d0bee757d0fb0de1529475b5426f.,
>> > replicaId => 0 } not in META, but deployed on
>> > hbase-region,16020,1536493017073...
>> > ERROR: hbase:namespace has no state in meta
>> > ERROR: table1 has no state in meta
>> > ERROR: table2 has no state in meta
>> > 2018-09-09 21:40:04,155 INFO [main] util.HBaseFsck: Handling overlap
>> > merges in parallel. set hbasefsck.overlap.merge.parallel to false to
>> > run serially.
>> > ERROR: There is a hole in the region chain between and . You need to
>> > create a new .regioninfo and region dir in hdfs to plug the hole.
>> > ERROR: Found inconsistency in table test3
>> >
>> > --
>> >
>> > BUT in 2.0.x HBAse version options *-repair, -fix, -fixHdfsHoles, etc*
>> was
>> > deprecated.
>> > How I can fix it without these options?
>> >
>> > The main question now - is how to fix "hole":
>> > --
>> > ERROR: There is a hole in the region chain between
>> > \x01mmdd=20180813|ad=  0|source=   rubicon|site=
>> >
>> >  demorgen.be|placement=   1892806 and \x01mmdd=20180817|ad=
>> >   0|source= index|site=
>> >  dailynews.co.th|placement=
>> >  1898.  You need to create a new .regioninfo and region dir in hdfs to
>> plug
>> > the hole.
>> > --
>> >
>> >
>> Yeah, hbck1 can't repair an hbase2 cluster.
>>
>> And for sure, you have a 'hole'? What happens if you fetch or scan that
>> section of the table?
>>
>> Please update to 2.0.2 also.
>>
>> S
>>
>>
>>
>>
>> > Thanks.
>> >
>>
>

Re: questions regarding hbase major compaction

[Josh Elser]

1. Yes
2. HDFS NN pressure, read slow down, general poor performance
3. Default configuration is weekly, if you don't explicitly know some 
reasons why weekly doesn't work, this is what you should follow ;)

4. No

I would be surprised if you need to do anything special with S3, but I 
don't know for sure.


On 9/10/18 2:19 PM, Antonio Si wrote:

Hello,

As I understand, the deleted records in hbase files do not get removed
until a major compaction is performed.

I have a few questions regarding major compaction:

1.   If I set a TTL and/or a max number of versions, the records are older
than the TTL or the
   expired versions will still be in the hbase files until the major
compaction is performed?
   Is my understanding correct?

2.   If a major compaction is never performed on a table, besides the size
of the table keep
   increasing, eventually, we will have too many hbase files and the
cluster will slow down.
   Is there any other implications?

3.   Is there any guidelines about how often should we run major compaction?

4.   During major compaction, do we need to pause all read/write operations
until major
   compaction is finished?

   I realize that if using S3 as the storage, after I run major
compaction, there is inconsistencies
   between s3 metadata and s3 file system and I need to run a "emrfs
sync" to synchronize them
   after major compaction is completed. Does it mean I need to pause all
read/write operations
   during this period?

Thanks.

Antonio.

Re: Extremely high CPU usage after upgrading to Hbase 1.4.4

[Srinidhi Muppalla]

It is during a period when the number of client operations was relatively low. 
It wasn’t zero, but it was definitely off peak hours. 

On 9/10/18, 12:16 PM, "Ted Yu"  wrote:

In the previous stack trace you sent, shortCompactions and longCompactions
threads were not active.

Was the stack trace captured during period when the number of client
operations was low ?

If not, can you capture stack trace during off peak hours ?

Cheers

On Mon, Sep 10, 2018 at 12:08 PM Srinidhi Muppalla 
wrote:

> Hi Ted,
>
> The highest number of filters used is 10, but the average is generally
> close to 1. Is it possible the CPU usage spike has to do with Hbase
> internal maintenance operations? It looks like post-upgrade the spike 
isn’t
> correlated with the frequency of reads/writes we are making, because the
> high CPU usage persisted when the number of operations went down.
>
> Thank you,
> Srinidhi
>
> On 9/8/18, 9:44 AM, "Ted Yu"  wrote:
>
> Srinidhi :
> Do you know the average / highest number of ColumnPrefixFilter's in 
the
> FilterList ?
>
> Thanks
>
> On Fri, Sep 7, 2018 at 10:00 PM Ted Yu  wrote:
>
> > Thanks for detailed background information.
> >
> > I assume your code has done de-dup for the filters contained in
> > FilterListWithOR.
> >
> > I took a look at JIRAs which
> > touched hbase-client/src/main/java/org/apache/hadoop/hbase/filter in
> > branch-1.4
> > There were a few patches (some were very big) since the release of
> 1.3.0
> > So it is not obvious at first glance which one(s) might be related.
> >
> > I noticed ColumnPrefixFilter.getNextCellHint (and
> > KeyValueUtil.createFirstOnRow) appearing many times in the stack
> trace.
> >
> > I plan to dig more in this area.
> >
> > Cheers
> >
> > On Fri, Sep 7, 2018 at 11:30 AM Srinidhi Muppalla <
> srinid...@trulia.com>
> > wrote:
> >
> >> Sure thing. For our table schema, each row represents one user and
> the
> >> row key is that user’s unique id in our system. We currently only
> use one
> >> column family in the table. The column qualifiers represent an item
> that
> >> has been surfaced to that user as well as additional information to
> >> differentiate the way the item has been surfaced to the user.
> Without
> >> getting into too many specifics, the qualifier follows the rough
> format of:
> >>
> >> “Channel-itemId-distinguisher”.
> >>
> >> The channel here is the channel through the item was previously
> surfaced
> >> to the user. The itemid is the unique id of the item that has been
> surfaced
> >> to the user. A distinguisher is some attribute about how that item
> was
> >> surfaced to the user.
> >>
> >> When we run a scan, we currently only ever run it on one row at a
> time.
> >> It was chosen over ‘get’ because (from our understanding) the
> performance
> >> difference is negligible, and down the road using scan would allow
> us some
> >> more flexibility.
> >>
> >> The filter list that is constructed with scan works by using a
> >> ColumnPrefixFilter as you mentioned. When a user is being
> communicated to
> >> on a particular channel, we have a list of items that we want to
> >> potentially surface for that user. So, we construct a prefix list
> with the
> >> channel and each of the item ids in the form of: “channel-itemId”.
> Then we
> >> run a scan on that row with that filter list using “WithOr” to get
> all of
> >> the matching channel-itemId combinations currently in that
> row/column
> >> family in the table. This way we can then know which of the items
> we want
> >> to surface to that user on that channel have already been surfaced
> on that
> >> channel. The reason we query using a prefix filter is so that we
> don’t need
> >> to know the ‘distinguisher’ part of the record when writing the
> actual
> >> query, because the distinguisher is only relevant in certain
> circumstances.
> >>
> >> Let me know if this is the information about our query pattern that
> you
> >> were looking for and if there is anything I can clarify or add.
> >>
> >> Thanks,
> >> Srinidhi
> >>
> >> On 9/6/18, 12:24 PM, "Ted Yu"  wrote:
> >>
> >> From the stack trace, ColumnPrefixFilter is used during scan.
> >>
> >> Can you illustrate how various filters are formed thru
> >> 

Re: HBase 2.0.1 INCONSISTENT issues

[Oleg Galitskiy]

Yes, bow main issue in "hole".

Could you give me example how I can fetch or scan section of the table?

Thanks.


On Mon, Sep 10, 2018 at 1:53 PM Stack  wrote:

> On Mon, Sep 10, 2018 at 1:32 PM Oleg Galitskiy 
> wrote:
>
> > Hello,
> >
> > Faced with inconsistent issues on HBase 2.0.1:
> > --
> >
> > ERROR: Region \{ meta => null, hdfs =>
> >
> >
> hdfs://master:50001/hbase/data/default/some_table/0646d0bee757d0fb0de1529475b5426f,
> > deployed =>
> >
> hbase-region,16020,1536493017073;some_table,,1534195327532.0646d0bee757d0fb0de1529475b5426f.,
> > replicaId => 0 } not in META, but deployed on
> > hbase-region,16020,1536493017073...
> > ERROR: hbase:namespace has no state in meta
> > ERROR: table1 has no state in meta
> > ERROR: table2 has no state in meta
> > 2018-09-09 21:40:04,155 INFO [main] util.HBaseFsck: Handling overlap
> > merges in parallel. set hbasefsck.overlap.merge.parallel to false to
> > run serially.
> > ERROR: There is a hole in the region chain between and . You need to
> > create a new .regioninfo and region dir in hdfs to plug the hole.
> > ERROR: Found inconsistency in table test3
> >
> > --
> >
> > BUT in 2.0.x HBAse version options *-repair, -fix, -fixHdfsHoles, etc*
> was
> > deprecated.
> > How I can fix it without these options?
> >
> > The main question now - is how to fix "hole":
> > --
> > ERROR: There is a hole in the region chain between
> > \x01mmdd=20180813|ad=  0|source=   rubicon|site=
> >
> >  demorgen.be|placement=   1892806 and \x01mmdd=20180817|ad=
> >   0|source= index|site=
> >  dailynews.co.th|placement=
> >  1898.  You need to create a new .regioninfo and region dir in hdfs to
> plug
> > the hole.
> > --
> >
> >
> Yeah, hbck1 can't repair an hbase2 cluster.
>
> And for sure, you have a 'hole'? What happens if you fetch or scan that
> section of the table?
>
> Please update to 2.0.2 also.
>
> S
>
>
>
>
> > Thanks.
> >
>

Re: HBase 2.0.1 INCONSISTENT issues

[Stack]

On Mon, Sep 10, 2018 at 1:32 PM Oleg Galitskiy 
wrote:

> Hello,
>
> Faced with inconsistent issues on HBase 2.0.1:
> --
>
> ERROR: Region \{ meta => null, hdfs =>
>
> hdfs://master:50001/hbase/data/default/some_table/0646d0bee757d0fb0de1529475b5426f,
> deployed =>
> hbase-region,16020,1536493017073;some_table,,1534195327532.0646d0bee757d0fb0de1529475b5426f.,
> replicaId => 0 } not in META, but deployed on
> hbase-region,16020,1536493017073...
> ERROR: hbase:namespace has no state in meta
> ERROR: table1 has no state in meta
> ERROR: table2 has no state in meta
> 2018-09-09 21:40:04,155 INFO [main] util.HBaseFsck: Handling overlap
> merges in parallel. set hbasefsck.overlap.merge.parallel to false to
> run serially.
> ERROR: There is a hole in the region chain between and . You need to
> create a new .regioninfo and region dir in hdfs to plug the hole.
> ERROR: Found inconsistency in table test3
>
> --
>
> BUT in 2.0.x HBAse version options *-repair, -fix, -fixHdfsHoles, etc* was
> deprecated.
> How I can fix it without these options?
>
> The main question now - is how to fix "hole":
> --
> ERROR: There is a hole in the region chain between
> \x01mmdd=20180813|ad=  0|source=   rubicon|site=
>
>  demorgen.be|placement=   1892806 and \x01mmdd=20180817|ad=
>   0|source= index|site=
>  dailynews.co.th|placement=
>  1898.  You need to create a new .regioninfo and region dir in hdfs to plug
> the hole.
> --
>
>
Yeah, hbck1 can't repair an hbase2 cluster.

And for sure, you have a 'hole'? What happens if you fetch or scan that
section of the table?

Please update to 2.0.2 also.

S




> Thanks.
>

HBase 2.0.1 INCONSISTENT issues

[Oleg Galitskiy]

Hello,

Faced with inconsistent issues on HBase 2.0.1:
--

ERROR: Region \{ meta => null, hdfs =>
hdfs://master:50001/hbase/data/default/some_table/0646d0bee757d0fb0de1529475b5426f,
deployed => 
hbase-region,16020,1536493017073;some_table,,1534195327532.0646d0bee757d0fb0de1529475b5426f.,
replicaId => 0 } not in META, but deployed on
hbase-region,16020,1536493017073...
ERROR: hbase:namespace has no state in meta
ERROR: table1 has no state in meta
ERROR: table2 has no state in meta
2018-09-09 21:40:04,155 INFO [main] util.HBaseFsck: Handling overlap
merges in parallel. set hbasefsck.overlap.merge.parallel to false to
run serially.
ERROR: There is a hole in the region chain between and . You need to
create a new .regioninfo and region dir in hdfs to plug the hole.
ERROR: Found inconsistency in table test3

--

BUT in 2.0.x HBAse version options *-repair, -fix, -fixHdfsHoles, etc* was
deprecated.
How I can fix it without these options?

The main question now - is how to fix "hole":
--
ERROR: There is a hole in the region chain between
\x01mmdd=20180813|ad=  0|source=   rubicon|site=

 demorgen.be|placement=   1892806 and \x01mmdd=20180817|ad=
  0|source= index|site=
 dailynews.co.th|placement=
 1898.  You need to create a new .regioninfo and region dir in hdfs to plug
the hole.
--

Thanks.

Re: Extremely high CPU usage after upgrading to Hbase 1.4.4

[Ted Yu]

In the previous stack trace you sent, shortCompactions and longCompactions
threads were not active.

Was the stack trace captured during period when the number of client
operations was low ?

If not, can you capture stack trace during off peak hours ?

Cheers

On Mon, Sep 10, 2018 at 12:08 PM Srinidhi Muppalla 
wrote:

> Hi Ted,
>
> The highest number of filters used is 10, but the average is generally
> close to 1. Is it possible the CPU usage spike has to do with Hbase
> internal maintenance operations? It looks like post-upgrade the spike isn’t
> correlated with the frequency of reads/writes we are making, because the
> high CPU usage persisted when the number of operations went down.
>
> Thank you,
> Srinidhi
>
> On 9/8/18, 9:44 AM, "Ted Yu"  wrote:
>
> Srinidhi :
> Do you know the average / highest number of ColumnPrefixFilter's in the
> FilterList ?
>
> Thanks
>
> On Fri, Sep 7, 2018 at 10:00 PM Ted Yu  wrote:
>
> > Thanks for detailed background information.
> >
> > I assume your code has done de-dup for the filters contained in
> > FilterListWithOR.
> >
> > I took a look at JIRAs which
> > touched hbase-client/src/main/java/org/apache/hadoop/hbase/filter in
> > branch-1.4
> > There were a few patches (some were very big) since the release of
> 1.3.0
> > So it is not obvious at first glance which one(s) might be related.
> >
> > I noticed ColumnPrefixFilter.getNextCellHint (and
> > KeyValueUtil.createFirstOnRow) appearing many times in the stack
> trace.
> >
> > I plan to dig more in this area.
> >
> > Cheers
> >
> > On Fri, Sep 7, 2018 at 11:30 AM Srinidhi Muppalla <
> srinid...@trulia.com>
> > wrote:
> >
> >> Sure thing. For our table schema, each row represents one user and
> the
> >> row key is that user’s unique id in our system. We currently only
> use one
> >> column family in the table. The column qualifiers represent an item
> that
> >> has been surfaced to that user as well as additional information to
> >> differentiate the way the item has been surfaced to the user.
> Without
> >> getting into too many specifics, the qualifier follows the rough
> format of:
> >>
> >> “Channel-itemId-distinguisher”.
> >>
> >> The channel here is the channel through the item was previously
> surfaced
> >> to the user. The itemid is the unique id of the item that has been
> surfaced
> >> to the user. A distinguisher is some attribute about how that item
> was
> >> surfaced to the user.
> >>
> >> When we run a scan, we currently only ever run it on one row at a
> time.
> >> It was chosen over ‘get’ because (from our understanding) the
> performance
> >> difference is negligible, and down the road using scan would allow
> us some
> >> more flexibility.
> >>
> >> The filter list that is constructed with scan works by using a
> >> ColumnPrefixFilter as you mentioned. When a user is being
> communicated to
> >> on a particular channel, we have a list of items that we want to
> >> potentially surface for that user. So, we construct a prefix list
> with the
> >> channel and each of the item ids in the form of: “channel-itemId”.
> Then we
> >> run a scan on that row with that filter list using “WithOr” to get
> all of
> >> the matching channel-itemId combinations currently in that
> row/column
> >> family in the table. This way we can then know which of the items
> we want
> >> to surface to that user on that channel have already been surfaced
> on that
> >> channel. The reason we query using a prefix filter is so that we
> don’t need
> >> to know the ‘distinguisher’ part of the record when writing the
> actual
> >> query, because the distinguisher is only relevant in certain
> circumstances.
> >>
> >> Let me know if this is the information about our query pattern that
> you
> >> were looking for and if there is anything I can clarify or add.
> >>
> >> Thanks,
> >> Srinidhi
> >>
> >> On 9/6/18, 12:24 PM, "Ted Yu"  wrote:
> >>
> >> From the stack trace, ColumnPrefixFilter is used during scan.
> >>
> >> Can you illustrate how various filters are formed thru
> >> FilterListWithOR ?
> >> It would be easier for other people to reproduce the problem
> given
> >> your
> >> query pattern.
> >>
> >> Cheers
> >>
> >> On Thu, Sep 6, 2018 at 11:43 AM Srinidhi Muppalla <
> >> srinid...@trulia.com>
> >> wrote:
> >>
> >> > Hi Vlad,
> >> >
> >> > Thank you for the suggestion. I recreated the issue and
> attached
> >> the stack
> >> > traces I took. Let me know if there’s any other info I can
> provide.
> >> We
> >> > narrowed the issue down to occurring when upgrading from
> 1.3.0 to
> >> any 1.4.x
> >> > version.
> >> >

Re: Extremely high CPU usage after upgrading to Hbase 1.4.4

[Srinidhi Muppalla]

Hi Ted, 

The highest number of filters used is 10, but the average is generally close to 
1. Is it possible the CPU usage spike has to do with Hbase internal maintenance 
operations? It looks like post-upgrade the spike isn’t correlated with the 
frequency of reads/writes we are making, because the high CPU usage persisted 
when the number of operations went down. 

Thank you, 
Srinidhi

On 9/8/18, 9:44 AM, "Ted Yu"  wrote:

Srinidhi :
Do you know the average / highest number of ColumnPrefixFilter's in the
FilterList ?

Thanks

On Fri, Sep 7, 2018 at 10:00 PM Ted Yu  wrote:

> Thanks for detailed background information.
>
> I assume your code has done de-dup for the filters contained in
> FilterListWithOR.
>
> I took a look at JIRAs which
> touched hbase-client/src/main/java/org/apache/hadoop/hbase/filter in
> branch-1.4
> There were a few patches (some were very big) since the release of 1.3.0
> So it is not obvious at first glance which one(s) might be related.
>
> I noticed ColumnPrefixFilter.getNextCellHint (and
> KeyValueUtil.createFirstOnRow) appearing many times in the stack trace.
>
> I plan to dig more in this area.
>
> Cheers
>
> On Fri, Sep 7, 2018 at 11:30 AM Srinidhi Muppalla 
> wrote:
>
>> Sure thing. For our table schema, each row represents one user and the
>> row key is that user’s unique id in our system. We currently only use one
>> column family in the table. The column qualifiers represent an item that
>> has been surfaced to that user as well as additional information to
>> differentiate the way the item has been surfaced to the user. Without
>> getting into too many specifics, the qualifier follows the rough format 
of:
>>
>> “Channel-itemId-distinguisher”.
>>
>> The channel here is the channel through the item was previously surfaced
>> to the user. The itemid is the unique id of the item that has been 
surfaced
>> to the user. A distinguisher is some attribute about how that item was
>> surfaced to the user.
>>
>> When we run a scan, we currently only ever run it on one row at a time.
>> It was chosen over ‘get’ because (from our understanding) the performance
>> difference is negligible, and down the road using scan would allow us 
some
>> more flexibility.
>>
>> The filter list that is constructed with scan works by using a
>> ColumnPrefixFilter as you mentioned. When a user is being communicated to
>> on a particular channel, we have a list of items that we want to
>> potentially surface for that user. So, we construct a prefix list with 
the
>> channel and each of the item ids in the form of: “channel-itemId”. Then 
we
>> run a scan on that row with that filter list using “WithOr” to get all of
>> the matching channel-itemId combinations currently in that row/column
>> family in the table. This way we can then know which of the items we want
>> to surface to that user on that channel have already been surfaced on 
that
>> channel. The reason we query using a prefix filter is so that we don’t 
need
>> to know the ‘distinguisher’ part of the record when writing the actual
>> query, because the distinguisher is only relevant in certain 
circumstances.
>>
>> Let me know if this is the information about our query pattern that you
>> were looking for and if there is anything I can clarify or add.
>>
>> Thanks,
>> Srinidhi
>>
>> On 9/6/18, 12:24 PM, "Ted Yu"  wrote:
>>
>> From the stack trace, ColumnPrefixFilter is used during scan.
>>
>> Can you illustrate how various filters are formed thru
>> FilterListWithOR ?
>> It would be easier for other people to reproduce the problem given
>> your
>> query pattern.
>>
>> Cheers
>>
>> On Thu, Sep 6, 2018 at 11:43 AM Srinidhi Muppalla <
>> srinid...@trulia.com>
>> wrote:
>>
>> > Hi Vlad,
>> >
>> > Thank you for the suggestion. I recreated the issue and attached
>> the stack
>> > traces I took. Let me know if there’s any other info I can provide.
>> We
>> > narrowed the issue down to occurring when upgrading from 1.3.0 to
>> any 1.4.x
>> > version.
>> >
>> > Thanks,
>> > Srinidhi
>> >
>> > On 9/4/18, 8:19 PM, "Vladimir Rodionov" 
>> wrote:
>> >
>> > Hi, Srinidhi
>> >
>> > Next time you will see this issue, take jstack of a RS several
>> times
>> > in a
>> > row. W/o stack traces it is hard
>> > to tell what was going on with your cluster after upgrade.
>> >
>> > -Vlad
>> >
>> >
>> >
>> > On Tue, Sep 4, 2018 at 3:50 PM Srinidhi Muppalla <
>> 

questions regarding hbase major compaction

[Antonio Si]

Hello,

As I understand, the deleted records in hbase files do not get removed
until a major compaction is performed.

I have a few questions regarding major compaction:

1.   If I set a TTL and/or a max number of versions, the records are older
than the TTL or the
  expired versions will still be in the hbase files until the major
compaction is performed?
  Is my understanding correct?

2.   If a major compaction is never performed on a table, besides the size
of the table keep
  increasing, eventually, we will have too many hbase files and the
cluster will slow down.
  Is there any other implications?

3.   Is there any guidelines about how often should we run major compaction?

4.   During major compaction, do we need to pause all read/write operations
until major
  compaction is finished?

  I realize that if using S3 as the storage, after I run major
compaction, there is inconsistencies
  between s3 metadata and s3 file system and I need to run a "emrfs
sync" to synchronize them
  after major compaction is completed. Does it mean I need to pause all
read/write operations
  during this period?

Thanks.

Antonio.

Re: Improving on MTTR of cluster [Hbase - 1.1.13]

[Ted Yu]

For the second config you mentioned, hbase.master.distributed.log.replay,
see http://hbase.apache.org/book.html#upgrade2.0.distributed.log.replay

FYI

On Mon, Sep 10, 2018 at 8:52 AM sahil aggarwal 
wrote:

> Hi,
>
> My cluster has around 50k regions and 130 RS. In case of unclean shutdown,
> the cluster take around 40 50 mins to come up(mostly slow on region
> assignment from observation). Trying to optimize it found following
> possible configs:
>
> *hbase.assignment.usezk:* which will co-host meta table and Hmaster and
> avoid zk interaction for region assignment.
> *hbase.master.distributed.log.replay:* to replay the edit logs in
> distributed manner.
>
>
> Testing *hbase.assignment.usezk* alone on small cluster(2200 regions, 4 RS)
> gave following results:
>
> hbase.assignment.usezk=true -> 12 mins
> hbase.assignment.usezk=false -> 9 mins
>
>
> From this blog
> , i
> was expecting better results so probably I am missing something. Will
> appreciate any pointers.
>
> Thanks,
> Sahil
>

Improving on MTTR of cluster [Hbase - 1.1.13]

[sahil aggarwal]

Hi,

My cluster has around 50k regions and 130 RS. In case of unclean shutdown,
the cluster take around 40 50 mins to come up(mostly slow on region
assignment from observation). Trying to optimize it found following
possible configs:

*hbase.assignment.usezk:* which will co-host meta table and Hmaster and
avoid zk interaction for region assignment.
*hbase.master.distributed.log.replay:* to replay the edit logs in
distributed manner.


Testing *hbase.assignment.usezk* alone on small cluster(2200 regions, 4 RS)
gave following results:

hbase.assignment.usezk=true -> 12 mins
hbase.assignment.usezk=false -> 9 mins


>From this blog
, i
was expecting better results so probably I am missing something. Will
appreciate any pointers.

Thanks,
Sahil