Re: ABORTING region server and following HBase cluster "crash"

[Batyrshin Alexander]

Correct me if im wrong.

But looks like if you have A and B region server that has index and primary 
table then possible situation like this.

A and B under writes on table with indexes
A - crash
B failed on index update because A is not operating then B starting aborting
A after restart try to rebuild index from WAL but B at this time is aborting 
then A starting aborting too
From this moment nothing happens (0 requests to region servers) and A and B is 
not responsible from Master-status web interface


> On 9 Sep 2018, at 04:38, Batyrshin Alexander <0x62...@gmail.com> wrote:
> 
> After update we still can't recover HBase cluster. Our region servers 
> ABORTING over and over:
> 
> prod003:
> Sep 09 02:51:27 prod003 hbase[1440]: 2018-09-09 02:51:27,395 FATAL 
> [RpcServer.default.FPBQ.Fifo.handler=92,queue=2,port=60020] 
> regionserver.HRegionServer: ABORTING region server 
> prod003,60020,1536446665703: Could not update the index table, killing server 
> region because couldn't write to an index table
> Sep 09 02:51:27 prod003 hbase[1440]: 2018-09-09 02:51:27,395 FATAL 
> [RpcServer.default.FPBQ.Fifo.handler=77,queue=7,port=60020] 
> regionserver.HRegionServer: ABORTING region server 
> prod003,60020,1536446665703: Could not update the index table, killing server 
> region because couldn't write to an index table
> Sep 09 02:52:19 prod003 hbase[1440]: 2018-09-09 02:52:19,224 FATAL 
> [RpcServer.default.FPBQ.Fifo.handler=82,queue=2,port=60020] 
> regionserver.HRegionServer: ABORTING region server 
> prod003,60020,1536446665703: Could not update the index table, killing server 
> region because couldn't write to an index table
> Sep 09 02:52:28 prod003 hbase[1440]: 2018-09-09 02:52:28,922 FATAL 
> [RpcServer.default.FPBQ.Fifo.handler=94,queue=4,port=60020] 
> regionserver.HRegionServer: ABORTING region server 
> prod003,60020,1536446665703: Could not update the index table, killing server 
> region because couldn't write to an index table
> Sep 09 02:55:02 prod003 hbase[957]: 2018-09-09 02:55:02,096 FATAL 
> [RpcServer.default.FPBQ.Fifo.handler=95,queue=5,port=60020] 
> regionserver.HRegionServer: ABORTING region server 
> prod003,60020,1536450772841: Could not update the index table, killing server 
> region because couldn't write to an index table
> Sep 09 02:55:18 prod003 hbase[957]: 2018-09-09 02:55:18,793 FATAL 
> [RpcServer.default.FPBQ.Fifo.handler=97,queue=7,port=60020] 
> regionserver.HRegionServer: ABORTING region server 
> prod003,60020,1536450772841: Could not update the index table, killing server 
> region because couldn't write to an index table
> 
> prod004:
> Sep 09 02:52:13 prod004 hbase[4890]: 2018-09-09 02:52:13,541 FATAL 
> [RpcServer.default.FPBQ.Fifo.handler=83,queue=3,port=60020] 
> regionserver.HRegionServer: ABORTING region server 
> prod004,60020,1536446387325: Could not update the index table, killing server 
> region because couldn't write to an index table
> Sep 09 02:52:50 prod004 hbase[4890]: 2018-09-09 02:52:50,264 FATAL 
> [RpcServer.default.FPBQ.Fifo.handler=75,queue=5,port=60020] 
> regionserver.HRegionServer: ABORTING region server 
> prod004,60020,1536446387325: Could not update the index table, killing server 
> region because couldn't write to an index table
> Sep 09 02:53:40 prod004 hbase[4890]: 2018-09-09 02:53:40,709 FATAL 
> [RpcServer.default.FPBQ.Fifo.handler=66,queue=6,port=60020] 
> regionserver.HRegionServer: ABORTING region server 
> prod004,60020,1536446387325: Could not update the index table, killing server 
> region because couldn't write to an index table
> Sep 09 02:54:00 prod004 hbase[4890]: 2018-09-09 02:54:00,060 FATAL 
> [RpcServer.default.FPBQ.Fifo.handler=89,queue=9,port=60020] 
> regionserver.HRegionServer: ABORTING region server 
> prod004,60020,1536446387325: Could not update the index table, killing server 
> region because couldn't write to an index table
> 
> prod005:
> Sep 09 02:52:50 prod005 hbase[3772]: 2018-09-09 02:52:50,661 FATAL 
> [RpcServer.default.FPBQ.Fifo.handler=65,queue=5,port=60020] 
> regionserver.HRegionServer: ABORTING region server 
> prod005,60020,153644649: Could not update the index table, killing server 
> region because couldn't write to an index table
> Sep 09 02:53:27 prod005 hbase[3772]: 2018-09-09 02:53:27,542 FATAL 
> [RpcServer.default.FPBQ.Fifo.handler=90,queue=0,port=60020] 
> regionserver.HRegionServer: ABORTING region server 
> prod005,60020,153644649: Could not update the index table, killing server 
> region because couldn't write to an index table
> Sep 09 02:54:00 prod005 hbase[3772]: 2018-09-09 02:53:59,915 FATAL 
> [RpcServer.default.FPBQ.Fifo.handler=7,queue=7,port=60020] 
> regionserver.HRegionServer: ABORTING region server 
> prod005,60020,153644649: Could not update the index table, killing server 
> region because couldn't write to an index table
> Sep 09 02:54:30 prod005 hbase[3772]: 2018-09-09 02:54:30,058 FATAL 
> [RpcServer.default.FPBQ.Fifo.handler=16,queue=6,port=60020] 
> reg

Re: SKIP_SCAN on variable length keys

[Batyrshin Alexander]

Thank you for reply

> On 4 Sep 2018, at 21:04, Sergey Soldatov  wrote:
> 
> SKIP SCAN doesn't use FuzzyRowFilter. It has its own SkipScanFilter. If you 
> see problems, please provide more details or file a JIRA for that. 
> 
> Thanks,
> Sergey
> 
> On Wed, Aug 29, 2018 at 2:17 PM Batyrshin Alexander <0x62...@gmail.com 
> > wrote:
>  Hello,
> Im wondering is there any issue with SKIP SCAN when variable length columns 
> used in composite key?
> My suspicion comes from FuzzyRowFilter that takes fuzzy row key template with 
> fixed positions

Unsubscribe

[William Li]

Unsubscribe 

Thanks,

William.