Thank for your reply.
Here is My test environment：
hbase:1.2.5
phoenix:4.12
Phoenix table with local index。
The region will wait for a long time when the region split。
The region server webapp page show below:
Start TimeDescriptionStateStatus
Mon Nov 06 17:49:27 CST 2017Closing region
TEST_TABLE_LOCAL,\x01,1509961563238.9070a4f2ace4516f985c6c9e31427eee.RUNNING
(since 1mins, 46sec ago)Finished memstore flush of ~39.94 MB/41879888,
currentsize=14.31 MB/15002912 for region
TEST_TABLE_LOCAL,\x01,1509961563238.9070a4f2ace4516f985c6c9e31427eee. in 856ms,
sequenceid=41861, compaction requested=true (since 1mins, 45sec ago)
The region server wait for a long time and log show below :
2017-11-06 18:02:06,450 INFO
[B.defaultRpcServer.handler=27,queue=0,port=16020]
index.PhoenixIndexFailurePolicy: Successfully disabled index TEST_INDEX_LOCAL
due to an exception while writing updates.
org.apache.phoenix.hbase.index.exception.MultiIndexWriteFailureException:
Failed to write to multiple index tables
at
org.apache.phoenix.hbase.index.write.TrackingParallelWriterIndexCommitter.write(TrackingParallelWriterIndexCommitter.java:229)
at
org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWriter.java:193)
at
org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:154)
at
org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:143)
at
org.apache.phoenix.hbase.index.Indexer.doPostWithExceptions(Indexer.java:653)
at org.apache.phoenix.hbase.index.Indexer.doPost(Indexer.java:610)
at
org.apache.phoenix.hbase.index.Indexer.postBatchMutateIndispensably(Indexer.java:593)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$37.call(RegionCoprocessorHost.java:1034)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1673)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1749)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1705)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutateIndispensably(RegionCoprocessorHost.java:1030)
at
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3322)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2881)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2823)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:758)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:720)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2168)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33656)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2188)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
at java.lang.Thread.run(Thread.java:748)
2017-11-06 18:02:06,493 WARN [hconnection-0x52dee0d4-shared--pool423-t2]
client.AsyncProcess: #394, table=TEST_TABLE_LOCAL, attempt=11/11 failed=40ops,
last exception: org.apache.hadoop.hbase.RegionTooBusyException:
org.apache.hadoop.hbase.RegionTooBusyException: failed to get a lock in 60000
ms.
regionName=TEST_TABLE_LOCAL,\x01,1509961563238.9070a4f2ace4516f985c6c9e31427eee.,
server=ht06,16020,1509939923675
at org.apache.hadoop.hbase.regionserver.HRegion.lock(HRegion.java:8176)
at org.apache.hadoop.hbase.regionserver.HRegion.lock(HRegion.java:8162)
at
org.apache.hadoop.hbase.regionserver.HRegion.startRegionOperation(HRegion.java:8071)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2866)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2823)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:758)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:720)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2168)
at
org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:128)
The region will go back to normal after above exception,then table's local
index turn disable.
------------------ Original ------------------
From: "Ted Yu";<[email protected]>;
Send time: Saturday, Nov 4, 2017 0:09 AM
To: "[email protected]"<[email protected]>;
Subject: Re: local index turn disable when region split
Can you give us more information ?
Release of hbase and Phoenix you use
bq. The local index turn disable
Can you pastebin the related exception(s) ?
Snippet from region server log would also help.
On Thu, Nov 2, 2017 at 11:31 PM, vergil <[email protected]> wrote:
Hi,all:
Here is my test table.
create table test_table_local(id varchar primary key,f1 varchar,f2 varchar)
salt_buckets=3;
Add local index on it at the first time.
create local index test_index_local on test_table_local(f1);
Then upsert data into it.
As the data increase,the region will split.
The local index turn disable when the region splits.
Local index data do not increase and it do not work.
Here is my configuration on each master and regionserver.
<property> <name>hbase.regionserver.wal.codec</name>
<value>org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec</value>
</property> <property>
<name>hbase.region.server.rpc.scheduler.factory.class</name>
<value>org.apache.hadoop.hbase.ipc.PhoenixRpcSchedulerFactory</value>
</property> <property> <name>hbase.rpc.controllerfactory.class</name>
<value>org.apache.hadoop.hbase.ipc.controller.ServerRpcControllerFactory</value>
</property>
I need your help,thx!
------------------
The harder, more fortunate
