Hi fangzhen, It seems like this thread is waiting for HBase coprocessor's result. Are you facing region split or something else at that time? Some HBase log maybe helpful for diagnose. I hope this can help, not sure whether I'm right.
Cheers, Yuhao. 2016-11-09 19:44 GMT+08:00 方震(大数据&滴滴研究院) <[email protected]>: > Guys, > > We met one issue: we have a table which has more than 100M rows, sometimes > it blocks at: > > hread 3303: (state = BLOCKED) > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information > may be imprecise) > - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, > line=175 (Compiled frame) > - java.util.concurrent.FutureTask.awaitDone(boolean, long) @bci=165, > line=429 (Compiled frame) > - java.util.concurrent.FutureTask.get() @bci=13, line=191 (Compiled > frame) > - org.apache.hadoop.hbase.client.HTable.coprocessorService(java.lang.Class, > byte[], byte[], org.apache.hadoop.hbase.client.coprocessor.Batch$Call, > org.apache.hadoop.hbase.client.coprocessor.Batch$Callback) @bci=159, > line=1732 (Compiled frame) > - org.apache.hadoop.hbase.client.HTable.coprocessorService(java.lang.Class, > byte[], byte[], org.apache.hadoop.hbase.client.coprocessor.Batch$Call) > @bci=31, line=1688 (Compiled frame) > - org.apache.phoenix.query.ConnectionQueryServicesImpl. > metaDataCoprocessorExec(byte[], > org.apache.hadoop.hbase.client.coprocessor.Batch$Call, > byte[]) @bci=38, line=1034 (Compiled frame) > - org.apache.phoenix.query.ConnectionQueryServicesImpl. > metaDataCoprocessorExec(byte[], > org.apache.hadoop.hbase.client.coprocessor.Batch$Call) > @bci=6, line=1014 (Compiled frame) > - org.apache.phoenix.query.ConnectionQueryServicesImpl. > getTable(org.apache.phoenix.schema.PName, byte[], byte[], long, long) > @bci=46, line=1287 (Compiled frame) > - org.apache.phoenix.schema.MetaDataClient.updateCache( > org.apache.phoenix.schema.PName, java.lang.String, java.lang.String, > boolean) @bci=167, line=417 (Compiled frame) > - org.apache.phoenix.schema.MetaDataClient.updateCache(java.lang.String, > java.lang.String, boolean) @bci=11, line=360 (Compiled frame) > - org.apache.phoenix.schema.MetaDataClient.updateCache(java.lang.String, > java.lang.String) @bci=4, line=356 (Compiled frame) > - org.apache.phoenix.compile.FromCompiler$BaseColumnResolver. > createTableRef(org.apache.phoenix.parse.NamedTableNode, boolean) @bci=63, > line=409 (Compiled frame) > - org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.< > init>(org.apache.phoenix.jdbc.PhoenixConnection, > org.apache.phoenix.parse.NamedTableNode, > boolean, int, java.util.Map) @bci=22, line=284 (Compiled frame) > - org.apache.phoenix.compile.FromCompiler.getResolverForQuery(org. > apache.phoenix.parse.SelectStatement, > org.apache.phoenix.jdbc.PhoenixConnection) > @bci=35, line=185 (Compiled frame) > > > Since it is not an easy reproduced issue, it is difficult for us to > troubleshoot. > > We found an old jira https://issues.apache.org/jira/browse/PHOENIX-1515 - > which says some select has slow response when calling the > coprocessorService. This jira marks “Resolved”, but we did not found any > commit there. Could someone gives us some clues on it. > > rgds > mac >
