Guys, We met one issue: we have a table which has more than 100M rows, sometimes it blocks at:
hread 3303: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame) - java.util.concurrent.FutureTask.awaitDone(boolean, long) @bci=165, line=429 (Compiled frame) - java.util.concurrent.FutureTask.get() @bci=13, line=191 (Compiled frame) - org.apache.hadoop.hbase.client.HTable.coprocessorService(java.lang.Class, byte[], byte[], org.apache.hadoop.hbase.client.coprocessor.Batch$Call, org.apache.hadoop.hbase.client.coprocessor.Batch$Callback) @bci=159, line=1732 (Compiled frame) - org.apache.hadoop.hbase.client.HTable.coprocessorService(java.lang.Class, byte[], byte[], org.apache.hadoop.hbase.client.coprocessor.Batch$Call) @bci=31, line=1688 (Compiled frame) - org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(byte[], org.apache.hadoop.hbase.client.coprocessor.Batch$Call, byte[]) @bci=38, line=1034 (Compiled frame) - org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(byte[], org.apache.hadoop.hbase.client.coprocessor.Batch$Call) @bci=6, line=1014 (Compiled frame) - org.apache.phoenix.query.ConnectionQueryServicesImpl.getTable(org.apache.phoenix.schema.PName, byte[], byte[], long, long) @bci=46, line=1287 (Compiled frame) - org.apache.phoenix.schema.MetaDataClient.updateCache(org.apache.phoenix.schema.PName, java.lang.String, java.lang.String, boolean) @bci=167, line=417 (Compiled frame) - org.apache.phoenix.schema.MetaDataClient.updateCache(java.lang.String, java.lang.String, boolean) @bci=11, line=360 (Compiled frame) - org.apache.phoenix.schema.MetaDataClient.updateCache(java.lang.String, java.lang.String) @bci=4, line=356 (Compiled frame) - org.apache.phoenix.compile.FromCompiler$BaseColumnResolver.createTableRef(org.apache.phoenix.parse.NamedTableNode, boolean) @bci=63, line=409 (Compiled frame) - org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.<init>(org.apache.phoenix.jdbc.PhoenixConnection, org.apache.phoenix.parse.NamedTableNode, boolean, int, java.util.Map) @bci=22, line=284 (Compiled frame) - org.apache.phoenix.compile.FromCompiler.getResolverForQuery(org.apache.phoenix.parse.SelectStatement, org.apache.phoenix.jdbc.PhoenixConnection) @bci=35, line=185 (Compiled frame) Since it is not an easy reproduced issue, it is difficult for us to troubleshoot. We found an old jira https://issues.apache.org/jira/browse/PHOENIX-1515 - which says some select has slow response when calling the coprocessorService. This jira marks “Resolved”, but we did not found any commit there. Could someone gives us some clues on it. rgds mac
