[ https://issues.apache.org/jira/browse/HBASE-26027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Wang updated HBASE-26027: ------------------------------- Description: The batch api of HTable contains a param named results to store result or exception, its type is Object[]. If user pass an array with other type, eg: org.apache.hadoop.hbase.client.Result, and we need to put an exception into it by some reason, then the ArrayStoreException will occur in AsyncRequestFutureImpl.updateResult, then the AsyncRequestFutureImpl.decActionCounter will be skipped, then in the AsyncRequestFutureImpl.waitUntilDone we will stuck at here checking the actionsInProgress again and again, forever. It is better to add an cutoff calculated by operationTimeout, instead of only depend on the value of actionsInProgress. BTW, this issue only for 2.x, since 3.x the implement has refactored. {code:java} [ERROR] [2021/06/22 23:23:00,676] hconnection-0x6b927fb-shared-pool3-t1 - id=1 error for test processing localhost,16020,1624343786295 java.lang.ArrayStoreException: org.apache.hadoop.hbase.DoNotRetryIOException at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.updateResult(AsyncRequestFutureImpl.java:1242) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.trySetResultSimple(AsyncRequestFutureImpl.java:1087) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.setError(AsyncRequestFutureImpl.java:1021) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.manageError(AsyncRequestFutureImpl.java:683) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.receiveGlobalFailure(AsyncRequestFutureImpl.java:716) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.access$1500(AsyncRequestFutureImpl.java:69) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncRequestFutureImpl.java:219) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266) at java.util.concurrent.FutureTask.run(FutureTask.java) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) [INFO ] [2021/06/22 23:23:10,375] main - #1, waiting for 10 actions to finish on table: test [INFO ] [2021/06/22 23:23:20,378] main - #1, waiting for 10 actions to finish on table: test [INFO ] [2021/06/22 23:23:30,384] main - #1, waiting for 10 actions to finish on table: [INFO ] [2021/06/22 23:23:40,387] main - #1, waiting for 10 actions to finish on table: test [INFO ] [2021/06/22 23:23:50,397] main - #1, waiting for 10 actions to finish on table: test [INFO ] [2021/06/22 23:24:00,400] main - #1, waiting for 10 actions to finish on table: test [INFO ] [2021/06/22 23:24:10,408] main - #1, waiting for 10 actions to finish on table: test [INFO ] [2021/06/22 23:24:20,413] main - #1, waiting for 10 actions to finish on table: test {code} was: The batch api of HTable contains a param named results to store result or exception, its type is Object[]. If user pass an array with other type, eg: org.apache.hadoop.hbase.client.Result, then the ArrayStoreException will occur in AsyncRequestFutureImpl.updateResult, then the AsyncRequestFutureImpl.decActionCounter will be skipped, then in the AsyncRequestFutureImpl.waitUntilDone we will stuck at here checking the actionsInProgress again and again, forever. It is better to add an cutoff calculated by operationTimeout, instead of only depend on the value of actionsInProgress. BTW, this issue only for 2.x, since 3.x the implement has refactored. {code:java} [ERROR] [2021/06/22 23:23:00,676] hconnection-0x6b927fb-shared-pool3-t1 - id=1 error for test processing localhost,16020,1624343786295 java.lang.ArrayStoreException: org.apache.hadoop.hbase.DoNotRetryIOException at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.updateResult(AsyncRequestFutureImpl.java:1242) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.trySetResultSimple(AsyncRequestFutureImpl.java:1087) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.setError(AsyncRequestFutureImpl.java:1021) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.manageError(AsyncRequestFutureImpl.java:683) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.receiveGlobalFailure(AsyncRequestFutureImpl.java:716) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.access$1500(AsyncRequestFutureImpl.java:69) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncRequestFutureImpl.java:219) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266) at java.util.concurrent.FutureTask.run(FutureTask.java) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) [INFO ] [2021/06/22 23:23:10,375] main - #1, waiting for 10 actions to finish on table: test [INFO ] [2021/06/22 23:23:20,378] main - #1, waiting for 10 actions to finish on table: test [INFO ] [2021/06/22 23:23:30,384] main - #1, waiting for 10 actions to finish on table: [INFO ] [2021/06/22 23:23:40,387] main - #1, waiting for 10 actions to finish on table: test [INFO ] [2021/06/22 23:23:50,397] main - #1, waiting for 10 actions to finish on table: test [INFO ] [2021/06/22 23:24:00,400] main - #1, waiting for 10 actions to finish on table: test [INFO ] [2021/06/22 23:24:10,408] main - #1, waiting for 10 actions to finish on table: test [INFO ] [2021/06/22 23:24:20,413] main - #1, waiting for 10 actions to finish on table: test {code} > The calling of HTable.batch blocked at AsyncRequestFutureImpl.waitUntilDone > caused by ArrayStoreException > --------------------------------------------------------------------------------------------------------- > > Key: HBASE-26027 > URL: https://issues.apache.org/jira/browse/HBASE-26027 > Project: HBase > Issue Type: Bug > Components: Client > Affects Versions: 2.2.7, 2.3.5, 2.4.4 > Reporter: Zheng Wang > Assignee: Zheng Wang > Priority: Major > > The batch api of HTable contains a param named results to store result or > exception, its type is Object[]. > If user pass an array with other type, eg: > org.apache.hadoop.hbase.client.Result, and we need to put an exception into > it by some reason, then the ArrayStoreException will occur in > AsyncRequestFutureImpl.updateResult, then the > AsyncRequestFutureImpl.decActionCounter will be skipped, then in the > AsyncRequestFutureImpl.waitUntilDone we will stuck at here checking the > actionsInProgress again and again, forever. > It is better to add an cutoff calculated by operationTimeout, instead of only > depend on the value of actionsInProgress. > BTW, this issue only for 2.x, since 3.x the implement has refactored. > {code:java} > [ERROR] [2021/06/22 23:23:00,676] hconnection-0x6b927fb-shared-pool3-t1 - > id=1 error for test processing localhost,16020,1624343786295 > java.lang.ArrayStoreException: org.apache.hadoop.hbase.DoNotRetryIOException > at > org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.updateResult(AsyncRequestFutureImpl.java:1242) > at > org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.trySetResultSimple(AsyncRequestFutureImpl.java:1087) > at > org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.setError(AsyncRequestFutureImpl.java:1021) > at > org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.manageError(AsyncRequestFutureImpl.java:683) > at > org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.receiveGlobalFailure(AsyncRequestFutureImpl.java:716) > at > org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.access$1500(AsyncRequestFutureImpl.java:69) > at > org.apache.hadoop.hbase.client.AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncRequestFutureImpl.java:219) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266) > at java.util.concurrent.FutureTask.run(FutureTask.java) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > [INFO ] [2021/06/22 23:23:10,375] main - #1, waiting for 10 actions to > finish on table: test > [INFO ] [2021/06/22 23:23:20,378] main - #1, waiting for 10 actions to > finish on table: test > [INFO ] [2021/06/22 23:23:30,384] main - #1, waiting for 10 actions to > finish on table: > [INFO ] [2021/06/22 23:23:40,387] main - #1, waiting for 10 actions to > finish on table: test > [INFO ] [2021/06/22 23:23:50,397] main - #1, waiting for 10 actions to > finish on table: test > [INFO ] [2021/06/22 23:24:00,400] main - #1, waiting for 10 actions to > finish on table: test > [INFO ] [2021/06/22 23:24:10,408] main - #1, waiting for 10 actions to > finish on table: test > [INFO ] [2021/06/22 23:24:20,413] main - #1, waiting for 10 actions to > finish on table: test > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)