[
https://issues.apache.org/jira/browse/HBASE-15104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15098492#comment-15098492
]
Jonathan Hsieh commented on HBASE-15104:
----------------------------------------
On 1.3 - there's these recent failures. [1][3]. It is definitely related. I
see this in here [2][4]
{code}
016-01-14 06:31:46,480 ERROR [RS_OPEN_REGION-asf900:40489-2]
handler.OpenRegionHandler(385): Failed open of
region=TestAcidGuarantees,,1452752899465.31ca9689983f4ee6fc51fba6cd695fa0.,
starting to roll back the global memstore size.
org.apache.hadoop.hbase.DoNotRetryIOException: Compression algorithm 'lzo'
previously failed test.
at
org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:91)
at
org.apache.hadoop.hbase.regionserver.HRegion.checkCompressionCodecs(HRegion.java:6561)
at org.
{code}
I see a similar thing in the 1.2 builds but with the 'snappy' or 'lz4' codecs.
{code}
org.apache.hadoop.hbase.DoNotRetryIOException: Compression algorithm 'lz4'
previously failed test.
at
org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:91)
at
org.apache.hadoop.hbase.regionserver.HRegion.checkCompressionCodecs(HRegion.java:6494)
{code}
[1]
https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.3-IT/it.test=IntegrationTestAcidGuarantees,jdk=latest1.8,label=Hadoop/435/consoleFull
[2]
https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.3-IT/it.test=IntegrationTestAcidGuarantees,jdk=latest1.8,label=Hadoop/435/artifact/hbase-it/target/failsafe-reports/org.apache.hadoop.hbase.IntegrationTestAcidGuarantees-output.txt
[3]
https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.3-IT/434/it.test=IntegrationTestBigLinkedList,jdk=latest1.7,label=Hadoop/
[4]
https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.3-IT/434/it.test=IntegrationTestBigLinkedList,jdk=latest1.7,label=Hadoop/artifact/hbase-it/target/failsafe-reports/org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList-output.txt
> Occasional failures due to NotServingRegionException in IT tests
> ----------------------------------------------------------------
>
> Key: HBASE-15104
> URL: https://issues.apache.org/jira/browse/HBASE-15104
> Project: HBase
> Issue Type: Bug
> Components: integration tests
> Affects Versions: 1.2.0
> Reporter: huaxiang sun
> Assignee: huaxiang sun
> Fix For: 2.0.0, 0.94.28, 1.2.0, 1.1.3, 1.0.4
>
> Attachments: HBASE-15104-v001.patch
>
>
> IntegrationTestAcidGuarantees fails when trying to cleanup with
> NotServerRegionExceptions giving up (after 36 attempts) .
> 5/11/09 09:19:24 INFO client.AsyncProcess: #33, waiting for some tasks to
> finish. Expected max=0, tasksInProgress=9
> 15/11/09 09:19:33 INFO client.AsyncProcess: #45, table=TestAcidGuarantees,
> attempt=10/35 failed=1ops, last exception:
> org.apache.hadoop.hbase.NotServingRegionException:
> org.apache.hadoop.hbase.NotServingRegionException: Region
> TestAcidGuarantees,test_row_1,1447089367019.032439ef4f3353cb894d20337ba043bc.
> is not online on node-4.internal,22101,1447089152259
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2786)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:922)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:1893)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2035)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> at java.lang.Thread.run(Thread.java:745)
> ...
> Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed
> after attempts=36, exceptions:
> Mon Nov 09 09:19:53 PST 2015, null, java.net.SocketTimeoutException:
> callTimeout=60000, callDuration=68104: row 'test_row_1'
> Looked at the RS log, the following exception is found:
> 2015-11-10 10:07:49,091 ERROR
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open
> of
> region=TestAcidGuarantees,,1447177733243.f1be6b850fe3958c5c9b5e330b5dfb00.,
> starting to roll back the global memstore size.
> org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.RuntimeException:
> java.lang.ClassNotFoundException: com.hadoop.compression.lzo.LzoCodec
> at
> org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:102)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkCompressionCodecs(HRegion.java:6011)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5995)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5967)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5938)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5894)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5845)
> at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:356)
> at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:126)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
