[
https://issues.apache.org/jira/browse/HBASE-6295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elliott Clark updated HBASE-6295:
---------------------------------
Attachment: hbase-ycsb-workloads Build time trend.png
So the good news is this patch really helped our perf on YCSB.
Build #91 is where this patch went in.
The bad news is that the addendum didn't fix integration tests on a real
cluster. They still fail.
{code}
2013-06-28 13:08:32,882 DEBUG [hbase-table-pool-175-thread-2]
client.AsyncProcess: Attempt #10/14 failed for 1 operations on server
a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1,
tableName=IntegrationTestDataIngestSlowDeterministic,
location=region=IntegrationTestDataIngestSlowDeterministic,bbbbbbb0,1372449851893.b7040e878147caf1f6de338faad504be.,
hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=1, last
exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException:
org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not
online: b7040e878147caf1f6de338faad504be
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
at
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
- sleeping 3205 ms.
2013-06-28 13:08:32,939 DEBUG [hbase-table-pool-173-thread-1]
client.ClientScanner: Scan table=.META.,
startRow=IntegrationTestDataIngestSlowDeterministic,22222220,00000000000000
2013-06-28 13:08:32,945 DEBUG [hbase-table-pool-173-thread-2]
client.AsyncProcess: Attempt #10/14 failed for 1 operations on server
a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1,
tableName=IntegrationTestDataIngestSlowDeterministic,
location=region=IntegrationTestDataIngestSlowDeterministic,22222220,1372449851892.c8a2a77e0690901df245ac9fff088a0e.,
hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=1, last
exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException:
org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not
online: c8a2a77e0690901df245ac9fff088a0e
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
at
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
- sleeping 3201 ms.
2013-06-28 13:08:32,996 DEBUG [hbase-table-pool-174-thread-1]
client.ClientScanner: Scan table=.META.,
startRow=IntegrationTestDataIngestSlowDeterministic,bbbbbbb0,00000000000000
2013-06-28 13:08:33,030 DEBUG [hbase-table-pool-174-thread-1]
client.ClientScanner: Finished region={ENCODED => 1028785192, NAME =>
'.META.,,1', STARTKEY => '', ENDKEY => ''}
2013-06-28 13:08:33,032 DEBUG [hbase-table-pool-174-thread-2]
client.AsyncProcess: Attempt #10/14 failed for 1 operations on server
a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1,
tableName=IntegrationTestDataIngestSlowDeterministic,
location=region=IntegrationTestDataIngestSlowDeterministic,bbbbbbb0,1372449851893.b7040e878147caf1f6de338faad504be.,
hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=1, last
exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException:
org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not
online: b7040e878147caf1f6de338faad504be
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
at
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
- sleeping 3201 ms.
2013-06-28 13:08:33,371 DEBUG [hbase-table-pool-177-thread-2]
client.ClientScanner: Scan table=.META.,
startRow=IntegrationTestDataIngestSlowDeterministic,b3333328,00000000000000
2013-06-28 13:08:33,408 DEBUG [hbase-table-pool-177-thread-2]
client.ClientScanner: Finished region={ENCODED => 1028785192, NAME =>
'.META.,,1', STARTKEY => '', ENDKEY => ''}
2013-06-28 13:08:33,411 DEBUG [hbase-table-pool-171-thread-1]
client.AsyncProcess: Attempt #10/14 failed for 1 operations on server
a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1,
tableName=IntegrationTestDataIngestSlowDeterministic,
location=region=IntegrationTestDataIngestSlowDeterministic,b3333328,1372449851893.c6df99964212adf9c8d9807926371a57.,
hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=1, last
exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException:
org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not
online: c6df99964212adf9c8d9807926371a57
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
at
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
- sleeping 3215 ms.
2013-06-28 13:08:33,411 DEBUG [hbase-table-pool-177-thread-1]
client.AsyncProcess: Attempt #10/14 failed for 1 operations on server
a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1,
tableName=IntegrationTestDataIngestSlowDeterministic,
location=region=IntegrationTestDataIngestSlowDeterministic,b3333328,1372449851893.c6df99964212adf9c8d9807926371a57.,
hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=1, last
exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException:
org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not
online: c6df99964212adf9c8d9807926371a57
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
at
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
- sleeping 3212 ms.
2013-06-28 13:08:35,941 DEBUG [hbase-table-pool-180-thread-2]
client.AsyncProcess: Attempt #11/14 failed for 1 operations on server
a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1,
tableName=IntegrationTestDataIngestSlowDeterministic,
location=region=IntegrationTestDataIngestSlowDeterministic,bbbbbbb0,1372449851893.b7040e878147caf1f6de338faad504be.,
hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=1, last
exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException:
org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not
online: b7040e878147caf1f6de338faad504be
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
at
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
- sleeping 6460 ms.
2013-06-28 13:08:35,956 DEBUG [hbase-table-pool-178-thread-2]
client.AsyncProcess: Attempt #11/14 failed for 1 operations on server
a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1,
tableName=IntegrationTestDataIngestSlowDeterministic,
location=region=IntegrationTestDataIngestSlowDeterministic,3bbbbbb8,1372449851893.ebd0ab2aba6df7f8ed2ee5e47872dd36.,
hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=53019, last
exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException:
org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not
online: ebd0ab2aba6df7f8ed2ee5e47872dd36
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
at
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
- sleeping 6410 ms.
2013-06-28 13:08:35,968 DEBUG [hbase-table-pool-172-thread-2]
client.AsyncProcess: Attempt #11/14 failed for 1 operations on server
a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1,
tableName=IntegrationTestDataIngestSlowDeterministic,
location=region=IntegrationTestDataIngestSlowDeterministic,66666660,1372449851893.6910db29befeea246245b02808ac45ec.,
hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=1, last
exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException:
org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not
online: 6910db29befeea246245b02808ac45ec
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
at
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
- sleeping 6421 ms.
2013-06-28 13:08:36,000 DEBUG [hbase-table-pool-176-thread-2]
client.ClientScanner: Scan table=.META.,
startRow=IntegrationTestDataIngestSlowDeterministic,b3333328,00000000000000
2013-06-28 13:08:36,051 DEBUG [hbase-table-pool-176-thread-2]
client.ClientScanner: Finished region={ENCODED => 1028785192, NAME =>
'.META.,,1', STARTKEY => '', ENDKEY => ''}
2013-06-28 13:08:36,053 DEBUG [hbase-table-pool-176-thread-1]
client.AsyncProcess: Attempt #11/14 failed for 1 operations on server
a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1,
tableName=IntegrationTestDataIngestSlowDeterministic,
location=region=IntegrationTestDataIngestSlowDeterministic,b3333328,1372449851893.c6df99964212adf9c8d9807926371a57.,
hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=1, last
exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException:
org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not
online: c6df99964212adf9c8d9807926371a57
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
at
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
- sleeping 6456 ms.
2013-06-28 13:08:36,054 DEBUG [hbase-table-pool-179-thread-1]
client.ClientScanner: Scan table=.META.,
startRow=IntegrationTestDataIngestSlowDeterministic,22222220,00000000000000
2013-06-28 13:08:36,061 DEBUG [hbase-table-pool-179-thread-2]
client.AsyncProcess: Attempt #11/14 failed for 1 operations on server
a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1,
tableName=IntegrationTestDataIngestSlowDeterministic,
location=region=IntegrationTestDataIngestSlowDeterministic,22222220,1372449851892.c8a2a77e0690901df245ac9fff088a0e.,
hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=1, last
exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException:
org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not
online: c8a2a77e0690901df245ac9fff088a0e
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
at
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
- sleeping 6422 ms.
2013-06-28 13:08:36,088 DEBUG [hbase-table-pool-175-thread-1]
client.AsyncProcess: Attempt #11/14 failed for 1 operations on server
a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1,
tableName=IntegrationTestDataIngestSlowDeterministic,
location=region=IntegrationTestDataIngestSlowDeterministic,bbbbbbb0,1372449851893.b7040e878147caf1f6de338faad504be.,
hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=1, last
exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException:
org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not
online: b7040e878147caf1f6de338faad504be
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
at
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
- sleeping 6445 ms.
2013-06-28 13:08:36,150 DEBUG [hbase-table-pool-173-thread-2]
client.ClientScanner: Scan table=.META.,
startRow=IntegrationTestDataIngestSlowDeterministic,22222220,00000000000000
2013-06-28 13:08:36,156 DEBUG [hbase-table-pool-173-thread-1]
client.AsyncProcess: Attempt #11/14 failed for 1 operations on server
a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1,
tableName=IntegrationTestDataIngestSlowDeterministic,
location=region=IntegrationTestDataIngestSlowDeterministic,22222220,1372449851892.c8a2a77e0690901df245ac9fff088a0e.,
hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=1, last
exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException:
org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not
online: c8a2a77e0690901df245ac9fff088a0e
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
at
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
- sleeping 6459 ms.
2013-06-28 13:08:36,237 DEBUG [hbase-table-pool-174-thread-2]
client.ClientScanner: Scan table=.META.,
startRow=IntegrationTestDataIngestSlowDeterministic,bbbbbbb0,00000000000000
2013-06-28 13:08:36,273 DEBUG [hbase-table-pool-174-thread-2]
client.ClientScanner: Finished region={ENCODED => 1028785192, NAME =>
'.META.,,1', STARTKEY => '', ENDKEY => ''}
2013-06-28 13:08:36,276 DEBUG [hbase-table-pool-174-thread-1]
client.AsyncProcess: Attempt #11/14 failed for 1 operations on server
a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1,
tableName=IntegrationTestDataIngestSlowDeterministic,
location=region=IntegrationTestDataIngestSlowDeterministic,bbbbbbb0,1372449851893.b7040e878147caf1f6de338faad504be.,
hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=1, last
exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException:
org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not
online: b7040e878147caf1f6de338faad504be
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
at
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
- sleeping 6441 ms.
2013-06-28 13:08:36,627 DEBUG [hbase-table-pool-177-thread-1]
client.ClientScanner: Scan table=.META.,
startRow=IntegrationTestDataIngestSlowDeterministic,b3333328,00000000000000
2013-06-28 13:08:36,663 DEBUG [hbase-table-pool-177-thread-1]
client.ClientScanner: Finished region={ENCODED => 1028785192, NAME =>
'.META.,,1', STARTKEY => '', ENDKEY => ''}
2013-06-28 13:08:36,666 DEBUG [hbase-table-pool-171-thread-2]
client.AsyncProcess: Attempt #11/14 failed for 1 operations on server
a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1,
tableName=IntegrationTestDataIngestSlowDeterministic,
location=region=IntegrationTestDataIngestSlowDeterministic,b3333328,1372449851893.c6df99964212adf9c8d9807926371a57.,
hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=1, last
exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException:
org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not
online: c6df99964212adf9c8d9807926371a57
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
at
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
- sleeping 6415 ms.
2013-06-28 13:08:36,667 DEBUG [hbase-table-pool-177-thread-2]
client.AsyncProcess: Attempt #11/14 failed for 1 operations on server
a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1,
tableName=IntegrationTestDataIngestSlowDeterministic,
location=region=IntegrationTestDataIngestSlowDeterministic,b3333328,1372449851893.c6df99964212adf9c8d9807926371a57.,
hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=1, last
exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException:
org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not
online: c6df99964212adf9c8d9807926371a57
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
at
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
- sleeping 6453 ms.
2013-06-28 13:08:37,368 INFO [Thread-399] util.MultiThreadedAction: [W:10]
Keys=6575, cols=68.0 K, time=00:00:50 Overall: [keys/s= 131, latency=46 ms],
insertedUpTo=74999
2013-06-28 13:08:42,367 INFO [HBaseWriterThread_7] client.AsyncProcess: :
Waiting for the global number of running tasks to be equals or less than 0,
tasksSent=6934, tasksDone=6933, currentTasksDone=6933,
tableName=IntegrationTestDataIngestSlowDeterministic
2013-06-28 13:08:42,368 DEBUG [hbase-table-pool-178-thread-1]
client.AsyncProcess: Attempt #12/14 failed for 1 operations on server
a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1,
tableName=IntegrationTestDataIngestSlowDeterministic,
location=region=IntegrationTestDataIngestSlowDeterministic,3bbbbbb8,1372449851893.ebd0ab2aba6df7f8ed2ee5e47872dd36.,
hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=53019, last
exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException:
org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not
online: ebd0ab2aba6df7f8ed2ee5e47872dd36
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
at
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
- sleeping 6401 ms.
2013-06-28 13:08:42,369 INFO [Thread-399] util.MultiThreadedAction: [W:10]
Keys=6575, cols=68.0 K, time=00:00:55 Overall: [keys/s= 119, latency=46 ms],
insertedUpTo=74999
2013-06-28 13:08:42,389 INFO [HBaseWriterThread_1] client.AsyncProcess: :
Waiting for the global number of running tasks to be equals or less than 0,
tasksSent=6883, tasksDone=6882, currentTasksDone=6882,
tableName=IntegrationTestDataIngestSlowDeterministic
2013-06-28 13:08:42,390 DEBUG [hbase-table-pool-172-thread-1]
client.AsyncProcess: Attempt #12/14 failed for 1 operations on server
a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1,
tableName=IntegrationTestDataIngestSlowDeterministic,
location=region=IntegrationTestDataIngestSlowDeterministic,66666660,1372449851893.6910db29befeea246245b02808ac45ec.,
hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=1, last
exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException:
org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not
online: 6910db29befeea246245b02808ac45ec
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
at
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
- sleeping 6432 ms.
2013-06-28 13:08:42,401 INFO [HBaseWriterThread_9] client.AsyncProcess: :
Waiting for the global number of running tasks to be equals or less than 0,
tasksSent=6886, tasksDone=6885, currentTasksDone=6885,
tableName=IntegrationTestDataIngestSlowDeterministic
2013-06-28 13:08:42,486 DEBUG [hbase-table-pool-179-thread-2]
client.ClientScanner: Scan table=.META.,
startRow=IntegrationTestDataIngestSlowDeterministic,22222220,00000000000000
2013-06-28 13:08:42,491 INFO [HBaseWriterThread_8] client.AsyncProcess: :
Waiting for the global number of running tasks to be equals or less than 0,
tasksSent=7006, tasksDone=7005, currentTasksDone=7005,
tableName=IntegrationTestDataIngestSlowDeterministic
2013-06-28 13:08:42,497 DEBUG [HBaseWriterThread_9] client.ClientScanner: Scan
table=.META.,
startRow=IntegrationTestDataIngestSlowDeterministic,b3333328,00000000000000
2013-06-28 13:08:42,533 DEBUG [HBaseWriterThread_9] client.ClientScanner:
Finished region={ENCODED => 1028785192, NAME => '.META.,,1', STARTKEY => '',
ENDKEY => ''}
2013-06-28 13:08:42,534 INFO [HBaseWriterThread_4] client.AsyncProcess: :
Waiting for the global number of running tasks to be equals or less than 0,
tasksSent=7105, tasksDone=7104, currentTasksDone=7104,
tableName=IntegrationTestDataIngestSlowDeterministic
2013-06-28 13:08:42,534 INFO [HBaseWriterThread_5] client.AsyncProcess: :
Waiting for the global number of running tasks to be equals or less than 0,
tasksSent=6812, tasksDone=6811, currentTasksDone=6811,
tableName=IntegrationTestDataIngestSlowDeterministic
2013-06-28 13:08:42,615 INFO [HBaseWriterThread_2] client.AsyncProcess: :
Waiting for the global number of running tasks to be equals or less than 0,
tasksSent=7169, tasksDone=7168, currentTasksDone=7168,
tableName=IntegrationTestDataIngestSlowDeterministic
2013-06-28 13:08:42,739 INFO [HBaseWriterThread_3] client.AsyncProcess: :
Waiting for the global number of running tasks to be equals or less than 0,
tasksSent=7091, tasksDone=7090, currentTasksDone=7090,
tableName=IntegrationTestDataIngestSlowDeterministic
2013-06-28 13:08:43,081 INFO [HBaseWriterThread_0] client.AsyncProcess: :
Waiting for the global number of running tasks to be equals or less than 0,
tasksSent=6907, tasksDone=6906, currentTasksDone=6906,
tableName=IntegrationTestDataIngestSlowDeterministic
2013-06-28 13:08:43,120 INFO [HBaseWriterThread_6] client.AsyncProcess: :
Waiting for the global number of running tasks to be equals or less than 0,
tasksSent=7102, tasksDone=7101, currentTasksDone=7101,
tableName=IntegrationTestDataIngestSlowDeterministic
2013-06-28 13:08:47,370 INFO [Thread-399] util.MultiThreadedAction: [W:10]
Keys=8191, cols=84.6 K, time=00:01:00 Overall: [keys/s= 136, latency=55 ms]
Current: [keys/s=323, latency=91 ms], insertedUpTo=74999
2013-06-28 13:08:52,370 INFO [Thread-399] util.MultiThreadedAction: [W:10]
Keys=10203, cols=105.5 K, time=00:01:05 Overall: [keys/s= 156, latency=53 ms]
Current: [keys/s=402, latency=43 ms], insertedUpTo=74999
2013-06-28 13:08:57,371 INFO [Thread-399] util.MultiThreadedAction: [W:10]
Keys=12404, cols=127.8 K, time=00:01:10 Overall: [keys/s= 177, latency=47 ms]
Current: [keys/s=440, latency=22 ms], insertedUpTo=74999
2013-06-28 13:09:02,373 INFO [Thread-399] util.MultiThreadedAction: [W:10]
Keys=14290, cols=146.8 K, time=00:01:15 Overall: [keys/s= 190, latency=44 ms]
Current: [keys/s=377, latency=26 ms], insertedUpTo=74999
Failed to write keys: 4
Failed to write key: 81469
Failed to write key: 81503
Failed to write key: 81518
Failed to write key: 81524
2013-06-28 13:09:04,376 ERROR [main] hbase.IngestIntegrationTestBase: Load
failed with error code 1
2013-06-28 13:10:31,691 INFO [Thread-4] util.ChaosMonkey: Sleeping for 84975
to add jitter
2013-06-28 13:11:56,667 DEBUG [main] hbase.IngestIntegrationTestBase: Restoring
the cluster
2013-06-28 13:11:56,679 DEBUG [main] hbase.IngestIntegrationTestBase: Done
restoring the cluster
E
Time: 466.629
There was 1 failure:
1)
testDataIngest(org.apache.hadoop.hbase.IntegrationTestDataIngestSlowDeterministic)
junit.framework.AssertionFailedError: Load failed with error code 1
at junit.framework.Assert.fail(Assert.java:57)
at
org.apache.hadoop.hbase.IngestIntegrationTestBase.runIngestTest(IngestIntegrationTestBase.java:114)
at
org.apache.hadoop.hbase.IntegrationTestDataIngestSlowDeterministic.testDataIngest(IntegrationTestDataIngestSlowDeterministic.java:72)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.junit.runners.Suite.runChild(Suite.java:127)
at org.junit.runners.Suite.runChild(Suite.java:26)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
at org.junit.runner.JUnitCore.run(JUnitCore.java:138)
at org.junit.runner.JUnitCore.run(JUnitCore.java:117)
at
org.apache.hadoop.hbase.IntegrationTestsDriver.doWork(IntegrationTestsDriver.java:111)
at
org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:108)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
{code}
> Possible performance improvement in client batch operations: presplit and
> send in background
> --------------------------------------------------------------------------------------------
>
> Key: HBASE-6295
> URL: https://issues.apache.org/jira/browse/HBASE-6295
> Project: HBase
> Issue Type: Improvement
> Components: Client, Performance
> Affects Versions: 0.95.2
> Reporter: Nicolas Liochon
> Assignee: Nicolas Liochon
> Labels: noob
> Fix For: 0.98.0, 0.95.2
>
> Attachments: 6295.addendum.patch, 6295.v11.patch, 6295.v12.patch,
> 6295.v14.patch, 6295.v15.patch, 6295.v1.patch, 6295.v2.patch, 6295.v3.patch,
> 6295.v4.patch, 6295.v5.patch, 6295.v6.patch, 6295.v8.patch, 6295.v9.patch,
> hbase-ycsb-workloads Build time trend.png
>
>
> today batch algo is:
> {noformat}
> for Operation o: List<Op>{
> add o to todolist
> if todolist > maxsize or o last in list
> split todolist per location
> send split lists to region servers
> clear todolist
> wait
> }
> {noformat}
> We could:
> - create immediately the final object instead of an intermediate array
> - split per location immediately
> - instead of sending when the list as a whole is full, send it when there is
> enough data for a single location
> It would be:
> {noformat}
> for Operation o: List<Op>{
> get location
> add o to todo location.todolist
> if (location.todolist > maxLocationSize)
> send location.todolist to region server
> clear location.todolist
> // don't wait, continue the loop
> }
> send remaining
> wait
> {noformat}
> It's not trivial to write if you add error management: retried list must be
> shared with the operations added in the todolist. But it's doable.
> It's interesting mainly for 'big' writes
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
