RE: Trafodion release2.0 Daily Test Result - 23 - Still Failing
Hi Steve,
The error today is this :
*** ERROR[8448] Unable to access Hbase interface. Call to
ExpHbaseInterface::scanOpen returned error HBASE_OPEN_ERROR(-704). Cause:
> java.lang.Exception: Cannot create Table Snapshot Scanner
> org.TRAFODION.sql.HTableClient.startScan(HTableClient.java:1003)
We have seen this when there is java memory pressure in the past.
A few days back this same snapshot scan creation failed with this : I
wonder if anyone can see pattern here or knows the causes of either of
these.
>>--snapshot
>>execute snp;
*** ERROR[8448] Unable to access Hbase interface. Call to
ExpHbaseInterface::scanOpen returned error HBASE_OPEN_ERROR(-704). Cause:
java.io.IOException: java.util.concurrent.ExecutionException:
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File
/bulkload/20160520102824/TRAFODION.HBASE.CUSTOMER_ADDRESS_SNAP111/6695c6f9-4bb5-4ad5-893b-adf07fc8a4b9/data/default/TRAFODION.HBASE.CUSTOMER_ADDRESS/7143c21b40a7bef21768685f7dc18e1c/.regioninfo
could only be replicated to 0 nodes instead of minReplication (=1). There
are 1 datanode(s) running and no node(s) are excluded in this operation.
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1541)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3289)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:668)
at
org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.addBlock(AuthorizationProviderProxyClientProtocol.java:212)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:483)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038)
org.apache.hadoop.hbase.util.ModifyRegionUtils.createRegions(ModifyRegionUtils.java:162)
org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.cloneHdfsRegions(RestoreSnapshotHelper.java:561)
org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:237)
org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:159)
org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.copySnapshotForScanner(RestoreSnapshotHelper.java:812)
org.apache.hadoop.hbase.client.TableSnapshotScanner.init(TableSnapshotScanner.java:156)
org.apache.hadoop.hbase.client.TableSnapshotScanner.<init>(TableSnapshotScanner.java:124)
org.apache.hadoop.hbase.client.TableSnapshotScanner.<init>(TableSnapshotScanner.java:101)
org.trafodion.sql.HTableClient$SnapshotScanHelper.createTableSnapshotScanner(HTableClient.java:222)
org.trafodion.sql.HTableClient.startScan(HTableClient.java:1009)
.
--- 0 row(s) selected.
>>log;
Sandhya
-----Original Message-----
From: Steve Varnau [mailto:[email protected] <[email protected]>]
Sent: Thursday, May 26, 2016 8:49 AM
To: [email protected]
Subject: RE: Trafodion release2.0 Daily Test Result - 23 - Still Failing
This hive regression behavior is still puzzling, however, I just realized
one thing that did change just before it started failing and is a test
environment change common to both branches. The VM image for cloudera was
updated to set a smaller ephemeral port range to reduce chance of port
conflict that was occasionally impacting HBase.
The range was set to 51000 - 59999, to avoid default port numbers that
Cloudera distro uses.
So how could this possibly be causing disaster in hive/TEST018? I have no
idea.
--Steve
> -----Original Message-----
> From: [email protected] [mailto:[email protected]
<[email protected]>]
> Sent: Thursday, May 26, 2016 1:36 AM
> To: [email protected]
> Subject: Trafodion release2.0 Daily Test Result - 23 - Still Failing
>
> Daily Automated Testing release2.0
>
> Jenkins Job: https://jenkins.esgyn.com/job/Check-Daily-release2.0/23/
> Archived Logs: http://traf-testlogs.esgyn.com/Daily-release2.0/23
> Bld Downloads: http://traf-builds.esgyn.com
>
> Changes since previous daily build:
> No changes
>
>
> Test Job Results:
>
> FAILURE core-regress-hive-cdh (55 min) SUCCESS build-release2.0-debug
> (24 min) SUCCESS build-release2.0-release (28 min) SUCCESS
> core-regress-charsets-cdh (28 min) SUCCESS core-regress-charsets-hdp
> (41 min) SUCCESS core-regress-compGeneral-cdh (36 min) SUCCESS
> core-regress-compGeneral-hdp (45 min) SUCCESS core-regress-core-cdh
> (39 min) SUCCESS core-regress-core-hdp (1 hr 10 min) SUCCESS
> core-regress-executor-cdh (56 min) SUCCESS core-regress-executor-hdp
> (1 hr 25 min) SUCCESS core-regress-fullstack2-cdh (13 min) SUCCESS
> core-regress-fullstack2-hdp (14 min) SUCCESS core-regress-hive-hdp (53
> min) SUCCESS core-regress-privs1-cdh (39 min) SUCCESS
> core-regress-privs1-hdp (59 min) SUCCESS core-regress-privs2-cdh (41
> min) SUCCESS core-regress-privs2-hdp (54 min) SUCCESS
> core-regress-qat-cdh (16 min) SUCCESS core-regress-qat-hdp (21 min)
> SUCCESS core-regress-seabase-cdh (57 min) SUCCESS
> core-regress-seabase-hdp (1 hr 16 min) SUCCESS core-regress-udr-cdh
> (28 min) SUCCESS core-regress-udr-hdp (31 min) SUCCESS jdbc_test-cdh
> (22 min) SUCCESS jdbc_test-hdp (40 min) SUCCESS phoenix_part1_T2-cdh
> (56 min) SUCCESS phoenix_part1_T2-hdp (1 hr 17 min) SUCCESS
> phoenix_part1_T4-cdh (46 min) SUCCESS phoenix_part1_T4-hdp (57 min)
> SUCCESS phoenix_part2_T2-cdh (53 min) SUCCESS phoenix_part2_T2-hdp (1
> hr 25 min) SUCCESS phoenix_part2_T4-cdh (44 min) SUCCESS
> phoenix_part2_T4-hdp (1 hr 0 min) SUCCESS pyodbc_test-cdh (11 min)
> SUCCESS pyodbc_test-hdp (23 min)
