Hi,
Moving this over to the dev list (but still copying user). Another thing that might be interesting is to see what UPDATE STATISTICS was doing at the time of the failure. To do that, before you run UPDATE STATISTICS, turn logging on: Cqd ustat_log ‘<your file name here>’ ; Update statistics log on; <your update statistics command here> If it fails again, check the log in <your file name here> to see what UPDATE STATISTICS was doing. You may be able to reproduce the problem outside of UPDATE STATISTICS by just running the statement seen there. Dave *From:* Dave Birdsall [mailto:[email protected]] *Sent:* Thursday, March 24, 2016 2:58 PM *To:* '[email protected]' < [email protected]> *Subject:* RE: problem with update statistics Hi, I’ve seen this when the HMaster has died on a workstation using the standalone Hadoop. Perhaps you’d see the same behavior if a RegionServer died on a cluster, I don’t know. You may want to check your HBase logs to see if there are any clues there. Given that you’re developing a Trafodion feature, it is possible that something Trafodion did or said to HBase made it blush. J Dave *From:* Eric Owhadi [mailto:[email protected] <[email protected]>] *Sent:* Thursday, March 24, 2016 2:32 PM *To:* [email protected] *Subject:* problem with update statistics Hi Trafodioneers, I am trying to create a 10M row table, 1KB each row 10 partitions to test a new feature (parallelScan). But on my dev (so single node), I stumble on update statistics error: update statistics for table t132 on every column sample; *** ERROR[9200] UPDATE STATISTICS for table TRAFODION.SCH.T132 encountered an error (8448) from statement HSCursor::fetchRowset(). *** ERROR[8448] Unable to access Hbase interface. Call to ExpHbaseInterface::nextRow returned error HBASE_ACCESS_ERROR(-706). Cause: java.util.concurrent.ExecutionException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions: Thu Mar 24 21:01:52 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Call to edevxx.xxxxx.local/10.1.xx.xx:38868 failed on local exception: java.io.EOFException Thu Mar 24 21:02:11 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:02:30 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:02:51 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:03:12 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:03:33 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:04:03 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:04:32 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:05:00 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:05:27 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:06:04 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:06:42 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:07:20 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:08:00 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:08:38 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:09:18 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:09:56 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:10:34 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:11:13 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:11:52 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:12:33 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:13:11 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:13:51 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:14:31 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:15:08 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:15:47 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:16:29 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:17:07 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:17:45 GMT 2016, org.apache.hadoop.hbase.clien'��k *** ERROR[9200] UPDATE STATISTICS for table TRAFODION.SCH.T132 encountered an error (8448) from statement READ_COLS_INTO_MEM. *** ERROR[8448] Unable to access Hbase interface. Call to ExpHbaseInterface::nextRow returned error HBASE_ACCESS_ERROR(-706). Cause: java.util.concurrent.ExecutionException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions: Thu Mar 24 21:01:52 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Call to xxxxx.xxxxx.local/10.1.xx.xx:38868 failed on local exception: java.io.EOFException Thu Mar 24 21:02:11 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:02:30 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:02:51 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:03:12 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:03:33 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:04:03 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:04:32 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:05:00 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:05:27 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:06:04 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:06:42 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:07:20 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:08:00 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:08:38 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:09:18 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:09:56 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:10:34 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:11:13 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:11:52 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:12:33 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:13:11 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:13:51 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:14:31 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:15:08 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:15:47 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:16:29 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:17:07 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:17:45 GMT 2016, org.apache.hadoop.hbase.clien'��k The table: create table t132helper (a int not null, primary key(a)); insert into t132helper values(1); create table t132 (k1 int not null, k2 int not null, a int not null, b int not null, c char(1000), primary key (k1,k2)) SALT USING 10 PARTITIONS ON (k1, k2) ATTRIBUTES ALIGNED FORMAT ; upsert using load into t132 select x1000*1000+ x10000*10000 + x100000*100000 + x1000000*1000000, x1+x10*10+x100*100, x1+x10*10+x100*100+ x1000*1000+ x10000*10000 + x100000*100000 + x1000000*1000000, x1+x10*10+x100*100+ x1000*1000+ x10000*10000 + x100000*100000 + x1000000*1000000, 'yo bro' from t132helper transpose 0,1,2,3,4,5,6,7,8,9 as x1 transpose 0,1,2,3,4,5,6,7,8,9 as x10 transpose 0,1,2,3,4,5,6,7,8,9 as x100 transpose 0,1,2,3,4,5,6,7,8,9 as x1000 transpose 0,1,2,3,4,5,6,7,8,9 as x10000 transpose 0,1,2,3,4,5,6,7,8,9 as x100000 transpose 0,1,2,3,4,5,6,7,8,9 as x1000000; update statistics for table t132 on every column sample; is that a known problem? A workaround available? Eric
