The exceptions seem to be another problem. They all happened on one node. And after the task attempts failed at that node, retried on other nodes and no exceptions. So that, the exception maybe have nothing to do with the performance issue.
On Wed, Nov 7, 2012 at 11:07 AM, Cheng Su <scarcer...@gmail.com> wrote: > Hi, all. > > I have a hive + hbase integration cluster. I met a performance issue. > I executed some complicated hql, but only the first one is actually running. > The rests are showed "running" on the job track web ui, but task > attempt failed with exceptions below: > > java.io.IOException: java.io.IOException: > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@5117a20 > closed > at > org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97) > at > org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57) > at > org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:243) > at > org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:522) > at > org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:197) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) > at org.apache.hadoop.mapred.Child$4.run(Child.java:255) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Unknown Source) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) > at org.apache.hadoop.mapred.Child.main(Child.java:249) > Caused by: java.io.IOException: > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@5117a20 > closed > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:794) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:782) > at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:249) > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:213) > at > org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat.getRecordReader(HiveHBaseTableInputFormat.java:92) > at > org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:240) > ... 9 more > > So actually the rests are queuing. > > I guess it might be because the count(?) of hbase handler is too > small, which handle access request from hive. > So that when a big job occupied all handles the rests have to wait > until the handlers are released. > > Is my assumption right? and what settings should I tuning? > > Thanks. > > -- > > Regards, > Cheng Su -- Regards, Cheng Su
