Corrosponding HBase bug: https://issues.apache.org/jira/browse/HBASE-12629
On Wed, Nov 23, 2016 at 1:55 PM, Mukesh Jha <me.mukesh....@gmail.com> wrote: > The solution is to disable region size caluculation check. > > hbase.regionsizecalculator.enable: false > > On Sun, Nov 20, 2016 at 9:29 PM, Mukesh Jha <me.mukesh....@gmail.com> > wrote: > >> Any ideas folks? >> >> On Fri, Nov 18, 2016 at 3:37 PM, Mukesh Jha <me.mukesh....@gmail.com> >> wrote: >> >>> Hi >>> >>> I'm accessing multiple regions (~5k) of an HBase table using spark's >>> newAPIHadoopRDD. But the driver is trying to calculate the region size >>> of all the regions. >>> It is not even reusing the hconnection and creting a new connection for >>> every request (see below) which is taking lots of time. >>> >>> Is there a better approach to do this? >>> >>> >>> 8 Nov 2016 22:25:22,759] [INFO Driver] RecoverableZooKeeper: Process >>> identifier=*hconnection-0x1e7824af* connecting to ZooKeeper ensemble= >>> hbase19.cloud.com:2181,hbase24.cloud.com:2181,hbase28.cloud.com:2181 >>> [18 Nov 2016 22:25:22,759] [INFO Driver] ZooKeeper: Initiating client >>> connection, connectString=hbase19.cloud.com:2181,hbase24.cloud.com:2181, >>> hbase28.cloud.com:2181 sessionTimeout=60000 >>> watcher=hconnection-0x1e7824af0x0, quorum=hbase19.cloud.com:2181, >>> hbase24.cloud.com:2181,hbase28.cloud.com:2181, baseZNode=/hbase >>> [18 Nov 2016 22:25:22,761] [INFO Driver-SendThread(hbase24.clou >>> d.com:2181)] ClientCnxn: Opening socket connection to server >>> hbase24.cloud.com/10.193.150.217:2181. Will not attempt to authenticate >>> using SASL (unknown error) >>> [18 Nov 2016 22:25:22,763] [INFO Driver-SendThread(hbase24.clou >>> d.com:2181)] ClientCnxn: Socket connection established, initiating >>> session, client: /10.193.138.145:47891, server: >>> hbase24.cloud.com/10.193.150.217:2181 >>> [18 Nov 2016 22:25:22,766] [INFO Driver-SendThread(hbase24.clou >>> d.com:2181)] ClientCnxn: Session establishment complete on server >>> hbase24.cloud.com/10.193.150.217:2181, sessionid = 0x2564f6f013e0e95, >>> negotiated timeout = 60000 >>> [18 Nov 2016 22:25:22,766] [INFO Driver] RegionSizeCalculator: >>> Calculating region sizes for table "message". >>> [18 Nov 2016 22:25:27,867] [INFO Driver] >>> ConnectionManager$HConnectionImplementation: >>> Closing master protocol: MasterService >>> [18 Nov 2016 22:25:27,868] [INFO Driver] >>> ConnectionManager$HConnectionImplementation: >>> Closing zookeeper sessionid=0x2564f6f013e0e95 >>> [18 Nov 2016 22:25:27,869] [INFO Driver] ZooKeeper: Session: >>> 0x2564f6f013e0e95 closed >>> [18 Nov 2016 22:25:27,869] [INFO Driver-EventThread] ClientCnxn: >>> EventThread shut down >>> [18 Nov 2016 22:25:27,880] [INFO Driver] RecoverableZooKeeper: Process >>> identifier=*hconnection-0x6a8a1efa* connecting to ZooKeeper ensemble= >>> hbase19.cloud.com:2181,hbase24.cloud.com:2181,hbase28.cloud.com:2181 >>> [18 Nov 2016 22:25:27,880] [INFO Driver] ZooKeeper: Initiating client >>> connection, connectString=hbase19.cloud.com:2181,hbase24.cloud.com:2181, >>> hbase28.cloud.com:2181 sessionTimeout=60000 >>> watcher=hconnection-0x6a8a1efa0x0, quorum=hbase19.cloud.com:2181, >>> hbase24.cloud.com:2181,hbase28.cloud.com:2181, baseZNode=/hbase >>> [18 Nov 2016 22:25:27,883] [INFO Driver-SendThread(hbase24.clou >>> d.com:2181)] ClientCnxn: Opening socket connection to server >>> hbase24.cloud.com/10.193.150.217:2181. Will not attempt to authenticate >>> using SASL (unknown error) >>> [18 Nov 2016 22:25:27,885] [INFO Driver-SendThread(hbase24.clou >>> d.com:2181)] ClientCnxn: Socket connection established, initiating >>> session, client: /10.193.138.145:47894, server: >>> hbase24.cloud.com/10.193.150.217:2181 >>> [18 Nov 2016 22:25:27,887] [INFO Driver-SendThread(hbase24.clou >>> d.com:2181)] ClientCnxn: Session establishment complete on server >>> hbase24.cloud.com/10.193.150.217:2181, sessionid = 0x2564f6f013e0e97, >>> negotiated timeout = 60000 >>> [18 Nov 2016 22:25:27,888] [INFO Driver] RegionSizeCalculator: >>> Calculating region sizes for table "message". >>> .... >>> >>> -- >>> Thanks & Regards, >>> >>> *Mukesh Jha <me.mukesh....@gmail.com>* >>> >> >> >> >> -- >> >> >> Thanks & Regards, >> >> *Mukesh Jha <me.mukesh....@gmail.com>* >> > > > > -- > > > Thanks & Regards, > > *Mukesh Jha <me.mukesh....@gmail.com>* > -- Thanks & Regards, *Mukesh Jha <me.mukesh....@gmail.com>*
