What version of HBase are you running?
On Tue, Jul 5, 2011 at 1:45 AM, praveenesh kumar <praveen...@gmail.com> wrote:
> Hi,
> I have a latest pig build from trunk. I have configured it to run on 12-node
> hadoop cluster.
> I am trying to access a hbase table, the map job is running fine for
> sometime...but after some time.. I am getting zookeeper exception
>
> hadoop@ub13:/usr/local/hadoop/hbase/conf$ pig
> log4j:WARN No appenders could be found for logger
> (org.apache.hadoop.conf.Configuration).
> log4j:WARN Please initialize the log4j system properly.
> 2011-07-05 12:40:23,737 [main] INFO org.apache.pig.Main - Logging error
> messages to: /usr/local/hadoop/hbase/conf/pig_1309849823721.log
> 2011-07-05 12:40:24,047 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
> to hadoop file system at: hdfs://ub13:54310
> 2011-07-05 12:40:24,241 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
> to map-reduce job tracker at: ub13:54311
>
> grunt> ht = load 'hbase://tweetinfo' using
> org.apache.pig.backend.hadoop.hbase.HBaseStorage('prim_tweetinfo:name',
> '-loadKey') as (id,name);
> grunt> dump ht | HEAD 100;
>
> 2011-07-05 12:40:38,234 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script:
> UNKNOWN
> 2011-07-05 12:40:38,481 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler -
> File concatenation threshold: 100 optimistic? false
> 2011-07-05 12:40:38,513 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
> - MR plan size before optimization: 1
> 2011-07-05 12:40:38,514 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
> - MR plan size after optimization: 1
> 2011-07-05 12:40:38,559 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to
> the job
> 2011-07-05 12:40:38,572 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
> 2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:zookeeper.version=3.3.2-1031432, built on 11/05/2010 05:32 GMT
> 2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:host.name=ub13
> 2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:java.version=1.6.0_20
> 2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:java.vendor=Sun Microsystems Inc.
> 2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:java.home=/usr/lib/jvm/java-6-openjdk/jre
> 2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:java.class.path=/usr/local/hadoop/pig-0.10/bin/../conf:/usr/lib/jvm/java-6-openjdk/lib/tools.jar:/usr/local/hadoop/pig-0.10/bin/../build/classes:/usr/local/hadoop/pig-0.10/bin/../build/test/classes:/usr/local/hadoop/pig-0.10/bin/../pig-*-core.jar:/usr/local/hadoop/pig-0.10/bin/../build/pig-0.10.0-SNAPSHOT.jar:/usr/local/hadoop/pig-0.10/bin/../lib/automaton.jar:/usr/local/hadoop/hadoop/hadoop-0.20-append-for-hbase-core.jar:/usr/local/hadoop/hbase/hbase-0.90.3.jar:/usr/local/hadoop/hbase/hbase-0.90.3-tests.jar:/usr/local/hadoop/hbase/lib/zookeeper-3.3.2.jar:/usr/local/hadoop/hadoop:/usr/local/hadoop/hadoop/conf
> 2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:java.library.path=/usr/lib/jvm/java-6-openjdk/jre/lib/i386/server:/usr/lib/jvm/java-6-openjdk/jre/lib/i386:/usr/lib/jvm/java-6-openjdk/jre/../lib/i386:/usr/java/packages/lib/i386:/usr/lib/jni:/lib:/usr/lib
> 2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:java.io.tmpdir=/tmp
> 2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:java.compiler=<NA>
> 2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:os.name=Linux
> 2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:os.arch=i386
> 2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:os.version=2.6.35-22-generic-pae
> 2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:user.name=hadoop
> 2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:user.home=/home/hadoop
> 2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:user.dir=/usr/local/hadoop/hbase/conf
> 2011-07-05 12:40:38,739 [main] INFO org.apache.zookeeper.ZooKeeper -
> Initiating client connection, connectString=localhost:2181
> sessionTimeout=180000 watcher=hconnection
> 2011-07-05 12:40:38,791 [main-SendThread()] INFO
> org.apache.zookeeper.ClientCnxn - Opening socket connection to server
> localhost/127.0.0.1:2181
> 2011-07-05 12:40:38,814 [main-SendThread(localhost:2181)] INFO
> org.apache.zookeeper.ClientCnxn - Socket connection established to
> localhost/127.0.0.1:2181, initiating session
> 2011-07-05 12:40:38,833 [main-SendThread(localhost:2181)] INFO
> org.apache.zookeeper.ClientCnxn - Session establishment complete on server
> localhost/127.0.0.1:2181, sessionid = 0x30f91a0a2c0009, negotiated timeout =
> 180000
> 2011-07-05 12:40:43,484 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> - Setting up single store job
> 2011-07-05 12:40:43,529 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 1 map-reduce job(s) waiting for submission.
> 2011-07-05 12:40:44,030 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 0% complete
> 2011-07-05 12:40:44,499 [Thread-13] INFO org.apache.zookeeper.ZooKeeper -
> Initiating client connection, connectString=localhost:2181
> sessionTimeout=180000 watcher=hconnection
> 2011-07-05 12:40:44,500 [Thread-13-SendThread()] INFO
> org.apache.zookeeper.ClientCnxn - Opening socket connection to server
> localhost/127.0.0.1:2181
> 2011-07-05 12:40:44,501 [Thread-13-SendThread(localhost:2181)] INFO
> org.apache.zookeeper.ClientCnxn - Socket connection established to
> localhost/127.0.0.1:2181, initiating session
> 2011-07-05 12:40:44,512 [Thread-13-SendThread(localhost:2181)] INFO
> org.apache.zookeeper.ClientCnxn - Session establishment complete on server
> localhost/127.0.0.1:2181, sessionid = 0x30f91a0a2c000a, negotiated timeout =
> 180000
> 2011-07-05 12:40:44,554 [Thread-13] INFO org.apache.zookeeper.ZooKeeper -
> Initiating client connection, connectString=localhost:2181
> sessionTimeout=180000 watcher=hconnection
> 2011-07-05 12:40:44,556 [Thread-13-SendThread()] INFO
> org.apache.zookeeper.ClientCnxn - Opening socket connection to server
> localhost/127.0.0.1:2181
> 2011-07-05 12:40:44,558 [Thread-13-SendThread(localhost:2181)] INFO
> org.apache.zookeeper.ClientCnxn - Socket connection established to
> localhost/127.0.0.1:2181, initiating session
> 2011-07-05 12:40:44,569 [Thread-13-SendThread(localhost:2181)] INFO
> org.apache.zookeeper.ClientCnxn - Session establishment complete on server
> localhost/127.0.0.1:2181, sessionid = 0x30f91a0a2c000b, negotiated timeout =
> 180000
> 2011-07-05 12:40:45,964 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - HadoopJobId: job_201107051228_0001
> 2011-07-05 12:40:45,964 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - More information at:
> http://ub13:50030/jobdetails.jsp?jobid=job_201107051228_0001
> 2011-07-05 12:41:04,624 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 1% complete
> 2011-07-05 12:41:07,638 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 3% complete
> 2011-07-05 12:41:08,141 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 4% complete
> 2011-07-05 12:41:10,652 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 7% complete
> 2011-07-05 12:41:11,155 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 9% complete
> 2011-07-05 12:41:16,681 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 12% complete
> 2011-07-05 12:41:17,183 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 13% complete
> 2011-07-05 12:41:19,933 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 15% complete
> 2011-07-05 12:41:22,444 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 18% complete
> 2011-07-05 12:41:22,947 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 19% complete
> 2011-07-05 12:41:26,463 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 21% complete
> 2011-07-05 12:41:28,473 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 24% complete
> 2011-07-05 12:41:29,478 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 25% complete
> 2011-07-05 12:41:31,488 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 27% complete
> 2011-07-05 12:41:37,516 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 28% complete
> 2011-07-05 12:41:43,544 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 30% complete
> 2011-07-05 12:41:56,124 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 31% complete
> 2011-07-05 12:41:58,635 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 33% complete
> 2011-07-05 12:41:59,140 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 34% complete
> 2011-07-05 12:42:02,155 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 36% complete
> 2011-07-05 12:42:10,700 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - job job_201107051228_0001 has failed! Stop running all dependent jobs
> 2011-07-05 12:42:10,701 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 100% complete
> 2011-07-05 12:42:10,748 [main] ERROR
> org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to
> recreate exception from backed error:
> org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
> connect to ZooKeeper but the connection closes immediately. This could be a
> sign that the server has too many connections (30 is the default). Consider
> inspecting your ZK server logs for that error and then make sure you are
> reusing HBaseConfiguration as often as you can. See HTable's javadoc for
> more information.
> 2011-07-05 12:42:10,748 [main] ERROR
> org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
> 2011-07-05 12:42:10,750 [main] INFO
> org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:
>
> HadoopVersion PigVersion UserId StartedAt FinishedAt
> Features
> 0.20-append-for-hbase 0.10.0-SNAPSHOT hadoop 2011-07-05 12:40:38
> 2011-07-05 12:42:10 UNKNOWN
>
> Failed!
>
> Failed Jobs:
> JobId Alias Feature Message Outputs
> job_201107051228_0001 ht MAP_ONLY Message: Job failed!
> hdfs://ub13:54310/tmp/temp-1630311766/tmp888118030,
>
> Input(s):
> Failed to read data from "hbase://tweetinfo"
>
> Output(s):
> Failed to produce result in
> "hdfs://ub13:54310/tmp/temp-1630311766/tmp888118030"
>
> Counters:
> Total records written : 0
> Total bytes written : 0
> Spillable Memory Manager spill count : 0
> Total bags proactively spilled: 0
> Total records proactively spilled: 0
>
> Job DAG:
> job_201107051228_0001
>
>
> 2011-07-05 12:42:10,750 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - Failed!
> 2011-07-05 12:42:10,783 [main] ERROR org.apache.pig.tools.grunt.Grunt -
> ERROR 2997: Unable to recreate exception from backed error:
> org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
> connect to ZooKeeper but the connection closes immediately. This could be a
> sign that the server has too many connections (30 is the default). Consider
> inspecting your ZK server logs for that error and then make sure you are
> reusing HBaseConfiguration as often as you can. See HTable's javadoc for
> more information.
> Details at logfile: /usr/local/hadoop/hbase/conf/pig_1309849823721.log
>
> LOG file details.....
>
>
> Backend error message
> ---------------------
> org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
> connect to ZooKeeper but the connection closes immediately. This could be a
> sign that the server has too many connections (30 is the default). Consider
> inspecting your ZK server logs for that error and then make sure you are
> reusing HBaseConfiguration as often as you can. See HTable's javadoc for
> more information.
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:155)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:990)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:303)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:294)
> at
> org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:156)
> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:168)
> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:146)
> at
> org.apache.pig.backend.hadoop.hbase.HBaseStorage.setLocation(HBaseStorage.java:457)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.mergeSplitSpecificConf(PigInputFormat.java:133)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.createRecordReader(PigInputFormat.java:111)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:588)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:809)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:837)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:903)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:133)
> ... 12 more
>
> Pig Stack Trace
> ---------------
> ERROR 2997: Unable to recreate exception from backed error:
> org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
> connect to ZooKeeper but the connection closes immediately. This could be a
> sign that the server has too many connections (30 is the default). Consider
> inspecting your ZK server logs for that error and then make sure you are
> reusing HBaseConfiguration as often as you can. See HTable's javadoc for
> more information.
>
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to
> open iterator for alias ht. Backend error : Unable to recreate exception
> from backed error: org.apache.hadoop.hbase.ZooKeeperConnectionException:
> HBase is able to connect to ZooKeeper but the connection closes immediately.
> This could be a sign that the server has too many connections (30 is the
> default). Consider inspecting your ZK server logs for that error and then
> make sure you are reusing HBaseConfiguration as often as you can. See
> HTable's javadoc for more information.
> at org.apache.pig.PigServer.openIterator(PigServer.java:890)
> at
> org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:655)
> at
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
> at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:67)
> at org.apache.pig.Main.run(Main.java:487)
> at org.apache.pig.Main.main(Main.java:108)
> Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2997:
> Unable to recreate exception from backed error:
> org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
> connect to ZooKeeper but the connection closes immediately. This could be a
> sign that the server has too many connections (30 is the default). Consider
> inspecting your ZK server logs for that error and then make sure you are
> reusing HBaseConfiguration as often as you can. See HTable's javadoc for
> more information.
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getErrorMessages(Launcher.java:221)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getStats(Launcher.java:151)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:341)
> at org.apache.pig.PigServer.launchPlan(PigServer.java:1313)
> at
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1298)
> at org.apache.pig.PigServer.storeEx(PigServer.java:995)
> at org.apache.pig.PigServer.store(PigServer.java:962)
> at org.apache.pig.PigServer.openIterator(PigServer.java:875)
> ... 7 more
> ================================================================================
>
>
> My hbase-site.xml looks like this
>
> <configuration>
> <property>
> <name>hbase.rootdir</name>
> <value>hdfs://ub13:54310/hbase</value>
> <final>true</final>
> </property>
>
>
> <property>
> <name>hbase.zookeeper.quorum</name>
> <value>ub13,ub12,ub11</value>
> <final>true</final>
> </property>
>
> <property>
> <name>hbase.zookeeper.property.maxClientCnxns</name>
> <value>200</value>
> <final>true</final>
> </property>
>
> <property>
> <name>hbase.cluster.distributed</name>
> <value>true</value>
> <description>The mode the cluster will be in. Possible values are
> false: standalone and pseudo-distributed setups with managed Zookeeper
> true: fully-distributed with unmanaged Zookeeper Quorum
> </description>
> </property>
>
>
> Zookeeper log -- http://pastebin.com/RRdzZYui
>
> I have increased the maximum number of zookeeper connections.. but still I
> am getting the same error.. I am not able to get enough out of zookeeper
> logs also.. its just saying.. end of stream exception..socket ended..
> something like that..
> The table I am trying to access consists of around 21 lacs record..
> If I try to access small tables.. I am able to access it..
>
> What can be the issue ?
>
> Thanks,
> Praveenesh
>
