>From the new log snippet you posted, the hbase client tried to connect to 192.168.0.24 whose node is not listed in the quorum.
Can you check the classpath for the maptask ? Looks like the effective hbase-site.xml wasn't on the classpath. On Wed, Jan 25, 2017 at 7:21 AM, Hernán Blanco <hernanblancola...@gmail.com> wrote: > Hello Ted, > > Thank you for your reply. Indeed, I wasn't sure if that property > existed, and in fact hbase-default.xml doesn't include it. I just > followed the advise from some webpage (I couldn't recall which) that > stated that you can include *any* property from the zoo.cfg native > config file for ZooKeeper standalone into hbase-site.xml by just adding > the "hbase.zookeeper.property." prefix. Even the comment in lines > 397-398 of > https://github.com/apache/hbase/blob/master/hbase- > common/src/main/resources/hbase-default.xml mentioned this, but I > probably misinterpreted it. > > I removed those incorrect properties from hbase-site.xml and reduced my > ZooKeeper quorum to only 3 nodes -the main and 2 others: > > > ... > <property> > <name>hbase.zookeeper.quorum</name> > <value>bwcloud-fip26.rz.uni-ulm.de,bwcloud-fip27.rz.uni-ulm.de, > bwcloud-fip144.rz.uni-ulm.de</value> > </property> > ... > > > But the problem remains. The MapReduce job keeps stuck, and same kind of > logs, both on HBase startup and during job execution, with an added > exception which is that those map tasks that are deployed on the nodes > that aren't running a ZooKeeper generate the following log: > > > ... > 2017-01-25 15:57:50,295 INFO [main-SendThread(bwcloud- > fip150.rz.uni-ulm.de:2181)] org.apache.zookeeper.ClientCnxn: Opening > socket connection to server bwcloud-fip150.rz.uni-ulm.de/192.168.0.24:2181. > Will not attempt to authenticate using SASL (unknown error) > 2017-01-25 15:57:50,296 WARN [main-SendThread(bwcloud- > fip150.rz.uni-ulm.de:2181)] org.apache.zookeeper.ClientCnxn: Session 0x0 > for server null, unexpected error, closing socket connection and attempting > reconnect > java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > at sun.nio.ch.SocketChannelImpl.finishConnect( > SocketChannelImpl.java:717) > at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport( > ClientCnxnSocketNIO.java:361) > at org.apache.zookeeper.ClientCnxn$SendThread.run( > ClientCnxn.java:1081) > 2017-01-25 15:57:51,400 INFO [main-SendThread(bwcloud- > fip150.rz.uni-ulm.de:2181)] org.apache.zookeeper.ClientCnxn: Opening > socket connection to server bwcloud-fip150.rz.uni-ulm.de/192.168.0.24:2181. > Will not attempt to authenticate using SASL (unknown error) > 2017-01-25 15:57:51,401 WARN [main-SendThread(bwcloud- > fip150.rz.uni-ulm.de:2181)] org.apache.zookeeper.ClientCnxn: Session 0x0 > for server null, unexpected error, closing socket connection and attempting > reconnect > java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > at sun.nio.ch.SocketChannelImpl.finishConnect( > SocketChannelImpl.java:717) > at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport( > ClientCnxnSocketNIO.java:361) > at org.apache.zookeeper.ClientCnxn$SendThread.run( > ClientCnxn.java:1081) > ... > > > I actually tried this before, but my conclusion was that all nodes that > are running a YARN NodeManager need to have ZooKeeper running. > > Best, > Hernán. > > > On Wed, 2017-01-25 at 06:23 -0800, Ted Yu wrote: > > > bq. <name>hbase.zookeeper.property.server.7</name> > > > > I searched 1.2 codebase but didn't find config parameter in the above > form. > > http://hbase.apache.org/book.html didn't mention it either. > > > > May I ask where you obtained such config ? > > > > For hbase.zookeeper.quorum, do you have zookeeper running on the 12 > nodes ? > > Normally 3 zookeeper nodes should be enough. > > > > Cheers > > > > On Wed, Jan 25, 2017 at 2:08 AM, Hernán Blanco < > hernanblancola...@gmail.com> > > wrote: > > > > > Hi all, > > > > > > I'm running HBase 1.2.4 on a distributed setup with 12 virtual machines > > > on the same local network. The "main" node (node26.example.com) runs > the > > > HMaster, while the other 11 machines run RegionServers. No backup > > > HMaster. This cluster also runs Hadoop 2.7.2 smoothly. > > > > > > Both HBase shell and accessing through the HBase client API run > > > properly, and even importing a TSV file into a table with > > > org.apache.hadoop.hbase.mapreduce.ImportTsv succeeds, remaining > > > registered in the YARN history. > > > > > > But the problem appears when trying to access from a MapReduce job to > an > > > HBase table (using Hadoop 2.7.2). Here a minimal code that produces the > > > issue by connecting and scanning an HBase table: > > > http://pastebin.com/pm8tbbTq. The maps hang until timeout and then > > > retries deploying new maps until failing, each map showing the > following > > > messages in the syslog: > > > > > > ---------- > > > > > > 2017-01-24 20:04:07,904 INFO [main] org.apache.hadoop.metrics2. > impl.MetricsConfig: > > > loaded properties from hadoop-metrics2.properties > > > 2017-01-24 20:04:08,186 INFO [main] org.apache.hadoop.metrics2. > impl.MetricsSystemImpl: > > > Scheduled snapshot period at 10 second(s). > > > 2017-01-24 20:04:08,186 INFO [main] org.apache.hadoop.metrics2. > impl.MetricsSystemImpl: > > > MapTask metrics system started > > > 2017-01-24 20:04:08,193 INFO [main] org.apache.hadoop.mapred. > YarnChild: > > > Executing with tokens: > > > 2017-01-24 20:04:08,193 INFO [main] org.apache.hadoop.mapred. > YarnChild: > > > Kind: mapreduce.job, Service: job_1485182272940_0336, Ident: > > > (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@6f03482 > ) > > > 2017-01-24 20:04:08,450 INFO [main] org.apache.hadoop.mapred. > YarnChild: > > > Sleeping for 0ms before retrying again. Got null now. > > > 2017-01-24 20:04:08,994 INFO [main] org.apache.hadoop.mapred. > YarnChild: > > > mapreduce.cluster.local.dir for child: /tmp/hadoop-hdadmin/nm-local- > > > dir/usercache/idstest/appcache/application_1485182272940_0336 > > > 2017-01-24 20:04:09,617 INFO [main] org.apache.hadoop.conf. > Configuration.deprecation: > > > session.id is deprecated. Instead, use dfs.metrics.session-id > > > 2017-01-24 20:04:10,968 INFO [main] org.apache.hadoop.mapreduce. > > > lib.output.FileOutputCommitter: File Output Committer Algorithm > version > > > is 1 > > > 2017-01-24 20:04:11,044 INFO [main] org.apache.hadoop.mapred.Task: > Using > > > ResourceCalculatorProcessTree : [ ] > > > 2017-01-24 20:04:11,455 INFO [main] org.apache.hadoop.mapred.MapTask: > > > Processing split: hdfs://node26.example.com/ > user/idstest/data/articles-50/ > > > 99322.txt:0+3757 > > > 2017-01-24 20:04:11,596 INFO [main] org.apache.hadoop.mapred.MapTask: > > > (EQUATOR) 0 kvi 26214396(104857584) > > > 2017-01-24 20:04:11,596 INFO [main] org.apache.hadoop.mapred.MapTask: > > > mapreduce.task.io.sort.mb: 100 > > > 2017-01-24 20:04:11,596 INFO [main] org.apache.hadoop.mapred.MapTask: > > > soft limit at 83886080 > > > 2017-01-24 20:04:11,596 INFO [main] org.apache.hadoop.mapred.MapTask: > > > bufstart = 0; bufvoid = 104857600 > > > 2017-01-24 20:04:11,596 INFO [main] org.apache.hadoop.mapred.MapTask: > > > kvstart = 26214396; length = 6553600 > > > 2017-01-24 20:04:11,612 INFO [main] org.apache.hadoop.mapred.MapTask: > Map > > > output collector class = org.apache.hadoop.mapred. > MapTask$MapOutputBuffer > > > 2017-01-24 20:04:12,211 INFO [main] org.apache.hadoop.hbase.zookeeper. > RecoverableZooKeeper: > > > Process identifier=hconnection-0x650eab8 connecting to ZooKeeper > > > ensemble=localhost:2181 > > > 2017-01-24 20:04:12,234 INFO [main] org.apache.zookeeper.ZooKeeper: > > > Client environment:zookeeper.version=3.4.6-1569965, built on > 02/20/2014 > > > 09:09 GMT > > > 2017-01-24 20:04:12,234 INFO [main] org.apache.zookeeper.ZooKeeper: > > > Client environment:host.name=node27.example.com > > > 2017-01-24 20:04:12,234 INFO [main] org.apache.zookeeper.ZooKeeper: > > > Client environment:java.version=1.8.0_77-Debian > > > 2017-01-24 20:04:12,234 INFO [main] org.apache.zookeeper.ZooKeeper: > > > Client environment:java.vendor=Oracle Corporation > > > 2017-01-24 20:04:12,234 INFO [main] org.apache.zookeeper.ZooKeeper: > > > Client environment:java.home=/usr/lib/jvm/java-8-openjdk-amd64/jre > > > 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper: > > > Client environment:java.class.path= /* ... */ > > > 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper: > > > Client environment:java.library.path=/tmp/hadoop-hdadmin/nm-local- > > > dir/usercache/idstest/appcache/application_ > 1485182272940_0336/container_ > > > 1485182272940_0336_01_000048:/opt/hadoop-2.7.2/lib/native:/ > > > usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/ > > > lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni: > /lib:/usr/lib > > > 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper: > > > Client environment:java.io.tmpdir=/tmp/hadoop-hdadmin/nm-local- > > > dir/usercache/idstest/appcache/application_ > 1485182272940_0336/container_ > > > 1485182272940_0336_01_000048/tmp > > > 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper: > > > Client environment:java.compiler=<NA> > > > 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper: > > > Client environment:os.name=Linux > > > 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper: > > > Client environment:os.arch=amd64 > > > 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper: > > > Client environment:os.version=3.16.0-4-amd64 > > > 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper: > > > Client environment:user.name=hdadmin > > > 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper: > > > Client environment:user.home=/home/hdadmin > > > 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper: > > > Client environment:user.dir=/tmp/hadoop-hdadmin/nm-local-dir/ > > > usercache/idstest/appcache/application_1485182272940_ > > > 0336/container_1485182272940_0336_01_000048 > > > 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper: > > > Initiating client connection, connectString=localhost:2181 > > > sessionTimeout=90000 watcher=hconnection-0x650eab80x0, > > > quorum=localhost:2181, baseZNode=/hbase > > > 2017-01-24 20:04:12,322 INFO [main-SendThread(node27.example.com:2181 > )] > > > org.apache.zookeeper.ClientCnxn: Opening socket connection to server > > > node27.example.com/192.168.0.17:2181. Will not attempt to authenticate > > > using SASL (unknown error) > > > 2017-01-24 20:04:12,323 INFO [main-SendThread(node27.example.com:2181 > )] > > > org.apache.zookeeper.ClientCnxn: Socket connection established to > > > node27.example.com/192.168.0.17:2181, initiating session > > > 2017-01-24 20:04:12,331 INFO [main-SendThread(node27.example.com:2181 > )] > > > org.apache.zookeeper.ClientCnxn: Session establishment complete on > server > > > node27.example.com/192.168.0.17:2181, sessionid = 0x159d07980b50189, > > > negotiated timeout = 90000 > > > 2017-01-24 20:04:51,657 INFO [hconnection-0x650eab8- > metaLookup-shared--pool2-t1] > > > org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, > > > tries=10, retries=35, started=38852 ms ago, cancelled=false, msg=row > > > 'idstest1,,99999999999999' on table 'hbase:meta' at > region=hbase:meta,,1.1588230740, > > > hostname=node149.example.com,16020,1485261342943, seqNum=0 > > > 2017-01-24 20:05:01,664 INFO [hconnection-0x650eab8- > metaLookup-shared--pool2-t1] > > > org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, > > > tries=11, retries=35, started=48860 ms ago, cancelled=false, msg=row > > > 'idstest1,,99999999999999' on table 'hbase:meta' at > region=hbase:meta,,1.1588230740, > > > hostname=node149.example.com,16020,1485261342943, seqNum=0 > > > 2017-01-24 20:05:40,044 INFO [hconnection-0x650eab8- > metaLookup-shared--pool2-t2] > > > org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, > > > tries=10, retries=35, started=38273 ms ago, cancelled=false, msg=row > > > 'idstest1,,99999999999999' on table 'hbase:meta' at > region=hbase:meta,,1.1588230740, > > > hostname=node149.example.com,16020,1485261342943, seqNum=0 > > > 2017-01-24 20:05:50,059 INFO [hconnection-0x650eab8- > metaLookup-shared--pool2-t2] > > > org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, > > > tries=11, retries=35, started=48288 ms ago, cancelled=false, msg=row > > > 'idstest1,,99999999999999' on table 'hbase:meta' at > region=hbase:meta,,1.1588230740, > > > hostname=node149.example.com,16020,1485261342943, seqNum=0 > > > 2017-01-24 20:06:28,543 INFO [hconnection-0x650eab8- > metaLookup-shared--pool2-t3] > > > org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, > > > tries=10, retries=35, started=38273 ms ago, cancelled=false, msg=row > > > 'idstest1,,99999999999999' on table 'hbase:meta' at > region=hbase:meta,,1.1588230740, > > > hostname=node149.example.com,16020,1485261342943, seqNum=0 > > > 2017-01-24 20:06:38,632 INFO [hconnection-0x650eab8- > metaLookup-shared--pool2-t3] > > > org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, > > > tries=11, retries=35, started=48362 ms ago, cancelled=false, msg=row > > > 'idstest1,,99999999999999' on table 'hbase:meta' at > region=hbase:meta,,1.1588230740, > > > hostname=node149.example.com,16020,1485261342943, seqNum=0 > > > ... > > > > > > ---------- > > > > > > My hbase-site.xml in the Master node node26.example.com (almost same > for > > > the other nodes, just referring as 0.0.0.0 to themselves): > > > > > > ---------- > > > > > > <configuration> > > > <property> > > > <name>hbase.rootdir</name> > > > <value>hdfs://node26.example.com:8020/hbase</value> > > > </property> > > > <property> > > > <name>hbase.zookeeper.property.dataDir</name> > > > <value>/home/hdadmin/zookeeper</value> > > > </property> > > > <property> > > > <name>hbase.cluster.distributed</name> > > > <value>true</value> > > > </property> > > > <property> > > > <name>hbase.zookeeper.quorum</name> > > > <value>node26.example.com,node27.example.com,node144.example.com, > > > node145.example.com,node146.example.com,node147.example.com, > > > node148.example.com,node149.example.com,node150.example.com, > > > node151.example.com,node152.example.com,node153.example.com</value> > > > </property> > > > <property> > > > <name>hbase.zookeeper.property.server.0</name> > > > <value>0.0.0.0:2888:3888</value> > > > </property> > > > <property> > > > <name>hbase.zookeeper.property.server.1</name> > > > <value>node27.example.com:2888:3888</value> > > > </property> > > > <property> > > > <name>hbase.zookeeper.property.server.2</name> > > > <value>node144.example.com:2888:3888</value> > > > </property> > > > <property> > > > <name>hbase.zookeeper.property.server.3</name> > > > <value>node145.example.com:2888:3888</value> > > > </property> > > > <property> > > > <name>hbase.zookeeper.property.server.4</name> > > > <value>node146.example.com:2888:3888</value> > > > </property> > > > <property> > > > <name>hbase.zookeeper.property.server.5</name> > > > <value>node147.example.com:2888:3888</value> > > > </property> > > > <property> > > > <name>hbase.zookeeper.property.server.6</name> > > > <value>node148.example.com:2888:3888</value> > > > </property> > > > <property> > > > <name>hbase.zookeeper.property.server.7</name> > > > <value>node149.example.com:2888:3888</value> > > > </property> > > > <property> > > > <name>hbase.zookeeper.property.server.8</name> > > > <value>node150.example.com:2888:3888</value> > > > </property> > > > <property> > > > <name>hbase.zookeeper.property.server.9</name> > > > <value>node151.example.com:2888:3888</value> > > > </property> > > > <property> > > > <name>hbase.zookeeper.property.server.10</name> > > > <value>node152.example.com:2888:3888</value> > > > </property> > > > <property> > > > <name>hbase.zookeeper.property.server.11</name> > > > <value>node153.example.com:2888:3888</value> > > > </property> > > > </configuration> > > > > > > ---------- > > > > > > After starting HBase using the start-hbase.sh script, the RegionServer > > > logs have no warnings or errors, but the different ZooKeeper instances > > > generate diverse outputs. 3 examples: > > > > > > > > > - The master node has an empty log: > > > > > > ---------- > > > > > > Tue Jan 24 20:52:38 CET 2017 Starting zookeeper on node26.example.com > > > core file size (blocks, -c) 0 > > > data seg size (kbytes, -d) unlimited > > > scheduling priority (-e) 0 > > > file size (blocks, -f) unlimited > > > pending signals (-i) 128914 > > > max locked memory (kbytes, -l) 64 > > > max memory size (kbytes, -m) unlimited > > > open files (-n) 65536 > > > pipe size (512 bytes, -p) 8 > > > POSIX message queues (bytes, -q) 819200 > > > real-time priority (-r) 0 > > > stack size (kbytes, -s) 8192 > > > cpu time (seconds, -t) unlimited > > > max user processes (-u) 128914 > > > virtual memory (kbytes, -v) unlimited > > > file locks (-x) unlimited > > > > > > ---------- > > > > > > > > > - node153.example.com has the following small ZooKeeper log: > > > > > > ---------- > > > > > > Tue Jan 24 20:52:38 CET 2017 Starting zookeeper on node153.example.com > > > core file size (blocks, -c) 0 > > > data seg size (kbytes, -d) unlimited > > > scheduling priority (-e) 0 > > > file size (blocks, -f) unlimited > > > pending signals (-i) 128914 > > > max locked memory (kbytes, -l) 64 > > > max memory size (kbytes, -m) unlimited > > > open files (-n) 65536 > > > pipe size (512 bytes, -p) 8 > > > POSIX message queues (bytes, -q) 819200 > > > real-time priority (-r) 0 > > > stack size (kbytes, -s) 8192 > > > cpu time (seconds, -t) unlimited > > > max user processes (-u) 128914 > > > virtual memory (kbytes, -v) unlimited > > > file locks (-x) unlimited > > > 2017-01-24 20:52:39,816 WARN [main] quorum.QuorumPeerConfig: > Non-optimial > > > configuration, consider an odd number of servers. > > > 2017-01-24 20:52:39,817 INFO [main] quorum.QuorumPeerConfig: > Defaulting > > > to majority quorums > > > 2017-01-24 20:52:39,984 INFO [main] quorum.QuorumPeerMain: Starting > > > quorum peer > > > 2017-01-24 20:52:39,994 INFO [main] server.NIOServerCnxnFactory: > binding > > > to port 0.0.0.0/0.0.0.0:2181 > > > > > > ---------- > > > > > > As an extra hint, this node is not generating > > > the /tmp/hbase-hdadmin-zookeeper.pid, so it shows an error when > running > > > the stop-hbase.sh script. > > > > > > > > > - And node144.example.com (as other nodes) has the following ZooKeeper > > > log with "connection refused" errors: > > > > > > ---------- > > > > > > Tue Jan 24 20:52:38 CET 2017 Starting zookeeper on node144.example.com > > > core file size (blocks, -c) 0 > > > data seg size (kbytes, -d) unlimited > > > scheduling priority (-e) 0 > > > file size (blocks, -f) unlimited > > > pending signals (-i) 128914 > > > max locked memory (kbytes, -l) 64 > > > max memory size (kbytes, -m) unlimited > > > open files (-n) 65536 > > > pipe size (512 bytes, -p) 8 > > > POSIX message queues (bytes, -q) 819200 > > > real-time priority (-r) 0 > > > stack size (kbytes, -s) 8192 > > > cpu time (seconds, -t) unlimited > > > max user processes (-u) 128914 > > > virtual memory (kbytes, -v) unlimited > > > file locks (-x) unlimited > > > 2017-01-24 20:52:39,473 WARN [main] quorum.QuorumPeerConfig: > Non-optimial > > > configuration, consider an odd number of servers. > > > 2017-01-24 20:52:39,473 INFO [main] quorum.QuorumPeerConfig: > Defaulting > > > to majority quorums > > > 2017-01-24 20:52:39,567 INFO [main] quorum.QuorumPeerMain: Starting > > > quorum peer > > > 2017-01-24 20:52:39,581 INFO [main] server.NIOServerCnxnFactory: > binding > > > to port 0.0.0.0/0.0.0.0:2181 > > > 2017-01-24 20:52:39,601 INFO [main] quorum.QuorumPeer: tickTime set to > > > 3000 > > > 2017-01-24 20:52:39,601 INFO [main] quorum.QuorumPeer: > minSessionTimeout > > > set to -1 > > > 2017-01-24 20:52:39,601 INFO [main] quorum.QuorumPeer: > maxSessionTimeout > > > set to 90000 > > > 2017-01-24 20:52:39,601 INFO [main] quorum.QuorumPeer: initLimit set > to 10 > > > 2017-01-24 20:52:39,613 INFO [main] persistence.FileSnap: Reading > > > snapshot /home/hdadmin/zookeeper2/version-2/snapshot.1d0000023f > > > 2017-01-24 20:52:39,714 INFO [Thread-2] quorum.QuorumCnxManager: My > > > election bind port: node144.example.com/192.168.0.19:3888 > > > 2017-01-24 20:52:39,729 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > 0:0:0:2181] > > > quorum.QuorumPeer: LOOKING > > > 2017-01-24 20:52:39,731 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > 0:0:0:2181] > > > quorum.FastLeaderElection: New election. My id = 2, proposed > > > zxid=0x1e0000258d > > > 2017-01-24 20:52:39,740 INFO [WorkerReceiver[myid=2]] > > > quorum.FastLeaderElection: Notification: 1 (message format version), 2 > > > (n.leader), 0x1e0000258d (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > 2017-01-24 20:52:39,740 INFO [WorkerReceiver[myid=2]] > > > quorum.FastLeaderElection: Notification: 1 (message format version), 0 > > > (n.leader), 0x1e0000258d (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > 2017-01-24 20:52:39,742 WARN [WorkerSender[myid=2]] > > > quorum.QuorumCnxManager: Cannot open channel to 3 at election address > > > node145.example.com/192.168.0.18:3888 > > > java.net.ConnectException: Connection refused > > > at java.net.PlainSocketImpl.socketConnect(Native Method) > > > at java.net.AbstractPlainSocketImpl.doConnect( > > > AbstractPlainSocketImpl.java:350) > > > at java.net.AbstractPlainSocketImpl.connectToAddress( > > > AbstractPlainSocketImpl.java:206) > > > at java.net.AbstractPlainSocketImpl.connect( > > > AbstractPlainSocketImpl.java:188) > > > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > > > at java.net.Socket.connect(Socket.java:589) > > > at org.apache.zookeeper.server.quorum.QuorumCnxManager. > > > connectOne(QuorumCnxManager.java:368) > > > at org.apache.zookeeper.server.quorum.QuorumCnxManager. > > > toSend(QuorumCnxManager.java:341) > > > at org.apache.zookeeper.server.quorum.FastLeaderElection$ > > > Messenger$WorkerSender.process(FastLeaderElection.java:449) > > > at org.apache.zookeeper.server.quorum.FastLeaderElection$ > > > Messenger$WorkerSender.run(FastLeaderElection.java:430) > > > at java.lang.Thread.run(Thread.java:745) > > > 2017-01-24 20:52:39,744 INFO [WorkerReceiver[myid=2]] > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11 > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 1 > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > 2017-01-24 20:52:39,745 INFO [WorkerReceiver[myid=2]] > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11 > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 1 > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > 2017-01-24 20:52:39,745 INFO [WorkerReceiver[myid=2]] > > > quorum.FastLeaderElection: Notification: 1 (message format version), 2 > > > (n.leader), 0x1e0000258d (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > 2017-01-24 20:52:39,746 INFO [WorkerSender[myid=2]] > > > quorum.QuorumCnxManager: Have smaller server identifier, so dropping > the > > > connection: (4, 2) > > > 2017-01-24 20:52:39,747 INFO [WorkerSender[myid=2]] > > > quorum.QuorumCnxManager: Have smaller server identifier, so dropping > the > > > connection: (5, 2) > > > 2017-01-24 20:52:39,747 INFO [node144.example.com/192.168.0.19:3888] > > > quorum.QuorumCnxManager: Received connection request / > 192.168.0.21:40064 > > > 2017-01-24 20:52:39,747 INFO [node144.example.com/192.168.0.19:3888] > > > quorum.QuorumCnxManager: Received connection request / > 192.168.0.22:54362 > > > 2017-01-24 20:52:39,747 INFO [WorkerSender[myid=2]] > > > quorum.QuorumCnxManager: Have smaller server identifier, so dropping > the > > > connection: (6, 2) > > > 2017-01-24 20:52:39,748 INFO [WorkerReceiver[myid=2]] > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11 > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 4 > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > 2017-01-24 20:52:39,748 INFO [WorkerReceiver[myid=2]] > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11 > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 5 > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > 2017-01-24 20:52:39,748 INFO [node144.example.com/192.168.0.19:3888] > > > quorum.QuorumCnxManager: Received connection request / > 192.168.0.23:53410 > > > 2017-01-24 20:52:39,749 INFO [WorkerReceiver[myid=2]] > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11 > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 5 > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > 2017-01-24 20:52:39,749 INFO [WorkerReceiver[myid=2]] > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11 > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 0 > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > 2017-01-24 20:52:39,749 INFO [WorkerSender[myid=2]] > > > quorum.QuorumCnxManager: Have smaller server identifier, so dropping > the > > > connection: (7, 2) > > > 2017-01-24 20:52:39,749 INFO [WorkerReceiver[myid=2]] > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11 > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 4 > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > 2017-01-24 20:52:39,749 INFO [WorkerReceiver[myid=2]] > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11 > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 6 > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > 2017-01-24 20:52:39,750 INFO [node144.example.com/192.168.0.19:3888] > > > quorum.QuorumCnxManager: Received connection request / > 192.168.0.25:33085 > > > 2017-01-24 20:52:39,750 INFO [WorkerReceiver[myid=2]] > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11 > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 6 > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > 2017-01-24 20:52:39,751 WARN [WorkerSender[myid=2]] > > > quorum.QuorumCnxManager: Cannot open channel to 8 at election address > > > node150.example.com/192.168.0.24:3888 > > > java.net.ConnectException: Connection refused > > > at java.net.PlainSocketImpl.socketConnect(Native Method) > > > at java.net.AbstractPlainSocketImpl.doConnect( > > > AbstractPlainSocketImpl.java:350) > > > at java.net.AbstractPlainSocketImpl.connectToAddress( > > > AbstractPlainSocketImpl.java:206) > > > at java.net.AbstractPlainSocketImpl.connect( > > > AbstractPlainSocketImpl.java:188) > > > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > > > at java.net.Socket.connect(Socket.java:589) > > > at org.apache.zookeeper.server.quorum.QuorumCnxManager. > > > connectOne(QuorumCnxManager.java:368) > > > at org.apache.zookeeper.server.quorum.QuorumCnxManager. > > > toSend(QuorumCnxManager.java:341) > > > at org.apache.zookeeper.server.quorum.FastLeaderElection$ > > > Messenger$WorkerSender.process(FastLeaderElection.java:449) > > > at org.apache.zookeeper.server.quorum.FastLeaderElection$ > > > Messenger$WorkerSender.run(FastLeaderElection.java:430) > > > at java.lang.Thread.run(Thread.java:745) > > > 2017-01-24 20:52:39,751 INFO [WorkerReceiver[myid=2]] > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11 > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 7 > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > 2017-01-24 20:52:39,752 INFO [WorkerReceiver[myid=2]] > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11 > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 7 > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > 2017-01-24 20:52:39,753 INFO [WorkerSender[myid=2]] > > > quorum.QuorumCnxManager: Have smaller server identifier, so dropping > the > > > connection: (9, 2) > > > 2017-01-24 20:52:39,753 INFO [WorkerSender[myid=2]] > > > quorum.QuorumCnxManager: Have smaller server identifier, so dropping > the > > > connection: (10, 2) > > > 2017-01-24 20:52:39,753 INFO [node144.example.com/192.168.0.19:3888] > > > quorum.QuorumCnxManager: Received connection request / > 192.168.0.26:35997 > > > 2017-01-24 20:52:39,754 INFO [WorkerSender[myid=2]] > > > quorum.QuorumCnxManager: Have smaller server identifier, so dropping > the > > > connection: (11, 2) > > > 2017-01-24 20:52:39,755 INFO [WorkerReceiver[myid=2]] > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11 > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 2 > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > 2017-01-24 20:52:39,755 INFO [WorkerSender[myid=2]] > > > quorum.QuorumCnxManager: Have smaller server identifier, so dropping > the > > > connection: (3, 2) > > > 2017-01-24 20:52:39,756 WARN [WorkerSender[myid=2]] > > > quorum.QuorumCnxManager: Cannot open channel to 8 at election address > > > node150.example.com/192.168.0.24:3888 > > > java.net.ConnectException: Connection refused > > > at java.net.PlainSocketImpl.socketConnect(Native Method) > > > at java.net.AbstractPlainSocketImpl.doConnect( > > > AbstractPlainSocketImpl.java:350) > > > at java.net.AbstractPlainSocketImpl.connectToAddress( > > > AbstractPlainSocketImpl.java:206) > > > at java.net.AbstractPlainSocketImpl.connect( > > > AbstractPlainSocketImpl.java:188) > > > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > > > at java.net.Socket.connect(Socket.java:589) > > > at org.apache.zookeeper.server.quorum.QuorumCnxManager. > > > connectOne(QuorumCnxManager.java:368) > > > at org.apache.zookeeper.server.quorum.QuorumCnxManager. > > > toSend(QuorumCnxManager.java:341) > > > at org.apache.zookeeper.server.quorum.FastLeaderElection$ > > > Messenger$WorkerSender.process(FastLeaderElection.java:449) > > > at org.apache.zookeeper.server.quorum.FastLeaderElection$ > > > Messenger$WorkerSender.run(FastLeaderElection.java:430) > > > at java.lang.Thread.run(Thread.java:745) > > > 2017-01-24 20:52:39,757 INFO [WorkerReceiver[myid=2]] > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11 > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 9 > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > 2017-01-24 20:52:39,757 INFO [WorkerSender[myid=2]] > > > quorum.QuorumCnxManager: Have smaller server identifier, so dropping > the > > > connection: (10, 2) > > > 2017-01-24 20:52:39,757 INFO [WorkerSender[myid=2]] > > > quorum.QuorumCnxManager: Have smaller server identifier, so dropping > the > > > connection: (11, 2) > > > 2017-01-24 20:52:39,758 INFO [node144.example.com/192.168.0.19:3888] > > > quorum.QuorumCnxManager: Received connection request / > 192.168.0.27:34404 > > > 2017-01-24 20:52:39,759 INFO [WorkerReceiver[myid=2]] > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11 > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 9 > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > 2017-01-24 20:52:39,768 INFO [node144.example.com/192.168.0.19:3888] > > > quorum.QuorumCnxManager: Received connection request / > 192.168.0.18:59032 > > > 2017-01-24 20:52:39,769 INFO [WorkerReceiver[myid=2]] > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11 > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 11 > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > 2017-01-24 20:52:39,769 INFO [WorkerReceiver[myid=2]] > > > quorum.FastLeaderElection: Notification: 1 (message format version), 3 > > > (n.leader), 0x1e0000258d (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > 2017-01-24 20:52:39,776 INFO [node144.example.com/192.168.0.19:3888] > > > quorum.QuorumCnxManager: Received connection request / > 192.168.0.27:34405 > > > 2017-01-24 20:52:39,776 WARN [RecvWorker:11] quorum.QuorumCnxManager: > > > Connection broken for id 11, my id = 2, error = > > > java.io.EOFException > > > at java.io.DataInputStream.readInt(DataInputStream.java:392) > > > at org.apache.zookeeper.server.quorum.QuorumCnxManager$ > > > RecvWorker.run(QuorumCnxManager.java:765) > > > 2017-01-24 20:52:39,777 WARN [RecvWorker:11] quorum.QuorumCnxManager: > > > Interrupting SendWorker > > > 2017-01-24 20:52:39,777 WARN [SendWorker:11] quorum.QuorumCnxManager: > > > Interrupted while waiting for message on queue > > > java.lang.InterruptedException > > > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ > > > ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer. > > > java:2014) > > > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ > > > ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) > > > at java.util.concurrent.ArrayBlockingQueue.poll( > > > ArrayBlockingQueue.java:418) > > > at org.apache.zookeeper.server.quorum.QuorumCnxManager. > > > pollSendQueue(QuorumCnxManager.java:849) > > > at org.apache.zookeeper.server.quorum.QuorumCnxManager. > > > access$500(QuorumCnxManager.java:64) > > > at org.apache.zookeeper.server.quorum.QuorumCnxManager$ > > > SendWorker.run(QuorumCnxManager.java:685) > > > 2017-01-24 20:52:39,777 INFO [node144.example.com/192.168.0.19:3888] > > > quorum.QuorumCnxManager: Received connection request / > 192.168.0.28:57974 > > > 2017-01-24 20:52:39,778 WARN [SendWorker:11] quorum.QuorumCnxManager: > > > Send worker leaving thread > > > 2017-01-24 20:52:39,778 INFO [node144.example.com/192.168.0.19:3888] > > > quorum.QuorumCnxManager: Received connection request / > 192.168.0.28:57975 > > > 2017-01-24 20:52:39,778 WARN [RecvWorker:10] quorum.QuorumCnxManager: > > > Connection broken for id 10, my id = 2, error = > > > java.io.EOFException > > > at java.io.DataInputStream.readInt(DataInputStream.java:392) > > > at org.apache.zookeeper.server.quorum.QuorumCnxManager$ > > > RecvWorker.run(QuorumCnxManager.java:765) > > > 2017-01-24 20:52:39,778 WARN [RecvWorker:10] quorum.QuorumCnxManager: > > > Interrupting SendWorker > > > 2017-01-24 20:52:39,778 WARN [SendWorker:10] quorum.QuorumCnxManager: > > > Exception when using channel: for id 10 my id = 2 error = > > > java.net.SocketException: Broken pipe > > > 2017-01-24 20:52:39,778 WARN [SendWorker:10] quorum.QuorumCnxManager: > > > Send worker leaving thread > > > 2017-01-24 20:52:39,779 INFO [WorkerReceiver[myid=2]] > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11 > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 3 > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > 2017-01-24 20:52:39,782 INFO [WorkerReceiver[myid=2]] > > > quorum.FastLeaderElection: Notification: 1 (message format version), 10 > > > (n.leader), 0x1e0000258d (n.zxid), 0x1 (n.round), LOOKING (n.state), 10 > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > 2017-01-24 20:52:39,788 INFO [WorkerReceiver[myid=2]] > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11 > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 11 > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > 2017-01-24 20:52:39,789 INFO [WorkerReceiver[myid=2]] > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11 > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 10 > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > 2017-01-24 20:52:39,815 INFO [node144.example.com/192.168.0.19:3888] > > > quorum.QuorumCnxManager: Received connection request / > 192.168.0.24:56503 > > > 2017-01-24 20:52:39,829 INFO [WorkerReceiver[myid=2]] > > > quorum.FastLeaderElection: Notification: 1 (message format version), 8 > > > (n.leader), 0x1e0000258d (n.zxid), 0x1 (n.round), LOOKING (n.state), 8 > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > 2017-01-24 20:52:39,829 INFO [WorkerReceiver[myid=2]] > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11 > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 8 > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > 2017-01-24 20:52:40,030 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > 0:0:0:2181] > > > quorum.QuorumPeer: FOLLOWING > > > 2017-01-24 20:52:40,035 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > 0:0:0:2181] > > > quorum.Learner: TCP NoDelay set to: true > > > 2017-01-24 20:52:40,040 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > 0:0:0:2181] > > > server.ZooKeeperServer: Server environment:zookeeper.version= > 3.4.6-1569965, > > > built on 02/20/2014 09:09 GMT > > > 2017-01-24 20:52:40,040 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > 0:0:0:2181] > > > server.ZooKeeperServer: Server environment:host.name=node144. > example.com > > > 2017-01-24 20:52:40,040 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > 0:0:0:2181] > > > server.ZooKeeperServer: Server environment:java.version=1.8. > 0_77-Debian > > > 2017-01-24 20:52:40,040 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > 0:0:0:2181] > > > server.ZooKeeperServer: Server environment:java.vendor=Oracle > Corporation > > > 2017-01-24 20:52:40,040 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > 0:0:0:2181] > > > server.ZooKeeperServer: Server environment:java.home=/usr/ > > > lib/jvm/java-8-openjdk-amd64/jre > > > 2017-01-24 20:52:40,040 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > 0:0:0:2181] > > > server.ZooKeeperServer: Server environment:java.class.path= /* ... */ > > > 2017-01-24 20:52:40,040 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > 0:0:0:2181] > > > server.ZooKeeperServer: Server environment:java.library.path= > > > /usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/ > > > lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni: > /lib:/usr/lib > > > 2017-01-24 20:52:40,040 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > 0:0:0:2181] > > > server.ZooKeeperServer: Server environment:java.io.tmpdir=/tmp > > > 2017-01-24 20:52:40,040 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > 0:0:0:2181] > > > server.ZooKeeperServer: Server environment:java.compiler=<NA> > > > 2017-01-24 20:52:40,040 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > 0:0:0:2181] > > > server.ZooKeeperServer: Server environment:os.name=Linux > > > 2017-01-24 20:52:40,040 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > 0:0:0:2181] > > > server.ZooKeeperServer: Server environment:os.arch=amd64 > > > 2017-01-24 20:52:40,040 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > 0:0:0:2181] > > > server.ZooKeeperServer: Server environment:os.version=3.16.0-4-amd64 > > > 2017-01-24 20:52:40,040 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > 0:0:0:2181] > > > server.ZooKeeperServer: Server environment:user.name=hdadmin > > > 2017-01-24 20:52:40,040 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > 0:0:0:2181] > > > server.ZooKeeperServer: Server environment:user.home=/home/hdadmin > > > 2017-01-24 20:52:40,040 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > 0:0:0:2181] > > > server.ZooKeeperServer: Server environment:user.dir=/opt/hbase-1.2.4 > > > 2017-01-24 20:52:40,042 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > 0:0:0:2181] > > > server.ZooKeeperServer: Created server with tickTime 3000 > minSessionTimeout > > > 6000 maxSessionTimeout 90000 datadir /home/hdadmin/zookeeper2/ > version-2 > > > snapdir /home/hdadmin/zookeeper2/version-2 > > > 2017-01-24 20:52:40,043 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > 0:0:0:2181] > > > quorum.Learner: FOLLOWING - LEADER ELECTION TOOK - 311 > > > 2017-01-24 20:52:40,046 WARN [QuorumPeer[myid=2]/0:0:0:0:0: > 0:0:0:2181] > > > quorum.Learner: Unexpected exception, tries=0, connecting to > > > node153.example.com/192.168.0.27:2888 > > > java.net.ConnectException: Connection refused > > > at java.net.PlainSocketImpl.socketConnect(Native Method) > > > at java.net.AbstractPlainSocketImpl.doConnect( > > > AbstractPlainSocketImpl.java:350) > > > at java.net.AbstractPlainSocketImpl.connectToAddress( > > > AbstractPlainSocketImpl.java:206) > > > at java.net.AbstractPlainSocketImpl.connect( > > > AbstractPlainSocketImpl.java:188) > > > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > > > at java.net.Socket.connect(Socket.java:589) > > > at org.apache.zookeeper.server.quorum.Learner. > > > connectToLeader(Learner.java:225) > > > at org.apache.zookeeper.server.quorum.Follower.followLeader( > > > Follower.java:71) > > > at org.apache.zookeeper.server.quorum.QuorumPeer.run( > > > QuorumPeer.java:786) > > > 2017-01-24 20:52:41,073 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > 0:0:0:2181] > > > quorum.Learner: Getting a diff from the leader 0x1e0000258d > > > 2017-01-24 20:52:41,116 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > 0:0:0:2181] > > > persistence.FileTxnSnapLog: Snapshotting: 0x1e0000258d to > > > /home/hdadmin/zookeeper2/version-2/snapshot.1e0000258d > > > 2017-01-24 20:52:42,408 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > 192.168.0.29:44336 > > > 2017-01-24 20:52:42,418 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > 192.168.0.29:44336 > > > 2017-01-24 20:52:42,421 WARN [QuorumPeer[myid=2]/0:0:0:0:0: > 0:0:0:2181] > > > quorum.Learner: Got zxid 0x1f00000001 expected 0x1 > > > 2017-01-24 20:52:42,421 INFO [SyncThread:2] persistence.FileTxnLog: > > > Creating new log file: log.1f00000001 > > > 2017-01-24 20:52:42,440 INFO [CommitProcessor:2] > server.ZooKeeperServer: > > > Established session 0x259d20997750000 with negotiated timeout 90000 for > > > client /192.168.0.29:44336 > > > 2017-01-24 20:52:42,734 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > 192.168.0.22:43377 > > > 2017-01-24 20:52:42,737 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > 192.168.0.22:43377 > > > 2017-01-24 20:52:42,742 INFO [CommitProcessor:2] > server.ZooKeeperServer: > > > Established session 0x259d20997750001 with negotiated timeout 90000 for > > > client /192.168.0.22:43377 > > > 2017-01-24 20:52:43,257 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > 192.168.0.23:59005 > > > 2017-01-24 20:52:43,258 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > 192.168.0.23:59005 > > > 2017-01-24 20:52:43,262 INFO [CommitProcessor:2] > server.ZooKeeperServer: > > > Established session 0x259d20997750002 with negotiated timeout 90000 for > > > client /192.168.0.23:59005 > > > > > > ---------- > > > > > > > > > I've manually checked connecting to the most relevant ports of HBase > and > > > ZooKeeper (using nc or telnet) and always succeeded. > > > > > > During a job execution, these are the type of errors that I'm getting: > > > > > > ---------- > > > > > > 2017-01-24 21:10:37,187 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > 192.168.0.29:44750 > > > 2017-01-24 21:10:37,193 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > 192.168.0.29:44750 > > > 2017-01-24 21:10:37,198 INFO [CommitProcessor:2] > server.ZooKeeperServer: > > > Established session 0x259d20997750003 with negotiated timeout 90000 for > > > client /192.168.0.29:44750 > > > 2017-01-24 21:10:38,136 WARN [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxn: caught end of stream exception > > > EndOfStreamException: Unable to read additional data from client > sessionid > > > 0x259d20997750003, likely client has closed socket > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > NIOServerCnxn.java:228) > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > NIOServerCnxnFactory.java:208) > > > at java.lang.Thread.run(Thread.java:745) > > > 2017-01-24 21:10:38,137 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxn: Closed socket connection for client / > > > 192.168.0.29:44750 which had sessionid 0x259d20997750003 > > > 2017-01-24 21:18:45,469 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > 192.168.0.19:49706 > > > 2017-01-24 21:18:45,479 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > 192.168.0.19:49706 > > > 2017-01-24 21:18:45,543 INFO [CommitProcessor:2] > server.ZooKeeperServer: > > > Established session 0x259d20997750004 with negotiated timeout 90000 for > > > client /192.168.0.19:49706 > > > 2017-01-24 21:18:45,684 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > 192.168.0.19:49708 > > > 2017-01-24 21:18:45,689 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > 192.168.0.19:49708 > > > 2017-01-24 21:18:45,820 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > 192.168.0.19:49709 > > > 2017-01-24 21:18:45,824 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > 192.168.0.19:49709 > > > 2017-01-24 21:18:45,825 INFO [CommitProcessor:2] > server.ZooKeeperServer: > > > Established session 0x259d20997750005 with negotiated timeout 90000 for > > > client /192.168.0.19:49708 > > > 2017-01-24 21:18:45,840 INFO [CommitProcessor:2] > server.ZooKeeperServer: > > > Established session 0x259d20997750006 with negotiated timeout 90000 for > > > client /192.168.0.19:49709 > > > 2017-01-24 21:18:45,892 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > 192.168.0.19:49710 > > > 2017-01-24 21:18:45,896 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > 192.168.0.19:49710 > > > 2017-01-24 21:18:45,965 INFO [CommitProcessor:2] > server.ZooKeeperServer: > > > Established session 0x259d20997750007 with negotiated timeout 90000 for > > > client /192.168.0.19:49710 > > > 2017-01-24 21:18:46,010 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > 192.168.0.19:49711 > > > 2017-01-24 21:18:46,012 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > 192.168.0.19:49711 > > > 2017-01-24 21:18:46,065 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > 192.168.0.19:49712 > > > 2017-01-24 21:18:46,067 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > 192.168.0.19:49712 > > > 2017-01-24 21:18:46,133 INFO [CommitProcessor:2] > server.ZooKeeperServer: > > > Established session 0x259d20997750008 with negotiated timeout 90000 for > > > client /192.168.0.19:49711 > > > 2017-01-24 21:18:46,169 INFO [CommitProcessor:2] > server.ZooKeeperServer: > > > Established session 0x259d20997750009 with negotiated timeout 90000 for > > > client /192.168.0.19:49712 > > > 2017-01-24 21:18:46,571 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > 192.168.0.19:49713 > > > 2017-01-24 21:18:46,578 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > 192.168.0.19:49713 > > > 2017-01-24 21:18:46,655 INFO [CommitProcessor:2] > server.ZooKeeperServer: > > > Established session 0x259d2099775000a with negotiated timeout 90000 for > > > client /192.168.0.19:49713 > > > 2017-01-24 21:18:46,875 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > 192.168.0.19:49714 > > > 2017-01-24 21:18:46,877 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > 192.168.0.19:49714 > > > 2017-01-24 21:18:46,894 INFO [CommitProcessor:2] > server.ZooKeeperServer: > > > Established session 0x259d2099775000b with negotiated timeout 90000 for > > > client /192.168.0.19:49714 > > > 2017-01-24 21:29:04,319 WARN [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxn: caught end of stream exception > > > EndOfStreamException: Unable to read additional data from client > sessionid > > > 0x259d20997750006, likely client has closed socket > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > NIOServerCnxn.java:228) > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > NIOServerCnxnFactory.java:208) > > > at java.lang.Thread.run(Thread.java:745) > > > 2017-01-24 21:29:04,320 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxn: Closed socket connection for client / > > > 192.168.0.19:49709 which had sessionid 0x259d20997750006 > > > 2017-01-24 21:29:04,344 WARN [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxn: caught end of stream exception > > > EndOfStreamException: Unable to read additional data from client > sessionid > > > 0x259d2099775000a, likely client has closed socket > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > NIOServerCnxn.java:228) > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > NIOServerCnxnFactory.java:208) > > > at java.lang.Thread.run(Thread.java:745) > > > 2017-01-24 21:29:04,345 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxn: Closed socket connection for client / > > > 192.168.0.19:49713 which had sessionid 0x259d2099775000a > > > 2017-01-24 21:29:04,361 WARN [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxn: caught end of stream exception > > > EndOfStreamException: Unable to read additional data from client > sessionid > > > 0x259d20997750004, likely client has closed socket > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > NIOServerCnxn.java:228) > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > NIOServerCnxnFactory.java:208) > > > at java.lang.Thread.run(Thread.java:745) > > > 2017-01-24 21:29:04,361 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxn: Closed socket connection for client / > > > 192.168.0.19:49706 which had sessionid 0x259d20997750004 > > > 2017-01-24 21:29:04,381 WARN [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxn: caught end of stream exception > > > EndOfStreamException: Unable to read additional data from client > sessionid > > > 0x259d20997750008, likely client has closed socket > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > NIOServerCnxn.java:228) > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > NIOServerCnxnFactory.java:208) > > > at java.lang.Thread.run(Thread.java:745) > > > 2017-01-24 21:29:04,381 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxn: Closed socket connection for client / > > > 192.168.0.19:49711 which had sessionid 0x259d20997750008 > > > 2017-01-24 21:29:04,392 WARN [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxn: caught end of stream exception > > > EndOfStreamException: Unable to read additional data from client > sessionid > > > 0x259d2099775000b, likely client has closed socket > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > NIOServerCnxn.java:228) > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > NIOServerCnxnFactory.java:208) > > > at java.lang.Thread.run(Thread.java:745) > > > 2017-01-24 21:29:04,393 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxn: Closed socket connection for client / > > > 192.168.0.19:49714 which had sessionid 0x259d2099775000b > > > 2017-01-24 21:29:04,412 WARN [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxn: caught end of stream exception > > > EndOfStreamException: Unable to read additional data from client > sessionid > > > 0x259d20997750005, likely client has closed socket > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > NIOServerCnxn.java:228) > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > NIOServerCnxnFactory.java:208) > > > at java.lang.Thread.run(Thread.java:745) > > > 2017-01-24 21:29:04,413 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxn: Closed socket connection for client / > > > 192.168.0.19:49708 which had sessionid 0x259d20997750005 > > > 2017-01-24 21:29:04,425 WARN [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxn: caught end of stream exception > > > EndOfStreamException: Unable to read additional data from client > sessionid > > > 0x259d20997750007, likely client has closed socket > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > NIOServerCnxn.java:228) > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > NIOServerCnxnFactory.java:208) > > > at java.lang.Thread.run(Thread.java:745) > > > 2017-01-24 21:29:04,426 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxn: Closed socket connection for client / > > > 192.168.0.19:49710 which had sessionid 0x259d20997750007 > > > 2017-01-24 21:29:04,445 WARN [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxn: caught end of stream exception > > > EndOfStreamException: Unable to read additional data from client > sessionid > > > 0x259d20997750009, likely client has closed socket > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > NIOServerCnxn.java:228) > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > NIOServerCnxnFactory.java:208) > > > at java.lang.Thread.run(Thread.java:745) > > > 2017-01-24 21:29:04,446 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxn: Closed socket connection for client / > > > 192.168.0.19:49712 which had sessionid 0x259d20997750009 > > > 2017-01-24 21:29:10,042 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > 192.168.0.19:49761 > > > 2017-01-24 21:29:10,044 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > 192.168.0.19:49761 > > > 2017-01-24 21:29:10,051 INFO [CommitProcessor:2] > server.ZooKeeperServer: > > > Established session 0x259d2099775000c with negotiated timeout 90000 for > > > client /192.168.0.19:49761 > > > 2017-01-24 21:29:12,150 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > 192.168.0.19:49765 > > > 2017-01-24 21:29:12,152 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > 192.168.0.19:49765 > > > 2017-01-24 21:29:12,165 INFO [CommitProcessor:2] > server.ZooKeeperServer: > > > Established session 0x259d2099775000d with negotiated timeout 90000 for > > > client /192.168.0.19:49765 > > > 2017-01-24 21:29:13,332 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > 192.168.0.19:49769 > > > 2017-01-24 21:29:13,334 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > 192.168.0.19:49769 > > > 2017-01-24 21:29:13,343 INFO [CommitProcessor:2] > server.ZooKeeperServer: > > > Established session 0x259d2099775000e with negotiated timeout 90000 for > > > client /192.168.0.19:49769 > > > 2017-01-24 21:29:13,933 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > 192.168.0.19:49772 > > > 2017-01-24 21:29:13,935 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > 192.168.0.19:49772 > > > 2017-01-24 21:29:13,939 INFO [CommitProcessor:2] > server.ZooKeeperServer: > > > Established session 0x259d2099775000f with negotiated timeout 90000 for > > > client /192.168.0.19:49772 > > > 2017-01-24 21:39:34,294 WARN [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxn: caught end of stream exception > > > EndOfStreamException: Unable to read additional data from client > sessionid > > > 0x259d2099775000e, likely client has closed socket > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > NIOServerCnxn.java:228) > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > NIOServerCnxnFactory.java:208) > > > at java.lang.Thread.run(Thread.java:745) > > > 2017-01-24 21:39:34,295 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxn: Closed socket connection for client / > > > 192.168.0.19:49769 which had sessionid 0x259d2099775000e > > > 2017-01-24 21:39:34,314 WARN [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxn: caught end of stream exception > > > EndOfStreamException: Unable to read additional data from client > sessionid > > > 0x259d2099775000d, likely client has closed socket > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > NIOServerCnxn.java:228) > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > NIOServerCnxnFactory.java:208) > > > at java.lang.Thread.run(Thread.java:745) > > > 2017-01-24 21:39:34,315 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxn: Closed socket connection for client / > > > 192.168.0.19:49765 which had sessionid 0x259d2099775000d > > > 2017-01-24 21:39:34,343 WARN [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxn: caught end of stream exception > > > EndOfStreamException: Unable to read additional data from client > sessionid > > > 0x259d2099775000f, likely client has closed socket > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > NIOServerCnxn.java:228) > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > NIOServerCnxnFactory.java:208) > > > at java.lang.Thread.run(Thread.java:745) > > > 2017-01-24 21:39:34,343 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxn: Closed socket connection for client / > > > 192.168.0.19:49772 which had sessionid 0x259d2099775000f > > > 2017-01-24 21:39:34,366 WARN [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxn: caught end of stream exception > > > EndOfStreamException: Unable to read additional data from client > sessionid > > > 0x259d2099775000c, likely client has closed socket > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > NIOServerCnxn.java:228) > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > NIOServerCnxnFactory.java:208) > > > at java.lang.Thread.run(Thread.java:745) > > > 2017-01-24 21:39:34,367 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxn: Closed socket connection for client / > > > 192.168.0.19:49761 which had sessionid 0x259d2099775000c > > > 2017-01-24 21:39:38,640 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > 192.168.0.19:49817 > > > 2017-01-24 21:39:38,642 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > 192.168.0.19:49817 > > > 2017-01-24 21:39:38,647 INFO [CommitProcessor:2] > server.ZooKeeperServer: > > > Established session 0x259d20997750010 with negotiated timeout 90000 for > > > client /192.168.0.19:49817 > > > 2017-01-24 21:39:40,386 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > 192.168.0.19:49821 > > > 2017-01-24 21:39:40,392 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > 192.168.0.19:49821 > > > 2017-01-24 21:39:40,399 INFO [CommitProcessor:2] > server.ZooKeeperServer: > > > Established session 0x259d20997750011 with negotiated timeout 90000 for > > > client /192.168.0.19:49821 > > > 2017-01-24 21:39:41,708 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > 192.168.0.19:49825 > > > 2017-01-24 21:39:41,712 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > 192.168.0.19:49825 > > > 2017-01-24 21:39:41,716 INFO [CommitProcessor:2] > server.ZooKeeperServer: > > > Established session 0x259d20997750012 with negotiated timeout 90000 for > > > client /192.168.0.19:49825 > > > 2017-01-24 21:39:42,420 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > 192.168.0.19:49828 > > > 2017-01-24 21:39:42,421 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > 192.168.0.19:49828 > > > 2017-01-24 21:39:42,426 INFO [CommitProcessor:2] > server.ZooKeeperServer: > > > Established session 0x259d20997750013 with negotiated timeout 90000 for > > > client /192.168.0.19:49828 > > > 2017-01-24 21:50:04,309 WARN [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxn: caught end of stream exception > > > EndOfStreamException: Unable to read additional data from client > sessionid > > > 0x259d20997750013, likely client has closed socket > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > NIOServerCnxn.java:228) > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > NIOServerCnxnFactory.java:208) > > > at java.lang.Thread.run(Thread.java:745) > > > 2017-01-24 21:50:04,309 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxn: Closed socket connection for client / > > > 192.168.0.19:49828 which had sessionid 0x259d20997750013 > > > 2017-01-24 21:50:04,328 WARN [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxn: caught end of stream exception > > > EndOfStreamException: Unable to read additional data from client > sessionid > > > 0x259d20997750012, likely client has closed socket > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > NIOServerCnxn.java:228) > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > NIOServerCnxnFactory.java:208) > > > at java.lang.Thread.run(Thread.java:745) > > > 2017-01-24 21:50:04,329 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxn: Closed socket connection for client / > > > 192.168.0.19:49825 which had sessionid 0x259d20997750012 > > > 2017-01-24 21:50:04,384 WARN [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxn: caught end of stream exception > > > EndOfStreamException: Unable to read additional data from client > sessionid > > > 0x259d20997750010, likely client has closed socket > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > NIOServerCnxn.java:228) > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > NIOServerCnxnFactory.java:208) > > > at java.lang.Thread.run(Thread.java:745) > > > 2017-01-24 21:50:04,387 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxn: Closed socket connection for client / > > > 192.168.0.19:49817 which had sessionid 0x259d20997750010 > > > 2017-01-24 21:50:04,396 WARN [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxn: caught end of stream exception > > > EndOfStreamException: Unable to read additional data from client > sessionid > > > 0x259d20997750011, likely client has closed socket > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > NIOServerCnxn.java:228) > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > NIOServerCnxnFactory.java:208) > > > at java.lang.Thread.run(Thread.java:745) > > > 2017-01-24 21:50:04,397 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxn: Closed socket connection for client / > > > 192.168.0.19:49821 which had sessionid 0x259d20997750011 > > > 2017-01-24 21:50:08,380 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > 192.168.0.19:49857 > > > 2017-01-24 21:50:08,382 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > 192.168.0.19:49857 > > > 2017-01-24 21:50:08,387 INFO [CommitProcessor:2] > server.ZooKeeperServer: > > > Established session 0x259d20997750014 with negotiated timeout 90000 for > > > client /192.168.0.19:49857 > > > 2017-01-24 21:50:09,791 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > 192.168.0.19:49862 > > > 2017-01-24 21:50:09,792 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > 192.168.0.19:49862 > > > 2017-01-24 21:50:09,799 INFO [CommitProcessor:2] > server.ZooKeeperServer: > > > Established session 0x259d20997750015 with negotiated timeout 90000 for > > > client /192.168.0.19:49862 > > > 2017-01-24 21:50:11,381 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > 192.168.0.19:49866 > > > 2017-01-24 21:50:11,383 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > 192.168.0.19:49866 > > > 2017-01-24 21:50:11,390 INFO [CommitProcessor:2] > server.ZooKeeperServer: > > > Established session 0x259d20997750016 with negotiated timeout 90000 for > > > client /192.168.0.19:49866 > > > 2017-01-24 21:50:12,189 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > 192.168.0.19:49869 > > > 2017-01-24 21:50:12,191 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > 192.168.0.19:49869 > > > 2017-01-24 21:50:12,196 INFO [CommitProcessor:2] > server.ZooKeeperServer: > > > Established session 0x259d20997750017 with negotiated timeout 90000 for > > > client /192.168.0.19:49869 > > > 2017-01-24 22:00:34,306 WARN [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxn: caught end of stream exception > > > EndOfStreamException: Unable to read additional data from client > sessionid > > > 0x259d20997750015, likely client has closed socket > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > NIOServerCnxn.java:228) > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > NIOServerCnxnFactory.java:208) > > > at java.lang.Thread.run(Thread.java:745) > > > 2017-01-24 22:00:34,307 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxn: Closed socket connection for client / > > > 192.168.0.19:49862 which had sessionid 0x259d20997750015 > > > 2017-01-24 22:00:34,330 WARN [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxn: caught end of stream exception > > > EndOfStreamException: Unable to read additional data from client > sessionid > > > 0x259d20997750017, likely client has closed socket > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > NIOServerCnxn.java:228) > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > NIOServerCnxnFactory.java:208) > > > at java.lang.Thread.run(Thread.java:745) > > > 2017-01-24 22:00:34,337 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxn: Closed socket connection for client / > > > 192.168.0.19:49869 which had sessionid 0x259d20997750017 > > > 2017-01-24 22:00:34,355 WARN [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxn: caught end of stream exception > > > EndOfStreamException: Unable to read additional data from client > sessionid > > > 0x259d20997750016, likely client has closed socket > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > NIOServerCnxn.java:228) > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > NIOServerCnxnFactory.java:208) > > > at java.lang.Thread.run(Thread.java:745) > > > 2017-01-24 22:00:34,356 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxn: Closed socket connection for client / > > > 192.168.0.19:49866 which had sessionid 0x259d20997750016 > > > 2017-01-24 22:00:34,369 WARN [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxn: caught end of stream exception > > > EndOfStreamException: Unable to read additional data from client > sessionid > > > 0x259d20997750014, likely client has closed socket > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > NIOServerCnxn.java:228) > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > NIOServerCnxnFactory.java:208) > > > at java.lang.Thread.run(Thread.java:745) > > > 2017-01-24 22:00:34,369 INFO [NIOServerCxn.Factory:0.0.0.0/ > 0.0.0.0:2181] > > > server.NIOServerCnxn: Closed socket connection for client / > > > 192.168.0.19:49857 which had sessionid 0x259d20997750014 > > > > > > ---------- > > > > > > The RegionServer logs are clean, no warnings. I assume this is a > > > ZooKeeper problem more than HBase, but I tried many different > > > configurations and nothing worked. Does anyone have an idea what could > > > be happening? > > > > > > Best, > > > Hernán. > > > > > >
