Hello all, I have recently updated my Storm cluster to use an ensemble of zookeeper servers (3 is the exact number). I am running my VMs on EC2, and in order to avoid updating the configuration files I used Elastic IPs for each of the Nimbus and ZooKeeper servers (every time an instance restarts, its IP address changes).
However, when I use an Elastic IP ZooKeeper server which are part of an ensemble (the election and ensemble ports need to be assigned), for some reason, the binding of the election address is unsuccessful. In fact, I have followed the instructions for setting up a proper zookeeper ensemble, and my configuration files are the following: # The number of milliseconds of each tick tickTime=2000 # The number of ticks that the initial # synchronization phase can take initLimit=5 # The number of ticks that can pass between # sending a request and getting an acknowledgement syncLimit=2 # the directory where the snapshot is stored. # do not use /tmp for storage, /tmp here is just # example sakes. dataDir=/var/lib/zookeeper # the port at which the clients will connect clientPort=2181 # the maximum number of client connections. # increase this if you need to handle more clients #maxClientCnxns=60 # # Be sure to read the maintenance section of the # administrator guide before turning on autopurge. # # http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance # # The number of snapshots to retain in dataDir #autopurge.snapRetainCount=3 # Purge task interval in hours # Set to "0" to disable auto purge feature #autopurge.purgeInterval=1 server.1=XXX.XX.XX.XX:2888:3888 server.2=YYY.YY.YY.YYY:2889:3889 server.3=ZZZ.ZZ.ZZ.ZZZ:2890:3890 where XXX.XX.XX.XX... are the elastic-IPs of my servers. In fact, when I start my ensemble with elastic IPs I get the following: ubuntu@ip-XXX-XX-XX-XX:/opt/zookeeper-3.4.6$ ./bin/zkServer.sh start-foreground JMX enabled by default Using config: /opt/zookeeper-3.4.6/bin/../conf/zoo.cfg 2015-06-18 17:16:07,596 [myid:] - INFO [main:QuorumPeerConfig@103] - Reading configuration from: /opt/zookeeper-3.4.6/bin/../conf/zoo.cfg 2015-06-18 17:16:07,600 [myid:] - INFO [main:QuorumPeerConfig@340] - Defaulting to majority quorums 2015-06-18 17:16:07,603 [myid:1] - INFO [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3 2015-06-18 17:16:07,604 [myid:1] - INFO [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0 2015-06-18 17:16:07,605 [myid:1] - INFO [main:DatadirCleanupManager@101] - Purge task is not scheduled. 2015-06-18 17:16:07,614 [myid:1] - INFO [main:QuorumPeerMain@127] - Starting quorum peer 2015-06-18 17:16:07,626 [myid:1] - INFO [main:NIOServerCnxnFactory@94] - binding to port 0.0.0.0/0.0.0.0:2181 2015-06-18 17:16:07,652 [myid:1] - INFO [main:QuorumPeer@959] - tickTime set to 2000 2015-06-18 17:16:07,652 [myid:1] - INFO [main:QuorumPeer@979] - minSessionTimeout set to -1 2015-06-18 17:16:07,652 [myid:1] - INFO [main:QuorumPeer@990] - maxSessionTimeout set to -1 2015-06-18 17:16:07,652 [myid:1] - INFO [main:QuorumPeer@1005] - initLimit set to 5 2015-06-18 17:16:07,676 [myid:1] - INFO [Thread-1:QuorumCnxManager$Listener@504] - My election bind port: /XXX.XX.XX.XX:3888 2015-06-18 17:16:07,677 [myid:1] - ERROR [/XXX.XX.XX.XX:3888:QuorumCnxManager$Listener@517] - Exception while listening java.net.BindException: Cannot assign requested address at java.net.PlainSocketImpl.socketBind(Native Method) at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376) at java.net.ServerSocket.bind(ServerSocket.java:376) at java.net.ServerSocket.bind(ServerSocket.java:330) at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:507) 2015-06-18 17:16:07,688 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumPeer@714] - LOOKING By the way, I have checked the AWS security group I am using and the proper rules have been set. Also, I tried to see (with netstat) if ports 2888 and 3888 are used, and they were not when I tried the above. Has anybody else had the same issue? If yes, did you manage to work around it and create your cluster with elastic IPs? It would be really awesome if I could use the elastic-IPs so that I do not have to update my IPs every time I restart my instances. Thank you, Nick P.S.: I have extensively searched on the web for answers, but not real solution exists.
