created a parallel independant zookeeper cluster on the same set of machines with different ports and that worked. This indicates the port was the issue.
On Mon, Apr 11, 2016 at 1:35 PM, s influxdb <elastic....@gmail.com> wrote: > reboot of the server didn't help > > On Thu, Apr 7, 2016 at 6:50 PM, s influxdb <elastic....@gmail.com> wrote: > >> I ran tcpdump on all the three nodes. >> It looks like that for every [PSH, ACK] there is a missing [ACK] from >> the other nodes to this 2nd node on port 3888. >> >> >> On Thu, Apr 7, 2016 at 1:29 PM, s influxdb <elastic....@gmail.com> wrote: >> >>> Thanks Flavio for your quick replies. >>> The zookeeper version is 3.4.6 >>> >>> >>> >>> On Thu, Apr 7, 2016 at 1:23 PM, Flavio P JUNQUEIRA <f...@apache.org> >>> wrote: >>> >>>> You need to determine why it is not receiving notification messages. >>>> From >>>> the information you've given, it doesn't look like a zookeeper code >>>> issue. >>>> >>>> BTW, which version are you using? >>>> >>>> -Flavio >>>> On 7 Apr 2016 21:20, "s influxdb" <elastic....@gmail.com> wrote: >>>> >>>> > nothin on the iptables firewall . >>>> > >>>> > What options do i have to reconnect this node to the cluster ? >>>> > >>>> > >>>> > On Thu, Apr 7, 2016 at 10:14 AM, s influxdb <elastic....@gmail.com> >>>> wrote: >>>> > >>>> > > telnet works on 2888 and 3888 to the other nodes. Now i see >>>> > > java.net.SocketTimeoutException: connect timed out messages in the >>>> logs >>>> > for >>>> > > node 2 >>>> > > >>>> > > On Thu, Apr 7, 2016 at 3:05 AM, Flavio Junqueira <f...@apache.org> >>>> wrote: >>>> > > >>>> > >> I only see notifications from the node to itself. It says that it >>>> is >>>> > >> connected to 1, but it doesn't seem to be receiving the >>>> notification >>>> > from >>>> > >> 1. It also doesn't seem to be receiving the connection request >>>> from 3. >>>> > >> >>>> > >> Last time I've seen something like this was due to iptables rules, >>>> but >>>> > if >>>> > >> it was working before and no configuration has changed, then I >>>> don't >>>> > know >>>> > >> what it could be. >>>> > >> >>>> > >> -Flavio >>>> > >> >>>> > >> > On 07 Apr 2016, at 05:43, s influxdb <elastic....@gmail.com> >>>> wrote: >>>> > >> > >>>> > >> > this is the pastie >>>> > >> > http://pastie.org/10788301 >>>> > >> > >>>> > >> > On Wed, Apr 6, 2016 at 9:41 PM, s influxdb < >>>> elastic....@gmail.com> >>>> > >> wrote: >>>> > >> > >>>> > >> >> We had one of the node giving OOM java.lang.OutOfMemoryError: >>>> unable >>>> > to >>>> > >> >> create new native thread and then being unresponsive. >>>> > >> >> >>>> > >> >> We tried to add the node back to the cluster but with no luck. >>>> > >> >> >>>> > >> >> It doesn't seem to "Receive any notification " messages from >>>> the >>>> > other >>>> > >> >> nodes. >>>> > >> >> Keeps "Sending notifications " in loop >>>> > >> >> >>>> > >> >> Please see attached the logs of the node that is out of >>>> rotation. >>>> > >> >> >>>> > >> >> Any inputs appreciated. >>>> > >> >> >>>> > >> >> Thanks >>>> > >> >> >>>> > >> >>>> > >> >>>> > > >>>> > >>>> >>> >>> >> >
