Re: Working around Leader election Listner thread death

2016-08-31 Thread Guy Laden
Hi Flavio, I think your idea of using iptables should help. I hope to have time to experiment with it. Thanks for your help. Guy On Wed, Aug 31, 2016 at 11:15 AM, Flavio Junqueira wrote: > Ok, I think I get what you're saying. Perhaps you're missing that this is > an issue that Guy encountered

Re: Working around Leader election Listner thread death

2016-08-31 Thread David Brower
OK, makes sense. On 8/31/2016 1:15 AM, Flavio Junqueira wrote: Ok, I think I get what you're saying. Perhaps you're missing that this is an issue that Guy encountered in 3.4.6 and that is fixed in a later release. We are discussing here a workaround for his 3.4.6 deployment, not a permanent s

Re: Working around Leader election Listner thread death

2016-08-31 Thread Flavio Junqueira
Ok, I think I get what you're saying. Perhaps you're missing that this is an issue that Guy encountered in 3.4.6 and that is fixed in a later release. We are discussing here a workaround for his 3.4.6 deployment, not a permanent solution. Does it make sense? -Flavio > On 31 Aug 2016, at 01:16

Re: Working around Leader election Listner thread death

2016-08-30 Thread David Brower
You'd be programming iptables to pass/accept things from a whitelist of peers you're willing to talk with. If you've got such a whitelist, you don't need to program iptables to look at the peer address from a packet/socket and drop it, you can just do it in your message processing code. The

Re: Working around Leader election Listner thread death

2016-08-30 Thread Flavio Junqueira
I'm not sure what you're suggesting, David. Could you be more specific, please? -Flavio > On 30 Aug 2016, at 23:54, David Brower wrote: > > Anything you could do with iptables you can do in the process by having it > drop connections from things not on a whitelist, and not having a thread > w

Re: Working around Leader election Listner thread death

2016-08-30 Thread David Brower
Anything you could do with iptables you can do in the process by having it drop connections from things not on a whitelist, and not having a thread waiting indefinitely for operations from any connection. -dB On 8/30/2016 2:46 PM, Flavio Junqueira wrote: I was trying to write down an analysi

Re: Working around Leader election Listner thread death

2016-08-30 Thread Flavio Junqueira
I was trying to write down an analysis and I haven't been able to come up with anything that is foolproof. Basically, the two main issues are: - A bad server is able to connect to a good server in the case it has a message outstanding and is trying to establish a connection to the good server. T

Re: Working around Leader election Listner thread death

2016-08-29 Thread Guy Laden
Hi Flavio, Thanks for your reply. The situation is that indeed all the servers are in a bad state so it looks like we will have to perform a cluster restart. We played with attempts to optimize the downtime along the lines you suggested. In testing it we ran into the issue where a server with no L

Re: Working around Leader election Listner thread death

2016-08-28 Thread Flavio Junqueira
Hi Guy, We don't have a way to restart the listener thread, so you really need to bounce the server. I don't think there is a way of doing this without forcing a leader election, assuming all your servers are in this bad state. To minimize downtime, one thing you can do is to avoid bouncing the

Working around Leader election Listner thread death

2016-08-24 Thread Guy Laden
Hi all, It looks like due to a security scan sending "bad" traffic to the leader election port, we have clusters in which the leader election Listener thread is dead (unchecked exception was thrown and thread died - seen in the log). (This seems to be fixed by fixed in https://issues.apache.org/ji