Re: how to force curernt session when active is down
Yep, that did the trick. No need for short rabbit client-level heartbeat. Thank you, Andrei On Sat, Jan 18, 2014 at 11:23 AM, Baptiste wrote: > Hi, > > You may want to try "on-marked-down shutdown-sessions". > I use it on long applications which require TCP connection established > for a long time. > > Baptiste > > > On Fri, Jan 17, 2014 at 9:25 AM, Andrei Chevenkov > wrote: > > Got it, thank you. > > > > On 17/01/2014 9:01 pm, "Dmitriy Samsonov" > > wrote: > >> > >> When you closing rabbit gracefully it sends 'DOWN' message to > >> connected clients and you notice your clients reconnecting. > >> I've tried in the past various ways to detect disconnects on HAProxy > >> side, including playing with various TCPKA settings but nothing > >> helped. (I was playing with 1.4) > >> > >> 2014/1/17 Andrei Chevenkov : > >> > Dmitry, thank you for the reply, but I would imagine that haproxy > would > >> > close all sessions on the DOWN nodes, regardless of the type of client > >> > connecting and the protocol? Can this be enforced? > >> > > >> > I did implement producer ack, but that slows down publishes big time. > >> > Have > >> > also tried heartbeat and that did help, but this is initiated by the > >> > client > >> > and I would like to see if haproxy can manage this. I.e. close all > >> > sessions > >> > on a node that is marked as DOWN. > >> > > >> > On 17/01/2014 8:09 pm, "Dmitriy Samsonov" > > >> > wrote: > >> >> > >> >> Hi! > >> >> > >> >> That's related to rabbiit's default heartbeat timeout. You can set it > >> >> to > >> >> lower value when connecting. Also there is a way to check if message > >> >> was > >> >> actually delivered to the broker, take a look at rabbiit's docs. > >> >> > >> >> > >> >> пятница, 17 января 2014 г. пользователь Andrei Chevenkov написал: > >> >>> > >> >>> Hi, > >> >>> > >> >>> I am pretty new to haproxy and trying to do ha setup for rabbitmq. > >> >>> > >> >>> Setup is pretyt simple: > >> >>> > >> >>> listen rabbitmq 192.168.69.106:5672 > >> >>> modetcp > >> >>> balance leastconn > >> >>> option tcplog > >> >>> option tcpka > >> >>> server rabbit01 192.168.69.107:5672 check inter 1000 downinter > 5000 > >> >>> fall 1 on-error mark-down > >> >>> server rabbit02 192.168.69.108:5672 check inter 1000 downinter > 5000 > >> >>> fall 1 on-error mark-down backup > >> >>> > >> >>> Everything seems to work well, but I noticed one strange behaviour. > If > >> >>> I > >> >>> gracefully shut down rabbit01, message producer (client) connected > to > >> >>> 192.168.69.106:5672 notices broken connection pretty much straight > >> >>> away, > >> >>> attempts a reconnect, hits rabbit02 and all is well. > >> >>> > >> >>> However if I simply "turn off" rabbit01, message producer (client) > >> >>> notices broken connection only in around 30 seconds or so. And while > >> >>> the > >> >>> break is undetected it keeps pushing messages thinking there is > still > >> >>> someone on another side. > >> >>> > >> >>> Now, from the statistic report I can see that as soon as I turn off > >> >>> the > >> >>> box, rabbit01 is marked as "DOWN", but its "Sessions Current" is > still > >> >>> set > >> >>> to 1? In the logs this comes up pretty much immediately after the > hard > >> >>> turn > >> >>> off: > >> >>> > >> >>> Jan 17 17:13:51 prodlb01 haproxy[38459]: Server rabbitmq/rabbit01 is > >> >>> DOWN, reason: Layer4 timeout, check duration: 1008ms. 0 active and 1 > >> >>> backup > >> >>> servers left. Running on backup. 1 sessions active, 0 requeued, 0 > >> >>> remaining > >> >>> in queue. > >> >>> > >> >>> After around 30 seconds, the connection is detected as broken and > >> >>> failover happens successfully. At that time "Sessions Current" is > >> >>> set to 0 > >> >>> fo rabbit01. > >> >>> > >> >>> Tried option nolinger and option abortonclose, but no luck. > >> >>> > >> >>> Any help would be much appreciated... > >> >>> > >> >>> Cheers, > >> >>> Andrei >
Re: how to force curernt session when active is down
They are in cluster and it's definitely to do with timeout on rabbit(ampq) clients, but I was after the HAProxy solution. I.e. I wanted all sessions to a DOWN node killed immediately when HAProxy detects that a node has gone DOWN. Baptiste's reply seems to hit the spot with this new feature in 1.5 http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#on-marked-down I can also see that there is http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#5.2-on-marked-up That is also very useful for my scenario as I would like all connections to failover back to rabbit01 when it goes back up. Resetting the connection to BACKUP will cause my client to reconnect the rabbit01 that's just came up. On Sat, Jan 18, 2014 at 11:56 AM, Thomas Spicer wrote: > This may more to due with Rabbit (or AMPQ clients) than HAProxy. Do you > have Rabbit 01 and 02 in a cluster or are they separate nodes? > > > On Jan 16, 2014, at 11:28 PM, Andrei Chevenkov wrote: > > Hi, > > I am pretty new to haproxy and trying to do ha setup for rabbitmq. > > Setup is pretyt simple: > > listen rabbitmq 192.168.69.106:5672 > modetcp > balance leastconn > option tcplog > option tcpka > server rabbit01 192.168.69.107:5672 check inter 1000 downinter 5000 > fall 1 on-error mark-down > server rabbit02 192.168.69.108:5672 check inter 1000 downinter 5000 > fall 1 on-error mark-down backup > > Everything seems to work well, but I noticed one strange behaviour. If I > gracefully shut down rabbit01, message producer (client) connected to > 192.168.69.106:5672 notices broken connection pretty much straight away, > attempts a reconnect, hits rabbit02 and all is well. > > However if I simply "turn off" rabbit01, message producer (client) notices > broken connection only in around 30 seconds or so. And while the break is > undetected it keeps pushing messages thinking there is still someone on > another side. > > Now, from the statistic report I can see that as soon as I turn off the > box, rabbit01 is marked as "DOWN", but its "Sessions Current" is still set > to 1? In the logs this comes up pretty much immediately after the hard turn > off: > > Jan 17 17:13:51 prodlb01 haproxy[38459]: Server rabbitmq/rabbit01 is DOWN, > reason: Layer4 timeout, check duration: 1008ms. 0 active and 1 backup > servers left. Running on backup. 1 sessions active, 0 requeued, 0 remaining > in queue. > > After around 30 seconds, the connection is detected as broken and failover > happens successfully. At that time "Sessions Current" is set to 0 fo > rabbit01. > > Tried option nolinger and option abortonclose, but no luck. > > Any help would be much appreciated... > > Cheers, > Andrei > > >
Re: how to force curernt session when active is down
This may more to due with Rabbit (or AMPQ clients) than HAProxy. Do you have Rabbit 01 and 02 in a cluster or are they separate nodes? On Jan 16, 2014, at 11:28 PM, Andrei Chevenkov wrote: > Hi, > > I am pretty new to haproxy and trying to do ha setup for rabbitmq. > > Setup is pretyt simple: > > listen rabbitmq 192.168.69.106:5672 > modetcp > balance leastconn > option tcplog > option tcpka > server rabbit01 192.168.69.107:5672 check inter 1000 downinter 5000 fall 1 > on-error mark-down > server rabbit02 192.168.69.108:5672 check inter 1000 downinter 5000 fall 1 > on-error mark-down backup > > Everything seems to work well, but I noticed one strange behaviour. If I > gracefully shut down rabbit01, message producer (client) connected to > 192.168.69.106:5672 notices broken connection pretty much straight away, > attempts a reconnect, hits rabbit02 and all is well. > > However if I simply "turn off" rabbit01, message producer (client) notices > broken connection only in around 30 seconds or so. And while the break is > undetected it keeps pushing messages thinking there is still someone on > another side. > > Now, from the statistic report I can see that as soon as I turn off the box, > rabbit01 is marked as "DOWN", but its "Sessions Current" is still set to 1? > In the logs this comes up pretty much immediately after the hard turn off: > > Jan 17 17:13:51 prodlb01 haproxy[38459]: Server rabbitmq/rabbit01 is DOWN, > reason: Layer4 timeout, check duration: 1008ms. 0 active and 1 backup servers > left. Running on backup. 1 sessions active, 0 requeued, 0 remaining in queue. > > After around 30 seconds, the connection is detected as broken and failover > happens successfully. At that time "Sessions Current" is set to 0 fo > rabbit01. > > Tried option nolinger and option abortonclose, but no luck. > > Any help would be much appreciated... > > Cheers, > Andrei
Re: how to force curernt session when active is down
Thank you! This looks exactly what I was after. Sadly this seems to be only available in 1.5. It will take a little while for me to test. Will let you know if that worked with rabbit once tested. On Sat, Jan 18, 2014 at 11:23 AM, Baptiste wrote: > Hi, > > You may want to try "on-marked-down shutdown-sessions". > I use it on long applications which require TCP connection established > for a long time. > > Baptiste > > > On Fri, Jan 17, 2014 at 9:25 AM, Andrei Chevenkov > wrote: > > Got it, thank you. > > > > On 17/01/2014 9:01 pm, "Dmitriy Samsonov" > > wrote: > >> > >> When you closing rabbit gracefully it sends 'DOWN' message to > >> connected clients and you notice your clients reconnecting. > >> I've tried in the past various ways to detect disconnects on HAProxy > >> side, including playing with various TCPKA settings but nothing > >> helped. (I was playing with 1.4) > >> > >> 2014/1/17 Andrei Chevenkov : > >> > Dmitry, thank you for the reply, but I would imagine that haproxy > would > >> > close all sessions on the DOWN nodes, regardless of the type of client > >> > connecting and the protocol? Can this be enforced? > >> > > >> > I did implement producer ack, but that slows down publishes big time. > >> > Have > >> > also tried heartbeat and that did help, but this is initiated by the > >> > client > >> > and I would like to see if haproxy can manage this. I.e. close all > >> > sessions > >> > on a node that is marked as DOWN. > >> > > >> > On 17/01/2014 8:09 pm, "Dmitriy Samsonov" > > >> > wrote: > >> >> > >> >> Hi! > >> >> > >> >> That's related to rabbiit's default heartbeat timeout. You can set it > >> >> to > >> >> lower value when connecting. Also there is a way to check if message > >> >> was > >> >> actually delivered to the broker, take a look at rabbiit's docs. > >> >> > >> >> > >> >> пятница, 17 января 2014 г. пользователь Andrei Chevenkov написал: > >> >>> > >> >>> Hi, > >> >>> > >> >>> I am pretty new to haproxy and trying to do ha setup for rabbitmq. > >> >>> > >> >>> Setup is pretyt simple: > >> >>> > >> >>> listen rabbitmq 192.168.69.106:5672 > >> >>> modetcp > >> >>> balance leastconn > >> >>> option tcplog > >> >>> option tcpka > >> >>> server rabbit01 192.168.69.107:5672 check inter 1000 downinter > 5000 > >> >>> fall 1 on-error mark-down > >> >>> server rabbit02 192.168.69.108:5672 check inter 1000 downinter > 5000 > >> >>> fall 1 on-error mark-down backup > >> >>> > >> >>> Everything seems to work well, but I noticed one strange behaviour. > If > >> >>> I > >> >>> gracefully shut down rabbit01, message producer (client) connected > to > >> >>> 192.168.69.106:5672 notices broken connection pretty much straight > >> >>> away, > >> >>> attempts a reconnect, hits rabbit02 and all is well. > >> >>> > >> >>> However if I simply "turn off" rabbit01, message producer (client) > >> >>> notices broken connection only in around 30 seconds or so. And while > >> >>> the > >> >>> break is undetected it keeps pushing messages thinking there is > still > >> >>> someone on another side. > >> >>> > >> >>> Now, from the statistic report I can see that as soon as I turn off > >> >>> the > >> >>> box, rabbit01 is marked as "DOWN", but its "Sessions Current" is > still > >> >>> set > >> >>> to 1? In the logs this comes up pretty much immediately after the > hard > >> >>> turn > >> >>> off: > >> >>> > >> >>> Jan 17 17:13:51 prodlb01 haproxy[38459]: Server rabbitmq/rabbit01 is > >> >>> DOWN, reason: Layer4 timeout, check duration: 1008ms. 0 active and 1 > >> >>> backup > >> >>> servers left. Running on backup. 1 sessions active, 0 requeued, 0 > >> >>> remaining > >> >>> in queue. > >> >>> > >> >>> After around 30 seconds, the connection is detected as broken and > >> >>> failover happens successfully. At that time "Sessions Current" is > >> >>> set to 0 > >> >>> fo rabbit01. > >> >>> > >> >>> Tried option nolinger and option abortonclose, but no luck. > >> >>> > >> >>> Any help would be much appreciated... > >> >>> > >> >>> Cheers, > >> >>> Andrei >
Re: how to force curernt session when active is down
Hi, You may want to try "on-marked-down shutdown-sessions". I use it on long applications which require TCP connection established for a long time. Baptiste On Fri, Jan 17, 2014 at 9:25 AM, Andrei Chevenkov wrote: > Got it, thank you. > > On 17/01/2014 9:01 pm, "Dmitriy Samsonov" > wrote: >> >> When you closing rabbit gracefully it sends 'DOWN' message to >> connected clients and you notice your clients reconnecting. >> I've tried in the past various ways to detect disconnects on HAProxy >> side, including playing with various TCPKA settings but nothing >> helped. (I was playing with 1.4) >> >> 2014/1/17 Andrei Chevenkov : >> > Dmitry, thank you for the reply, but I would imagine that haproxy would >> > close all sessions on the DOWN nodes, regardless of the type of client >> > connecting and the protocol? Can this be enforced? >> > >> > I did implement producer ack, but that slows down publishes big time. >> > Have >> > also tried heartbeat and that did help, but this is initiated by the >> > client >> > and I would like to see if haproxy can manage this. I.e. close all >> > sessions >> > on a node that is marked as DOWN. >> > >> > On 17/01/2014 8:09 pm, "Dmitriy Samsonov" >> > wrote: >> >> >> >> Hi! >> >> >> >> That's related to rabbiit's default heartbeat timeout. You can set it >> >> to >> >> lower value when connecting. Also there is a way to check if message >> >> was >> >> actually delivered to the broker, take a look at rabbiit's docs. >> >> >> >> >> >> пятница, 17 января 2014 г. пользователь Andrei Chevenkov написал: >> >>> >> >>> Hi, >> >>> >> >>> I am pretty new to haproxy and trying to do ha setup for rabbitmq. >> >>> >> >>> Setup is pretyt simple: >> >>> >> >>> listen rabbitmq 192.168.69.106:5672 >> >>> modetcp >> >>> balance leastconn >> >>> option tcplog >> >>> option tcpka >> >>> server rabbit01 192.168.69.107:5672 check inter 1000 downinter 5000 >> >>> fall 1 on-error mark-down >> >>> server rabbit02 192.168.69.108:5672 check inter 1000 downinter 5000 >> >>> fall 1 on-error mark-down backup >> >>> >> >>> Everything seems to work well, but I noticed one strange behaviour. If >> >>> I >> >>> gracefully shut down rabbit01, message producer (client) connected to >> >>> 192.168.69.106:5672 notices broken connection pretty much straight >> >>> away, >> >>> attempts a reconnect, hits rabbit02 and all is well. >> >>> >> >>> However if I simply "turn off" rabbit01, message producer (client) >> >>> notices broken connection only in around 30 seconds or so. And while >> >>> the >> >>> break is undetected it keeps pushing messages thinking there is still >> >>> someone on another side. >> >>> >> >>> Now, from the statistic report I can see that as soon as I turn off >> >>> the >> >>> box, rabbit01 is marked as "DOWN", but its "Sessions Current" is still >> >>> set >> >>> to 1? In the logs this comes up pretty much immediately after the hard >> >>> turn >> >>> off: >> >>> >> >>> Jan 17 17:13:51 prodlb01 haproxy[38459]: Server rabbitmq/rabbit01 is >> >>> DOWN, reason: Layer4 timeout, check duration: 1008ms. 0 active and 1 >> >>> backup >> >>> servers left. Running on backup. 1 sessions active, 0 requeued, 0 >> >>> remaining >> >>> in queue. >> >>> >> >>> After around 30 seconds, the connection is detected as broken and >> >>> failover happens successfully. At that time "Sessions Current" is >> >>> set to 0 >> >>> fo rabbit01. >> >>> >> >>> Tried option nolinger and option abortonclose, but no luck. >> >>> >> >>> Any help would be much appreciated... >> >>> >> >>> Cheers, >> >>> Andrei
Re: how to force curernt session when active is down
Got it, thank you. On 17/01/2014 9:01 pm, "Dmitriy Samsonov" wrote: > When you closing rabbit gracefully it sends 'DOWN' message to > connected clients and you notice your clients reconnecting. > I've tried in the past various ways to detect disconnects on HAProxy > side, including playing with various TCPKA settings but nothing > helped. (I was playing with 1.4) > > 2014/1/17 Andrei Chevenkov : > > Dmitry, thank you for the reply, but I would imagine that haproxy would > > close all sessions on the DOWN nodes, regardless of the type of client > > connecting and the protocol? Can this be enforced? > > > > I did implement producer ack, but that slows down publishes big time. > Have > > also tried heartbeat and that did help, but this is initiated by the > client > > and I would like to see if haproxy can manage this. I.e. close all > sessions > > on a node that is marked as DOWN. > > > > On 17/01/2014 8:09 pm, "Dmitriy Samsonov" > > wrote: > >> > >> Hi! > >> > >> That's related to rabbiit's default heartbeat timeout. You can set it to > >> lower value when connecting. Also there is a way to check if message was > >> actually delivered to the broker, take a look at rabbiit's docs. > >> > >> > >> пятница, 17 января 2014 г. пользователь Andrei Chevenkov написал: > >>> > >>> Hi, > >>> > >>> I am pretty new to haproxy and trying to do ha setup for rabbitmq. > >>> > >>> Setup is pretyt simple: > >>> > >>> listen rabbitmq 192.168.69.106:5672 > >>> modetcp > >>> balance leastconn > >>> option tcplog > >>> option tcpka > >>> server rabbit01 192.168.69.107:5672 check inter 1000 downinter 5000 > >>> fall 1 on-error mark-down > >>> server rabbit02 192.168.69.108:5672 check inter 1000 downinter 5000 > >>> fall 1 on-error mark-down backup > >>> > >>> Everything seems to work well, but I noticed one strange behaviour. If > I > >>> gracefully shut down rabbit01, message producer (client) connected to > >>> 192.168.69.106:5672 notices broken connection pretty much straight > away, > >>> attempts a reconnect, hits rabbit02 and all is well. > >>> > >>> However if I simply "turn off" rabbit01, message producer (client) > >>> notices broken connection only in around 30 seconds or so. And while > the > >>> break is undetected it keeps pushing messages thinking there is still > >>> someone on another side. > >>> > >>> Now, from the statistic report I can see that as soon as I turn off the > >>> box, rabbit01 is marked as "DOWN", but its "Sessions Current" is still > set > >>> to 1? In the logs this comes up pretty much immediately after the hard > turn > >>> off: > >>> > >>> Jan 17 17:13:51 prodlb01 haproxy[38459]: Server rabbitmq/rabbit01 is > >>> DOWN, reason: Layer4 timeout, check duration: 1008ms. 0 active and 1 > backup > >>> servers left. Running on backup. 1 sessions active, 0 requeued, 0 > remaining > >>> in queue. > >>> > >>> After around 30 seconds, the connection is detected as broken and > >>> failover happens successfully. At that time "Sessions Current" is > set to 0 > >>> fo rabbit01. > >>> > >>> Tried option nolinger and option abortonclose, but no luck. > >>> > >>> Any help would be much appreciated... > >>> > >>> Cheers, > >>> Andrei >
Re: how to force curernt session when active is down
When you closing rabbit gracefully it sends 'DOWN' message to connected clients and you notice your clients reconnecting. I've tried in the past various ways to detect disconnects on HAProxy side, including playing with various TCPKA settings but nothing helped. (I was playing with 1.4) 2014/1/17 Andrei Chevenkov : > Dmitry, thank you for the reply, but I would imagine that haproxy would > close all sessions on the DOWN nodes, regardless of the type of client > connecting and the protocol? Can this be enforced? > > I did implement producer ack, but that slows down publishes big time. Have > also tried heartbeat and that did help, but this is initiated by the client > and I would like to see if haproxy can manage this. I.e. close all sessions > on a node that is marked as DOWN. > > On 17/01/2014 8:09 pm, "Dmitriy Samsonov" > wrote: >> >> Hi! >> >> That's related to rabbiit's default heartbeat timeout. You can set it to >> lower value when connecting. Also there is a way to check if message was >> actually delivered to the broker, take a look at rabbiit's docs. >> >> >> пятница, 17 января 2014 г. пользователь Andrei Chevenkov написал: >>> >>> Hi, >>> >>> I am pretty new to haproxy and trying to do ha setup for rabbitmq. >>> >>> Setup is pretyt simple: >>> >>> listen rabbitmq 192.168.69.106:5672 >>> modetcp >>> balance leastconn >>> option tcplog >>> option tcpka >>> server rabbit01 192.168.69.107:5672 check inter 1000 downinter 5000 >>> fall 1 on-error mark-down >>> server rabbit02 192.168.69.108:5672 check inter 1000 downinter 5000 >>> fall 1 on-error mark-down backup >>> >>> Everything seems to work well, but I noticed one strange behaviour. If I >>> gracefully shut down rabbit01, message producer (client) connected to >>> 192.168.69.106:5672 notices broken connection pretty much straight away, >>> attempts a reconnect, hits rabbit02 and all is well. >>> >>> However if I simply "turn off" rabbit01, message producer (client) >>> notices broken connection only in around 30 seconds or so. And while the >>> break is undetected it keeps pushing messages thinking there is still >>> someone on another side. >>> >>> Now, from the statistic report I can see that as soon as I turn off the >>> box, rabbit01 is marked as "DOWN", but its "Sessions Current" is still set >>> to 1? In the logs this comes up pretty much immediately after the hard turn >>> off: >>> >>> Jan 17 17:13:51 prodlb01 haproxy[38459]: Server rabbitmq/rabbit01 is >>> DOWN, reason: Layer4 timeout, check duration: 1008ms. 0 active and 1 backup >>> servers left. Running on backup. 1 sessions active, 0 requeued, 0 remaining >>> in queue. >>> >>> After around 30 seconds, the connection is detected as broken and >>> failover happens successfully. At that time "Sessions Current" is set to 0 >>> fo rabbit01. >>> >>> Tried option nolinger and option abortonclose, but no luck. >>> >>> Any help would be much appreciated... >>> >>> Cheers, >>> Andrei
Re: how to force curernt session when active is down
Dmitry, thank you for the reply, but I would imagine that haproxy would close all sessions on the DOWN nodes, regardless of the type of client connecting and the protocol? Can this be enforced? I did implement producer ack, but that slows down publishes big time. Have also tried heartbeat and that did help, but this is initiated by the client and I would like to see if haproxy can manage this. I.e. close all sessions on a node that is marked as DOWN. On 17/01/2014 8:09 pm, "Dmitriy Samsonov" wrote: > Hi! > > That's related to rabbiit's default heartbeat timeout. You can set it to > lower value when connecting. Also there is a way to check if message was > actually delivered to the broker, take a look at rabbiit's docs. > > > пятница, 17 января 2014 г. пользователь Andrei Chevenkov написал: > >> Hi, >> >> I am pretty new to haproxy and trying to do ha setup for rabbitmq. >> >> Setup is pretyt simple: >> >> listen rabbitmq 192.168.69.106:5672 >> modetcp >> balance leastconn >> option tcplog >> option tcpka >> server rabbit01 192.168.69.107:5672 check inter 1000 downinter 5000 >> fall 1 on-error mark-down >> server rabbit02 192.168.69.108:5672 check inter 1000 downinter 5000 >> fall 1 on-error mark-down backup >> >> Everything seems to work well, but I noticed one strange behaviour. If I >> gracefully shut down rabbit01, message producer (client) connected to >> 192.168.69.106:5672 notices broken connection pretty much straight away, >> attempts a reconnect, hits rabbit02 and all is well. >> >> However if I simply "turn off" rabbit01, message producer (client) >> notices broken connection only in around 30 seconds or so. And while the >> break is undetected it keeps pushing messages thinking there is still >> someone on another side. >> >> Now, from the statistic report I can see that as soon as I turn off the >> box, rabbit01 is marked as "DOWN", but its "Sessions Current" is still set >> to 1? In the logs this comes up pretty much immediately after the hard turn >> off: >> >> Jan 17 17:13:51 prodlb01 haproxy[38459]: Server rabbitmq/rabbit01 is >> DOWN, reason: Layer4 timeout, check duration: 1008ms. 0 active and 1 backup >> servers left. Running on backup. 1 sessions active, 0 requeued, 0 remaining >> in queue. >> >> After around 30 seconds, the connection is detected as broken and >> failover happens successfully. At that time "Sessions Current" is set to >> 0 fo rabbit01. >> >> Tried option nolinger and option abortonclose, but no luck. >> >> Any help would be much appreciated... >> >> Cheers, >> Andrei >> >
Re: how to force curernt session when active is down
Hi! That's related to rabbiit's default heartbeat timeout. You can set it to lower value when connecting. Also there is a way to check if message was actually delivered to the broker, take a look at rabbiit's docs. пятница, 17 января 2014 г. пользователь Andrei Chevenkov написал: > Hi, > > I am pretty new to haproxy and trying to do ha setup for rabbitmq. > > Setup is pretyt simple: > > listen rabbitmq 192.168.69.106:5672 > modetcp > balance leastconn > option tcplog > option tcpka > server rabbit01 192.168.69.107:5672 check inter 1000 downinter 5000 > fall 1 on-error mark-down > server rabbit02 192.168.69.108:5672 check inter 1000 downinter 5000 > fall 1 on-error mark-down backup > > Everything seems to work well, but I noticed one strange behaviour. If I > gracefully shut down rabbit01, message producer (client) connected to > 192.168.69.106:5672 notices broken connection pretty much straight away, > attempts a reconnect, hits rabbit02 and all is well. > > However if I simply "turn off" rabbit01, message producer (client) notices > broken connection only in around 30 seconds or so. And while the break is > undetected it keeps pushing messages thinking there is still someone on > another side. > > Now, from the statistic report I can see that as soon as I turn off the > box, rabbit01 is marked as "DOWN", but its "Sessions Current" is still set > to 1? In the logs this comes up pretty much immediately after the hard turn > off: > > Jan 17 17:13:51 prodlb01 haproxy[38459]: Server rabbitmq/rabbit01 is DOWN, > reason: Layer4 timeout, check duration: 1008ms. 0 active and 1 backup > servers left. Running on backup. 1 sessions active, 0 requeued, 0 remaining > in queue. > > After around 30 seconds, the connection is detected as broken and failover > happens successfully. At that time "Sessions Current" is set to 0 fo > rabbit01. > > Tried option nolinger and option abortonclose, but no luck. > > Any help would be much appreciated... > > Cheers, > Andrei >
how to force curernt session when active is down
Hi, I am pretty new to haproxy and trying to do ha setup for rabbitmq. Setup is pretyt simple: listen rabbitmq 192.168.69.106:5672 modetcp balance leastconn option tcplog option tcpka server rabbit01 192.168.69.107:5672 check inter 1000 downinter 5000 fall 1 on-error mark-down server rabbit02 192.168.69.108:5672 check inter 1000 downinter 5000 fall 1 on-error mark-down backup Everything seems to work well, but I noticed one strange behaviour. If I gracefully shut down rabbit01, message producer (client) connected to 192.168.69.106:5672 notices broken connection pretty much straight away, attempts a reconnect, hits rabbit02 and all is well. However if I simply "turn off" rabbit01, message producer (client) notices broken connection only in around 30 seconds or so. And while the break is undetected it keeps pushing messages thinking there is still someone on another side. Now, from the statistic report I can see that as soon as I turn off the box, rabbit01 is marked as "DOWN", but its "Sessions Current" is still set to 1? In the logs this comes up pretty much immediately after the hard turn off: Jan 17 17:13:51 prodlb01 haproxy[38459]: Server rabbitmq/rabbit01 is DOWN, reason: Layer4 timeout, check duration: 1008ms. 0 active and 1 backup servers left. Running on backup. 1 sessions active, 0 requeued, 0 remaining in queue. After around 30 seconds, the connection is detected as broken and failover happens successfully. At that time "Sessions Current" is set to 0 fo rabbit01. Tried option nolinger and option abortonclose, but no luck. Any help would be much appreciated... Cheers, Andrei
