Re: How to disable backend servers without health check
On Thu, Jul 16, 2015 at 5:06 PM, Pavlos Parissis pavlos.paris...@gmail.com wrote: On 16/07/2015 04:02 μμ, Krishna Kumar (Engineering) wrote: Hi John, Your suggestion works very well, and exactly what I was looking for. Thank you very much. You could also try https://github.com/unixsurfer/haproxytool Cheers, Pavlos +1 to Pavlos' tool for this type of task Baptiste
How to disable backend servers without health check
Hi all, We have a large set of machines running haproxy (1.5.12), and each of them have hundreds of backends, many of which are the same across systems. nbproc is set to 12 at present for our 48 core systems. We are planning a centralized health check, and disable the same in haproxy, to avoid each process on each server doing health check for the same backend servers. Is there any way to disable a single backend from the command line, such that each haproxy instance finds that this backend is disabled? Using socat with the socket only makes the handling process set it's status of the backend as MAINT, but others don't get this information. Appreciate if someone can show if this can be done. Regards, - Krishna Kumar -- -- This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Although Flipkart has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments
Re: How to disable backend servers without health check
Krishna,
I've recently had to deal with this as well. Our solution involves a
couple of aspects. Firstly, one must configure an admin socket per
process. In our case we run with 20 processes, so we've got a
configuration that looks similar to this in our global section:
stats socket /var/run/haproxy_admin1.sock mode 600 level admin process 1
stats socket /var/run/haproxy_admin2.sock mode 600 level admin process 2
stats socket /var/run/haproxy_admin3.sock mode 600 level admin process 3
stats socket /var/run/haproxy_admin4.sock mode 600 level admin process 4
Counting all the way up to 20...
After that we can do a simple one liner that disables a single server;
using bash:
for i in {1..20}; do echo 'disable server the_backend/the_server' | socat
/var/run/haproxy_admin$i.sock stdio; done
This loops through each admin socket and disables 'the_server' from
'the_backend'.
I hope this gets you started in looking for a solution.
I like your route of accomplishing this though. With our 20 proc
configuration we've decided to deal with the pain of 20 health checks which
has caused us some issues, but nothing being a show stopper.
On Thu, Jul 16, 2015 at 5:53 AM, Krishna Kumar (Engineering)
krishna...@flipkart.com wrote:
Hi all,
We have a large set of machines running haproxy (1.5.12), and each of
them have hundreds of backends, many of which are the same across
systems. nbproc is set to 12 at present for our 48 core systems. We are
planning a centralized health check, and disable the same in haproxy, to
avoid each process on each server doing health check for the same
backend servers.
Is there any way to disable a single backend from the command line, such
that each haproxy instance finds that this backend is disabled? Using
socat with the socket only makes the handling process set it's status of
the
backend as MAINT, but others don't get this information.
Appreciate if someone can show if this can be done.
Regards,
- Krishna Kumar
--
This email and any files transmitted with it are confidential and intended
solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error please notify the system manager.
This message contains confidential information and is intended only for the
individual named. If you are not the named addressee you should not
disseminate, distribute or copy this e-mail. Please notify the sender
immediately by e-mail if you have received this e-mail by mistake and
delete this e-mail from your system. If you are not the intended recipient
you are notified that disclosing, copying, distributing or taking any
action in reliance on the contents of this information is strictly
prohibited. Although Flipkart has taken reasonable precautions to ensure no
viruses are present in this email, the company cannot accept responsibility
for any loss or damage arising from the use of this email or attachments
--
[image: rally-logo-68x68.jpg]
John T Skarbek | jskar...@rallydev.com
Infrastructure Engineer, Engineering
1101 Haynes Street, Suite 105, Raleigh, NC 27604
720.921.8126 Office
Re: How to disable backend servers without health check
Hi John,
Your suggestion works very well, and exactly what I was looking for.
Thank you very much.
Regards,
- Krishna Kumar
On Thu, Jul 16, 2015 at 5:48 PM, John Skarbek jskar...@rallydev.com wrote:
Krishna,
I've recently had to deal with this as well. Our solution involves a
couple of aspects. Firstly, one must configure an admin socket per
process. In our case we run with 20 processes, so we've got a
configuration that looks similar to this in our global section:
stats socket /var/run/haproxy_admin1.sock mode 600 level admin process 1
stats socket /var/run/haproxy_admin2.sock mode 600 level admin process 2
stats socket /var/run/haproxy_admin3.sock mode 600 level admin process 3
stats socket /var/run/haproxy_admin4.sock mode 600 level admin process 4
Counting all the way up to 20...
After that we can do a simple one liner that disables a single server;
using bash:
for i in {1..20}; do echo 'disable server the_backend/the_server' | socat
/var/run/haproxy_admin$i.sock stdio; done
This loops through each admin socket and disables 'the_server' from
'the_backend'.
I hope this gets you started in looking for a solution.
I like your route of accomplishing this though. With our 20 proc
configuration we've decided to deal with the pain of 20 health checks which
has caused us some issues, but nothing being a show stopper.
On Thu, Jul 16, 2015 at 5:53 AM, Krishna Kumar (Engineering)
krishna...@flipkart.com wrote:
Hi all,
We have a large set of machines running haproxy (1.5.12), and each of
them have hundreds of backends, many of which are the same across
systems. nbproc is set to 12 at present for our 48 core systems. We are
planning a centralized health check, and disable the same in haproxy, to
avoid each process on each server doing health check for the same
backend servers.
Is there any way to disable a single backend from the command line, such
that each haproxy instance finds that this backend is disabled? Using
socat with the socket only makes the handling process set it's status of
the
backend as MAINT, but others don't get this information.
Appreciate if someone can show if this can be done.
Regards,
- Krishna Kumar
--
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they are
addressed. If you have received this email in error please notify the
system manager. This message contains confidential information and is
intended only for the individual named. If you are not the named addressee
you should not disseminate, distribute or copy this e-mail. Please notify
the sender immediately by e-mail if you have received this e-mail by
mistake and delete this e-mail from your system. If you are not the
intended recipient you are notified that disclosing, copying, distributing
or taking any action in reliance on the contents of this information is
strictly prohibited. Although Flipkart has taken reasonable precautions to
ensure no viruses are present in this email, the company cannot accept
responsibility for any loss or damage arising from the use of this email or
attachments
--
[image: rally-logo-68x68.jpg]
John T Skarbek | jskar...@rallydev.com
Infrastructure Engineer, Engineering
1101 Haynes Street, Suite 105, Raleigh, NC 27604
720.921.8126 Office
--
--
This email and any files transmitted with it are confidential and intended
solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error please notify the system manager.
This message contains confidential information and is intended only for the
individual named. If you are not the named addressee you should not
disseminate, distribute or copy this e-mail. Please notify the sender
immediately by e-mail if you have received this e-mail by mistake and
delete this e-mail from your system. If you are not the intended recipient
you are notified that disclosing, copying, distributing or taking any
action in reliance on the contents of this information is strictly
prohibited. Although Flipkart has taken reasonable precautions to ensure no
viruses are present in this email, the company cannot accept responsibility
for any loss or damage arising from the use of this email or attachments
Re: How to disable backend servers without health check
On 16/07/2015 04:02 μμ, Krishna Kumar (Engineering) wrote: Hi John, Your suggestion works very well, and exactly what I was looking for. Thank you very much. You could also try https://github.com/unixsurfer/haproxytool Cheers, Pavlos signature.asc Description: OpenPGP digital signature
Re: How to disable backend servers without health check
Thanks Pavlos, this looks very promising, I will take a look on how we can use this. Regards, - Krishna Kumar On Thu, Jul 16, 2015 at 8:36 PM, Pavlos Parissis pavlos.paris...@gmail.com wrote: On 16/07/2015 04:02 μμ, Krishna Kumar (Engineering) wrote: Hi John, Your suggestion works very well, and exactly what I was looking for. Thank you very much. You could also try https://github.com/unixsurfer/haproxytool Cheers, Pavlos -- -- This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Although Flipkart has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments
