maybe something here http://lnxmon.com/haproxy/
Thanks,
; Yuan
On 07/25/2015 12:10 PM, Igor Cicimov wrote:
You need to run haproxy as root to bind to ports lower than 1024
On 25/07/2015 1:36 PM, "Tim Dunphy" <bluethu...@gmail.com> wrote:
Hi Yuan,
Nice.
Do you use selinux in prod.
regards,
; Yuan
Yep! Actually I use it every chance I get. Prod/stage/dev and my own hobby
environments. And right now actually what I was discussing was a hobby
environment.
And actually if I could bother you guys one more time, I do have one more
issue to solve. LOL
And this time it's guaranteed not to be an SELinux issue. Because I tried
running haproxy with SELInux on and off this time.
But what's happening now, is that HA/Proxy is not creating the http port
for the 'stats' interface. I've setup stats to listen on port 80. But for
some reason that's not happening.
Here's my config one more time, with the trouble part in bold:
global
log 127.0.0.1 local0 notice
user haproxy
group haproxy
defaults
log global
retries 2
timeout connect 3000
timeout server 5000
timeout client 5000
listen mysql-cluster
bind 0.0.0.0:3306
mode tcp
option mysql-check user haproxy_check
balance roundrobin
server mysql-1 52.3.28.48:3306 check
server mysql-2 52.2.0.176:3306 check
*listen 0.0.0.0:80 <http://0.0.0.0:80> mode http stats enable
stats uri / stats realm Strictly\ Private stats auth admin:secret*
Currently haproxy is listening on the first port specified* - 3306 - *but
not listening on port 80.
Observe:
[root@ha1:/etc/haproxy] #lsof -i :3306
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
*haproxy 11653 haproxy 4u IPv4 7145270 0t0 TCP *:mysql (LISTEN)*
[root@ha1:/etc/haproxy] #lsof -i :80
[root@ha1:/etc/haproxy] #
[root@ha1:/etc/haproxy] #telnet localhost 80
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused
Port 80 simply isn't listening.
And this time, I can't blame it on SELinux being on:
[root@ha1:/etc/haproxy] #getenforce
Permissive
I've grepped thru /var/log/messages but not turned up any clues to this
one.
And I really would like to get the stats interface up and running.
Any thoughts here? I'm wondering what I can do to get stats working.
Thanks,
Tim
On Fri, Jul 24, 2015 at 10:52 PM, Gmail <longwuy...@gmail.com> wrote:
Nice.
Do you use selinux in prod.
regards,
; Yuan
On 07/25/2015 09:17 AM, Tim Dunphy wrote:
Bingo!!!
The problem was with SELinux. Not sure what took me so long to think of
it...!!!
So set the mysql listener back to port 3306. Turned off SELinux with
setenforce 0. Then it started right up!!! And port 3306 was listening.
Then I consulted with audit2why and saw the following:
type=AVC msg=audit(1437786617.963:28856863): avc: denied {
name_connect }
for pid=29175 comm="haproxy" dest=3306
scontext=system_u:system_r:haproxy_t:s0
tcontext=system_u:object_r:mysqld_port_t:s0 tclass=tcp_socket
Was caused by:
The boolean haproxy_connect_any was set incorrectly.
Description:
Allow haproxy to connect any
Allow access by executing:
# *setsebool -P haproxy_connect_any 1*
I just ran that command you see above in bold, and then all was right
with
the world.
[root@ha1:/etc/haproxy] #systemctl status haproxy
haproxy.service - HAProxy Load Balancer
Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled)
Active: active (running) since Sat 2015-07-25 01:14:53 UTC; 33s ago
Main PID: 30618 (haproxy-systemd)
CGroup: /system.slice/haproxy.service
├─30618 /usr/sbin/haproxy-systemd-wrapper -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid
├─30619 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p
/run/haproxy.pid -Ds
└─30620 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p
/run/haproxy.pid -Ds
Jul 25 01:14:53 ha1 systemd[1]: Starting HAProxy Load Balancer...
Jul 25 01:14:53 ha1 systemd[1]: Started HAProxy Load Balancer.
Jul 25 01:14:53 ha1 haproxy-systemd-wrapper[30618]:
haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
[root@ha1:/etc/haproxy] #lsof -i :3306
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
haproxy 30620 haproxy 1u IPv4 7075172 0t0 TCP
ha1.example.com:55499->ec2-52-2-0-xxx.compute-1.amazonaws.com:mysql
(SYN_SENT)
haproxy 30620 haproxy 4u IPv4 7074731 0t0 TCP *:mysql (LISTEN)
Thanks for nudging me in the right direction. All I had to hear was the
word 'selinux' and from there it all fell into place!
Thanks!!
Tim
On Fri, Jul 24, 2015 at 8:20 PM, Gmail <longwuy...@gmail.com> wrote:
I could be completely wrong here and I am curious to know the answer
myself. Please don't take this as a solution, just my thoughts.
First, you can not use backend ip-address of 10.x.x.x subnet because
each
account's VPC is seggregated. If you do want to use 10.X.X.X ipadress
you
have to setup a inter VPC endpoint in AWS. I would just use EIP.
For the port 3306, try to use nc to listen on that port or iperf. Do yo
uhave iptables turned on.
I would check "systemctl -l status haproxy.service"
I would check lsof -i why can't bind to 3306 on loopback ipaddress.
I would check iptables or selinux preventing the bind.
It wil be interesting to know the source ipaddress of MySQL client ec2
instance.
Interesting if you can Copy/paste output of "telnet
<haproxynode_ipaddress> 3306" from mysql client ec2 instance , here.
Interesting if you can Copy/paste output of "telnet 10.10.10.10 3306"
from haproxy ec2 instances, here.
Interesting if you can Copy/paste output of "telnet 10.10.10.11 3306"
from haproxy ec2 instances, here.
I I was doing this, maybe I would consider testing something like ;
..
frontend mysql_lb_fe 0.0.0.0:3306
....
acl host_myql_lb hdr(host) -i mysql-lb
..
..
use_backend mysql_lb_backend if host mysql_lb
..
..
backend mysql_lb_be
..
..
option mysql-check user haproxy_check
balance roundrobin
server mysql-1 10.10.10.10:3306 check
server mysql-2 10.10.10.11:3306 check
Thanks,
; Yuan
On 07/25/2015 06:41 AM, Tim Dunphy wrote:
Hello Nenad,
Jul 24 03:44:18 ha1 haproxy-systemd-wrapper[25034]: [ALERT]
204/034418 (25035) : *Starting proxy mysql-cluster: cannot
bind
s...:3306]*
Nothing listening on the port I'm trying to bind to: 3306
[root@ha1:~] #ss -lpt | fgrep 3306
[root@ha1:~] #lsof -i :3306
[root@ha1:~] #netstat -tulpn | grep -i listen | grep 3306
[root@ha1:~] #
While we're on the subject of listening ports, here's a list of all
listening ports on the haproxy host:
[root@ha1:~] #netstat -tulpn | grep -i listen
tcp 0 0 0.0.0.0:35145 0.0.0.0:*
LISTEN -
tcp 0 0 0.0.0.0:56814 0.0.0.0:*
LISTEN 16346/rpc.statd
tcp 0 0 0.0.0.0:111 0.0.0.0:*
LISTEN 16455/rpcbind
tcp 0 0 0.0.0.0:22 0.0.0.0:*
LISTEN 16396/sshd
tcp6 0 0 :::49349 :::*
LISTEN 16346/rpc.statd
tcp6 0 0 :::111 :::*
LISTEN 16455/rpcbind
tcp6 0 0 :::47314 :::*
LISTEN -
tcp6 0 0 :::22 :::*
LISTEN 16396/sshd
I thought I was beginning to understand this problem. That haproxy was
trying to bind on port 3306 from the mysql host on another machine. But
come to think of it, that doesn't make a lot of sense.
Because I already have haproxy setup for some web servers, and there it
creates port 80 on the haproxy node. It's not trying to connect to a
foreign source. Not sure where I got that idea!!
I also tried binding the mysql section to another port that wasn't in
use.
I tried port 3307,3308. I even tried binding the mysql section of the
config to a weird port I just grabbed off of the top of my head. I
tried
binding it to port 4444.
And there I still got a bind error:
[ALERT] 204/223303 (13081) : Starting proxy mysql-cluster: cannot
bind
socket [0.0.0.0:4444]
Now watch this!! If I bind the mysql section to port 80 instead of
any
other port.. haproxy starts up without complaint!
listen mysql-cluster
bind 0.0.0.0:80
mode tcp
option mysql-check user haproxy_check
balance roundrobin
server mysql-1 10.0.0.xxx :3306 check
server mysql-2 10.0.0.xxx:3306 check
[root@ha1:/etc/haproxy] #systemctl status haproxy
haproxy.service - HAProxy Load Balancer
Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled)
Active: active (running) since Fri 2015-07-24 22:35:03 UTC; 4s ago
Main PID: 13213 (haproxy-systemd)
CGroup: /system.slice/haproxy.service
├─13213 /usr/sbin/haproxy-systemd-wrapper -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid
├─13214 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p
/run/haproxy.pid -Ds
└─13215 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p
/run/haproxy.pid -Ds
Jul 24 22:35:03 ha1 systemd[1]: Starting HAProxy Load Balancer...
*Jul 24 22:35:03 ha1 systemd[1]: Started HAProxy Load Balancer.*
Jul 24 22:35:03 ha1 haproxy-systemd-wrapper[13213]:
haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
Ok. What...the...heck!!
So why do you think that haproxy is only happy starting up on port 80?
I
would think that I should be able to specify any arbitrary port for it
to
listen on in a 'listen' sub-block.
I guess I could have my app contact the database using port 80. But
that's
a little... weird. I installed haproxy using yum from the 'updates'
repository. Is there any reason anyone can think of as to why haproxy
refuses to start on any port other than port 80??
Thanks,
Tim
On Fri, Jul 24, 2015 at 4:59 PM, Nenad Merdanovic <ni...@nimzo.info>
wrote:
Hello Tim,
On Fri, Jul 24, 2015 at 1:46 PM, Tim Dunphy <
bluethu...@gmail.com
<mailto:bluethu...@gmail.com>> wrote:
listen mysql-cluster
bind 127.0.0.1:3306 <http://127.0.0.1:3306>
mode tcp
option mysql-check user haproxy_check
balance roundrobin
server mysql-1 10.10.10.10:3306 <
http://10.10.10.10:3306>
check
server mysql-2 10.10.10.11:3306 <
http://10.10.10.11:3306>
check
Jul 24 03:44:18 ha1 haproxy-systemd-wrapper[25034]: [ALERT]
204/034418 (25035) : *Starting proxy mysql-cluster: cannot
bind
s...:3306]*
Can you check if something is listening on 127.0.0.1:3306
(netstat, ss,
lsof)? For example:
ss -lpt | fgrep 3306
Regards,
Nenad
--
GPG me!!
gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
