Nice. Do you use selinux in prod. regards, ; Yuan On 07/25/2015 09:17 AM, Tim Dunphy wrote:
Bingo!!!The problem was with SELinux. Not sure what took me so long to think of it...!!! So set the mysql listener back to port 3306. Turned off SELinux with setenforce 0. Then it started right up!!! And port 3306 was listening. Then I consulted with audit2why and saw the following: type=AVC msg=audit(1437786617.963:28856863): avc: denied { name_connect } for pid=29175 comm="haproxy" dest=3306 scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:object_r:mysqld_port_t:s0 tclass=tcp_socket Was caused by: The boolean haproxy_connect_any was set incorrectly. Description: Allow haproxy to connect any Allow access by executing: # *setsebool -P haproxy_connect_any 1* I just ran that command you see above in bold, and then all was right with the world. [root@ha1:/etc/haproxy] #systemctl status haproxy haproxy.service - HAProxy Load Balancer Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled) Active: active (running) since Sat 2015-07-25 01:14:53 UTC; 33s ago Main PID: 30618 (haproxy-systemd) CGroup: /system.slice/haproxy.service ├─30618 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid ├─30619 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds └─30620 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds Jul 25 01:14:53 ha1 systemd[1]: Starting HAProxy Load Balancer... Jul 25 01:14:53 ha1 systemd[1]: Started HAProxy Load Balancer. Jul 25 01:14:53 ha1 haproxy-systemd-wrapper[30618]: haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds [root@ha1:/etc/haproxy] #lsof -i :3306 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME haproxy 30620 haproxy 1u IPv4 7075172 0t0 TCP ha1.example.com:55499->ec2-52-2-0-xxx.compute-1.amazonaws.com:mysql (SYN_SENT) haproxy 30620 haproxy 4u IPv4 7074731 0t0 TCP *:mysql (LISTEN) Thanks for nudging me in the right direction. All I had to hear was the word 'selinux' and from there it all fell into place! Thanks!! Tim On Fri, Jul 24, 2015 at 8:20 PM, Gmail <longwuy...@gmail.com> wrote:I could be completely wrong here and I am curious to know the answer myself. Please don't take this as a solution, just my thoughts. First, you can not use backend ip-address of 10.x.x.x subnet because each account's VPC is seggregated. If you do want to use 10.X.X.X ipadress you have to setup a inter VPC endpoint in AWS. I would just use EIP. For the port 3306, try to use nc to listen on that port or iperf. Do yo uhave iptables turned on. I would check "systemctl -l status haproxy.service" I would check lsof -i why can't bind to 3306 on loopback ipaddress. I would check iptables or selinux preventing the bind. It wil be interesting to know the source ipaddress of MySQL client ec2 instance. Interesting if you can Copy/paste output of "telnet <haproxynode_ipaddress> 3306" from mysql client ec2 instance , here. Interesting if you can Copy/paste output of "telnet 10.10.10.10 3306" from haproxy ec2 instances, here. Interesting if you can Copy/paste output of "telnet 10.10.10.11 3306" from haproxy ec2 instances, here. I I was doing this, maybe I would consider testing something like ; .. frontend mysql_lb_fe 0.0.0.0:3306 .... acl host_myql_lb hdr(host) -i mysql-lb .. .. use_backend mysql_lb_backend if host mysql_lb .. .. backend mysql_lb_be .. .. option mysql-check user haproxy_check balance roundrobin server mysql-1 10.10.10.10:3306 check server mysql-2 10.10.10.11:3306 check Thanks, ; Yuan On 07/25/2015 06:41 AM, Tim Dunphy wrote:Hello Nenad, Jul 24 03:44:18 ha1 haproxy-systemd-wrapper[25034]: [ALERT]204/034418 (25035) : *Starting proxy mysql-cluster: cannot bind s...:3306]*Nothing listening on the port I'm trying to bind to: 3306 [root@ha1:~] #ss -lpt | fgrep 3306 [root@ha1:~] #lsof -i :3306 [root@ha1:~] #netstat -tulpn | grep -i listen | grep 3306 [root@ha1:~] # While we're on the subject of listening ports, here's a list of all listening ports on the haproxy host: [root@ha1:~] #netstat -tulpn | grep -i listen tcp 0 0 0.0.0.0:35145 0.0.0.0:* LISTEN - tcp 0 0 0.0.0.0:56814 0.0.0.0:* LISTEN 16346/rpc.statd tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 16455/rpcbind tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 16396/sshd tcp6 0 0 :::49349 :::* LISTEN 16346/rpc.statd tcp6 0 0 :::111 :::* LISTEN 16455/rpcbind tcp6 0 0 :::47314 :::* LISTEN - tcp6 0 0 :::22 :::* LISTEN 16396/sshd I thought I was beginning to understand this problem. That haproxy was trying to bind on port 3306 from the mysql host on another machine. But come to think of it, that doesn't make a lot of sense. Because I already have haproxy setup for some web servers, and there it creates port 80 on the haproxy node. It's not trying to connect to a foreign source. Not sure where I got that idea!! I also tried binding the mysql section to another port that wasn't in use. I tried port 3307,3308. I even tried binding the mysql section of the config to a weird port I just grabbed off of the top of my head. I tried binding it to port 4444. And there I still got a bind error: [ALERT] 204/223303 (13081) : Starting proxy mysql-cluster: cannot bind socket [0.0.0.0:4444] Now watch this!! If I bind the mysql section to port 80 instead of any other port.. haproxy starts up without complaint! listen mysql-cluster bind 0.0.0.0:80 mode tcp option mysql-check user haproxy_check balance roundrobin server mysql-1 10.0.0.xxx :3306 check server mysql-2 10.0.0.xxx:3306 check [root@ha1:/etc/haproxy] #systemctl status haproxy haproxy.service - HAProxy Load Balancer Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled) Active: active (running) since Fri 2015-07-24 22:35:03 UTC; 4s ago Main PID: 13213 (haproxy-systemd) CGroup: /system.slice/haproxy.service ├─13213 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid ├─13214 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds └─13215 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds Jul 24 22:35:03 ha1 systemd[1]: Starting HAProxy Load Balancer... *Jul 24 22:35:03 ha1 systemd[1]: Started HAProxy Load Balancer.* Jul 24 22:35:03 ha1 haproxy-systemd-wrapper[13213]: haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds Ok. What...the...heck!! So why do you think that haproxy is only happy starting up on port 80? I would think that I should be able to specify any arbitrary port for it to listen on in a 'listen' sub-block. I guess I could have my app contact the database using port 80. But that's a little... weird. I installed haproxy using yum from the 'updates' repository. Is there any reason anyone can think of as to why haproxy refuses to start on any port other than port 80?? Thanks, Tim On Fri, Jul 24, 2015 at 4:59 PM, Nenad Merdanovic <ni...@nimzo.info> wrote: Hello Tim,On Fri, Jul 24, 2015 at 1:46 PM, Tim Dunphy <bluethu...@gmail.com<mailto:bluethu...@gmail.com>> wrote: listen mysql-cluster bind 127.0.0.1:3306 <http://127.0.0.1:3306> mode tcp option mysql-check user haproxy_check balance roundrobin server mysql-1 10.10.10.10:3306 <http://10.10.10.10:3306>checkserver mysql-2 10.10.10.11:3306 <http://10.10.10.11:3306>checkJul 24 03:44:18 ha1 haproxy-systemd-wrapper[25034]: [ALERT] 204/034418 (25035) : *Starting proxy mysql-cluster: cannot bind s...:3306]*Can you check if something is listening on 127.0.0.1:3306 (netstat, ss, lsof)? For example: ss -lpt | fgrep 3306 Regards, Nenad
