> > By run I meant you have to start it as root user which you are doing > anyway. Can you run: > # nc -l -p 80 > as root just to confirm you can bind to port 80?
If I stop haproxy and run that command this is what I get: [root@ha1:~] #nc -l -p 80 POST /index.php?title=Special%3ARunJobs&tasks=jobs&maxjobs=1&sigexpiry=1437841644&signature=2f2d2dbd28afbc4ecf7e1f59708ff018a30427a9 HTTP/1.1 Host: wiki.example.com Connection: Close Content-Length: 0 Odd, since haproxy isnt' running currently. On Sat, Jul 25, 2015 at 12:07 PM, Igor Cicimov < ig...@encompasscorporation.com> wrote: > By run I meant you have to start it as root user which you are doing > anyway. Can you run: > > # nc -l -p 80 > > as root just to confirm you can bind to port 80? > On 25/07/2015 2:10 PM, "Igor Cicimov" <ig...@encompasscorporation.com> > wrote: > >> You need to run haproxy as root to bind to ports lower than 1024 >> On 25/07/2015 1:36 PM, "Tim Dunphy" <bluethu...@gmail.com> wrote: >> >>> Hi Yuan, >>> >>> Nice. >>>> Do you use selinux in prod. >>>> regards, >>>> ; Yuan >>> >>> >>> Yep! Actually I use it every chance I get. Prod/stage/dev and my own >>> hobby environments. And right now actually what I was discussing was a >>> hobby environment. >>> >>> And actually if I could bother you guys one more time, I do have one >>> more issue to solve. LOL >>> >>> And this time it's guaranteed not to be an SELinux issue. Because I >>> tried running haproxy with SELInux on and off this time. >>> >>> But what's happening now, is that HA/Proxy is not creating the http port >>> for the 'stats' interface. I've setup stats to listen on port 80. But for >>> some reason that's not happening. >>> >>> Here's my config one more time, with the trouble part in bold: >>> >>> global >>> log 127.0.0.1 local0 notice >>> user haproxy >>> group haproxy >>> >>> defaults >>> log global >>> retries 2 >>> timeout connect 3000 >>> timeout server 5000 >>> timeout client 5000 >>> >>> listen mysql-cluster >>> bind 0.0.0.0:3306 >>> mode tcp >>> option mysql-check user haproxy_check >>> balance roundrobin >>> server mysql-1 52.3.28.48:3306 check >>> server mysql-2 52.2.0.176:3306 check >>> >>> >>> >>> >>> >>> >>> >>> >>> *listen 0.0.0.0:80 <http://0.0.0.0:80> mode http stats enable >>> stats uri / stats realm Strictly\ Private stats auth admin:secret* >>> Currently haproxy is listening on the first port specified* - 3306 - *but >>> not listening on port 80. >>> >>> Observe: >>> >>> [root@ha1:/etc/haproxy] #lsof -i :3306 >>> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME >>> *haproxy 11653 haproxy 4u IPv4 7145270 0t0 TCP *:mysql >>> (LISTEN)* >>> >>> [root@ha1:/etc/haproxy] #lsof -i :80 >>> [root@ha1:/etc/haproxy] # >>> >>> [root@ha1:/etc/haproxy] #telnet localhost 80 >>> Trying 127.0.0.1... >>> telnet: connect to address 127.0.0.1: Connection refused >>> >>> Port 80 simply isn't listening. >>> >>> And this time, I can't blame it on SELinux being on: >>> >>> [root@ha1:/etc/haproxy] #getenforce >>> Permissive >>> >>> I've grepped thru /var/log/messages but not turned up any clues to this >>> one. >>> >>> And I really would like to get the stats interface up and running. >>> >>> Any thoughts here? I'm wondering what I can do to get stats working. >>> >>> Thanks, >>> Tim >>> >>> >>> >>> On Fri, Jul 24, 2015 at 10:52 PM, Gmail <longwuy...@gmail.com> wrote: >>> >>>> Nice. >>>> Do you use selinux in prod. >>>> regards, >>>> ; Yuan >>>> >>>> On 07/25/2015 09:17 AM, Tim Dunphy wrote: >>>> >>>>> Bingo!!! >>>>> >>>>> The problem was with SELinux. Not sure what took me so long to think of >>>>> it...!!! >>>>> >>>>> So set the mysql listener back to port 3306. Turned off SELinux with >>>>> setenforce 0. Then it started right up!!! And port 3306 was listening. >>>>> >>>>> Then I consulted with audit2why and saw the following: >>>>> >>>>> type=AVC msg=audit(1437786617.963:28856863): avc: denied { >>>>> name_connect } >>>>> for pid=29175 comm="haproxy" dest=3306 >>>>> scontext=system_u:system_r:haproxy_t:s0 >>>>> tcontext=system_u:object_r:mysqld_port_t:s0 tclass=tcp_socket >>>>> >>>>> Was caused by: >>>>> The boolean haproxy_connect_any was set incorrectly. >>>>> Description: >>>>> Allow haproxy to connect any >>>>> >>>>> Allow access by executing: >>>>> # *setsebool -P haproxy_connect_any 1* >>>>> >>>>> >>>>> I just ran that command you see above in bold, and then all was right >>>>> with >>>>> the world. >>>>> >>>>> [root@ha1:/etc/haproxy] #systemctl status haproxy >>>>> haproxy.service - HAProxy Load Balancer >>>>> Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled) >>>>> Active: active (running) since Sat 2015-07-25 01:14:53 UTC; 33s ago >>>>> Main PID: 30618 (haproxy-systemd) >>>>> CGroup: /system.slice/haproxy.service >>>>> ├─30618 /usr/sbin/haproxy-systemd-wrapper -f >>>>> /etc/haproxy/haproxy.cfg -p /run/haproxy.pid >>>>> ├─30619 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p >>>>> /run/haproxy.pid -Ds >>>>> └─30620 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p >>>>> /run/haproxy.pid -Ds >>>>> >>>>> Jul 25 01:14:53 ha1 systemd[1]: Starting HAProxy Load Balancer... >>>>> Jul 25 01:14:53 ha1 systemd[1]: Started HAProxy Load Balancer. >>>>> Jul 25 01:14:53 ha1 haproxy-systemd-wrapper[30618]: >>>>> haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f >>>>> /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds >>>>> >>>>> [root@ha1:/etc/haproxy] #lsof -i :3306 >>>>> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME >>>>> haproxy 30620 haproxy 1u IPv4 7075172 0t0 TCP >>>>> ha1.example.com:55499->ec2-52-2-0-xxx.compute-1.amazonaws.com:mysql >>>>> (SYN_SENT) >>>>> haproxy 30620 haproxy 4u IPv4 7074731 0t0 TCP *:mysql >>>>> (LISTEN) >>>>> >>>>> >>>>> Thanks for nudging me in the right direction. All I had to hear was the >>>>> word 'selinux' and from there it all fell into place! >>>>> >>>>> Thanks!! >>>>> Tim >>>>> >>>>> On Fri, Jul 24, 2015 at 8:20 PM, Gmail <longwuy...@gmail.com> wrote: >>>>> >>>>> I could be completely wrong here and I am curious to know the answer >>>>>> myself. Please don't take this as a solution, just my thoughts. >>>>>> >>>>>> First, you can not use backend ip-address of 10.x.x.x subnet because >>>>>> each >>>>>> account's VPC is seggregated. If you do want to use 10.X.X.X ipadress >>>>>> you >>>>>> have to setup a inter VPC endpoint in AWS. I would just use EIP. >>>>>> >>>>>> For the port 3306, try to use nc to listen on that port or iperf. Do >>>>>> yo >>>>>> uhave iptables turned on. >>>>>> >>>>>> I would check "systemctl -l status haproxy.service" >>>>>> >>>>>> I would check lsof -i why can't bind to 3306 on loopback ipaddress. >>>>>> >>>>>> I would check iptables or selinux preventing the bind. >>>>>> >>>>>> It wil be interesting to know the source ipaddress of MySQL client ec2 >>>>>> instance. >>>>>> Interesting if you can Copy/paste output of "telnet >>>>>> <haproxynode_ipaddress> 3306" from mysql client ec2 instance , here. >>>>>> Interesting if you can Copy/paste output of "telnet 10.10.10.10 3306" >>>>>> from haproxy ec2 instances, here. >>>>>> Interesting if you can Copy/paste output of "telnet 10.10.10.11 3306" >>>>>> from haproxy ec2 instances, here. >>>>>> >>>>>> I I was doing this, maybe I would consider testing something like ; >>>>>> .. >>>>>> frontend mysql_lb_fe 0.0.0.0:3306 >>>>>> .... >>>>>> acl host_myql_lb hdr(host) -i mysql-lb >>>>>> .. >>>>>> .. >>>>>> use_backend mysql_lb_backend if host mysql_lb >>>>>> .. >>>>>> .. >>>>>> backend mysql_lb_be >>>>>> .. >>>>>> .. >>>>>> >>>>>> option mysql-check user haproxy_check >>>>>> balance roundrobin >>>>>> server mysql-1 10.10.10.10:3306 check >>>>>> server mysql-2 10.10.10.11:3306 check >>>>>> >>>>>> Thanks, >>>>>> ; Yuan >>>>>> >>>>>> >>>>>> On 07/25/2015 06:41 AM, Tim Dunphy wrote: >>>>>> >>>>>> Hello Nenad, >>>>>>> >>>>>>> Jul 24 03:44:18 ha1 haproxy-systemd-wrapper[25034]: [ALERT] >>>>>>> >>>>>>> 204/034418 (25035) : *Starting proxy mysql-cluster: >>>>>>>> cannot bind >>>>>>>> s...:3306]* >>>>>>>> >>>>>>>> Nothing listening on the port I'm trying to bind to: 3306 >>>>>>> >>>>>>> [root@ha1:~] #ss -lpt | fgrep 3306 >>>>>>> [root@ha1:~] #lsof -i :3306 >>>>>>> [root@ha1:~] #netstat -tulpn | grep -i listen | grep 3306 >>>>>>> [root@ha1:~] # >>>>>>> >>>>>>> While we're on the subject of listening ports, here's a list of all >>>>>>> listening ports on the haproxy host: >>>>>>> >>>>>>> [root@ha1:~] #netstat -tulpn | grep -i listen >>>>>>> tcp 0 0 0.0.0.0:35145 0.0.0.0:* >>>>>>> LISTEN - >>>>>>> tcp 0 0 0.0.0.0:56814 0.0.0.0:* >>>>>>> LISTEN 16346/rpc.statd >>>>>>> tcp 0 0 0.0.0.0:111 0.0.0.0:* >>>>>>> LISTEN 16455/rpcbind >>>>>>> tcp 0 0 0.0.0.0:22 0.0.0.0:* >>>>>>> LISTEN 16396/sshd >>>>>>> tcp6 0 0 :::49349 :::* >>>>>>> LISTEN 16346/rpc.statd >>>>>>> tcp6 0 0 :::111 :::* >>>>>>> LISTEN 16455/rpcbind >>>>>>> tcp6 0 0 :::47314 :::* >>>>>>> LISTEN - >>>>>>> tcp6 0 0 :::22 :::* >>>>>>> LISTEN 16396/sshd >>>>>>> >>>>>>> I thought I was beginning to understand this problem. That haproxy >>>>>>> was >>>>>>> trying to bind on port 3306 from the mysql host on another machine. >>>>>>> But >>>>>>> come to think of it, that doesn't make a lot of sense. >>>>>>> >>>>>>> Because I already have haproxy setup for some web servers, and there >>>>>>> it >>>>>>> creates port 80 on the haproxy node. It's not trying to connect to a >>>>>>> foreign source. Not sure where I got that idea!! >>>>>>> >>>>>>> I also tried binding the mysql section to another port that wasn't >>>>>>> in use. >>>>>>> I tried port 3307,3308. I even tried binding the mysql section of the >>>>>>> config to a weird port I just grabbed off of the top of my head. I >>>>>>> tried >>>>>>> binding it to port 4444. >>>>>>> >>>>>>> And there I still got a bind error: >>>>>>> >>>>>>> [ALERT] 204/223303 (13081) : Starting proxy mysql-cluster: cannot >>>>>>> bind >>>>>>> socket [0.0.0.0:4444] >>>>>>> >>>>>>> >>>>>>> Now watch this!! If I bind the mysql section to port 80 instead >>>>>>> of any >>>>>>> other port.. haproxy starts up without complaint! >>>>>>> >>>>>>> listen mysql-cluster >>>>>>> bind 0.0.0.0:80 >>>>>>> mode tcp >>>>>>> option mysql-check user haproxy_check >>>>>>> balance roundrobin >>>>>>> server mysql-1 10.0.0.xxx :3306 check >>>>>>> server mysql-2 10.0.0.xxx:3306 check >>>>>>> >>>>>>> [root@ha1:/etc/haproxy] #systemctl status haproxy >>>>>>> haproxy.service - HAProxy Load Balancer >>>>>>> Loaded: loaded (/usr/lib/systemd/system/haproxy.service; >>>>>>> enabled) >>>>>>> Active: active (running) since Fri 2015-07-24 22:35:03 UTC; 4s >>>>>>> ago >>>>>>> Main PID: 13213 (haproxy-systemd) >>>>>>> CGroup: /system.slice/haproxy.service >>>>>>> ├─13213 /usr/sbin/haproxy-systemd-wrapper -f >>>>>>> /etc/haproxy/haproxy.cfg -p /run/haproxy.pid >>>>>>> ├─13214 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p >>>>>>> /run/haproxy.pid -Ds >>>>>>> └─13215 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p >>>>>>> /run/haproxy.pid -Ds >>>>>>> >>>>>>> Jul 24 22:35:03 ha1 systemd[1]: Starting HAProxy Load Balancer... >>>>>>> *Jul 24 22:35:03 ha1 systemd[1]: Started HAProxy Load Balancer.* >>>>>>> >>>>>>> Jul 24 22:35:03 ha1 haproxy-systemd-wrapper[13213]: >>>>>>> haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f >>>>>>> /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds >>>>>>> >>>>>>> Ok. What...the...heck!! >>>>>>> >>>>>>> So why do you think that haproxy is only happy starting up on port >>>>>>> 80? I >>>>>>> would think that I should be able to specify any arbitrary port for >>>>>>> it to >>>>>>> listen on in a 'listen' sub-block. >>>>>>> >>>>>>> I guess I could have my app contact the database using port 80. But >>>>>>> that's >>>>>>> a little... weird. I installed haproxy using yum from the 'updates' >>>>>>> repository. Is there any reason anyone can think of as to why haproxy >>>>>>> refuses to start on any port other than port 80?? >>>>>>> >>>>>>> Thanks, >>>>>>> Tim >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Fri, Jul 24, 2015 at 4:59 PM, Nenad Merdanovic <ni...@nimzo.info> >>>>>>> wrote: >>>>>>> >>>>>>> Hello Tim, >>>>>>> >>>>>>>> On Fri, Jul 24, 2015 at 1:46 PM, Tim Dunphy < >>>>>>>> bluethu...@gmail.com >>>>>>>> >>>>>>>>> <mailto:bluethu...@gmail.com>> wrote: >>>>>>>>> listen mysql-cluster >>>>>>>>> bind 127.0.0.1:3306 <http://127.0.0.1:3306> >>>>>>>>> >>>>>>>>> mode tcp >>>>>>>>> option mysql-check user haproxy_check >>>>>>>>> balance roundrobin >>>>>>>>> server mysql-1 10.10.10.10:3306 < >>>>>>>>> http://10.10.10.10:3306> >>>>>>>>> >>>>>>>>> check >>>>>>>> >>>>>>>> server mysql-2 10.10.10.11:3306 < >>>>>>>>> http://10.10.10.11:3306> >>>>>>>>> >>>>>>>>> check >>>>>>>> >>>>>>>> Jul 24 03:44:18 ha1 haproxy-systemd-wrapper[25034]: >>>>>>>>> [ALERT] >>>>>>>>> 204/034418 (25035) : *Starting proxy mysql-cluster: >>>>>>>>> cannot bind >>>>>>>>> s...:3306]* >>>>>>>>> >>>>>>>>> Can you check if something is listening on 127.0.0.1:3306 >>>>>>>> (netstat, ss, >>>>>>>> lsof)? For example: >>>>>>>> ss -lpt | fgrep 3306 >>>>>>>> >>>>>>>> Regards, >>>>>>>> Nenad >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>> >>> >>> >>> -- >>> GPG me!! >>> >>> gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B >>> >>> -- GPG me!! gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
