You need to run haproxy as root to bind to ports lower than 1024 On 25/07/2015 1:36 PM, "Tim Dunphy" <bluethu...@gmail.com> wrote:
> Hi Yuan, > > Nice. >> Do you use selinux in prod. >> regards, >> ; Yuan > > > Yep! Actually I use it every chance I get. Prod/stage/dev and my own hobby > environments. And right now actually what I was discussing was a hobby > environment. > > And actually if I could bother you guys one more time, I do have one more > issue to solve. LOL > > And this time it's guaranteed not to be an SELinux issue. Because I tried > running haproxy with SELInux on and off this time. > > But what's happening now, is that HA/Proxy is not creating the http port > for the 'stats' interface. I've setup stats to listen on port 80. But for > some reason that's not happening. > > Here's my config one more time, with the trouble part in bold: > > global > log 127.0.0.1 local0 notice > user haproxy > group haproxy > > defaults > log global > retries 2 > timeout connect 3000 > timeout server 5000 > timeout client 5000 > > listen mysql-cluster > bind 0.0.0.0:3306 > mode tcp > option mysql-check user haproxy_check > balance roundrobin > server mysql-1 52.3.28.48:3306 check > server mysql-2 52.2.0.176:3306 check > > > > > > > > > *listen 0.0.0.0:80 <http://0.0.0.0:80> mode http stats enable > stats uri / stats realm Strictly\ Private stats auth admin:secret* > Currently haproxy is listening on the first port specified* - 3306 - *but > not listening on port 80. > > Observe: > > [root@ha1:/etc/haproxy] #lsof -i :3306 > COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME > *haproxy 11653 haproxy 4u IPv4 7145270 0t0 TCP *:mysql (LISTEN)* > > [root@ha1:/etc/haproxy] #lsof -i :80 > [root@ha1:/etc/haproxy] # > > [root@ha1:/etc/haproxy] #telnet localhost 80 > Trying 127.0.0.1... > telnet: connect to address 127.0.0.1: Connection refused > > Port 80 simply isn't listening. > > And this time, I can't blame it on SELinux being on: > > [root@ha1:/etc/haproxy] #getenforce > Permissive > > I've grepped thru /var/log/messages but not turned up any clues to this > one. > > And I really would like to get the stats interface up and running. > > Any thoughts here? I'm wondering what I can do to get stats working. > > Thanks, > Tim > > > > On Fri, Jul 24, 2015 at 10:52 PM, Gmail <longwuy...@gmail.com> wrote: > >> Nice. >> Do you use selinux in prod. >> regards, >> ; Yuan >> >> On 07/25/2015 09:17 AM, Tim Dunphy wrote: >> >>> Bingo!!! >>> >>> The problem was with SELinux. Not sure what took me so long to think of >>> it...!!! >>> >>> So set the mysql listener back to port 3306. Turned off SELinux with >>> setenforce 0. Then it started right up!!! And port 3306 was listening. >>> >>> Then I consulted with audit2why and saw the following: >>> >>> type=AVC msg=audit(1437786617.963:28856863): avc: denied { >>> name_connect } >>> for pid=29175 comm="haproxy" dest=3306 >>> scontext=system_u:system_r:haproxy_t:s0 >>> tcontext=system_u:object_r:mysqld_port_t:s0 tclass=tcp_socket >>> >>> Was caused by: >>> The boolean haproxy_connect_any was set incorrectly. >>> Description: >>> Allow haproxy to connect any >>> >>> Allow access by executing: >>> # *setsebool -P haproxy_connect_any 1* >>> >>> >>> I just ran that command you see above in bold, and then all was right >>> with >>> the world. >>> >>> [root@ha1:/etc/haproxy] #systemctl status haproxy >>> haproxy.service - HAProxy Load Balancer >>> Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled) >>> Active: active (running) since Sat 2015-07-25 01:14:53 UTC; 33s ago >>> Main PID: 30618 (haproxy-systemd) >>> CGroup: /system.slice/haproxy.service >>> ├─30618 /usr/sbin/haproxy-systemd-wrapper -f >>> /etc/haproxy/haproxy.cfg -p /run/haproxy.pid >>> ├─30619 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p >>> /run/haproxy.pid -Ds >>> └─30620 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p >>> /run/haproxy.pid -Ds >>> >>> Jul 25 01:14:53 ha1 systemd[1]: Starting HAProxy Load Balancer... >>> Jul 25 01:14:53 ha1 systemd[1]: Started HAProxy Load Balancer. >>> Jul 25 01:14:53 ha1 haproxy-systemd-wrapper[30618]: >>> haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f >>> /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds >>> >>> [root@ha1:/etc/haproxy] #lsof -i :3306 >>> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME >>> haproxy 30620 haproxy 1u IPv4 7075172 0t0 TCP >>> ha1.example.com:55499->ec2-52-2-0-xxx.compute-1.amazonaws.com:mysql >>> (SYN_SENT) >>> haproxy 30620 haproxy 4u IPv4 7074731 0t0 TCP *:mysql (LISTEN) >>> >>> >>> Thanks for nudging me in the right direction. All I had to hear was the >>> word 'selinux' and from there it all fell into place! >>> >>> Thanks!! >>> Tim >>> >>> On Fri, Jul 24, 2015 at 8:20 PM, Gmail <longwuy...@gmail.com> wrote: >>> >>> I could be completely wrong here and I am curious to know the answer >>>> myself. Please don't take this as a solution, just my thoughts. >>>> >>>> First, you can not use backend ip-address of 10.x.x.x subnet because >>>> each >>>> account's VPC is seggregated. If you do want to use 10.X.X.X ipadress >>>> you >>>> have to setup a inter VPC endpoint in AWS. I would just use EIP. >>>> >>>> For the port 3306, try to use nc to listen on that port or iperf. Do yo >>>> uhave iptables turned on. >>>> >>>> I would check "systemctl -l status haproxy.service" >>>> >>>> I would check lsof -i why can't bind to 3306 on loopback ipaddress. >>>> >>>> I would check iptables or selinux preventing the bind. >>>> >>>> It wil be interesting to know the source ipaddress of MySQL client ec2 >>>> instance. >>>> Interesting if you can Copy/paste output of "telnet >>>> <haproxynode_ipaddress> 3306" from mysql client ec2 instance , here. >>>> Interesting if you can Copy/paste output of "telnet 10.10.10.10 3306" >>>> from haproxy ec2 instances, here. >>>> Interesting if you can Copy/paste output of "telnet 10.10.10.11 3306" >>>> from haproxy ec2 instances, here. >>>> >>>> I I was doing this, maybe I would consider testing something like ; >>>> .. >>>> frontend mysql_lb_fe 0.0.0.0:3306 >>>> .... >>>> acl host_myql_lb hdr(host) -i mysql-lb >>>> .. >>>> .. >>>> use_backend mysql_lb_backend if host mysql_lb >>>> .. >>>> .. >>>> backend mysql_lb_be >>>> .. >>>> .. >>>> >>>> option mysql-check user haproxy_check >>>> balance roundrobin >>>> server mysql-1 10.10.10.10:3306 check >>>> server mysql-2 10.10.10.11:3306 check >>>> >>>> Thanks, >>>> ; Yuan >>>> >>>> >>>> On 07/25/2015 06:41 AM, Tim Dunphy wrote: >>>> >>>> Hello Nenad, >>>>> >>>>> Jul 24 03:44:18 ha1 haproxy-systemd-wrapper[25034]: [ALERT] >>>>> >>>>> 204/034418 (25035) : *Starting proxy mysql-cluster: cannot >>>>>> bind >>>>>> s...:3306]* >>>>>> >>>>>> Nothing listening on the port I'm trying to bind to: 3306 >>>>> >>>>> [root@ha1:~] #ss -lpt | fgrep 3306 >>>>> [root@ha1:~] #lsof -i :3306 >>>>> [root@ha1:~] #netstat -tulpn | grep -i listen | grep 3306 >>>>> [root@ha1:~] # >>>>> >>>>> While we're on the subject of listening ports, here's a list of all >>>>> listening ports on the haproxy host: >>>>> >>>>> [root@ha1:~] #netstat -tulpn | grep -i listen >>>>> tcp 0 0 0.0.0.0:35145 0.0.0.0:* >>>>> LISTEN - >>>>> tcp 0 0 0.0.0.0:56814 0.0.0.0:* >>>>> LISTEN 16346/rpc.statd >>>>> tcp 0 0 0.0.0.0:111 0.0.0.0:* >>>>> LISTEN 16455/rpcbind >>>>> tcp 0 0 0.0.0.0:22 0.0.0.0:* >>>>> LISTEN 16396/sshd >>>>> tcp6 0 0 :::49349 :::* >>>>> LISTEN 16346/rpc.statd >>>>> tcp6 0 0 :::111 :::* >>>>> LISTEN 16455/rpcbind >>>>> tcp6 0 0 :::47314 :::* >>>>> LISTEN - >>>>> tcp6 0 0 :::22 :::* >>>>> LISTEN 16396/sshd >>>>> >>>>> I thought I was beginning to understand this problem. That haproxy was >>>>> trying to bind on port 3306 from the mysql host on another machine. But >>>>> come to think of it, that doesn't make a lot of sense. >>>>> >>>>> Because I already have haproxy setup for some web servers, and there it >>>>> creates port 80 on the haproxy node. It's not trying to connect to a >>>>> foreign source. Not sure where I got that idea!! >>>>> >>>>> I also tried binding the mysql section to another port that wasn't in >>>>> use. >>>>> I tried port 3307,3308. I even tried binding the mysql section of the >>>>> config to a weird port I just grabbed off of the top of my head. I >>>>> tried >>>>> binding it to port 4444. >>>>> >>>>> And there I still got a bind error: >>>>> >>>>> [ALERT] 204/223303 (13081) : Starting proxy mysql-cluster: cannot >>>>> bind >>>>> socket [0.0.0.0:4444] >>>>> >>>>> >>>>> Now watch this!! If I bind the mysql section to port 80 instead of >>>>> any >>>>> other port.. haproxy starts up without complaint! >>>>> >>>>> listen mysql-cluster >>>>> bind 0.0.0.0:80 >>>>> mode tcp >>>>> option mysql-check user haproxy_check >>>>> balance roundrobin >>>>> server mysql-1 10.0.0.xxx :3306 check >>>>> server mysql-2 10.0.0.xxx:3306 check >>>>> >>>>> [root@ha1:/etc/haproxy] #systemctl status haproxy >>>>> haproxy.service - HAProxy Load Balancer >>>>> Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled) >>>>> Active: active (running) since Fri 2015-07-24 22:35:03 UTC; 4s ago >>>>> Main PID: 13213 (haproxy-systemd) >>>>> CGroup: /system.slice/haproxy.service >>>>> ├─13213 /usr/sbin/haproxy-systemd-wrapper -f >>>>> /etc/haproxy/haproxy.cfg -p /run/haproxy.pid >>>>> ├─13214 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p >>>>> /run/haproxy.pid -Ds >>>>> └─13215 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p >>>>> /run/haproxy.pid -Ds >>>>> >>>>> Jul 24 22:35:03 ha1 systemd[1]: Starting HAProxy Load Balancer... >>>>> *Jul 24 22:35:03 ha1 systemd[1]: Started HAProxy Load Balancer.* >>>>> >>>>> Jul 24 22:35:03 ha1 haproxy-systemd-wrapper[13213]: >>>>> haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f >>>>> /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds >>>>> >>>>> Ok. What...the...heck!! >>>>> >>>>> So why do you think that haproxy is only happy starting up on port 80? >>>>> I >>>>> would think that I should be able to specify any arbitrary port for it >>>>> to >>>>> listen on in a 'listen' sub-block. >>>>> >>>>> I guess I could have my app contact the database using port 80. But >>>>> that's >>>>> a little... weird. I installed haproxy using yum from the 'updates' >>>>> repository. Is there any reason anyone can think of as to why haproxy >>>>> refuses to start on any port other than port 80?? >>>>> >>>>> Thanks, >>>>> Tim >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Fri, Jul 24, 2015 at 4:59 PM, Nenad Merdanovic <ni...@nimzo.info> >>>>> wrote: >>>>> >>>>> Hello Tim, >>>>> >>>>>> On Fri, Jul 24, 2015 at 1:46 PM, Tim Dunphy < >>>>>> bluethu...@gmail.com >>>>>> >>>>>>> <mailto:bluethu...@gmail.com>> wrote: >>>>>>> listen mysql-cluster >>>>>>> bind 127.0.0.1:3306 <http://127.0.0.1:3306> >>>>>>> >>>>>>> mode tcp >>>>>>> option mysql-check user haproxy_check >>>>>>> balance roundrobin >>>>>>> server mysql-1 10.10.10.10:3306 < >>>>>>> http://10.10.10.10:3306> >>>>>>> >>>>>>> check >>>>>> >>>>>> server mysql-2 10.10.10.11:3306 < >>>>>>> http://10.10.10.11:3306> >>>>>>> >>>>>>> check >>>>>> >>>>>> Jul 24 03:44:18 ha1 haproxy-systemd-wrapper[25034]: [ALERT] >>>>>>> 204/034418 (25035) : *Starting proxy mysql-cluster: cannot >>>>>>> bind >>>>>>> s...:3306]* >>>>>>> >>>>>>> Can you check if something is listening on 127.0.0.1:3306 >>>>>> (netstat, ss, >>>>>> lsof)? For example: >>>>>> ss -lpt | fgrep 3306 >>>>>> >>>>>> Regards, >>>>>> Nenad >>>>>> >>>>>> >>>>>> >>>>> >>> >> > > > -- > GPG me!! > > gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B > >
