cheers, ; Yuan
On 07/26/2015 12:13 AM, Gmail wrote:
I am uncertain about syntax but the diff is "appname" of sorts for the port 80 listener. Someone may comment with more details ;Earlier = *listen 0.0.0.0:80 [ no app name string ]Now = listen jokefire 0.0.0.0:80 [ app name == jokefire and also no astrisk visible ]Maybe attempt restart without any appname and also with/without that asterisk.Deep dives need more awareness. Glad all settled. Cheers, ; Yuan On 07/25/2015 11:51 PM, Tim Dunphy wrote:Yuan, maybe something here http://lnxmon.com/haproxy/Thanks, ; YuanI modified a config from your blog that you showed me and came up with this:global log 127.0.0.1 local0 notice maxconn 2000 user haproxy group haproxy defaults log global mode http option httplog option dontlognull retries 3 option redispatch timeout connect 5000 timeout client 10000 timeout server 10000 listen jokefire 0.0.0.0:80 mode http stats enable stats uri /haproxy?stats stats realm Strictly\ Private stats auth admin:secret balance roundrobin option httpclose option forwardfor server varnish1 10.10.10.5:80 check server varnish2 10.10.10.6:80 check listen mysql-cluster bind 0.0.0.0:3306 mode tcp balance roundrobin maxconn 5200 option mysql-check user haproxy_root server mysql-1 10.10.10.7:3306 check server mysql-2 10.10.10.8:3306 check And that seemed to work. I can see that both ports are listening now: [root@ha1:/etc/haproxy] #lsof -i :80 -i :3306 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME haproxy 27136 haproxy 4u IPv4 7563913 0t0 TCP *:http (LISTEN) haproxy 27136 haproxy 6u IPv4 7563915 0t0 TCP *:mysql (LISTEN)Although I am not aware of the real difference between this and my previousconfig that allows this to work is.Not a huge issue at this point since it's working. But if anyone wants totake a stab at this, be my guest! Thanks, Tim On Sat, Jul 25, 2015 at 12:15 AM, Gmail <longwuy...@gmail.com> wrote:maybe something here http://lnxmon.com/haproxy/ Thanks, ; Yuan On 07/25/2015 12:10 PM, Igor Cicimov wrote:You need to run haproxy as root to bind to ports lower than 1024 On 25/07/2015 1:36 PM, "Tim Dunphy" <bluethu...@gmail.com> wrote: Hi Yuan,Nice.Do you use selinux in prod. regards, ; YuanYep! Actually I use it every chance I get. Prod/stage/dev and my own hobbyenvironments. And right now actually what I was discussing was a hobbyenvironment.And actually if I could bother you guys one more time, I do have one moreissue to solve. LOLAnd this time it's guaranteed not to be an SELinux issue. Because I triedrunning haproxy with SELInux on and off this time.But what's happening now, is that HA/Proxy is not creating the http port for the 'stats' interface. I've setup stats to listen on port 80. But forsome reason that's not happening. Here's my config one more time, with the trouble part in bold: global log 127.0.0.1 local0 notice user haproxy group haproxy defaults log global retries 2 timeout connect 3000 timeout server 5000 timeout client 5000 listen mysql-cluster bind 0.0.0.0:3306 mode tcp option mysql-check user haproxy_check balance roundrobin server mysql-1 52.3.28.48:3306 check server mysql-2 52.2.0.176:3306 check *listen 0.0.0.0:80 <http://0.0.0.0:80> mode http stats enablestats uri / stats realm Strictly\ Private stats auth admin:secret* Currently haproxy is listening on the first port specified* - 3306 - *butnot listening on port 80. Observe: [root@ha1:/etc/haproxy] #lsof -i :3306 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME *haproxy 11653 haproxy 4u IPv4 7145270 0t0 TCP *:mysql (LISTEN)* [root@ha1:/etc/haproxy] #lsof -i :80 [root@ha1:/etc/haproxy] # [root@ha1:/etc/haproxy] #telnet localhost 80 Trying 127.0.0.1... telnet: connect to address 127.0.0.1: Connection refused Port 80 simply isn't listening. And this time, I can't blame it on SELinux being on: [root@ha1:/etc/haproxy] #getenforce PermissiveI've grepped thru /var/log/messages but not turned up any clues to thisone. And I really would like to get the stats interface up and running. Any thoughts here? I'm wondering what I can do to get stats working. Thanks, Tim On Fri, Jul 24, 2015 at 10:52 PM, Gmail <longwuy...@gmail.com> wrote: Nice.Do you use selinux in prod. regards, ; Yuan On 07/25/2015 09:17 AM, Tim Dunphy wrote: Bingo!!!The problem was with SELinux. Not sure what took me so long to think ofit...!!!So set the mysql listener back to port 3306. Turned off SELinux with setenforce 0. Then it started right up!!! And port 3306 was listening.Then I consulted with audit2why and saw the following: type=AVC msg=audit(1437786617.963:28856863): avc: denied { name_connect } for pid=29175 comm="haproxy" dest=3306 scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:object_r:mysqld_port_t:s0 tclass=tcp_socket Was caused by: The boolean haproxy_connect_any was set incorrectly. Description: Allow haproxy to connect any Allow access by executing: # *setsebool -P haproxy_connect_any 1*I just ran that command you see above in bold, and then all was rightwith the world. [root@ha1:/etc/haproxy] #systemctl status haproxy haproxy.service - HAProxy Load BalancerLoaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled) Active: active (running) since Sat 2015-07-25 01:14:53 UTC; 33sago Main PID: 30618 (haproxy-systemd) CGroup: /system.slice/haproxy.service ├─30618 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid├─30619 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p/run/haproxy.pid -Ds└─30620 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p/run/haproxy.pid -Ds Jul 25 01:14:53 ha1 systemd[1]: Starting HAProxy Load Balancer... Jul 25 01:14:53 ha1 systemd[1]: Started HAProxy Load Balancer. Jul 25 01:14:53 ha1 haproxy-systemd-wrapper[30618]: haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds [root@ha1:/etc/haproxy] #lsof -i :3306 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME haproxy 30620 haproxy 1u IPv4 7075172 0t0 TCP ha1.example.com:55499->ec2-52-2-0-xxx.compute-1.amazonaws.com:mysql (SYN_SENT) haproxy 30620 haproxy 4u IPv4 7074731 0t0 TCP *:mysql (LISTEN)Thanks for nudging me in the right direction. All I had to hear was theword 'selinux' and from there it all fell into place! Thanks!! TimOn Fri, Jul 24, 2015 at 8:20 PM, Gmail <longwuy...@gmail.com> wrote:I could be completely wrong here and I am curious to know the answermyself. Please don't take this as a solution, just my thoughts.First, you can not use backend ip-address of 10.x.x.x subnet becauseeachaccount's VPC is seggregated. If you do want to use 10.X.X.X ipadressyou have to setup a inter VPC endpoint in AWS. I would just use EIP.For the port 3306, try to use nc to listen on that port or iperf. Doyo uhave iptables turned on. I would check "systemctl -l status haproxy.service"I would check lsof -i why can't bind to 3306 on loopback ipaddress.I would check iptables or selinux preventing the bind.It wil be interesting to know the source ipaddress of MySQL client ec2instance. Interesting if you can Copy/paste output of "telnet<haproxynode_ipaddress> 3306" from mysql client ec2 instance , here. Interesting if you can Copy/paste output of "telnet 10.10.10.10 3306"from haproxy ec2 instances, here.Interesting if you can Copy/paste output of "telnet 10.10.10.11 3306"from haproxy ec2 instances, here.I I was doing this, maybe I would consider testing something like ;.. frontend mysql_lb_fe 0.0.0.0:3306 .... acl host_myql_lb hdr(host) -i mysql-lb .. .. use_backend mysql_lb_backend if host mysql_lb .. .. backend mysql_lb_be .. .. option mysql-check user haproxy_check balance roundrobin server mysql-1 10.10.10.10:3306 check server mysql-2 10.10.10.11:3306 check Thanks, ; Yuan On 07/25/2015 06:41 AM, Tim Dunphy wrote: Hello Nenad,Jul 24 03:44:18 ha1 haproxy-systemd-wrapper[25034]: [ALERT]204/034418 (25035) : *Starting proxy mysql-cluster: cannotbind s...:3306]* Nothing listening on the port I'm trying to bind to: 3306[root@ha1:~] #ss -lpt | fgrep 3306 [root@ha1:~] #lsof -i :3306 [root@ha1:~] #netstat -tulpn | grep -i listen | grep 3306 [root@ha1:~] #While we're on the subject of listening ports, here's a list of alllistening ports on the haproxy host: [root@ha1:~] #netstat -tulpn | grep -i listen tcp 0 0 0.0.0.0:35145 0.0.0.0:* LISTEN - tcp 0 0 0.0.0.0:56814 0.0.0.0:* LISTEN 16346/rpc.statd tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 16455/rpcbind tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 16396/sshd tcp6 0 0 :::49349 :::* LISTEN 16346/rpc.statd tcp6 0 0 :::111 :::* LISTEN 16455/rpcbind tcp6 0 0 :::47314 :::* LISTEN - tcp6 0 0 :::22 :::* LISTEN 16396/sshdI thought I was beginning to understand this problem. That haproxywastrying to bind on port 3306 from the mysql host on another machine.But come to think of it, that doesn't make a lot of sense.Because I already have haproxy setup for some web servers, and thereitcreates port 80 on the haproxy node. It's not trying to connect to aforeign source. Not sure where I got that idea!!I also tried binding the mysql section to another port that wasn't inuse.I tried port 3307,3308. I even tried binding the mysql section of the config to a weird port I just grabbed off of the top of my head. Itried binding it to port 4444. And there I still got a bind error:[ALERT] 204/223303 (13081) : Starting proxy mysql-cluster: cannotbind socket [0.0.0.0:4444]Now watch this!! If I bind the mysql section to port 80 insteadof any other port.. haproxy starts up without complaint! listen mysql-cluster bind 0.0.0.0:80 mode tcp option mysql-check user haproxy_check balance roundrobin server mysql-1 10.0.0.xxx :3306 check server mysql-2 10.0.0.xxx:3306 check [root@ha1:/etc/haproxy] #systemctl status haproxy haproxy.service - HAProxy Load Balancer Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled)Active: active (running) since Fri 2015-07-24 22:35:03 UTC; 4sago Main PID: 13213 (haproxy-systemd) CGroup: /system.slice/haproxy.service ├─13213 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid├─13214 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg-p /run/haproxy.pid -Ds└─13215 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg-p /run/haproxy.pid -Ds Jul 24 22:35:03 ha1 systemd[1]: Starting HAProxy Load Balancer... *Jul 24 22:35:03 ha1 systemd[1]: Started HAProxy Load Balancer.* Jul 24 22:35:03 ha1 haproxy-systemd-wrapper[13213]: haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds Ok. What...the...heck!!So why do you think that haproxy is only happy starting up on port80? Iwould think that I should be able to specify any arbitrary port forit to listen on in a 'listen' sub-block.I guess I could have my app contact the database using port 80. Butthat'sa little... weird. I installed haproxy using yum from the 'updates' repository. Is there any reason anyone can think of as to why haproxyrefuses to start on any port other than port 80?? Thanks, TimOn Fri, Jul 24, 2015 at 4:59 PM, Nenad Merdanovic <ni...@nimzo.info>wrote: Hello Tim, On Fri, Jul 24, 2015 at 1:46 PM, Tim Dunphy <bluethu...@gmail.com <mailto:bluethu...@gmail.com>> wrote:listen mysql-cluster bind 127.0.0.1:3306 <http://127.0.0.1:3306> mode tcp option mysql-check user haproxy_check balance roundrobin server mysql-1 10.10.10.10:3306 < http://10.10.10.10:3306> checkserver mysql-2 10.10.10.11:3306 <http://10.10.10.11:3306> checkJul 24 03:44:18 ha1 haproxy-systemd-wrapper[25034]: [ALERT]204/034418 (25035) : *Starting proxy mysql-cluster: cannot bind s...:3306]* Can you check if something is listening on 127.0.0.1:3306(netstat, ss, lsof)? For example: ss -lpt | fgrep 3306 Regards, Nenad-- GPG me!! gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
