Re: cannot bind socket - Need help with config file
Hello, On 11 January 2018 at 16:36, Jonathan Matthews wrote: > On 11 January 2018 at 00:03, Imam Toufique wrote: >> So, I have everything in the listen section commented out: >> >> frontend main >>bind :2200 >>default_backend sftp >>timeout client 5d >> >> >> #listen stats >> # bind *:2200 >> # mode tcp >> # maxconn 2000 >> # option redis-check >> # retries 3 >> # option redispatch >> # balance roundrobin >> >> #use_backend sftp_server >> backend sftp >> balance roundrobin >> server web 10.0.15.21:2200 check weight 2 >> server nagios 10.0.15.15:2200 check weight 2 >> >> Is that what I need, right? > > I suspect you won't need to have your *backend*'s ports changed to > 2200. Your SSH server on those machines is *probably* also your SFTP > server That's exactly right, your backend destination port should probably 22, there is no need to bump that one to 2200. > As an aside, it's not clear why you're trying to do this. You've > already hit the host-key-changing problem, and unless you have a > *very* specific use case, your users will hit the "50% of the time I > connect, my files have gone away" problem soon. So you've probably got > to solve the shared-storage problem on your backends ... which turns > them in to stateless SFTP-to-FS servers. > > In my opinion adding haproxy as a TCP proxy in your architecture adds > very little, if anything. If I were you, I'd strongly consider just > sync'ing the same host key to each server, putting their IPs in a > low-TTL DNS record, and leaving haproxy out of the setup. With DNS round-robin instead of haproxy you have the same exact requirements regarding SSH keys and filesystem synchronization, with all the disadvantages (no health checks, no direct control of the actual load-balancing, no stats, no logs, etc). I'm really not sure why you'd recommend DNS RR instead of haproxy here. Load-balancing a single-port TCP protocol between 2 backends is a bread and butter use-case for haproxy. Regards, Lukas
Re: cannot bind socket - Need help with config file
On 11 January 2018 at 00:03, Imam Toufique wrote: > So, I have everything in the listen section commented out: > > frontend main >bind :2200 >default_backend sftp >timeout client 5d > > > #listen stats > # bind *:2200 > # mode tcp > # maxconn 2000 > # option redis-check > # retries 3 > # option redispatch > # balance roundrobin > > #use_backend sftp_server > backend sftp > balance roundrobin > server web 10.0.15.21:2200 check weight 2 > server nagios 10.0.15.15:2200 check weight 2 > > Is that what I need, right? I suspect you won't need to have your *backend*'s ports changed to 2200. Your SSH server on those machines is *probably* also your SFTP server. I don't recall if you can serve a different/sync'd host key per port in sshd, but this might be a reason to run a different daemon on a higher port as you're doing. As an aside, it's not clear why you're trying to do this. You've already hit the host-key-changing problem, and unless you have a *very* specific use case, your users will hit the "50% of the time I connect, my files have gone away" problem soon. So you've probably got to solve the shared-storage problem on your backends ... which turns them in to stateless SFTP-to-FS servers. In my opinion adding haproxy as a TCP proxy in your architecture adds very little, if anything. If I were you, I'd strongly consider just sync'ing the same host key to each server, putting their IPs in a low-TTL DNS record, and leaving haproxy out of the setup. J
Re: cannot bind socket - Need help with config file
Thanks, Lukas! Sorry, I think I have been just replying to you by accidentally hitting the 'reply' button. So, I have everything in the listen section commented out: frontend main bind :2200 default_backend sftp timeout client 5d #listen stats # bind *:2200 # mode tcp # maxconn 2000 # option redis-check # retries 3 # option redispatch # balance roundrobin #use_backend sftp_server backend sftp balance roundrobin server web 10.0.15.21:2200 check weight 2 server nagios 10.0.15.15:2200 check weight 2 Is that what I need, right? thanks. On Wed, Jan 10, 2018 at 4:00 PM, Lukas Tribus wrote: > Hello Imam, > > > On Wed, Jan 10, 2018 at 11:49 PM, Imam Toufique > wrote: > > Lukas, > > > > Sorry to keep on dragging this, I am confused here. I will admit that I > > have not had the time to read the documentation on this. From what I was > > able to read, I slapped togather this config to get me started. > > > > I am not sure exactly what the 'listen' part do. From what I can > gather, I > > found this in the user documentation: > > Again please "Reply-All" so the mailing list remains CC'ed. > > > The frontend and listen functionality overlap, they can do the same > thing, with a slightly different syntax. You either use a frontend OR > a listen section. You don't use both for the same exact purpose. > > The frontend is fine, just delete everything related to the listen > section and that's it. > > > > > Regards, > Lukas > -- Regards, *Imam Toufique* *213-700-5485*
Re: cannot bind socket - Need help with config file
Hello Imam, On Wed, Jan 10, 2018 at 11:49 PM, Imam Toufique wrote: > Lukas, > > Sorry to keep on dragging this, I am confused here. I will admit that I > have not had the time to read the documentation on this. From what I was > able to read, I slapped togather this config to get me started. > > I am not sure exactly what the 'listen' part do. From what I can gather, I > found this in the user documentation: Again please "Reply-All" so the mailing list remains CC'ed. The frontend and listen functionality overlap, they can do the same thing, with a slightly different syntax. You either use a frontend OR a listen section. You don't use both for the same exact purpose. The frontend is fine, just delete everything related to the listen section and that's it. Regards, Lukas
Re: cannot bind socket - Need help with config file
Hi Imam, On Tue, Jan 9, 2018 at 6:54 PM, Imam Toufique wrote: > Hi Lukus, > > thanks again for your continued help and support! Here is my config file > with updates now: > > frontend main >bind :2200 >default_backend sftp >timeout client 5d > > > listen stats >bind *:2200 >mode tcp >maxconn 2000 >option redis-check >retries 3 >option redispatch >balance roundrobin > > > Please correct me if you see something that is not right. That's wrong. You are again configuring 2 services on a single port. In this case, the kernel will load-balance between the two causing chaos. What is the "listen stats" section supposed to do anyway in your configuration? Why do you need a main frontend and this listen section? > You asked about my SSH/SFTP use-case. Basically, here is my use-case. I > have several SFTP servers that I would like to load-balance. I was thinking > about using HAProxy to load-balance SFTP connections between my SFTP > servers. As I was testing my setup yesterday, I was sending sftp file > transfers to the HAproxy node, I noticed that HAProxy node CPU usage was > pretty high. I am beginning to wonder if it is the right setup for my > environment. > Is HAProxy is the right solution for SFTP server load-balancing? Load-balancing SSH/SFTP generally should be very easy to do, as SSH only uses a single port and doesn't have any layering violations (as opposed to FTP). The only thing to be aware of is the public key issue with different servers, as you are load-balancing between them. Use the same private key on all the backend server to avoid this problem. As for the high CPU usage, I'd recommend fixing the configuration first, before troubleshooting the CPU load. You may see strange effects due to unintended load-balancing. The rule is is simple: you are specifying the same listening port more than once in the configuration, then something is and will go wrong. You must have one single reference to port 2200 only. Lukas
Re: cannot bind socket - Need help with config file
Hi Lukus, thanks again for your continued help and support! Here is my config file with updates now: frontend main bind :2200 default_backend sftp timeout client 5d listen stats bind *:2200 mode tcp maxconn 2000 option redis-check retries 3 option redispatch balance roundrobin Please correct me if you see something that is not right. You asked about my SSH/SFTP use-case. Basically, here is my use-case. I have several SFTP servers that I would like to load-balance. I was thinking about using HAProxy to load-balance SFTP connections between my SFTP servers. As I was testing my setup yesterday, I was sending sftp file transfers to the HAproxy node, I noticed that HAProxy node CPU usage was pretty high. I am beginning to wonder if it is the right setup for my environment. Is HAProxy is the right solution for SFTP server load-balancing? thanks On Tue, Jan 9, 2018 at 2:12 AM, Lukas Tribus wrote: > Hello Imam, > > > On Tue, Jan 9, 2018 at 2:30 AM, Imam Toufique wrote: > > > > Hi Jonathan, and Lucas, > > > > Thanks for your replies. With your help, I was able to get it work > > partially. > > Please always CC the mailing list though. > > > > > frontend main *:2200 > >#bind *:22 > >default_backend sftp > >timeout client 1h > > While this works, it's causing a lot of confusion. Please do follow my > advice and DON'T specify the port in the frontend/listen line. Use the > bind directive instead. > So in this case: > > > frontend main > >bind :2200 > >default_backend sftp > >timeout client 1h > > It's much more readable like this. > > > > > listen stats > > #bind *:22 > > You disbled your stats section with this configuration. Either decide > for a port, or remove it if you don't need it. > > > > > But haproxy starts and I was able to get ssh to one of the servers. Now > I > > have a different problem where I get a ssh ket fingerprint error warning > and > > my connection drops. > > > > I get the error below: > > > > [vagrant@db ~]$ ssh file -p 2200 > > @@@ > > @WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ > > @@@ > > IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! > > Someone could be eavesdropping on you right now (man-in-the-middle > attack)! > > It is also possible that a host key has just been changed. > > The fingerprint for the RSA key sent by the remote host is > > SHA256:MHkXThp4cSltDn0/mRsq7Se+qcDz6cz1dD+kCiyE9e0. > > Please contact your system administrator. > > Add correct host key in /home/vagrant/.ssh/known_hosts to get rid of this > > message. > > Offending ECDSA key in /home/vagrant/.ssh/known_hosts:4 > > RSA host key for [file]:2200 has changed and you have requested strict > > checking. > > Host key verification failed > > > > It looks like host keys are changing, and the host key becomes unknown to > > both servers that are behind HAProxy. what do you recommend doing in a > case > > like this? > > That's what happens when you load-balance between 2 different SSH > servers with a different private key. What is it that you want to > achieve in the first place? > > > > cheers, > lukas > -- Regards, *Imam Toufique* *213-700-5485*
Re: cannot bind socket - Need help with config file
Hello Imam, On Tue, Jan 9, 2018 at 2:30 AM, Imam Toufique wrote: > > Hi Jonathan, and Lucas, > > Thanks for your replies. With your help, I was able to get it work > partially. Please always CC the mailing list though. > frontend main *:2200 >#bind *:22 >default_backend sftp >timeout client 1h While this works, it's causing a lot of confusion. Please do follow my advice and DON'T specify the port in the frontend/listen line. Use the bind directive instead. So in this case: > frontend main >bind :2200 >default_backend sftp >timeout client 1h It's much more readable like this. > listen stats > #bind *:22 You disbled your stats section with this configuration. Either decide for a port, or remove it if you don't need it. > But haproxy starts and I was able to get ssh to one of the servers. Now I > have a different problem where I get a ssh ket fingerprint error warning and > my connection drops. > > I get the error below: > > [vagrant@db ~]$ ssh file -p 2200 > @@@ > @WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ > @@@ > IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! > Someone could be eavesdropping on you right now (man-in-the-middle attack)! > It is also possible that a host key has just been changed. > The fingerprint for the RSA key sent by the remote host is > SHA256:MHkXThp4cSltDn0/mRsq7Se+qcDz6cz1dD+kCiyE9e0. > Please contact your system administrator. > Add correct host key in /home/vagrant/.ssh/known_hosts to get rid of this > message. > Offending ECDSA key in /home/vagrant/.ssh/known_hosts:4 > RSA host key for [file]:2200 has changed and you have requested strict > checking. > Host key verification failed > > It looks like host keys are changing, and the host key becomes unknown to > both servers that are behind HAProxy. what do you recommend doing in a case > like this? That's what happens when you load-balance between 2 different SSH servers with a different private key. What is it that you want to achieve in the first place? cheers, lukas
Re: cannot bind socket - Need help with config file
Hello Imam, On Mon, Jan 8, 2018 at 11:24 AM, Jonathan Matthews wrote: > On Mon, 8 Jan 2018 at 08:29, Imam Toufique wrote: >> >> [ALERT] 007/081940 (1416) : Starting frontend sftp-server: cannot bind >> socket [0.0.0.0:22] >> [ALERT] 007/081940 (1416) : Starting proxy stats: cannot bind socket >> [10.0.15.23:22] >> [ALERT] 007/081940 (1416) : Starting proxy stats: cannot bind socket >> [0.0.0.0:22] > > > I would strongly suspect that the server already has something bound to port > 22. It's probably your SSH daemon. > > You'll need to fix that, by dedicating either a different port or interface > to the SFTP listener. Correct. Also: - you can't bind the stats socket to the same port as your actual frontend - you are binding twice for the stats socket already (you must not have "bind :ABC" AND listen stats 1.2.3.4:ABC as that will cause 2 different sockets to be created - don't specify IP and port in the "listen" line to avoid that kind of confusing) Lukas
Re: cannot bind socket - Need help with config file
On Mon, 8 Jan 2018 at 08:29, Imam Toufique wrote: > [ALERT] 007/081940 (1416) : Starting frontend sftp-server: cannot bind > socket [0.0.0.0:22] > [ALERT] 007/081940 (1416) : Starting proxy stats: cannot bind socket [ > 10.0.15.23:22] > [ALERT] 007/081940 (1416) : Starting proxy stats: cannot bind socket [ > 0.0.0.0:22] > I would strongly suspect that the server already has something bound to port 22. It's probably your SSH daemon. You'll need to fix that, by dedicating either a different port or interface to the SFTP listener. J > -- Jonathan Matthews London, UK http://www.jpluscplusm.com/contact.html
