Re: [Bulk] Re: ssh error - Write failed: Broken pipe
On 24/05/12 04:24 AM, elbbit wrote: On 23/05/12 22:45, Gary Dale wrote: On 23/05/12 04:53 PM, elbbit wrote: On your server put this line in your sshd_config after "Port 22": Port 443 Do a "service ssh restart" or "/etc/init.d/ssh restart". Connect using: ssh -p 443 user@host That has the same issue as the keepalive - it presumes that the connection is at fault when the problem only seems to happen when I'm doing large amounts of data movement on the server (NOT over the connection). Are you saying that if you log in by SSH then issue a command like cp large_file4gb large_copy4gb that even though the command produces no output you then get broken pipe? If so the connection is likely being terminated because it's idle (check your internet provider). The keepalive should prevent this. If you log in and do a ls -R / do you get the output or broken pipe? If you get broken pipe before the command completes this again suggests (to me) that you are subject to packet inspection. For me, putting sshd on different ports helps here like ports 22, 81, 443, 2022, 3389 and 20222. Some 3G providers here (UK) block port 22 altogether. One works best on port 81, on others 443 seems to work most of the time. Some network admins prefer to administer Windows boxes and (thus) leave port 3389 untouched. YMMV. Either way, in my experience broken pipe is usually a network problem. Fascinating. The keep-alive seems to prevent the error, which is occurring at the end of whatever command I executed on the server. This would be when the remote shell is returning to the command prompt. For example, if I mount usb drive on the remote server then copy some files to it, the pipe breaks after the copy completes. If I put the sequence in a script, complete it with a umount command at the end, the copy completes and the device is not mounted when the pipe breaks. This would suggest that it's the sending of data (the shell prompt) from the remote machine that is triggering the broken pipe. This is confirmed because I can leave the remote session sitting for hours before coming back to enter another command without problems. It's just when it's the remote server initiating data transfer after an extended period of inactivity that I get a broken pipe. Clearly the keep-alive is helping by preventing an extended period of inactivity. However, I'm puzzled by why the timeout only seems to operate against the server and not against my workstation. The remote servers are running Squeeze in all cases, while my workstation is running Wheezy. Has something changed between the two releases? -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4fbfd299.3030...@rogers.com
Re: ssh error - Write failed: Broken pipe
Karl E. Jorgensen wrote: > I've seen that before and it took us ages to narrow down - in our case > (admittedly your case may be different), it was the combination of > network interface MTU and packet fragmentation. I was just going to suggest exactly the same thing. And it took us ages to track down, too! Chris -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/a6v199x2pt@news.roaima.co.uk
Re: ssh error - Write failed: Broken pipe
On Wed, May 23, 2012 at 08:25:20PM +0100, Gary Dale wrote: > On 23/05/12 03:07 PM, elbbit wrote: > > On 23/05/12 19:46, Gary Dale wrote: > >> This is repeatable 100% of the time. > > I get this error on my ISP. I have found they are inspecting the > > internet packets and terminating if there is too little back-and-forth > > traffic. My ISP is a mobile phone network over 3G so they are probably > > trying to keep the bandwidth for phones only. > > > > I do "-o ServerAliveInterval=5" which has my ssh client ping the server > > every 5 seconds to keep it alive. > > > > To summarise, I get rid of broken pipe errors by using this command line > > > > ssh -o ServerAliveInterval=5 u...@server.com > > > > Hope this helps, > > elbbit > I can try it, but as I said, I have a stable connection when I'm not > doing anything. I can leave an ssh shell open overnight and it will be > there in the morning. It's only when I'm moving bits around on the > remote machine that I get disconnected. ah. I misunderstood your scenario in my previous mail... sorry. So the ssh breakage is (somehow) triggered by moving bits around on the remote machine only - not by moving lots of bits through the SSH connection. Anything in the logs on the remote machine? these files may contain relevant stuff: /var/log/syslog /var/log/daemon.log /var/log/kern.log .. basically all of them... look for files in /var/log which were touched around the time (or later) of the problem. Hope this helps -- Karl E. Jorgensen IT Operations -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120524114601.GR17862@hawking
Re: ssh error - Write failed: Broken pipe
Hi - entering this thread late... On Wed, May 23, 2012 at 07:46:29PM +0100, Gary Dale wrote: > I'm connecting at various times to different Debian/Squeeze servers from > my Debian/Wheezy workstation using ssh. No matter which server I connect > to, I find that if I move a lot of data on the remote server, I get > kicked out of my ssh session with a "Write failed: Broken pipe" error. I've seen that before and it took us ages to narrow down - in our case (admittedly your case may be different), it was the combination of network interface MTU and packet fragmentation. Reducing the MTU on our local network interface (on the server which runs the SSH client) to 1450 (rather than the default 1500) solved it consistently: sudo ifconfig eth0 mtu 1450 (to make the effect permanent, tweak /etc/network/interfaces and/or network manager config) Unfortunately we never found the root cause of the problem: Packet fragmentation is normally handled transparently by the TCP layer and there were no VPNs or VLANs involved. > This doesn't happen immediately. It usually occurs after some period of > time, during which time the cp or rsync command is moving data from one > location to another (gigabytes typically). So far as I can tell, the > operation finished successfully. Interesting. > > This is repeatable 100% of the time. > > typical commands: > cp /tmp/backups/* /media/dvd > rsync /tmp/backups/file /media/dvd/file Hm. These commands would not use the network - unless /tmp or /media happen to be NFS mounted, ISCSI-backed or similar - and neither involves ssh... > I have even encountered it with > if cmp /media/backupmedia/backup-set /home/shares/backup/backup-set; > then echo "OK"; fi Hm.. this example does not involve ssh at all!? And (apparantly) not network either? > although this sometimes works. I think it depends on the exact size of > the file. The cmp operation seems to not trigger it as fast as rsync or cp. > > If I'm not moving large amounts of data on the server, the ssh session > is stable and will sit for as long as I want. I only get disconnected > when I'm moving data around on the remote machine. That is consistent with the symptoms we saw before decreasing the MTU. Hope this helps -- Karl E. Jorgensen IT Operations -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120524113910.GQ17862@hawking
Re: [Bulk] Re: ssh error - Write failed: Broken pipe
On 23/05/12 22:45, Gary Dale wrote: > On 23/05/12 04:53 PM, elbbit wrote: >> On your server put this line in your sshd_config after "Port 22": >> >> Port 443 >> >> Do a "service ssh restart" or "/etc/init.d/ssh restart". Connect using: >> >> ssh -p 443 user@host >> > That has the same issue as the keepalive - it presumes that the > connection is at fault when the problem only seems to happen when I'm > doing large amounts of data movement on the server (NOT over the > connection). Are you saying that if you log in by SSH then issue a command like cp large_file4gb large_copy4gb that even though the command produces no output you then get broken pipe? If so the connection is likely being terminated because it's idle (check your internet provider). The keepalive should prevent this. If you log in and do a ls -R / do you get the output or broken pipe? If you get broken pipe before the command completes this again suggests (to me) that you are subject to packet inspection. For me, putting sshd on different ports helps here like ports 22, 81, 443, 2022, 3389 and 20222. Some 3G providers here (UK) block port 22 altogether. One works best on port 81, on others 443 seems to work most of the time. Some network admins prefer to administer Windows boxes and (thus) leave port 3389 untouched. YMMV. Either way, in my experience broken pipe is usually a network problem. -- elb...@gmail.com -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4fbdf036.7080...@gmail.com
Re: ssh error - Write failed: Broken pipe
On 23/05/12 03:25 PM, Gary Dale wrote: On 23/05/12 03:07 PM, elbbit wrote: On 23/05/12 19:46, Gary Dale wrote: This is repeatable 100% of the time. I get this error on my ISP. I have found they are inspecting the internet packets and terminating if there is too little back-and-forth traffic. My ISP is a mobile phone network over 3G so they are probably trying to keep the bandwidth for phones only. I do "-o ServerAliveInterval=5" which has my ssh client ping the server every 5 seconds to keep it alive. To summarise, I get rid of broken pipe errors by using this command line ssh -o ServerAliveInterval=5 u...@server.com Hope this helps, elbbit I can try it, but as I said, I have a stable connection when I'm not doing anything. I can leave an ssh shell open overnight and it will be there in the morning. It's only when I'm moving bits around on the remote machine that I get disconnected. Strange. When I turned on the keepalive, the cmp completed faster / I got the remote prompt back faster with no broken pipe. The stable connection could be because the operation returned faster or it could be that the keepalive fixed the problem. I'll have to check some more. -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4fbd5bb0.5010...@rogers.com
Re: [Bulk] Re: ssh error - Write failed: Broken pipe
On 23/05/12 04:53 PM, elbbit wrote: On 23/05/12 20:25, Gary Dale wrote: On 23/05/12 03:07 PM, elbbit wrote: On 23/05/12 19:46, Gary Dale wrote: This is repeatable 100% of the time. I get this error on my ISP. I have found they are inspecting the internet packets and terminating if there is too little back-and-forth traffic. My ISP is a mobile phone network over 3G so they are probably trying to keep the bandwidth for phones only. I do "-o ServerAliveInterval=5" which has my ssh client ping the server every 5 seconds to keep it alive. To summarise, I get rid of broken pipe errors by using this command line ssh -o ServerAliveInterval=5 u...@server.com Hope this helps, elbbit I can try it, but as I said, I have a stable connection when I'm not doing anything. I can leave an ssh shell open overnight and it will be there in the morning. It's only when I'm moving bits around on the remote machine that I get disconnected. In that case you may have the same problem I did some months ago. I'm on T-Mobile here and they like to packet inspect the TCP sessions. I found by putting all SSH traffic through port 443 I subverted their system -- they don't inspect encrypted traffic for which port 443 is the default for secure http. On your server put this line in your sshd_config after "Port 22": Port 443 Do a "service ssh restart" or "/etc/init.d/ssh restart". Connect using: ssh -p 443 user@host See if that helps. elbbit That has the same issue as the keepalive - it presumes that the connection is at fault when the problem only seems to happen when I'm doing large amounts of data movement on the server (NOT over the connection). -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4fbd5a89.3060...@rogers.com
Re: ssh error - Write failed: Broken pipe
On 23/05/12 20:25, Gary Dale wrote: > On 23/05/12 03:07 PM, elbbit wrote: >> On 23/05/12 19:46, Gary Dale wrote: >>> This is repeatable 100% of the time. >> I get this error on my ISP. I have found they are inspecting the >> internet packets and terminating if there is too little back-and-forth >> traffic. My ISP is a mobile phone network over 3G so they are probably >> trying to keep the bandwidth for phones only. >> >> I do "-o ServerAliveInterval=5" which has my ssh client ping the server >> every 5 seconds to keep it alive. >> >> To summarise, I get rid of broken pipe errors by using this command line >> >> ssh -o ServerAliveInterval=5 u...@server.com >> >> Hope this helps, >> elbbit > I can try it, but as I said, I have a stable connection when I'm not > doing anything. I can leave an ssh shell open overnight and it will be > there in the morning. It's only when I'm moving bits around on the > remote machine that I get disconnected. > In that case you may have the same problem I did some months ago. I'm on T-Mobile here and they like to packet inspect the TCP sessions. I found by putting all SSH traffic through port 443 I subverted their system -- they don't inspect encrypted traffic for which port 443 is the default for secure http. On your server put this line in your sshd_config after "Port 22": Port 443 Do a "service ssh restart" or "/etc/init.d/ssh restart". Connect using: ssh -p 443 user@host See if that helps. elbbit -- elb...@gmail.com -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4fbd4e46.4080...@gmail.com
Re: ssh error - Write failed: Broken pipe
On 23/05/12 03:07 PM, elbbit wrote: On 23/05/12 19:46, Gary Dale wrote: This is repeatable 100% of the time. I get this error on my ISP. I have found they are inspecting the internet packets and terminating if there is too little back-and-forth traffic. My ISP is a mobile phone network over 3G so they are probably trying to keep the bandwidth for phones only. I do "-o ServerAliveInterval=5" which has my ssh client ping the server every 5 seconds to keep it alive. To summarise, I get rid of broken pipe errors by using this command line ssh -o ServerAliveInterval=5 u...@server.com Hope this helps, elbbit I can try it, but as I said, I have a stable connection when I'm not doing anything. I can leave an ssh shell open overnight and it will be there in the morning. It's only when I'm moving bits around on the remote machine that I get disconnected. -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4fbd39a0.3050...@rogers.com
Re: ssh error - Write failed: Broken pipe
On 23/05/12 19:46, Gary Dale wrote: > This is repeatable 100% of the time. I get this error on my ISP. I have found they are inspecting the internet packets and terminating if there is too little back-and-forth traffic. My ISP is a mobile phone network over 3G so they are probably trying to keep the bandwidth for phones only. I do "-o ServerAliveInterval=5" which has my ssh client ping the server every 5 seconds to keep it alive. To summarise, I get rid of broken pipe errors by using this command line ssh -o ServerAliveInterval=5 u...@server.com Hope this helps, elbbit -- elb...@gmail.com -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4fbd3582.6060...@gmail.com
