Re: Transfering large files (was: Unison hangs on copy)
On Tue, Jul 26, 2011 at 03:24:11PM +0200, Claudius Hubig wrote:
Victor Munoz vmu...@macul.ciencias.uchile.cl wrote:
Both replicas are very large, and the few listed changes showed no
large files, or so I thought. But it turns out I was wrong, and a 20M
file was involved. I deleted the cache file, and reconstructed the
mirror, and I finally discovered that unison hangs when this
particular large files is involved. There are other large (and larger,
4 times larger, for instance) files in the replicas, but they are
identical, and unison doesn't complain. Only with this file, which
differs.
So I tried rsyncinc another file, 23 M in size (with --progress
option), and surprise, it stops when 11% transfer is reached. Tried
with scp, and same magic number: 11% and it stops.
What happens if you try to copy this file locally or on some other
device or force reading it with, for example, dd (dd
if=/path/to/yourfile of=/dev/null). Maybe the storage device on which
it is located has some bad blocks. You could also try to run scp
within strace to see where or what hangs.
I tried various combinations of this. I tried several different files,
all above 1M, none was fully transfered by rsync or scp :-( Looks like
an issue with keeping the connection alive. Using dd doesn't give
problems. scp within strace shows, at the end, messages like this:
write(6, )\377K\373)\377X\373#\377\373
\377n\373$\377\206\373\\377\231\373\36\377\257\373\\377\323\373...,
16384) = -1 EAGAIN (Resource temporarily unavailable)
poll([{fd=6, events=POLLOUT}], 1, -1) = ? ERESTART_RESTARTBLOCK (To
be restarted)
--- SIGALRM (Alarm clock) @ 0 (0) ---
By this time the copy is stalled.
Victor
--
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive:
http://lists.debian.org/20110730230655.gb18...@llacolen.ciencias.uchile.cl
Re: Transfering large files (was: Unison hangs on copy)
On Tuesday 26 July 2011 09:28:49 am Victor Munoz wrote: Fine, thanks for the speculation and the interest. You're welcome, and good luck! Randy Kramer -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/201107271853.41032.rhkra...@gmail.com
Re: Transfering large files (was: Unison hangs on copy)
I've reordered some of your comments for my convenience in responding. On Tuesday 26 July 2011 12:32:18 am Victor Munoz wrote: Thought you guys who helped me with my original question would like to know how this evolved. Thank you! So I tried rsyncinc another file, 23 M in size (with --progress option), and surprise, it stops when 11% transfer is reached. Tried with scp, and same magic number: 11% and it stops. I'm somewhat surprised, because when I was thinking of large files, I was thinking of CD image size files. (I used to transfer these over a 33 kbps modem, requiring about 65 hours per image, spread out over (typically) 5 nights. Oh, the good old days ;-) I don't really have any good thoughts to offer. I'd be looking to make sure that no temporary storage areas have been filled (presumably /tmp on either machine), in fact, I'd probably do a df for all partitions on both machines and make sure all of them have plenty of space (a nice big multiple of 23 MB--I don't really know how much storage rsync, or scp need, but I'm just grasping at straws looking for possibilities. While rsync may do some thinking (i.e., pause) during a 23 MB file transfer, I don't think scp would, but there still might be something that is timing out. All from me is just random speculation, however. Good luck--I hope someone else will have some better suggestions for you. Randy Kramer So now I think I have a problem with transfering large files. Somehow the upgrade caused this, because almost all previous transfers involved that 20M file, but now I am unable to copy this and other large files between both machines. I'm also losing my connection to the remote machine as I type this mail, so I have some kind of issue with the connection, and I really hope it is the the reason, and not the size of the files itself. -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/201107260725.41630.rhkra...@gmail.com
Re: Transfering large files (was: Unison hangs on copy)
Victor Munoz vmu...@macul.ciencias.uchile.cl wrote: Both replicas are very large, and the few listed changes showed no large files, or so I thought. But it turns out I was wrong, and a 20M file was involved. I deleted the cache file, and reconstructed the mirror, and I finally discovered that unison hangs when this particular large files is involved. There are other large (and larger, 4 times larger, for instance) files in the replicas, but they are identical, and unison doesn't complain. Only with this file, which differs. So I tried rsyncinc another file, 23 M in size (with --progress option), and surprise, it stops when 11% transfer is reached. Tried with scp, and same magic number: 11% and it stops. What happens if you try to copy this file locally or on some other device or force reading it with, for example, dd (dd if=/path/to/yourfile of=/dev/null). Maybe the storage device on which it is located has some bad blocks. You could also try to run scp within strace to see where or what hangs. So now I think I have a problem with transfering large files. Somehow the upgrade caused this, because almost all previous transfers involved that 20M file, but now I am unable to copy this and other large files between both machines. Is only this one file the problem or maybe other files as well? What happens if you copy (not move!) that file to another location on your drive and then try to scp it over? Is the content of the file thus that you could send it via plain netcat? If so, did you try something along the lines of: destination# nc -l yourport | md5sum source# dd if=/path/to/mysterious/file | nc otherhost.example.org yourport Please note that I have not tested the commands given here, so you might want to have a look at the manpages to make sure you got them correct. Best regards, Claudius -- If only you had a personality instead of an attitude. Please use GPG: ECB0C2C7 4A4C4046 446ADF86 C08112E5 D72CDBA4 http://chubig.net/ http://nightfall.org -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110726152411.23a3d...@ares.home.chubig.net
Re: Transfering large files (was: Unison hangs on copy)
On Tue, Jul 26, 2011 at 07:25:41AM -0400, Randy Kramer wrote: So I tried rsyncinc another file, 23 M in size (with --progress option), and surprise, it stops when 11% transfer is reached. Tried with scp, and same magic number: 11% and it stops. I'm somewhat surprised, because when I was thinking of large files, I was thinking of CD image size files. (I used to transfer these over a 33 kbps modem, requiring about 65 hours per image, spread out over (typically) 5 nights. Oh, the good old days ;-) Indeed, this is very frustrating, 20M is not a huge file, but it's the size which is giving me trouble (ok, it could be less, haven't tried, but I don't want to be too obsessive :-) I don't really have any good thoughts to offer. I'd be looking to make sure that no temporary storage areas have been filled (presumably /tmp on either machine), in fact, I'd probably do a df for all partitions on both machines and make sure all of them have plenty of space (a nice big multiple of 23 MB--I don't really know how much storage rsync, or scp need, but I'm just grasping at straws looking for possibilities. While rsync may do some thinking (i.e., pause) during a 23 MB file transfer, I don't think scp would, but there still might be something that is timing out. I thought exactly the same thing, that's why I tried scp after rsync. One of the machines reports this: df -h FilesystemSize Used Avail Use% Mounted on /dev/sda1 178G 146G 24G 87% / tmpfs1007M 0 1007M 0% /lib/init/rw udev 1002M 172K 1002M 1% /dev tmpfs1007M 0 1007M 0% /dev/shm This is the remote machine I'm trying to synchronize with every night. I will check at home the other one when I can. All from me is just random speculation, however. Good luck--I hope someone else will have some better suggestions for you. Fine, thanks for the speculation and the interest. Victor -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110726132849.gc11...@llacolen.ciencias.uchile.cl
Re: Transfering large files (was: Unison hangs on copy)
On Tue, Jul 26, 2011 at 03:24:11PM +0200, Claudius Hubig wrote: Victor Munoz vmu...@macul.ciencias.uchile.cl wrote: So I tried rsyncinc another file, 23 M in size (with --progress option), and surprise, it stops when 11% transfer is reached. Tried with scp, and same magic number: 11% and it stops. What happens if you try to copy this file locally or on some other device or force reading it with, for example, dd (dd if=/path/to/yourfile of=/dev/null). Maybe the storage device on which it is located has some bad blocks. You could also try to run scp within strace to see where or what hangs. Ok, I will try this and your other suggestions when I'm home later in the day, since I don't have access to that machine now. So now I think I have a problem with transfering large files. Somehow the upgrade caused this, because almost all previous transfers involved that 20M file, but now I am unable to copy this and other large files between both machines. Is only this one file the problem or maybe other files as well? What happens if you copy (not move!) that file to another location on your drive and then try to scp it over? After I noticed unison had problems with this file, I took another file, about the same size, in other directory, and did the rsync and scp experiments with those files. So I don't think it is a problem with that particular file or disk sector. But I will check your other suggestions as soon as I can. Thanks, Victor -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110726133425.gd11...@llacolen.ciencias.uchile.cl
Transfering large files (was: Unison hangs on copy)
On Fri, Jul 22, 2011 at 11:07:58PM -0400, Victor Munoz wrote: Today, I lost the ability to synchronize between two machines, one had lenny until today morning, and the other was sid. I was using apt-pinning to keep both unison versions compatible. I few days ago I posted a problem with unison, which appeared after a particular update I did. Someone suggested it could be a problem with large files, and I said no, no large files involved. Ok, I should I was wrong. Both replicas are very large, and the few listed changes showed no large files, or so I thought. But it turns out I was wrong, and a 20M file was involved. I deleted the cache file, and reconstructed the mirror, and I finally discovered that unison hangs when this particular large files is involved. There are other large (and larger, 4 times larger, for instance) files in the replicas, but they are identical, and unison doesn't complain. Only with this file, which differs. So I tried rsyncinc another file, 23 M in size (with --progress option), and surprise, it stops when 11% transfer is reached. Tried with scp, and same magic number: 11% and it stops. So now I think I have a problem with transfering large files. Somehow the upgrade caused this, because almost all previous transfers involved that 20M file, but now I am unable to copy this and other large files between both machines. I'm also losing my connection to the remote machine as I type this mail, so I have some kind of issue with the connection, and I really hope it is the the reason, and not the size of the files itself. Thought you guys who helped me with my original question would like to know how this evolved. Regards, Victor -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110726043218.ga7...@llacolen.ciencias.uchile.cl
