Re: rsync 10 times slower in Etch than in Sarge!
On Sat, Jan 19, 2008 at 03:23:57PM -0800, Daniel Burrows wrote: My theory about flushing even makes a testable prediction: if you run umount(8) first and have no other removable media, then sync(1) should complete almost immediately. I don't think I've ever done this, unlike the two points above, but you're welcome to try it yourself and let us know if it works. :-) Um, so a device does not need to be mounted for a sync to work? -- Chris. == -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: rsync 10 times slower in Etch than in Sarge!
On Jan 22, 2008 7:33 PM, Chris Bannister [EMAIL PROTECTED] wrote: On Sat, Jan 19, 2008 at 03:23:57PM -0800, Daniel Burrows wrote: My theory about flushing even makes a testable prediction: if you run umount(8) first and have no other removable media, then sync(1) should complete almost immediately. I don't think I've ever done this, unlike the two points above, but you're welcome to try it yourself and let us know if it works. :-) Um, so a device does not need to be mounted for a sync to work? Chris, I think this is some kind of joke :) If there are no big buffers to flush then 'sync' should run very fast. N.- -- http://arhuaco.org -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: rsync 10 times slower in Etch than in Sarge!
Chris writes: so a device does not need to be mounted for a sync to work? Sync only affects file systems on mounted devices. -- John Hasler -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: rsync 10 times slower in Etch than in Sarge!
on Fri, Jan 18, 2008 at 12:16:28PM -0500, Douglas A. Tutty wrote: Why issue a 'sync' instead of just unmounting and waiting until the thing stops flashing? On Fri, Jan 18, 2008 at 09:24:22AM -0900, Ken Irving [EMAIL PROTECTED] was heard to say: sync blocks, so you can tell from the command line when the job is done. Daniel Burrows [EMAIL PROTECTED] writes: So does umount. (I see you said that below, but I wanted to underline it again; it looks like Rudolfo overlooked it) After umount completes, it's safe to remove the stick. Rodolfo: I didn't overlook this, but: 1) in my experience, some time ago I used to extract the pendrive after `umount' completes and the device often got corrupted. When I asked this list for help to recover it, someone suggested to apply the `sync' option to the mount option: since then I put that option in my fstab, until this thread advised not to do so; Daniel Burrows [EMAIL PROTECTED] writes: I've been using Linux for 10 years, and I've always, always, always heard that umount had a side effect of flushing dirty buffers to disk and syncing the filesystem. But I did a little research before I replied, and I came up with some surprising information. [...] Rodolfo: 2) I measured the time, and it seems to me that the `sync' command takes a little longer than `umount' to stop. Daniel: I don't know for sure, but this could be because sync flushes the write buffers on all devices, not just the one you're pulling out. Thanks for taking care of the important matter. Then I presume that if I use both issues, i.e. first do `sync' and then `umount', I should feel quite safe: is that right? I did so some times now (with no `sync' option left in my fstab) and nothing bad has happened up to now. Bye Rodolfo -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: rsync 10 times slower in Etch than in Sarge!
On Fri, Jan 18, 2008 at 09:24:22AM -0900, Ken Irving [EMAIL PROTECTED] was heard to say: n Fri, Jan 18, 2008 at 12:16:28PM -0500, Douglas A. Tutty wrote: Why issue a 'sync' instead of just unmounting and waiting until the thing stops flashing? sync blocks, so you can tell from the command line when the job is done. So does umount. (I see you said that below, but I wanted to underline it again; it looks like Rudolfo overlooked it) After umount completes, it's safe to remove the stick. I guess this thread took a turn to USB drives, which would presumably be local, but the rsync operation might be on a remote system where the lights aren't visible. If you're rsyncing to a remote system, it doesn't matter whether you sync or not unless you've directly mounted it on your computer using something like NFS. sync flushes the local computer's write buffers, but if you've rsynced over ssh, all the actual writes took place on the remote machine. You'd have to ssh over there and run sync to ensure that your data was written to disk. Daniel -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: rsync 10 times slower in Etch than in Sarge!
on Fri, Jan 18, 2008 at 12:16:28PM -0500, Douglas A. Tutty wrote: Why issue a 'sync' instead of just unmounting and waiting until the thing stops flashing? On Fri, Jan 18, 2008 at 09:24:22AM -0900, Ken Irving [EMAIL PROTECTED] was heard to say: sync blocks, so you can tell from the command line when the job is done. Daniel Burrows [EMAIL PROTECTED] writes: So does umount. (I see you said that below, but I wanted to underline it again; it looks like Rudolfo overlooked it) After umount completes, it's safe to remove the stick. I didn't overlook this, but: 1) in my experience, some time ago I used to extract the pendrive after `umount' completes and the device often got corrupted. When I asked this list for help to recover it, someone suggested to apply the `sync' option to the mount option: since then I put that option in my fstab, until this thread advised not to do so; 2) I measured the time, and it seems to me that the `sync' command takes a little longer than `umount' to stop. For these two reasons I presume that it is not safe to extract the device just after `umounting' and without the sync option in fstab. What do you think? Thanks, Rodolfo -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: rsync 10 times slower in Etch than in Sarge!
On Sat, Jan 19, 2008 at 09:59:50PM +, Rodolfo Medina [EMAIL PROTECTED] was heard to say: on Fri, Jan 18, 2008 at 12:16:28PM -0500, Douglas A. Tutty wrote: Daniel Burrows [EMAIL PROTECTED] writes: So does umount. (I see you said that below, but I wanted to underline it again; it looks like Rudolfo overlooked it) After umount completes, it's safe to remove the stick. I apologize for misspelling your name -- I checked twice and still did it! I didn't overlook this, but: 1) in my experience, some time ago I used to extract the pendrive after `umount' completes and the device often got corrupted. When I asked this list for help to recover it, someone suggested to apply the `sync' option to the mount option: since then I put that option in my fstab, until this thread advised not to do so; I've been using Linux for 10 years, and I've always, always, always heard that umount had a side effect of flushing dirty buffers to disk and syncing the filesystem. But I did a little research before I replied, and I came up with some surprising information. First, neither the manpage of umount(8) nor the manpage of mount/umount/umount2(2) mention flushing buffers; neither does the libc documentation on these functions. Linux system-level documentation is usually fairly thorough, so I'd expect an important detail like this to be mentioned if it were true. On the other hand, this is such a fundamental thing, that I could believe the manpage author forgot to mention it, the same way that the unlink(2) manpage doesn't mention that future calls to open(2) on the unlinked name will fail. So I hit Google. As I said, I'd picked up a long time ago the notion that umount(8) was how to make it safe to yank removable media, and this belief is widely spread in the community. (e.g., Google for umount removable corruption and you'll find this advice being purveyed in many locations) So I figured that a little Googling would quickly turn up people squawking about their expectations being violated if it were true that umount(8) was not flushing. I didn't find much; most of the , but I did find a very strong claim by some guy named Christian Neumair that umount(8) does *not* flush write buffers, but that eject(1) does and should be used on USB media: http://blogs.gnome.org/cneumair/2006/02/11/ioctl-fsync-how-to-flush-block-device-buffers/#comment-94 An Ubuntu user named Toby Smithe claims here that umount should be avoided in favor of eject, also with no citation to supporting evidence: http://ubuntuforums.org/showpost.php?p=1737197postcount=5 However, virtually all the hits claim (also without any support) that umount *does* flush buffers. There are some intriguing references to a BLKFLSBUF ioctl that's supposed to be related to this (perhaps is flushes dirty buffers for a block device?), but no documentation on what it actually does that I can find. The .h says flush buffer cache, but there are a couple different things I could imagine that meaning. I also found a comment on an Ubuntu bug by John Dong that appears to say that sync(1) should be run after unmounting. However, I didn't read all the context -- it looks to me like perhaps he's talking about a situation where the GUI does an asynchronous umount in the background, so users don't know whether the umount is actually complete. https://bugs.launchpad.net/ubuntu/+source/kdebase/+bug/61946/comments/27 I also found a Sun manpage saying that Solaris needs you to run eject(1) on USB devices. Obviously this doesn't apply here until Debian/Solaris gets off the ground. Since it doesn't exist right now according to http://www.debian.org/ports, that will take a while. :-) So, the situation as I see it is that there's no authoritative information one way or another -- everyone's just repeating hearsay of one sort or another, as often happens on the Internet. Christian Neumair might have actual information, but he didn't say where he got his information from, so I don't know whether to believe him or not. I've also skimmed over the source for eject(1), the program being touted as somehow safer than umount() -- I haven't had time to read it thoroughly, nor do I intend to do so (although it's fairly short at 1449 lines of code), but I did do some quick searches in the source code. It makes no references to flush or sync anywhere in comments or source code: grep -ir flush and grep -ir sync come up empty. If it is doing something magical, though, I'd expect it to be via an ioctl. The only ioctls I see in there are: (1) a bunch of ioctls that are obviously CD-ROM manipulation (e.g., CDROMEJECT) and don't apply here. (2) some ioctls for sg devices (SG_*). These could be relevant. It appears that eject(1) sends the commands ALLOW_MEDIUM_REMOVAL, START_STOP, and START_STOP again. I don't know SCSI well, and there's no documentation for this, so it could be doing just about
Re: rsync 10 times slower in Etch than in Sarge!
On 2008-01-18 15:32 +0100, Rodolfo Medina wrote: I do my daily backup with the following command: $ rsync -vr --delete Mail News /mnt/sda (where `sda' is the pendrive), and I've calculated that in my new Debian Etch system the process is about 10 times slower than it was in Sarge! It is the same with both my desktop PC and my laptop. In /etc/fstab I have the following entry: /dev/sda/mnt/sdavfatsync,rw,user,noauto 0 0 . The backup that would take only a few minutes now takes almost one hour! Can anybody please explain this strange phenomena? Sven Joachim [EMAIL PROTECTED] writes: IIRC it's an issue with vfat file systems and the sync mount option in 2.6 kernels. I noticed the same phenomenon when I switched from 2.4 to 2.6; I see... the solution is not to use the sync option, that also increases your drive's lifetime. I put the `sync' option just to avoid drive corruption. If it is necessary to toggle it off, how can I be sure the process is terminated before I unmount the drive? You may want to add sync as the last command in your backup script do avoid data loss. I looked into `sync' manual but couldn't find out how to use this command. What should I say instead of `rsync -vr --delete Mail News /mnt/sda'? Thanks for your help Rodolfo -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: rsync 10 times slower in Etch than in Sarge!
the solution is not to use the sync option, that also increases your drive's lifetime. I put the `sync' option just to avoid drive corruption. If it is necessary to toggle it off, how can I be sure the process is terminated before I unmount the drive? This was discussed sometime ago in this list (someone wanted to be able to remove the USB drive anytime). Using sync is not a good idea with the USB pendrive. Check this thread: http://marc.info/?t=11792503843r=1w=2 ( I was wondering why I was aware of this old thread and I just noticed that I was the one who suggested using sync! ). You may want to add sync as the last command in your backup script do avoid data loss. I looked into `sync' manual but couldn't find out how to use this command. What should I say instead of `rsync -vr --delete Mail News /mnt/sda'? You use the same rsync command. You execute sync _after_ the rsync command. Cheers, N.- -- http://arhuaco.org -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: rsync 10 times slower in Etch than in Sarge!
On 2008-01-18 15:32 +0100, Rodolfo Medina wrote: I do my daily backup with the following command: $ rsync -vr --delete Mail News /mnt/sda (where `sda' is the pendrive), and I've calculated that in my new Debian Etch system the process is about 10 times slower than it was in Sarge! It is the same with both my desktop PC and my laptop. In /etc/fstab I have the following entry: /dev/sda/mnt/sda vfatsync,rw,user,noauto 0 0 . The backup that would take only a few minutes now takes almost one hour! Can anybody please explain this strange phenomena? IIRC it's an issue with vfat file systems and the sync mount option in 2.6 kernels. I noticed the same phenomenon when I switched from 2.4 to 2.6; the solution is not to use the sync option, that also increases your drive's lifetime. You may want to add sync as the last command in your backup script do avoid data loss. Sven -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: rsync 10 times slower in Etch than in Sarge!
n Fri, Jan 18, 2008 at 12:16:28PM -0500, Douglas A. Tutty wrote: Why issue a 'sync' instead of just unmounting and waiting until the thing stops flashing? sync blocks, so you can tell from the command line when the job is done. I guess this thread took a turn to USB drives, which would presumably be local, but the rsync operation might be on a remote system where the lights aren't visible. For that matter, umount also blocks... I often use sync before umount, but I suppose it's not really necessary. Ken -- Ken Irving, [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: rsync 10 times slower in Etch than in Sarge!
On Fri, Jan 18, 2008 at 11:53:10AM -0500, Nelson Castillo wrote: You may want to add sync as the last command in your backup script do avoid data loss. I looked into `sync' manual but couldn't find out how to use this command. What should I say instead of `rsync -vr --delete Mail News /mnt/sda'? You use the same rsync command. You execute sync _after_ the rsync command. Why issue a 'sync' instead of just unmounting and waiting until the thing stops flashing? Doug. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
rsync 10 times slower in Etch than in Sarge!
I do my daily backup with the following command: $ rsync -vr --delete Mail News /mnt/sda (where `sda' is the pendrive), and I've calculated that in my new Debian Etch system the process is about 10 times slower than it was in Sarge! It is the same with both my desktop PC and my laptop. In /etc/fstab I have the following entry: /dev/sda/mnt/sda vfatsync,rw,user,noauto 0 0 . The backup that would take only a few minutes now takes almost one hour! Can anybody please explain this strange phenomena? Thanks Rodolfo -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: rsync 10 times slower in Etch than in Sarge!
Rodolfo Medina [EMAIL PROTECTED] writes: I do my daily backup with the following command: $ rsync -vr --delete Mail News /mnt/sda (where `sda' is the pendrive), and I've calculated that in my new Debian Etch system the process is about 10 times slower than it was in Sarge! It is the same with both my desktop PC and my laptop. In /etc/fstab I have the following entry: /dev/sda/mnt/sda vfatsync,rw,user,noauto 0 0 . The backup that would take only a few minutes now takes almost one hour! Can anybody please explain this strange phenomena? Sven Joachim [EMAIL PROTECTED] writes: IIRC it's an issue with vfat file systems and the sync mount option in 2.6 kernels. I noticed the same phenomenon when I switched from 2.4 to 2.6; the solution is not to use the sync option, that also increases your drive's lifetime. Rodolfo: I put the `sync' option just to avoid drive corruption. If it is necessary to toggle it off, how can I be sure the process is terminated before I unmount the drive? Nelson Castillo [EMAIL PROTECTED] writes: This was discussed sometime ago in this list (someone wanted to be able to remove the USB drive anytime). Using sync is not a good idea with the USB pendrive. Check this thread: http://marc.info/?t=11792503843r=1w=2 Sven Joachim [EMAIL PROTECTED] writes: You may want to add sync as the last command in your backup script do avoid data loss. Nelson Castillo [EMAIL PROTECTED] writes: You use the same rsync command. You execute sync _after_ the rsync command. Douglas A. Tutty [EMAIL PROTECTED] writes: Why issue a 'sync' instead of just unmounting and waiting until the thing stops flashing? Ken Irving [EMAIL PROTECTED] writes: sync blocks, so you can tell from the command line when the job is done. I guess this thread took a turn to USB drives, which would presumably be local, but the rsync operation might be on a remote system where the lights aren't visible. For that matter, umount also blocks... I often use sync before umount, but I suppose it's not really necessary. Thanks to all for your help. I put off the `sync' option from my /etc/fstab. Douglas, my pendrive never stops flashing as long as it is inserted in the usb location: he must be in love with the PC. So I'm going to run the `sync' command after the backup and wait until the prompt returns back. Then I will unmount the drive. Will it this way be really _safe_? I.e., am I sure that the copy is finished when the `sync' command turns the prompt back? Thanks, Rodolfo -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
