Re: files not moved immediately to final destination from temp location after rsync returns with success
Yes, we are not trying to directly access the data via calls outside the file system so the VFS should have handled it correctly. Our logs don't show any type of errors related to SD card access or file system access in general. Thanks, Alistair On Sat, Apr 23, 2011 at 9:04 AM, Cameron Simpson c...@zip.com.au wrote: On 20Apr2011 19:29, Tony Abernethy t...@servasoftware.com wrote: | OK, I'll bite. | With all file system designs, there is a tradeoff between speed and safety. | This tradeoff occurs at all levels where there might be something that buffers information. | Writing into the middle of a structure can be incredibly slow if everything is done safely. | | Enter disk caches (Operating System, Disk controller, Disk itself) | Much much faster, BUT if mother nature pulls the plug, very weird things can happen. | | Expect everything to be very non-informative about the strategy used. | | With an SD card I would expect everything in the chain to NOT wait for the SD card to be finished. | Might even take a few minutes for the disk cache to be flushed to the SD card. | Makes no difference unless you pull the SD card prematurely | (or possibly in some cases actually look at what is actually on the card) And none of this matters. The view from within the OS should be consistent, regardless of the state of the disc buffering. Any other behaviour is a bug. To be explicit, if I went: mv huge_file /mnt/the-sd-card and the mv completes (the command exits), no examination of the sd card _via the filesystem_ (directory listings etc) will not show the huge_file on the card and the huge_file gone from its original location. I/O copying of data to the card may be happening in the background, but the logical view will be as it it were all complete. Of course, doing direct access to the card device, _outside the filesystem_, can should incompleteness. But that's not what the OP is doing AFAIK. Cheers, -- Cameron Simpson c...@zip.com.au DoD#743 http://www.cskk.ezoshosting.com/cs/ Some people have all the luck. When I find the guy with mine, I'm gonna kick his teeth in. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: files not moved immediately to final destination from temp location after rsync returns with success
The two rsync calls had a delay of 180 seconds between them but when the issue was seen the debug prints from the second rsync call were interspersed with the debug logs of the first rsync call. Our system startup scripts start just one instance of the script which has the two rsync calls. Also when the issue was later detected there was one instance of that script running. Have you come across errors with bash scripts where they run the command in the background even when not specified to. If so then that seems to be an explanation as to what took place. We did not see any SD card errors or file system errors in general. Regards, Alistair On Sat, Apr 23, 2011 at 9:06 PM, Wayne Davison way...@samba.org wrote: On Wed, Apr 20, 2011 at 12:44 PM, Alistair Dsouza alista...@gmail.comwrote: I came across an issue where it seems that the rsync call returned with a success but the files that it pulled are not moved immediately to its final destination. I think it more likely that you had 2 instances of the script running at the same time. You could use something like shlock to prevent that from happening. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: files not moved immediately to final destination from temp location after rsync returns with success
On 4/25/2011 8:10 AM, Alistair Dsouza wrote: The two rsync calls had a delay of 180 seconds between them but when the issue was seen the debug prints from the second rsync call were interspersed with the debug logs of the first rsync call. Our system startup scripts start just one instance of the script which has the two rsync calls. Also when the issue was later detected there was one instance of that script running. Have you come across errors with bash scripts where they run the command in the background even when not specified to. If so then that seems to be an explanation as to what took place. We did not see any SD card errors or file system errors in general. Regards, Alistair I use rsync a lot in a lot of scripts for all manner of special purposes, on various different platforms too, and I never had or even heard of any such problem with rsync as you describe. I never saw or heard of shell running jobs in the backgound asynchronously without being asked to so I don't think it's that either. (oh it certainly might be running in the background, but I doubt it's without being told to) But I have had no small number of similar problems that were plain my error in scripting or coding, or in configuring cron or some other service, or in a couple cases bugs in cron that resulted in duplicate jobs. So I think you have a scripting or other systemic problem to track down. It's possible you discovered a way to make rsync misbehave in that way, but you will have to prove it in order to provide the info needed for anyone else to debug it anyways. And in setting up and using the instrumentation to gather that proof, I bet you will end up discovering what's really wrong. -- bkw On Sat, Apr 23, 2011 at 9:06 PM, Wayne Davison way...@samba.org mailto:way...@samba.org wrote: On Wed, Apr 20, 2011 at 12:44 PM, Alistair Dsouza alista...@gmail.com mailto:alista...@gmail.com wrote: I came across an issue where it seems that the rsync call returned with a success but the files that it pulled are not moved immediately to its final destination. I think it more likely that you had 2 instances of the script running at the same time. You could use something like shlock to prevent that from happening. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
