Re: rsync as a deliberately slow copy?
Le 29 sept. 10 à 22:35, Henri Shustak a écrit : I'm looking for a way to deliberately copy a large directory tree of files somewhat slowly, rather than as fast as the hardware will allow. Just do it to localhost - that way it's still a network connection, and you can use --bwlimit. Also, you could try nice to lower the priority rsync runs at Yes SSH (depending upon the system) may be resource intensive. As such, the suggestion of using nice is a really good suggestion! Although, if you are limiting the bandwidth then the resources required by SSH will be lower. It will really depend on a number of factors. Bottom line is nice is a great suggestion. You can do network rsync without ssh to localhost. But I don't know about efficiency and material saving of this solution. nice will not work to slow the transfer if rsync is the only intense process running at the moment in the computer. You would need also to create a fake process that do nothing with a big nice value. But, what I really doubt is about the efficacy of slowing rsync to preserve the material. Do you really think that running slower process help preserving the material? Best regards, Vitorio -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync as a deliberately slow copy?
On Thu 30 Sep 2010, Henri Shustak wrote: Yes SSH (depending upon the system) may be resource intensive. As such, the suggestion of using nice is a really good suggestion! Also, if running on linux, investigate ionice. This, given CFQ disk scheduling (the default in recent kernels, I believe) will cause rsync to play nice with IO resources, whereas nice handles CPU load. rsync is often much more IO intensive than CPU bound. I often run rsync with 'ionice -c3 rsync ' Paul -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync: link_stat . (in sprengel) failed: Permission denied (13)
Le 30 sept. 10 à 11:39, andreas.moro...@sb-brixen.it a écrit : Verify the user doing the transfer have write access on the sprengel module. Hello, do you mean the user on the client ( while testing it is root ) on the server where I started rsync as daemon as root or does it change group according to rsyncd.conf when running as daemon ? The user as whom the rsync session is starting at the server side. Regards, Vitorio -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Excluding most and including some problems continue.
Here is my rsync command as it currently stands. /usr/local/bin/rsync -vvv --stats -Pzrtpl --delete --password-file=/export/home/webuser/.appprod --log-file=/export/home/webuser/logs/rsync-log -F /export/home/ webu...@appprod::dprweb_extranet/ rsync-test Here is the current .rsync-filter file. + das + em + enf + internal + itb + medtox + pml + psb + reg + whs + htdocs + docs - /* - das/* - em/* - enf/* - internal/* - itb/* - medtox/* - pml/* - psb/* - reg/* - whs/* - htdocs/* This is doing pretty close to what I want it to do. Which is to mirror only the */htdocs/docs* in each of the ten directories (das,em,enf,etc.) in the base path of */export/home* and exclude the rest. Unfortunately there are some subdirectories in some of these selected */htdocs/docs* directories that are unintentionally being excluded by these rules. I.E. */export/home/enf/htdocs/docs/county/internal/*. [sender] hiding file enf/htdocs/docs/county/internal/gis0402.pdf because of pattern internal/* [per-dir .rsync-filter] [sender] hiding file enf/htdocs/docs/county/internal/gis1201.pdf because of pattern internal/* [per-dir .rsync-filter] [sender] hiding directory enf/htdocs/docs/county/internal/gis1201 because of pattern internal/* [per-dir .rsync-filter] [sender] hiding directory enf/htdocs/docs/county/internal/gis0402 because of pattern internal/* [per-dir .rsync-filter] Is there an easy way to remedy this in the base .rsync-filter file and|or the rsync command? Someway to say only exclude the base */export/home/internal/* directory, not any lower internal directories? OR is the only way to create sub .rsync-filter files in other directories? My concern with the latter option is that users are in control of these directories and can add and modify them at will. If I find all the special cases today, this is no guarantee that there won't be more special cases tomorrow. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
-r required when -a already specified in --files-from?
Hi all, I am using rsync v.3.0.7 (protocol v30) on ubuntu 10.04. My rsync line looks like this, cd /home/shantanu/ rsync -av --files-from=bin/include-to-rsync.dat /home/shantanu/ \ /media/transcend/backup_laptop/shantanu/ include-to-rsync had directories in this format, bank/ dwhelper/ clients/ When I ran the script I noticed that a files in bank/10-11 were not getting synced. Upon adding -r with -av the files got synced properly. I think when one supplies -a, -r is implied. I reconfirmed this from man-page where it says -a is equal to -rlptgoD. Am I missing anything from the man-page for this? Thanks, Shantanu -- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Excluding most and including some problems continue.
In 4ca449ea.2858.00a...@cdpr.ca.gov, on 09/30/10 at 08:27 AM, Ian Skinner iskin...@cdpr.ca.gov said: Hi, Here is my rsync command as it currently stands. /usr/local/bin/rsync -vvv --stats -Pzrtpl --delete --password-file=/export/home/webuser/.appprod --log-file=/export/home/webuser/logs/rsync-log -F /export/home/ webu...@appprod::dprweb_extranet/ rsync-test This is doing pretty close to what I want it to do. Which is to mirror only the */htdocs/docs* in each of the ten directories (das,em,enf,etc.) in the base path of */export/home* and exclude the rest. I'm not sure I entirely understand what you want, but what about + das/**/htdocs/docs* + em/**/htdocs/docs* etc. or possibly + das/**htdocs/docs* + em/**/htdocs/docs* etc. I'm not sure if the addtional slash is required without setting up a testcase. If you really want just the files matching */htdocs/docs/*, the above needs to change slightly. Steven -- -- Steven Levine stev...@earthlink.net eCS/Warp/DIY etc. www.scoug.com www.ecomstation.com -- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Excluding most and including some problems continue.
Steven Levine stev...@earthlink.net Thursday, September 30, 2010 11:14 AM I'm not sure I entirely understand what you want, but what about + das/**/htdocs/docs* + em/**/htdocs/docs* etc. or possibly + das/**htdocs/docs* + em/**/htdocs/docs* etc. I'm not sure what the difference between the first example and the second example is supposed to be? I don't see how that would address my needs, but I'm not sure what the double ** symbols represent? But there is no extra directories between the das and the htdocs directories in my use case. Maybe this representation would be clearer. /export /home /excluded_A /em /exclude_em-1 /htdocs /exclude_em_htdocs-1 /exclude_em_htdocs-2 /docs /exclude_em-2 /exclude_C /enf /exclude_enf-1 /htdocs /exclude_enf_htdocs-1 /exclude_enf_htdocs-2 /docs /exclude_enf-2 /exclude_E /das /exclude_das-1 /htdocs /exclude_das_htdocs-1 /exclude_das_htdocs-2 /docs /exclude_das-2 I want to mirror the following directories from the above example and exclude everything else. /export/home/em/htdocs/docs/* /export/home/enf/htdocs/docs/* /export/home/das/htdocs/docs/* (And seven more similar directories) I just tried this filter file somewhat based on your previous suggestion but it excluded everything. + das/htdocs/docs/* + em/htdocs/docs/* + enf/htdocs/docs/* + internal/htdocs/docs/* + itb/htdocs/docs/* + medtox/htdocs/docs/* + pml/htdocs/docs/* + psb/htdocs/docs/* + reg/htdocs/docs/* + whs/htdocs/docs/* - /* TIA Ian -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: -r required when -a already specified in --files-from?
Kulkarni Shantanu (m...@lists.shantanukulkarni.org) wrote on 30 September 2010 22:05: I am using rsync v.3.0.7 (protocol v30) on ubuntu 10.04. My rsync line looks like this, cd /home/shantanu/ rsync -av --files-from=bin/include-to-rsync.dat /home/shantanu/ \ /media/transcend/backup_laptop/shantanu/ include-to-rsync had directories in this format, bank/ dwhelper/ clients/ When I ran the script I noticed that a files in bank/10-11 were not getting synced. Upon adding -r with -av the files got synced properly. I think when one supplies -a, -r is implied. I reconfirmed this from man-page where it says -a is equal to -rlptgoD. Am I missing anything from the man-page for this? The man page entry for -a is pretty clear: The only exception to the above equivalence is when --files-from is specified, in which case -r is not implied. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: recent discussion regarding 'checksums'
Also as I think Wayne pointed out, using -c isn't very common in the first place. transfer. The delay due to I/O wait is going to be orders of magnitude higher than any reasonable hash computation. In a parallel life, I also mirrored some websites generated from database backends. For some unknown reason, they often changes embedded fixed length strings in the html, but the last-mod date didn't change. I never caught this till using -c for grins. Then I ran it against the entire filesystem and turned up some more interesting diffs. It was just a thousand or so small files so there wasn't much I/O time involved in -c. I've also compared find -ls's for grins [reported most recently, albeit indirectly as: Subject: Abysmal sparse file performance!] So the various use cases do exist out there. I've solved media file integrity with ZFS sha256 checksums on top of crypto block devices. And due to having some CPU to spare, still make and compare strong checksum indexes to catch new things as mentioned two paragraphs up. It's only marginally faster to do it separately than with rsync -c overhead. And the index is a bonus used for fastfind, etc. Anyways :) -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Excluding most and including some problems continue.
In 4ca4958c.2858.00a...@cdpr.ca.gov, on 09/30/10 at 01:50 PM, Ian Skinner iskin...@cdpr.ca.gov said: Hi, or possibly + das/**htdocs/docs* + em/**/htdocs/docs* etc. I'm not sure what the difference between the first example and the second example is supposed to be? That's my bad eyes. This should have been + das/**htdocs/docs* + em/**htdocs/docs* but it's not going to do what you really want. I don't see how that would address my needs, but I'm not sure what the double ** symbols represent? I recommend you read the man page. ** and *** can be very useful. But there is no extra directories between the das and the htdocs directories in my use case. OK. That's why I said I was not sure what you were asking. I want to mirror the following directories from the above example and exclude everything else. /export/home/em/htdocs/docs/* /export/home/enf/htdocs/docs/* /export/home/das/htdocs/docs/* (And seven more similar directories) OK. This is easier. I just tried this filter file somewhat based on your previous suggestion but it excluded everything. It's close, but you need to augment is a bit. Try + das/htdocs/docs/* + em/htdocs/docs/* etc. + */ - * and add --prune-empty-dirs to the command line. Also, if you really only want the contents of specific directories and not the content of any of the subdirecories, you can often avoid the recursive scan and use the --relative option and just list the source directories on the command line. Steven -- -- Steven Levine stev...@earthlink.net eCS/Warp/DIY etc. www.scoug.com www.ecomstation.com -- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Excluding most and including some problems continue.
Steven Levine stev...@earthlink.net Thursday, September 30, 2010 3:22 PM It's close, but you need to augment is a bit. Try + das/htdocs/docs/* + em/htdocs/docs/* etc. + */ - * Thanks for the suggestion, but that did not seem to produce the desired results. I did not look into why in detail, but a dry run produced files from directories I wanted to exclude and apparently not all the files I wanted to include. After a day of trail and error and internet searching I now have this that is really close. It copies all the directories I earlier identifies that where being falsely excluded. There are still two or three individual files that are not copying for some reason. I am looking into those now. + das + em + enf + internal + itb + medtox + pml + psb + reg + whs + htdocs + docs - /* - /das/* - /em/* - /enf/* - /internal/* - /itb/* - /medtox/* - /pml/* - /psb/* - /reg/* - /whs/* - htdocs/* -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: -r required when -a already specified in --files-from?
* Carlos Carvalho car...@fisica.ufpr.br [101001 06:16]: The man page entry for -a is pretty clear: The only exception to the above equivalence is when --files-from is specified, in which case -r is not implied. Thanks a lot Carlos. I am still not sure how I missed that in the manual. Thanks. Shantanu -- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Excluding most and including some problems continue.
In 4ca4be08.2858.00a...@cdpr.ca.gov, on 09/30/10 at 04:42 PM, Ian Skinner iskin...@cdpr.ca.gov said: Hi, + das/htdocs/docs/* + em/htdocs/docs/* etc. + */ - * Thanks for the suggestion, but that did not seem to produce the desired results. I did not look into why in detail, but a dry run produced files from directories I wanted to exclude and apparently not all the files I wanted to include. Did you add --prune-empty-directorys to the command line? This filter setup along with --prune-empty-directories will copy only the files in the named directores, which is my understanding of what you want. After a day of trail and error and internet searching I now have this that is really close. Looks overly complex to me. Taking your example layout and using this filter set + das/htdocs/docs/* + em/htdocs/docs/* + enf/htdocs/docs/* + internal/htdocs/docs/* + itb/htdocs/docs/* + medtox/htdocs/docs/* + pml/htdocs/docs/* + psb/htdocs/docs/* + reg/htdocs/docs/* + whs/htdocs/docs/* + */ - * and this command line rsync --dry-run --prune-empty-dirs --itemize-changes -a -F export\ to\ I get .d..t.. ./ cd+ home/ cd+ home/das/ cd+ home/das/htdocs/ cd+ home/das/htdocs/docs/ f+ home/das/htdocs/docs/SHLNotes.txt cd+ home/em/ cd+ home/em/htdocs/ cd+ home/em/htdocs/docs/ f+ home/em/htdocs/docs/SHLNotes.txt Which I think is what you want. Every subdirectory contains a file. Good luck, Steven -- -- Steven Levine stev...@earthlink.net eCS/Warp/DIY etc. www.scoug.com www.ecomstation.com -- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
