[PATCH] Exceeding white spaces and a typo
From: Andrea Gelmini --- rsync3.txt | 22 +++--- 1 files changed, 11 insertions(+), 11 deletions(-) diff --git a/rsync3.txt b/rsync3.txt index 42d77dc..293680f 100644 --- a/rsync3.txt +++ b/rsync3.txt @@ -1,6 +1,6 @@ -*- indented-text -*- -Notes towards a new version of rsync +Notes towards a new version of rsync Martin Pool , September 2001. @@ -13,7 +13,7 @@ Good things about the current implementation: - Fairly reliable. - - The choice of runnning over a plain TCP socket or tunneling over + - The choice of running over a plain TCP socket or tunneling over ssh. - rsync operations are idempotent: you can always run the same @@ -51,7 +51,7 @@ Bad things about the current implementation: hard to modify/extend - Both the program and the protocol assume a single non-interactive -one-way transfer +one-way transfer - A list of all files are held in memory for the entire transfer, which cripples scalability to large file trees @@ -88,7 +88,7 @@ Protocol philosophy: Questionable features: - These are neat, but not necessarily clean or worth preserving. + These are neat, but not necessarily clean or worth preserving. - The remote rsync can be wrapped by some other program, such as in tridge's rsync-mail scripts. The general feature of sending and @@ -100,7 +100,7 @@ Desirable features: These don't really require architectural changes; they're just something to keep in mind. - + - Synchronize ACLs and extended attributes - Anonymous servers should be efficient @@ -122,7 +122,7 @@ Desirable features: Alternatively, as long as transfers are idempotent, we can just restart the whole thing. [NFSv4] - - Scripting support. + - Scripting support. - Propagate atimes and do not modify them. This is very ugly on Unix. It might be better to try to add O_NOATIME to kernels, and @@ -224,7 +224,7 @@ Scripting hooks: - What basis file to use - Logging - + - Whether to allow transfers (for public servers) - Authentication @@ -275,7 +275,7 @@ Pie-in-the-sky features: These might have a severe impact on the protocol, and are not clearly in our core requirements. It looks like in many of them - having scripting hooks will allow us + having scripting hooks will allow us - Transport over UDP multicast. The hard part is handling multiple destinations which have different basis files. We can look at @@ -381,7 +381,7 @@ Uses: http://www.ietf.org/proceedings/00jul/00july-133.htm#P24510_1276764 - Sync with PDA - + - Network backup systems - CVS filemover @@ -431,7 +431,7 @@ Atomic updates: Scalability: - + We should aim to work well on machines in use in a year or two. That probably means transfers of many millions of files in one batch, and gigabytes or terabytes of data. @@ -466,4 +466,4 @@ Related work: - http://freshmeat.net/search/?site=Freshmeat&q=mirror§ion=projects - BitTorrent -- p2p mirroring -http://bitconjurer.org/BitTorrent/ \ No newline at end of file +http://bitconjurer.org/BitTorrent/ -- 1.7.0.2 -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
include/exclude problems
Dear Wayne, I am using rsync to synchronize my working files from home computer to the work computer. I want to sync only the files in some directories and no .files. However, I do need to sync several files from .mozilla director (addressbook, calendar and bookmarks) which are located in the .directory. I use the following command and the "excluded-utk.txt" file rsync -aunvz --delete -e ssh --exclude-from 'excluded-utk.txt' /home/vitaly/ utk:/home/vitaly from excluded-utk.txt file: + .mozilla-thunderbird/ + .mozilla-thunderbird/afc5fv2p.default/ + .mozilla-thunderbird/afc5fv2p.default/abook.mab + .mozilla-thunderbird/afc5fv2p.default/storage.sdb - .mozilla-thunderbird/afc5fv2p.default/ImapMail - .mozilla-thunderbird/afc5fv2p.default/Mail - .mozilla-thunderbird/afc5fv2p.default/lock - .mozilla-thunderbird/afc5fv2p.default/extensions - lena - data - doctoral - install - local - stuff - temp - web - Hosts - .* The output is: ./ .mozilla-thunderbird/ .mozilla-thunderbird/afc5fv2p.default/ .mozilla-thunderbird/afc5fv2p.default/10357267.s .mozilla-thunderbird/afc5fv2p.default/abook.mab .mozilla-thunderbird/afc5fv2p.default/blocklist.xml .mozilla-thunderbird/afc5fv2p.default/files .mozilla-thunderbird/afc5fv2p.default/junklog.html .mozilla-thunderbird/afc5fv2p.default/localstore.rdf .mozilla-thunderbird/afc5fv2p.default/panacea.dat .mozilla-thunderbird/afc5fv2p.default/storage.sdb .mozilla-thunderbird/afc5fv2p.default/training.dat refs/computers/ refs/computers/excluded-utk.txt refs/computers/excluded-utk.txt-def No "storage.sdb" is transferred. Can you help me with figuring out why this is the case? Thanks a lot, Vitaly -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: include/exclude problems
On Thu 25 Mar 2010, Vitaly V. Ganusov wrote: > > from excluded-utk.txt file: > > + .mozilla-thunderbird/ > + .mozilla-thunderbird/afc5fv2p.default/ > + .mozilla-thunderbird/afc5fv2p.default/abook.mab > + .mozilla-thunderbird/afc5fv2p.default/storage.sdb > - .mozilla-thunderbird/afc5fv2p.default/ImapMail > - .mozilla-thunderbird/afc5fv2p.default/Mail [snip] > > The output is: > > ./ > .mozilla-thunderbird/ > .mozilla-thunderbird/afc5fv2p.default/ > .mozilla-thunderbird/afc5fv2p.default/10357267.s > .mozilla-thunderbird/afc5fv2p.default/abook.mab > .mozilla-thunderbird/afc5fv2p.default/blocklist.xml > .mozilla-thunderbird/afc5fv2p.default/files > .mozilla-thunderbird/afc5fv2p.default/junklog.html > .mozilla-thunderbird/afc5fv2p.default/localstore.rdf > .mozilla-thunderbird/afc5fv2p.default/panacea.dat > .mozilla-thunderbird/afc5fv2p.default/storage.sdb [snip] > No "storage.sdb" is transferred. Can you help me with figuring out I definitely see it in your output. Why do you say it's not transferred? Paul (not Wayne :) -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: include/exclude problems
You are right, the reason that I did not see that before is that the file on the server was newer than on my desktop. It now all works. However, it seems like a very messy way to sync a given file (by excluding bunch of other files that are not needed). Would you suggest a better way? Could you suggest a simple way/script to generate such a file (see attached) with only a few files being included in the rsync line? Many thanks in advance, Vitaly PS. My current "exclude-utk.txt" file is attached. Paul Slootman wrote: On Thu 25 Mar 2010, Vitaly V. Ganusov wrote: from excluded-utk.txt file: + .mozilla-thunderbird/ + .mozilla-thunderbird/afc5fv2p.default/ + .mozilla-thunderbird/afc5fv2p.default/abook.mab + .mozilla-thunderbird/afc5fv2p.default/storage.sdb - .mozilla-thunderbird/afc5fv2p.default/ImapMail - .mozilla-thunderbird/afc5fv2p.default/Mail [snip] The output is: ./ .mozilla-thunderbird/ .mozilla-thunderbird/afc5fv2p.default/ .mozilla-thunderbird/afc5fv2p.default/10357267.s .mozilla-thunderbird/afc5fv2p.default/abook.mab .mozilla-thunderbird/afc5fv2p.default/blocklist.xml .mozilla-thunderbird/afc5fv2p.default/files .mozilla-thunderbird/afc5fv2p.default/junklog.html .mozilla-thunderbird/afc5fv2p.default/localstore.rdf .mozilla-thunderbird/afc5fv2p.default/panacea.dat .mozilla-thunderbird/afc5fv2p.default/storage.sdb [snip] No "storage.sdb" is transferred. Can you help me with figuring out I definitely see it in your output. Why do you say it's not transferred? Paul (not Wayne :) #+ local/ #+ local/bin/ #+ local/bin/* + .mozilla-thunderbird/ + .mozilla-thunderbird/afc5fv2p.default/ + .mozilla-thunderbird/afc5fv2p.default/abook.mab + .mozilla-thunderbird/afc5fv2p.default/storage.sdb - .mozilla-thunderbird/afc5fv2p.default/ImapMail - .mozilla-thunderbird/afc5fv2p.default/Mail - .mozilla-thunderbird/afc5fv2p.default/lock - .mozilla-thunderbird/afc5fv2p.default/extensions - .mozilla-thunderbird/afc5fv2p.default/10357267.s - .mozilla-thunderbird/afc5fv2p.default/blocklist.xml - .mozilla-thunderbird/afc5fv2p.default/files - .mozilla-thunderbird/afc5fv2p.default/junklog.html - .mozilla-thunderbird/afc5fv2p.default/localstore.rdf - .mozilla-thunderbird/afc5fv2p.default/panacea.dat - .mozilla-thunderbird/afc5fv2p.default/training.dat + .mozilla/ - .mozilla/Default User/ + .mozilla/firefox/ - .mozilla/extensions/ - .mozilla/sunbird/ - .mozilla/plugins/ + .mozilla/firefox/5dpdgl81.default/ - .mozilla/firefox/5dpdgl81.default/extensions/ - .mozilla/firefox/5dpdgl81.default/chrome/ - .mozilla/firefox/5dpdgl81.default/bookmarkbackups/ - .mozilla/firefox/5dpdgl81.default/forecasefox/ - .mozilla/firefox/5dpdgl81.default/Cache/ - .mozilla/firefox/5dpdgl81.default/searchplugins/ - .mozilla/firefox/5dpdgl81.default/minidumps/ - .mozilla/firefox/Crash Reports/ - .mozilla/appreg - .mozilla/mozver.dat - .mozilla/pluginreg.dat - .mozilla/firefox/5dpdgl81.default/urlclassifier3.sqlite - .mozilla/firefox/5dpdgl81.default/signons.sqlite - .mozilla/firefox/5dpdgl81.default/cookies.sqlite-journal - .mozilla/firefox/5dpdgl81.default/XUL.mfasl - .mozilla/firefox/5dpdgl81.default/blocklist.xml - .mozilla/firefox/5dpdgl81.default/bookmarks-1.html - .mozilla/firefox/5dpdgl81.default/bookmarks.bak - .mozilla/firefox/5dpdgl81.default/cert8.db - .mozilla/firefox/5dpdgl81.default/cert_override.txt - .mozilla/firefox/5dpdgl81.default/components.ini - .mozilla/firefox/5dpdgl81.default/content-prefs.sqlite - .mozilla/firefox/5dpdgl81.default/cookies.sqlite - .mozilla/firefox/5dpdgl81.default/defaults.ini - .mozilla/firefox/5dpdgl81.default/downloads.sqlite - .mozilla/firefox/5dpdgl81.default/extensions.log - .mozilla/firefox/5dpdgl81.default/formhistory.dat - .mozilla/firefox/5dpdgl81.default/history.dat - .mozilla/firefox/5dpdgl81.default/install.log - .mozilla/firefox/5dpdgl81.default/kf.txt - .mozilla/firefox/5dpdgl81.default/localstore.rdf - .mozilla/firefox/5dpdgl81.default/lock - .mozilla/firefox/5dpdgl81.default/mimeTypes.rdf - .mozilla/firefox/5dpdgl81.default/pluginreg.dat - .mozilla/firefox/5dpdgl81.default/search.rdf - .mozilla/firefox/5dpdgl81.default/search.sqlite - .mozilla/firefox/5dpdgl81.default/secmod.db - .mozilla/firefox/5dpdgl81.default/sessionstore.bak - .mozilla/firefox/5dpdgl81.default/sessionstore.js - .mozilla/firefox/5dpdgl81.default/sessionstore.tmp - .mozilla/firefox/5dpdgl81.default/signons2.txt - .mozilla/firefox/5dpdgl81.default/signons3.txt - .mozilla/firefox/5dpdgl81.default/urlclassifier2.sqlite - .mozilla/firefox/5dpdgl81.default/urlclassifierkey3.txt - .mozilla/firefox/5dpdgl81.default/OfflineCache/ - .mozilla/firefox/5dpdgl81.default/forecastfox/ - .mozilla/firefox/5dpdgl81.default/places.sqlite-journal + .mozilla/firefox/5dpdgl81.default/bookmarks.html + .mozilla/firefox/5dpdgl81.default/places.sqlite - lena - data - doctoral - install - local - stuff - temp - web - Hosts - .* -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change option
rsync give option mkdir -p
Hello how i can use rsync whith mkdir -p for nackup dirs ? when i use rsync / /backup/month/day i need automatic creation month and day dirs. mkdir -p can do that, but rsync can't -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync give option mkdir -p
Alex, to create directories for a series of backups and other backup administration tasks, I think you should consider using a backup program or shell script - there are several. I am author of one, called "snap2." It uses rsync as backup "engine." You have find it here: http://www.linuxbackups.org. It includes a GUI for easy configuration of desktop backups to local or remote media, or it can be used without the GUI to backup a server. On Thu, 25 Mar 2010 17:12:41 -0600, Alex wrote: Hello how i can use rsync whith mkdir -p for nackup dirs ? when i use rsync / /backup/month/day i need automatic creation month and day dirs. mkdir -p can do that, but rsync can't -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Test post - ignore
Posted a message earlier, seen others since but not mine. Test post to see if this makes it to the list or not. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync issue with lots of small files
(Sorry if this is a repost - didn't see my earlier post, so thinking it got lost somehow.) Wayne, et al: First posting, and I'm a n00b with rSync. I was having issues with rSync from a PC (3.0.7 either via DeltaCopy or cwRsync) to a Freenas box (3.0.6). I've since replicated and isolated the problem attempting to sync PC to PC with various flavors of rSync on both sides. The executive summary is "daemon side processes are hanging up with 100% cpu utilization when sending many small files." I can replicate this 100% of the time. What I've done: Created 2 directories with 20K files in each, 17 bytes long numbered 1.txt through 2.txt. This is the source sync directory. Here was one of many iterations of copy lines, this one happened to use DeltaCopy's version of rsync, but I can repeat it with versions of cwRsync as well: C:\DeltaCopy>rsync -rld -t --stats -v --timeout=360 "/cygdrive/e/SyncTest" sassy::Sync System named Sassy is the target machine. Here is one of his lines when server side started by hand: C:\Program Files\ICW\Bin>rsync.exe --config ..\rsyncd.conf -vv --daemon --detach - --log-file=e:\temp\rsync.txt Yes, I'm typing all those -v lines - trying to get some debug / verbosity out of the server side, which is where I believe the root of the problem is, and I'm not getting much in the way of debugging clues from the server end. I did some analysis of the network data with Wireshark on the client side. Started @ 12 seconds into the capture @ 13 seconds, started sending the file list. @ 27 seconds, file data starts flowing @ 41 seconds, I'm seeing the first TCP ZeroWindow on the sender, which frees up and stops a few times between this and the next line as the socket is full or empties on the server side. @ 122 seconds, the zero window happens and this time it no longer frees up. The client's "-v" output screen shows it is sending file 13159.txt and is frozen until the timeout or I hit ^C. Meanwhile, back at the server side ... I'm monitoring the rsync server with ProcessMonitor from SysInternals.com (now Microsoft), so I can see all the file opens, reads, writes, etc. The last file written was 10819.txt, so he wrote 913 total files. Next time I'll zero-fill my file names so the counts are accurate. Sorted in order, they go 1.txt, 10.txt, 100.txt, 1000.txt, 1.txt, 10001.txt, ..., but that is why 10819 really means 913 files. The process on the server side is spinning and taking up as much CPU as it can. I tried forcing affinity in case this was a deadlock type issue, but that didn't fix the problem. I think one version of rSync had both spinning, and the one I'm looking at right now had only one of the processes spinning, but I can't say for sure. I can replicate this failure anytime and believe the lab setup replicates the failure in the production network. --bwlimit= did not improve the situation. The lowest was --bwlimit=10 with no change other than where the hang happens. Lab has been tried at 100 Mb/s, 1 Gb/s, and I'm very confident I don't have cable / network / switch issues as I'm seeing the same failure in both locations with compeltely different hardware in each world. I can get a stack trace from ProcessExplorer, and all I can tell is the thread is "WaitForSingleObject" but not much else. Stack trace (from Process Explorer) of the spinning thread: ntkrnlpa.exe+0x8db2e ntkrnlpa.exe+0x29a82 ntkrnlpa.exe+0x331a4 ntkrnlpa.exe+0x312a9 ntkrnlpa.exe+0x28c73 ntkrnlpa.exe+0x29c72 ntkrnlpa.exe+0x138d10 ntkrnlpa.exe+0x897cc ntdll.dll!KiFastSystemCallRet kernel32.dll!WaitForSingleObject+0x12 cygwin1.dll!cygwin_internal+0x25f3d I'd get more debug out of the server side if the - options would work, but they don't seem to on the server side nor can I find a way to have the client pass more - onto the server and get more debug. Timeout or ^C will eventually kill the client, and the server's child processes do stop when I kill the client, but the logs on the server aren't chatty: Rsync.log on server: [--snip--] 2010/03/25 13:03:59 [3912] connect from uber.dascc2.local (192.168.2.148) [I sopped the server with ^C] 2010/03/25 13:23:07 [5500] [Receiver] _exit_cleanup(code=20, file=rsync.c, line=543): entered 2010/03/25 13:23:07 [5500] rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(543) [Receiver=3.0.7] 2010/03/25 13:23:07 [5500] [Receiver] _exit_cleanup(code=20, file=rsync.c, line=543): about to call exit(20) Rsyncd.log on server: [--snip--] 2010/03/25 13:05:35 [3912] recv uber.dascc2.local [192.168.2.148] Sync () SyncTest/11774.txt 17 2010/03/25 13:05:36 [3912] recv uber.dascc2.local [192.168.2.148] Sync () SyncTest/11775.txt 17 2010/03/25 13:05:36 [3912] recv uber.dascc2.local [192.168.2.148] Sync () SyncTest/11776.txt 17 [I stopped the client] 2010/03/25 13:22:39 [3912] rsync: writefd_unbuffered failed to write 4092 bytes to socket [generator]: Connection reset by peer (104) 2010/03/25 13:22:39 [3912] rsync error: error in
Re: UDT4 and rsync? support to socket.c?
Jan Wagner wrote:
> Jamie Lokier kirjoitti:
> >Jan Wagner wrote:
> >
> >>Hi, has anyone of the devels considered adding UDT4 fast reliable udp
> >>transport to socket.c, as a user-selectable alternative to using default
> >>slow TCP?
> >>
> >>It could give a 4 to 10-fold throughput improvement to rsync speed over
> >>wide area networks.
> >>
> >
> >If you're seeing that, you must be using an old OS at one/both ends,
> >or poor TCP settings, or a broken router, or have a very unreliable link.
> >
> Seeing it, with 10G links to the Internet and 1G/10G machines on the
> links, in Europe and over different continents, lots of small files (few
> gbyte per file), reasonably new Ubuntu/Debian typically at the ends,
> NICs typically Myrinet and Chelsio, single computer disk subsys' can
> sustain about 6 Gbps.
>
> Sure, on home cable and short distances TCP at least ramps up reasonably
> fast and rate recovery does not take ages and it fares pretty well vs
> UDP rates. Can't get quite the same out of TCP on 10G (except locally).
> In some cases non-ssh rsync throughput is closer to 100Mbps (was on
> Japan-Finland i think) than to 500..900Mbps/~1G nic/udp/udt4/tsunami rate.
At those speeds and distances you probably need to raise at least the
third value in /proc/sys/net/ipv4/tcp{r,w}mem quite a lot (gigabytes =
bandwidth * delay), and you'll need enough RAM for it (so will UDT4).
I'm not sure if /proc/sys/net/ipv4/tcp_mem also needs changing, and
/proc/sys/net/core/{r,w}mem_max.
There are also several choices of congestion control algorithm; the
default might not be best for your network.
> >Anyway, if you really want to use UDT4, it is probably better to use a
> >generic TCP<->UDT4 proxy and point rsync to that.
> >
> Thanks, hmm, actually there seems to be
> http://sourceforge.net/projects/udtgate/ I'll give it a go...!
Do let us know how it performs!
I'll be interested if TCP cannot reach anywhere near the same
performance level, because TCP is hardware assisted if you have
sensible 10G hardware, and I can't imagine UDT4 needing any less RAM
on an unreliable network.
-- Jamie
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: UDT4 and rsync? support to socket.c?
Jamie Lokier wrote: > I'll be interested if TCP cannot reach anywhere near the same > performance level, because TCP is hardware assisted if you have > sensible 10G hardware, and I can't imagine UDT4 needing any less RAM > on an unreliable network. Having just read the presentation poster, UDT does look quite interesting. Pity I don't have a network big enough to use it on :-/ -- Jamie -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
