Re: rsync support in authprogs - feedback requested

[Karl O. Pinc via rsync]

On Thu, 18 Feb 2021 12:22:33 -0500
Kevin Korb via rsync  wrote:

> You should both look into rrsync.  It comes with rsync and is designed
> to do exactly this.  

I'm not really interested in restricting rsync to particular
directories.  That seems to be what rrsync is for, although
it's a little hard to tell -- there's also a read-only option.

>  Unfortunately some Linux distros are maintained
> by insane people who install rrsync as if it was documentation
> (compressed and not executable) instead of a helper script which is
> what it is.  

FWIW, my uninformed guess is that Debian does not install
rrsync as an executable script because nobody has gotten around to
writing a it a man page.  A man page is required by Debian
policy for every executable.  

The good news is that rrsync is not compressed in Debian.  :)  The bad
news is I don't even see a bug report requesting rrsync, or
anything else in /usr/share/doc/rsync/scripts/, be executable.

Regards,

Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync support in authprogs - feedback requested

[Karl O. Pinc via rsync]

On Wed, 17 Feb 2021 21:52:06 -0800
Bri Hatch via rsync  wrote:

> I recently added initial rsync support to authprogs.

> I'd be very interested in feedback 

For some 15 years+ (?) I've had a /root/.ssh/authorized keys line
that starts with:

"no-pty,no-agent-forwarding,no-port-forwarding,no-user-rc,no-X11-forwarding,command="rsync
 --server --daemon ."

Occasionally I frob the ssh restrictions as new ones are
introduced.

The remote end uses rsync to backup (with --link-dest) the
entire file system.  The idea (iirc) was to restrict
the given key so that it would only run rsync.
And I think this also forces the local end to use
/etc/rsyncd.conf, where there's an additional layer
of security via a secrets file and read-only can
be set to provide some control.

The remote end always runs rsync -- the direction of 
transfer is static, per-host-pair, but can be either
in or out. (Push or pull backups.) The above authorized_keys 
line does not enforce direction, which might be useful.

I only rarely think about tweaking the authorized_keys line, 
and the rsync options used haven't changed since I got them to work.
Without really thinking about it it seems that your
authprogs development might be useful.  

My purpose with this email is to let you do all the 
thinking and tell me of all the wonderful utility
your authprogs work can provides, either now or
in the future.  ;-)  Or at least give you some
background in case you want to develop in a direction
that you think would helpful to me.  If something comes
of this I might even turn my brain on again and
modify my systems.  :)

Regards,

Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Enabling easier contributions to rsync

[Karl O. Pinc via rsync]

On Tue, 26 May 2020 00:43:41 +0200
uxio prego via rsync  wrote:

> > On 25 May 2020, at 23:55, Wayne Davison via rsync
> >  wrote:
> > 
> > I've decided to give hosting it on github a try, especially since
> > there's been a lot of nice contributions lately.  Hopefully this
> > will make it easier for both the people sending patches as well as
> > for me to snag the changes.  I'll continue to push changes to the
> > samba git as well.

> Excellent call; for GitHub's temporary, but freedom's forever!

FWIW, I find sending an email with an attachment
a lot easier than having to login to github, clone, push,
and open a pull request.  I thought people went to github
because they like the workflow when _accepting_ patches.

Regards,

Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: [PATCH] Optimized assembler version of md5_process() for x86-64

[Karl O. Pinc via rsync]

On Sat, 23 May 2020 10:21:31 -0700
Wayne Davison via rsync  wrote:

> Adding optional support for openssl's crypto library is also a good
> idea.

There is also libressl to consider, if you're considering libraries.

Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Much improved speeds of rsync via SSH - something to consider

[Karl O. Pinc via rsync]

On Mon, 2 Apr 2018 15:05:14 +0200
Thanassis Tsiodras via rsync  wrote:

> I recently concluded a bug hunt to trace why my rsync-ing to an SBC
> was much slower than the corresponding iperf3-reported speeds. To
> give a concise summary of the situation, in slow wifi links using SSH
> with ProxyCommand tremendously speeds up things:

> $ rsync -avz --progress -e 'ssh -o "ProxyCommand nc %h %p"'
> ./sample.data root@192.168.1.150:/dev/shm/

> If it isn't clear - the speed of the upload went from 543 kbytes/sec
> to 2690 kbytes/sec.
> 
> If you want to see why - and how I traced this down - you can read my
> complete report in the UNIX StackExchange forum. To avoid being
> classified as a spammer, I won't include a link - search for question
> 434825 in the search bar. The executive summary is that SSH disables
> Nagle's algorithm by default - and in slow links this can cause
> tremendous impact (as you see above).
> 
> Hope this helps people that backup over slow links (wi-fi or
> otherwise).

In the interest of collecting ssh-related issues in a single
thread I add this note:

The HPN patches to the portable OpenSSH implementation can
improve speed over high speed links with long round trip times.

  https://www.psc.edu/hpn-ssh

It also helps in those cases where CPU is a bottleneck and
encryption of transmitted data unnecessary.

The FAQ is also interesting.

The executive summary is:

  SCP and the underlying SSH2 protocol implementation in OpenSSH is
  network performance limited by statically defined internal flow
  control buffers.

Regards,

Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Help - rsync runs from command line, fails from task scheduler, hangs at msg checking charset: UTF-8

[Karl O. Pinc via rsync]

On Sat, 29 Jul 2017 19:45:22 -0700 (PDT)
leonv12 via rsync  wrote:

> I don't get why it runs from the command line but not from a
> scheduled task. Any suggestions for a fix or a work-around?

Talk to someone who knows about task scheduling and its
failure modes?

Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Help - rsync runs from command line, fails from task scheduler, hangs at msg checking charset: UTF-8

[Karl O. Pinc via rsync]

On Fri, 28 Jul 2017 14:47:02 -0700 (PDT)
leonv12 via rsync  wrote:

>(18 args)*msg
> checking charset: UTF-8*[sender] io timeout after 3000 seconds --
> exiting[sender] _exit_cleanup(code=30, file=io.c, line=195):
> enteredrsync error: timeout in data send/receive (code 30) at
> io.c(195) [sender=3.1.2][sender] _exit_cleanup(code=30, file=io.c,
> line=195): about to call exit(30)Help will be much appreciated!

Reminds me of previous struggles I've had with MS Windows and
byte order marks.  (Something about the way it requires UTF-16
but breaks unless you use UTF-8, or something.)

https://en.wikipedia.org/wiki/Byte_order_mark


Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: [Bug 12819] [PATCH] sync() on receiving side for data consistency

[Karl O. Pinc via rsync]

On Fri, 16 Jun 2017 12:34:40 +0200
Ben RUBSON via rsync  wrote:

> > On 15 Jun 2017, at 19:29, Karl O. Pinc via rsync
> >  wrote:

> > The problem is that the --server (and, especially,
> > --daemon) documentation has gone away.  Or at least
> > left the man page. (v3.1.1, Debian 8, Jessie)  Except
> > for a hint that --server exists at the bottom.  
> 
> Are you looking for `man rsyncd.conf` ?

No, that tells me what --daemon does; how to run rsync
as a server.  It does not tell me how to invoke rsync at the 
remote end manually without doing server-side things
such as the reading of rsyncd.conf.

What I want documened is how to use a customized
transport that does not allow the client side to
send arbirtrary commands to the remote end.
The sort of thing done when using
ssh with keys and the command= option within an
authorized_keys file.

As mentioned, now I use command="rsync --server --daemon ."
in my authorized_keys file.
I once figured this out from old rsync man pages, but don't
see how to glean this command sequence from a more recent
man page.

Again, I might (eventually) get around to sending
in a man page patch if somebody explains how it's done.

Regards,

Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: [Bug 12819] [PATCH] sync() on receiving side for data consistency

[Karl O. Pinc via rsync]

On Thu, 15 Jun 2017 13:23:44 +
just subscribed for rsync-qa from bugzilla via rsync
 wrote:

> https://bugzilla.samba.org/show_bug.cgi?id=12819
> 
> --- Comment #7 from Ben RUBSON  ---

> Note that my patch simply adds a sync() just after recv_files(), so
> one sync() per connection, not per write operation.

> But we could make this a rsync option, so that one can enable /
> disable it on its own.

I think the "right" rsync option to add (because rsync does
not have enough options already ;-) is a --hook-post option.
It would run something (a `sync` in your case) on the
remote end after finishing.  There are clear security issues
here.

Rather than having --hook-post and having to do something
(a server side config option that says what --hook-post
can do?) to address the security concerns it seems much
simpler to improve the rsync documentation regarding running
the rsync server side.

I'm still using command="rsync --server --daemon ." in my
~/.ssh/authorized_keys file on the remote end.  It'd be simple 
enough to add, say, a "sync" to the end of this to force a sync
when rsync finishes.  The problem is that the --server (and, especially,
--daemon) documentation has gone away.  Or at least
left the man page. (v3.1.1, Debian 8, Jessie)  Except
for a hint that --server exists at the bottom.

If the server side of rsync was better documented then
perhaps a simple inetd rsync service (or --rsync-path
or -e value, etc.) would be easy for the end-user to 
cobble together to meet needs such as this.

Can somebody please explain --server?  (And --sender, I guess.)
I might (possibly) be motivated to send in a man page patch.

Regards,

Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: How to detect hanging rsync from bash-script ?

[Karl O. Pinc via rsync]

On Mon, 10 Apr 2017 06:50:24 + (UTC)
reiner otto via rsync  wrote:

> I am using rsync on an unreliable mobile WAN connection, which causes
> rsync receiver to hang for long time, when connection is broken
> during transfere. As the option "--timeout=" here does not hit for
> me, is there a recommended "best" method to detect such scenario ?

One good way would be to use rsync's --rsh option to
invoke ssh yourself with something like -o ServerAliveInterval.

It's not clear to me how you'd get the ssh error code back
out of rsync, in other words what error rsync returns in the
case of transport failure.  Maybe 10?  Perhaps someone else
can chime in here?


Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: [Bug 12576] popt aliases allow users to bypass sudo argument restrictions

[Karl O. Pinc]

On Sun, 12 Feb 2017 02:18:35 +
samba-b...@samba.org wrote:

> https://bugzilla.samba.org/show_bug.cgi?id=12576
> 
> --- Comment #4 from Paul Donohue  ---
> That's an interesting solution, but it doesn't really work well for
> my use case.  I would like my users to be able to maintain their own
> SSH keys (this solution would require me to manage users' SSH keys in
> /root/.ssh/authorized_keys), and I don't particularly want to set
> "PermitRootLogin yes" in /etc/ssh/sshd_config.  I also already have
> scripts to manage sudo permissions, and I would have to make some
> significant changes to support centrally managing authorized_keys.

You don't have to set "PermitRootLogin yes".  You can set
"PermitRootLogin forced-commands-only".  Far more secure.

Yes, you probably would need a script, or something, that takes a ssh
key and somehow modifies /root/.ssh/authorized_keys.  

One secure way to do this is to have a script (or any
other interface) that drops a file containing the key 
(and other requisite information) into a directory
that is watched with, e.g., incron.  incron can then run a script
that parses the file, pulls out the requisite information, and
has the permissions to modify /root/.ssh/authorized_keys.
(And then cleans up by deleting the file.)  This way the process
with permissions to modify /root/.ssh/authorized_keys can
verify the data it's putting in there and validate the credentials
of the user putting the data in.

You could instead leverage your sudo management scripts to
give users permissions to run scripts as root that modify
/root/.ssh/authorized_keys.  Here it's sudo, instead of
the script run by incron, controlling access. Anybody without
authorization gets a permission error.  As above, this approach
avoids the insecurity of suid root scripts.

Either way, the script the user runs must pick up the user's
username (or home directory, or whatever) itself.  This avoids
the problem of allowing one user to set permissions for another.
The trick for the script run via sudo would be to use logname(1).

So, writing a script (or scripts) to add/remove/replace lines in
/root/.ssh/authorized_keys does not sound all that much work,
although of course there are always devilish details.  flock(1) is
probably your friend if you write in shell.  You probably
want to log with logger(1), etc.

Regards,

Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Alternatives to rsync. Was: Huge directory tree: Get files to sync via tools like sysdig

[Karl O. Pinc]

On Fri, 10 Feb 2017 11:38:34 +0100
Thomas Güttler  wrote:

> Yes, I think rsync is coming to the edge of its capabilities here. I
> guess a different strategy is needed.
> 
> I see these alternatives to rsync:
> 
>   - Incremental Snapshots at block-level device is one of them.
>   - We get the application ported to access a storage server, and not
> file server any more.
>   - 
> 
> Do you see other alternatives?

I thought the Lsyncd + rsync alternative proposed by
Axel Kittenberger workable.  Depending on your level of
paranoia you might want to be sure that you could also
run a plain rsync now and again to validate that what
you put in place is doing what you think it is.  I
say this because I've no experience at all with
Lsyncd.

Regards,

Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Huge directory tree: Get files to sync via tools like sysdig

[Karl O. Pinc]

On Fri, 10 Feb 2017 12:38:32 +1300
Henri Shustak  wrote:

> As Ben mentioned, ZFS snapshots is one possible approach. Another
> approach is to have a faster storage system. I have seen considerable
> speed improvements with rsync on similar data sets by say upgrading
> the storage sub system.

Another possibility could be to use lvm and lvmcache to throw a ssd in
front of the spinning disks.  This would only improve things if
you didn't otherwise fill up the cache with data -- you want
the cache to contain inodes.  So this might work only if your
ssd cache was larger than whatever amount of data you typically
write between rsync runs, plus enough to hold all the inodes
in your rsync-ed fs.

I've not tried this.  I'm not even certain it's a good idea.  It's
just a thought.

Regards,

Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Huge directory tree: Get files to sync via tools like sysdig

[Karl O. Pinc]

On Thu, 9 Feb 2017 14:43:57 +0100
Axel Kittenberger  wrote:

> >
> > Not only that, but inotify is not guaranteed.  (At least not on
> > 3.16.0.  Can't say regards later versions.)  So you might miss some
> > changes.
> >  
> 
> Got any info on that?
> 
> I noted that MOVE_FROM and MOVE_TO events are not guaranted to arrive
> in order, or even the file descriptor might briefly close with "no
> more events" inbetween them, but I never ever heared of anybody
> encountering an issue of an event in a watched directory on not being
> correctly reported, without getting the information of an overlfow
> with an OVERFLOW event, which results in case of Lsyncd in a full
> rescan of everything.

Not much.  inotify(7) on my system says:

   With careful programming, an application can use inotify to
   efficiently monitor and cache the state of a set of filesystem
   objects.   However, robust applications should allow for the
   fact that bugs in the monitor‐ ing logic or races of the kind
   described  below  may  leave  the  cache inconsistent  with  the
   filesystem state.  It is probably wise to to do some consistency
   checking, and rebuild the cache  when  inconsistencies are
   detected.

I think one of the pretty much unavoidable race conditions is
sub-directory creation; the sub-directory can have files added
to it before the monitoring process is able to set a watch
on it.  Of course this is an application level race.

I've had incron (which uses inotify) regularly fail to
catch all monitored fs changes on a busy system.  And
the monitored system does not involve creating sub-directories --
and I don't think I'm exceeding the system's inotify event limit
either.  But I could be wrong about either of these.

So perhaps the take-away is that inotify is "hard", or even
"impossible" to rely on as the sole method for change monitoring.
It may not be right to say it's "unreliable" as I did above.
I'm not the expert here.  But I can say that my limited
experience with it makes me want to look very closely
before relying on it.

Regards,

Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Huge directory tree: Get files to sync via tools like sysdig

[Karl O. Pinc]

On Thu, 9 Feb 2017 10:55:51 +0100
Axel Kittenberger  wrote:

> > Has someone experience with collecting the changed files
> > with a third party tool which detects which files were changed?  
> 
> I don't know of sysdig but am the developer of Lsyncd which does
> exactly that, collect file changes via inotify event mechanism and
> then calls rsync with a matching filter mask.
> 
> However, since you say, your directory tree is hugh, the main issue
> is that for every directory an inotify watch must be created, taking
> about 1KB of kernel memory per watch.

Not only that, but inotify is not guaranteed.  (At least not on
3.16.0.  Can't say regards later versions.)  So you might miss some
changes.


Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: man page

[Karl O. Pinc]

Let me start with a reference to Nazis, just so we can reach
the Godwin-point and be done with it.

If somebody thinks they can improve the documentation let them
submit a patch.  This provides something concrete for people
to comment on and the maintainers can make a decision
as to whether they want to make the proposed change.

Until I see something in writing I've no opinion one way or
another.  As a rule, getting rid of extra words makes
for clearer sentences.   The sticking point is in what is
considered "extra".

On Thu, 28 Jul 2016 14:15:51 -0400
Tony Reed  wrote:

> Nah, this is the first time I've posted here in years. Gennaly things
> just tick over with the application moving along in a nice
> straightforward manner, improvment upon improvement. Rsync is one of
> the mose important and usefu apps available for Linux and OS X
> (powered by APPLE); followed by ffmpeg and vlc (and Apache, I
> guess, but I don't do  much of that anymore). The docs are very
> well-written and I'm sorry that you and Herr Obermeister Schtallman
> have had your sad little feelings hurt by the mere mention
> of MICROSOFT! I used to maintain a couple of packages for
> CRUX and there was a lot of that misplaced zealotry in those parts. I
> don't beweeve in the widda baby jesus neither. Neener. I used to be
> Juanita's old boyfriend, and all that.
> 
> On Wed, Jul 27, 2016 at 3:45 PM, Marcus Fonzarelli <
> marcus.fonzare...@yandex.com> wrote:  
> 
> > So I guess you're the bully of the community here. Strong opinions
> > without understanding, talk shit, and nothing useful to contribute.
> >
> >
> > 27.07.2016, 21:54, "Tony Reed" :
> >
> > I would rather the maintainers would spend their time improving the
> > application, and not worrying aobut the OP's hurt little
> > open-sourced religious feelings. I use rsync to transfer pretty
> > massive (wait for it) ADOBE PHOTOSHOP files around my LAN. Neener
> > neener fucken neener.
> >
> >  
> 
> 




Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: man page

[Karl O. Pinc]

On Wed, 27 Jul 2016 14:56:07 +0300
Marcus Fonzarelli  wrote:
> Therefore I'd like to suggest that the man page be changed to "which
> consists of large documents and mail folders", or at least mention
> another software such as LibreOffice.

Submitting a patch would be a good start.  Gotta make it easy on the
maintainers.

Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Yet another filter question

[Karl O. Pinc]

On Wed, 4 May 2016 21:26:26 -0500
"Karl O. Pinc"  wrote:

> On Wed, 4 May 2016 21:09:44 -0400
> Kevin Korb  wrote:
> 
> > That wording from the man page makes almost no sense without the
> > examples directly after it (and I have read it many times and know
> > what it is saying).  

> Patch appended. (Which does not perfectly match my comments
> above.)

2nd version of the patch appended. (I also line-wrapped and filled.
This makes it harder to see the exact details of the difference,
but there's enough changes now that it does not really matter.
Use wdiff for an exact comparison.)

The wording now contains some redundancy.  Or maybe
the redundancy that was there is now more apparent.  I don't think
this is necessarily bad when explaining something that people
find confusing.

Regards,

Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein
diff --git a/rsync.yo b/rsync.yo
index 0ec5e55..166b58e 100644
--- a/rsync.yo
+++ b/rsync.yo
@@ -2829,17 +2829,19 @@ itemization(
   version 2.6.7.
 )
 
-Note that, when using the bf(--recursive) (bf(-r)) option (which is implied by
-bf(-a)), every subcomponent of every path is visited from the top down, so
-include/exclude patterns get applied recursively to each subcomponent's
-full name (e.g. to include "/foo/bar/baz" the subcomponents "/foo" and
-"/foo/bar" must not be excluded).
-The exclude patterns actually short-circuit the directory traversal stage
-when rsync finds the files to send.  If a pattern excludes a particular
-parent directory, it can render a deeper include pattern ineffectual
-because rsync did not descend through that excluded section of the
-hierarchy.  This is particularly important when using a trailing '*' rule.
-For instance, this won't work:
+Note that, when using the bf(--recursive) (bf(-r)) option (which is
+implied by bf(-a)), every component of every pathname is visited left
+to right; directories are examined before their content.  In this way
+include/exclude patterns are applied recursively to the full pathname
+of each node in the filesystem's tree.  The exclude patterns
+short-circuit the directory traversal stage as rsync finds the files
+to send.  (E.g. to include "/foo/bar/baz", "/foo" and "/foo/bar" must
+not be excluded.  Excluding either prevents examination of their
+content.)  If a pattern excludes a particular parent directory this
+will render a deeper include pattern ineffectual because rsync does
+not descend through the excluded section of the hierarchy.  This is
+particularly important when using a trailing '*' rule.  For instance,
+this won't work:
 
 quote(
 tt(+ /some/path/this-file-will-not-be-found)nl()

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Yet another filter question

[Karl O. Pinc]

On Wed, 4 May 2016 21:09:44 -0400
Kevin Korb  wrote:

> That wording from the man page makes almost no sense without the
> examples directly after it (and I have read it many times and know
> what it is saying).

Makes sense to me.  The only thing I'd change is to use "in a depth
first fasion" instead of "from the top down", "depth first" (v.s.
"breadth first") being the standard idiom when talking about
this tree traversal strategy.  It also matches up with some
of the vocabulary (e.g. "deeper") used later in the paragraph.

Er, ok.  It could also say "component" instead of "subcomponent",
in the first instance.  And "to the full name of each node in the
filesystem's tree" instead of "to each subcomponents full name",
in the second instance.  And eliminate "the subcomponents" in the
third instance.

Patch attached. (Which does not perfectly match my comments
above.)

> On 05/04/2016 09:03 PM, Wayne Davison wrote:

> > From the man page near the start of the "INCLUDE/EXCLUDE PATTERN
> > RULES" section:
> > 
> > /Note that, when using the --recursive (-r) option (which is
> > implied by -a), every subcomponent of every path is visited from
> > the top down, so include/exclude patterns get applied recursively
> > to each subcomponent’s full name (e.g. to include "/foo/bar/baz" the
> > subcomponents "/foo" and "/foo/bar" must not be excluded). The
> > exclude patterns actually short-circuit the directory traversal
> > stage when rsync finds the files to send./
> > 

Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein
diff --git a/rsync.yo b/rsync.yo
index 0ec5e55..3f742ab 100644
--- a/rsync.yo
+++ b/rsync.yo
@@ -2830,14 +2830,16 @@ itemization(
 )
 
 Note that, when using the bf(--recursive) (bf(-r)) option (which is implied by
-bf(-a)), every subcomponent of every path is visited from the top down, so
-include/exclude patterns get applied recursively to each subcomponent's
-full name (e.g. to include "/foo/bar/baz" the subcomponents "/foo" and
+bf(-a)), every component of every pathname is visited
+in a depth-first fashion.  In this way
+include/exclude patterns are applied recursively to the full name of each
+node in the filesystem's tree.
+(E.g. to include "/foo/bar/baz", "/foo" and
 "/foo/bar" must not be excluded).
-The exclude patterns actually short-circuit the directory traversal stage
+The exclude patterns short-circuit the directory traversal stage
 when rsync finds the files to send.  If a pattern excludes a particular
-parent directory, it can render a deeper include pattern ineffectual
-because rsync did not descend through that excluded section of the
+parent directory this can render a deeper include pattern ineffectual
+because rsync does not descend through the excluded section of the
 hierarchy.  This is particularly important when using a trailing '*' rule.
 For instance, this won't work:
 
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Yet another filter question

[Karl O. Pinc]

On Wed, 4 May 2016 21:09:44 -0400
Kevin Korb  wrote:

> That wording from the man page makes almost no sense without the
> examples directly after it (and I have read it many times and know
> what it is saying).

Makes sense to me.  The only thing I'd change is to use "in a depth
first fasion" instead of "from the top down", "depth first" (v.s.
"breadth first") being the standard idiom when talking about
this tree traversal strategy.  It also matches up with some
of the vocabulary (e.g. "deeper") used later in the paragraph.

Er, ok.  It could also say "component" instead of "subcomponent",
in the first instance.  And "to the full name of each node in the
filesystem's tree" instead of "to each subcomponents full name",
in the second instance.  And eliminate "the subcomponents" in the
third instance.

Patch appended. (Which does not perfectly match my comments
above.)

> On 05/04/2016 09:03 PM, Wayne Davison wrote:

> > From the man page near the start of the "INCLUDE/EXCLUDE PATTERN
> > RULES" section:
> > 
> > /Note that, when using the --recursive (-r) option (which is
> > implied by -a), every subcomponent of every path is visited from
> > the top down, so include/exclude patterns get applied recursively
> > to each subcomponent’s full name (e.g. to include "/foo/bar/baz" the
> > subcomponents "/foo" and "/foo/bar" must not be excluded). The
> > exclude patterns actually short-circuit the directory traversal
> > stage when rsync finds the files to send./



Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein


diff --git a/rsync.yo b/rsync.yo
index 0ec5e55..3f742ab 100644
--- a/rsync.yo
+++ b/rsync.yo
@@ -2830,14 +2830,16 @@ itemization(
 )
 
 Note that, when using the bf(--recursive) (bf(-r)) option (which is implied by
-bf(-a)), every subcomponent of every path is visited from the top down, so
-include/exclude patterns get applied recursively to each subcomponent's
-full name (e.g. to include "/foo/bar/baz" the subcomponents "/foo" and
+bf(-a)), every component of every pathname is visited
+in a depth-first fashion.  In this way
+include/exclude patterns are applied recursively to the full name of each
+node in the filesystem's tree.
+(E.g. to include "/foo/bar/baz", "/foo" and
 "/foo/bar" must not be excluded).
-The exclude patterns actually short-circuit the directory traversal stage
+The exclude patterns short-circuit the directory traversal stage
 when rsync finds the files to send.  If a pattern excludes a particular
-parent directory, it can render a deeper include pattern ineffectual
-because rsync did not descend through that excluded section of the
+parent directory this can render a deeper include pattern ineffectual
+because rsync does not descend through the excluded section of the
 hierarchy.  This is particularly important when using a trailing '*' rule.
 For instance, this won't work:
 

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: [PATCH] Consider nanoseconds when quick-checking for unchanged files

[Karl O. Pinc]

On Wed, 20 Jan 2016 23:04:20 -0800
Wayne Davison  wrote:
> 
> The problem is that if you transfer from a filesystem that has
> nanoseconds to one that does not support it, rsync would consider
> most of the files to be constantly different, since the nanosecond
> values would only match if the source file happened to have 0
> nanoseconds. So, the logic has to be improved to somehow detect such
> a case and treat the truncated values as equal. One possible
> improvement would be to skip the nanosecond check if the destination
> file has a nanosecond value of 0.  That could possibly be improved if
> we figure out if a particular device ID supports nanoseconds
> somehow.

Seems to me that nanoseconds are the sort of thing that could cause
sysadms crazy headaches.   My thought is to have a declaration
in the rsync configuration file (that can be overridden on
the command line).Something like "--nanosecond".
It'd have the following values:

ignore   : Ignore nanoseconds.  (default)

update   : Ignore nanoseconds, but update destination timestamps
   when nanoseconds differ.

heuristic: Check nanoseconds with Wayne's spiffy heuristic.

check: Check nanoseconds.

When there is a conflict between the conf files of the 2 endpoints the
topmost of the above options has priority.  (When no configuration
is specified on at least one endpoint there is no conflict.)

To provide control over conflict management you could have another
option, say, --nanosecond-force, to force your endpoint's choice.  If
both ends force then the later of ignore, update, heuristic, check has
priority.

I don't know how this would work with the existing rsync protocol.
Perhaps it'd be easier to have only the destination end's config
matter, although this does not provide a lot of flexibility from the
command line.  The motivation is to be able to keep things simple,
or as simple as they can be.  Already my ideas seem overly complicated.
Perhaps someone can improve them.

It makes some sense to be able to configure nanosecond related
behavior on a per-directory (i.e., mountpoint) basis, as a substitute
for knowing about every possible filesystem type and being able to
detect file system type.  But this introduces yet more complication.

The default is backward compatible.  Distros could set their
own default in the rsync.conf file they install.

---

Maybe the thing to do is to give up on runtime complication and
just do testing on the destination filesystem when rsync initializes.
This would be on by default but could be turned off by command line.

Since the problem is with destination filesystems that don't support
nanoseconds, and destination filesystems are by definition written to,
then test for nanosecond support once when rsync starts.  Write a file
with non-zero nanoseconds and read it back and see if the nanoseconds
are zero.  Then delete the test file.  Easy if the destination is
empty, harder if there's an existing directory hierarchy.  (But rsync
can already tell if it's crossing a filesystem boundary so)

Trouble is that unconditionally writing a file would affect directory
timestamps.  If you wait until you know you're writing to a directory
then the approach here is starting to sound suspiciously like Wayne's
heuristic

Hope these thoughts are helpful to somebody.

Regards,

Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: [Bug 11378] Please add a '--line-buffered' option to rsync to make logging/output more friendly with pipes/syslog/CI systems/etc.

[Karl O. Pinc]

On Sat, 04 Jul 2015 17:56:25 +
samba-b...@samba.org wrote:

> --- Comment #2 from Nathan Neulinger  ---
> Perhaps the naming is not correct on my suggested option (and I'll
> admit, I completely missed the outbuf option) - unfortunately, outbuf
> doesn't actually solve the problem. 
> 
> The goal is to get incremental progress output while running rsync
> through a build system or similar.

What would happen if you piped the rsync output through
tr and changed \r to \n?



Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Downloading a great number of files from different rsync servers for good loadbalancing and high efficiency.

[Karl O. Pinc]

On Sat, 4 Apr 2015 15:21:21 +0800
Hongyi Zhao  wrote:

> I'm using Debian, I want to make a local repository which can let me
> install packages more conveniently.

Your solution will not work for mirroring debian since it does
not do a 2-stage mirroring process.  This is
described in: https://www.debian.org/mirror/ftpmirrors

Further, your solution is a bad idea for many reasons.  If
you want to know more about this I suggest asking on
the Debian mailing lists or on the #debian irc
channel on irc.freenode.net.

Better would be to use the Debian recommended ftpsync
script.  This can be found at:
https://ftp-master.debian.org/ftpsync.tar.gz
The instructions are at:
https://www.debian.org/mirror/ftpmirrors

The Debian people know how to best mirror Debian.
Best to follow their guidance.  Depending on
your purposes you might not even want a mirror,
you might be better served with a cache.
Again, ask the Debian people for guidance.

Regards,


Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: [Bug 11013] New: [patch] Mention that privileges are dropped, when "use chroot" is enabled in rsyncd.conf manpage

[Karl O. Pinc]

On 12/16/2014 08:45:15 PM, Kevin Korb wrote:
> Only root can chown.  If rsync isn't running as root then it ignores
> the --owner part of --archive.  This also makes --numeric-ids inert.
> 
> Simply put, if you aren't running as root then you can only create
> files owned by your UID.  Rsync knows this.  Rsync assumes that if 
> you
> aren't root you didn't intend the --owner (or --group) part of
> --archive and it ignores those features.

Rsync has enough options that it seems it can be smarter than
it's users.  It could be worth adding a --stupid
option to tell rsync to be stupid and complain instead of doing
smart things.  This would help people diagnose just what
rsync is doing.

I'm sure this is a lot more work than it sounds.  There might
be better ways of reporting what's happening
than by complaining. A displayed table of which options are on/off
or have what value comes to mind.  (Too bad that
--stupid would not be the right option name for this.  :-)

Just a thought.



Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Aw: Re: encrypted rsyncd - why was it never implemented?

[Karl O. Pinc]

On 12/03/2014 01:37:58 PM, Kevin Korb wrote:
> As far as a backup provider goes I wouldn't expect them to use rsync
> over SSL unless that were built into rsync in the future (and has 
> been
> around long enough that most users would have it).
> 
> I would expect them to either use rsync over ssh secured by rrsync or
> rsyncd over ssh with them managing the rsyncd.conf file.  Either way
> the server side command would be forced and no other ssh 
> functionality
> would be allowed.



> I am thinking of something like this with in sshd_config with
> whichever ForceCommand they would pick:
> 
> Match Group backupusers
>   X11Forwarding no
>   AllowTcpForwarding no
>   ForceCommand /usr/bin/rsync --server --daemon .
>   ForceCommand /usr/bin/rrsync-wrapper
> 
> Note that a wrapper or modification would be needed for rrsync since
> sshd_config doesn't support %u or %h in ForceCommand :(

I am using command="rsync --server --daemon ." 
in ~/ssh/authorized_keys.  Correct me if I'm wrong,
but I believe this eliminates the need for %u or %h
and ForceCommand.

It does mean that key based authentication is required,
but this does not seem burdensome for a backup oriented
solution.


Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Using --fuzzy

[Karl .O Pinc]

The backups can be on separate partitions.  What must be on one partition is 
the file and it's hard link.

On November 16, 2014 6:58:26 PM CST, Joe  wrote:
>Great idea which I will keep in mind for other cases!
>
>In this case, however, the backups are on separate partitions on
>external USB drives (I have a notebook), so hard links won't work.
>
>Joe
>
>On 11/16/2014 07:38 PM, Karl O. Pinc wrote:
>> On 11/16/2014 03:53:12 PM, Joe wrote:
>>> I have a lot of files (and directories) (up to a few hundred at a
>>> time)
>>> that I get from various sources. Some time after I get them (after
>>> they
>>> are already backed up), I often have to move them around and 
>>> normalize
>>> their names.
>>>
>>> When I do this, rsync sees them as unrelated to the copies of these
>>> files which are already on the backup destination. 
>> I don't know if it suits your use case but
>> you could consider using hardlinks.
>>
>> If, instead of moving the files, you hardlinked them
>> then rsync with -H would see the files as being the same.
>>
>> (Hardlinking can only be done within a filesystem.)
>>
>> Then you'd have to delete the original filenames and
>> rsync again.
>>
>> This is only practicable if it's easy to delete
>> the old filenames, say, if all the new files
>> arrive in a single directory that can later
>> be deleted.
>>
>>
>>
>> Karl 
>> Free Software:  "You don't pay back, you pay forward."
>>  -- Robert A. Heinlein

Karl 
Free Software: "You don't pay back, you pay forward."
-- Robert A. Heinlein
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Using --fuzzy

[Karl O. Pinc]

On 11/16/2014 03:53:12 PM, Joe wrote:
> I have a lot of files (and directories) (up to a few hundred at a
> time)
> that I get from various sources. Some time after I get them (after
> they
> are already backed up), I often have to move them around and 
> normalize
> their names.
> 
> When I do this, rsync sees them as unrelated to the copies of these
> files which are already on the backup destination. 

I don't know if it suits your use case but
you could consider using hardlinks.

If, instead of moving the files, you hardlinked them
then rsync with -H would see the files as being the same.

(Hardlinking can only be done within a filesystem.)

Then you'd have to delete the original filenames and
rsync again.

This is only practicable if it's easy to delete
the old filenames, say, if all the new files
arrive in a single directory that can later
be deleted.



Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Concern: rsync failing to find some attributes in a file transfer?

[Karl O. Pinc]

On 07/26/2014 03:34:23 PM, Kevin Korb wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
> 
> I wasn't objecting to the use of multiple file systems.  I have a
> bunch of them too.  I was objecting to the use of partitions to
> achieve multiple files systems.  Logical volume management has been
> available for a long time and now we also have access to file systems
> that include such features.

I too like logical volume management but that does not mean
it's right for everyone.  E.g. chasing badspot block numbers
back and forth between the underlying media and the file
system makes me cranky.



Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: [Bug 10637] rsync --link-dest should break hard links when encountering "Too many links"

[Karl O. Pinc]

On 06/06/2014 12:08:46 AM, L. A. Walsh wrote:
> samba-b...@samba.org wrote:
> 
> > https://bugzilla.samba.org/show_bug.cgi?id=10637
> >
> > --- Comment #1 from Karl O. Pinc  2014-05-28 19:05:04
> UTC ---
> > Yum is also rsync happy.  That's where our --link-dest backups
> always break due
> > to too many hard links.
> 
> --
> 
> What would be "too many"? -- a few million?  I have files in a test
> setup that have over 7000 files hard linked (only an 8 byte file, but
> since
> minimum block size is 4K, that would be 28MB w/o the hard link v. 4k
> with.

Too many would be when the link(2) call returns EMLINK
because your filesystem has hit it's hard link limit.

The ext2/3 limit is 32,000, the ext4 limit is twice that.
(EXT3_LINK_MAX, EXT4_LINK_MAX)

This is low enough that I can't fill a 4TB drive with
--link-dest backups without EMLINK error.

> 
> It would NOT be good for rsync to start breaking
> links w/standard options.  Maybe a new option
> to allow link breaking?

Rsync has no choice.  If you hit the filesystem link
limit then you can't make more hardlinks.  The only
question is what link to break.  The proposal is
to break the --link-dest links.  Currently rsync
instead almost always breaks the -H links.  Because
there's only 1 --link-dest link but almost always
many -H links you're simply more likely to run
out when linking -H links.  If you're preserving
hard links you probably really want an accurate
copy of your hard links and would much rather,
since you're forced to sacrifice something,
forgo the --link-dest for that particular file.

It would be nice to have an option, after this
proposal is implemented, to control whether rsync
reports an error when breaking --link-dest links.
It might even be nice to have the default be to _not_
report errors when breaking --link-dest links.

However, once started down the path of controlling
which errors rsync reports and which it does not,
it's probably better to implement an option that
controls/filters error reporting in general instead
of having separate options for each kind of error
that rsync might report.

Regards,

Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Alternative rsync client for Windows

[Karl O. Pinc]

So long as your compiling a list of things to
overcome to restore a MS Windows box you need
to add the following:

You'd have to have an (exact?)
copy of the original hardware onto which
you wish to restore.  I try to stay away
from Windows so am no expert, but I believe
that the OEMs are the ones that put the
hardware drivers onto the box.  Trying
to restore one MS Windows system onto
different hardware results in a system that
won't run on the new hardware.

On 04/11/2014 07:24:15 PM, L. A. Walsh wrote:
> Kevin Korb wrote:
> > -BEGIN PGP SIGNED MESSAGE- Hash: SHA1
> > 
> > I come from the Linux world.  If one of my computers were to simply
> > evaporate into nothingness or have complete storage failure then
> once
> > the hardware problem is dealt with I would network boot
> SystemRescueCD
> > then restore my backups that I made with rsync.
> > 
> > I understand that things are more complicated in Windows but if say
> my
> > laptop (it is the only computer I have that both boots and stores
> > Windows) were similarly destroyed or blanked I would still network
> > boot SystemRescueCD and restore my backups that I made with
> ntfsclone.
> > 
> > My hesitation with backing up a Windows system with rsync is that
> > I have absolutely no idea to go from "I have a blank computer and
> > a copy of all my files" to "I have a working computer with all my
> > stuff".  I might be asking for something as simple as "Install
> > Windows, install Acrosync, restore everything including the Windows
> > configuration from backups" or maybe some kind of rescue disc or
> maybe
> > some kind of custom WinPE disc.  I don't know.  I know just enough
> > about Windows to figure out how to use what I know from Linux to
> make
> > things sorta work.
> ---
> I wouldn't suggest trying to restore windows w/rsync.  It might be
> possible, but first issue is that whatever media you rsync things to,
> needs to support full NT security and be able to create arbitrary
> users/acl's to fully replicate the source.
> 
> Second issue is that MS deliberately uses things the location of
> something on the disk as a "security option".  I don't know what
> software uses it, but I remember discussion about "media licenses"
> (❝𝑐𝑜𝑛𝑡𝑒𝑛𝑡❞) using the feature to prohibit any "copy" of them from
> working:
> only the original in its original 'licensed' location would work.  
> The
> whole way NTFS is designs it's locking of files is very unlike how
> it's
> done on linux/unix.  When it locks a file, it isn't, like in linux, 
> at
> the inode level+offset; it's at a physical location on the disk that
> gets locked... it's really primitive, (and is why one needs to often
> reboot a system to replace binaries -- because the bytes on the disk
> ARE
> the file and they are locked -- vs. on linux, usually you have an
> inode
> that points to sectors where the file is, and by changing where the
> inode points, you can change the content.
> 
> That said, my primary concern would be the first issue (for me, not
> using licensed content on windows, I've not run into the problem, so
> that's mostly from memory about how it was implemented.  VERY often,
> when doing copies with rsync or cp -a from one sys to another, I'll
> find
> permissions or such won't get transferred "quite" the same way.
> 
> I have used rsync from/to the same disk to restore & repair a broken
> windows install -- the part that has problems is storing the extended
> stuff and ACL's on a foreign media.
> 
> (Also have to make sure on restore that rsync has all needed
> rights&privileges.  Cygwin takes care of alot of that -- removing
> a file or such that in the windows command line, you'd have a pretty
> hard time doing... or setting permissions on all the files in the
> windows/system32 dir despite not "owning them" - under the posix
> model,
> ownership doesn't matter for 'root'.. so cygwin tries to emulate that
> as
> much as possible -- probably why I've seen cygwin listed as a
> "security
> hacking tool"...;-)  (really!, letting a user control their own
> system,
> how absurd!)..
> 
> 
> 
> -- 
> Please use reply-all for most replies to avoid omitting the mailing
> list.
> To unsubscribe or change options: https://lists.samba.org/mailman/
> listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-
> questions.html




Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: silent data corruption with rsync

[Karl O. Pinc]

On 03/11/2014 11:02:28 AM, Sig Pam wrote:
> Hi everbody!
> 
> I'm currently working in a project which has to copy huge amounts of
> data from one storage to another. For a reason I cannot validate any
> longer, there is a roumor that "rsync may silently corrupt data".
> Personally, I don't believe that.
> 
> "They" explain it this way: "rsync does an in-stream data
> deduplication. It creates a checksum for each data block to transfer,
> and if a block with the same checksum has already been transferred
> sooner, this old block will be re-used to save bandwidth. But, for 
> any
> reason, two diffent blocks can produce the same checksum even if the
> source data is not the same, effectively corrupting the data stream".

Well, yeah.  It works that way if you're transferring data over
the network.

The question is: "how often will this problem exhibit itself?"
The answer is: "Usually, never within the lifetime of the Universe."

You're a lot more likely to have data corruption due to a 
cosmic ray hitting your box.

There are some cases where the answer is: "Maybe more often."  The only 
time I can think of that you'd want to worry about
is if you're researching MD5
checksum collisions and have a lot of data on disk that has
collisions in the checksumming.  In other words,
if you're actively trying to cause problems it might be an issue.

(The older rsyncs used MD4.)

If you're actually _copying_ data rather than backing it up then
avoid the issue by not using rsync.  Otherwise the tradeoff
is worth the risk.

Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Need hint for my question regarding the working of rsync.

[Karl O. Pinc]

necessarily 11/13/2013 01:04:29 PM, Kevin Korb wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
> 
> Is there a hard links limit?  I have been in the 70-80 million range
> on ext4 without a problem (other than performance which is why I
> switched to ZFS for that use case).

It's a per-file limit

I'm mostly using ext3.  Limit of 32,000.

According to fs/ext4/ext4.h ext4 has a limit of 65,000,
although googling about implies that there's a 16
bit counter somewhere so it could be 65,536.
And if you're getting 70-80 million then
that's obviously wrong.

Trying:

  for ((i=1;i=$i+1;i<0)); do ln foo foo$i || break; done

I get 65,000.  But I know that's the directory
size limit so this is not necessarily a good test
since all the links are in a single directory.

I tend to hit the hardlink limit on my ext3 fs-s
around 300 --link-dest-ed backups.


Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Need hint for my question regarding the working of rsync.

[Karl O. Pinc]

On 11/13/2013 12:03:21 PM, Kevin Korb wrote:
> OK, in the case of using v3 with --link-dest and not --checksum most
> of the initial activity on the sender would be doing calls to stat()
> to index what is there.
> 
> The receiving side would be doing 2x the stat() calls (you have 2
> --link-dest dirs for it to check) and link() calls every time it 
> finds
> a matching file.

Am I correct in my impression that the sender and receiver
are doing the above serially, not concurrently?

> stat() is an expensive call in terms of time spent (especially when
> multiplied by millions of files) but it doesn't really translate into
> much disk IO since it is such a small amount of actual data.  The
> link() call is pretty much the same except it is a write op instead 
> of
> a read op.  So, you wouldn't show much MB/sec usage of your disks
> until rsync found a new or different file but there would be many
> small operations.

My thought is to save wall time by increasing concurrency.

No doubt there are tradeoffs involved.  If forced to
choose between features what I really want is entirely 
different; for -H to have "priority" over
--link-dest so that when the fs surpasses its hardlink
limit the end result is that the -H links exist
and the --link-dest links do not.  Future --link-dest
operations would then work and, most importantly,
the result of the running rsync operation would
be a good copy of the source.  This would allow
many --link-dest-ed backups of a fs used by 
hardlink-happy applications.  (Like yum.)


Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Need hint for my question regarding the working of rsync.

[Karl O. Pinc]

On 11/12/2013 04:13:01 PM, Karl O. Pinc wrote:
> On 11/12/2013 03:50:20 PM, Wayne Davison wrote
> 
> > Yes, the receiver sends all the checksums that it generates at once
> 
> > For really big files it would be interesting to amend this rule to 
> > one
> > where the sending side waits only long enough for a certain number
> of
> > checksums to arrive before it begins its work (and perhaps pauses 
> if
> > it
> > gets too far ahead of the arriving checksums).
> 
> Based on the behavior I see when using rsync, without
> really knowing what's going on, it seems that the
> sending side first does a lot of disk access,
> then the receiving side does, and then
> the sync begins over the network.  It would
> save a lot of wall-clock time if the sending
> and receiving side could both be hitting
> the disk at once.  At least in my use case,
> with whatever version of rsync I happen to be
> using.

Note that I don't see this per-file, but per-filesystem/
rsync command.

Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Need hint for my question regarding the working of rsync.

[Karl O. Pinc]

On 11/12/2013 03:50:20 PM, Wayne Davison wrote

> Yes, the receiver sends all the checksums that it generates at once

> For really big files it would be interesting to amend this rule to 
> one
> where the sending side waits only long enough for a certain number of
> checksums to arrive before it begins its work (and perhaps pauses if
> it
> gets too far ahead of the arriving checksums).

Based on the behavior I see when using rsync, without
really knowing what's going on, it seems that the
sending side first does a lot of disk access,
then the receiving side does, and then
the sync begins over the network.  It would
save a lot of wall-clock time if the sending
and receiving side could both be hitting
the disk at once.  At least in my use case,
with whatever version of rsync I happen to be
using.


Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: contribute to rsync

[Karl O. Pinc]

On 05/19/2013 10:19:10 PM, garvit sharma wrote:
> Hello All,
> 
>Myself Garvit Sharma working in hyderabad central
> university(HCU) at DCIS. I have started using rsync to sync the data
> and i
> found it very interesting. After using rsync a lot i realized to
> contribute
> to rsync by adding some extra features into it. Every time for
> synchronization we need to do it manually by executing command on the
> command line but instead of doing it manually imagine if it does
> automatically sync whenever it find changes in the single file.
> To do this we need to start a daemon on both the source and
> destination and
> these daemon will check the time stamp of last modification of the
> directory we want and if it finds the new time stamp then it will
> sync.

You might also consider invoking rsync from incron.


Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: High Speed WAN Rsync now possible!!!

[Karl O. Pinc]

On 02/22/2013 09:54:03 AM, Erich Weiler wrote:
> Folks-
> 
> Just wanted to plug a totally awesome software package from a group I 
> know: UDR (UDT Enabled Rsync).
> 
> For those not familiar with UDT, it is a low level network protocol 
> based on UDP that allows for high speed transfers over high latency
> WAN 
> networks:
> 
> http://udt.sourceforge.net/

Too bad nobody's dropped UDT into socat.  Then it wouldn't
have to be ported into specific applications like rsync.

http://www.dest-unreach.org/socat/




Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: A --exclude-checksum option?

[Karl O. Pinc]

On 02/17/2013 01:59:02 PM, Wayne Davison wrote:
> On Tue, Feb 12, 2013 at 12:42 PM, Karl O. Pinc  wrote:
> 
> > It occurs to me that a handy solution might be to have an rsync
> option,
> > similar to the --exclude option, which would allow checksumming to
> happen
> > throughout most of the  ackup process but would do "regular"
> size/timestamp
> > based backups on certain directories.
> >
> 
> One thing you could do is to apply the checksum-reading.diff patch 
> and
> then
> generate a checksum file (using the included perl script) for the /
> tmp
> dir
> right before doing your rsync run.  That would cache the file's
> checksums
> and not re-read the files unless the perl script notices that the
> ctime has
> changed (which will cause it to update the cache file).  A patched
> rsync
> will read the checksum from the cache file instead of re-computing 
> it,
> which should solve your issue.

Thanks much, but what I've done is persuaded the sysadm
to mount the fs relatime -- which makes the problem go away. :-)

Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: A --exclude-checksum option?

[Karl O. Pinc]

On 02/12/2013 02:48:35 PM, Kevin Korb wrote:
> My first thought is why are you backing up /tmp at all?

Because I put stuff in /tmp I might want, and whatever
I put there goes away by itself.  It stays very handy for a while,
then it's on backup and less handy, then it's gone

> My second thought is why are you using atime for anything?  It can be
> touched by almost anything and running a filesystem with atime 
> enabled
> is a huge performance detriment as it adds a directory write 
> operation
> to every file read operation.

Not my box, not my choice.  (I tend to like relatime)

> 
> My final thought is maybe you want a file verification tool (I like
> cfv) instead of rsync --checksum.  Rsync's --checksum is kinda
> mindless in terms of performance.  It checksums everything.  This is
> rather pointless as a file that is a different size will obviously
> have a different checksum.  Rsync even checksums files that only 
> exist
> on one side of the transfer.

Wouldn't I have to do something with cfv as well so that checksums
only happens on files of different sizes?  Sounds like complication
when the backup is on a box reachable only via ssh.

I'm lazy, it was easy to incorporate verification into the
rsync backup process.  And checksumming everything means
that everything is verified -- may as well do it in rsync as
anywhere else.

It's a backup.  If it's corrupted then --checksum will fix it
and it won't be corrupted.  Regardless of whether the backup
side fs is broken.  (Presumably the backup side hardware/
fs will be fixed quickly.)  And I don't care that --checksum means
that the rsync takes longer once a week.

Sounds like you're leaning toward "it's a niche feature
and let's not clutter up rsync (further)".

> 
> On 02/12/13 15:42, Karl O. Pinc wrote:
> > Hi,
> >
> > I use rsync with hardlinks for backup, once a week doing checksums
> > to ensure there's no filesystem corruption in the backed-up data.
> >
> >
> > I also use tmpwatch, or something similar, to clean up /tmp, it
> > removes files that have not been accessed recently. (atime older
> > than some configured limit). I backup /tmp because I throw stuff in
> > tmp that I might possibly need again but don't want to bother
> > having to remember to delete -- and if I'm expecting to have useful
> > data somewhere I want it backed up.
> >
> > However, rsync's checksumming (naturally) updates the atimes of the
> > files in /tmp, and so tmpwatch never deletes them.
> >
> > It occurs to me that a handy solution might be to have an rsync
> > option, similar to the --exclude option, which would allow
> > checksumming to happen throughout most of the backup process but
> > would do "regular" size/timestamp based backups on certain
> > directories.
> >
> > What do people think of such an option? Is there a better design.
> > (E.g. an option that, er, preserves atime when checksumming?) Is
> > rsync just too overloaded with options already and it would be
> > better instead to run two instances of rsync?  Is there a
> > bug/feature in process already that would address the use-case
> > above?
> >
> > I'd like to have a sensible design before even thinking about
> > patching.
> >
> > Thanks for the feedback.
> >
> > Regards,
> >
> > Karl  Free Software:  "You don't pay back, you pay
> > forward." -- Robert A. Heinlein
> >
> 
> --
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-
> *~'`^`'~*-,._.,-*~
>   Kevin Korb  Phone:(407) 252-6853
>   Systems Administrator   Internet:
>   FutureQuest, Inc.   ke...@futurequest.net  (work)
>   Orlando, Floridak...@sanitarium.net (personal)
>   Web page:   http://www.sanitarium.net/
>   PGP public key available on web site.
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-
> *~'`^`'~*-,._.,-*~
> 




Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

A --exclude-checksum option?

[Karl O. Pinc]

Hi,

I use rsync with hardlinks for backup, once a week doing checksums
to ensure there's no filesystem corruption in the
backed-up data.  

I also use tmpwatch, or something similar, to clean up /tmp,
it removes files that have not been accessed recently.
(atime older than some configured limit).
I backup /tmp because I throw stuff in tmp that
I might possibly need again but don't want to bother
having to remember to delete -- and if I'm expecting
to have useful data somewhere I want it backed up.

However, rsync's checksumming (naturally) updates
the atimes of the files in /tmp, and so tmpwatch
never deletes them.

It occurs to me that a handy solution might be
to have an rsync option, similar to
the --exclude option, which would allow checksumming
to happen throughout most of the backup process but
would do "regular" size/timestamp based backups
on certain directories.

What do people think of such an option?
Is there a better design.  (E.g. an option that, er,
preserves atime when checksumming?)
Is rsync just too overloaded with options already
and it would be better instead to run
two instances of rsync?  Is there a bug/feature
in process already that would address the use-case
above?

I'd like to have a sensible design before even
thinking about patching.

Thanks for the feedback.

Regards,

Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Is there a howto/tutorial on backups/rsync that covers the use of hard and soft links?

[Karl O. Pinc]

Here's my rsync based backup system.

http://wikisend.com/download/377440/rsync_backup-0.26.tar.gz

It's an rsync based backup system utilizing hard links
to reduce storage requirements.  It supports both push
and pull.  It uses public keys with ssh for the transport.

It works and I've used it for years, but haven't released it
since there's some changes that should be made so that
it is fully generalized.  These planned changes will require
existing installations to change things (like file names).
(See the TODO file.)

Suggestions/patches are welcome.

Note that the above link expires in 90 days.  If people
want continued access let me know and I'll put it somewhere
more permanent.


On 02/01/2013 10:04:31 AM, Joe wrote:
> Thanks.  I'm taking a look at it.  May take awhile to see if it fits
> my
> needs.  It sounds promising.
> 
> Joe
> 
> On 01/31/2013 04:36 PM, Henri Shustak wrote:
> > You may be interested in having a look at LBackup
> , an open source (released under the GNU GPL)
> backup system.
> >
> > Essentially, LBackup is a wrapper for rsync. If you are working on
> your own script. Feel free to look at how LBackup works (primely
> written in bash at present) and use the features, ideas or even code
> snippets within your project in order to make your project even 
> better
> than LBackup.
> >
> > All the best with your project. Looking forward to seeing what you
> come up with.
> >
> >
> > 
> 
> > This email is protected by LBackup, an open source backup solution
> > http://www.lbackup.org
> 
> -- 
> Please use reply-all for most replies to avoid omitting the mailing
> list.
> To unsubscribe or change options: https://lists.samba.org/mailman/
> listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-
> questions.html
> 




Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Re: rsyncssl

[Karl O. Pinc]

On 01/28/2013 04:24:13 AM, devz...@web.de wrote:
> >The only place that an SSL would make some sense, is if you are 
> going
> to do it to/from an rsync daemon,
>  
> yes, exactly.
>  
>  
> >but then how would that be "better" than a ssh-only account with
> keys/etc. only allowing the rsync to execute?
>  
> I think that`s far more secure by design, because you won`t allow
> shell-access which needs hardening afterwards. 
>  
> For me it`s like giving someone the key to my house and then trying 
> to
> keep him in the hallway , hoping all the other doors being properly
> closed
>  
> I feel extremely uncomfortable with allowing other people shell 
> access
> to a box, where they need nothing but filetransfer into some 
> dedicated
> subdir.

Put this at the beginning of the ~/.ssh/authorized_keys line, before
the key:

no-pty,no-agent-forwarding,no-port-forwarding,no-user-rc,no-X11-
forwarding,command="rsync --server --daemon ." 

Yes, you do have to lock them out of shell, but once you know
how to do it it's painless to repeat at will.

On the other hand the documentation for the rsync command to use
in this circumstance could be better.  I recall figuring it out
from the man pages but can't at the moment find anything to point
to in the docs which are succinct, clear, and obvious.
That might mean there's now a better way to do it.  Others
may chime in.

And of course you're free do use another method; a separate vpn
like openvpn would be the way I would go.  Trying to build
everything into a single tool, like rsync, is the wrong way.
Instead plug together versatile components.  (It's the Un*x
way!  :)

Meanwhile restricting to nothing but filetransfer into  some dedicated
subdirectory is a completely separate matter than
restricting the shell.  For that, no matter what,
you need to mess around with rsyncd.conf.

Regards,

Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Is there a howto/tutorial on backups/rsync that covers the use of hard and soft links?

[Karl O. Pinc]

On 01/23/2013 02:15:06 AM, Voelker, Bernhard wrote:
> Kevin Korb wrote:
> > On 01/22/13 18:12, Kevin Korb wrote:
> > > That is the old way that pre-dates --link-dest.  Instead of cp -
> al
> 
> > > daily.02 daily.01 you can do a mkdir daily.01 then an rsync ... 
> > > --link-dest=../daily.02 daily.01

> I'm using a similar script on some desktop PCs. The problem is,
> that - due to the fact that it's a desktop PC - the backup can
> be killed during a shutdown ... leaving the last backup uncomplete.
> That's not a problem per se, but with the next backup, the missing
> files are copied freshly from the source and thus exhausting backup
> space.
> Therefore, the script I'm using rotates the snapshot *after* a
> successful run of rsync.

The solution is to use --link-dest twice, or more, to older
versions of the backup.  If the file is not in the most recent
backup then the older version(s) are hardlinked to.

Regards,

Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Does rsync need a --ignore-unreadable-files option?

[Karl O. Pinc]

On 01/11/2013 09:08:52 AM, Allen Supynuk wrote:
> > While you're at it, why not a more generic
> > --ignore-errno foo
> > where foo is (a defined subset of) the Posix error numbers
> > listed in errno(3).  The names, alternately, could be
> > used.
> >
> > So your application would use
> >
> > --ignore-errno EACCES
> 
> Karl,
> 
> I love this suggestion, and will do it this way if there is 
> consensus.

If there's confusion about what this option is supposed to
do then write the patch for the man page
and send it around so people know
just what is is you will do.

Regards,


Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Does rsync need a --ignore-unreadable-files option?

[Karl O. Pinc]

Hi,

On 01/10/2013 08:07:18 PM, Allen Supynuk wrote:
> It strikes me though, that what I really want is an rsync option
> '--ignore-files-with-no-read-perms'. 


While you're at it, why not a more generic
--ignore-errno foo
where foo is (a defined subset of) the Posix error numbers
listed in errno(3).  The names, alternately, could be
used.

So your application would use

--ignore-errno EACCES

Regards,

Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync in conjunction with the --link-dest option does not output deleted files

[Karl O. Pinc]

On 01/08/2013 07:17:15 AM, Kevin Korb wrote:
> You might find this useful:
> http://sanitarium.net/unix_stuff/rspaghetti_backup/diff_backup.pl.txt
> It is a script I wrote that diffs 2 backup directories and will tell
> you what is missing between them. 

There's always some simple shell (bash) scripts:

diff -u <(find olddir) <(find newdir) | grep '^- '

Will probably work.  I've not tried it.

Throw a pipe to cut on the end if you want to remove the leading
'- 's.

Regards,

Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: vanilla rsync 3.0.9 hangs after transferring ~2000 files

[Karl O. Pinc]

On 11/08/2012 05:42:08 PM, Christian Iversen wrote:

> 
> It's vanilla ext4, with:
> 
> $ df -i /data
> 
> FilesystemInodes   IUsed   IFree IUse% Mounted on
> /dev/mapper/storage-data
>  16384000 1475262 14908738   10% /data
> 
> and:
> 
> $ file /dev/mapper/storage-data
> /dev/mapper/storage-data: Linux rev 1.0 ext4 filesystem data,
> UUID=8b748eca-bdfe-4165-b7b3-debfb9463366 (needs journal recovery)
> (extents) (large files) (huge files)

It probably does not matter, but tune2fs -l would provide more
data with which to confuse the issue.  ;-)



Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: vanilla rsync 3.0.9 hangs after transferring ~2000 files

[Karl O. Pinc]

On 11/02/2012 08:33:13 AM, Christian Iversen wrote:
> Hello rsync folks

> However, 1 server is giving me a lot of trouble. It has a directory
> with (currently) 734088 files in it, and every time I try to backup
> this dir, rsync hangs after transferring roughly 2000 files. 

Since it might be filesystem related you could also tell us what the
filesystem is, and try doing a fsck on it and see if that helps.


Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: vanilla rsync 3.0.9 hangs after transferring ~2000 files

[Karl O. Pinc]

On 11/02/2012 08:33:13 AM, Christian Iversen wrote:
> Hello rsync folks
> 

> Do you have any ideas what the problem might be? Or how I can help
> debug it?

I have no idea regards the problem, but it never hurts to do a
tcpdump (-w, possibly -s) and look at what's on the wire.  
Perhaps you can figure out more by using wireshark to
view the end of the tcpdumps and compare both ends.
Make sure acks are getting through, and so forth.
This would at least eliminate the network as the source
of the problem.

But, as I say, I don't know what I'm doing.  This is
just a thought can could cause you to spend time
going down a blind alley.

Regards,

Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync has big problem with leaving lots of empty dirs around...

[Karl O. Pinc]

On 10/19/2012 03:39:18 PM, Linda Walsh wrote:
> Karl O. Pinc wrote:
> > If you want help with your problem you're going to have
> > to describe exactly what you're doing; what command
> > you're running, what version of rsync, and so forth.
> ---
> 
> Thanks...was going to file a bug report on it when I got
> time... just haven't gotten to it.

Then, more appropriate than my previous link, this is, IMO,
canonical.

How to Report Bugs Effectively
by Simon Tatham
http://www.chiark.greenend.org.uk/~sgtatham/bugs.html

Regards,

Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync has big problem with leaving lots of empty dirs around...

[Karl O. Pinc]

If you want help with your problem you're going to have
to describe exactly what you're doing; what command
you're running, what version of rsync, and so forth.

See:
http://www.catb.org/esr/faqs/smart-questions.html

On 10/17/2012 04:43:18 AM, Linda Walsh wrote:
> 
> 
> 
> 
> 
> The volume I am differencing a snapshot with has
> 595,400 dirs (424994 w/link count=2)
> & 5,703,820 files.
> 
> The diff volume starts with
> 233,764 files (a large multi-day diff)
> but 289,215 dirs!
> 
> I ran a script to clean out all the empty dirs...final tally:
> same # of files: 233764, but
> #dirs: 5462.
> 
> Thats about a huge amount of extra dirs -- I'm seeing
> almost a 10% reduction in the diff size by removing all
> the empty dirs.
> 
> At this point will likely run a clean up script on the diff vol 
> before
> I copy it to longer term storage, but it would be nice if the
> no-empty-dirs option worked... ;-)




Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Problem rsyncing 450GB file to my NAS: 'connection unexpectedly closed'

[Karl O. Pinc]

On 10/12/2012 01:06:03 PM, Justin T Pryzby wrote:

> It sounds like a daemon may be timing out; is there a timeout
> specified in rsyncd.conf?  Is there a remote logfile with any useful
> content?

I've gotten a 12 error code when a lame firewall broke the connection.


Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync hashing / collision handling?

[Karl O. Pinc]

On 10/02/2012 03:58:04 PM, Andrew Pennebaker wrote:
> Have recent versions of rsync considered using a more robust
> implementation
> of file change monitoring, like Dropbox
>  does?

The above link says Dropbox does a "binary diff".
How are you proposing to do a binary diff,
which implies a bit-for-bit comparison,
when the 2 copies of the data to be compared
are on different machines?

Regards,

Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Is --sparse suitable for general purpose use?

[Karl O. Pinc]

Hi,

I'm using rsync with --link-dest to do backups.
I don't have any sparse files, but someday I might.
Should I be using --sparse?

I notice that -S is not implied by -a.  This makes
me suspicious that --sparse is not (yet?) suitable
for general purpose use.  There also seem to be
outstanding bugs related to --sparse.

Thanks.


Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: BDsync: Block Device sync

[Karl O. Pinc]

On 06/27/2012 08:48:30 AM, Karl O. Pinc wrote:
> On 06/27/2012 06:51:29 AM, Rolf Fokkens wrote:
> 
> > After having wrestled with rsync and several patches I found a
> > solution to
> > synchronize block devices: BDsync.
> > 
> > Bdsync can be used to synchronize block devices over a network. It
> > generates
> > a "binary diff" in an efficient way by comparing  MD5 checksums of 
> > 32k
> > blocks of block devices. This binary diff can be applied to
> > "destination"
> > after which  the local blockdevices are synchronized.
> 
> > 
> > Interrested people can download the sources from:
> > http://bdsync.rolf-fokkens.nl/
> 
> I'm curious.  Why would use use bdsync instead of drdb?

Oops.  drbd.  Sorry.



Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: BDsync: Block Device sync

[Karl O. Pinc]

On 06/27/2012 06:51:29 AM, Rolf Fokkens wrote:

> After having wrestled with rsync and several patches I found a
> solution to
> synchronize block devices: BDsync.
> 
> Bdsync can be used to synchronize block devices over a network. It
> generates
> a "binary diff" in an efficient way by comparing  MD5 checksums of 
> 32k
> blocks of block devices. This binary diff can be applied to
> "destination"
> after which  the local blockdevices are synchronized.

> 
> Interrested people can download the sources from:
> http://bdsync.rolf-fokkens.nl/

I'm curious.  Why would use use bdsync instead of drdb?



Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Hlink node data for 2282618 already has path=...

[Karl O. Pinc]

On 06/24/2012 10:40:11 AM, Brian J. Murrell wrote:
> So, like a lot of people, I am using --link-dest to do backups.  My
> backup target is ext4 so with a hard link limit of 64K.  I do end up
> with trying to create too many links at some point though and get the
> following sequence of events:

> 

> It seems awfully heavy-handed for this "Too many links" condition to
> be treated as an outright error.  Surely it's sufficient to just fail
> to hard-link and move on, yes?

Typically the pathnames that run out of hard links are those that
are already hard linked in the source file system.
At least if you're using -H as well to ensure that
your backup is a real copy of the source system.

The problem is that "just fail(ing) to hard link" can cause subsequent
backup(s) to fail depending on the state in which the failure
leaves the pathnames in question, more than one pathname
because I'm assuming the source system has
hardlinking going on at the point where the problem
occurs, not to mention the
unknown state of the pathnames in the initial rsync.
Although "moving on" does get you something, you're not
going to get a good backup without some manual work.

Speaking as a sysadmin I want some predictability
in my backup system and even a small problem with a
backup will put me to work until I resolve the issue.

Agreed, a "mostly good" backup is better than no backup
at all. Even better though would be to have an actual
good backup all the time.  This could be done by
making --link-dest an "atomic(ish)" operation;
either all of the source pathhames sharing an inode
would be --link-dest-ed or none of them would.
The kludgey way to do this is to stat(2) before
each --link-dest to see if there's enough free
links before proceeding.  I say kludgey because
of the possibility of race conditions.
(I also have not looked into how you'd discover
the hard link limit for the filesystem in question.)  An
alternate approach is to, on "too many links"
error, "undo" all the --link-dests
on the "problem paths".  (I've looked at
the code and, at first glance, this seems
like it might be sane.  I posted an email,
of unknown clarity, to this list on this subject
and never heard back.)  The goal is to have
--link-dest work when possible and when not
to copy the source and have the same set of
hard links on the destination side as the source
side.

Note that this is all hot air on my part.  I've
no time now to try to code anything -- even assuming
that a patch on this matter would be accepted.

Apologies if the above is not clearly written.
I have limited opportunity to write and
thought a quick brain dump would be better than
nothing.

Regards,

Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein
e looked at
the code and 
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Using rsync to mirror directories where root owns file, using non-root user to initiate session

[Karl O. Pinc]

On 06/20/2012 10:40:57 PM, Kevin Korb wrote:
> On 06/20/12 21:53, Karl O. Pinc wrote:
> > On 06/20/2012 05:29:09 PM, Kevin Korb wrote:

> > Somehow or another you need root access on the remote side in order
> > to properly set permissions. 


> Not permissions, ownership.

Quite right.  I shouldn't be writing emails when otherwise occupied.
Sorry.


> > rsync -av -e "ssh -l ssh-user" rsync-user@host::module /dest
> 
> Now you are talking rsyncd over ssh. Still as root.  The benefit is
> minimal at best.

My point here is to show the '-e "ssh -l ssh-user"', allowing the local 
end to be non-root while the remote end is root; an example invocation
independent of whether rrsync is the command
executed on the remote end or not.  (I'm a bit confused.
Minimal as compared to what, rrsync?)

I agree that rrsync is probably the best option for
the original poster's use case, at least if he
wants to stick with userspace solutions.  I agree that command=
I supplied at the top of my post does not provide
much in the way of security on the remote end, short
of using chroot in rsyncd.conf.   I should have
been more careful in writing the post.  

The "right way", ideally, avoids all the kludgeyness of restricted
shell-like things, chroots, and so forth, and instead
uses a linux container (lxc) on the remote side for every
user on the local side.  The local user would connect
in as root to the remote container and the container
would prevent shenanigans.  It's what containers
are for.  How much sense this makes for RH 5 I can't say.



Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Using rsync to mirror directories where root owns file, using non-root user to initiate session

[Karl O. Pinc]

On 06/20/2012 05:29:09 PM, Kevin Korb wrote:
> http://www.sanitarium.net/rsyncfaq/#sudo

Along these lines...

Somehow or another you need root access on the
remote side in order to properly set permissions.
You can use ssh public keys to invoke a rsync daemon.
In /root/.ssh/authorized_keys you put the public key on
a line like:

no-pty,command="rsync --server --daemon ." ssh-dss 

This allows someone with the private key on the local
end to run a remote rsync server as root.  To invoke
on the client side see the rsync man page section:
USING RSYNC-DAEMON FEATURES VIA A REMOTE-SHELL CONNECTION

rsync -av -e "ssh -l ssh-user" rsync-user@host::module /dest

In addition to no-pty you may also wish to use
no-agent-forwarding, no-port-forwarding, no-user-rc,
no-X11-forwarding and, possibly, from="pattern-list".

The problem with the above is that the remote end
does run as root, so you're relying on your remote
rsync config to keep the user from doing bad
things on the remote end.

See also the rrsync script in the rsync examples
directory.  rrsync can be used as the command=
value instead of the above rsync command.


> 
> On 06/20/12 18:26, PEOPLES, MICHAEL P wrote:
> > I have spent a day researching and attempting to debug this issue.
> > I am hoping someone can tell me how (or disabuse me of the delusion
> > that it's possible) to do the following:
> >
> > - Mirror the contents of a directory on one server to a remote
> > server where there are diverse ownership and permissions
> >
> > - File and directory ownership on both the source and destination
> > servers would normally prevent the user account initiating the
> > rsync session from accessing, modifying, or changing attributes of
> > the files and directories in question
> >
> > - Session authentication of the initiating user on the remote
> > server must be by public key
> >
> > - No root logins are permitted on either server
> >
> > I can successfully transfer the files with the user account, but if
> > the files have ownership attributes that need to be set on the
> > remote (destination) server, using the --owner, --group, and/or
> > --perms options produces errors indicating the "Operation is not
> > permitted".  When logged into the remote server as the user, I
> > still cannot modify the attributes, only root (super user) can do
> > this.  The "--super" command line option appears to have no
> > effect.
> >
> > Both servers are Red Hat Linux.  I am using rsync 3.0.9.
> >
> > The only way I can conceive of doing this would be to record the
> > file attributes, transfer the files (along with a record of their
> > attributes), then run a script using sudo that would move the files
> > into their final location and set the attributes.  This, however,
> > would seem to defeat much of the purpose of rsync.
> >
> > The manuals suggest there is a way to invoke super user
> > functionality when contacting a daemon instance, but I could not
> > get this to work.  However, this appears to require contacting an
> > rsync daemon started by root.  Attempting to perform the rsync,
> > while simultaneously using the public key, which can only be used
> > when "ssh" is invoked, seems to exclude the use of the daemon on
> > the remote side, effectively running the entire rsync session as
> > the user without elevated privileges.
> >
> > In short, I want to copy files from one server to another, and have
> > all ownership and permissions preserved (including root), using
> > rsync to perform "privileged" operations to set file attributes
> > properly and a public key to authenticate the user.
> >
> > Thanks.
> >
> >
> > Michael Peoples (mp4783) Senior Systems Manager AT&T - ATTSI
> > Office/Cell:  614-886-0923
> > mpeop...@att.com
> >
> > This e-mail and any files transmitted with it are AT&T property,
> > are confidential, and are intended solely for the use of the
> > individual or entity to whom this email is addressed. If you are
> > not one of the named recipient(s) or otherwise have reason to
> > believe that you have received this message in error, please notify
> > the sender and delete this message immediately from your computer.
> > Any other use, retention, dissemination, forwarding, printing, or
> > copying of this e-mail is strictly prohibited."
> >
> >
> >
> 
> --
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-
> *~'`^`'~*-,._.,-*~
>   Kevin Korb  Phone:(407) 252-6853
>   Systems Administrator   Internet:
>   FutureQuest, Inc.   ke...@futurequest.net  (work)
>   Orlando, Floridak...@sanitarium.net (personal)
>   Web page:   http://www.sanitarium.net/
>   PGP public key available on web site.
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-
> *~'`^`'~*-,._.,-*~
> 




Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for m

Re: Getting messages "failed to stat" only in cron execution

[Karl O. Pinc]

> > rsync: recv_generator: failed to stat
> > "/mnt/tmp/backup/museum/documenten/secretariaat/vrijwilligers/
> medewerkersboek
> > en poster/losse foto's medewerkers/oude foto's medewerkers die we
> willen
> > bewaren/wienk, ari\#303\#253l .jpg": Invalid or incomplete 
> multibyte
> or
> > wide
> > character (84)
> >
> > If I rerun the same file from my command shell, I get none of 
> these.
> >
> > I suspect that one of my variables makes the difference, but has
> anyone an
> > idea, where to look? There are so many of them...
> >

Feels like a locale(1) thing, but that's a guess.


Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync takes long time to finish

[Karl O. Pinc]

On 04/12/2012 04:36:44 PM, vijay patel wrote:
> 
> We are running Kernel 2.6.18-308.1.1.el5 which is latest in RHEL 5.8
> on both the server. I think i might have to explore option of using
> ext4.

Before you do anything you want to figure out why it
is slow so you can solve the real problem.  vmstat, iostat
and so forth are your friends.


Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync takes long time to finish

[Karl O. Pinc]

On 04/12/2012 04:19:47 PM, Kevin Korb wrote:
> Yes, you have the feature in your filesystem.  Good.  

(Note that you will want to check the filesystems on
both ends.)




Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync takes long time to finish

[Karl O. Pinc]

On 04/12/2012 03:28:18 PM, vijay patel wrote:
> 
> Thanks friends.  We are using Redhat Linux 5.8 on Production and
> Disaster Recovery side.  By drilling down we have found out it is
> taking lot of time to check what has changed while data tranfer is
> very fast.  As i mentioned data in these folders is very less (hardly
> 40GB) and whenever new file is created, it is of max 30KB. 
>  
> Since we have to sync production environment to DR every 10 mins as
> per Business requirement i have to schedule it via cron. This already
> distributed folder structure i am using. I already have another rsync
> job which runs every 5 mins on another folder structure. It is 
> running
> fine. Is there any option i can use with rsync to make this folder
> check fast?

No.  Per the response below you need to look at your filesystems.

Use "tune2fs -l" and see if the dir_index option is on.  If not,
then turn it on using tune2fs.  This probably won't fix the existing
directories.  If this is the problem you'll have to do a
backup/restore, or a move of all the files into a new directory
hierarchy and then replace the old hierarchy, or 
something else to fix all the existing directories.
(I don't think e2fsck will help, but I've not looked.  As I say,
there may also be some other approach.)

> > If it is mostly looking for something to transfer then you need
> > filesystem optimizations. Such as directory indexing. You didn't
> > specify the OS or anything but if you are on Linux this is where an
> > ext3 > ext4 conversion would be helpful.


Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Run rsync even not connected

[Karl O. Pinc]

On 04/11/2012 09:23:57 PM, Kevin Korb wrote:
> Cron will allow for an rsync to be running in the background.
> However, there are additional steps (such as a lock file) you should
> take to make sure two don't end up running at the same time.

You could also use "at" if you want something to run once,
but not right now.



Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: link(2) EMLINK error behavior with --link-dest and --hard-links

[Karl O. Pinc]

Should I submit a bug?  Will you take a patch?

On 03/24/2012 11:11:47 PM, Karl O. Pinc wrote:
> Hi,
> 
> I'm having a problem using --link-dest and --hard-links when
> the fs hits the hard link limit (link(2) returns EMLINK).
> 
> Using rsync 3.0.7 an error is thrown and the target file is
> not created.  Glancing at git head it _looks_ like things
> could now be a little nicer.  Perhaps the target file is copied
> instead of hard linked when hardlinking fails -- I've not
> tested it.
> 
> Anyway, the behavior I desire when using both --link-dest and
> --hard-links and when running out of links is to get an error,
> but to "have the --link-dest fail".  In other words, the
> end result, for that particular target file, would be
> as if --link-dest was not specified.  This leaves the hard links
> in the source preserved in the target.  (At least an
> "exact copy" happens, which is more desirable than
> creating a copy at whatever point the hard link limit
> is reached and then hardlinking from that point forward --
> if that's what the git head code really does.)
> 
> Glancing over the code, it seems to me the way to do this 
> is to have the hard_link_one() return value indicate
> when failure is EMLINK.  This can then be tested for
> during the execution of finish_hard_link().  When
> link exhaustion is detected in finish_hard_link(),
> && link_dest, then report an error but also
> copy the source file and re-run
> finish_hard_link() to undo/fix-up the hard links
> already created.  This assumes that the
> hard links created so far can be overwritten and
> won't break things.  (And there needs to be some
> sort of flag to avoid an infinite loop
> should link exhaustion happen again.)
> 
> If this works there shouldn't be any performance
> impact unless hard links are exhausted.
> 
> I really don't understand how the code works,
> my suggestion could be completely wrong.
> I still wanted to supply a brain-dump here
> in case my suggestion is useful and to learn
> something from the feedback.
> 
> Regards,
> 
> Karl 
> Free Software:  "You don't pay back, you pay forward."
>  -- Robert A. Heinlein
> 
> -- 
> Please use reply-all for most replies to avoid omitting the mailing
> list.
> To unsubscribe or change options: https://lists.samba.org/mailman/
> listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-
> questions.html
> 
> 
> 
> 




Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

link(2) EMLINK error behavior with --link-dest and --hard-links

[Karl O. Pinc]

Hi,

I'm having a problem using --link-dest and --hard-links when
the fs hits the hard link limit (link(2) returns EMLINK).

Using rsync 3.0.7 an error is thrown and the target file is
not created.  Glancing at git head it _looks_ like things
could now be a little nicer.  Perhaps the target file is copied
instead of hard linked when hardlinking fails -- I've not
tested it.

Anyway, the behavior I desire when using both --link-dest and
--hard-links and when running out of links is to get an error,
but to "have the --link-dest fail".  In other words, the
end result, for that particular target file, would be
as if --link-dest was not specified.  This leaves the hard links
in the source preserved in the target.  (At least an
"exact copy" happens, which is more desirable than
creating a copy at whatever point the hard link limit
is reached and then hardlinking from that point forward --
if that's what the git head code really does.)

Glancing over the code, it seems to me the way to do this 
is to have the hard_link_one() return value indicate
when failure is EMLINK.  This can then be tested for
during the execution of finish_hard_link().  When
link exhaustion is detected in finish_hard_link(),
&& link_dest, then report an error but also
copy the source file and re-run
finish_hard_link() to undo/fix-up the hard links
already created.  This assumes that the
hard links created so far can be overwritten and
won't break things.  (And there needs to be some
sort of flag to avoid an infinite loop
should link exhaustion happen again.)

If this works there shouldn't be any performance
impact unless hard links are exhausted.

I really don't understand how the code works,
my suggestion could be completely wrong.
I still wanted to supply a brain-dump here
in case my suggestion is useful and to learn
something from the feedback.

Regards,

Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html