Re: A question about rsync filters, not sure if I understand the man page
On Tue, Jul 16, 2024 at 05:56:28PM +0200, francis.montag...@inria.fr wrote: > Hi. > > On Tue, 16 Jul 2024 16:00:47 +0100 Chris Green via rsync wrote: > > > I run a daily backup using 'rsync -a -F ' > > > I want to exclude everything in ~/.local/share **except** the file:- > > >/home/chris/.local/share/evolution/calendar/system/calendar.ics > > > I have the following in my rsync-filter file to exclude ~/.local/share > > > - .local/share > > > Can I simply add the following before the exclude line:- > > > + /home/chris/.local/share/evolution/calendar/system/calendar.ics > > No. Assuming you are doing this backup from your homedir, you should > add to your .rsync-filter file: > > + /.local/ > + /.local/share/ > + /.local/share/evolution/ > + /.local/share/evolution/calendar/ > + /.local/share/evolution/calendar/system/ > + /.local/share/evolution/calendar/system/calendar.ics > - /.local/share/** > > The leading / does not means the / of the machine, but the root of the > transfer (assuming this is your homedir). > Brilliant, thank you, that clarifies the explanation in the man page which did *seem* to be saying what you have laid out above but I wasn't quite sure. You have made it very clear! :-) -- Chris Green -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
A question about rsync filters, not sure if I understand the man page
I run a daily backup using 'rsync -a -F ' I want to exclude everything in ~/.local/share **except** the file:- /home/chris/.local/share/evolution/calendar/system/calendar.ics I have the following in my rsync-filter file to exclude ~/.local/share - .local/share Can I simply add the following before the exclude line:- + /home/chris/.local/share/evolution/calendar/system/calendar.ics It's not quite clear (to me anyway) if this will work or do I have to somehow add something further to allow the calendar.ics file to be found? It's the following bit that **seems** to be saying the above won't work:- It is also important to understand that the include/exclude rules are applied to every file and directory that the sender is recursing into. Thus, if you want a particular deep file to be included, you have to make sure that none of the directories that must be traversed on the way down to that file are excluded or else the file will never be discovered to be included. As an example, if the directory "a/path" was given as a transfer argument and you want to ensure that the file "a/path/down/deep/wanted.txt" is a part of the transfer, then the sender must not exclude the directories "a/path", "a/path/down", or "a/path/down/deep" as it makes it way scanning through the file tree. -- Chris Green -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Is it possible to use rsync daemon running on 64-bit machine from a 32-bit machine?
Paul Slootman via rsync wrote: > On Sun 02 Jun 2024, Chris Green via rsync wrote: > > > I have an rsync daemon running on a 64-bit (x86_64) system which I > > successfully use for backups from several other 64-bit systems on my > > LAN. > > > > I want to use it for backups from a BeagleBone Black (32-bit, armv7l) > > but it fails as follows:- > > > > root@bbb:~# rsync -a /etc chris@backup::bbb > > Password: > > pre-xfer exec returned failure (256) > > rsync error: requested action not supported (code 4) at > > clientserver.c(1171) [Receiver=3.2.7] > > rsync: [sender] read error: Connection reset by peer (104) > > root@bbb:~# > > > > Is it simply not possible to do what I'm trying to do or is there some > > way to tell the rsync daemon to work with connections from 32-bit > > clients? > > There should be no problem communicating between 32-bit and 64-bit > systems with rsync. > > Make sure that the versions of rsync aren't too different, you may run > into problems with rsync options not being supported on the old one. > > The error message says that there was a problem with the pre-xfer exec. > Check what that's doing (in the daemon's rsyncd.conf), most probably > that is the problem. > Yes, I saw that the error is in the 'pre-xfer exec' but it's exactly the same as works with all the 64-bit clients. I.e. it's the same rsyncd and configuration on the server. after a lot of digging around I found the problem. The first pass of the pre-xfer exec generates a warning error message which was failing because it was trying to send it to a system that no longer exists. The 64-bit/32-bit difference was a red herring, it was just that the 64-bit systems had been doing backups for a while so didn't generate any error messages (and so no mail failures). Thanks for making me dig further! :-) -- Chris Green · -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Is it possible to use rsync daemon running on 64-bit machine from a 32-bit machine?
I have an rsync daemon running on a 64-bit (x86_64) system which I successfully use for backups from several other 64-bit systems on my LAN. I want to use it for backups from a BeagleBone Black (32-bit, armv7l) but it fails as follows:- root@bbb:~# rsync -a /etc chris@backup::bbb Password: pre-xfer exec returned failure (256) rsync error: requested action not supported (code 4) at clientserver.c(1171) [Receiver=3.2.7] rsync: [sender] read error: Connection reset by peer (104) root@bbb:~# Is it simply not possible to do what I'm trying to do or is there some way to tell the rsync daemon to work with connections from 32-bit clients? -- Chris Green -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[PATCH] Fix INET6 detection on recent clang
The implicit int return for main() in the configure test for INET6 is
now a hard error on recent clang, breaking the detection of IPv6 support.
Update this to int main(void) like the other configure tests.
The problem this causes in practice is quite subtle and easy to miss.
ssh is always run with -4 which works fine except when there is only v6
connectivity between a pair of hosts, whereupon ssh between them works
but rsync unexpectedly fails with 'Name has no usable address'.
Signed-off-by: Chris Webb
---
configure.ac | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/configure.ac b/configure.ac
index ccad7f13..9a766c94 100644
--- a/configure.ac
+++ b/configure.ac
@@ -392,7 +392,7 @@ AS_HELP_STRING([--disable-ipv6],[disable to omit ipv6
support]),
#include
#include
#include
-main()
+int main(void)
{
if (socket(AF_INET6, SOCK_STREAM, 0) < 0)
exit(1);
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Can rsync write to a FIFO?
I have searched a little and read the man page but I can't really find a good definite answer to this. Can rsync write to a FIFO? Obviously one needs the --inplace to do this, does one also need --write-devices? It would be very handy if one can do this, to use as a simple message passing mechanism. Write something to a file on system A and rsync it to a FIFO on system B where there is a simple script reading the FIFO. The script gets the contents of the file every time it's written. (this is all within a LAN behind a reasonably secure firewall) -- Chris Green -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync using a lot of memory at receiving end (receiving end is cPanel ssh login)
Chris Green via rsync wrote: > I have been using rsync to copy some web site files to a new (to me) > hosting platform. Yesterday I was doing this and noticed that my ssh > login to cPanel in another terminal window was unresponsive. > > On looking at the browser display of my cPanel admin window I saw that > the 'Physical Memory Usage' was banging up against my limit of 1Gb. > No other limits are being reached at all, at most there's 4 or 5 > processes running, CPU load and I/O use is negligable. > > I'm running rsync version 3.0.9 protocol version 30 Oops, I meant to change that! I'm running rsync version 3.2.3 protocol version 31. > and the cPanel > receiving end is rsync version 3.0.9 protocol version 30. > > Is this to be expected (using so much memory) or is something going > wrong somewhere? More to the point is there anything I can do about > it? > -- > Chris Green > -- Chris Green · -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync using a lot of memory at receiving end (receiving end is cPanel ssh login)
I have been using rsync to copy some web site files to a new (to me) hosting platform. Yesterday I was doing this and noticed that my ssh login to cPanel in another terminal window was unresponsive. On looking at the browser display of my cPanel admin window I saw that the 'Physical Memory Usage' was banging up against my limit of 1Gb. No other limits are being reached at all, at most there's 4 or 5 processes running, CPU load and I/O use is negligable. I'm running rsync version 3.0.9 protocol version 30 and the cPanel receiving end is rsync version 3.0.9 protocol version 30. Is this to be expected (using so much memory) or is something going wrong somewhere? More to the point is there anything I can do about it? -- Chris Green -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: How to manage root<-->root rsync keeping permissions?
Greg Minshall via rsync wrote: > >If you only do backups at 1am (or whenever), why would your > > backup machine enable ssh outside of the range 12:59 - 01:01? > > Greg's rule of windows: the narrower the window, the more likely it will > be hit. :) > But I use Linux, not windows.. :-) -- Chris Green · -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: How to manage root<-->root rsync keeping permissions?
On Sat, Aug 07, 2021 at 08:10:47AM -0700, L A Walsh wrote: > On 2021/08/07 03:44, Chris Green via rsync wrote: > > L A Walsh via rsync wrote: > > > It seems to me, a safer bet would be to generate an ssh-cert > > > that allows a passwdless login from your sys to the remote. > > > > > The trouble with that is that it leaves a big security hole. > > >If you only do backups at 1am (or whenever), why would your > backup machine enable ssh outside of the range 12:59 - 01:01? > Because cron/anacron isn't perfect and the machine being backed up nay not be turned on all the time so the time that it tries to backup is most definitely not fixed accurately! > > > > If (for example) I leave my laptop turned on somewhere, or someone > > wanders into my study where my desktop machine is they have instant, > > passwordless access to the remote backup machine. > >If your desktop machine is that open to casual wanderers, perhaps > you should enable a passwd locked screen saver activating after a few > minutes? I keep my home computer unlocked all the time as well, but I > don't have walk-through visitors that might mess with it. > Neither do I, though we do have family and friends around the place quite a lot. I agree, in general, my desktop machine isn't particularly accessible or vulnerable but it *might* get hacked or accessed by an intruder and I thus try my best to protect the backup machine from it. > > I try very hard to make my backups secure from attack so that if my > > desktop or laptop is compromised somehow the (remote) backups are > > still secure. > --- >Excellent! In my case, my laptop/desktop (used to be a laptop) is > thoroughly entwined with the server such that one has trouble functioning > without the other. > >In your case, though, I was thinking of a backup process that would > only be used when my laptop was on a secure network (like @ home). > Yes, but as above, if my laptop is compromised in any way (hopefully unlikely but still possible) I want my backups to be safe still. >If there is risk to your laptop while @ home, hopefully it has a > short-timeout that bounces it to the screen saver that requires a > password to unlock?t > > > > The backup system that runs the rsync daemon has its rsync configured > > with 'refuse options = delete' > --- >Ahh...I thought you were actually trying to keep them in sync. > Maybe you might think about using an actual backup prog like tar. > In my case, the Users/groups are the same. Tar handles ext attrs and > acls and can keep track of backing files up that have actually changed > rather than relying on time/date stamps. > My *backups* of important data are incremental backups done once a day for every machine. I also do hourly incremental backups on my desktop machine but that is more for protecting myself against myself than for protecting against intruders or hardware failure. The original point of this thread is about something closer to synchronising my (small, Raspberry Pi) DNS server so that if it fails I can get a DNS server back up and running as quickly as possible. > > so not only does someone with access to > > my desktop/laptop need to know the rsyncd username and password but > > they also cannot delete my existing backups. It runs incremental > > backups so nothing is ever overwritten either. > >BTW, incremental backups aren't really the same as 'update' backups, > they keep track of the state of the file system (including files no longer > there) > so you can restore your desktop to a specific day before some unwanted > updated was introduced and kept by an update-only backup system. > Yes, exactly, or more to the point (in my case anyway) I can restore a specific file to a few hours ago after I've scrambled it in some disastrous way! :-) I use the rsync --link-dest option to make the incremental backups (and, yes, I know this means that I only really have one copy of unchanging files. I do have more than one backup) > Constructed it using rsync, but it really was too much work for > too little feature. > Mine too, as I said, is rsync with 'before the backup' and 'after the backup' python scripts that do the housekeeping like thinning out backups as they get older. -- Chris Green -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: How to manage root<-->root rsync keeping permissions?
L A Walsh via rsync wrote: > On 2021/08/03 07:09, Chris Green via rsync wrote: > > I already have an rsync daemon server running elsewhere, I can add > > this requirement to that I think. Thank you. > > > > > It seems to me, a safer bet would be to generate an ssh-cert > that allows a passwdless login from your sys to the remote. > The trouble with that is that it leaves a big security hole. If (for example) I leave my laptop turned on somewhere, or someone wanders into my study where my desktop machine is they have instant, passwordless access to the remote backup machine. I try very hard to make my backups secure from attack so that if my desktop or laptop is compromised somehow the (remote) backups are still secure. The backup system that runs the rsync daemon has its rsync configured with 'refuse options = delete' so not only does someone with access to my desktop/laptop need to know the rsyncd username and password but they also cannot delete my existing backups. It runs incremental backups so nothing is ever overwritten either. -- Chris Green · -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: How to manage root<-->root rsync keeping permissions?
Andy Smith via rsync wrote: > > > I've set it up so chris can run rsync with root permissions. > > However I'm not quite sure how to get it to work as one needs to say > > "sudo rsync" to get the root privilege. How do you do that? > > The first link I sent you had an example of that: --rsync-path="sudo > rsync" > Ah, oops, I'm sure I looked for that in the man page! :-) I thought it should be possible but couldn't find it. Thanks. -- Chris Green · -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: How to manage root<-->root rsync keeping permissions?
Andy Smith via rsync wrote: > Hi Chris, > > On Tue, Aug 03, 2021 at 11:48:31AM +0100, Chris Green via rsync wrote: > > If I used the --super option (in a command like the one above) and > > chris can run rsync as root on the remote end (via options in the > > sudoers file) will this do what I want? I guess I can go away and try > > it! :-) > > You don't need --super if the remote side actually is running as > root (either because you logged in as "root" or you logged in as > "chris" but told it to execute "sudo rsync"). > Remember, as I said, this is all Debianland with no real root login, while I could add one I'd prefer not to. > If you're going to use sudo then you'll want to set it NOPASSWD so > it doesn't ask for a sudo password. Possibly restricting that only > to uses of rsync or a specific script, otherwise it is giving > "chris" blanket sudo access without a password. > Yes, I've set it up so chris can run rsync with root permissions. However I'm not quite sure how to get it to work as one needs to say "sudo rsync" to get the root privilege. How do you do that? -- Chris Green · -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: How to manage root<-->root rsync keeping permissions?
Paul Slootman via rsync wrote: > On Tue 03 Aug 2021, Chris Green via rsync wrote: > > > Is there a way to copy (for example) the /etc hierarchy from one > > system to another preserving root ownership of files and without > > revealing root passwords all over the place? > > Best way is to run an rsync daemon on the source system, and be sure to > use "uid = 0" so that the daemon reads the source as root. > > > So, it's easy for the sending end to be run as root as it's going to be > > run by a script in /etc/cron.daily, so it can access all the files in > > /etc even if only readable by root. > > Hmm I prefer to use "pull" mechanisms as that's more secure (harder to > screw up the destination). > > So create a /etc/rsyncd.conf file with the appropriate config, something > like: > > [etc] > path = /etc > read only = yes > hosts allow = another-system > uid = 0 > > If using systemd then enable and start the daemon: > > systemctl enable rsync.service > systemctl start rsync.service > > Then on another-system as root run rsync: > > rsync -a one-system::etc/ /backups/etc/ > > I usually also use -H for hard links, but /etc usually won't have those. > > You can also use an rsync password to make this a bit more secure so > that not everyone on another-system can read all of /etc from > one-system. Details in the manpage. > I already have an rsync daemon server running elsewhere, I can add this requirement to that I think. Thank you. -- Chris Green · -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: How to manage root<-->root rsync keeping permissions?
Andy Smith via rsync wrote: > Hi Chris, > > On Tue, Aug 03, 2021 at 09:48:37AM +0100, Chris Green via rsync wrote: > > But how do you handle the other end to restore the root ownership etc.? > > The script has to do something like:- > > > > rsync -a /etc/ chris@remote:backups/etc/ > > > > So at the remote end it only has chris' privileges. > > A couple of options: > > > https://strugglers.net/~andy/blog/2021/04/10/rsync-and-sudo-without-x-forwarding/ > > Since you want to automate it I'd go with letting root log in by ssh > key only, and force the key to work only with a specific script. > > Here is an example forced command that only allows rsync > > https://www.guyrutenberg.com/2014/01/14/restricting-ssh-access-to-rsync/ > > This is still vulnerable to doing anything that rsync can do. You > can secure it further by making a script that only does the specific > things you need rsync to do, e.g. the exact parameters and paths, > and force that script instead. > Ah yes, I've done this elsewhere using 'rrsync' at the receiving end, it's another possible approach to investigate, thanks. -- Chris Green · -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: How to manage root<-->root rsync keeping permissions?
Chris Green via rsync wrote: > Is there a way to copy (for example) the /etc hierarchy from one > system to another preserving root ownership of files and without > revealing root passwords all over the place? > > This is actually from and to Debian based systems (from Raspberry Pi > to Xubuntu) so there's no actual root user login anyway, it's all sudo > from privileged user. > > So, it's easy for the sending end to be run as root as it's going to be > run by a script in /etc/cron.daily, so it can access all the files in > /etc even if only readable by root. > > But how do you handle the other end to restore the root ownership etc.? > The script has to do something like:- > > rsync -a /etc/ chris@remote:backups/etc/ > > So at the remote end it only has chris' privileges. > > > I want to automate this, I don't want any manual intervention to be > needed. > If I used the --super option (in a command like the one above) and chris can run rsync as root on the remote end (via options in the sudoers file) will this do what I want? I guess I can go away and try it! :-) -- Chris Green · -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
How to manage root<-->root rsync keeping permissions?
Is there a way to copy (for example) the /etc hierarchy from one system to another preserving root ownership of files and without revealing root passwords all over the place? This is actually from and to Debian based systems (from Raspberry Pi to Xubuntu) so there's no actual root user login anyway, it's all sudo from privileged user. So, it's easy for the sending end to be run as root as it's going to be run by a script in /etc/cron.daily, so it can access all the files in /etc even if only readable by root. But how do you handle the other end to restore the root ownership etc.? The script has to do something like:- rsync -a /etc/ chris@remote:backups/etc/ So at the remote end it only has chris' privileges. I want to automate this, I don't want any manual intervention to be needed. -- Chris Green · -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Simplest way to copy only .log files, and then delete them?
I can see very complex ways to do this but I can't see a reasonably simple way to do it. I want to copy all *.log files from a directory hierarchy and then delete them. I want to preserve the hierarchy at the destination so (if I ever need to) I can associate the log files with the place they came from. Is the only way to exclude all files and then include '*.log' files? -- Chris Green · -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[no subject]
On Sun, 25 Apr 2021 at 17:53, Chris Cowan via rsync
https://lists.samba.org/mailman/listinfo/rsync>> wrote:
>>* I’ve also been looking at several solutions that try to sandbox
>>openssh/rsync.These include rssh (which should not be used anymore,
>>because it's Abandon-ware. But, it is what I am most familiar with), GNU
>>rush, and daethnir/authprogs on github.None of these seems to be able to
>>provide me the control, with rsync, when –protect-args is used. Unless I’m
>>mistaken, the filtering has to be done by the rsync --server --sender process
>>itself, since it's the only thing that has visibility to the filepath passed
>>in the ssh channel.
*
>
> I like to use rsync in daemon mode over ssh for that type of thing, because:
>
> * you don't need a shim, just make the ssh forced command "rsync
> --server --daemon --config /path/to/some/rsyncd.conf ."
> * the --daemon turns on extra server side security checks
> * you always have --protect-args when in daemon mode
> * you can sandbox the transfer root and other things with settings in
> the rsyncd.conf
>
> Example rsync.conf for allowing reading of /var/lib/{foo,bar} but
> writes to only /var/lib/foo :
>
> use chroot = no
> [foo]
> path = /var/lib/foo
> read only = no
> [bar]
> path = /var/lib/bar
> read only = yes
> On the client side you use the :: syntax to specify a module in an
> rsync daemon along with "-e ssh" to get daemon mode over ssh, for
> example to write to /var/lib/foo/someplace you could:
> rsync -e ssh [OTHER OPTIONS] /tmp/new-foo-things ${hostname}::foo/someplace
I was aware of similar features with sshd_config for sftp, but never
considered this. (Never ran rsync in daemon mode, to be honest). The
other solutions gave me separate control over scp, sftp, svn+ssh, git, etc
Will have to read the docs to see what modules are available.
--
Chris Cowan
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
The GPFS attribute patch and sandboxing rsync when running in --server mode
First my apologies if this ended up getting double posted! I'm looking at the GPFS attribute patches that Ronnie Sahlberg originally created. The last reference I've seen on this list was using rsync v3.0.9. (Orlando Richards mentions looking at it, and I know there is also a patch on github, in the gpfsug/gpfstools repo.) We had a working version of this 3.0.9 hack for years, but I'm trying to forward port this to v3.2.3. I'm wondering: - Has anyone else forward ported this to a version > 3.0.9. - Any caveats I need to be aware of? - Still trying to get my head around the attribute cache, and the various refactorings that occurred between 3.0.3 and 3.0.9. The GPFS hack seems to have two separate options --gpfs-attrs and --gpfs-attrs-cache. (I know Ronnie, so hopefully I'll be able to get in touch with him, and he'll remember more specifics.) I’ve also been looking at several solutions that try to sandbox openssh/rsync.These include rssh (which should not be used anymore, because it's Abandon-ware. But, it is what I am most familiar with), GNU rush, and daethnir/authprogs on github.None of these seems to be able to provide me the control, with rsync, when –protect-args is used. Unless I’m mistaken, the filtering has to be done by the rsync --server --sender process itself, since it's the only thing that has visibility to the filepath passed in the ssh channel. This got me thinking, has anyone ever considered a “plug-in” / “shim” interface for rsync, to allow an admin to insert some extra security control on the src filepath? If not a plugin, perhaps a special command line flag to rsync, to identify an "exit". Perhaps something like openssh's Local Command option? I realize that if done incorrectly, it could cause more problems than it fixed. -- Chris Cowan -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Is there a default for 'pid file' for rsync in daemon mode?
Francis.Montagnac--- via rsync wrote: > > Hi. > > On Fri, 01 Jan 2021 09:57:38 + Chris Green via rsync wrote: > > > My backup system crashed a couple of nights ago due to a power cut > > (can't really blame it!) and I went and restarted it after the power > > came back. However, as I note above, files in /home/chris/tmp/pid > > aren't cleared out so rsync refused to run when systemd attempted to > > start it up. > > > I guess the pid file should either be in /tmp or somewhere in > > /run/user. However shouldn't there be some sort of sensible default > > if it isn't set in rsyncd.conf? > > Since rsyncd is launched by systemd, I would say that no pid file is > needed: systemd will prevent simultaneous launch of rsyncd. > > Can you try without spicifying "pid file" in rsyncd.conf ? > > What is the content of the rsyncd.service file on your system ? > > On Fedora it is a system (not user) service: > > cat /lib/systemd/system/rsyncd.service > [Unit] > Description=fast remote file copy program daemon > ConditionPathExists=/etc/rsyncd.conf > Wants=network-online.target > After=network-online.target > > [Service] > EnvironmentFile=/etc/sysconfig/rsyncd > ExecStart=/usr/bin/rsync --daemon --no-detach "$OPTIONS" > > [Install] > WantedBy=multi-user.target > You're quite right, simply removing the 'pid file' setting from rsynd.conf seems to work perfectly. Maybe this should be noted somwhere in the rsyncd.conf man page. -- Chris Green · -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Is there a default for 'pid file' for rsync in daemon mode?
I just got bitten by a (fairly) subtle problem due to having an inappropriate location for 'pid file' in my rsyncd.conf. What I had was:- pid file = /home/chris/tmp/pid This works fine until the system running rsync in daemon mode crashes rather than being properly shut down. The pid file doesn't get removed and rsync doesn't restart because it thinks it's already running. My backup system crashed a couple of nights ago due to a power cut (can't really blame it!) and I went and restarted it after the power came back. However, as I note above, files in /home/chris/tmp/pid aren't cleared out so rsync refused to run when systemd attempted to start it up. I guess the pid file should either be in /tmp or somewhere in /run/user. However shouldn't there be some sort of sensible default if it isn't set in rsyncd.conf? -- Chris Green · -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Is there any way to restore/create hardlinks lost in incremental backups?
Guillaume Outters via rsync wrote: > On 2020-12-11 12:53, Chris Green wrote : > > > […] wrote a trivial[ish] script that copied > > all the backups to a new destination sequentially (using --link-dest) > > and then removed the original tree, having checked the new backups > > were OK of course. > > With the same cause as yours, I once worked out exactly the same > solution. > > But then, having to automate it, I worked a bit more on it, and ended > up having a shell script that: > - recursively listed files as "file size - inode - path" > - with sort and awk, output the list of "every size that has different > inodes" > - for each output size, cksumed one file for each inode > - if two different inodes (with the same file size) had their cksum > match, then it replaced every file for the last inode, with a link to > the first inode > > If you have to run it frequently, you may want to implement something > similar. > Although it ignores mtime info (and thus strips it when lning), > it has the great benefit of finding every duplicate, be it renamed and > move to another dir > (as in > ./her.2020-12-01/Library/Mail/…/Sent.mbox/…/Attachments/…/PhotoDeFamille.JPG > versus ./his.2020-11-26/perso/photos/100_.JPG). > > (and by the way I reimplemented it in C, "just for fun" and for speed > too: https://github.com/outtersg/dude/ . Hmm, in C but in French) > The program jdupes will do it for you as well. The disadvantage (for me) of jdupes is that, given 40 or so incremental backups (which is what I had when I saw the problem) each with many tens of thousands of files in them it will take a *very* long time to do its job. Like your solution it's general, files can have different names and be in totally different places in the directory hierarchy and it will find the duplicates. In my case the files which should be duplicates (and thus be hard linked) are always ones with the same name in the same place in the hierarchy. It feels as if there should be a better/faster way of addressing this particular case but I don't know what it is. -- Chris Green · -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Is there any way to restore/create hardlinks lost in incremental backups?
Paul Slootman via rsync wrote: > On Thu 10 Dec 2020, Chris Green via rsync wrote: > > > > Occasionally, because I've moved things around or because I've done > > something else that breaks things, the hard links aren't created as > > they should be and I get a very space consuming backup increment. > > > > Is there any easy way that one can restore hard links in the *middle* > > of a series? For example say I have:- > > > > day1/pictures > > day2/pictures > > day3/pictures > > day4/pictures > > day5/pictures > > > > and I notice that day4/pictures is using as much space as > > day1/pictures but all the others are relatively small, i.e. > > day2 day3 and day5 have correctly hard linked to the previous day but > > day4 hasn't. > > > > It needs a tool that can scan day4, check a file is identical with the > > one in day3 then hardlink it without losing the link from day5. > > If you have these files that are hardlinked: > > day1/pictures/1.jpg > day2/pictures/1.jpg > day3/pictures/1.jpg > > And these are hardlinked, but to a different inode: > > day4/pictures/1.jpg > day5/pictures/1.jpg > > then there is no way of linking the second group to the first in one > step; you will have to individually link day3/pictures/1.jpg to > day4/pictures/1.jpg and then day3/pictures/1.jpg (or > day4/pictures/1.jpg) to day5/pictures/1.jpg. > > It's not like a group of directory entries that are hardlinked to one > inode are some sort of actual group; they just happen to be directory > entries that point to the same inode number. There is no other relation > between those directory entries. > > So you will have to incrementally process each next day against the > previous day. > Yes, that's what I have done, wrote a trivial[ish] script that copied all the backups to a new destination sequentially (using --link-dest) and then removed the original tree, having checked the new backups were OK of course. Fortunately I have lots of spare space on the backup system at the moment having just upgraded it with a new 8Tb drive, so duplicating the whole backup wasn't an issue (though rather slow because it was from and to the same drive). > > If I make a significant change in such a directory structure (e.g. > renaming a directory) I try to remember to do the same thing on the > backup which some say is wrong, but it saves a lot of space, like you > discovered :) > Yes, I've sometimes done that. -- Chris Green · -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Is there any way to restore/create hardlinks lost in incremental backups?
I run a simple self written incremental backup system using rsync's --link-dest option. Occasionally, because I've moved things around or because I've done something else that breaks things, the hard links aren't created as they should be and I get a very space consuming backup increment. Is there any easy way that one can restore hard links in the *middle* of a series? For example say I have:- day1/pictures day2/pictures day3/pictures day4/pictures day5/pictures and I notice that day4/pictures is using as much space as day1/pictures but all the others are relatively small, i.e. day2 day3 and day5 have correctly hard linked to the previous day but day4 hasn't. It needs a tool that can scan day4, check a file is identical with the one in day3 then hardlink it without losing the link from day5. There's jdupes but that does lose the link from day5 so you'd have to apply it to all the directories after the one that's lost the links. -- Chris Green · -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: implied-dirs tail component being chown'd
Thanks Francis,
Neither of those really do what I'd like: to properly sync/preserve
ownership/attributes for files below the root of the transfer (which are not
necessarily me:me -- I use groups for sharing between users), but to leave the
root attributes untouched, and to use filters anchored at the root.
Upon re-reading the man page, I see I misunderstood how /./ and
--no-implied-dirs works. I thought I could use them to solve this problem,
but it doesn't look like it.
>From my perspective it seems like a mis-match between where filters are
>anchored ("exclude the root") and where attributes are touched ("include the
>root").
This is a situation I encounter a fair bit. On Windows and Mac there are a
lot of user directories that are system-owned or have special system ACLs,
xattrs, etc: $HOME, Downloads, Desktop, Library, etc,
and I would like rsync to avoid touching those ownership/attributes, but also
to use filters that are anchored at the root of those (so that they can be
applied globally to any user).
E.g.
backup:
rsync -a ". HomeDir.rfilter"/Users//./
server:BACKUPDIR/Users//
restore:
rsync -a ". HomeDir.rfilter" server:BACKUPDIR/Users//./
/Users//
where HomeDir.rfilter is anchored at a user's home directory (and therefore
applies to any user) and attributes to the left of the /./ remain untouched
(which was my misunderstanding -- it doesn't work that way).
I hope this illustrates the issue I'm trying to solve.I had thought this
might be a somewhat common problem, but it doesn't seem like there is a good
solution to this. The easiest workaround for me so far is just to overwrite
the Mac/Windows system-owned attributes and make them owned by
:, but it's not ideal.
-- Chris
On Tue Aug 27 2019, at 11:31 PM, francis.montag...@inria.fr wrote:
>
> Hi.
>
> On Tue, 27 Aug 2019 14:56:25 -0700 Chris Roehrig via rsync wrote:
>
>> rsync -a --super --relative --no-implied-dirs "--filter=. HomeWin.rfilter"
>> /cygdrive/c/Users/me/./ myserver:/WinBACKUP/Users/me/
>
> If you are connecting to myserver as you (not root), simply
> suppressing the --super option should work.
>
> Otherwise (with a recent version of rsync) add: --chown me:me
>
> --
> francis
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
implied-dirs tail component being chown'd
I use rsync in a fairly complex scripted situation and am trying to figure out
how to avoid changing ownership/permissions just on the directories specified
on the command line (but operate normally for everything underneath).
I've been using --relative --no-implied-dirs with some success in other
situations, but here it still seems to try to chown the last path component of
the implied-dirs, even when the /./ separator is appended. When I use
verbose, I see that rsync does a chdir() to the implied dir on the receiving
end, but then also tries to chown('.') which I don't expect it to be doing. I
expected rsync to not try to change permissions or ownership on everything
before the /./ separator.
E.g. backing up and restoring my Windows home directory which is owned by
SYSTEM:SYSTEM with everything underneath owned by me:me to a remote Linux
server with everything owned by me:me:
rsync -a --super --relative --no-implied-dirs "--filter=. HomeWin.rfilter"
/cygdrive/c/Users/me/./ myserver:/WinBACKUP/Users/me/
where HomeWin.rfilter is anchored at a Windows home directory and contains
something like:
+ /Pictures/
- /*
This ends up failing to chown SYSTEM:SYSTEM /WinBACKUP/Users/me (permission
denied; no such user/group).
I have a usable workaround which is just to chown me:me /cygdrive/c/Users/me,
but it would be nice to figure out a proper solution without risking Windows
issues.
Any help, ideas would be appreciated.
Thanks,
-- Chris
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [Bug 12569] Missing directory errors not ignored
? (https://mail.edison.tech/chat-invite.html?invitorName=Chris%20Goodman=cgoody2...@gmail.com=?=572b82e3c44e452eab297ad5de8726d6=Rsync-Qa=rsync...@samba.org) Sent With Edison Mail > > On Oct 9, 2018 at 4:17 PM, rsync (mailto:rsync@lists.samba.org)> wrote: > > > > https://bugzilla.samba.org/show_bug.cgi?id=12569 > > --- Comment #10 from Marc Krämer--- > @Axel: cool, I've played a bit with your tool, but for my needs with many > directories inotify was the pitfall. > > I'm coauthor on sfs (https://github.com/mokraemer/sfs) which uses fuse for > signaling. And then, as you do, rsync for synchronization. So we have both > the > same problem here :( > > -- > You are receiving this mail because: > You are the QA Contact for the bug. > > -- > Please use reply-all for most replies to avoid omitting the mailing list. > To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html > -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Ensuring that rsync doesn't try to write to an unmounted drive
On Tue, 2018-09-11 at 12:56 -0400, Kevin Korb via rsync wrote:
> --timeout is about network connection timeouts. You aren't using the
> network so it doesn't apply at all. Even if you were networking an
> unmounted filesystem is an empty directory as far as rsync is
> concerned
> and rsync would treat it that way with no idea that you intended to
> have
> something mounted there.
Thanks Kevin, didn't realize that. Missed that when reading about --
timeout.
>
> Now, I see at the top of your script you check for the existence of
> the
> target directory. If that isn't the root of a filesystem then you
> are
> good because you are already checking for that. If it is the root of
> the filesystem then it will exist either as an empty directory or a
> mount point and you need to check for those possibilities.
I hope I'm answering your question. The mount info on the drive is
/dev/sdb1 on /media/chris/backup2 type ext4
(rw,nosuid,nodev,relatime,data=ordered,uhelper=udisks2)
However, in the script it's actually checking for the folder the backup
is written to 'snapshot'
BACKUP_DIR="/media/chris/backup2/snapshot/"
if [ ! -d ${BACKUP_DIR} ];then
echo "Backup destination directory ${BACKUP_DIR} not exist."
echo "run 'sudo mkdir ${BACKUP_DIR}' to create. "
exit 1
So if I understand the script is checking for the 'snapshot' folder on
the mount point 'backup2'. Is that correct? If rsync can't see the
'snapshot' folder would it still turn around and write or attempt to
write the backup to the hard drive or would it gracefully fail?
I had asked a question on LQ.org about the script earlier because I
couldn't see why it wasn't making a log file. Note-this script was
written by someone here on the list for me about 4 or so years ago.
Yes, it took me that long to notice. One of the replies to my question
suggested that I add this to the script:
mount_point='/media/chris/backup2'
df -h | grep $mount_point > /dev/null
if [ $? -eq 0 ]
then
rsync.
echo "mount point $mount_point exists, rsync started"
else
echo "Error: mount point $mount_point does not exist, rsync
operation skipped"
So, would it be ok to leave it as is or do I need to add the above to
check for the actual mount point?
>
> On 09/11/2018 12:28 PM, Chris via rsync wrote:
> > I have a script that runs nightly as a cronjob to backup my drive
> > to a
> > USB drive https://pastebin.com/yivqrGUC On the command line I use
> > the
> > --timeout option. Is this sufficient to ensure that if the external
> > drive somehow becomes unmounted that rsync will gracefully fail
> > without
> > trying to write to the hard drive instead of the USB drive?
> >
> > rsync -vaWSHpl --timeout=15 --delete-excluded --filter "merge
> > ${EXC_FILE}" / "${BACKUP_DIR}" > /home/chris/rsyncbackup.log 2>
> > /home/chris/rsyncbackup.errors.log
> >
> > If this is sufficient or would it be better if I lowered the
> > 'timeout'
> > to 5 seconds?
> >
> > Thanks for any suggestions/advice.
> >
> > Chris
> >
> >
> >
>
>
--
Chris
KeyID 0xE372A7DA98E6705C
31.11972; -97.90167 (Elev. 1092 ft)
12:50:18 up 19:57, 1 user, load average: 0.82, 0.92, 1.29
Description:Ubuntu 18.04.1 LTS, kernel 4.15.0-34-generic
signature.asc
Description: This is a digitally signed message part
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Ensuring that rsync doesn't try to write to an unmounted drive
I have a script that runs nightly as a cronjob to backup my drive to a
USB drive https://pastebin.com/yivqrGUC On the command line I use the
--timeout option. Is this sufficient to ensure that if the external
drive somehow becomes unmounted that rsync will gracefully fail without
trying to write to the hard drive instead of the USB drive?
rsync -vaWSHpl --timeout=15 --delete-excluded --filter "merge
${EXC_FILE}" / "${BACKUP_DIR}" > /home/chris/rsyncbackup.log 2>
/home/chris/rsyncbackup.errors.log
If this is sufficient or would it be better if I lowered the 'timeout'
to 5 seconds?
Thanks for any suggestions/advice.
Chris
--
Chris
KeyID 0xE372A7DA98E6705C
31.11972; -97.90167 (Elev. 1092 ft)
11:18:25 up 18:25, 1 user, load average: 1.18, 1.41, 1.43
Description:Ubuntu 18.04.1 LTS, kernel 4.15.0-34-generic
signature.asc
Description: This is a digitally signed message part
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
"Total File Size" Statistic counts each instance of hard linked files
Hi, This is a question is seeking clarification of intended behaviour. Right now, rsync reports a statistic of "Total file size". This represents "the total sum of all file sizes in the transfer" (as described in the man page). A case I've hit in using this statistic is that it counts each instance of a file even when it has multiple hard links. We are using --hard-links to preserve hard links on the destination. As a result we get a statistic of, for instance, 2TB when the actual sum on disk (counted with du, using the default behaviour of counting hard linked files only once) is only around 80GB. I'm using the statistic for generating backup disk usage numbers that eventually become billing data, so this has generated a few surprise cases. There are a few alternatives for my use-case, but I was wondering if counting hard links multiple times is actually correct behaviour? My feeling is no, but this consideration isn't apparent in the source or docs that I've read. Appreciate any comments, particularly from the project maintainers. Thanks, Chris -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync hangs on select() system call
Hi Thomas (and the list members), I have exactly the same problem. It's very very annoying as it breaks backups. Did you resolve it ? Regards, Chris -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync hangs on select() system call
Le 02/05/2015 01:09, Chris a écrit :
Hi Thomas (and the list members),
I have exactly the same problem. It's very very annoying as it breaks
backups. Did you resolve it ?
I updated on both sides to rsync 3.1.1 compiled manually, and now I get
one more line just before the select and the hang :
gettimeofday({1430522930, 618188}, NULL) = 0
select(4, [], [3], [], {60, 0}
I just ntpdated on both side just before the rsync and verified the time
is identical on both sides.
Any idea ?
Regards,
Chris
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Backup scripts
I have two scripts that a kind soul on this list wrote for me over 4yrs ago. I got to looking at them the other day because my old box crashed and had to build a new one also got a new backup USB drive since I'm still copying over things from the old one. The first one is for a full backup: http://pastebin.com/XF6Zm42A Works great, does exactly what it's supposed to do. The second is for a 'snapshot' which is where I get a bit confused. I would think that a 'snapshot' would be just the changed files either since the last full backup or since the last 'snapshot' the night before. It seems though that it's actually the same as a full backup. I don't profess to be a script person so I have no idea if it's doing what it should or something needs to be changed. Below is the 2nd script: http://pastebin.com/MkBzJnux Any advice would be appreciated. Chris -- Chris 31.11°N 97.89°W (Elev. 1092 ft) 11:21:45 up 1 day, 2:52, 1 user, load average: 0.08, 0.21, 0.27 Ubuntu 14.04 LTS, kernel 3.13.0-35-generic -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Backup scripts
On Thu, 2014-09-04 at 12:46 -0400, Kevin Korb wrote: The scripts you posted look the same to me. And I don't see any form of snapshotting. The $NOW variable is set and is echoed but it is never actually used. For an rsync snapshot I would expect to see either rsync --link-dest or a cp -al depending on the age of the script. The only reason to need two scripts would be because the snapshot would need 3 dirs to work with (the source, the target, and the previous backup) but that could also be handled with 1 script and a simple existing check. On 09/04/2014 12:30 PM, Chris wrote: I have two scripts that a kind soul on this list wrote for me over 4yrs ago. I got to looking at them the other day because my old box crashed and had to build a new one also got a new backup USB drive since I'm still copying over things from the old one. The first one is for a full backup: http://pastebin.com/XF6Zm42A Works great, does exactly what it's supposed to do. The second is for a 'snapshot' which is where I get a bit confused. I would think that a 'snapshot' would be just the changed files either since the last full backup or since the last 'snapshot' the night before. It seems though that it's actually the same as a full backup. I don't profess to be a script person so I have no idea if it's doing what it should or something needs to be changed. Below is the 2nd script: http://pastebin.com/MkBzJnux Any advice would be appreciated. Chris Thanks Kevin, I guess for now I'll leave them as they are until I can get smart on scripting. Chris -- Chris 31.11°N 97.89°W (Elev. 1092 ft) 15:57:38 up 1 day, 7:28, 1 user, load average: 0.23, 0.25, 0.28 Ubuntu 14.04 LTS, kernel 3.13.0-35-generic -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Backup scripts
On Thu, 2014-09-04 at 17:00 -0400, Kevin Korb wrote: Do you actually have any snapshots currently? From the scripts you posted it seems to just be rsyncing to the same dir every run and only claiming to be making snapshots. If you do have snapshots now then something else is happening in addition to this script. It was pointed out to me in a direct message that I had inadvertently posted the same scripts (full backup) in pastebin. Here are the correct one. Full http://pastebin.com/dEk7kBip Snapshot http://pastebin.com/H7SuABN1 On 09/04/2014 04:58 PM, Chris wrote: On Thu, 2014-09-04 at 12:46 -0400, Kevin Korb wrote: The scripts you posted look the same to me. And I don't see any form of snapshotting. The $NOW variable is set and is echoed but it is never actually used. For an rsync snapshot I would expect to see either rsync --link-dest or a cp -al depending on the age of the script. The only reason to need two scripts would be because the snapshot would need 3 dirs to work with (the source, the target, and the previous backup) but that could also be handled with 1 script and a simple existing check. On 09/04/2014 12:30 PM, Chris wrote: I have two scripts that a kind soul on this list wrote for me over 4yrs ago. I got to looking at them the other day because my old box crashed and had to build a new one also got a new backup USB drive since I'm still copying over things from the old one. The first one is for a full backup: http://pastebin.com/XF6Zm42A Works great, does exactly what it's supposed to do. The second is for a 'snapshot' which is where I get a bit confused. I would think that a 'snapshot' would be just the changed files either since the last full backup or since the last 'snapshot' the night before. It seems though that it's actually the same as a full backup. I don't profess to be a script person so I have no idea if it's doing what it should or something needs to be changed. Below is the 2nd script: http://pastebin.com/MkBzJnux Any advice would be appreciated. Chris Thanks Kevin, I guess for now I'll leave them as they are until I can get smart on scripting. Chris -- Chris 31.11°N 97.89°W (Elev. 1092 ft) 16:13:32 up 1 day, 7:44, 1 user, load average: 0.13, 0.27, 0.33 Ubuntu 14.04 LTS, kernel 3.13.0-35-generic -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Problem copying hard-linked symlinks
This begins to feel like sinking further and further into the tar pit ...
Forcibly setting #define CAN_HARDLINK_SYMLINK 1 in config.h after
configure and then make'ing gives an rsync that is only slightly better
behaved. It doesn't hang any longer (in any combination so far tried),
but rather than hardlinking the first symlink it hardlinks its *target*
(or fails if the target doesn't exist [yet]). Maybe some other config.h
change is necessary as well?
As for why CAN_HARDLINK_SYMLINK doesn't get set in the first place,
that's also still a mystery...
On Jun 16 2014, Wayne Davison wrote:
I'd imagine that your configure run was done on a filesystem that didn't
support hardlinked symlinks, since the test program it runs is super simple
and to the point.
Super simple indeed (although it should probably unlink(FILENAME 2) as
well as FILENAME, but that doesn't seem to help). Here is the relevant
part of config.log
configure.sh:7897: checking whether link() can hard-link symlinks
configure.sh:7922: cc -xc99=all -o conftest -O -DHAVE_CONFIG_H conftest.c -lsec
-lsocket -lnsl 5
conftest.c, line 166: warning: old-style declaration or incorrect type for:
main
configure.sh:7922: $? = 0
configure.sh:7922: ./conftest
configure.sh:7922: $? = 1
configure.sh: program exited with status 1
configure.sh: failed program was:
| /* confdefs.h */
| #define PACKAGE_NAME
| #define PACKAGE_TARNAME
[... a whole lot more of these, available if wanted! ...]
| #define ICONV_OPTION NULL
| #define UTF8_CHARSET UTF-8
| /* end confdefs.h. */
|
| #if HAVE_UNISTD_H
| # include unistd.h
| #endif
| #include stdlib.h
| #include errno.h
| #define FILENAME conftest.dangle
| main() {
| unlink(FILENAME);
| if (symlink(conftest.no-such, FILENAME) 0) abort();
| if (link(FILENAME, FILENAME 2) 0) exit(1);
| exit(0);
| }
configure.sh:7932: result: no
But when I run that exact same program (in the build directory, with all
the #defines's) it gives exit code 0.
Some historical research indicates that something changed here between
rsync 3.0.5 and 3.0.6. I found some old build directories on one
Solaris 10_x86 system (not all made by me, so it doesn't seem to be
a personal idioscyncracy). Running configure now for rsync 3.0.5 still
ends up with CAN_HARDLINK_SYMLINK defined in config.h, so it isn't
an OS patch level thing either.
--
Chris Thompson
Email: c...@cam.ac.uk
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Problem copying hard-linked symlinks
On Jun 14 2014, Wayne Davison wrote: On Fri, Jun 13, 2014 at 12:29 PM, Chris Thompson c...@cam.ac.uk wrote: When copying these with -H, rsync gets more than a little confused: Your rsync must be configured without CAN_HARDLINK_SYMLINK being defined (see config.h), as the normal code has been working fine for quite a while. Yes - it contains /* Define to 1 if link() can hard-link symlinks. */ /* #undef CAN_HARDLINK_SYMLINK */ But this suggests something wrong with the configure script, as Solaris 10_x86 is perfectly capable of hardlinking symlinks. (This was rsync compiled from source, using various Sun Studio C compilers.) Apparently the we-don't-support-it code wasn't tested very well, since I also saw some issues trying it out. I've checked-in a fix for this into the git repo. Thanks for the report! Meanwhile, I'll try forcibly modifying config.h, then! Thanks. -- Chris Thompson Email: c...@cam.ac.uk -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Problem copying hard-linked symlinks
This has been niggling me for a while, but am only now getting around to reporting it. I have reproduced the problem with rsync 3.1.0 and nothing in the news file for 3.1.1pre2 looks hopeful... In some filing systems it is possible to have symlinks which are themselves hard linked, i.e. have a link count greater than 1. For example, $ ls -ali /tmp/rsync-in/ total 40 4063036241 drwx-- 2 cet1 cet1 302 Jun 13 20:00 . 4033877245 drwxrwxrwt 11 root sys 1104 Jun 13 20:07 .. 4063674164 -rw-r--r-- 1 cet1 cet1 19 Jun 13 19:59 data 4068304496 lrwxrwxrwx 2 cet1 cet1 4 Jun 13 20:00 name1 - data 4068304496 lrwxrwxrwx 2 cet1 cet1 4 Jun 13 20:00 name2 - data When copying these with -H, rsync gets more than a little confused: $ rsync -aHv --delete /tmp/rsync-in/ /tmp/rsync-out/ sending incremental file list ./ rsync: link /tmp/rsync-out/name2 = name1 failed: No such file or directory (2) data name1 - data sent 184 bytes received 131 bytes 630.00 bytes/sec total size is 27 speedup is 0.09 rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1165) [sender=3.1.0] And a repeat attempt after that (with /tmp/rsync-out populated by the first attempt) is even more dire: $ rsync -aHv --delete /tmp/rsync-in/ /tmp/rsync-out/ sending incremental file list name2 - data name1 = name2 ... and at this point rsync hangs! On breaking out, name1 and name2 in the target directory are hard links to data - no symlinks to be seen. All this is with Solaris 10_x86 systems, but I have no reason to believe it is OS-specific. It certainly doesn't seem to depend on fstype (tmpfs as in the toy example above, ufs or zfs) or whether the target is local or remote. I have variants where -R is being used as well as -H. Before trying to officially report this, I thought I would ask on this mailing list whether anyone can reproduce the problem. -- Chris Thompson Email: c...@cam.ac.uk -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: /usr/bin/ssh not found when rsync is executed within rsnapshot
On 10/02/14 05:38, Lorenz wrote: i have a problem. But let me first describe my setup. [...rsnapshot configuration...] cmd_ssh/usr/bin/ssh ssh_args-i /home/backupuser/.ssh/id_rsa rsync: Failed to exec /usr/bin/ssh -i /home/backupuser/.ssh/id_rsa: No such file or directory (2) There's a bug in some versions of rsnapshot that causes it to try and execute cmd_ssh + ssh_args as a single command. The workaround is either to put as much of ssh_args as possible into $HOME/.ssh/config for the specific target host, or to create a /usr/local/bin/ssh-for-rsnapshot type script that incorporates the call to ssh with the relevant arguments. Chris smime.p7s Description: S/MIME Cryptographic Signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync seems to overwhelm a failing hard disk
Hello rsync people Today I was recovering data from a beginning-to-fail external USB hard disk. I started with my usual 'rsync -av --ignore-errors source dest', and that was fine until it got to the first I/O errors. It paused but continued after the first couple of errors, but then the disk started buzzing and rsync gave error messages for every file (I'm afraid I didn't note the exact text of the messages). After that, every run of rsync (trying to --exclude the faulty files and folders) started the disk buzzing again, and rsync couldn't copy anything. The curious thing was that after replugging the disk, I was able to manually drag-and-drop a lot more files from the disk via the Debian desktop. Trying rsync again quickly caused the buzzing problem again. In other words, a simple file copy was able to read the disk when rsync gave errors. Is that to be expected? Is there a way to make rsync more gentle on a fragile disk? cheers Chris -- Chris Dennis cgden...@btinternet.com Fordingbridge, Hampshire, UK -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync equivalent of 'cp -al' ?
Hello rsync people I thought I knew how to use rsync, but I can't work out how to use it to do the equivalent of cp -al dir1 dir2 where dir1 and dir2 are both local and on the same disk. In other words I want to make dir2 a copy of dir1, with every file hard-linked to its counterpart in dir1. Why not just use cp? Because I want to be able to do it as a user who has sudo permission to run rsync but not cp. cheers Chris -- Chris Dennis cgden...@btinternet.com Fordingbridge, Hampshire, UK -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync equivalent of 'cp -al' ?
On 18/06/13 15:02, Kevin Korb wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 rsync -vai --lin-dest=/path/to/source/ /path/to/source/ /path/to/target/ Note that if you try it with relative paths the link-dest will be relative to the source not . Thank you Kevin. I'd forgotten that --link-dest needed a relative path. This works for me: rsync -Haxv --stats --link-dest ../dir1 dir1/ dir2 The / on the end of the second dir1 is, of course, essential. cheers Chris On 06/18/13 09:39, Chris Dennis wrote: Hello rsync people I thought I knew how to use rsync, but I can't work out how to use it to do the equivalent of cp -al dir1 dir2 where dir1 and dir2 are both local and on the same disk. In other words I want to make dir2 a copy of dir1, with every file hard-linked to its counterpart in dir1. Why not just use cp? Because I want to be able to do it as a user who has sudo permission to run rsync but not cp. cheers Chris - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.20 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlHAaIgACgkQVKC1jlbQAQcKLQCfSaHopk/J2bhPCUFuwR3KSIMM daQAnjpVXhgeRz0bvb2/ttKVNZHAmuq6 =5f7j -END PGP SIGNATURE- -- Chris Dennis cgden...@btinternet.com Fordingbridge, Hampshire, UK -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync equivalent of 'cp -al' ?
On 18/06/13 16:53, Kevin Korb wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 The -H there isn't needed and could only cause increased memory usage. I realise that --link-dest implies hard links between directories, but I use -H as well to maintain any hard links within the source directory. Does that make sense? cheers Chris On 06/18/13 11:49, Chris Dennis wrote: On 18/06/13 15:02, Kevin Korb wrote: rsync -vai --lin-dest=/path/to/source/ /path/to/source/ /path/to/target/ Note that if you try it with relative paths the link-dest will be relative to the source not . Thank you Kevin. I'd forgotten that --link-dest needed a relative path. This works for me: rsync -Haxv --stats --link-dest ../dir1 dir1/ dir2 The / on the end of the second dir1 is, of course, essential. cheers Chris -- Chris Dennis cgden...@btinternet.com Fordingbridge, Hampshire, UK -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync equivalent of 'cp -al' ?
On 18/06/13 17:35, Kevin Korb wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 If you make a link to a link you make a link to all of its links. The effect is the same. Good point -- I hadn't thought of it like that. Thanks for the tip. cheers Chris On 06/18/13 12:27, Chris Dennis wrote: On 18/06/13 16:53, Kevin Korb wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 The -H there isn't needed and could only cause increased memory usage. I realise that --link-dest implies hard links between directories, but I use -H as well to maintain any hard links within the source directory. Does that make sense? cheers Chris On 06/18/13 11:49, Chris Dennis wrote: On 18/06/13 15:02, Kevin Korb wrote: rsync -vai --lin-dest=/path/to/source/ /path/to/source/ /path/to/target/ Note that if you try it with relative paths the link-dest will be relative to the source not . Thank you Kevin. I'd forgotten that --link-dest needed a relative path. This works for me: rsync -Haxv --stats --link-dest ../dir1 dir1/ dir2 The / on the end of the second dir1 is, of course, essential. cheers Chris - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.20 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlHAjG4ACgkQVKC1jlbQAQcvVgCgks/5JRXax+OLMVH3chMOqge8 vMsAoJYat6BZWtDdDGeg2kTQvr0gT5Rx =xaMU -END PGP SIGNATURE- -- Chris Dennis cgden...@btinternet.com Fordingbridge, Hampshire, UK -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: speedup is always 0.99
On 22/02/13 21:23, Grant wrote: I'm syncing from a USB disk to my hard disk like this: rsync -vr /path/to/usb/disk/dir/ /path/to/hard/disk/dir/ But the speedup is always 0.99 which I think means it is just copying the files each time instead of syncing them. What could be wrong? - Grant If either of the disks is formatted with FAT, then you'll need --modify-window=1 -- see the man page or http://serverfault.com/questions/54949/how-can-i-use-rsync-with-a-fat-file-system cheers Chris -- Chris Dennis cgden...@btinternet.com Fordingbridge, Hampshire, UK -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
--list-only ordering
Hello rsync people I've noticed an apparent inconsistency in the ordering of output from the --list-only option. For example: $ ls d1 d2 d2-x d3 f1 f2 f2-x f3 $ rsync --list-only . drwxr-xr-x4096 2012/12/17 15:18:05 . -rw-r--r-- 0 2012/12/17 15:17:52 f1 -rw-r--r-- 0 2012/12/17 15:17:52 f2 -rw-r--r-- 0 2012/12/17 15:17:52 f2-x -rw-r--r-- 0 2012/12/17 15:17:52 f3 drwxr-xr-x4096 2012/12/17 15:17:40 d1 drwxr-xr-x4096 2012/12/17 15:17:40 d2-x drwxr-xr-x4096 2012/12/17 15:17:40 d2 drwxr-xr-x4096 2012/12/17 15:17:40 d3 Note that the files are listed in a 'sensible' order, with f2-x coming after f2. But for the directories, d2-x comes before d2. Is this a bug or a feature? (I'm running rsync 3.0.9 on 64-bit Arch Linux.) And more importantly, can I rely on the output staying the same in the future? cheers Chris -- Chris Dennis cgden...@btinternet.com Fordingbridge, Hampshire, UK -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --list-only ordering
On 17/12/12 17:09, Paul Slootman wrote: On Mon 17 Dec 2012, Chris Dennis wrote: I've noticed an apparent inconsistency in the ordering of output from the --list-only option. For example: $ ls d1 d2 d2-x d3 f1 f2 f2-x f3 $ rsync --list-only . drwxr-xr-x4096 2012/12/17 15:18:05 . -rw-r--r-- 0 2012/12/17 15:17:52 f1 -rw-r--r-- 0 2012/12/17 15:17:52 f2 -rw-r--r-- 0 2012/12/17 15:17:52 f2-x -rw-r--r-- 0 2012/12/17 15:17:52 f3 drwxr-xr-x4096 2012/12/17 15:17:40 d1 drwxr-xr-x4096 2012/12/17 15:17:40 d2-x drwxr-xr-x4096 2012/12/17 15:17:40 d2 drwxr-xr-x4096 2012/12/17 15:17:40 d3 Note that the files are listed in a 'sensible' order, with f2-x coming after f2. But for the directories, d2-x comes before d2. I wouldn't be surprised if rsync appends a slash to the directory names internally; '-' sorts before '/' in ASCII. That would explain it. I just hope that the behaviour doesn't change. Thanks for your help. Chris -- Chris Dennis cgden...@btinternet.com Fordingbridge, Hampshire, UK -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: dynamic bwlimit with rsync
On 28/09/12 16:06, Satish Shukla wrote: Our pipe is limited and we don't want to give everything to rsync, We have situation where we want rsync transfers to scale up or down within a specified bandwidth limit. For e.g. if there are two (or multiple) rsyncs running parallel they should be contained in max bwlimit (say like 1Mbps) and if there is only one running it should use the entire allocated bandwidth limit. Instead of trying to use --bwlimit, take a look at trickle. It's old but works well Chris smime.p7s Description: S/MIME Cryptographic Signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Link-dest breaks on uid/gid mismatches
On 11/06/12 21:24, Kevin Korb wrote: You might want to look into --fake-super. It can store metadata things (like ownership) in the file attributes instead of setting them on the file. Thank you for the suggestion. Again, that seems to be a work-around (albeit possibly a better one)? Chris -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Link-dest breaks on uid/gid mismatches
I want to use rsync to backup a couple of Windows systems to a central point. To this end I installed the DeltaCopy implementation of an rsync server on the Windows boxes. On the central server I have rsync 3.0.7 running on a Debian GNU/Linux distribution. (I also have a similar configuration but with a QNAP - a NAS with bells on - as the central backup point. Since both exhibit the same behaviour I'm going to stay with the more common scenario.) I also have rsnapshot in the mix but to keep things simple I'm going to ignore that for now. A cut-down example rsync command could be something like this, rsync -avxH --delete --numeric-ids --relative --delete-excluded --modify-window=1 --password-file=/etc/rsnapshot.secret --link-dest=/backup/last/ rsync://rsync@windows/DocumentsAndSettings /backup/this/ If this call contacts a Linux based system, the hard links between files in /backup/last/... and their unchanged counterparts in /backup/this/ are created by the --link-dest option as expected. If this contacts a Windows based system running DeltaCopy they are not linked. Ever. I believe I have narrowed this down to a mismatch between the UID/GID values on the Windows side and the Linux side. What appears to be happening is that the Windows system is providing UID/GID values in a range that cannot be handled by the Linux system, and so it is truncating/converting them to 0/0. Then, when the next rsync runs, the UID/GID values provided by the Windows system do not match 0/0 and so copy is made of the file. (This is fast as it's built from the --link-dest item, but nevertheless it's a copy.) The resulting copy cannot have its UID/GID values applied correctly as they are out of the valid range on the Linux side, and so they are again truncated/converted to 0/0. And so on. The work-around is to apply --no-owner and --no-group to the rsync command line, at which point the files are owned by the caller of the rsync command on the central server and --link-dest can work as expected. However, I believe a better solution would, if possible, be to force the UID/GID values into a valid range before comparing them to an existing file referenced via --link-dest. Additional comments - 1. I haven't tried an alternative Windows client (e.g. cygwin/rsync) as this is outside the scope of what I can change on these systems. Therefore I cannot be sure whether this is a client or server issue, i.e. I cannot confirm that this fault report belongs to rsync rather than to DeltaCopy 2. I did get a UID/GID range error at one point but I can neither reproduce it nor remember it sufficiently accurately to want to cite it here Any other thoughts, please? Should I report this as a bug? Cheers, Chris -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync to a Remote NAS
If the NAS is 192.168.123.6 your command on the other side would be: rsync --verbose --progress --stats --compress --recursive --times --perms --links --delete /Share/ 192.168.123.6::Backup/EdensLandCorp I run this command and get failed to connect to 192.168.123.6: Connection refused. I shouldnt have to open any ports because this is coming across the VPN, right? Chris Arnold schrieb: Forgive me if this has been addressed here before. We have a remote office that we need to backup to our NAS. We have a site to site certificate VPN. The remote site has over 51gb that needs to be backed up to our NAS over that VPN. I have tried this command: rsync --verbose --progress --stats --compress --recursive --times --perms --links --delete /Share/* / smb://192.168.123.6/Backup/EdensLandCorp and it just sits there and appears to do nothing. Does rsync make a tarball first and then put it where it is told to put it or does it just copy the files/folders over? Maybe it is the smb://xx.xx.xx.xx/whatever that is breaking it..the bottom line is i need to copy/rsync a directory to a remote server through a VPN. How is this accomplished? -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Run rsync even not connected
On Apr 11, 2012, at 10:48 PM, Brian K. White br...@aljex.com wrote: Now just run the rsync command. no nohup, no . Just so completely understand, it should be: rsync --options x no nohup no Or am I completely wrong? Btw, Brian K. White, that looks familiar. You a member of the opensuse or postfix lists? -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync to a Remote NAS
Are you saying the current way we are doing it does NOT support incremental backups after the first full backup? One of the NAS devices is a readynas duo rnd2100. In the backup section of the gui, it does say backup:remote::rsync but when i select that and fill in the info and click test connection, it does not connect. Does this one support the rsync daemon? - Original Message - From: Joachim Otahal (privat) j...@gmx.net To: Chris Arnold carn...@electrichendrix.com Cc: rsync@lists.samba.org Sent: Thursday, April 12, 2012 3:28:42 PM Subject: Re: Rsync to a Remote NAS This is like mounting the remote drive via samba and then do a sync, this is like doing a normal copy job without the deltra transfer benefits of rsync. If at all possible you should run an rsync daemon on the NAS box and then run the rsync command on the other side of the VPN. rsync uses port 873 by default. Or use an extra box connected via LAN (not vpn) to mount the NAS and run the rsync daemon. If the NAS is 192.168.123.6 your command on the other side would be: rsync --verbose --progress --stats --compress --recursive --times --perms --links --delete /Share/ 192.168.123.6::Backup/EdensLandCorp You can also turn it around to let the NAS poll the backup, you need to run an rsync server on the main site then, but only a few officially support it. regards, Joachim Otahal Chris Arnold schrieb: Forgive me if this has been addressed here before. We have a remote office that we need to backup to our NAS. We have a site to site certificate VPN. The remote site has over 51gb that needs to be backed up to our NAS over that VPN. I have tried this command: rsync --verbose --progress --stats --compress --recursive --times --perms --links --delete /Share/* / smb://192.168.123.6/Backup/EdensLandCorp and it just sits there and appears to do nothing. Does rsync make a tarball first and then put it where it is told to put it or does it just copy the files/folders over? Maybe it is the smb://xx.xx.xx.xx/whatever that is breaking it..the bottom line is i need to copy/rsync a directory to a remote server through a VPN. How is this accomplished? -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Rsync to a Remote NAS
Forgive me if this has been addressed here before. We have a remote office that we need to backup to our NAS. We have a site to site certificate VPN. The remote site has over 51gb that needs to be backed up to our NAS over that VPN. I have tried this command: rsync --verbose --progress --stats --compress --recursive --times --perms --links --delete /Share/* / smb://192.168.123.6/Backup/EdensLandCorp and it just sits there and appears to do nothing. Does rsync make a tarball first and then put it where it is told to put it or does it just copy the files/folders over? Maybe it is the smb://xx.xx.xx.xx/whatever that is breaking it..the bottom line is i need to copy/rsync a directory to a remote server through a VPN. How is this accomplished? -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync to a Remote NAS
So what would plan A be? - Original Message - From: Kevin Korb k...@sanitarium.net To: rsync@lists.samba.org Sent: Wednesday, April 11, 2012 5:39:31 PM Subject: Re: Rsync to a Remote NAS -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 You mean mount -t cifs ;) This is a good plan B if the remote system can't support rsyncd or rsync over ssh. But unfortunately it would mean running with --whole-file On 04/11/12 17:36, Greg Deback (rsync) wrote: Hi, I don't think rsync natively supports samba shared volumes. You should probably start by mounting your shared volume, using mount -t smbfs, then sync. See http://users.softlab.ece.ntua.gr/~ttsiod/backup.html Greg On Wed, Apr 11, 2012 at 11:31 PM, Chris Arnold carn...@electrichendrix.com mailto:carn...@electrichendrix.com wrote: Forgive me if this has been addressed here before. We have a remote office that we need to backup to our NAS. We have a site to site certificate VPN. The remote site has over 51gb that needs to be backed up to our NAS over that VPN. I have tried this command: rsync --verbose --progress --stats --compress --recursive --times --perms --links --delete /Share/* / smb://192.168.123.6/Backup/EdensLandCorp http://192.168.123.6/Backup/EdensLandCorp and it just sits there and appears to do nothing. Does rsync make a tarball first and then put it where it is told to put it or does it just copy the files/folders over? Maybe it is the smb://xx.xx.xx.xx/whatever that is breaking it..the bottom line is i need to copy/rsync a directory to a remote server through a VPN. How is this accomplished? -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html http://www.catb.org/%7Eesr/faqs/smart-questions.html - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (GNU/Linux) iEYEARECAAYFAk+F+hIACgkQVKC1jlbQAQcAFQCdFacX7oMLOIIxgmUJRf4IV5/b KY0An3yzom6ymZG6aRAQP3zdH4u6Dt83 =REhL -END PGP SIGNATURE- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Run rsync even not connected
I hopethis hope this makes sense. How do you make rsync run even when not physically connected to the server? In other words, I run rsync from the terminal via vnc and when I log out of the connection, rsync stops running. Is there a script or something I can use? Sent from my iPhone -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Run rsync even not connected
I just thought about cron! Will a cron job accomplish this? Sent from my iPhone On Apr 11, 2012, at 10:07 PM, Kevin Korb k...@sanitarium.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Use screen or tmux. You can start rsync (or anything else) running then detach. Later you can log back in and re-attach. On 04/11/12 22:05, Chris Arnold wrote: I hopethis hope this makes sense. How do you make rsync run even when not physically connected to the server? In other words, I run rsync from the terminal via vnc and when I log out of the connection, rsync stops running. Is there a script or something I can use? Sent from my iPhone - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin KorbPhone:(407) 252-6853 Systems AdministratorInternet: FutureQuest, Inc.ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page:http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk+GOM0ACgkQVKC1jlbQAQdnuQCgjEWWux7uqcpDgjUWFeBL2CkN qXcAn3+9vIe46JvdwCjPDPtcRLNiVZol =Sg0J -END PGP SIGNATURE- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
--compress data-duplication bug?
Hi, I just noticed this commit for 3.1: --- commit cbdff74b44b25ce713739b9c1fb4db67610c675e Author: Wayne Davison way...@samba.org Date: Mon Nov 21 09:13:11 2011 -0800 Fix --compress data-duplication bug. --- Is there some reference as to what that is actually fixing, e.g. bug number? Specifically, I'm wondering if this bug might have caused any corruption in existing data sets. Cheers, Chris -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
question about why rsync log doesn't include remot ip or remote host
Hello all, This is my first post to this mailing list. I have been using rsync for a bit and have it mostly working the way I want. I have a backup server that I run all scripts from. It rsyncs data from four different Linux servers via cron. A sample of a script is here: rsync -avz --log-format=%t %b %f %l %M --delete --stats -e ssh -i /root/rsync/authors-rsync-key bac...@mydomain.com:/disk2/www /backup/seic37 21 /var/log/rsync/rsync.log This works fine and I get the log file entry that shows this: 2011/11/22 13:20:51 2089 acad0304/alllibdirstaff.MYI 2048 2011/08/31-12:46:20 I would like to include the IP and/or hostname of the machine being backed up, but when I use %a (remote IP) or %h (remote host) in the -log-format=, it just writes the literal %a or %h in those fields. Could this be because I am running scripts from the machine that is on the receiving end. Inspire me. Thanks. Christopher Adams Library Systems Coordinator Oregon State Library chris.a.ad...@state.or.us 503-378-5031 503-932-1004 (IT support) -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Use rsync's checksums to deduplicate across backups
On Thu, Nov 03, 2011 at 09:34:53AM -0500, Alex Waite wrote: Not a direct answer, but this may do what you want: http://gitweb.samba.org/?p=rsync-patches.git;a=blob;f=link-by-hash.diff This patch adds the --link-by-hash=DIR option, which hard links received files in a link farm arranged by MD4 file hash. The result is that the system will only store one copy of the unique contents of each file, regardless of the file's name. This does look like what I was describing, though it seems it was never included into rsync. Is that correct? Yes, rsync-patches is stuff that is deemed to be not yet ready (i.e. it may go in after it's been polished), or not at all suitable (e.g. it's too esoteric for general usage), for rsync proper. Chris -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Use rsync's checksums to deduplicate across backups
On 2011-11-03, Alex Waite alexq...@gmail.com wrote: I apologize if this has already been discussed before, but as of yet I have been unable to find any info on the topic. I have a very simple (and common) disk based backup system using rsync, hard links, and a little bit of perl to glue it together. Remote machines are backed up regularly using hardlinks across each snapshot to reduce disk usage. Recently I learned that rsync does a checksum of every file transferred. I thought it might be interesting to record the path and checksum of each file in a table. On future backups, the checksum of a file being backed up could be looked up in the table. If there's a matching checksum, a hard link will be created to the match instead of storing a new copy. This means that the use of hard link won't be limited to just the immediately preceding snapshot (as is the case with my current setup). Instead a hard link could be created to an identical file located in a different machine's snapshot. My initial concerns were that doing the checksums would be too CPU expensive, but if rsync is already doing them then that isn't a concern. My next thought was that the checksums would be susceptible to collisions, thus leading to potential data loss by linking to a non-identical file. However, from what I've read on wikipedia, rsync does both a MD5 and a rolling checksum. These two together make it /very/ unlikely to have a collision, thus accidentally linking to a non-identical file is unlikely. Is this approach even possible, or am I missing something? I know my labs have a lot of duplicate data across many machines, so this could save me hundreds of GiBs, maybe even a TiB or two. If this is possible, how can I save the resulting checksum of a file from rsync? Thank you for your time. I look forward to hearing your thoughts. ---Alex Not a direct answer, but this may do what you want: http://gitweb.samba.org/?p=rsync-patches.git;a=blob;f=link-by-hash.diff This patch adds the --link-by-hash=DIR option, which hard links received files in a link farm arranged by MD4 file hash. The result is that the system will only store one copy of the unique contents of each file, regardless of the file's name. Cheers, Chris -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Brandysnap -- a new rsync-based snapshot management script
On 04/07/11 02:24, Henri Shustak wrote: Hi Chris, https://github.com/StarsoftAnalysis/brandysnap I am involved with the LBackup project. Would you be okay with a link being generated to the Brandy on github page from the LBackup alternativeshttp://www.lbackup.org/alternatives page? Yes, that would be great. Depending upon the license you release Brandysnap under, perhaps in the future Brandysnap functionality could be incorporated into the LBackup project? It's GPL'd -- share and enjoy! cheers Chris -- Chris Dennis cgden...@btinternet.com Fordingbridge, Hampshire, UK -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Brandysnap -- a new rsync-based snapshot management script
Brandysnap is an rsync-based script with a difference. Unlike dirvish, it does not assign importance to snapshots when they are created. All snapshots are created equal, and then they are managed so that the required number of old snapshots is maintained. Unlike rsnapshot, brandysnap does not get its snapshots out of sync if runs are missed, and it is very easy to configure. The link between brandysnap and cron is very simple: just make sure brandysnap is run often enough to create the first level of snapshots. There is no need for separate runs to rotate the snapshot names at the required intervals. Defining which snapshots to keep is simple and intuitive. For example 4d7,7w3,4m11,3y means 'keep 4 snapshots a day for the last 7 days, 7 a week for the 3 weeks before that, then 4 a month for 11 months, then 3 a year forever'. This example is referred to in the explanations below. Such specifications can be as simple or as complex as you like. Brandysnap is flexible. It can be ultra-cautious, refusing to delete old snapshots unless there are enough to completely fill the requested schedule. Or it can be more relaxed, and apply the rules more simply. The periods 'day', 'week', 'month', etc. can be aligned on real calendar periods, or they can be considered as simple spans of time working backwards from 'now'. Brandysnap is effectively stateless: it does not keep any sort of list or database of information about previous runs. It uses the existing snapshots as the basis for deciding what needs to be done. The snapshots it creates are simple directories, which can be accessed without any special tools. Brandysnap is a Perl script designed to run on Linux. If there is demand, I hope to make it work on other operating systems too. It uses rsync (http://rsync.samba.org/) to do all the hard work, including the network and authentication stuff. Most of the cleverness in brandysnap is in deciding which snapshots to keep. As time passes, existing snapshots move into the realm of later specs, which require fewer snapshots to be kept. For example, after a week of keeping 4 snapshots a day, only 7 snapshots out of the existing 28 may still be required. Brandysnap chooses 7 that are spread out across the week to give the most useful set for future recovery. But, and this is the important bit, if runs have been missed for any reason, brandysnap will skip over incomplete periods. For example, if the destination is an external USB drive and someone forgot to plug it in, there will be snapshots missing. Brandysnap does not assume that a '4d7' spec has been completed after 7 days: it only does so when it has found 7 days each with at least 4 snapshots. Only then does it start looking for weeks with at least 7 snapshots. If you don't intend to create snapshots at weekends, specify something like 4d5,7w3,4m11,3y and brandysnap will automatically skip over the weekend days. There are options to fine-tune the way brandysnap behaves. The output includes an analysis of the existing snapshots and the way snapshots have been chosen to be kept or deleted. At present (3 July 2011), brandysnap is still being developed and should be considered 'experimental'. Do not use it for anything important (yet). Brandysnap can be downloaded from GitHub at https://github.com/StarsoftAnalysis/brandysnap where you can follow its development and read more details. Hopefully it will prove useful. I look forward to hearing comments and ideas. cheers Chris -- Chris Denniscgden...@btinternet.com Fordingbridge, Hampshire, UK -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: keep 2 dirs in sync
On 01/07/11 17:44, Paul Slootman wrote: On Fri 01 Jul 2011, Michael Makuch wrote: I have two hosts (my portable and my desktop). I work on both hosts at different times and so I keep a few dirs sync'd between the two. I have a docs dir where I may be modifying files, adding files, renaming files and deleting files on *either* host. You may want to investigate unison, which is designed for such usage. Or syncany -- www.syncany.org cheers Chris -- Chris Dennis cgden...@btinternet.com Fordingbridge, Hampshire, UK -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync and many files
On 2011-06-07, Kevin Korb k...@sanitarium.net wrote: A lot of this has to do with the filesystems and operating systems involved. Since you didn't specify I will guess Linux with ext3. If that is the case run don't walk to ext4. Also, mount the filesystems with the noatime and nodiratime options. This will prevent every stat() call from also writing to the filesystem which can be a huge performance benefit. Note: noatime implies nodiratime, i.e. no need to use both. E.g., see http://lwn.net/Articles/245002/ -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync and many files
Kevin Korb kmk at sanitarium.net writes: A lot of this has to do with the filesystems and operating systems involved. Since you didn't specify I will guess Linux with ext3. If that is the case run don't walk to ext4. Also, mount the filesystems with the noatime and nodiratime options. This will prevent every stat() call from also writing to the filesystem which can be a huge performance benefit. Note: noatime includes nodiratime, i.e. there's no need to include both. E.g. see http://lwn.net/Articles/245002/ -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync 3.0.7 hangs with unreadable hard-links files
Hello rsync list I've stumbled across a fairly obscure situation in which rsync hangs -- it just waits until I press ctrl-C. Here's a transcript. Note that directory foo contains two small files that are hard-linked together, and are unreadable. -- $ uname -a Linux ferox 2.6.35-28-generic-pae #50-Ubuntu SMP Fri Mar 18 20:43:15 UTC 2011 i686 GNU/Linux $ rsync --version rsync version 3.0.7 protocol version 30 Copyright (C) 1996-2009 by Andrew Tridgell, Wayne Davison, and others. Web site: http://rsync.samba.org/ Capabilities: 64-bit files, 64-bit inums, 32-bit timestamps, 64-bit long ints, socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace, append, ACLs, xattrs, iconv, symtimes rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions. See the GNU General Public Licence for details. $ ls -dl foo drwxr-xr-x 2 chris chris 4096 2011-06-09 23:23 foo $ ls -il foo total 8 15958048 --w--- 2 chris chris 193 2011-05-25 19:19 bd 15958048 --w--- 2 chris chris 193 2011-05-25 19:19 bd2 $ rsync -avv --hard-links foo /tmp sending incremental file list delta-transmission disabled for local transfer or --whole-file rsync: send_files failed to open /home/chris/Dropbox/perlwork/brandysnap/foo/bd2: Permission denied (13) ^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(543) [sender=3.0.7] -- If this is not a known bug, then hopefully this information will be of use. regards Chris -- Chris Dennis cgden...@btinternet.com Fordingbridge, Hampshire, UK -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: can't create a local mirror
On Sat, 2010-09-25 at 14:42 -0700, Tim Hanson wrote: On Saturday, September 25, 2010 02:35:16 pm you wrote: rsync@lists.samba.org !$#%^#$#! I have a workaround but I'm not that thrilled with it. It involves logging in to a root desktop and doing the script from there. IOW, everything else stops until I do the daily backup. Add this to your 'exclude.rules' file: - /home/*/.gvfs -- Chris KeyID 0xE372A7DA98E6705C signature.asc Description: This is a digitally signed message part -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync command failing as a Cron Tab
On Tue, 2010-09-21 at 15:38 -0700, Ian Skinner wrote: We have this Rsync command that does what is desired when ran either directly OR with a shell script. /usr/local/bin/rsync -vvv -P --stats -zrtpl --delete --password-file=/export/home/webuser/.appprod - -log-file=/exp*/h*/web*/logs/rsync-log --exclude *.htacc* /exp*/h*/xter*/htdocs/ webu...@appprod:: dprweb_www /exp*/h*/web*/logs/rsync-output But no matter how we put it into a cron task, we get nothing. No errors, no logs, no output, just nothing. 14 15 * * * cd /usr/local/doc/rsync/; ./rsyncjob.sh #WebMirror Can anybody shed some light on where we can find out why this doesn't do anything? TIA Ian I'm using the below as the command for my backup cron job. Are you sure you need the ./? Try using the whole path as I've done below and see if that works. cd /usr/local/bin; /usr/local/bin/rsync_local_backup_v2.sh HTH Chris -- Chris KeyID 0xE372A7DA98E6705C signature.asc Description: This is a digitally signed message part -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Backing up a stand alone system to a usb drive with rsync
On Thu, 2010-09-09 at 13:57 +0300, edac...@gmail.com wrote: On 09/09/2010 03:16 AM, Chris wrote: I have a stand alone system with two drives / and /home on one and /var on the the other. I'd like to backup the complete system to a usb drive. I've tried dd however since it copies over everything, even empty space it didn't seem very practical since for instance root only has 10GiB out of 70 used. Would rsync be a suitable application to use for this or will it only work over a network? Thank you for any information See http://edacval.wikidot.com/backup-whole-system-with-rsync Thanks again for the link, a couple of questions: 1 - My backup usb drive is mounted in /media/backup, shouldn't cause a problem correct? 2 - Since my system is spaced across two drives will they both be backed up or would I require some additional parameters? Chris -- Chris KeyID 0xE372A7DA98E6705C signature.asc Description: This is a digitally signed message part -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Backing up a stand alone system to a usb drive with rsync
On Thu, 2010-09-09 at 13:57 +0300, edac...@gmail.com wrote: On 09/09/2010 03:16 AM, Chris wrote: I have a stand alone system with two drives / and /home on one and /var on the the other. I'd like to backup the complete system to a usb drive. I've tried dd however since it copies over everything, even empty space it didn't seem very practical since for instance root only has 10GiB out of 70 used. Would rsync be a suitable application to use for this or will it only work over a network? Thank you for any information See http://edacval.wikidot.com/backup-whole-system-with-rsync Thanks very much for the link, I've printed it and am printing the rsync man page now. I may have a few more questions if that's ok. Thanks Chris -- Chris KeyID 0xE372A7DA98E6705C signature.asc Description: This is a digitally signed message part -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Backing up a stand alone system to a usb drive with rsync
I have a stand alone system with two drives / and /home on one and /var on the the other. I'd like to backup the complete system to a usb drive. I've tried dd however since it copies over everything, even empty space it didn't seem very practical since for instance root only has 10GiB out of 70 used. Would rsync be a suitable application to use for this or will it only work over a network? Thank you for any information -- Chris KeyID 0xE372A7DA98E6705C signature.asc Description: This is a digitally signed message part -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Check/compare modtime before deleting?
Thanks Matt. I'll do some research on unison. - Chris On Thu, Aug 19, 2010 at 8:27 PM, Matt McCutchen m...@mattmccutchen.net wrote: On Thu, 2010-08-19 at 11:31 -0700, Chris wrote: I need to keep two identical copies of files on my desktop and laptop, so I want to use the -delete option. But I don't want to delete files that are created after last sync. Is there a way to keep files with modtime later than a file of a different name (a flag file created before each sync) when using the -delete option? No. As soon as you have one side making arbitrary changes that you don't want to clobber, you need a stateful change-propagation tool such as unison (http://www.cis.upenn.edu/~bcpierce/unison/). -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Check/compare modtime before deleting?
I need to keep two identical copies of files on my desktop and laptop, so I want to use the -delete option. But I don't want to delete files that are created after last sync. Is there a way to keep files with modtime later than a file of a different name (a flag file created before each sync) when using the -delete option? Thanks and I appreciate any help. -Chris -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rSync Issue with lots of small files
G'day David, David A. Soussan das at dascomputerconsultants.com writes: First posting, and I'm a n00b with rSync. I was having issues with rSync from a PC (3.0.7 either via DeltaCopy or cwRsync) to a Freenas box (3.0.6). I've since replicated and isolated the problem attempting to sync PC to PC with various flavors of rSync on both sides. The executive summary is daemon side processes are hanging up with 100% cpu utilization when sending many small files. I can replicate this 100% of the time. What I've done: Created 2 directories with 20K files in each, 17 bytes long numbered 1.txt through 2.txt. This is the source sync directory. You may have resolved your problem by now, but for the record and in case anyone else ends up here chasing the same issue... The short answer is: use cwRsync 4.0.4 (or later). The problem is, or was, with cygwin rather than rsync, and specifically, in the communication between rsync and ssh. I could duplicate this problem when using cwRsync 4.0.3 on the sending side (using a single directory with 10,000 empty files). I wasn't able to duplicate the problem when running direct to a remote rsync daemon, i.e. not running via ssh. I haven't been able to duplicate the problem with cwRsync 4.0.4, which includes Cygwin 1.7.2-2, which in turn includes fixes in the problem area: o Fix some hangs and a potential crash using pipes and FIFOs. o Fix multiple socket problems: o Potential blocking of non-blocking sockets. o Non-working renaming of AF_UNIX socket files. o Using MSG_PEEK on UDP sockets leads to data loss. See: http://www.itefix.no/i2/node/12243 Hopefully that's the end of the reports of hangs when running rsync under Windows! Cheers, Chris. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [PATCH] Don't (wrongly) retouch dir permissions with --fake-super.
Once upon a time, Matt McCutchen m...@mattmccutchen.net said: --- generator.c |7 +-- 1 files changed, 5 insertions(+), 2 deletions(-) Hmm, I've applied this patch on both sides, but I'm still getting the same error. Is this because I'm using fake-super only on one side? # rsync -v -axH --delete --rsync-path='rsync --fake-super' / u...@remote:/dest/backup/new/root/ sending incremental file list proc/ rsync: failed to read xattr user.rsync.%stat for /dest/backup/new/root/tcb/lib: Permission denied (13) rsync: failed to read xattr user.rsync.%stat for /dest/backup/new/root/tcb/lib: Permission denied (13) rsync: failed to read xattr user.rsync.%stat for /dest/backup/new/root/tcb/lib: Permission denied (13) tcb/lib/ rsync: failed to read xattr user.rsync.%stat for /dest/backup/new/root/tcb/lib: Permission denied (13) rsync: failed to read xattr user.rsync.%stat for /dest/backup/new/root/tcb/lib: Permission denied (13) sent 50682 bytes received 561 bytes 20497.20 bytes/sec total size is 73719753 speedup is 1438.63 rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1039) [sender=3.0.6] # ls -ld /tcb/lib d--x--x--x2 bin bin 8192 Aug 20 2003 /tcb/lib/ # ssh u...@remote 'ls -ld /dest/backup/new/root/tcb/lib' d--x--x--x 2 user users 4096 Aug 20 2003 /dest/backup/new/root/tcb/lib -- Chris Adams cmad...@hiwaay.net Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [PATCH] Don't (wrongly) retouch dir permissions with --fake-super.
Once upon a time, Wayne Davison way...@samba.org said: On Sun, Nov 29, 2009 at 5:26 PM, Chris Adams cmad...@hiwaay.net wrote: Hmm, I've applied this patch on both sides, but I'm still getting the same error. The errors should go away on the next run No, the errors went away when I actually used the patched rsync on both ends (somehow missed it on one end - d'oh!). Sorry for the noise, and thanks for the fix. -- Chris Adams cmad...@hiwaay.net Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Problem with permissions
I'm trying to set up a full server backup using rsync 3.0.6. I'm using --fake-super, and SSH keys to access a remote server as a normal user. My problem is that there is a local directory that has permissions 0111 (d--x--x--x), and rsync throws an error trying to set the xattr: rsync: failed to read xattr user.rsync.%stat for /roach/backup/root/tcb/lib: Permission denied (13) It appears that rsync knows to create remote files with u+rw, but not directories. Is this a bug, or am I missing something? The full command I'm using is: rsync --stats -v -axH --inplace --delete --rsync-path=rsync --fake-super / u...@remote:/backup/root/ I have to put --fake-super in the remote command only, because the system I'm backing up doesn't support extended attributes. -- Chris Adams cmad...@hiwaay.net Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync, --sparse and VM disk images
Hi Bas, I'm not sure if this is of interest, but I also had issues with VM disk-image sparse-files (in my case KVM, rather than VMWare), which I've now resolved. http://www.finalcog.com/rsync-vm-sparse-inplace-kvm-vmware All the best, Chris Dew. P.S. Apologies for any breach of etiquette - I could not see Bas' email address on http://archives.free.net.ph/message/20090318.080211.28dac829.en.html so I've replied via this list. In reply to: Author: Bas Bahlmann || Steady IT Systeembeheer Date: 2009-03-18 08:022009-03-18 08:02 -000UTC To: rsync Subject: Is it possible to make rsync VMware split .vmdk's aware? Hi, I am using rsync for my customers to have disaster recovery off-site with files from a VMware Server (under Linux). All works very well, but when I defragment the VM's (once a week) or Exchange defragments it's datastore the disk layout changes offcourse and sometimes a lot. What do I do: - I am making a local copy with vmware-vdiskmanager to an USB disk in the split thin-disk format of the vmdk's - Then I start rsync to our datacenter to replicate the split thin-disk vmdk's What happens: Sometimes, because of the defragment within the VM or Exchange, the disk layout changes so much that a split .vmdk file that was very little and now becomes filled with 2Gb data. As a result rsync has to transfer 2Gb of data for that .vmdk which takes a lot of time. In my opnion that's not nessesary because the data is probably available in another split .vmdk because it was moved across the virtual disk. My question: Is it possible to make an option in Rsync which reads out the vmdk config file for the split disks so it can search for known data across all the split .vmdk files within one virtual disk? If this is possible this will improve the rsync process in a major way! The .vmdk config file looks like this: Contents of PVSBS2K3-1.vmdk: # Disk DescriptorFile version=1 CID=ee057ac0 parentCID= createType=twoGbMaxExtentSparse # Extent description RW 4192256 SPARSE PVSBS2K3-1-s001.vmdk RW 4192256 SPARSE PVSBS2K3-1-s002.vmdk RW 4192256 SPARSE PVSBS2K3-1-s003.vmdk RW 4192256 SPARSE PVSBS2K3-1-s004.vmdk RW 4192256 SPARSE PVSBS2K3-1-s005.vmdk RW 4192256 SPARSE PVSBS2K3-1-s006.vmdk RW 4192256 SPARSE PVSBS2K3-1-s007.vmdk RW 4192256 SPARSE PVSBS2K3-1-s008.vmdk RW 4192256 SPARSE PVSBS2K3-1-s009.vmdk RW 4192256 SPARSE PVSBS2K3-1-s010.vmdk RW 4192256 SPARSE PVSBS2K3-1-s011.vmdk RW 4192256 SPARSE PVSBS2K3-1-s012.vmdk RW 2147202 SPARSE PVSBS2K3-1-s013.vmdk # The Disk Data Base #DDB ddb.geometry.biosHeads = 255 ddb.geometry.biosSectors = 63 ddb.geometry.biosCylinders = 3265 ddb.uuid = 60 00 C2 92 f3 f3 f2 72-66 dc e5 10 bd 92 16 44 ddb.virtualHWVersion = 4 ddb.toolsVersion = 6535 ddb.geometry.cylinders = 3265 ddb.geometry.heads = 255 ddb.geometry.sectors = 63 ddb.adapterType = lsilogic I am looking forward to your answer, Thanks in advance, Bas Bahlmann The Netherlands -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- http://www.finalcog.com/ -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync docs typo
http://rsync.samba.org/ftp/rsync/rsync.html --super This tells the receiving side to attempt super-user activities even if the receiving rsync wasn't run by the super-user. These activities include: preserving users via the --owner option, preserving all groups (not just the current user's groups) via the --groups option, and copying devices via the --devices option. This is useful for systems that allow such activities without being the super-user, and also for ensuring that you will get errors if the receiving side isn't being running as the super-user. To turn off super-user activities, the super-user can use --no-super. the receiving side isn't being running as the receiving side isn't being run as -- Chris Pepper:http://cbio.mskcc.org/ http://www.extrapepperoni.com/ -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync binaries for AIX rs6000
Anyone know where I can find rsync binaries for AIX rs6000? This is for AIX 6.1 and I can't seem to find a C compiler on this box. Thanks. CONFIDENTIALITY: This e-mail (including any attachments) may contain confidential, proprietary and privileged information, and unauthorized disclosure or use is prohibited. If you receive this e-mail in error, please notify the sender and delete this e-mail from your system. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Using a CD to make initial copy
On Sun, Dec 07, 2008 at 06:58:57PM -0600, Jay Strauss wrote: Hi, I have 20Gb of data I need to back up. It takes too copy it across the internet. Is there a way I can copy it locally to some removable media like a couple of DVDs, then bring them to the target machine, copy onto the target, then run rsync to grab any updates? What if the target doesn't have the same users as the source. Is there a suggested recipe for this? See Batch Mode in the man page. -chris -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: CVS access to rsync repository
thanks Matt! On Mon, Okt 13, 2008, Matt McCutchen wrote: On Mon, 2008-10-13 at 22:39 +0200, Chris Joelly wrote: i see lots of error messages with io.c(1123) showing up, but i don't know where to get the exact revision of this io.c file? Can anybody tell me where i can find the CVS repository for example rsync 2.6.9 to get the exact revision of the mentioned file for debugging purposes? The repository uses git version control. It can be downloaded like this: git clone git://git.samba.org/rsync.git or browsed on the Web at: http://gitweb.samba.org/?p=rsync.git io.c(1123) in rsync 2.6.9 is at: http://gitweb.samba.org/?p=rsync.git;a=blob;f=io.c;hb=v2.6.9#l1123 Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
IO error / connection unexpectedly closed
Hello, i use rsync from Windows to rsync on Linux via ssh and get IO Errors and connection unexpectedly closed errors on almost every sync run. Can anybody give me some hints on how to track down the problem? thx, Chris 2008/10/11 09:41:04 [13744] rsync to backup// from UNKNOWN (w.x.y.z) 2008/10/11 09:41:04 [13744] receiving file list 2008/10/11 09:42:48 [13744] IO error encountered -- skipping file deletion 2008/10/11 09:43:59 [13744] Programme/Archiv/backend/logs/ 2008/10/11 09:49:23 [13744] rsync: connection unexpectedly closed (5324192 bytes received so far) [receiver] 2008/10/11 09:49:24 [13744] rsync: writefd_unbuffered failed to write 85 bytes [generator]: Broken pipe (32) 2008/10/11 09:49:24 [13744] rsync error: error in rsync protocol data stream (code 12) at io.c(1123) [generator=2.6.9] 2008/10/11 09:49:24 [15583] rsync to backup// from UNKNOWN (w.x.y.z) 2008/10/11 09:49:24 [15583] receiving file list 2008/10/11 09:51:08 [15583] IO error encountered -- skipping file deletion 2008/10/11 10:01:33 [15583] rsync warning: some files vanished before they could be transferred (code 24) at main.c(872) [generator=2.6.9] 2008/10/11 10:01:34 [18301] rsync to backup// from UNKNOWN (w.x.y.z) 2008/10/11 10:01:34 [18301] receiving file list 2008/10/11 10:03:16 [18301] IO error encountered -- skipping file deletion 2008/10/11 10:08:50 [18301] rsync: connection unexpectedly closed (5324186 bytes received so far) [receiver] 2008/10/11 10:08:50 [18301] rsync: writefd_unbuffered failed to write 85 bytes [generator]: Broken pipe (32) 2008/10/11 10:08:50 [18301] rsync error: error in rsync protocol data stream (code 12) at io.c(1123) [generator=2.6.9] rsync on the client: rsync version 2.6.6 protocol version 29 Copyright (C) 1996-2005 by Andrew Tridgell, Wayne Davison, and others. http://rsync.samba.org/ Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles, inplace, no IPv6, 64-bit system inums, 64-bit internal inums rsync on the server: rsync version 2.6.9 protocol version 29 Copyright (C) 1996-2006 by Andrew Tridgell, Wayne Davison, and others. http://rsync.samba.org/ Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles, inplace, IPv6, ACLs, 64-bit system inums, 64-bit internal inums -- The greatest proof that intelligent life other that humans exists in the universe is that none of it has tried to contact us! -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: IO error / connection unexpectedly closed
Hello again, now i also tried to sync from an 32bit linux machine to a 64bit linux machine and i got the same error message reported on the server side, so i think it isn't a problem due to different operating systems. the last log messages are: client: recv_generator(public/backups/cms1.it-factory.local/home/ajoelly/var/cache/apt/archives/sendmail-base_8.13.4-3sarge3_all.deb,2064) set modtime of public/backups/cms1.it-factory.local/etc/.printcap.H4IUXB to (1100788395) Thu Nov 18 15:33:15 2004 renaming public/backups/cms1.it-factory.local/etc/.printcap.H4IUXB to public/backups/cms1.it-factory.local/etc/printcap recv_generator(public/backups/cms1.it-factory.local/home/ajoelly/var/cache/apt/archives/sendmail-bin_8.13.1-16_i386.deb,2065) recv_files(public/backups/cms1.it-factory.local/etc/profile) got file_sum recv_generator(public/backups/cms1.it-factory.local/home/ajoelly/var/cache/apt/archives/sendmail-bin_8.13.1-20_i386.deb,2066) set modtime of public/backups/cms1.it-factory.local/etc/.profile.RboUbQ to (1100790549) Thu Nov 18 16:09:09 2004 rsync: connection unexpectedly closed (28352 bytes received so far) [sender] _exit_cleanup(code=12, file=io.c, line=420): entered rsync error: error in rsync protocol data stream (code 12) at io.c(420) _exit_cleanup(code=12, file=io.c, line=420): about to call exit(12) server: 2008/10/13 22:03:01 [7682] got file_sum 2008/10/13 22:03:01 [7682] set modtime of public/backups/cms1.it-factory.local/home/ajoelly/var/lib/defoma/gs.d/dirs/fonts/p052003l.pfb to (1181557381) Mon Jun 11 12:23:01 2007 2008/10/13 22:03:01 [7682] set modtime of public/backups/cms1.it-factory.local/etc/X11/xkb/symbols/.dev.vKDTmI to (1103138010) Wed Dec 15 20:13:30 2004 2008/10/13 22:03:01 [7682] public/backups/cms1.it-factory.local/home/ajoelly/var/lib/defoma/gs.d/dirs/fonts/p052003l.pfb - /rsyncd-munged//usr/share/fonts/type1/gsfonts/p052003l.pfb 2008/10/13 22:03:01 [7682] recv_generator(public/backups/cms1.it-factory.local/home/ajoelly/var/lib/defoma/gs.d/dirs/fonts/p052004l.pfb,2388) 2008/10/13 22:03:01 [7682] renaming public/backups/cms1.it-factory.local/etc/X11/xkb/symbols/.dev.vKDTmI to public/backups/cms1.it-factory.local/etc/X11/xkb/symbols/dev 2008/10/13 22:03:01 [7682] recv_files(public/backups/cms1.it-factory.local/etc/X11/xkb/symbols/dk) 2008/10/13 22:03:01 [7682] set modtime of public/backups/cms1.it-factory.local/home/ajoelly/var/lib/defoma/gs.d/dirs/fonts/p052004l.pfb to (1181557381) Mon Jun 11 12:23:01 2007 2008/10/13 22:03:01 [7682] got file_sum 2008/10/13 22:03:01 [7682] public/backups/cms1.it-factory.local/home/ajoelly/var/lib/defoma/gs.d/dirs/fonts/p052004l.pfb - /rsyncd-munged//usr/share/fonts/type1/gsfonts/p052004l.pfb 2008/10/13 22:03:01 [7682] recv_generator(public/backups/cms1.it-factory.local/home/ajoelly/var/lib/defoma/gs.d/dirs/fonts/p052023l.pfb,2389) 2008/10/13 22:03:01 [7682] set modtime of public/backups/cms1.it-factory.local/etc/X11/xkb/symbols/.dk.ziOtI2 to (1103138010) Wed Dec 15 20:13:30 2004 2008/10/13 22:03:01 [7682] renaming public/backups/cms1.it-factory.local/etc/X11/xkb/symbols/.dk.ziOtI2 to public/backups/cms1.it-factory.local/etc/X11/xkb/symbols/dk 2008/10/13 22:06:38 [7682] rsync: writefd_unbuffered failed to write 144 bytes [generator]: Broken pipe (32) 2008/10/13 22:06:38 [7682] rsync error: error in rsync protocol data stream (code 12) at io.c(1123) [generator=2.6.9] 2008/10/13 22:06:38 [7682] _exit_cleanup(code=12, file=io.c, line=1123): about to call exit(12) client version of rsync used now: rsync version 2.6.4 protocol version 29 Copyright (C) 1996-2005 by Andrew Tridgell and others http://rsync.samba.org/ Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles, inplace, IPv6, 64-bit system inums, 64-bit internal inums On Mon, Okt 13, 2008, Chris Joelly wrote: Hello, i use rsync from Windows to rsync on Linux via ssh and get IO Errors and connection unexpectedly closed errors on almost every sync run. Can anybody give me some hints on how to track down the problem? thx, Chris 2008/10/11 09:41:04 [13744] rsync to backup// from UNKNOWN (w.x.y.z) 2008/10/11 09:41:04 [13744] receiving file list 2008/10/11 09:42:48 [13744] IO error encountered -- skipping file deletion 2008/10/11 09:43:59 [13744] Programme/Archiv/backend/logs/ 2008/10/11 09:49:23 [13744] rsync: connection unexpectedly closed (5324192 bytes received so far) [receiver] 2008/10/11 09:49:24 [13744] rsync: writefd_unbuffered failed to write 85 bytes [generator]: Broken pipe (32) 2008/10/11 09:49:24 [13744] rsync error: error in rsync protocol data stream (code 12) at io.c(1123) [generator=2.6.9] 2008/10/11 09:49:24 [15583] rsync to backup// from UNKNOWN (w.x.y.z) 2008/10/11 09:49:24 [15583] receiving file list 2008/10/11 09:51:08 [15583] IO error encountered -- skipping file deletion 2008/10/11 10:01:33 [15583] rsync warning: some files vanished before they could be transferred (code 24
CVS access to rsync repository
Hello, i see lots of error messages with io.c(1123) showing up, but i don't know where to get the exact revision of this io.c file? Can anybody tell me where i can find the CVS repository for example rsync 2.6.9 to get the exact revision of the mentioned file for debugging purposes? thx, Chris -- The greatest proof that intelligent life other that humans exists in the universe is that none of it has tried to contact us! -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Bug with crtimes and hard links?
Thanks Wayne, Your patch didn't fix it, but this one seem to work: crtimes-bug.patch Description: Binary data (It turns out that file_struct does have a crtime embedded in it; I just didn't see it at first). -- Chris On Sep 27, 2008, at 6:28, Wayne Davison wrote: On Fri, Sep 26, 2008 at 05:42:40PM -0700, Chris Roehrig wrote: I've been getting spurious unnecessary copying of files on OSX when using the crtimes patch and the --crtimes -H options (version 3.0.4). This appears to be a lack of initializing the stat_x crtime value to 0 in the hlink.c code. Please try out the attached patch. ..wayne.. crtime.patch -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Bug with crtimes and hard links?
I've been getting spurious unnecessary copying of files on OSX when using the crtimes patch and the --crtimes -H options (version 3.0.4). I can reliably demonstrate it (on OSX 10.5) by doing this several times (as root): rsync -v -N -axHAX --delete-during --fileflags --force-change /usr/ bin//tmp/foo/ I think I've tracked it down to the hard-link processing code in recv_file_entry() in flist.c around line 751: Essentially, all hard links but the first will get their modtime set correctly (from first-modtime), but their crtime is inherited (incorrectly) from the previous invocation of recv_file_entry. The obvious idea would be to add a crtime field to file_struct but that looks like it has some large potential impact and I'm afraid I don't know the rsync codebase sufficiently to want to attempt a fix. -- Chris -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[EMAIL PROTECTED]: Cron [EMAIL PROTECTED] rsync -r --exclude /In/ --exclude /Lirsync error message that I don't understand
I'm getting this error message and I don't really understand what rsync is trying to tell me:- rsync: link_stat /rdiffBackup/gradwell/Mail/. failed: No such file or directory (2) rsync error: some files could not be transferred (code 23) at main.c(977) [sender=2.6.9] Can anyone explain what it's saying please. /rdiffBackup/gradwell/Mail/ does exist and is readable by the user running rsync. Oh, this is rsync 3.0.0 and the command line is:- rsync -r --exclude /In/ --exclude /Li/ --exclude /Tm/ /rdiffBackup/gradwell/Mail/ /home/chris/MailArchive -- Chris Green -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [EMAIL PROTECTED]: Cron [EMAIL PROTECTED] rsync -r --exclude /In/ --exclude /Lirsync error message that I don't understand
On Tue, Mar 25, 2008 at 12:41:25PM +0100, Paul Slootman wrote: On Tue 25 Mar 2008, Chris G wrote: I'm getting this error message and I don't really understand what rsync is trying to tell me:- rsync: link_stat /rdiffBackup/gradwell/Mail/. failed: No such file or directory (2) rsync error: some files could not be transferred (code 23) at main.c(977) [sender=2.6.9] Can anyone explain what it's saying please. /rdiffBackup/gradwell/Mail/ does exist and is readable by the user running rsync. Oh, this is rsync 3.0.0 and the command line is:- rsync -r --exclude /In/ --exclude /Li/ --exclude /Tm/ /rdiffBackup/gradwell/Mail/ /home/chris/MailArchive Does /rdiffBackup/gradwell/Mail/ exist below /home/chris/MailArchive/ ? No:- chris$ ls /home/chris/MailArchive/ boating friends holidays isbd money selling software vehicles family gardenhouseHome jobs personal sentmail telecoms work france hardware internet leisure ridingshopping trimble chris$ ls /rdiffBackup/gradwell/Mail/ In Tm family friends hardware houseHome isbd leisure personal selling software vehicles Li boating france garden holidays internet jobs money riding shopping telecoms work -- Chris Green -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [EMAIL PROTECTED]: Cron [EMAIL PROTECTED] rsync -r --exclude /In/ --exclude /Lirsync error message that I don't understand
On Tue, Mar 25, 2008 at 01:31:37PM +0100, Paul Slootman wrote: On Tue 25 Mar 2008, Chris G wrote: On Tue, Mar 25, 2008 at 12:41:25PM +0100, Paul Slootman wrote: On Tue 25 Mar 2008, Chris G wrote: I'm getting this error message and I don't really understand what rsync is trying to tell me:- rsync: link_stat /rdiffBackup/gradwell/Mail/. failed: No such file or directory (2) rsync error: some files could not be transferred (code 23) at main.c(977) [sender=2.6.9] Can anyone explain what it's saying please. /rdiffBackup/gradwell/Mail/ does exist and is readable by the user running rsync. Oh, this is rsync 3.0.0 and the command line is:- rsync -r --exclude /In/ --exclude /Li/ --exclude /Tm/ /rdiffBackup/gradwell/Mail/ /home/chris/MailArchive Does /rdiffBackup/gradwell/Mail/ exist below /home/chris/MailArchive/ ? No:- Oops, sorry, I don't know what I was thinking... (before my 1st mug of coffee, I think :-) chris$ ls /home/chris/MailArchive/ boating friends holidays isbd money selling software vehicles family gardenhouseHome jobs personal sentmail telecoms work france hardware internet leisure ridingshopping trimble chris$ ls /rdiffBackup/gradwell/Mail/ In Tm family friends hardware houseHome isbd leisure personal selling software vehicles Li boating france garden holidays internet jobs money riding shopping telecoms work Could you do an ls -la /rdiffBackup/gradwell/Mail/ ? and perhaps also without the trailing slash. It's the same with and without the trailing /. chris$ ls -la /rdiffBackup/gradwell/Mail total 104 drwxr-xr-x 26 chris chris 4096 2007-12-23 18:56 . drwxr-xr-x 26 chris chris 4096 2008-03-24 17:46 .. drwx-- 8 chris chris 4096 2008-02-24 19:15 In drwx-- 37 chris chris 4096 2008-02-26 12:37 Li drwx-- 11 chris chris 4096 2008-03-05 12:05 Tm drwx-- 5 chris chris 4096 2007-12-12 23:16 boating drwx-- 14 chris chris 4096 2007-12-12 23:22 family drwx-- 14 chris chris 4096 2008-01-29 13:47 france drwx-- 47 chris chris 4096 2007-12-12 23:20 friends drwx-- 8 chris chris 4096 2007-12-12 23:16 garden drwx-- 13 chris chris 4096 2007-12-14 13:21 hardware drwx-- 8 chris chris 4096 2007-12-12 23:26 holidays drwx-- 13 chris chris 4096 2007-12-12 23:18 houseHome drwx-- 10 chris chris 4096 2007-12-12 23:15 internet drwx-- 30 chris chris 4096 2008-03-10 09:58 isbd drwx-- 21 chris chris 4096 2007-12-12 23:13 jobs drwx-- 3 chris chris 4096 2007-12-12 23:22 leisure drwx-- 9 chris chris 4096 2007-12-13 09:17 money drwx-- 6 chris chris 4096 2007-12-12 23:18 personal drwx-- 28 chris chris 4096 2007-12-12 23:18 riding drwx-- 7 chris chris 4096 2007-12-12 23:18 selling drwx-- 20 chris chris 4096 2008-03-24 10:35 shopping drwx-- 7 chris chris 4096 2007-12-12 23:28 software drwx-- 8 chris chris 4096 2007-12-12 23:18 telecoms drwx-- 8 chris chris 4096 2007-12-12 23:12 vehicles drwx-- 4 chris chris 4096 2007-12-12 23:22 work Also running rsync with multiple -v options may help show what's wrong. I have just tried from the command line and got no error, the error was from a cron run lat night. I have added some -v to the crontab and we'll see if I get a repeat of the error tonight. BTW, is there a reason why you're doing -r and not -a ? I don't want some of the options that -a gives me, in particular not the -D and -l. -- Chris Green -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [EMAIL PROTECTED]: Cron [EMAIL PROTECTED] rsync -r --exclude /In/ --exclude /Lirsync error message that I don't understand
On Tue, Mar 25, 2008 at 02:32:34PM +0100, Paul Slootman wrote: On Tue 25 Mar 2008, Chris G wrote: BTW, is there a reason why you're doing -r and not -a ? I don't want some of the options that -a gives me, in particular not the -D and -l. That's what the -no-* things are for. You could use -a -no-D -no-l Maybe, but it's simpler just to put -r, does all I want! :-) -- Chris Green -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync 3.0.0 released
On Sat, Mar 01, 2008 at 04:58:04PM -0500, Matt McCutchen wrote: On Sat, 2008-03-01 at 13:29 -0800, Wayne Davison wrote: Yes, it's finally that time -- rsync 3.0.0 has been released. This is a feature release that also includes quite a few bug fixes. I'd like thank everyone who participated in the development and testing of rsync. I hope that you enjoy this latest version! Woohoo! This is an occasion to celebrate, but I'm also keenly aware of all the bugs and deficiencies that remain in 3.0.0, and I remain committed to fixing them for future releases. Wayne, I hope you will be open to releasing 3.0.1, etc. as additional improvements are made. (Make no mistake: I think it was the right decision to go ahead and release 3.0.0. It's been tested plenty, and now people in production environments will be more comfortable trying it. Also, it is just in time to make Fedora 9 Beta, which is a big deal for me.) As usual, my RPMS are here: http://mattmccutchen.net/rsync/#rsync-packages As a 'lowly' user I'm pleased to see rsync 3.0.0 as I've been using the release candidates for a while now to enable me to run backups to a CIFS mounted NAS system where 2.6.9 wouldn't work. So thank you all! :-) -- Chris Green -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync version executable for Debian
Thanks for that I am using Dreamhost and there is rsync installed, but not version 3. I understand that they do not upgrade the operating system or packages until there is a definite need. Thanks Chris Chris A Harris Adelaide Australia Paul Slootman wrote: On Wed 30 Jan 2008, Chris A Harris wrote: I will wait for a Debian package and extract the file; I do not have enough experience to compile from source. You can download an rsync .deb via http://packages.debian.org/rsync (be sure to select the correct distribution). You can then extract the contents of the package with: dpkg-deb --extract rsync*.deb destdir/ destdir/usr/bin/rsync will then be the executable you want. Note that you may run into problems with additional shared libraries that rsync needs, e.g. libacl1, libpopt0; if they're not already installed, you need to download those as well and get creative with $LD_LIBRARY_PATH. Have you asked your hoster whether he is willing to install rsync? I'm surprised, actually, that rsync isn't already installed. I wonder how he makes his backups. (There are of course alternatives, but esp. for a hosting server rsync is very useful.) Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync version executable for Debian
Thanks for the reply. I will wait for a Debian package and extract the file; I do not have enough experience to compile from source. Regards Chris On Wed, 2008-01-30 at 13:43 +1030, Chris A Harris wrote: I am using a shared host running on Debian sarge. I can not install packages. Is there an executable for that call be used? Yes. You can compile an executable for any version of rsync you like from the sources at http://rsync.samba.org/ftp/rsync/ , or you can download the Debian package you want and extract the executable from it manually (a Debian package is just an ar archive containing a tarball data.tar.gz containing the installed files). Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync version executable for Debian
I am using a shared host running on Debian sarge. I can not install packages. Is there an executable for that call be used? Regards Chris Adelaide Australia -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Backslash in filename causes error
rsync: recv_generator: failed to stat /freecom/backup/www/html/chris/info/computer/hardware/networkStorage/freecom/objects\5151.pdf: Invalid argument The file exists OK. -- Chris Green -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Problem with filenames with commas in them
This is a continuation of my previous problem where cp could copy files whereas rsync couldn't. It turns out that the problem is with files which have commas in their names, rsync can write the initial version of the file but it can't check/rewrite them. Here is the error I get from rsync when trying to overwrite the files (using --inplace) :- home# rsync -r --inplace .in /freecom/backup/home/chris/Mtest/ rsync: open /freecom/backup/home/chris/Mtest/.in/audacity/cur/1172007150.76430_0.newred:2, failed: Input/output error (5) rsync: open /freecom/backup/home/chris/Mtest/.in/audacity/cur/1172012773.6188_0.newred:2, failed: Input/output error (5) rsync: open /freecom/backup/home/chris/Mtest/.in/audacity/cur/1172032604.34523_0.newred:2, failed: Input/output error (5) rsync: open /freecom/backup/home/chris/Mtest/.in/audacity/cur/1172038263.69147_0.newred:2, failed: Input/output error (5) rsync: open /freecom/backup/home/chris/Mtest/.in/freecycle/cur/1172005922.70482_0.newred:2,S failed: Input/output error (5) rsync error: some files could not be transferred (code 23) at main.c(977) [sender=2.6.9] Now if I do a find for the maildir files in the destination directory I see:- ./audacity/cur/1172007150.76430_0.newred:2, ./audacity/cur/1172012773.6188_0.newred:2, ./audacity/cur/1172032604.34523_0.newred:2, ./audacity/cur/1172038263.69147_0.newred:2, ./cafe/new/1172050891.76512_0.newred ./cafe/new/.1172050891.76512_0.newred.Db1dJL ./chris/new/1172049386.67574_0.newred ./chris/new/1172049415.67719_0.newred ./chris/new/1172049422.67809_0.newred ./chris/new/1172049736.69687_0.newred ./chris/new/1172050647.75211_0.newred ./chris/new/1172050647.75212_0.newred ./chris/new/1172052822.87331_0.newred ./chris/new/1172052833.87374_0.newred ./chris/new/.1172052839.87385_0.newred.6zXpLZ ./freecycle/cur/1172005922.70482_0.newred:2,S ./ixion/new/1172048011.60081_0.newred ./ixion/new/1172048786.64308_0.newred ./ixion/new/1172048991.65223_0.newred ./ixion/new/1172049388.67581_0.newred ./ixion/new/1172050165.71835_0.newred ./ixion/new/1172050725.75518_0.newred ./ixion/new/1172050830.76247_0.newred ./ixion/new/1172051518.79746_0.newred [snip lots more] The five errors are on the only five files that have commas in their names. Now this *may* be because the destination can't cope with filenames with commas in very well but the files are certainly accessible on the destination and appear quite normal. An rsync -r without the --inplace just gets a different error:- rsync: rename /freecom/backup/home/chris/Mtest/.in/audacity/cur/.1172007150.76430_0.newred:2,.V9Pd4n - .in/audacity/cur/1172007150.76430_0.newred:2,: Input/output error (5) and *anyway* why is rsync trying to copy these files at all, they're unchanged! -- Chris Green -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Problem with filenames with commas in them
On Wed, Dec 12, 2007 at 09:33:14AM +, Chris G wrote: This is a continuation of my previous problem where cp could copy files whereas rsync couldn't. It turns out that the problem is with files which have commas in their names, rsync can write the initial version of the file but it can't check/rewrite them. Here is the error I get from rsync when trying to overwrite the files (using --inplace) :- home# rsync -r --inplace .in /freecom/backup/home/chris/Mtest/ rsync: open /freecom/backup/home/chris/Mtest/.in/audacity/cur/1172007150.76430_0.newred:2, failed: Input/output error (5) rsync: open /freecom/backup/home/chris/Mtest/.in/audacity/cur/1172012773.6188_0.newred:2, failed: Input/output error (5) rsync: open /freecom/backup/home/chris/Mtest/.in/audacity/cur/1172032604.34523_0.newred:2, failed: Input/output error (5) rsync: open /freecom/backup/home/chris/Mtest/.in/audacity/cur/1172038263.69147_0.newred:2, failed: Input/output error (5) rsync: open /freecom/backup/home/chris/Mtest/.in/freecycle/cur/1172005922.70482_0.newred:2,S failed: Input/output error (5) rsync error: some files could not be transferred (code 23) at main.c(977) [sender=2.6.9] ... I have found a solution, I had to install rsync version 3.0.0 to get it though. If I use the --temp-dir option to put temporary files on the local ext3 filesystem then the rsync succeeds, hurrah! :-) home# /usr/local/bin/rsync -r --temp-dir=/tmp .in /freecom/backup/home/chris/Mtest home# (I have the version 3.0.0pre6 installed in /usr/local/bin at the moment) While I'm about it the alternative -T option for --temp-dir doesn't work, rsync just says -T=/tmp: unknown option. Still I seem to have a fix for my problem. -- Chris Green -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
