Re: Is there a better way to transfer data that doesn't use so much cache?

On Wed, Aug 3, 2022 at 5:41 PM Robin Lee Powell via rsync < rsync@lists.samba.org> wrote: > On Wed, Aug 03, 2022 at 02:04:22PM -0400, Rob Campbell via rsync wrote: > > The problem isn't that there are many syncs because the problem happens > on > > the first one that runs. > > You didn't actually

Re: Merging three slightly different directories

I suspect you want a duplicate finder more than a file transfer tool. EG: https://stromberg.dnsalias.org/~strombrg/equivalence-classes.html On Tue, Jun 7, 2022 at 5:36 PM hput via rsync wrote: > I want to merge 3 slightly different directories of mostly images. > > Not just mostly but the

Re: Trying to elevate rsync privileges when connecting over ssh without using NOPASSWD in sudoers

On Sat, Mar 12, 2022 at 12:23 PM Dr. Mark Asbach via rsync < rsync@lists.samba.org> wrote: > Hi there, hi past me, > > > My (non-working) attempt: > > […] > > So it seems the "-l" is dropped into the void letting ssh assume USER > was the target host? I don’t actually get what I can do. > > Turns

Re: Trying to elevate rsync privileges when connecting over ssh without using NOPASSWD in sudoers

Why not rsync directly as root? Then you can use a passwordless, passphraseless RSA (or similar) keypair. On Fri, Mar 11, 2022 at 4:58 AM Dr. Mark Asbach via rsync < rsync@lists.samba.org> wrote: > Hi there, > > We are using ansible to deploy system configuration and web application > source

Re: Confused as to why rsync thinks time, owner and group of many files differ

On Thu, Feb 3, 2022 at 3:50 PM Andy Smith via rsync wrote: > I am tempted to blow away the btrfs filesystem and just do xfs to > xfs, to rule out weird issues there. It would be a shame though as > I was hoping to use btrfs's compression here. > You might be able to do a partial transfer to a

Windows backups?

Can rsync back up an NTFS using a Windows 10 kernel? So far I've had good luck backing up NTFS filesystems on a dual boot system when booted into Linux, but not when booted into Windows. I've been bitten in the past by /usr/bin/find (for EG) having problems with Windows junctions over sshfs.

Re: BackUp with "inverse" increments

rsync --link-dest is fast and simple, good stuff. I recommend using it with some sort of wrapper script for rotations, like Backup.rsync: https://stromberg.dnsalias.org/~strombrg/Backup.remote.html On Wed, Oct 6, 2021 at 7:29 AM Kevin Korb via rsync wrote: > See --link-dest. That is what

Re: Question about rsync -uav dir1/. dir2/.: possib to link?

I was thinking --link-dest too. Sometimes this can be done with cpio too; check out the -pdlv options. On Sat, Sep 4, 2021 at 4:57 PM Kevin Korb via rsync wrote: > Rsync does almost everything cp does but since it is designed to network > it never got that feature. I was thinking maybe

Re: Which method (rsync-based) is better for backing up Linux servers to Synology NAS?

I suppose I may as well mention: https://stromberg.dnsalias.org/~dstromberg/Backup.remote It just does rsync snapshotting with --link-dest, and keeps the last n snapshots. It's smart enough to resume a previously interrupted snapshot. It's pretty simple - both to set up and to use. I used to

Re: Utility of --backup

In Backup.rsync, which of course is a wrapper around rsync that can be used for backups, I do not use --backup, but I do use --link-dest: https://stromberg.dnsalias.org/~strombrg/Backup.remote.html On Sun, Jul 18, 2021 at 11:13 PM Lisa via rsync wrote: > I would like some feedback about the

Re: Syntax Question on 'rsyncd.conf' on Windows

This is probably more of a Cygwin question than an rsync question. On Cygwin, E: should show up automatically as /cygdrive/e You can test that by opening a Cygwin terminal and cd'ing to /cygdrive/e On Tue, Apr 6, 2021 at 1:32 PM Tim Evans via rsync wrote: > Cygwin distribution of rsync for

Re: '--address' option on client side.

les multicat, it doesn't seem to be able to handle multiple interfaces, > if I read the docs correctly. > Am I wrong? > harry > > On Fri, Mar 26, 2021 at 12:29 PM Dan Stromberg > wrote: > >> >> Hi Harry. Are you the person I worked with at UCI a bit? >> >> Anyw

Re: '--address' option on client side.

Hi Harry. Are you the person I worked with at UCI a bit? Anyway, you might consider trying mrsync; it's intended to do rsync over multicast. HTH. On Fri, Mar 26, 2021 at 12:22 PM Harry Mangalam via rsync < rsync@lists.samba.org> wrote: > Spent an hour trying to find the answer to this on the

Re: Rsync checksum issue

On Tue, Dec 15, 2020 at 3:25 AM Laurent B via rsync wrote: > Dear all, > > I'm encountering a problem with one of my backup. For some files, the > checksum calculation is failing leading to the following error : > It sounds a little like a bug, but perhaps if you share the command you're using

Re: Is there any way to restore/create hardlinks lost in incremental backups?

On Sun, Dec 13, 2020 at 11:59 AM Wayne Davison via rsync < rsync@lists.samba.org> wrote: > I should also mention that there are totally valid reasons why the dir > might be huge on day4. For instance, if someone changed the mode on the > files from 664 to 644 then the files cannot be hard-linked

Re: Is there any way to restore/create hardlinks lost in incremental backups?

Hi. Is it possible that, if day4 is consuming too much space, that day3 was an incomplete backup? The rsync wrapper I wrote goes to a little trouble to make sure that incomplete backups aren't allowed. It's called Backup.rsync, and can be found at:

Re: link-dest and batch-file

On Tue, Jun 26, 2018 at 12:02 PM, Дугин Сергей via rsync < rsync@lists.samba.org> wrote: > I am launching a cron bash script that does the following: > > Day 1 > /usr/bin/rsync -aH --link-dest /home/backuper/.BACKUP/009/2018-06-25 > root@192.168.1.103:/home/

GPL-compatible library implementing the rsync-over-ssh protocol?

Is there such a thing? I saw librsync, which appears to be the right algorithm, but not the protocol. And I saw the acrosync-library, which appears to be the protocol, but it's not GPL-compatible. Are there others? Thanks! -- Please use reply-all for most replies to avoid omitting the mailing

Re: Rsync between 2 datacenters not working

gt; > -Original Message- > From: Kevin Korb via rsync [mailto:rsync@lists.samba.org] > Sent: zondag 25 maart 2018 22:42 > To: rsync@lists.samba.org > Subject: Re: Rsync between 2 datacenters not working > > Note that if you do this you are stuck with --whole-file > >

Re: Rsync between 2 datacenters not working

You could try using an automounter, like autofs, in combination with sshfs. It'll be slower, possibly a lot slower, but it should be more reliable over an unreliable connection. I've been using: remote

Re: [Bug 13317] rsync returns success when target filesystem is full

On Mon, Mar 5, 2018 at 3:09 PM, just subscribed for rsync-qa from bugzilla via rsync wrote: > https://bugzilla.samba.org/show_bug.cgi?id=13317 > > --- Comment #6 from Rui DeSousa --- > (In reply to Rui DeSousa from comment #5) > > It looks like no

Re: [Bug 8512] rsync is slower than cp -- Reduce overhead to cp for transfer of large files

Why not enable Jumbo Frames? http://stromberg.dnsalias.org/~strombrg/jumbo.html For NFS, you can use http://stromberg.dnsalias.org/~strombrg/nfs-test.html to get some fast settings. The script could be modified to do CIFS I suppose. HTH -- Please use reply-all for most replies to avoid

Re: [Bug 10785] New: [PATCH] Add a flag to use numeric sort

On Tue, Aug 26, 2014 at 10:01 AM, samba-b...@samba.org wrote: https://bugzilla.samba.org/show_bug.cgi?id=10785 Summary: [PATCH] Add a flag to use numeric sort Product: rsync Version: 3.1.1 Platform: All OS/Version: All Status:

Re: increasing the write block size for high latency

On Tue, Jul 8, 2014 at 8:07 AM, Adam Edgar aed...@research.att.com wrote: It seems the issue is indeed in the ssh layer. scp has the same issue and some work has been done in “fixing” that: http://www.psc.edu/index.php/hpn-ssh From the papers abstract: SCP and the underlying SSH2

Re: Concern: rsync failing to find some attributes in a file transfer?

On Fri, Jul 25, 2014 at 11:35 PM, Kevin Korb k...@sanitarium.net wrote: I just moved my home partition to a new harddisk w/more space. Home Partition? Are we in 1995? Why would you have a partition mounted anywhere other than /boot ? That's a bit harsh, particularly considering that having

Re: Changing permissions on existing backup source

On Wed, Jan 1, 2014 at 10:09 AM, Charles Marcus cmar...@media-brokers.com wrote: If I change the permissions on the source maildirs, will this cause everything to be transferred again? Meaning, will rsync see everything as 'modified', thus creating a new copy of the entire mail store on the

Re: Changing permissions on existing backup source

First off, thanks much for your suggestions. On Thu, Jan 2, 2014 at 6:19 PM, Kevin Korb k...@sanitarium.net wrote: First, a new column for the old cp -al then rsync on top of it method that --link-dest mostly replaced. It is slower since all the hard links get made and then some get replaced

Re: rsync seems to overwhelm a failing hard disk

On Mon, Nov 25, 2013 at 1:41 PM, Chris Dennis cgden...@btinternet.comwrote: Hello rsync people Today I was recovering data from a beginning-to-fail external USB hard disk. I started with my usual 'rsync -av --ignore-errors source dest', and that was fine until it got to the first I/O

Re: collecting the differences between a local host and remote archive to third local location (a usb device)

I wonder if rdiff would do this nicely...? http://librsync.sourcefrog.net/doc/rdiff.html On Thu, Sep 27, 2012 at 12:33 AM, Benjamin Ward b...@forward.net.au wrote: Hi everyone, I'm trying to archive to a remote host but the size of the data to copy is prohibitively large to consider doing

Re: Time rsYnc Machine (tym)

: Dan Stromberg wrote: FWIW, it might be nice to add a hardlink detecting bloom filter to rsync at some point. This makes the process of detecting hardlinks less expensive. Another way to narrow down the field is to just look at st_nlink. What's a bloom filter? and how / why would

Re: Time rsYnc Machine (tym)

On Sat, Jul 28, 2012 at 3:48 AM, M. Carrasco c...@dragoman.org wrote: 3. Cron It can run properly run from cron as it is demonized. What's this about? I've never had problems running run of the mill scripts from cron, once the environment is adequately replicated. --hard-links is not used

Re: Time rsYnc Machine (tym)

Ah. Not being interactive is important for running in cron; I believe stdin will probably immediately EOF. But redirecting stdout and stderr is unnecessary - the output just goes to a cron e-mail with most cron's. Sometimes it's better to redirect to a file, but that's more of a user

Re: rsync hanging after working reliably for years on one of my shares

On Fri, Jul 13, 2012 at 1:21 PM, Rob J. Caskey rcas...@athenshousing.orgwrote: Howdy all, ** ** I’m using rsync in conjunction with backuppc and have been backing up this share without incident for almost 3 years and it has decided to go and hang on me. Other shares on the same

Re: rsync based on checksum only

On Sat, Jul 7, 2012 at 4:15 PM, Henri Shustak henri.shus...@gmail.comwrote: Wow! Thanks for making it so easy. I will try that asap. If you do not have any luck with the patched version of rsync there are various projects which spring to mind which offer this kind of functionality. For

Re: Rsync takes long time to finish

I've heard lots of good suggestions already - another thing that I've not seen mentioned is, upgrading your kernel may help. Somewhere shortly before kernel 3.0, pathname lookups got noticeably faster. You could also try an alternative filesystem like xfs. It's supposed to be pretty good at

Re: incremental backup to nas server

This is perhaps more a matter of which rsync wrapper you choose, than rsync itself. rsync provides just enough functionality to enable this kind of behavior, if your wrapper feeds it the right directories to work on. I used to use my Backup.rsync wrapper to get good resumption of interrupted

Re: Use rsync's checksums to deduplicate across backups

On Sun, Nov 6, 2011 at 2:29 PM, Dan Stromberg drsali...@gmail.com wrote: http://stromberg.dnsalias.org/~strombrg/backshift/documentation/comparison/index.html I've updated the above URL to include a comparison against Lessfs and git wrappers. The table has also become easier to navigate

Re: Use rsync's checksums to deduplicate across backups

On Sat, Nov 19, 2011 at 5:43 AM, Andrea Gelmini andrea.gelm...@gmail.comwrote: 2011/11/3 Alex Waite alexq...@gmail.com: Recently I learned that rsync does a checksum of every file transferred. I thought it might be interesting to record the path and checksum of each file in a table.

Re: Rsync with direct I/O

Direct I/O (assuming you mean O_DIRECT on open) can be a bit fiddly, but I doubt it's out of reach. The main difficulty is allocating a buffer with appropriate alignment. I put together a library to facilitate O_DIRECT I/O a while back: http://stromberg.dnsalias.org/~dstromberg/odirect/ odirect

Graphical rsync --link-dest frontends?

What are people's favorite rsync --link-dest frontends? Why? I personally want something based on GTK+ and/or HTML, with scheduling, but I don't necessarily want to limit the discussion to frontends with these attributes. Here's a list of some of them, but I doubt it's very complete:

O_DIRECT

It'd be pretty cool if rsync supported use of O_DIRECT on platforms that support it, with or without my odirect package: http://dcs.nac.uci.edu/~strombrg/odirect/ I say this because rsync is sometimes used to move a mountain of data, just once. So there's little point in rsync toasting one's

Re: Copy all local filesystems, but ignore NFS mounts?

On Wed, 03 Aug 2005 14:15:36 -0400, Keith Warno wrote: * [EMAIL PROTECTED] [02/08/2005 2008EDT]: What's the easiest way, with rsync, to back up all local filesystems to another host, ignoring anything on an NFS volume. For example, is there a way of giving a list of mount points but still

Re: Copy all local filesystems, but ignore NFS mounts?

On Wed, 03 Aug 2005 16:30:28 -0400, Keith Warno wrote: * [EMAIL PROTECTED] [03/08/2005 1531EDT]: On Wed, 03 Aug 2005 14:15:36 -0400, Keith Warno wrote: * [EMAIL PROTECTED] [02/08/2005 2008EDT]: What's the easiest way, with rsync, to back up all local filesystems to another host,

Re: Copy all local filesystems, but ignore NFS mounts?

On Wed, 03 Aug 2005 15:53:06 -0700, Wayne Davison wrote: On Wed, Aug 03, 2005 at 03:27:13PM -0700, Dan Stromberg wrote: skipping directory /. You didn't specify -r, so it's skipping all directories. See also -a. Doh! I know better than that. Dumb mistake. Thanks for the clear, patient

Copy all local filesystems, but ignore NFS mounts?

What's the easiest way, with rsync, to back up all local filesystems to another host, ignoring anything on an NFS volume. For example, is there a way of giving a list of mount points but still have --one-file-system work? Thanks! -- To unsubscribe or change options:

data recovery and rsync, among other things

I've been trying to extract some of my clients' data from a flakey lustre filesystem, and as a result, took some time to write up a page about pulling data out of a semi-crashy filesystem. The page includes a variety of ways of doing so, those ways including but not being limited to rsync and a

rsync - missing one or more directories?

I am running: + cd /oas + rsync -a --rsh=/usr/bin/ssh --rsync-path=/dcs/packages/gnu/bin/rsync --progress [EMAIL PROTECTED]:/oas . ...and I can see that it sometimes transfers some files, however, it appears to be consistently missing this directory: /oas/projects/pers/oct2004 What might

Re: rsync - missing one or more directories?

On Thu, 17 Mar 2005 13:04:03 -0800, Dan Stromberg wrote: I am running: + cd /oas + rsync -a --rsh=/usr/bin/ssh --rsync-path=/dcs/packages/gnu/bin/rsync --progress [EMAIL PROTECTED]:/oas . ...and I can see that it sometimes transfers some files, however, it appears to be consistently

Re: rsync creating 0-length files?

On Wed, 2004-12-01 at 01:04 -0600, John Van Essen wrote: On Tue, 30 Nov 2004, Dan Stromberg [EMAIL PROTECTED] wrote: We were doing a roughly 1 terabyte transfer, and upon running a python script to verify the integrity of the transfer, we discovered a small number of files that were 0

rsync creating 0-length files?

Is there any precedent for rsync creating 0 length files that should've had content in them? IE, has anyone ever seen this before? We were doing a roughly 1 terabyte transfer, and upon running a python script to verify the integrity of the transfer, we discovered a small number of files that

Re: rsync creating 0-length files?

On Tue, 2004-11-30 at 12:45 -0800, Wayne Davison wrote: On Tue, Nov 30, 2004 at 12:10:05PM -0800, Dan Stromberg wrote: Is there any precedent for rsync creating 0 length files that should've had content in them? Are you using 2.6.3? Older rsync versions did not check the return

Copying from one networked filesystem to another

What sorts of options might I give to rsync, to optimize transferring data from one networked filesystem to another? Transferring directly between the fileservers is not an option in this case, for technical reasons. What sorts of options (or system modifications) might I use to help rsync get

Re: Copying from one networked filesystem to another

On Tue, 2004-11-23 at 15:00 -0800, Dan Stromberg wrote: What sorts of options might I give to rsync, to optimize transferring data from one networked filesystem to another? Transferring directly between the fileservers is not an option in this case, for technical reasons. What sorts

8 bit filenames?

Are there any characters that can occur in filenames that will choke rsync? We're transferring lots of data, and some of our users' filenames appear to have their high bit set. I don't expect it to cope with filenames having a \0 or / in them (sometimes created over appletalk shares - strange

Re: rsync recopying files it shouldn't?

On Tue, 2004-10-26 at 16:15, Wayne Davison wrote: On Sun, Oct 24, 2004 at 10:46:01AM -0700, Dan Stromberg wrote: We're copying from 3 NFS mounts of 3 GFS volumes, into 1 NFS mount of 1 Lustre volume. Since you're merging 3 sources into one destination, unless all 3 sources have unique

rsync recopying files it shouldn't?

files)? Or could it mean that our source or destination filesystem(s) have data integrity problems? Thanks! -- Dan Stromberg DCS/NACS/UCI [EMAIL PROTECTED] -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs

Re: Progress bar

as a crude estimate). Such a pre-pass would slow down rsync, so it's not something that would be universally useful (probably only for smallish sets of files over a slow net connection). ..wayne.. -- Dan Stromberg DCS/NACS/UCI [EMAIL PROTECTED] #!/dcs/bin/python import select import sys

large blocksize for transferring files from one filesystem to another on the same disk?

double the throughput on an AIX system (1M vs 64K). Thanks. -- Dan Stromberg DCS/NACS/UCI [EMAIL PROTECTED] signature.asc Description: This is a digitally signed message part -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http

Re: rsync process never quits! BUG? PROBLEM?

are hung up in select()'s. Any ideas? -- Dan Stromberg DCS/NACS/UCI [EMAIL PROTECTED] signature.asc Description: This is a digitally signed message part -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart

Re: ssh - how to send password ?

the password can show up in ps or shell histories. -- Dan Stromberg DCS/NACS/UCI [EMAIL PROTECTED] signature.asc Description: This is a digitally signed message part -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs

Re: strip setuid/setgid bits on backup (was Re: small security-related rsync extension)

On Tue, Jul 16, 2002 at 10:50:03AM +1000, Martin Pool wrote: On 15 Jul 2002, Dan Stromberg [EMAIL PROTECTED] wrote: The issue was that demand paging would glitch from .nfs* for no good reason. That is an extremely unconvincing argument for changing rsync. Is it possible to just

Re: strip setuid/setgid bits on backup (was Re: small security-related rsync extension)

On Sat, Jul 13, 2002 at 10:22:29AM +1000, Martin Pool wrote: On 12 Jul 2002, Dan Stromberg [EMAIL PROTECTED] wrote: Because when we update, for example, bash, everbody's bash is going to die on them if we don't keep around backups (segfault as you demand page from a binary that has

Re: strip setuid/setgid bits on backup (was Re: small security-related rsync extension)

On Fri, Jul 12, 2002 at 02:59:11PM +1000, Martin Pool wrote: On 11 Jul 2002, Dan Stromberg [EMAIL PROTECTED] wrote: I don't get what you are doing. Where did these insecure suid root files come from in the first place? Have you ever read bugtraq on a regular basis? They're coming

Re: strip setuid/setgid bits on backup (was Re: small security-related rsync extension)

to add the dumptruck, palmpilot, and grapefruit spoon features. I want to remove the misfeature that throws broken glass in front of your own wheels. Is that so bad? -- Dan Stromberg UCI/NACS/DCS msg04549/pgp0.pgp Description: PGP signature

Re: strip setuid/setgid bits on backup (was Re: small security-related rsync extension)

On Tue, Jul 09, 2002 at 02:36:22PM -0700, jw schultz wrote: On Tue, Jul 09, 2002 at 11:03:25AM -0700, Dan Stromberg wrote: On Mon, Jul 08, 2002 at 02:04:57PM -0700, jw schultz wrote: The default behavior should not modify files. The general purpose is to have the copies be the same

Re: strip setuid/setgid bits on backup (was Re: small security-related rsync extension)

language design. -- Dan Stromberg UCI/NACS/DCS msg04518/pgp0.pgp Description: PGP signature

Re: strip setuid/setgid bits on backup (was Re: small security-related rsync extension)

. Having an insecure setuid root ~ file is a vastly larger problem than not being able to run some ~ file without minimal sysadmin intervention first. -- Dan Stromberg UCI/NACS/DCS msg04519/pgp0.pgp Description: PGP signature

small security-related rsync extension

$Wc_c fi exit 0 -- Dan Stromberg UCI/NACS/DCS msg04403/pgp0.pgp Description: PGP signature

Re: symlinks?

--force appears to take care of this. On Mon, Apr 22, 2002 at 06:23:08PM -0700, Dan Stromberg wrote: I'm trying to switch to using rsync for updating a huge software library containing binaries, text files, symlinks, and so on. We've been using something homegrown which I'm not that happy

symlinks?

) : File exists rsync: symlink packages/ctime-5.0 - ctime-5.0b3: File exists rsync error: partial transfer (code 23) at main.c(883) Is there a reasonably simple way around this? TIA. -- Dan Stromberg UCI/NACS/DCS -- To unsubscribe