Re: rsync support in authprogs - feedback requested

On Thu, 18 Feb 2021 12:22:33 -0500 Kevin Korb via rsync wrote: > You should both look into rrsync. It comes with rsync and is designed > to do exactly this. I'm not really interested in restricting rsync to particular directories. That seems to be what rrsync is for, although it's a little h

Re: rsync support in authprogs - feedback requested

On Wed, 17 Feb 2021 21:52:06 -0800 Bri Hatch via rsync wrote: > I recently added initial rsync support to authprogs. > I'd be very interested in feedback For some 15 years+ (?) I've had a /root/.ssh/authorized keys line that starts with: "no-pty,no-agent-forwarding,no-port-forwarding,no-user-

Re: Enabling easier contributions to rsync

On Tue, 26 May 2020 00:43:41 +0200 uxio prego via rsync wrote: > > On 25 May 2020, at 23:55, Wayne Davison via rsync > > wrote: > > > > I've decided to give hosting it on github a try, especially since > > there's been a lot of nice contributions lately. Hopefully this > > will make it easier

Re: [PATCH] Optimized assembler version of md5_process() for x86-64

On Sat, 23 May 2020 10:21:31 -0700 Wayne Davison via rsync wrote: > Adding optional support for openssl's crypto library is also a good > idea. There is also libressl to consider, if you're considering libraries. Karl Free Software: "You don't pay back, you pay forward." -- R

Re: Much improved speeds of rsync via SSH - something to consider

On Mon, 2 Apr 2018 15:05:14 +0200 Thanassis Tsiodras via rsync wrote: > I recently concluded a bug hunt to trace why my rsync-ing to an SBC > was much slower than the corresponding iperf3-reported speeds. To > give a concise summary of the situation, in slow wifi links using SSH > with ProxyComma

Re: Help - rsync runs from command line, fails from task scheduler, hangs at msg checking charset: UTF-8

On Sat, 29 Jul 2017 19:45:22 -0700 (PDT) leonv12 via rsync wrote: > I don't get why it runs from the command line but not from a > scheduled task. Any suggestions for a fix or a work-around? Talk to someone who knows about task scheduling and its failure modes? Karl Free Software: "You don't

Re: Help - rsync runs from command line, fails from task scheduler, hangs at msg checking charset: UTF-8

On Fri, 28 Jul 2017 14:47:02 -0700 (PDT) leonv12 via rsync wrote: >(18 args)*msg > checking charset: UTF-8*[sender] io timeout after 3000 seconds -- > exiting[sender] _exit_cleanup(code=30, file=io.c, line=195): > enteredrsync error: timeout in data send/receive (code 30) at > io.c(195) [sender=3

Re: [Bug 12819] [PATCH] sync() on receiving side for data consistency

On Fri, 16 Jun 2017 12:34:40 +0200 Ben RUBSON via rsync wrote: > > On 15 Jun 2017, at 19:29, Karl O. Pinc via rsync > > wrote: > > The problem is that the --server (and, especially, > > --daemon) documentation has gone away. Or at least > > left the man p

Re: [Bug 12819] [PATCH] sync() on receiving side for data consistency

On Thu, 15 Jun 2017 13:23:44 + just subscribed for rsync-qa from bugzilla via rsync wrote: > https://bugzilla.samba.org/show_bug.cgi?id=12819 > > --- Comment #7 from Ben RUBSON --- > Note that my patch simply adds a sync() just after recv_files(), so > one sync() per connection, not per wr

Re: How to detect hanging rsync from bash-script ?

On Mon, 10 Apr 2017 06:50:24 + (UTC) reiner otto via rsync wrote: > I am using rsync on an unreliable mobile WAN connection, which causes > rsync receiver to hang for long time, when connection is broken > during transfere. As the option "--timeout=" here does not hit for > me, is there a rec

Re: [Bug 12576] popt aliases allow users to bypass sudo argument restrictions

On Sun, 12 Feb 2017 02:18:35 + samba-b...@samba.org wrote: > https://bugzilla.samba.org/show_bug.cgi?id=12576 > > --- Comment #4 from Paul Donohue --- > That's an interesting solution, but it doesn't really work well for > my use case. I would like my users to be able to maintain their own

Re: Alternatives to rsync. Was: Huge directory tree: Get files to sync via tools like sysdig

On Fri, 10 Feb 2017 11:38:34 +0100 Thomas Güttler wrote: > Yes, I think rsync is coming to the edge of its capabilities here. I > guess a different strategy is needed. > > I see these alternatives to rsync: > > - Incremental Snapshots at block-level device is one of them. > - We get the app

Re: Huge directory tree: Get files to sync via tools like sysdig

On Fri, 10 Feb 2017 12:38:32 +1300 Henri Shustak wrote: > As Ben mentioned, ZFS snapshots is one possible approach. Another > approach is to have a faster storage system. I have seen considerable > speed improvements with rsync on similar data sets by say upgrading > the storage sub system. Anot

Re: Huge directory tree: Get files to sync via tools like sysdig

On Thu, 9 Feb 2017 14:43:57 +0100 Axel Kittenberger wrote: > > > > Not only that, but inotify is not guaranteed. (At least not on > > 3.16.0. Can't say regards later versions.) So you might miss some > > changes. > > > > Got any info on that? > > I noted that MOVE_FROM and MOVE_TO events a

Re: Huge directory tree: Get files to sync via tools like sysdig

On Thu, 9 Feb 2017 10:55:51 +0100 Axel Kittenberger wrote: > > Has someone experience with collecting the changed files > > with a third party tool which detects which files were changed? > > I don't know of sysdig but am the developer of Lsyncd which does > exactly that, collect file changes

Re: man page

Let me start with a reference to Nazis, just so we can reach the Godwin-point and be done with it. If somebody thinks they can improve the documentation let them submit a patch. This provides something concrete for people to comment on and the maintainers can make a decision as to whether they wa

Re: man page

On Wed, 27 Jul 2016 14:56:07 +0300 Marcus Fonzarelli wrote: > Therefore I'd like to suggest that the man page be changed to "which > consists of large documents and mail folders", or at least mention > another software such as LibreOffice. Submitting a patch would be a good start. Gotta make it

Re: Yet another filter question

On Wed, 4 May 2016 21:26:26 -0500 "Karl O. Pinc" wrote: > On Wed, 4 May 2016 21:09:44 -0400 > Kevin Korb wrote: > > > That wording from the man page makes almost no sense without the > > examples directly after it (and I have read it many times and know >

Re: Yet another filter question

On Wed, 4 May 2016 21:09:44 -0400 Kevin Korb wrote: > That wording from the man page makes almost no sense without the > examples directly after it (and I have read it many times and know > what it is saying). Makes sense to me. The only thing I'd change is to use "in a depth first fasion" inst

Re: Yet another filter question

On Wed, 4 May 2016 21:09:44 -0400 Kevin Korb wrote: > That wording from the man page makes almost no sense without the > examples directly after it (and I have read it many times and know > what it is saying). Makes sense to me. The only thing I'd change is to use "in a depth first fasion" inst

Re: [PATCH] Consider nanoseconds when quick-checking for unchanged files

On Wed, 20 Jan 2016 23:04:20 -0800 Wayne Davison wrote: > > The problem is that if you transfer from a filesystem that has > nanoseconds to one that does not support it, rsync would consider > most of the files to be constantly different, since the nanosecond > values would only match if the sour

Re: [Bug 11378] Please add a '--line-buffered' option to rsync to make logging/output more friendly with pipes/syslog/CI systems/etc.

On Sat, 04 Jul 2015 17:56:25 + samba-b...@samba.org wrote: > --- Comment #2 from Nathan Neulinger --- > Perhaps the naming is not correct on my suggested option (and I'll > admit, I completely missed the outbuf option) - unfortunately, outbuf > doesn't actually solve the problem. > > The go

Re: Downloading a great number of files from different rsync servers for good loadbalancing and high efficiency.

On Sat, 4 Apr 2015 15:21:21 +0800 Hongyi Zhao wrote: > I'm using Debian, I want to make a local repository which can let me > install packages more conveniently. Your solution will not work for mirroring debian since it does not do a 2-stage mirroring process. This is described in: https://www.

Re: [Bug 11013] New: [patch] Mention that privileges are dropped, when "use chroot" is enabled in rsyncd.conf manpage

On 12/16/2014 08:45:15 PM, Kevin Korb wrote: > Only root can chown. If rsync isn't running as root then it ignores > the --owner part of --archive. This also makes --numeric-ids inert. > > Simply put, if you aren't running as root then you can only create > files owned by your UID. Rsync knows

Re: Aw: Re: encrypted rsyncd - why was it never implemented?

On 12/03/2014 01:37:58 PM, Kevin Korb wrote: > As far as a backup provider goes I wouldn't expect them to use rsync > over SSL unless that were built into rsync in the future (and has > been > around long enough that most users would have it). > > I would expect them to either use rsync over ssh

Re: Using --fuzzy

>external USB drives (I have a notebook), so hard links won't work. > >Joe > >On 11/16/2014 07:38 PM, Karl O. Pinc wrote: >> On 11/16/2014 03:53:12 PM, Joe wrote: >>> I have a lot of files (and directories) (up to a few hundred at a >>> time) >>>

Re: Using --fuzzy

On 11/16/2014 03:53:12 PM, Joe wrote: > I have a lot of files (and directories) (up to a few hundred at a > time) > that I get from various sources. Some time after I get them (after > they > are already backed up), I often have to move them around and > normalize > their names. > > When I do thi

Re: Concern: rsync failing to find some attributes in a file transfer?

On 07/26/2014 03:34:23 PM, Kevin Korb wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > I wasn't objecting to the use of multiple file systems. I have a > bunch of them too. I was objecting to the use of partitions to > achieve multiple files systems. Logical volume management has be

Re: [Bug 10637] rsync --link-dest should break hard links when encountering "Too many links"

On 06/06/2014 12:08:46 AM, L. A. Walsh wrote: > samba-b...@samba.org wrote: > > > https://bugzilla.samba.org/show_bug.cgi?id=10637 > > > > --- Comment #1 from Karl O. Pinc 2014-05-28 19:05:04 > UTC --- > > Yum is also rsync happy. That's where our --link-

Re: Alternative rsync client for Windows

So long as your compiling a list of things to overcome to restore a MS Windows box you need to add the following: You'd have to have an (exact?) copy of the original hardware onto which you wish to restore. I try to stay away from Windows so am no expert, but I believe that the OEMs are the ones

Re: silent data corruption with rsync

On 03/11/2014 11:02:28 AM, Sig Pam wrote: > Hi everbody! > > I'm currently working in a project which has to copy huge amounts of > data from one storage to another. For a reason I cannot validate any > longer, there is a roumor that "rsync may silently corrupt data". > Personally, I don't believe

Re: Need hint for my question regarding the working of rsync.

necessarily 11/13/2013 01:04:29 PM, Kevin Korb wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > Is there a hard links limit? I have been in the 70-80 million range > on ext4 without a problem (other than performance which is why I > switched to ZFS for that use case). It's a per-file

Re: Need hint for my question regarding the working of rsync.

On 11/13/2013 12:03:21 PM, Kevin Korb wrote: > OK, in the case of using v3 with --link-dest and not --checksum most > of the initial activity on the sender would be doing calls to stat() > to index what is there. > > The receiving side would be doing 2x the stat() calls (you have 2 > --link-dest d

Re: Need hint for my question regarding the working of rsync.

On 11/12/2013 04:13:01 PM, Karl O. Pinc wrote: > On 11/12/2013 03:50:20 PM, Wayne Davison wrote > > > Yes, the receiver sends all the checksums that it generates at once > > > For really big files it would be interesting to amend this rule to > > one > > where

Re: Need hint for my question regarding the working of rsync.

On 11/12/2013 03:50:20 PM, Wayne Davison wrote > Yes, the receiver sends all the checksums that it generates at once > For really big files it would be interesting to amend this rule to > one > where the sending side waits only long enough for a certain number of > checksums to arrive before it

Re: contribute to rsync

On 05/19/2013 10:19:10 PM, garvit sharma wrote: > Hello All, > >Myself Garvit Sharma working in hyderabad central > university(HCU) at DCIS. I have started using rsync to sync the data > and i > found it very interesting. After using rsync a lot i realized to > contribute > to rsyn

Re: High Speed WAN Rsync now possible!!!

On 02/22/2013 09:54:03 AM, Erich Weiler wrote: > Folks- > > Just wanted to plug a totally awesome software package from a group I > know: UDR (UDT Enabled Rsync). > > For those not familiar with UDT, it is a low level network protocol > based on UDP that allows for high speed transfers over hig

Re: A --exclude-checksum option?

On 02/17/2013 01:59:02 PM, Wayne Davison wrote: > On Tue, Feb 12, 2013 at 12:42 PM, Karl O. Pinc wrote: > > > It occurs to me that a handy solution might be to have an rsync > option, > > similar to the --exclude option, which would allow checksumming to > happen &g

Re: A --exclude-checksum option?

l fix it and it won't be corrupted. Regardless of whether the backup side fs is broken. (Presumably the backup side hardware/ fs will be fixed quickly.) And I don't care that --checksum means that the rsync takes longer once a week. Sounds like you're leaning toward "it'

A --exclude-checksum option?

Hi, I use rsync with hardlinks for backup, once a week doing checksums to ensure there's no filesystem corruption in the backed-up data. I also use tmpwatch, or something similar, to clean up /tmp, it removes files that have not been accessed recently. (atime older than some configured limit).

Re: Is there a howto/tutorial on backups/rsync that covers the use of hard and soft links?

Here's my rsync based backup system. http://wikisend.com/download/377440/rsync_backup-0.26.tar.gz It's an rsync based backup system utilizing hard links to reduce storage requirements. It supports both push and pull. It uses public keys with ssh for the transport. It works and I've used it for

Re: Re: rsyncssl

On 01/28/2013 04:24:13 AM, devz...@web.de wrote: > >The only place that an SSL would make some sense, is if you are > going > to do it to/from an rsync daemon, > > yes, exactly. > > > >but then how would that be "better" than a ssh-only account with > keys/etc. only allowing the rsync to

Re: Is there a howto/tutorial on backups/rsync that covers the use of hard and soft links?

On 01/23/2013 02:15:06 AM, Voelker, Bernhard wrote: > Kevin Korb wrote: > > On 01/22/13 18:12, Kevin Korb wrote: > > > That is the old way that pre-dates --link-dest. Instead of cp - > al > > > > daily.02 daily.01 you can do a mkdir daily.01 then an rsync ... > > > --link-dest=../daily.02 daily.

Re: Does rsync need a --ignore-unreadable-files option?

On 01/11/2013 09:08:52 AM, Allen Supynuk wrote: > > While you're at it, why not a more generic > > --ignore-errno foo > > where foo is (a defined subset of) the Posix error numbers > > listed in errno(3). The names, alternately, could be > > used. > > > > So your application would use > > > > --ig

Re: Does rsync need a --ignore-unreadable-files option?

Hi, On 01/10/2013 08:07:18 PM, Allen Supynuk wrote: > It strikes me though, that what I really want is an rsync option > '--ignore-files-with-no-read-perms'. While you're at it, why not a more generic --ignore-errno foo where foo is (a defined subset of) the Posix error numbers listed in errno(

Re: rsync in conjunction with the --link-dest option does not output deleted files

On 01/08/2013 07:17:15 AM, Kevin Korb wrote: > You might find this useful: > http://sanitarium.net/unix_stuff/rspaghetti_backup/diff_backup.pl.txt > It is a script I wrote that diffs 2 backup directories and will tell > you what is missing between them. There's always some simple shell (bash) scr

Re: vanilla rsync 3.0.9 hangs after transferring ~2000 files

On 11/08/2012 05:42:08 PM, Christian Iversen wrote: > > It's vanilla ext4, with: > > $ df -i /data > > FilesystemInodes IUsed IFree IUse% Mounted on > /dev/mapper/storage-data > 16384000 1475262 14908738 10% /data > > and: > > $ file /dev/mapper/storage-

Re: vanilla rsync 3.0.9 hangs after transferring ~2000 files

On 11/02/2012 08:33:13 AM, Christian Iversen wrote: > Hello rsync folks > However, 1 server is giving me a lot of trouble. It has a directory > with (currently) 734088 files in it, and every time I try to backup > this dir, rsync hangs after transferring roughly 2000 files. Since it might be fil

Re: vanilla rsync 3.0.9 hangs after transferring ~2000 files

On 11/02/2012 08:33:13 AM, Christian Iversen wrote: > Hello rsync folks > > Do you have any ideas what the problem might be? Or how I can help > debug it? I have no idea regards the problem, but it never hurts to do a tcpdump (-w, possibly -s) and look at what's on the wire. Perhaps you can fi

Re: rsync has big problem with leaving lots of empty dirs around...

On 10/19/2012 03:39:18 PM, Linda Walsh wrote: > Karl O. Pinc wrote: > > If you want help with your problem you're going to have > > to describe exactly what you're doing; what command > > you're running, what version of rsync, and so forth. > --- > >

Re: rsync has big problem with leaving lots of empty dirs around...

If you want help with your problem you're going to have to describe exactly what you're doing; what command you're running, what version of rsync, and so forth. See: http://www.catb.org/esr/faqs/smart-questions.html On 10/17/2012 04:43:18 AM, Linda Walsh wrote: > > > > > > The volume I am di

Re: Problem rsyncing 450GB file to my NAS: 'connection unexpectedly closed'

On 10/12/2012 01:06:03 PM, Justin T Pryzby wrote: > It sounds like a daemon may be timing out; is there a timeout > specified in rsyncd.conf? Is there a remote logfile with any useful > content? I've gotten a 12 error code when a lame firewall broke the connection. Karl Free Software: "You d

Re: rsync hashing / collision handling?

On 10/02/2012 03:58:04 PM, Andrew Pennebaker wrote: > Have recent versions of rsync considered using a more robust > implementation > of file change monitoring, like Dropbox > does? The above link says Dropbox does a "binary diff". How are you proposing to do a

Is --sparse suitable for general purpose use?

Hi, I'm using rsync with --link-dest to do backups. I don't have any sparse files, but someday I might. Should I be using --sparse? I notice that -S is not implied by -a. This makes me suspicious that --sparse is not (yet?) suitable for general purpose use. There also seem to be outstanding bug

Re: BDsync: Block Device sync

On 06/27/2012 08:48:30 AM, Karl O. Pinc wrote: > On 06/27/2012 06:51:29 AM, Rolf Fokkens wrote: > > > After having wrestled with rsync and several patches I found a > > solution to > > synchronize block devices: BDsync. > > > > Bdsync can be used to synchron

Re: BDsync: Block Device sync

On 06/27/2012 06:51:29 AM, Rolf Fokkens wrote: > After having wrestled with rsync and several patches I found a > solution to > synchronize block devices: BDsync. > > Bdsync can be used to synchronize block devices over a network. It > generates > a "binary diff" in an efficient way by comparing

Re: Hlink node data for 2282618 already has path=...

On 06/24/2012 10:40:11 AM, Brian J. Murrell wrote: > So, like a lot of people, I am using --link-dest to do backups. My > backup target is ext4 so with a hard link limit of 64K. I do end up > with trying to create too many links at some point though and get the > following sequence of events: >

Re: Using rsync to mirror directories where root owns file, using non-root user to initiate session

On 06/20/2012 10:40:57 PM, Kevin Korb wrote: > On 06/20/12 21:53, Karl O. Pinc wrote: > > On 06/20/2012 05:29:09 PM, Kevin Korb wrote: > > Somehow or another you need root access on the remote side in order > > to properly set permissions. > Not permissions, own

Re: Using rsync to mirror directories where root owns file, using non-root user to initiate session

On 06/20/2012 05:29:09 PM, Kevin Korb wrote: > http://www.sanitarium.net/rsyncfaq/#sudo Along these lines... Somehow or another you need root access on the remote side in order to properly set permissions. You can use ssh public keys to invoke a rsync daemon. In /root/.ssh/authorized_keys you put

Re: Getting messages "failed to stat" only in cron execution

> > rsync: recv_generator: failed to stat > > "/mnt/tmp/backup/museum/documenten/secretariaat/vrijwilligers/ > medewerkersboek > > en poster/losse foto's medewerkers/oude foto's medewerkers die we > willen > > bewaren/wienk, ari\#303\#253l .jpg": Invalid or incomplete > multibyte > or > > wide >

Re: Rsync takes long time to finish

On 04/12/2012 04:36:44 PM, vijay patel wrote: > > We are running Kernel 2.6.18-308.1.1.el5 which is latest in RHEL 5.8 > on both the server. I think i might have to explore option of using > ext4. Before you do anything you want to figure out why it is slow so you can solve the real problem. vms

Re: Rsync takes long time to finish

On 04/12/2012 04:19:47 PM, Kevin Korb wrote: > Yes, you have the feature in your filesystem. Good. (Note that you will want to check the filesystems on both ends.) Karl Free Software: "You don't pay back, you pay forward." -- Robert A. Heinlein -- Please use reply-all f

Re: Rsync takes long time to finish

On 04/12/2012 03:28:18 PM, vijay patel wrote: > > Thanks friends. We are using Redhat Linux 5.8 on Production and > Disaster Recovery side. By drilling down we have found out it is > taking lot of time to check what has changed while data tranfer is > very fast. As i mentioned data in these fol

Re: Run rsync even not connected

On 04/11/2012 09:23:57 PM, Kevin Korb wrote: > Cron will allow for an rsync to be running in the background. > However, there are additional steps (such as a lock file) you should > take to make sure two don't end up running at the same time. You could also use "at" if you want something to run on

Re: link(2) EMLINK error behavior with --link-dest and --hard-links

Should I submit a bug? Will you take a patch? On 03/24/2012 11:11:47 PM, Karl O. Pinc wrote: > Hi, > > I'm having a problem using --link-dest and --hard-links when > the fs hits the hard link limit (link(2) returns EMLINK). > > Using rsync 3.0.7 an error is thrown and t

link(2) EMLINK error behavior with --link-dest and --hard-links

Hi, I'm having a problem using --link-dest and --hard-links when the fs hits the hard link limit (link(2) returns EMLINK). Using rsync 3.0.7 an error is thrown and the target file is not created. Glancing at git head it _looks_ like things could now be a little nicer. Perhaps the target file is