Re: Removing folder at destination

[Fabian Cenedese via rsync]

>Why not exclude 2_ASSETS from the first run, and then do a second run to
>rsync the _contents_ of 2_ASSETS to the target 01_us_eng directory?
>
>Add /01_us_eng/2_ASSETS/ to the exclude.txt file.
>
>rsync -rvz --exclude-from=exclude.txt source destination
>rsync -rvz source/01_us_eng/2_ASSETS/ destination/01_us_eng/
>
>(perhaps add some variation of the exclude.txt file if you're excluding
>stuff under the 2_ASSETS directory)
>
>> My humble question to samba.org is this, can I add an rsync command to move
>> folders 3 and 4 into 01... and delete the 2_ASSETS folder so I dont have to
>> manually do this 200 times each day?
>
>You can't use rsync to perform remote rename operations, which is what
>you're essentially asking for.

You can execute remote commands using ssh/rsh. If the names
stay the same then you can add the rename commands to a script
file being executed after the rsync run.

bye  Fabi


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync many processes and slow backup

[Fabian Cenedese via rsync]

At 10:56 12.07.2019, Hannes Hutmacher via rsync wrote:

>Hi all! :-)
> 
>I have a small rsync script to sync my data to a usb-disk. It works fine, when 
>I start it in console. I get 3 rsync processes (look in top) and the backup 
>takes ~25 min. But, when I add the script in cron to start it at 1am at night 
>it takes 7 - 9 hours and I see up to 180 processes. When I look in top I see a 
>hight load of 60 - 80 and 40 - 60 waits. Why? Can someone explain why it takes 
>so long when it starts with cron?
> 
>This is my rsync command:  rsync -azc --delete "$QUELLORDNER" "$ZIELORDNER" 
>This is the entry in cron (crontab -e): * 2 * * * /root/backupscript/backup.sh

I'd say this starts rsync on every minute between 2:00 and 3:00.
You should change that to e.g. 0 2 ...

>Data to sync: 18 Gb, 185.000 files.
> 
>When I look in the log files I see errors like this: 
> 
>rsync: rename "/media/usb/sicherung/var/lib/fail2ban/.fail2ban.sqlite3.JCzY1c" 
>-> "var/lib/fail2ban/fail2ban.sqlite3": No such file or directory (2) 

That's probably because several rsync instances are stomping
on each other's feet.

bye  Fabi


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Exclude and delete or not delete

[Fabian Cenedese via rsync]

>You need to switch to filters where you have the additional options
>hide, show, risk, and protect.  See the filter rules section of man rsync.

Thanks for the hint. Though I've been using rsync for years this part has
somehow escaped me. I've read up on the filters and they do what I
need. The docs could use a few samples but I did get it to work.

Thanks

bye  Fabi


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Exclude and delete or not delete

[Fabian Cenedese via rsync]

>> I'm looking for a way to exclude and delete some items but not delete
>> other items. Use case:
>> 
>> I'm on Windows where there are folders like recycle.bin and system
>> volume information on every drive. I can exclude them while backing
>> up my data drive. But I now also have other data that is excluded
>> and that I want to delete from my existing backup. However if I use
>> --delete-excluded then rsync tries to delete the (write protected)
>> system volume information from the destination disk.
>> 
>> Is there a way to get around this with only rsync parameters?
>> Do I have to move the destination into a directory to get rid of
>> the interfering drive root folders in the sync process?
>
>Perhaps use read only attributes on the items that you don't want to
>allow to be deleted?
>
>Also maybe user .rsync-filter ?

The destination system folders are already write protected, that's
not the problem. But if rsync fails to delete those folders then it will
also skip deleting other data that I'd like to get rid of.

I don't think that filters will help me as they are just another way of
writing --include and --exclude but not what will happen with the
excluded items.

Thanks

bye  Fabi


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Exclude and delete or not delete

[Fabian Cenedese via rsync]

Hello

I'm looking for a way to exclude and delete some items but not delete
other items. Use case:

I'm on Windows where there are folders like recycle.bin and system
volume information on every drive. I can exclude them while backing
up my data drive. But I now also have other data that is excluded
and that I want to delete from my existing backup. However if I use
--delete-excluded then rsync tries to delete the (write protected)
system volume information from the destination disk.

Is there a way to get around this with only rsync parameters?
Do I have to move the destination into a directory to get rid of
the interfering drive root folders in the sync process?

Thanks

bye  Fabi


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync of big image files from virtual hosts takes lot of time

[Fabian Cenedese via rsync]

>This is the output with --stats for a 50GB image.
>Appended also the output from time.
>You see it's horrible slow.

Please give also the source and destination parameters from
your rsync command. These define whether it's a local sync
or not (for rsync), the hardware setup doesn't really matter.

bye  Fabi


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Invalid argument with version 3.1.3

[Fabian Cenedese via rsync]

At 23:39 29.09.2018, Daniel Feenberg via rsync wrote:
>Content-Transfer-Encoding: 7bit
>
>
>We reently upgraded our rsync client from Freebsd 10.2 to 11.2 which upgraded 
>our rsync from 3.0.6 protocol version 30 to 3.1.3 protocol version 31. The 
>server remains Freebsd 11.1 with rsync 3.1.2 protocol version 31. Now we get 
>numerous messages like this:
>
> rsync: readlink_stat("/disk/homedirs/nber/bstevens/public_html/old/test/press
> reaction/Foreign 
> Language/Los_empleados_no_se_casan,_s\#363lo_se_divorcian.PDF")
> failed: Invalid argument (22)
>
>Apparently the filename includes a non-ASCII character , such as windows might 
>do (via samba). My understanding is that Unix filenames can contain any 
>sequence of bytes except for the null byte, so why should this cause a 
>problem? Is there an rsync option to allow these files to transfer? Should I 
>go back to rsync 3.0.6?
>
>Rsync commands are (aproximately):
>
>  rsync --server -logDtprze.iLsfxC --delete . /disk/
>
>and
>
>  rsync -a -z -t --delete -e  ssh -l root host:/disk/

Two options that might help you:

--iconv=CONVERT_SPECrequest charset conversion of filenames

and if that doesn't help

--protocol=NUM  force an older protocol version to be used

bye  Fabi


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync to my external eSATA HD is crashing/freezing my system...

[Fabian Cenedese via rsync]

>Since the move to fc26, this command has caused the complete freeze/crash of 
>the system.  I tried it under multi-user non-graphic mode in case it was 
>something to do with new graphics systems (don't know why).  The output of the 
>above always stops at:
>sending incremental file list
>.d..t.. Music/FLACs/Various_Artists/Temple_Sampler/
>>f+ 
>>Music/FLACs/Various_Artists/Temple_Sampler/13-brian_mcneill-the_butterfly_chain.flac
>
>(total 183 bytes file size, always)
>
>By complete freeze/crash I mean:
>* hard-drive light indicates spinning, but no noise (activity)
>* unable to ssh in
>* unable to ping
>* in multi-user text, I'm first able to switch to another login console, but 
>then unable to progress past the login password; after switching to other 
>consoles, the original login screen attempt has gone blank other than a 
>blinking cursor top left
>* in graphic mode, there's no (inter)activity whatsoever possible - though 
>screen has desktop on it
>* the only apparent solution is a hard reset
>
>The external drive is a WD 750GB hard-drive in a StarTech.com external 
>hard-drive caddy connected via eSATA cable to an old Dell Vostro 3500.

Did you check the drive for SMART info/errors? Sounds to me like a
hardware problem.

bye  Fabi


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync time machine backup permissions

[Fabian Cenedese via rsync]

At 11:22 19.03.2018, Andre Althoff via rsync wrote:

>Dear rsync users,
>
>I'm trying to copy my Time Machine Backup from my old USB hard drive to a new 
>one. I got error messages, because rsync don't create group and other 
>permissions.
>
>Did I use the correct parameters? Who has an idea, what mistake?
>
>rsync: failed to set times on "/Volumes/G-DRIVE Thunderbolt 
>3/Backups.backupdb/iMac/Latest": Operation not permitted (1)

Looks to me more like you have not enough rights to set times etc.
So you'd need to run the script as root.

bye  Fabi


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Bug: rsync erroneously changes modification time

[Fabian Cenedese via rsync]

At 08:11 12.06.2017, max.power--- via rsync wrote:
>Content-Transfer-Encoding: 7bit
>Content-Disposition: inline
>
>How exactly does rsync determine that the copy has the incorrect  
>timestamp and not the source file?
>Does it assume that the copy must be incorrect or are there other  
>criteria that have to be considered?

rsync is only synching one way, it's always going from source to dest.
Therefore source is always 'correct' whereas dest will be adjusted to
match source. If you want a two way synching you'd need to call
rsync twice with different parameters (and switched source/dest)
or use a different tool.

bye  Fabi


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Aw: rsync show files changed during transfer - how?

[Fabian Cenedese]

At 17:00 19.12.2016, devz...@web.de wrote:

>>http://olstrans.sourceforge.net/release/OLS2000-rsync/OLS2000-rsync.html
>
> But the filename twice can happen under other circumstances; if you've 
>seen this happen, it's almost certainly because the file changed during 
>transfer. Rsync does no locking. Which means that: if you are modifying a file 
>while it's being transferred, then probably the checksum will fail and it'll 
>go round again. And if it goes around twice, and it still fails, then it 
>prints a message saying; Error, checksum failed, file changed during transfer? 
>And it's probably a file like a log file that's being constantly updated and 
>so the checksums didn't match because it's never going to be able to get it 
>exact; it means that what you've got on the other end is something which will 
>approximate some snapshot of the file, but because it's not doing any locking, 
>it can't guarantee that it's got a particular snapshot of the file, because 
>you can't have an atomic read of the whole file. [31m, 49s] 
>nope, this is wrong, there is no message "checksum failed"

Actually rsync does that (from a log on our server):
...
PUBLIC_SERVER_BACKUP/wiki/data/index/i10.idx
WARNING: PUBLIC_SERVER_BACKUP/wiki/data/index/i10.idx failed verification -- 
update retained (will try again).
...
PUBLIC_SERVER_BACKUP/wiki/data/index/i10.idx

So it did recognize a change and synched it a second time. Don't know what 
happens
if the file changes again though.

>to be a valuable backup solution, i`d like some networker a`like behaviour 
>which gives "file changed during save"
> 
>i wonder what`s so difficult for rsync to "look" at the timestamp or checksum 
>a second time after transfer and if it changed it should spit out a warning
> 
>isn`t this something which _could_ be implemented, if somebody is able to do 
>ist ?

Maybe it's a question of informational flags (-v, --progress, --stats etc).

bye  Fabi


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

rsync on Windows 10 Linux subsystem

[Fabian Cenedese]

Hi

Has anybody had the chance to test the new Linux subsystem
on Windows 10 and compare the rsync performance to e.g. a
Cygwin or MinGW compiled rsync?

bye  Fabi


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync on Windows questions

[Fabian Cenedese]

At 10:35 01.07.2016, Fabian Cenedese wrote:
>At 10:12 01.07.2016, Simon Hobson wrote:
>>> I'm having problems with the user rights. The backup works
>>> ok but after restoring some files I can't access them.
>>
>>That's to be expected.
>>NTFS has a very rich permissions system and rsync won't be capturing that. 
>>While it's a PITA, your best best is resetting permissions after restore. 
>>It's probably as 'simple' (for your own documents etc) to select the top 
>>level folder they are in, go to permissions, and have Windows re-apply them 
>>recursively - but as to the exact steps, I can't remember as Windows is 
>>something I avoid as much as I can get away with.
>
>I know how to do it (or at least managed to do it manually), but this backup
>and restore should also be usable for "normal" users that don't know about
>or don't want to fiddle with access rights.

I'm now trying the MinGW/MSYS rsync. That is older (3.0.8) but seems to
work much better concerning user rights.

Thanks

bye  Fabi


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync on Windows questions

[Fabian Cenedese]

At 10:12 01.07.2016, Simon Hobson wrote:
>> I'm having problems with the user rights. The backup works
>> ok but after restoring some files I can't access them.
>
>That's to be expected.
>NTFS has a very rich permissions system and rsync won't be capturing that. 
>While it's a PITA, your best best is resetting permissions after restore. It's 
>probably as 'simple' (for your own documents etc) to select the top level 
>folder they are in, go to permissions, and have Windows re-apply them 
>recursively - but as to the exact steps, I can't remember as Windows is 
>something I avoid as much as I can get away with.

I know how to do it (or at least managed to do it manually), but this backup
and restore should also be usable for "normal" users that don't know about
or don't want to fiddle with access rights.

>Overall, while rsync is a good tool for many things, for backups in the 
>Windows world it's probably not a good fit except for your own documents where 
>it's "easy" to reset file permissions after a restore. Trying to do other than 
>documents (ie the system and applications) is pretty well guaranteed to lead 
>to security permissions screwed up to the point where it's not worth doing the 
>backup in the first place.

This is meant to copy local documents so they're not lost in case of a
locky attack. This is not a complete backup, just kind of a Windows poor
man's time-machine where the files on the server are not accessible to
viruses. It'd be perfect if I could just force rsync to run as the local user
instead of some linux/cygwin user. OK, the best thing would be a native
rsync for Windows :)

There are several tools out there that use a cygwin rsync and just offer
a GUI for easier handling. I was wondering if they don't have this problem
or if they compile a special rsync without the user rights stuff.

Thanks

bye  Fabi


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

rsync on Windows questions

[Fabian Cenedese]

Hi

I'm using a cygwin compiled rsync on Windows to backup
and restore files from my local Windows computer. I'm trying
several destination types:

- remote with rsync://
- Samba share \\Server\Share
- local drive

I'm having problems with the user rights. The backup works
ok but after restoring some files I can't access them. I've
read that this is because rsync runs as cygwin user and the
*ix rights are completely different than the Windows ACLs.
I've tried various combinations of -o -g -acl -p etc. But none
worked so I could access my files immediately. Some people
had success when adding in their cygwin /etc/fstab a mount
parameter noacl. However I don't have a full cygwin, only the
rsync binary and the necessary cygwin dlls. The only
parameter that helped somehow was

--chmod=a=rwx,Da+x

Is there no other possibility? Can I maybe tell cygwin to use
a custom fstab from my installation directory? How do other
people use rsync on Windows?

And a second question: Do --copy-dest or --link-dest
help on local targets? Or does this create even
more stress on the disk?

Thanks

bye  Fabi



-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync failure on corrupted source data

[Fabian Cenedese]

At 05:55 28.06.2016, Paul J. Durack wrote:

>I am trying to copy data from a partially corrupt backup (time machine) disk 
>on a os x system to another.
>And am getting the following error:
>
>Backups.backupdb/durack1ml/2016-02-10-091749/durack1ml_hdd/Applications/Adobe 
>Media Encoder CC 2015/Adobe Media Encoder CC 
>2015.app/Contents/Resources/pdfl/CMaps/ETen-B5-UCS2
>rsync: writefd_unbuffered failed to write 14550 bytes [sender]: Broken pipe 
>(32)
>rsync: write failed on 
>"/Volumes/durack1ml_bak/160405_1234/Backups.backupdb/durack1ml/2016-02-10-091749/durack1ml_hdd/Applications/Adobe
> Media Encoder CC 2015/Adobe Media Encoder CC 
>2015.app/Contents/Resources/pdfl/CMaps/ETen-B5-UCS2": Result too large (34)
>rsync error: error in file IO (code 11) at 
>/BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-47/rsync/receiver.c(268)
> [receiver=2.6.9]
>rsync: connection unexpectedly closed (14352612 bytes received so far) [sender]
>rsync error: error in rsync protocol data stream (code 12) at 
>/BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-47/rsync/io.c(453) 
>[sender=2.6.9]
>
>Is there a way to get rsync to continue when such an error is encountered?

Are you sure that the problem is with the corrupted source data? Because the
error happens on writing the data on the destination.

rsync: write failed on ... : Result too large (34)

However I don't know what that error means.

bye  Fabi


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Operation not supported (95)

[Fabian Cenedese]

>> In addition to what Kevin said, if you are rsync-ing to a SAMBA share, you
>> will be limited to what the SAMBA server supports and not all are
>> configured to enable ACL support.
>
>The 'backup' Samba share on the destination machine is configured with
>
>inherit acls = yes
>inherit permissions = yes

Why don't you start a rsync daemon on the destination and connect with
the rsync protocol? This circumvents Samba limitations and also gives
you the differential transfer algorithm. It's probably also faster because
only the necessary data has to be sent over the line.

bye  Fabi


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Delete remote folder

[Fabian Cenedese]

At 10:27 02.06.2016, Fabian Cenedese wrote:
>Content-Transfer-Encoding: 7bit
>
>Hi
>
>I'm trying to build a backup system based on rsync. It should
>work with different destinations (local, share, remote etc) that's
>why I'm looking for ways how every operation can be controlled
>from the client side.
>
>I want to create full backups and incremental backups. For the
>incremental backups I use --compare-dest lastfullbackup. This
>works nicely except that the whole directory tree is created
>even if no files are copied. This is mentioned several times
>on the Internet and the usual answer is that there's no way
>around it because of the way that rsync works. I don't mind
>that they are created if there's a (rsync-)way to delete them
>again afterwards, something like
>
>rsync -a --delete --prune-empty-dirs dest dest
>
>Unfortunately at least one place has to be local, doing a
>remote-remote is not possible.

So far I haven't found a solution to this problem. I've now seen
this release note for the last version 3.1.2:

- Don't create an empty backup dir for a transferred file that doesn't
  exist yet.

I'm wondering now if that would help in my case with --compare-dest.
But I don't have it compiled for Windows. Can anybody confirm if the
newest rsync does not create empty folders anymore?

Thanks

bye  Fabi


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Delete remote folder

[Fabian Cenedese]

At 13:22 02.06.2016, you wrote:
>Content-Transfer-Encoding: 7bit
>
>Fabian Cenedese <cened...@indel.ch> wrote:
>
>> This script is bash and also uses the "remote shell hacks" using SSH.
>> As I want to run it also from Windows I'm looking for a rsync solution.
>
>Assuming you have control of the server, can you do a bit of semaphore ?
>
>Eg, do your backup with rsync, then when it's complete, touch a file (or put 
>something in it) and rsync that to the server. The server then runs a process 
>that does some action.
>
>Eg, you touch a file "/etc/ivedoneabackup" and sync it, when the server sees 
>that, it runs a process which links/moves/whatever stuff around to suit.
>The client side can be scripted in whatever language is available on that 
>platform.

There are several ways to solve it on the server, from simple cron jobs
to do the cleanup to more sophisticated solutions. But what if I don't
have a server but e.g. just a USB disk? Then it has to be done from
the client. Of course now you have full access to the backups and can
do whatever you want. However that depends again on the operating
system and maybe even file system or so. I don't want to (but maybe
have to) implement various solutions depending on local/remote,
source Windows/Linux, destination Windows/Linux etc.

But thanks for your inputs, I may have to do some more thinking.

bye  Fabi


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Delete remote folder

[Fabian Cenedese]

At 10:52 02.06.2016, Hans-Christian Jehg wrote:
>Content-Transfer-Encoding: 7bit
>
>Try having a look at this, for inspiration.
>
>It does require the capability of hard-links on the remote end though... and 
>it does not resolve your "delete target directory" either, but it might save 
>you some fiddeling around :-)
>
>https://blog.interlinked.org/tutorials/rsync_time_machine.html

Thanks, I already saw that before. I looked for existing backup programs
but none really did what I wanted that's why I'm trying to do it myself.

This script is bash and also uses the "remote shell hacks" using SSH.
As I want to run it also from Windows I'm looking for a rsync solution.

bye  Fabi


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Delete remote folder

[Fabian Cenedese]

Hi

I'm trying to build a backup system based on rsync. It should
work with different destinations (local, share, remote etc) that's
why I'm looking for ways how every operation can be controlled
from the client side.

I want to create full backups and incremental backups. For the
incremental backups I use --compare-dest lastfullbackup. This
works nicely except that the whole directory tree is created
even if no files are copied. This is mentioned several times
on the Internet and the usual answer is that there's no way
around it because of the way that rsync works. I don't mind
that they are created if there's a (rsync-)way to delete them
again afterwards, something like

rsync -a --delete --prune-empty-dirs dest dest

Unfortunately at least one place has to be local, doing a
remote-remote is not possible.

The second problem is that I want to delete older backups.
I can use

rsync -a --delete empty_local_folder remote_folder_to_delete

to get rid of the contents. However the folder itself still remains.
So after a while I'd have a lot of empty folders.

As I want this to work on various systems I can't (and don't
want to) use remote shell hacks like "do_something && rsync ".
Also the policy of what and when to delete should be configurable
from the client.

Are there pure rsync ways to solve these two problems?

Thanks

bye  Fabi


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Combine exclude files

[Fabian Cenedese]

At 14:19 25.05.2016, N.J. van der Horn (Nico) wrote:
>Content-Transfer-Encoding: 8bit
>
>Hi Fabi,
>
> From the man-page, at the end of FILTER RULES:
>
>---
>Note also that the --filter, --include, and --exclude options take one 
>rule/pattern each.
>To add multiple ones, you can repeat the options on the command-line,
>use the merge-file syntax of the --filter option, or the 
>--include-from/--exclude-from options.
>---
>
>This implies that --exclude/include/filter options can be repeated on the 
>commandline.

I don't know. As I read this I interpret it that I can repeat single --exclude 
options,
and for several excludes I can use the --exclude-from option. But it doesn't say
anything about repeated --exclude-from options.

bye  Fabi


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Combine exclude files

[Fabian Cenedese]

Hi

I'd like to have a list of excluded items that I can always use
and combine them with job-specific excludes (or generally filters).

Is it possible to use --exclude-from more than once? Do I have
to combine the exclude files into one and give this to rsync?
Is there another possibility (that also works on Windows)?

Thanks

bye  Fabi


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Synch group with different gid

[Fabian Cenedese]

At 08:30 11.05.2016, Fabian Cenedese wrote:
>Content-Transfer-Encoding: 7bit
>
>Hello
>
>I have a problem while synching between two NAS boxes with Linux.
>First NAS: group staff=50 (3.0.3pre1)
>Second NAS: groups ftp=50, staff=500 (rsync 3.1.1)
>
>I synched with -rptgo, without --numeric-ids. The expected result
>was that the items on the second group would be staff as well
>even with a different gid. However it looks like the 50 was taken
>literally and now all staff items are ftp items.
>
>1. How could this happen? Is this expected, did I do something wrong?
>2. How can I now synch the ownership correctly?
>
>I tried with -a, with --super, but so far the ownership is not synched.

I can now synch the group by using --groupmap=50:staff. Just doing
--groupmap=ftp:staff or leaving it off doesn't change anything on the
receiving side. However then I don't understand the default behaviour.
Shouldn't that already have happened in the normal use case without
--numeric-ids and --groupmap?

Thanks

bye  Fabi


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Synch group with different gid

[Fabian Cenedese]

Hello

I have a problem while synching between two NAS boxes with Linux.
First NAS: group staff=50 (3.0.3pre1)
Second NAS: groups ftp=50, staff=500 (rsync 3.1.1)

I synched with -rptgo, without --numeric-ids. The expected result
was that the items on the second group would be staff as well
even with a different gid. However it looks like the 50 was taken
literally and now all staff items are ftp items.

1. How could this happen? Is this expected, did I do something wrong?
2. How can I now synch the ownership correctly?

I tried with -a, with --super, but so far the ownership is not synched.

Thanks

bye  Fabi


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: User controlled i/o block size?

[Fabian Cenedese]

At 01:33 12.04.2016, Greg Freemyer wrote:
>Content-Transfer-Encoding: 7bit
>
>I'm just doing a local copy:
>rsync -avp --progress  

Just as side information: In local copies all files are copied wholly,
the diff algorithm is not in effect. So if a file changes then it still is
copied completely (without --partial, --no-whole-file etc).

Second thing: From what I remember rsync does a lot of stat calls to
get every file's properties. This is more expensive on cygwin/Windows
than on linux directly. Rsync also uses processes/threads which are
easier/faster to create and switched to on linux than on Windows.

A Windows native implementation of rsync could run faster than the
original rsync with cygwin layer. Some time ago somebody announced
a new program using the rsync algorithm. But I never used it so I don't
know about the features or speed.

http://www.acrosync.com/windows.html

bye  Fabi


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Failed verification message

[Fabian Cenedese]

Hi

We synch between servers, some are NAS with RAIDs, some are
simple Linux servers. Sometimes I get messages like:

WARNING: path/to/file failed verification -- update retained (will try again).

What are the possible reasons for this message? I looked on
the Internet and found old posts about a problem with big files.
I do get this message with big files (GB) but also with small files
(<1MB). Of course if a file is changed while being synched then
a problem is to be expected. But most files are surely not written
at this moment. The error usually only happens once for one
file, the retry or a second synch run okay.

Does this mean that the file content is wrong? Is this a problem
of the connection? What else can be done to improve the reliability
of the backup?

Thanks

bye  Fabi


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync with overlay tree

[Fabian Cenedese]

>Since you say the number is small, I suspect that something simple like:
>
>  rsync -hia /path/to/sync/common remotehost:
>  rsync -hia --ignore-times /path/to/sync/groupN remotehost:
>
>will be close to optimal.
>
>Another option is to build an --exclude-from list for the files in the
>special group and apply it to the first rsync.  Then you can run the
>second rsync without the --ignore-times, which may result in faster
>overall throughput.

Another way could be to build the specific directories and in there
create links to the common files if there aren't too many. Then on
rsyncing you need to dereference the links. That way you only
have one directory (structure) which rsync can already handle.

bye  Fabi


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Slow backup

[Fabian Cenedese]

>>>  I have a large directory with about 0 Milillion files in it to backup by 
>>> network. Server and client both windows.
>>> 
>>>In order to make backup faster, I created many rsync intances with "-avvu" 
>>>options. It seems worked, and only take about 5 hours to complete backup 
>>>job. 
>>>When I restarted the client bat script, and expect very little time to 
>>>complete backup, but it still cost about 3 hours. 
>>>It seems that concurrence makes no effect, I can not figure out what 
>>>happend, the file checking is so slow?
>>>Does anybody encounted cases like this? Any help will be thankful.
>>
>>If this is just one directory on one hard disk then the disk head has to
>>move constantly from one place to another to another to another for
>>each process. It's possible that it will run even faster if you have
>>just one process. On the other hand Windows will cache file info
>>in RAM so a second run could be faster then the first one. This
>>depends one your RAM, how much have you got? Have you tried
>>with just one process?
>
>Thanks for your reply. I have tried one process, but have no improvement in 
>essence. So the question is
>why the file comparison consumes so much time? In my mind, building file list 
>may consumes some time,
>but comparison should very fast, what is the problem?

Please keep it on the mailing list. I'm not an expert, others might
know more than me.

rsync has to stat each file and directory. I don't know what exactly
takes much time. But maybe you can give more information about
your setup: What OS, what file system, how many files and dirs,
small/big files, connection speed, full command line with all
rsync options, RAM, CPU etc.

Are you using cwRsync? With the CygWin emulation? This
is known to be slower. But even then 5h seems slow.

Just as an example: Here's a log from a backup running on a
slow NAS to another NAS in the local LAN not doing much
as the files are already the same. But both are running Linux
or actually BusyBox:

Number of files: 3445179
Number of files transferred: 0
Total file size: 349083766943 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 71253503
File list generation time: 0.016 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 71953972
Total bytes received: 700464

sent 71953972 bytes  received 700464 bytes  66079.52 bytes/sec
total size is 349083766943  speedup is 4804.71

real18m18.760s
user0m55.883s
sys 2m10.016s

So over 3 million files in 20 minutes. What are your --stats?

bye  Fabi


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: why rsync update large directory so slow?

[Fabian Cenedese]

At 14:22 24.11.2015, =?gb18030?B?t+fQzg==?= wrote:
>Hi:
>  I have a large directory with about 0 Milillion files in it to backup by 
> network. Server and client both windows.
> 
>In order to make backup faster, I created many rsync intances with "-avvu" 
>options. It seems worked, and only take about 5 hours to complete backup job. 
>When I restarted the client bat script, and expect very little time to 
>complete backup, but it still cost about 3 hours. 
>It seems that concurrence makes no effect, I can not figure out what happend, 
>the file checking is so slow?
>Does anybody encounted cases like this? Any help will be thankful.

If this is just one directory on one hard disk then the disk head has to
move constantly from one place to another to another to another for
each process. It's possible that it will run even faster if you have
just one process. On the other hand Windows will cache file info
in RAM so a second run could be faster then the first one. This
depends one your RAM, how much have you got? Have you tried
with just one process?

bye  Fabi


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: cut-off time for rsync ?

[Fabian Cenedese]

At 10:32 30.06.2015, Dirk van Deun wrote:
Hi,

I used to rsync a /home with thousands of home directories every
night, although only a hundred or so would be used on a typical day,
and many of them have not been used for ages.  This became too large a
burden on the poor old destination server, so I switched to a script
that uses find -ctime -7 on the source to select recently used homes
first, and then rsyncs only those.  (A week being a more than good
enough safety margin in case something goes wrong occasionally.)

Doing it this way you can't delete files that have disappeared or been
renamed.

Is there a smarter way to do this, using rsync only ?  I would like to
use rsync with a cut-off time, saying if a file is older than this,
don't even bother checking it on the destination server (and the same
for directories -- but without ending a recursive traversal).  Now
I am traversing some directories twice on the source server to lighten
the burden on the destination server (first find, then rsync).

I would split up the tree into several sub trees and snyc them
normally, like /home/a* etc. You can then distribute the calls
over several days. If that is still too much then maybe to the
find call but then sync the whole user's home instead of just
the found files.

bye  Fabi

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Comparing FLAC header before syncing

[Fabian Cenedese]

At 02:24 02.12.2014, Mike Garey wrote:
Hi all, I'd like to modify rsync to add a flag to compare the MD5 signature of 
the unencoded audio data in the header of a FLAC file to determine whether or 
not to transfer a file. Â 

The reason being that I've got a large number of FLAC files, many of which are 
corrupted in the destination volume, but many of which are valid but the tags 
have been modified.  The sizes of both the source and destination files are 
the exact same, regardless of whether they're corrupted or not.  If I were to 
rsync all files based on the modification date, it would overwrite these valid 
FLAC files whose tags have been altered, which I want to prevent.  I only 
want to rsync new files and files whose MD5 signatures don't match.

If anyone could point me in the right direction of where in the source code I 
should concentrate my efforts to add this modification, it would help greatly.

You'd probably get a result faster with less problems (future updates) if you
separate your logic into a single program instead of incorporating it into 
rsync.
If you have a way of accessing both the source and dest then you can create
a list of files to sync and feed this to rsync with --files-from. Or you run 
your
script/program on the destination, check all files and create the list which
you can then transfer to the source and let rsync run. Or you have rsync run
on the destination if you have a server on or direct access to the source.

If you really want to change rsync you can probably look at -c (checksum)
which you may be able to adjust to use the header's checksum instead
of generating a file's checksum. But I don't know the source code.

bye  Fabi



-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Alternative rsync client for Windows

[Fabian Cenedese]

At 18:41 08. 04. 2014, Gilbert (Gang) Chen wrote:

Hi, there,


We're pleased to announce the open beta of Acrosync, a native rsync client 
 for Windows.  Key features include: 
* Easy install, no more dependency on cygwin 
* Simple GUI, one click to start the sync 
* sync over ssh for security, or directly with an rsync daemon 
* For ssh, no need to set up password-less login 
* Built-in file system monitor to upload changed files instantly (just 
 like Dropbox) 
* Support hourly incremental snapshot creation (just like Time Machine) 
* Can invoke Volume Shadow Copy Service to upload locked files
Do you have speed comparisons against cwrsync? Especially with many
small files, big directories etc. If bandwidth is limiting then this may not
make a big difference. However a (local) check if the most part is already
uptodate could be sped up.

Thanks

bye  Fabi

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Strangely fast run without error

[Fabian Cenedese]

Hi

I have a NAS with rsync 3.0.3pre1. I know that is not new,
but a NAS is not easy to update. This usually works fine.
I have set up jobs to backup various stuff to another NAS.

This is the statistics of one run. Though there were seen
files (file size, vmware images) it finished almost immediately
which lead to the funny transfer speed.

receiving incremental file list

Number of files: 125
Number of files transferred: 0
Total file size: 25013819983 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 2382
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 63
Total bytes received: 2425

sent 63 bytes  received 2425 bytes  ,')* bytes/sec
total size is 25013819983  speedup is 10053786.17

This was a one time only thing, afterwards it worked
fine again. Still might be interesting to know what it was
that rsync finished so quickly without an error.

Thanks

bye  Fabi

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync --delete not working (no wildcards)

[Fabian Cenedese]

At 22:02 28.01.2013 +, Stewart Jeacocke wrote:
Hi,

I'm trying to copy some directories with rsync and would like to
delete files from the destination if they no longer exist in the
source.

I'm using the following command:

rsync -v --archive --hard-links --delete --force
--filter=':/.rsync-filter' /opt/ /mnt/backup/opt/

/opt does not contain any files
the destination contains one file

stewart@attobeast:/etc$ ls /mnt/backup/opt/
test-file

after running the rsync command I would expect test-file to have been
deleted but it isn't. I'm sure I'm missing something obvious in the
man page that explains this behaviour but I can't figure it out. Can
anyone enlighten me?

I'd say you have to give the parent folder as source to detect
missing files:

rsync ... /opt /mnt/backup/

Note the missing backslash of /opt.

bye  Fabi

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Implicit --itemize

[Fabian Cenedese]

At 07:44 18.01.2011 -0800, you wrote:
In 5.2.0.9.1.20110118091956.04afb940@localhost, on 01/18/11
   at 09:32 AM, Fabian Cenedese cened...@indel.ch said:

Hi,

Is it possible that SSH -i key could also be scanned by rsync and
interpreted as --itemize?

That is what is happening.  You have

 -e $SSH

and your shell is interpreting this as multiple arguments after expansion. 
Depending on your shell, changing this to

 -e \$SSH\

might be sufficient to avoid the issue.

I use the same construct in other scripts without having this problem.
So I think the escaping is already correct. The main reason might be
the logfile command as Matt pointed out. I will try and see.

Thanks

bye  Fabi

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Implicit --itemize

[Fabian Cenedese]

Hi

From reading the man pages I couldn't find such a hint, but is
there a way that --itemize is implicitely activated e.g. with using
another parameter?

Is it possible that SSH -i key could also be scanned by rsync
and interpreted as --itemize?

Can it be switched off again with --no-i? That didn't seem to have
an effect.

I used these params in a script with rsync 3.0.3pre1 on a NAS.

MYRSYNC_ARGS=-rptgo --stats --modify-window=1 --timeout=1800 
--log-file=/tmp/rsbackup.log --rsync-path=/path/to/rsync --timeout=3600
SSH=ssh -l user -ax -i /path/to/key -o ClearAllForwardings=yes -o 
StrictHostKeyChecking=no

rsync $MYRSYNC_ARGS -l --delete -e $SSH --no-i $SOURCE/mod $DEST:$SOURCE/mod 

And that still results in:
2011/01/17 20:00:08 [3918] .d..t.. server/Directory/
2011/01/17 20:00:13 [3918] f..t.. server/Directory/File
etc

Thanks for any help. rsync works as expected, it just blows up the
logfile which makes it harder to see errors. The SSH part also works
or we shouldn't get a connection to the other server.

bye  Fabi

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: data deduplication

[Fabian Cenedese]

At 06:41 25.05.2010 -0400, Mag Gam wrote:
I know rsync can do many things but I was wondering if anyone is using
it for data deduplication on a large filesystem. I have a filesystem
which is about 2TB and I want to make sure I don't have the same data
in a different place of a filesystem. Is there an algorithm for that?

I think for just finding same files you might be better off with SHA, MD5
oder other hashes. Even if there are same hashes I'd check first before
removing one of them, or at least make a full compare once the hashes
match. I don't think rsync is up to the task unless you want e.g. merge
two slightly different trees and then delete the remainder. rsync could
help in that case but only if the file names match. If you want to find
same files with different names I think you can't do it with rsync.

bye  Fabi

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Where to download the rsync binaries for Windows but NOT the cwrsync version?

[Fabian Cenedese]

At 07:07 20.05.2010 -0700, juligo wrote:

BenXS wrote:
 
 When I go to page 
 
 http://rsync.samba.org/download.html
 
 then there is no download of the windows binaries except the cwRsync
 version.
 
 However there must be such a pure-rsync binary version for Windows because
 I have already an old, outdated rsync.exe v3.0.4
 
 So is there elsewhere a download site for the newest rsync for windows?

Hi, 
quickly I only found a german version. 
Have a look at: http://it-em.net/cms/front_content.php?idart=43
Somewhere you can click on Download von rsync.zip and you will get
rsync_3.0.6
I' m not sure if there is a 3.0.7 version for windows?

This version is also built with cygwin, that's why the example uses 
/cygdrive/...
There is no pure Windows binary of rsync as it relies on Posix functionality.
However there's no problem using the cygwin version, be it standalone in
cwrsync or with the full cygwin installation.

bye  Fabi

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: --timeout not honoured

[Fabian Cenedese]

At 14:19 16.12.2009 -0800, Wayne Davison wrote:
On Tue, Dec 15, 2009 at 11:41 PM, Fabian Cenedese 
mailto:cened...@indel.chcened...@indel.ch wrote:
I think I know now where my confusion comes from. The reported time is not
the actual time without action but the time since the start of the rsync 
command.


It shouldn't be, since it is reporting the time elapsed since the last input 
was received. Â  I'd suggest running an strace on the rsync and seeing what it 
is doing. Â I'd imagine that some system call does not return for an extended 
amount of time, and then, when rsync finally gets control back and checks for 
the elapsed time since the last received input, it is a really long amount.

I now have a trace with a timeout case (almost 300MB). What do I need to look
for? I don't see immediately what part lead to the timeout. A short snippet:

13:18:51 ioctl(1, TIOCGPGRP, [17723])   = 0
13:18:51 write(1,955908096  45%   44.22kB/s   ..., 41) = 41
13:18:51 select(5, NULL, [4], [4], {60, 0}) = 1 (out [4], left {60, 0})
13:18:51 write(4, \374\17\0\7, 4) = 4
...
13:18:52 select(5, NULL, [4], [4], {60, 0}) = 1 (out [4], left {60, 0})
13:18:52 write(4, 
qo\223\16\352\240\31z`\374;\350\203\374\2551\252\354\335..., 4092) = 4092
13:18:52 time(NULL) = 1261138732
13:18:52 select(5, NULL, [4], [4], {60, 0}) = 1 (out [4], left {60, 0})
13:18:52 write(4, \374\17\0\7, 4) = 4
13:18:52 time(NULL) = 1261138732
13:18:52 select(5, NULL, [4], [4], {60, 0}) = 1 (out [4], left {60, 0})
13:18:52 write(4, \367\344\356\325UUM3\273*17\7\256D\255\225x0\345^{\203..., 
4092) = 4092
13:18:52 time(NULL) = 1261138732
13:18:52 select(5, NULL, [4], [4], {60, 0}) = 0 (Timeout)
13:19:52 time(NULL) = 1261138792
13:19:52 write(2, io timeout after 9720 seconds --..., 40) = 40
13:19:52 write(2, \n, 1)  = 1
13:19:52 rt_sigaction(SIGUSR1, {SIG_IGN}, NULL, 8) = 0
13:19:52 rt_sigaction(SIGUSR2, {SIG_IGN}, NULL, 8) = 0

Should I trace with different flags?

Thanks

bye  Fabi

PS: I'll be away for two weeks so I can only give more info in the new year.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

--timeout not honoured

[Fabian Cenedese]

Hi

I already wrote about this problem half a year ago but didn't get
an answer: http://lists.samba.org/archive/rsync/2009-June/023412.html

I'm using rsync 3.0.3 on a NAS. In the parameter list I use --timeout=1800.
But still I often have rsync stall much longer than that.

Some examples:

2009/12/14 23:14:35 [8707] io timeout after 11670 seconds -- exiting
2009/12/14 23:14:35 [8707] rsync error: timeout in data send/receive (code 30) 
at io.c(239) [sender=3.0.3pre1]

2009/12/14 00:20:04 [19046] io timeout after 15485 seconds -- exiting
2009/12/14 00:20:04 [19046] rsync error: timeout in data send/receive (code 30) 
at io.c(239) [sender=3.0.3pre1]

2009/12/12 22:15:21 [29732] io timeout after 8082 seconds -- exiting
2009/12/12 22:15:21 [29732] rsync error: timeout in data send/receive (code 30) 
at io.c(239) [sender=3.0.3pre1]

2009/12/11 23:14:23 [30216] io timeout after 11613 seconds -- exiting
2009/12/11 23:14:23 [30216] rsync error: timeout in data send/receive (code 30) 
at io.c(239) [sender=3.0.3pre1]

I know that 3.0.3pre1 is not the newest version, but as this is running
on a NAS it's not that easy to update or compile a newer version.

However, why is it that rsync waits much longer than given in --timeout?
Shouldn't timeout work that way?

Thanks

bye  Fabi

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: --timeout not honoured

[Fabian Cenedese]

At 10:27 15.12.2009 +0100, Fabian Cenedese wrote:
Hi

I already wrote about this problem half a year ago but didn't get
an answer: http://lists.samba.org/archive/rsync/2009-June/023412.html

I'm using rsync 3.0.3 on a NAS. In the parameter list I use --timeout=1800.
But still I often have rsync stall much longer than that.

2009/12/14 23:14:35 [8707] io timeout after 11670 seconds -- exiting
2009/12/14 23:14:35 [8707] rsync error: timeout in data send/receive (code 30) 
at io.c(239) [sender=3.0.3pre1]

2009/12/14 00:20:04 [19046] io timeout after 15485 seconds -- exiting
2009/12/14 00:20:04 [19046] rsync error: timeout in data send/receive (code 
30) at io.c(239) [sender=3.0.3pre1]

However, why is it that rsync waits much longer than given in --timeout?
Shouldn't timeout work that way?

I think I know now where my confusion comes from. The reported time is not
the actual time without action but the time since the start of the rsync 
command.

2009/12/15 20:00:02 [4260] building file list
2009/12/15 20:00:03 [4260] .d..t.. vmware/FTPServer2/
2009/12/15 20:23:45 [4260] f.st.. vmware/FTPServer2/FTPServer2-s017.vmdk
2009/12/15 22:12:25 [4260] f+ vmware/FTPServer2/FTPServer2-s018.vmdk
2009/12/15 23:10:19 [4260] io timeout after 11372 seconds -- exiting
2009/12/15 23:10:19 [4260] rsync error: timeout in data send/receive (code 30) 
at io.c(239) [sender=3.0.3pre1]

11372 seconds is 189.something minutes, so 20:00+3h09 makes about the 23:10
timeout time. Same seems to be true for my previous examples, all started at 
20:00.

So --timeout does seem to be honoured, just the reported timeout time is a bit
misleading. If this is not to be changed maybe an explaining comment in the
docs might be a good idea.

Thanks

bye  Fabi

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: retrieve files without local dir compare

[Fabian Cenedese]

We receive meteorological data from a remote server to a local directory
(every 5 min).
If the data is here, it is imported by a special software, after the
import it will be deleted from that directory. The deleting can't be disabled.
Normally I would say, ok download all again, but we get 80GB data per day.

If rsync compares the local dir it will download all again, because it's empty.
So rsync has to know what is already downloaded, and only get the new
files WITHOUT the dir compare.
Does anybody know a way how to realize that?

One way: keep a local copy which rsync can update. Therefore only the new
or changed files will get transferred. Then from this (e.g. from the rsync 
output)
copy the new files into a separate folder where they will be imported from (and
deleted).

Another way: As Paul mentioned you first need to find out the files to copy,
e.g. have a remote script that gathers all new files into a textfile. Then you
first get this file and then feed it into rsync with --files-from.

I thought there was a way to tell rsync to only sync files from a specific
period (as cp can do) but I couldn't find it, maybe not possible.

bye  Fabi

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Limit rsync running time

[Fabian Cenedese]

Hi

I'd like to rsync a large amount of data over a slow connection,
but only during night hours. I couldn't find a parameter that limits
the time that rsync is running, only the timeout on idle time.
I guess the way to go would be to start rsync, get the process
ID and kill the process later on.

Has anybody already written a bash script that would do something
like that? Are there other ways? I don't want to kill all rsync
processes as there might be other syncs going on.

Thanks

bye  Fabi

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Limit rsync running time

[Fabian Cenedese]

At 15:14 17.09.2009 +0200, Paul Slootman wrote:
On Thu 17 Sep 2009, Fabian Cenedese wrote:
 
 Has anybody already written a bash script that would do something
 like that? Are there other ways? I don't want to kill all rsync
 processes as there might be other syncs going on.

There is for example a timeout package available in Debian:

Package: timeout
Description: run a command with a time limit
 timeout executes a command and imposes an elapsed time limit. When the time
 limit is reached, timeout sends a predefined signal to the target process.
Homepage: http://www.porcupine.org/forensics/tct.html

That would be what I need. But I have rsync running on a NAS with
BusyBox, so I'd need to cross-compile it. That's why I was looking
for a bash script.

If other people have the same need: I found a script on the net:
http://twoday.tuwien.ac.at/jo/stories/320762/

Thanks

bye  Fabi

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Status of running rsync

[Fabian Cenedese]

At 22:02 15.09.2009 +, Andrew Gideon wrote:

I find lsof very informative with respect to rsync's status.

That's a good one, that really helps.

Thanks

bye  Fabi

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Status of running rsync

[Fabian Cenedese]

Hi

Is there a way to get the status of a running rsync process? I believe
it has to be done by sending a signal, but I can't find it anymore. I'd
like to find out if a rsync process is hung or just transferring a lot of
data.
This is rsync 3.0.3(I think pre1) on a NAS with BusyBox.

Thanks

bye  Fabi

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Status of running rsync

[Fabian Cenedese]

At 03:04 15.09.2009 -0400, Matt McCutchen wrote:
On Tue, 2009-09-15 at 08:37 +0200, Fabian Cenedese wrote:
 Is there a way to get the status of a running rsync process?

AFAIK, rsync has never had such a feature.  One thing you can do is
temporarily attach strace.

Too bad. I don't know if strace would help on the NAS. I guess I'll just
give it some more time before I kill it. Then I can still restart it with -v.

 I believe
 it has to be done by sending a signal, but I can't find it anymore.

Maybe you're thinking of dd, which prints its status when it receives
SIGUSR1.

I don't know anymore what tool it was, just that I used it before.
Would be a nice addition to rsync though :)

Thanks

bye  Fabi

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Big timeout time

[Fabian Cenedese]

At 09:07 15.06.2009 +0200, Fabian Cenedese wrote:
Hi

I'm using rsync 3.0.3 on a NAS. In the parameter list I use --timeout=1800.
However sometimes I get very big timeout times like this one:

io timeout after 12220 seconds -- exiting
rsync error: timeout in data send/receive (code 30) at io.c(239) 
[sender=3.0.3pre1]

rsynd.conf on the receiving side only defines modules, no timing parameters.
How can that happen? What can I do to prevent this? As this is running on a
NAS it's not that easy to update rsync.

Does nobody know what to do about this? I had another error like that.
The rsync command was running for quite some time until it dropped out:

io timeout after 2598 seconds -- exiting
rsync error: timeout in data send/receive (code 30) at io.c(239) 
[sender=3.0.3pre1]

real279m16.896s
user62m48.660s
sys 2m18.113s

This command is part several rsync commands for a daily backup.
All commands sync to the same server, the other ones run without
problem.

Here is the full command (with some protection):
RSYNC_ARGS=-rptgo --stats --modify-window=1 --timeout=1800
RSYNC_ARGS=$RSYNC_ARGS -z --exclude=repo.*.svndmp.gz
SSH=ssh -l mylogin -p myport -ax -i /path/to/serverkey -o 
ClearAllForwardings=yes -o StrictHostKeyChecking=no
time $RSYNC_BIN $RSYNC_ARGS --delete -e $SSH -l $DATA/module 
$BACKUPSERVER:$DATA 21 | grep -v skipping non-regular file

The other commands look the same except for different modules. So if at all
it must be something with the data (big files, many files etc). But if I run the
same command on the console it finishes without problem. Of course it can
be a problem with the connection over the Internet itself, but that still leaves
the question with the big timeout times.

I'd be thankful for any hint on this as this prevents automatic backup from
completeing.

bye  Fabi

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Big timeout time

[Fabian Cenedese]

Hi

I'm using rsync 3.0.3 on a NAS. In the parameter list I use --timeout=1800.
However sometimes I get very big timeout times like this one:

io timeout after 12220 seconds -- exiting
rsync error: timeout in data send/receive (code 30) at io.c(239) 
[sender=3.0.3pre1]

rsynd.conf on the receiving side only defines modules, no timing parameters.
How can that happen? What can I do to prevent this? As this is running on a
NAS it's not that easy to update rsync.

Thanks

bye  Fabi

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

rsync+cygwin 1.5-1.7 performance

[Fabian Cenedese]

Hi

Has anybody ever made comparisons about the performance of
cygwin 1.7 compared to 1.5? I believe there were some changes
that should make it faster, but I don't know if they affect rsync too.

Thanks

bye  Fabi

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync Runs Out of Space Because of Temp File

[Fabian Cenedese]

At 10:19 08.04.2009 -0700, philvh wrote:

There are still problem to work out for example where data in the
destination are moved, and those data needs to be moved first before the
transfer of data take place.  This will ensure that data is not lost and
only the same space as the source is needed. This probably is the same as
doing a defragment of a drive with limited extra space.

You didn't say what kind of virtual machine it is, but with vmware you can
convert it to be split into 2GB chunks of data instead of one big file. Then
rsync won't have any problems as it syncs each file separately and 10GB
is enough to create a new 2GB file. Of course you first need another
50GB for the conversion...

bye  Fabi

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: best rsync for use with windows (client) and long file names

[Fabian Cenedese]

At 15:57 26.03.2009 -0700, dnk wrote:
Good day,

I have been doing some tests with rsync on windows (client) to backup  
to a linux server.

I started out with cwrsync, but ran into the dreaded File name too  
long (91) error. Apparently due to an issue with the cygwin portion  
of it. So I was then testing with a newer version of cygwin (1.7 beta)  
as the long file name issue was resolved.

In testing that with some large files (PST in this case), rsync seems  
to take over the whole file, and not just the differences.

The delta algorithm is only used if you access a remote server
(e.g. with server::dest/path). For local syncs it is usually faster
to just copy the whole file than to reconstruct it from the old
file plus differences. There are switches to ease this behaviour
a bit (like change dest in-file instead of copying it, append at
the end) but unless you know what you're doing you should keep
the default behaviour.

I also use cwrsync but the few cases where the path was too
long I just shortened the source. Of course that may not be
possible for you.

bye  Fabi

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: proposal to speed rsync with lots of files

[Fabian Cenedese]

At 07:58 06.03.2009 -0800, Wayne Davison wrote:
On Thu, Mar 05, 2009 at 03:27:50PM -0800, Peter Salameh wrote:
 My proposal is to first send a checksum of the file list for each
 directory.  If is found to be identical to the same checksum on the
 remote side then the list need not be sent for that directory!

My rZync source does something like that for directories:  it treats a
directory-list transfer like a file transfer.  That means that the
receiving side sends a set of checksums to the sending side telling it
what it's version of the directory looks like, and then the sender sends
a normal set of delta data that lets the receiver reconstruct the
sender's version of the directory (which it compares to its own).  One
potential drawback is having to deal with false checksum-matches (which
should be rare, but would require the dir data to be resent) I hadn't
optimized it for block size or (possibly) data order to make it more
efficient, but it is an interesting idea for speeding up a slow
connection.  I'm not sure if it would really help out that much for a
more modern, faster connection, because rsync sends the file-list data
at the same time as it is being scanned, and sometimes the scan is the
bottle-neck.

To find out whether the scanning or the transferring is the bottleneck,
would it be possible to give in the statistics a hint like what threads
needed to wait longer, what action took more time? Something that
would give a hint that e.g. enabling/disabling compression might give
a faster overall transfer. I don't know if this internal data can be collected
or if the trial-and-change method is the only way to do it.

Thanks

bye  Fabi

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Converting file system

[Fabian Cenedese]

At 14:35 04.03.2009 +0100, David de Lama wrote:
Hi @all!

I tested what happens with a file which is saved at a FAT32 partition and then 
this partition is converted to NTFS.
So first I transfered the file with rsync from the FAT32 partition to my Linux 
/home folder. Then I converted the FAT32 partition to NTFS.
After the convertation I transfered the file again to see what has changed 
(amount of data).
I was surprised Nothing has changed!!! rsync only transfered the checksums.

How can it be?
Are the files on another layer? So that the file system doen't affect the file?

What did you expect should have changed? It wouldn't be much worth if the
conversion changed the file data. If you put a new shelf in the cupboard the
cups will still be the same...

bye  Fabi

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: AW: Sync files not folders non recursively

[Fabian Cenedese]

I am now trying to find a solution to my problem for quite some time. I hope
you can help. What I am trying to do is to simply sync files with a special
suffix to a destination folder. So I dont want to transfer recursively but I
want to delete files from the destination which are not in the source
anymore.

Closest I have gotten is using this command:

/usr/bin/rsync -dnolptgvze --delete --delete-excluded --include='*.htm' --in
clude='*.html' --exclude='*' /SOURCE/ /DESTINATION/

BUT this deletes Files from folder /DESTINATION/A which are not present in
the Source. I dont want this to happen. Also it syncs folders to DESTINATION
but I only want to sync files not folders.

/usr/bin/rsync -dnolptgvze --delete --delete-excluded --include='*.htm' --in
clude='*.html' --exclude='*' /SOURCE/* /DESTINATION/

This command does not delete Files from folder /DESTINATION/A but in this
care --delete is not working - files which are not in the source anymore are
not deleted from destination.

I think you should use /SOURCE /DESTINATION (no trailing slashes).

bye  Fabi

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: AW: AW: Sync files not folders non recursively

[Fabian Cenedese]

At 15:35 04.03.2009 +0100, Ehlers, Kolja wrote:
Thanks for helping but using:

/usr/bin/rsync -dnolptgvze --delete --delete-excluded --include='*.htm' --in
clude='*.html' --exclude='*.*' /SOURCE /DESTINATION

is not syncing anything

Why do you use -d? You want to sync files not directories.

 -d, --dirs  transfer directories without recursing

If this is not the problem then the exclude might be too aggressive.
But I'm no expert on filter rules. (I guess the -e is an oversight). 

bye  Fabi

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Delete only run

[Fabian Cenedese]

 Is there a way to take advantage of incremental, even if it means
 running rsync twice?

You can use --ignore-existing --ignore-non-existing --del to perform an
incremental deletion run:

--existing, --ignore-non-existing
  This tells rsync to skip creating files (including directories) that
  do not exist yet on the destination.  If this  option  is  combined
  with the --ignore-existing option, no files will be updated (which
  can be useful if all you want to do is to delete extraneous files).

I'll give that a try. I have already read the manual when I was setting
up my backup, but it seems I should read it again more closely :)

Thanks

bye  Fabi

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Delete only run

[Fabian Cenedese]

Hi

I have a backup on a NAS that is quite full. So when I try to backup
changed stuff it may run out of space, but only because rsync may
try to write a new file before deleting the old one. That not only means
changed files in the same place but also moved files, so delete-before
wouldn't help.

Is there a possibility to have a complete rsync run that only deletes
files to make space for the new files being synched in a second run?
Could this be done with the batch mode and some file sorting?

Thanks

bye  Fabi

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Delete only run

[Fabian Cenedese]

Sorry Matt for mailing you personally.

At 03:58 08.01.2009 -0500, you wrote:
On Thu, 2009-01-08 at 09:42 +0100, Fabian Cenedese wrote:
 I have a backup on a NAS that is quite full. So when I try to backup
 changed stuff it may run out of space, but only because rsync may
 try to write a new file before deleting the old one. That not only means
 changed files in the same place but also moved files, so delete-before
 wouldn't help.

You may be misunderstanding --delete-before.  It makes rsync perform all
necessary deletions at the beginning of the run, and yours is precisely
the use case that motivated it.

Ack. Seems to be so. Sorry for that. Just a small thing:

http://samba.anu.edu.au/ftp/rsync/rsync.html
--delete-before receiver deletes before transfer (default)

So it should already be like that. However:

Because of this, the default delete mode when you specify --delete is now 
--delete-during when both ends of the connection are at least 3.0.0

The options summary is therefore a bit misleading.

The second thing:
Some options require rsync to know the full file list, so these options 
disable the incremental recursion mode. like --delete-before

Is there a way to take advantage of incremental, even if it means
running rsync twice?

Thanks

bye  Fabi

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync + cifs problem

[Fabian Cenedese]

At 11:31 20.11.2008 +0100, Christian Pinedo wrote:
I finally have installed cygwin in the windows 2003 machine with rsync,
ssh + cron in order to periodically launch the rsync bash script. I'm
launching the rsync script from the destination machine. This is a
vmware machine.

rsync -aqzb --backup-dir=../old_backups --ignore-errors --force --delete
  --delete-excluded -e ssh [EMAIL PROTECTED]:/dir/ /cygdrive/c/Documents\
and\ Settings/versia/Desktop/pruebas_backup/current

But I'm suffering the same problem, a lot of following errors:

rsync: mkstemp /cygdrive/c/Document and
Settings/versia/Desktop/pruebas_backup/current/dir/files/.1751_pruebas2_000101_2008-08-07_11:04:21_2008-08-07_11:04:36.mp3.uM0RoN
failed: No such file or directory (2)

Any idea?

I don't know if that is the reason and if so what you can do about it
but it's not allowed to create a folder/file with ':' int it. Try to rename
the source and see it that works better.

bye  Fabi


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: backup file to win98 system

[Fabian Cenedese]

Hi

Please stay on the list as there are more knowledgeable persons than me.

Hi...rsync working fine for my fedora 5 box to fedora 9 box.  Now i want to 
take back of my folder /home/rajiv to a windows 98 box...Shared a folder in 
win98 system for full access but do not know how to take backup...tried with 
the following but got error message
[EMAIL PROTECTED] ~]# rsync -aPrv /data/stock/ 
//http://192.168.1.75/dirhttp://192.168.1.75/dirhttp://192.168.1.75/dir192.168.1.75/dir
sending incremental file list
rsync: mkdir 
//http://192.168.1.75/dirhttp://192.168.1.75/dirhttp://192.168.1.75/dir192.168.1.75/dir
 failed: No such file or directory (2)
rsync error: error in file IO (code 11) at main.c(567) [receiver=3.0.2]
rsync: connection unexpectedly closed (9 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(635) 
[sender=3.0.2]

rsync -aPrv /home/rajiv/ 
//http://192.168.1.75/dir/http://192.168.1.75/dir/http://192.168.1.75/dir/192.168.1.75/dir/

ip 75 is my win98 systemis it possible to do this?

You need to mount your Win98 partition, then you can access it like:

rsync /source /mnt/Win98partition

If you don't want that you need to run rsync on Win98 as server.
Look out for cwrsync (rsync with cygwin). But I don't know if
or how good that works on Win98.

thanks for the quick reply.  However can you tell me how to open password 
protected user and password based folder (from a NAS)

SET USER='me'
SET RSYNC_PASSWORD='me1234'
rsync -aPrv /cygdrive/c/para1  /cygdrive/y/test

tried with the above and found not working..

Now you're talking about windows. I don't know how to authenticate
programmatically. I'd access the shared drive manually in explorer
so Windows will ask me for the password. Once access is granted
rsync shouldn't have any problems either.

What NAS are you using? Doesn't it have a builtin rsync daemon
running? Or maybe you could add one?

bye  Fabi


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: backup file to win98 system

[Fabian Cenedese]

At 14:35 17.09.2008 +0530, you wrote:
Hi...rsync working fine for my fedora 5 box to fedora 9 box.  Now i want to 
take back of my folder /home/rajiv to a windows 98 box...Shared a folder in 
win98 system for full access but do not know how to take backup...tried with 
the following but got error message
[EMAIL PROTECTED] ~]# rsync -aPrv /data/stock/ 
//http://192.168.1.75/dir192.168.1.75/dir
sending incremental file list
rsync: mkdir //http://192.168.1.75/dir192.168.1.75/dir failed: No such 
file or directory (2)
rsync error: error in file IO (code 11) at main.c(567) [receiver=3.0.2]
rsync: connection unexpectedly closed (9 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(635) 
[sender=3.0.2]

rsync -aPrv /home/rajiv/ //http://192.168.1.75/dir/192.168.1.75/dir/

ip 75 is my win98 systemis it possible to do this?

You need to mount your Win98 partition, then you can access it like:

rsync /source /mnt/Win98partition

If you don't want that you need to run rsync on Win98 as server.
Look out for cwrsync (rsync with cygwin). But I don't know if
or how good that works on Win98.

bye  Fabi


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync as a remote file system protocol?

[Fabian Cenedese]

At 12:07 10.09.2008 -0700, Adam J. Richter wrote:
Does anyone know of an existing facility to access a remote
rsync server as a virtual filesystem through gnome-vfs, FUSE, lufs, or
similar that supports listing the contents of directories?

In case anyone is curious, I want to monitor a bunch of
sourceforge directories for changes, which I used to do by the
gnomevfs-ftp back when ftp1.sourceforge.net at least allowed directory
listings.

Maybe you could also do something like 'svn --dry-run update' to
check for changes but for that you need a checkout of the project
you want to monitor.

bye  Fabi


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: where can I find complied rsync for windows

[Fabian Cenedese]

At 13:48 10.06.2008 -0700, Lopez David E wrote:
using rsync on solaris. but am in need of rsync client on windows.
can you point me to one?

Look for cwrsync.

bye  Fabi


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Status of --ignorcase option in main tree/build?

[Fabian Cenedese]

* If you use --delete-before or --delete-during, rsync will delete FOO
as extraneous and then copy foo.  The overall effect is to
(inefficiently) update the case of destination files, which may be
desirable.  If you change the case of a destination file by hand to work
around case sloppiness in a program (see
https://bugzilla.samba.org/show_bug.cgi?id=3444#c5 ) and don't want
rsync to reverse your change, you can use a protect filter to block the
individual deletion.

With --ignore-case, the patched version of rsync will leave FOO alone
regardless of the delete mode.  --ignore-case also makes filter matching
case-insensitive.

Is there a way that rsync will find different case files in destination and will
update the filename without copying the whole file? Kind of a special case of
the find existing patch. This is something that might be useful even on
case-sensitive filesystems as on linux. As only the case has changed
it may be a lot easier to implement than a full search for the file.

bye  Fabi


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync bandwidth usage

[Fabian Cenedese]

At 10:37 22.04.2008 +0100, Joao Ferreira gmail wrote:
Hello all,

I'm experiencing about 20MBit/s on a 100MBit/s ethernet connection when
rsync'ing a lager number on _new_ files.

I don't have much exeprience with rsync. I'dd just like to know if thi
is an acceptable bw usage and what could be the limiting factors... in
order to try to increase this speed.

I'm considering using 1000MBit/s  dedicated connections and so I'dd like
to know if I will or won't get faster operations, or if I will still
have this 20MBit bottleneck.

Furthermore... is there a way to ask rsync to be slower ?!?

How do you call rsync? Do you use zipping? What are the used machines
(CPU type/freq)? We had the case that rsyncing to a NAS was not that
fast because the used CPU couldn't hold up. Another reason might be
the used rsync protocol, what rsync version do you use?

bye  Fabi


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Check free space on destination

[Fabian Cenedese]

Hi

I use rsync to backup some data to an external NAS which has a
rsync daemon running. The problem is that I sometimes have too much
data to store, so the rsync fills up the whole NAS drive until that doesn't
react anymore as it has no space left for the working system.

Is there a possibility to tell rsync to leave a certain amount of space
on the destination? I couldn't find a parameter, either for client or
for daemon. Maybe it could be done by outputting a filelist and
then have a script check before executing every file if the available
space is big enough for that operation? Has anybody ever done
something like that?

I'm using Windows for the client, but I could also create a little
C-program in case the needed features for such a script exceed
the DOS-shell's capabilities (which is likely). If possible I'd prefer
a client side solution as I can't change the rsync installation on
the NAS. I can just get the available space over an smb-share.

Would such a check be a useful addition to rsync?

Thanks

bye  Fabi


-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

RE: accurate progress for UI

[Fabian Cenedese]

At 06:35 30.01.2008 -0500, Robert DuToit wrote:

On Jan 30, 2008, at 3:46 AM, Paul Slootman wrote:

Do note that disabling the incremental recursion will impact the
performance, esp. with a large number of files.

I did some tests (I am becoming the OSX rsync_3 benchmark guy!) and  
for 15GB Home folder, there was only a slight difference in  
performance but see that could grow considerably with larger numbers.  
Without incremental recursion, the initial file count seems to take  
forever, though it catches up a bit after that. The incremental  
backups are almost the same in speed at this size. The one option, I  
mentioned before, that does seem to affect performance is the osx- 
creation-dates patch. Oddly, it is much slower for the incremental  
backup scans (2+ times) than for the initial full copy, proportionally.

If you have a lot of files then rsync will build up the whole file list in
memory (without incremental). So the memory limit may be more
important than any performance hit. If you leave the decision to the
users you should add a note about this, that for many files they
won't have any other option and that the progress may be a little
off.

bye  Fabi


-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync iconv (Cygwin) (file has vanished)

[Fabian Cenedese]

At 13:23 27.01.2008 +0900, Brendan Grieve wrote:
Wayne Davison wrote:
On Thu, Jan 24, 2008 at 02:09:30PM +0900, Brendan Grieve wrote:
  
I get the following error on files that have russian cryllic letters: -
file has vanished: /cygdrive/D/Data_Tier1/Home/xxx/???  

See the prior discussions about how MS Windows is lying to rsync about
what the filenames are (giving it names that it can't use in an open()
call).  I have heard that there is an executable flag you can set to get
MS Windows to stop this, but I don't use MS Windows to know for sure.

Just to save people in the future searching, a link to a post that goes into a 
little detail about this problem is: -
http://lists.samba.org/archive/rsync/2005-May/012638.html

I use the hacked cygwin1.dll to work around it. I was hoping the iconv flag 
would fix this issue, but its definitely more to do with the interaction 
between MS Windows and Cygwin rather than anything to do with rsync.

This may also be of help (or at least worth a try):

http://lists.samba.org/archive/rsync/2007-September/018656.html

bye  Fabi


-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync cheatsheets

[Fabian Cenedese]

At 21:45 11.01.2008 +0100, David Tonhofer wrote:
Hello,

I have just put reviewed versions of the rsync cheatsheets which I posted a 
couple of years ago
(if I correctly keep track of time), right here:

http://public.m-plify.net/rsync_cheatsheets/

- Sheet one lists the possible combinations of SOURCE and DEST arguments that 
may be given to rsync.
- Sheet two lists the behaviour of rsync for different cases of SOURCE and 
DEST.

Look nice. In addition to what gets copied you could add cases on what gets
deleted as a deleted subitem gets deleted with src but not with src/ or
things like that.

bye  Fabi


-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: --delete not working - due to 200+G of files?

[Fabian Cenedese]

At 12:54 04.12.2007 -0800, Wayne Davison wrote:
On Thu, Nov 22, 2007 at 04:01:29PM -0500, Eric Praetzel wrote:
 ssh Solaris_box cd /home;rsync -az --force --delete * 
 Centos_Box::remote_mach
 ssh Solaris_box rsync -az --force --delete /home Centos_Box::remote_mach

Those are two very different things -- the first copies items from
inside the /home dir (and usually only non-dot names), and the latter
copies the /home directory itself, which will put all the copied items
an extra level deep (you probably wanted a trailing slash after /home).

But then rsync won't delete folders that disappeared in /home, like
/home/someuser. This gets only deleted if rsync starts a level above, right?

bye  Fabi


-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync hangs when accessing through SSH (Leopard OS X)

[Fabian Cenedese]

At 15:39 02.12.2007 -0500, Greg Loesch wrote:
To preface, I'm running Leopard OSX and am using rsync  version 2.6.9   
protocol version 29.

Whenever I attempt to sync (using rsync obviously) a folder on my  
local drive to my web server (netfirms.com is my web host) using SSH,  
the process hangs. I have to force kill it in order to continue using  
Terminal. I can connect to my server space using SSH just fine by  
itself, and I ran rsync to sync two folders locally just fine. Is  
there some special syntax I need to use when using rsync with SSH?  
I've searched google and the mailing lists, and I didn't find much.  
I'm fairly inexperienced with Unix, so I believe that I am the culprit  
of this problem :-)

Please look in the mailing list's archive, rsync stalls keep coming up.
e.g. http://lists.samba.org/archive/rsync/2007-November/019243.html
or http://lists.samba.org/archive/rsync/2007-October/018770.html

bye  Fabi


-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync hangs when accessing through SSH (Leopard OS X)

[Fabian Cenedese]

At 09:50 03.12.2007 +0100, Izidor Jerebic wrote:

On 3.12.2007, at 9:08, Fabian Cenedese wrote:

At 15:39 02.12.2007 -0500, Greg Loesch wrote:
To preface, I'm running Leopard OSX and am using rsync  version 2.6.9
protocol version 29.

Whenever I attempt to sync (using rsync obviously) a folder on my
local drive to my web server (netfirms.com is my web host) using SSH,
the process hangs. I have to force kill it in order to continue using
Terminal. I can connect to my server space using SSH just fine by
itself, and I ran rsync to sync two folders locally just fine. Is
there some special syntax I need to use when using rsync with SSH?
I've searched google and the mailing lists, and I didn't find much.
I'm fairly inexperienced with Unix, so I believe that I am the  
culprit
of this problem :-)

Please look in the mailing list's archive, rsync stalls keep coming  
up.
e.g. http://lists.samba.org/archive/rsync/2007-November/019243.html
or http://lists.samba.org/archive/rsync/2007-October/018770.html

This is Mac, so it has nothing to do with common stall problems  
(Windows and Cygwin). Except if web company uses Windows for servers  
(which I doubt very much).

If you read up in the mentioned mails you'll see that I also had a stall
on Linux with 2.6.9 and SSH and it helped me to use the then current
version from cvs. So I still recommend to use the new version if possible.

bye  Fabi


-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Syncing to multiple servers

[Fabian Cenedese]

At 14:46 27.11.2007 +0200, Pournaris Charalampos wrote:
Is there a way to send first to Server2 from Server1 and after the rsync is 
finished to start syncing from Server2 to Server3 extacly the same files with 
the same options ?

I tried the post-xfer exec and pre-xfer exec options in the rsyncd.conf file 
but there is no way to get the original sync folder (/home/./bigfile.big in 
the following example).

Example:

Server1: rsync -avzR /home/./bigfile.big  Server2::home --progress --delete
 Syncing ..
Sync done, start internal  (lan) sync: Server2:  rsync -avzR 
/home/./bigfile.big  Server3::home --progress --delete

Just thinking, I haven't tried this. You can specify the binary that is called
on the remote side if you go with SSH. Like that you could create a
script file that syncs from 2 to 3 and call that from 1 after you have finished
synching from 1 to 2. The script itself sits and runs on 2.
If 3 is a mirror of 2 you don't need exactly the same options as you used
for synching from 1 to 2, just create an exact copy (-a --delete etc).

bye   Fabi


-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync stalls

[Fabian Cenedese]

At 16:33 27.11.2007 -0500, Joe Demeny wrote:
I am trying to rsync a machine running CYGWIN_NT-5.2 server 1.5.24(0.156/4/2) 
to another which runs FreeBSD 6.2-STABLE; both with rsync 2.6.9.

I'm trying to pull the files from the Cygwin machine with:

/usr/local/bin/rsync -avz --delete --delete-excluded -e ssh 
[EMAIL PROTECTED]:/cygdrive/e/Shared /home/[host]

However, rsync stalls. This seems to be happening when there are some new 
large file(s) on [host] - or perhaps too many files? The total is about 30 GB 
and 55,000 files.

On the FreeBSD box the connection state is FIN_WAIT_1. On the Cygwin side the 
connection state is ESTABLISHED. The connections seem to be hanging around 
indefinitely.

If I push the files from the Cygwin machine, rsync runs fine and exits with 
no error.

One thing is a possible interaction problem with cygwin's pipes (I think). There
are many others that have a rsync stall on cygwin, but nobody has yet found
out exactly where the problem lies or how to fix it.
The other possibility is an error in rsync that was fixed early 2007. If 
possible
please try the actual development version which is in last testing stage 
(3.0.0).
This has helped me with a stall I had.

bye  Fabi


-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: monitore rsync

[Fabian Cenedese]

At 20:20 25.11.2007 +0100, Martin zhsg wrote:
Hi

We have one upload-server (client uploads file via FTP). As soon as a new file 
is on this server, it needs to be replicated to seven other servers. 
The file size is typically around 5-10MB and and the volume to replicate is 
around 200MB/day. 
For this purpose we plan to use rsync. My question: 

We want to monitore rsync and want to get a message by email as soon something 
is not running smoothly. 

Create a script that makes the various rsync calls and check the return
code (exit code), send a mail if something's wrong. You can even redirect
the output of rsync in a file and send this logfile as attachment, so you
already see what may be wrong.

bye  Fabi


-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync error when sending lot's of small files

[Fabian Cenedese]

At 11:57 15.11.2007 +0100, you wrote:
Hi All,

I have a problem when transferring files from a rsyncd.  This is my setup:

What is the error you get?

bye  Fabi


-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsyncing files that might change

[Fabian Cenedese]

At 23:09 31.10.2007 -0400, Matt McCutchen wrote:
On Thu, 2007-11-01 at 10:35 +1000, Franc Carter wrote:
 If am rsyncing a file and I have the the following sequence of events
 happen in
 the same second
 
1. rsync starts
2. rsync sends some chunk of data to the other end
3 a local process modifies the chunk that has just been sent

 So, my question - is this case a race condition in which a subsequent 
 run of rsync may miss that the file has been modified and hence skip
 it ?

Unfortunately, yes.

Shouldn't that be caught by the fact that the source file has a new
(or at least different) time stamp now?

bye   Fabi


-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync stops/hangs (from windows to debian)

[Fabian Cenedese]

At 01:26 05.10.2007 -0700, Miles Raymond wrote:
I'm not using rsync through ssh since this is on an internal network.  Would 
pipes still be used?

The only difference I can tell between my situation and Alain's is that my 
case the windows client is sending files instead of receiving.

Are there any suggestions for a work-around other than switch all the 
computers to *nix (which I'm in the process of doing anyway)?  I'm hoping for 
a more immediate solution...

You can try to build (or find) a cygwin version of the new 3.0.0 or the
cvs version. I also had stalls using the Linux 2.6.9 that were solved
using the newer rsync (so no cygwin involved). I don't know if a simple
bugfix was responsible for this or the new protocol 30. In the latter case
you need to have the new rsync on both ends or they use an earlier
protocol.

bye  Fabi


-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: why not a gui for rsync

[Fabian Cenedese]

At 00:04 01.10.2007 -0500, Robert wrote:
snip

Is there some works in the world trying to make a good gui for
rsync and to let all people using this very good method to make
a backup.
   
If somebody is interesting to create this gui, let me know

   --Suresh

Delta Copy: http://www.aboutmyip.com/AboutMyXApp/DeltaCopy.jsp

It has a gui already. No need to reinvent the wheel.

Delta Copy unfortunately only works in client/server mode. As I also
need local mode this is no option (though it looks nice). So me too
I still haven't found a simple, user-friendly GUI.

bye  Fabi


-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Problem with hard links

[Fabian Cenedese]

At 09:50 28.09.2007 +0200, limule pika wrote:
Hello,

I have a problem with rsync and hard links :

I have 1 folder : P, with 2 subfolders : S1 and S2

S2 contains a lot of hard links to file stored in folder S1.

   P : -S1
-S2 (S2 files hard links to S1 files) 

I would like to rsync the folder P to another computer, but each sub folder S1 
(110 Go) and S2 (10 Go + hard link to 100 Go of S1) contains thousands of 
thousands of files, and when i try to rsync the folder P i have an out of 
memory error. 

The command used is : rsync --recursive --hard-links -e ssh --stats --delete 
--links --perms --times

So i try to rsync the subfolder S1 and S2 in two rsync commands (with same 
argument as above), and then the hard links between S2 and S1 are not 
preserved. 

Is there a solution to keep the hard links between S2 and S1 when running two 
separated command ?

I don't know an answer to this, but if possible you can use an rsync from cvs.
The actual development version uses incremental file list (in remote mode
both rsync binaries have to support this). This should save you from the
memory problems and you can do it in one step.

bye  Fabi


-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: hebrew characters

[Fabian Cenedese]

At 18:01 25.09.2007 -0500, Robert wrote:
I'm trying to use rsync (version 2.6.9) to back up some files. However, a few 
of them contain Hebrew characters. While I can back them up, the filenames 
come through scrambled (I don't have Hebrew characters anymore). I've tried 
the -8 switch, but that didn't work. Any suggestions?

Background:

The files are created on Windows and saved on a samba share (v3.0.24). I can 
copy them around in Windows just fine. Konqueror, on the machine the files 
are physically on shows the file names properly. But when I copy them using 
rsync, the file names get scrambled (Hebrew characters only; English 
characters are fine).

I guess you're using cwrsync which uses the cygwin1.dll. The normal release
is not unicode aware and therefore has problems with non-ascii characters.
I also had this problem with German umlauts. There is a unicode-release
of the cygwin1.dll which you can copy over your existing cygwin1.dll.
In my case the problems went away. The characters still look strange when
looked at in Linux (different codepage, encoding...) but at least from the
samba share they're the same.

http://www.okisoft.co.jp/esc/utf8-cygwin/

bye  Fabi


-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: delta copies

[Fabian Cenedese]

Not necessarily.  Depending on how pg_dump works, it could be that
small changes to the database are resulting in unnecessarily large
changes to the dump.  Make sure you are using the uncompressed format
because most compression algorithms defeat the delta-transfer
algorithm almost completely.  Then you might take a look at two
consecutive dumps and check whether records common to both appear in
the same order in each dump.  (If pg_dump is dumping the records in a
different order each time, that would also defeat the delta-transfer
algorithm because no block of several consecutive records could be
matched.)

I was wondering if it would be possible to add a switch (probably coupled
to -v(+) ) that would report the number of matched blocks per file. Maybe
even with the offset of the block. That of course wouldn't help much in a
production environment, but it could help while setting up the whole
backup strategy. This would have helped the original poster finding
out if rsync is working correctly.

bye  Fabi


-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: delta copies

[Fabian Cenedese]

At 07:51 24.09.2007 -0400, Matt McCutchen wrote:
On 9/24/07, Fabian Cenedese [EMAIL PROTECTED] wrote:
 I was wondering if it would be possible to add a switch (probably coupled
 to -v(+) ) that would report the number of matched blocks per file. Maybe
 even with the offset of the block.

Rsync already lists all matched blocks by their offsets and lengths if
given -vvv.

Right, sorry about that. I didn't find it because the manual doesn't state
what info is given for more than 2 v's, only used for debugging :)

bye  Fabi


-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Data corruption check

[Fabian Cenedese]

At 08:12 19.09.2007 -0700, Keith Lofstrom wrote:
On 9/18/07, Fabian Cenedese [EMAIL PROTECTED] wrote:
 I was wondering what happens if a file that is regularly synched but
 seldom changes gets corrupted in the copy.

On Wed, Sep 19, 2007 at 09:23:28AM +0200, Fabian Cenedese wrote:
 I was asking because I'm responsible for our backups. The
 current solution with rsync works nicely. While the RAID storage
 also monitor the HD's SMART state I was still wondering
 about a way to detect otherwise unknown data corruption.

I run rsync inside of dirvish (www.dirvish.org) for automated
backups.  I also run osiris (osiris.shmoo.com) which scans for
modified files, both checking the metadata and a hash of the 
actual data, finding all changes relative to a database on a
central osiris server.

We already have a backup system that creates incremental backups
as well as mirroring the file server as well as the backups to a
second RAID. I think the osiris is a bit overkill for our application.
But thanks for the info.

bye  Fabi


-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Data corruption check

[Fabian Cenedese]

At 15:15 18.09.2007 -0400, Matt McCutchen wrote:
On 9/18/07, Fabian Cenedese [EMAIL PROTECTED] wrote:
 I was wondering what happens if a file that is regularly synched but
 seldom changes gets corrupted in the copy.

Are you referring to rsync writing corrupted data to the destination
file or a problem with the destination filesystem or disk causing the
file to read as data different from what was written?

I was thinking of any problem, even a transport error.

 There's also the -I, --ignore-times switch. If I use this but without -c
 what method is used for checking then? Or does -I imply -c?

-I rewrites the destination file no matter what, while -c computes its
MD4 or MD5 checksum first and then rewrites it only if its checksum
differs from that of the source file.  Either option gives the same
end result for the destination file.  However, they may have different
performance (-c uses more disk reading but potentially less disk
writing and slightly less network traffic), and -I logs more transfers
than -c and interferes with --link-dest.

Thanks for the explanations. That means that -l and -c are not
usable together as they contradict themselves, right?

I was asking because I'm responsible for our backups. The
current solution with rsync works nicely. While the RAID storage
also monitor the HD's SMART state I was still wondering
about a way to detect otherwise unknown data corruption.

I guess if I first made a normal rsync and then a rsync --dry-run -c
I could find file differences that shouldn't be (provided there
wasn't any real change otherwise, like in the middle of the night).
Of course that wouldn't tell me what side had changed, but still
something worth considering doing once a month or so...

Thanks for your help

bye  Fabi


-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Data corruption check

[Fabian Cenedese]

Hi

I was wondering what happens if a file that is regularly synched but
seldom changes gets corrupted in the copy. As it seldom (or never)
changes the mod time will always be the same. But if the content
changes (bit flip, bad HD...) will rsync get this and synch it again?

Would I need the -c (crc) flag for this to work? That of course slows
things down quite a bit. Is this the only way to ensure that the
contents are the same on both sides?

There's also the -I, --ignore-times switch. If I use this but without -c
what method is used for checking then? Or does -I imply -c?

Thanks

bye  Fabi


-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Win32 port?

[Fabian Cenedese]

Hi

Has anybody ever tried (and most probably failed) to port rsync to Win32?
I know there are versions built with cygwin and there are also other
Windows programs that include the rsync algorithm. But neither of them
make me really happy. The cygwin version is dead slow on my AMD
3500+ and the Windows programs all make something differently, can't
be scripted and/or have an unusable GUI.

I know that especially the process handling, signals etc is different and
may be difficult to get around. But socket communication and file
handling shouldn't be so hard to adjust. Looking at gnuwin32 there
are many *x tools already ported to Win32, unfortunately not rsync.

Are there other big difficulties in porting?
Thanks for any insight.

bye  Fabi


-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync stops randomly (using vanished files bash script)

[Fabian Cenedese]

At 06:58 24.08.2007 +0200, Samuel Vogel wrote:
I don't think this is possible. Pretty much no services except ssh are running 
on the machine that I'm running the rsync client on. The partition I'm 
transferring the files to hast 400GB of space left and 3.7 GB of my 4GB memory 
are still free...
Any other suggestions?

Do you think it could be possible that the machine which I'm fetching the data 
from run out of ram? I think the the client should have thrown some error 
message at least.

What rsync version are you using? Up to he last official (2.6.9) there
seems to be an error that rsync can deadlock, see here:

http://marc.info/?l=rsyncm=116932447411768w=2

I also had this problem, especially with big files over ssh. I now tried
the current cvs version and it works much better, it never stalled anymore
in the last few days.

bye  Fabi


-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync stops randomly (using vanished files bash script)

[Fabian Cenedese]

At 16:43 24.08.2007 +0200, Samuel Vogel wrote:
I installed the latest CVS Version (today about 1pm). Again, the same problem 
occured...

Did you have it on both sides? Did rsync say sending incremental
file list (probably only with -v)? It only does so if the rsync on the other
side understands it as well.

bye  Fabi


-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html