Re: rsync failure on corrupted source data

[Simon Hobson]

Fabian Cenedese  wrote:

>> rsync: write failed on 
>> "/Volumes/durack1ml_bak/160405_1234/Backups.backupdb/durack1ml/2016-02-10-091749/durack1ml_hdd/Applications/Adobe
>>  Media Encoder CC 2015/Adobe Media Encoder CC 
>> 2015.app/Contents/Resources/pdfl/CMaps/ETen-B5-UCS2": Result too large (34)

> Are you sure that the problem is with the corrupted source data? Because the
> error happens on writing the data on the destination.
> 
> rsync: write failed on ... : Result too large (34)
> 
> However I don't know what that error means.

Ditto.


But sort of answering the original question, if it is a corrupted source, then 
my favourite tool is gnu ddrescue which is specifically designed for recovering 
as much as possible from failing/damaged media. It's saved the day for me (or 
friends/clients) many times :-)
https://www.gnu.org/software/ddrescue/
Note: always use it with a log file - then it keeps track of what's been 
copied, what's a "hard" fail, and what's still left to try. You can stop and 
restart (it'll pick up where it left off), and force it to retry stuff 
(sometimes, letting a drive cool off will allow a few more bad blocks to be 
read).

Not to be confused with the confusingly similarly named dd_rescue which doesn't 
have half the features or abilities.


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: --partial not working?

[Paul Slootman]

On Fri 24 Jun 2016, Kevin Korb wrote:

> Again, --partial only means don't delete the incomplete file if rsync is
> aborted.  Normally rsync will delete the incomplete file so you don't
> have bogus files laying around.
> 
> When you rsync to or from a network mount to rsync that is a local copy.
>  To use rsync over the network either your source or your target would
> be hostname:/path (for rsync over ssh) or hostname::module (for an
> rsyncd server).
> 
> With a local copy rsync forces --whole-file because that is a simple
> read the file from one place and write it to the other place.  If you

If you are 100% sure that the source file has not changed in the
meantime, you could use -P --append as that tells rsync that the
destination file is identical to the source file as far as the data
exists. So rsync will just append the remaining data to the file without
bothering to check it.

According to a test I did --append isn't overridden by --whole-file so
it should work.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

RE: --partial not working?

[McDowell, Blake]

Thanks Paul. That should help!

Kevin, 

For a local copy is running a plain rsync transfer ( rsync   ) 
essentially the same as a "drag-and-drop"?

The benefits of using rsync in that situation would all come from choosing 
flags appropriate to the desired transfer?

Thanks,
Blake

-Original Message-
From: rsync [mailto:rsync-boun...@lists.samba.org] On Behalf Of Paul Slootman
Sent: Tuesday, June 28, 2016 7:36 AM
To: rsync@lists.samba.org
Subject: Re: --partial not working?

On Fri 24 Jun 2016, Kevin Korb wrote:

> Again, --partial only means don't delete the incomplete file if rsync 
> is aborted.  Normally rsync will delete the incomplete file so you 
> don't have bogus files laying around.
> 
> When you rsync to or from a network mount to rsync that is a local copy.
>  To use rsync over the network either your source or your target would 
> be hostname:/path (for rsync over ssh) or hostname::module (for an 
> rsyncd server).
> 
> With a local copy rsync forces --whole-file because that is a simple 
> read the file from one place and write it to the other place.  If you

If you are 100% sure that the source file has not changed in the meantime, you 
could use -P --append as that tells rsync that the destination file is 
identical to the source file as far as the data exists. So rsync will just 
append the remaining data to the file without bothering to check it.

According to a test I did --append isn't overridden by --whole-file so it 
should work.


Paul

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: --partial not working?

[Kevin Korb]

Drag and drop depends on a bunch of things like OS, desktop environment,
and file manager.  But in general it is about the same as a cp without
any options.

The upside is that cp and similar are usually faster than rsync since a
client/server model without any networking isn't very efficient.

The downside is that without copying timestamps if you ever want to use
rsync on the same dirs it won't have any idea what is different or not.

Essentially, 'rsync -au --inplace' is pretty much the same as 'cp -au'.
The big problem with either (same for rsync -au --partial) is that since
a partial file is left in place with a current time stamp when you run
it again the incomplete file is newer than the original and therefore it
is never fixed unless the source file gets touched to a newer date.

If I had a bunch of stuff to copy locally I would probably use cp -vau
and just have to remember to rm the incomplete file if I aborted the cp
or if it failed.

Of course there are other features rsync has that cp doesn't have such
as include/exclude/filter and --delete.  So, there are times when it is
still a significant advantage locally despite the performance hit.

On 06/28/2016 03:10 PM, McDowell, Blake wrote:
> Thanks Paul. That should help!
> 
> Kevin, 
> 
> For a local copy is running a plain rsync transfer ( rsync   ) 
> essentially the same as a "drag-and-drop"?
> 
> The benefits of using rsync in that situation would all come from choosing 
> flags appropriate to the desired transfer?
> 
> Thanks,
> Blake
> 
> -Original Message-
> From: rsync [mailto:rsync-boun...@lists.samba.org] On Behalf Of Paul Slootman
> Sent: Tuesday, June 28, 2016 7:36 AM
> To: rsync@lists.samba.org
> Subject: Re: --partial not working?
> 
> On Fri 24 Jun 2016, Kevin Korb wrote:
> 
>> Again, --partial only means don't delete the incomplete file if rsync 
>> is aborted.  Normally rsync will delete the incomplete file so you 
>> don't have bogus files laying around.
>>
>> When you rsync to or from a network mount to rsync that is a local copy.
>>  To use rsync over the network either your source or your target would 
>> be hostname:/path (for rsync over ssh) or hostname::module (for an 
>> rsyncd server).
>>
>> With a local copy rsync forces --whole-file because that is a simple 
>> read the file from one place and write it to the other place.  If you
> 
> If you are 100% sure that the source file has not changed in the meantime, 
> you could use -P --append as that tells rsync that the destination file is 
> identical to the source file as far as the data exists. So rsync will just 
> append the remaining data to the file without bothering to check it.
> 
> According to a test I did --append isn't overridden by --whole-file so it 
> should work.
> 
> 
> Paul
> 
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: 
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   http://www.sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Fwd: rsync seem to be broken on sparc64

[alexmcwhirter]

 Original Message 
Subject: rsync seem to be broken on sparc64
Date: 2016-06-27 23:43
From: alexmcwhir...@triadic.us
To: debian-sp...@lists.debian.org

I posted about this in the kernel lists a few months ago to no avail. I 
see it on gentoo as well with any kernel newer than 3.18. I came across 
this when using lxc on sparc64. The debian template uses rsync to move 
the cache's rootfs to the actual container directory.


I've since modified the template to use "cp -a" instead of rsync, which 
works. However this could be an issue for quite a lot of people that use 
rsync as a backup solution. It really needs to be addressed if we want 
sparc64 to be a release platform.


Here's the gist of it from back then...

On Sun, Feb 21, 2016 at 01:52:55PM -0500, Alex McWhirter wrote:

On 02/14/2016 07:02 PM, Alex McWhirter wrote:
> I having a strange issue where using any 4.X kernel causes rsync to
> appear to die on a select syscall. Not sure why, maybe it's getting a
> wrong file descriptor or something. Unfortunately this starts pushing
> outside of my knowledge level of linux so bear with me. This is on a Sun
> V215 but i have also tested it on a Sun Blade 150 and a Sun Ultra 45
> with the same results. These are all sun4u boxes of course, i haven't
> tried any sun4v boxes. I''l try to spin up a T5120 this week and find
> out if it's also an issue on sun4v.
>
> Here's what I've tested.
>
> 3.14.58 "gentoo" - Works
> 3.18.26 "vanilla"  - Works
> 4.1.15   "gentoo" - Dead
> 4.1.17   "vanilla"  - Dead
> 4.4.1 "vanilla"  - Dead
>
> I don't mind hacking away at kernel sources if anyone can point me in
> the right direction. It's also worth noting that this only happens when
> the folder i am attempting to rsync is significantly large in regards to
> the amount of sub-folders and files. The Gentoo portage tree in particular.
>
> Attached is the strace output of a failing rsync job.
>
>

I've traced this down a bit further.

Kernel 3.18.26 is working but 3.19.0 is not. Git bisect traced it down
to this commit.

e5a4b0bb803b39a36478451eae53a880d2663d5b is the first bad commit
commit e5a4b0bb803b39a36478451eae53a880d2663d5b


here is the gist of that commit...

https://lkml.org/lkml/2014/12/5/25

here is the output of rsync when the error occurs.

root@Magi-01:~# rsync -a /export/test/* /export/test2
rsync: [sender] write error: Broken pipe (32)
rsync error: error in socket IO (code 10) at io.c(820) [sender=3.1.1]
root@Magi-01:~#

here is the output of rsync when executed via gdb

root@Magi-01:~# gdb /usr/bin/rsync
GNU gdb (Debian 7.11.1-2) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 


This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show 
copying"

and "show warranty" for details.
This GDB was configured as "sparc64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
/root/.gdbinit:1: Error in sourced command file:
No executable file specified.
Use the "file" or "exec-file" command.
Reading symbols from /usr/bin/rsync...(no debugging symbols 
found)...done.

(gdb) set args -a /export/test/* /export/test2
(gdb) run
Starting program: /usr/bin/rsync -a /export/test/* /export/test2

Program received signal SIGPIPE, Broken pipe.
0xf80100528fb4 in __write_nocancel () at 
../sysdeps/unix/syscall-template.S:84

84  T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
(gdb)



T thought i would bring this up here as well. Does anyone have any input 
on this?


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Fwd: rsync seem to be broken on sparc64

[Kevin Korb]

I know almost nothing about modern SPARC64 systems especially when they
are running linux.  But, can you try this command line and see if it
gives more information before it blows up:
rsync -vvai /export/test/ /export/test2/

On 06/28/2016 05:39 PM, alexmcwhir...@triadic.us wrote:
>  Original Message 
> Subject: rsync seem to be broken on sparc64
> Date: 2016-06-27 23:43
> From: alexmcwhir...@triadic.us
> To: debian-sp...@lists.debian.org
> 
> I posted about this in the kernel lists a few months ago to no avail. I
> see it on gentoo as well with any kernel newer than 3.18. I came across
> this when using lxc on sparc64. The debian template uses rsync to move
> the cache's rootfs to the actual container directory.
> 
> I've since modified the template to use "cp -a" instead of rsync, which
> works. However this could be an issue for quite a lot of people that use
> rsync as a backup solution. It really needs to be addressed if we want
> sparc64 to be a release platform.
> 
> Here's the gist of it from back then...
> 
> On Sun, Feb 21, 2016 at 01:52:55PM -0500, Alex McWhirter wrote:
>> On 02/14/2016 07:02 PM, Alex McWhirter wrote:
>> > I having a strange issue where using any 4.X kernel causes rsync to
>> > appear to die on a select syscall. Not sure why, maybe it's getting a
>> > wrong file descriptor or something. Unfortunately this starts pushing
>> > outside of my knowledge level of linux so bear with me. This is on a
>> Sun
>> > V215 but i have also tested it on a Sun Blade 150 and a Sun Ultra 45
>> > with the same results. These are all sun4u boxes of course, i haven't
>> > tried any sun4v boxes. I''l try to spin up a T5120 this week and find
>> > out if it's also an issue on sun4v.
>> >
>> > Here's what I've tested.
>> >
>> > 3.14.58 "gentoo" - Works
>> > 3.18.26 "vanilla"  - Works
>> > 4.1.15   "gentoo" - Dead
>> > 4.1.17   "vanilla"  - Dead
>> > 4.4.1 "vanilla"  - Dead
>> >
>> > I don't mind hacking away at kernel sources if anyone can point me in
>> > the right direction. It's also worth noting that this only happens when
>> > the folder i am attempting to rsync is significantly large in
>> regards to
>> > the amount of sub-folders and files. The Gentoo portage tree in
>> particular.
>> >
>> > Attached is the strace output of a failing rsync job.
>> >
>> >
>>
>> I've traced this down a bit further.
>>
>> Kernel 3.18.26 is working but 3.19.0 is not. Git bisect traced it down
>> to this commit.
>>
>> e5a4b0bb803b39a36478451eae53a880d2663d5b is the first bad commit
>> commit e5a4b0bb803b39a36478451eae53a880d2663d5b
> 
> here is the gist of that commit...
> 
> https://lkml.org/lkml/2014/12/5/25
> 
> here is the output of rsync when the error occurs.
> 
> root@Magi-01:~# rsync -a /export/test/* /export/test2
> rsync: [sender] write error: Broken pipe (32)
> rsync error: error in socket IO (code 10) at io.c(820) [sender=3.1.1]
> root@Magi-01:~#
> 
> here is the output of rsync when executed via gdb
> 
> root@Magi-01:~# gdb /usr/bin/rsync
> GNU gdb (Debian 7.11.1-2) 7.11.1
> Copyright (C) 2016 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later
> 
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "sparc64-linux-gnu".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> .
> Find the GDB manual and other documentation resources online at:
> .
> For help, type "help".
> Type "apropos word" to search for commands related to "word"...
> /root/.gdbinit:1: Error in sourced command file:
> No executable file specified.
> Use the "file" or "exec-file" command.
> Reading symbols from /usr/bin/rsync...(no debugging symbols found)...done.
> (gdb) set args -a /export/test/* /export/test2
> (gdb) run
> Starting program: /usr/bin/rsync -a /export/test/* /export/test2
> 
> Program received signal SIGPIPE, Broken pipe.
> 0xf80100528fb4 in __write_nocancel () at
> ../sysdeps/unix/syscall-template.S:84
> 84T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
> (gdb)
> 
> 
> 
> T thought i would bring this up here as well. Does anyone have any input
> on this?
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   http://www.sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



signature.asc
Description: OpenPGP digital signature
-- 
Please use repl

Re: rsync failure on corrupted source data

[Henri Shustak]

+1 for ddrescue :)

> https://www.gnu.org/software/ddrescue/



This email is protected by LBackup, an open source backup solution
http://www.lbackup.org


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Fwd: rsync seem to be broken on sparc64

[alexmcwhirter]

On 2016-06-28 18:10, Kevin Korb wrote:

I know almost nothing about modern SPARC64 systems especially when they
are running linux.  But, can you try this command line and see if it
gives more information before it blows up:
rsync -vvai /export/test/ /export/test2/


I'm not sure how helpful this will be, but it certainly doesn't get very 
far.


root@Magi-01:/var/cache/lxc/debian/rootfs-sid-sparc64# rsync -vvai ./* 
/export/test/

sending incremental file list
delta-transmission disabled for local transfer or --whole-file
cd+ bin/

f+ bin/bash

rsync: [sender] write error: Broken pipe (32)
rsync error: error in socket IO (code 10) at io.c(820) [sender=3.1.1]
root@Magi-01:/var/cache/lxc/debian/rootfs-sid-sparc64#

here's the relevant strace output of the error.

getdents(3, /* 0 entries */, 32768) = 0
close(3)= 0
write(1, ".d..t.. bin/\n", 17.d..t.. bin/
)  = 17
open("bin/bash", O_RDONLY)  = 3
fstat64(3, {st_mode=01, st_size=0, ...}) = 0
write(1, ">f+ bin/bash\n", 21>f+ bin/bash
)  = 21
mmap(NULL, 270336, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 
0) = 0xf8010289e000
read(3, 
"\177ELF\2\2\1\0\0\0\0\0\0\0\0\0\0\2\0+\0\0\0\1\0\0\0\0\0\21\373P"..., 
262144) = 262144

select(6, [5], [4], [5], {60, 0})   = 1 (out [4], left {59, 87})
write(4, 
"\223\213\0\7\377\0368\r\7E/Eterm\0\333\10W\233\27g\244\201\0\0008\24\6-c"..., 
35735) = -1 EPIPE (Broken pipe)
--- SIGPIPE {si_signo=SIGPIPE, si_code=SI_USER, si_pid=10421, si_uid=0} 
---
write(2, "rsync: [sender] write error: Bro"..., 45rsync: [sender] write 
error: Broken pipe (32)) = 45

write(2, "\n", 1
)   = 1
rt_sigaction(SIGUSR1, {SIG_IGN, [], 0}, NULL, 0xf801004839b8, 8) = 0
rt_sigaction(SIGUSR2, {SIG_IGN, [], 0}, NULL, 0xf801004839b8, 8) = 0
waitpid(10422, 0x7feffd71954, WNOHANG)  = 0
getpid()= 10421
kill(10422, SIGUSR1)= 0
write(2, "rsync error: error in socket IO "..., 69rsync error: error in 
socket IO (code 10) at io.c(820) [sender=3.1.1]) = 69

write(2, "\n", 1
)   = 1
exit_group(10)  = ?
+++ exited with 10 +++

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html