Re: IO error encountered, skipping file deletion - WTF?
On Sun, May 29, 2011 at 11:04:25PM -0700, travis+ml-rs...@subspacefield.org wrote: Could a dangling symlink cause this problem? To answer my own question, yes, a dangling symlink on the source was the cause of the problem, and I had --copy-unsafe-symlinks is on. Thanks for the help people, that one was baffling me. -- http://www.subspacefield.org/~travis/ He who lives by the computer, dies by the computer. If you are a spammer, please email j...@subspacefield.org to get blacklisted. pgpfiQHMa9lhH.pgp Description: PGP signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: IO error encountered, skipping file deletion - WTF?
On Wed, Apr 06, 2011 at 11:44:45AM +0200, Paul Slootman wrote: On Tue 05 Apr 2011, travis+ml-rs...@subspacefield.org wrote: Anyone know what this is supposed to mean? 366305 files to consider IO error encountered -- skipping file deletion I keep getting this, and so it refuses to delete any files from a mirror. But what is the supposed I/O error? Can we have it give us some kind of clue? It probably already does. Really? Where, exactly, since it's not in stderr/stdout? Unreadable files/directories (for e.g. permissions reasons, it doesn't have to be a hardware IO error) will cause this. Okay, but which files/directories are the problem? Note that I am running this as root, at least on the remote system, so it does not seem like it could be on the target server. However, IO error is fairly generic, and on an archive of 300,000 files, does not help me much. If the error is local, it'd be nice if rsync used the standard perror to show the exact error, and prefix it with the directory/filename. If remote, perhaps the remote end should use perror and hte local side pick it up. Could a dangling symlink cause this problem? To answer the other question, no, it's not a rsync daemon. -- http://www.subspacefield.org/~travis/ He who lives by the computer, dies by the computer. If you are a spammer, please email j...@subspacefield.org to get blacklisted. pgpVutHAkcE6C.pgp Description: PGP signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
IO error encountered, skipping file deletion - WTF?
Anyone know what this is supposed to mean? 366305 files to consider IO error encountered -- skipping file deletion I keep getting this, and so it refuses to delete any files from a mirror. But what is the supposed I/O error? Can we have it give us some kind of clue? dmesg doesn't show any clues. -vv --stats --progress doesn't help much. I suppose I could strace/ktrace it, but is it local or remote? I can't even tell where to start. rsync should give us a better error message. -- Effing the ineffable since 1997. | http://www.subspacefield.org/~travis/ My emails do not usually have attachments; it's a digital signature that your mail program doesn't understand. If you are a spammer, please email j...@subspacefield.org to get blacklisted. pgpw9j7L1Dtrm.pgp Description: PGP signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
two bugs in rsync 3.0.6 - ampersands in filenames, double quoting required
1) deleting path/to/(Something) Word Word Anotherword--Word Word.pdf IO error encountered -- skipping file deletion I think either the -- is causing it to fail, or more likely, the next filename, which it didn't bother to print out, which has an ampersand in the name, causes it to fail. I am using SSH as a substrate, of course, and I think you guys failed to escape things, so the ampersand is hosing your protocol somehow. 2) I am also aware that when rsyncing to a foreign system, you have to double escape filenames - if the target has a space in it, you must use not one, but two sets of quotes. Since you can't nest double quotes or single quotes, this means that you have to use both. This is fairly annoying, for a number of reasons, including the difficulty of doing variable substitution (single quotes prevent it, so you gotta bust out for the variable and then back in), and the fact that it's not intuitive at all. -- I'm not sleeping; I'm regenerating. | http://www.subspacefield.org/~travis/ My emails do not usually have attachments; it's a digital signature that your mail program doesn't understand. If you are a spammer, please email j...@subspacefield.org to get blacklisted. pgpxzejIwIh98.pgp Description: PGP signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
maybe a Unicode problem? was Re: two bugs in rsync 3.0.6 - ampersands in filenames, double quoting required
On Fri, Mar 11, 2011 at 06:41:26PM -0800, travis+ml-rs...@subspacefield.org wrote: deleting path/to/(Something) Word Word Anotherword--Word Word.pdf IO error encountered -- skipping file deletion I think either the -- is causing it to fail, or more likely, the next filename, which it didn't bother to print out, which has an ampersand in the name, causes it to fail. I am using SSH as a substrate, of course, and I think you guys failed to escape things, so the ampersand is hosing your protocol somehow. Actually, I'm unable to replicate the error with some test files with the exact same names. Just to test the escaping, I tried creating this dir: (Word) Word Word Word--Word Word (Word Word.).ext idid idid`id` I was expecting to find some shell-escapes, so I threw those last two in for good measure. I was able to create and delete those filenames on the target properly. So I didn't find the shell escape I was looking for. This time it failed when trying to delete a file containing the word G\#303\#266del I wonder if there's some issues with Unicode... -- I'm not sleeping; I'm regenerating. | http://www.subspacefield.org/~travis/ My emails do not usually have attachments; it's a digital signature that your mail program doesn't understand. If you are a spammer, please email j...@subspacefield.org to get blacklisted. pgpoDrybyBy0Y.pgp Description: PGP signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: maybe a Unicode problem? was Re: two bugs in rsync 3.0.6 - ampersands in filenames, double quoting required
On Fri, Mar 11, 2011 at 07:00:57PM -0800, travis+ml-rs...@subspacefield.org wrote: This time it failed when trying to delete a file containing the word G\#303\#266del I wonder if there's some issues with Unicode... Or possibly not. This time, the last line before the I/O error was a filename with nothing special in it, just spaces and a hyphen. If anyone can suggest a troubleshooting technique I'd appreciate it. Both src and target are Ubuntu, the source is rsync 3.0.6 and the target is 3.0.7 -- I'm not sleeping; I'm regenerating. | http://www.subspacefield.org/~travis/ My emails do not usually have attachments; it's a digital signature that your mail program doesn't understand. If you are a spammer, please email j...@subspacefield.org to get blacklisted. pgpT2zvevdX2S.pgp Description: PGP signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
does rsync work in lockstep? parallel could improve high latency performance
Just curious if the protocol sends a request and waits for a response, or whether it can multiplex multiple requests on a single connection. This could improve performance over high latency links, if a lot of time is spent waiting for the response (i.e. when the hashes match). -- Travis is an organic computer peripheral. My emails do not have attachments; it's a digital signature that your mail program doesn't understand. | http://www.subspacefield.org/~travis/ If you are a spammer, please email j...@subspacefield.org to get blacklisted. pgpcsJe6CbjfX.pgp Description: PGP signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: does rsync work in lockstep? parallel could improve high latency performance
Oh, and if it buffers requests sent to its STDIN, that's fine. To actually complete requests in parallel, you'd need a multithread or multiprocess app, and that's complicated. The proper term for what I'm suggesting is probably pipelining. -- Travis is an organic computer peripheral. My emails do not have attachments; it's a digital signature that your mail program doesn't understand. | http://www.subspacefield.org/~travis/ If you are a spammer, please email j...@subspacefield.org to get blacklisted. pgptfOifydSQC.pgp Description: PGP signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
HDB, a hard drive backup program
Hey all, I'm writing a tool: http://www.subspacefield.org/security/hdb/ It is very similar in some ways to rsync, except it's meant for backing up locally to removable HDDs, and it keeps metadata around when the HDD is removed. I figured I'd ping people here to see if they are interested in participating in the brainstorming sessions. I figure you know a lot more than me about this kind of stuff. -- A Weapon of Mass Construction My emails do not have attachments; it's a digital signature that your mail program doesn't understand. | http://www.subspacefield.org/~travis/ If you are a spammer, please email j...@subspacefield.org to get blacklisted. pgp3b8fiiFUvX.pgp Description: PGP signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
feature request: remote user is root, make remote owner is foo
I often push files from my user account over SSH to my web server, and want them owned by www-user, which may not have a login shell, should never accept remote logins, and who may not have a ~/.ssh directory (and if it did, it would be under the wwwroot, ack!). Currently I push as root and then do a chmod, but isn't there a better way? While I'm doing this, the files are temporarily unavailable, since they aren't readable by www-user as they exist on the local system. -- A Weapon of Mass Construction My emails do not have attachments; it's a digital signature that your mail program doesn't understand. | http://www.subspacefield.org/~travis/ If you are a spammer, please email j...@subspacefield.org to get blacklisted. pgpDEunZZoPBe.pgp Description: PGP signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: feature request: remote user is root, make remote owner is foo
On Wed, Aug 11, 2010 at 01:34:44PM -0400, Brian Cuttler wrote: As a matter of principle, SOP, we don't like to ssh/rsync as root and generally don't allow root ssh/rsync into a box. Better/safer to move the security stuff to a lower powered user if you can. I'm familiar with the argument. Let me give you my take on it: http://www.subspacefield.org/security/security_concepts/index.html#tth_sEc11.9.9 Downside: Direct root logins make accountability harder - you have only the source IP to go on. Upside: You can back up the entire [file] system remotely. You can rsync stuff owned by users without valid login shells or authorized_keys. For me, I'm the only root user, and only allow key-based logins, so there's no downside. I'll look into your SGID directory idea for group ownership. PS: rsync kinda assumes when doing --preserve-uids that UIDs (or maybe it was user names) map. When they don't exist on target system, you either get owned by destination user (no --preserve-uids), or owned by wrong user, both of which have drawbacks. It'd be nice to have a way to map users, but not a must-have. -- A Weapon of Mass Construction My emails do not have attachments; it's a digital signature that your mail program doesn't understand. | http://www.subspacefield.org/~travis/ If you are a spammer, please email j...@subspacefield.org to get blacklisted. pgpxXcIkHOPhd.pgp Description: PGP signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: feature request: remote user is root, make remote owner is foo
On Wed, Aug 11, 2010 at 01:32:42PM -0400, Brian Cuttler wrote:
[Set u+s on directories, don't worry about owners]
It seems to work relatively well. I get an error about not being
able to chgrp the files owned by other users, and, in my case,
the group ends up wrong because it's not supposed to be the same
on both ends (www-data on destination, something else on source).
However, because of the former problem, at least the extant files
stay readable until I can fix-up the group problem caused by the
latter. If I further chown them to www-data, then they don't get
their gid hosed the second time around.
Not an ideal solution, but seems workable for the time being.
Aside: chown really needs a flag that says set GID on directories but
nothing else. Right now I do this, which is the only right thing to
do when filenames might contain anything, including whitespace or
newlines:
setperms () {
local perms=$1
local fileperm=$2
local dirperm=$3
local own=$4
local dir
shift 4
for dir in $@
do
if test -d $dir
then
find $dir -xdev -type d -print0 | xargs -0 --no-run-if-empty chmod
$perms,$dirperm --
find $dir -xdev -type f -print0 | xargs -0 --no-run-if-empty chmod
$perms,$fileperm --
chown -R $own $dir
fi
done
}
setperms u=rwX,o=rX g=rwX g=rwxs owner:group /path/to/whatever
--
A Weapon of Mass Construction
My emails do not have attachments; it's a digital signature that your mail
program doesn't understand. | http://www.subspacefield.org/~travis/
If you are a spammer, please email j...@subspacefield.org to get blacklisted.
pgpqbaSRuWBh3.pgp
Description: PGP signature
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
