Re: Shellscript escaping problem
On Fri, Aug 03, 2012 at 10:41:09PM -0700, Philip Guenther wrote: On Fri, Aug 3, 2012 at 9:35 PM, David Diggles da...@elven.com.au wrote: ... here's an example of how not to script rsync, when just starting to learn how to script it got over complicated over time. i should rewrite it sometime :) ... I guess I don't understand the point of sending that out. It's like a generic ghost story: ...and the code walks the corridors of the office building to this day! There it is! Ah! If the goal is to help the inexperienced shell script writer avoid that fate, you must provide instruction and suggestions, not just set up your prior works up as warning. The beginner will be dazzled by the mess-o'-punctuation, but that doesn't help them see what they should do instead when they, in turn, find their own scripts crawling into the morass. Philip Guenther There were already excellent examples of what to do provided by others. What is wrong with an example of how bad spaghetti scripting looks like? I think there can be value in seeing an exagerrated example of what not to do. Furthermore, I provided my own script, not someone elses, because I am happy to own my own mistakes. Another good thing to teach. IMO, a teaching method that only ever teaches what to do, and never what not to do, and only ever provides instructions and guidance, risks creating the kind of box that breeds idiots with no ability to think for themselves.
Re: Shellscript escaping problem
On Sat, Aug 4, 2012 at 11:00 PM, David Diggles da...@elven.com.au wrote: What is wrong with an example of how bad spaghetti scripting looks like? I think there can be value in seeing an exagerrated example of what not to do. I don't think it's obvious to everyone that that was bad code. Indeed, there was that person in the past that wrote that code and decided it was the best they could do at that moment. What's to say some other person won't look at your code and say huh, looks better than some of my other scripts; I don't get what the problem is. Furthermore, I provided my own script, not someone elses, because I am happy to own my own mistakes. Another good thing to teach. True. IMO, a teaching method that only ever teaches what to do, and never what not to do, and only ever provides instructions and guidance, risks creating the kind of box that breeds idiots with no ability to think for themselves. In my experience as a student and teacher, the discussions of about what made something (a program, a work of art, a mathematical proof, a piece of music) better or worse were the most valuable part of reviewing the existing works. If just seeing the bad stuff without thinking about and recognizing what made it bad was enough to let you create good stuff, then Sturgeon's law would be false. Philip Guenther
Re: Shellscript escaping problem
If you're doing complex rsync setups, you might also consider using rsnapshot as a wrapper. I just joined the maintainer list on that, it's a very useful old perl tool, well organized for frequent and well managed backups. Rsnapshot is a nice tool, but for my current usage rsync suffices. Kind regards, Martijn Rijkeboer
Re: Shellscript escaping problem
Philip Guenther guent...@gmail.com wrote: On Fri, Aug 3, 2012 at 9:35 PM, David Diggles da...@elven.com.au wrote: ... here's an example of how not to script rsync, when just starting to learn how to script it got over complicated over time. i should rewrite it sometime :) ... I guess I don't understand the point of sending that out. +1
Re: Shellscript escaping problem
RSYNC_CMD=/usr/local/bin/rsync -v -n \
--rsync-path='rsync sudo' \
This doesn't do what you think it does. The single quotes are getting
literally passed to rsync, they're not reinterpreted after $RSYNC_CMD
is interpolated.
Yep. $RSYNC_CMD will be splitted by space, tab or newline by default, or
by the contents of $IFS, if set.
if you need it to be reusable, I'd suggest making it a function or so:
synchronize() {
/usr/local/bin/rsync -v -n \
--rsync-path='/usr/bin/sudo /usr/local/bin/rsync' \
--archive --one-file-system --compress --hard-links \
--numeric-ids --human-readable --delete-after \
--backup --backup-dir=deleted/$TODAY \
--exclude=deleted \
$@
}
synchronize backup@fqdn:$DATADIR/dir/ $DATADIR/dir
Using a function works without problems. Thanks to all who gave input.
Kind regards,
Martijn Rijkeboer
Re: Shellscript escaping problem
On Fri, Aug 3, 2012 at 4:10 AM, Martijn Rijkeboer mart...@bunix.org wrote:
RSYNC_CMD=/usr/local/bin/rsync -v -n \
--rsync-path='rsync sudo' \
This doesn't do what you think it does. The single quotes are getting
literally passed to rsync, they're not reinterpreted after $RSYNC_CMD
is interpolated.
Yep. $RSYNC_CMD will be splitted by space, tab or newline by default, or
by the contents of $IFS, if set.
if you need it to be reusable, I'd suggest making it a function or so:
synchronize() {
/usr/local/bin/rsync -v -n \
--rsync-path='/usr/bin/sudo /usr/local/bin/rsync' \
--archive --one-file-system --compress --hard-links \
--numeric-ids --human-readable --delete-after \
--backup --backup-dir=deleted/$TODAY \
--exclude=deleted \
$@
}
synchronize backup@fqdn:$DATADIR/dir/ $DATADIR/dir
Using a function works without problems. Thanks to all who gave input.
Kind regards,
Martijn Rijkeboer
If you're doing complex rsync setups, you might also consider using
rsnapshot as a wrapper. I just joined the maintainer list on that,
it's a very useful old perl tool, well organized for frequent and well
managed backups.
Re: Shellscript escaping problem
On Thu, Aug 02, 2012 at 11:21:01PM +0200, Martijn Rijkeboer wrote:
Hi,
here's an example of how not to script rsync, when just starting
to learn how to script
it got over complicated over time. i should rewrite it sometime :)
#!/bin/bash
SCRIPT=${0##*/}
BASE=/archive0/_backup
SSHOPTS=-q -o Ciphers=arcfour256 -o MACs=umac...@openssh.com
RSYNCOPTS=--delete -avxlr
CONF=/etc/$SCRIPT
f_f() {
[ -d $1 ] \
for SRC in $(ls $1|grep -v ^\.); do
echo \# $SRC \#
if [ $# -ge 3 ]; then
eval $(echo eval $3)
DST=$2/$REV/${HOST%%.*}
local i=4;while [ $i -le $# ]; do
eval $(echo eval $$i)
((i++));done
fi
done
}
f_f $CONF/rsyncd $BASE local HOST=\${SRC##*@} local PKG=\$(ssh \$SSHOPTS
\$SRC \uname -nr;pkg_info -t 2/dev/null|cut -f1 -d ' '\) local
REV=\$(echo \$PKG|cut -f2 -d ' ') mkdir -p \$DST;cd \$DST;[[ -n \$PKG ]]
echo \$PKGpkg_info rsync \$RSYNCOPTS --files-from=\$CONF/rsyncd/\$SRC
\$HOST::rsk \$DST
f_f $CONF/rsh $BASE local HOST=\${SRC##*@} local PKG=\$(rsh -l \$SCRIPT
\$HOST \uname -nr;pkg_info -t 2/dev/null|cut -f1 -d ' '\) local
REV=\$(echo \$PKG|cut -f2 -d ' ') mkdir -p \$DST;cd \$DST;[[ -n \$PKG ]]
echo \$PKGpkg_info rsync \$RSYNCOPTS --files-from=\$CONF/rsh/\$SRC
\$HOST::rsk \$DST
f_f $CONF/rsync $BASE local HOST=\${SRC##*@} local PKG=\$(ssh \$SSHOPTS
\$SRC \uname -nr;pkg_info -t 2/dev/null|cut -f1 -d ' '\) local
REV=\$(echo \$PKG|cut -f2 -d ' ') mkdir -p \$DST;cd \$DST;[[ -n \$PKG ]]
echo \$PKGpkg_info rsync -e \ssh \$SSHOPTS\ \$RSYNCOPTS
--files-from=\$CONF/rsync/\$SRC \$SRC:/ \$DST
f_f $CONF/ssh $BASE local HOST=\${SRC##*@} local PKG=\$(ssh \$SSHOPTS \$SRC
\uname -nr;pkg_info -t 2/dev/null|cut -f1 -d ' '\) local REV=\$(echo
\$PKG|cut -f2 -d ' ') mkdir -p \$DST;cd \$DST;[[ -n \$PKG ]] echo
\$PKGpkg_info ssh \$SSHOPTS \$SRC \tar cpf - 2/dev/null \$(echo \$(
\$CONF/ssh/\$SRC))\|tar xpf - rm -rf \$DST mv \$DST.tmp \$DST
Re: Shellscript escaping problem
On Fri, Aug 3, 2012 at 9:35 PM, David Diggles da...@elven.com.au wrote: ... here's an example of how not to script rsync, when just starting to learn how to script it got over complicated over time. i should rewrite it sometime :) ... I guess I don't understand the point of sending that out. It's like a generic ghost story: ...and the code walks the corridors of the office building to this day! There it is! Ah! If the goal is to help the inexperienced shell script writer avoid that fate, you must provide instruction and suggestions, not just set up your prior works up as warning. The beginner will be dazzled by the mess-o'-punctuation, but that doesn't help them see what they should do instead when they, in turn, find their own scripts crawling into the morass. Philip Guenther
Shellscript escaping problem
Hi, I'm using the script below and it keeps giving me the following error: Unexpected remote arg: backup@fqdn:/data/dir/ rsync error: syntax or usage error (code 1) at main.c(1214) [sender=3.0.9] It looks like something is going wrong with my escaping, because when I remove the --rsync-path line everything works fine. It also works fine if I echo the last line of the script and copy and paste it in my shell (KSH). I've tried all kinds of different escaping and nothing seems to work. Any ideas on what I'm doing wrong? Kind regards, Martijn Rijkeboer #!/bin/sh DATADIR=/data TODAY=`/bin/date +%Y%m%d` RSYNC_CMD=/usr/local/bin/rsync -v -n \ --rsync-path='rsync sudo' \ --rsh=ssh \ --archive --one-file-system --compress --hard-links --numeric-ids \ --human-readable --delete-after --backup --backup-dir=deleted/$TODAY \ --exclude=deleted $RSYNC_CMD backup@fqdn:$DATADIR/dir/ $DATADIR/dir
Re: Shellscript escaping problem
On Thu, Aug 2, 2012 at 2:21 PM, Martijn Rijkeboer mart...@bunix.org wrote: RSYNC_CMD=/usr/local/bin/rsync -v -n \ --rsync-path='rsync sudo' \ This doesn't do what you think it does. The single quotes are getting literally passed to rsync, they're not reinterpreted after $RSYNC_CMD is interpolated. This is similar to running something like: rsync --rsync-path=\'rsync sudo\' backup@fqdn...
Re: Shellscript escaping problem
On 08/02/12 23:34, Matthew Dempsky wrote:
On Thu, Aug 2, 2012 at 2:21 PM, Martijn Rijkeboer mart...@bunix.org wrote:
RSYNC_CMD=/usr/local/bin/rsync -v -n \
--rsync-path='rsync sudo' \
This doesn't do what you think it does. The single quotes are getting
literally passed to rsync, they're not reinterpreted after $RSYNC_CMD
is interpolated.
This is similar to running something like:
rsync --rsync-path=\'rsync sudo\' backup@fqdn...
Yep. $RSYNC_CMD will be splitted by space, tab or newline by default, or
by the contents of $IFS, if set.
And no, that is _not_ a suggestion to fiddle with $IFS. :-)
if you need it to be reusable, I'd suggest making it a function or so:
...
synchronize() {
/usr/local/bin/rsync -v -n \
--rsync-path='/usr/bin/sudo /usr/local/bin/rsync' \
--archive --one-file-system --compress --hard-links \
--numeric-ids --human-readable --delete-after \
--backup --backup-dir=deleted/$TODAY \
--exclude=deleted \
$@
}
synchronize backup@fqdn:$DATADIR/dir/ $DATADIR/dir
synchronize backup2@host2:/home/ $DIR2/home/
synchronize --ignore-existing backup2@host3:/dir3/ $DIR3/dir3/
...
/Alexander
