Re: [leaf-devel] apkg and configdb.lrp, symlinks and empty directories (readlink required)
I updated the patch as I was relying on stat to determine if a file was empty,
which was lame because you can do this with [ -s. Besides, busybox isn't setup
with stat.
Nevertheless, READLINK is not in busybox as a standard component. So this will
only work if you have readlink support.
I think readlink is essential and should be part of busybox. There are other
ways to get the link name without readlink, but from what I've seen, they're
royal kludges and prone to failure if/when ls output changes.
--- apkg.orig2014-09-18 23:20:09.0 -0700
+++ apkg2014-10-15 18:14:56.719869756 -0700
@@ -200,24 +200,42 @@
# figure out which files have changed from this package
package_changed_files () {
-local tmp
+local tmp tmpdir mktmpd s_syms s_dirs s_files
tmp=$TMP/$SESSIONID.sha1
+tmpdir=$TMP/$SESSIONID.tar
(
cd $PKGROOT
echo > $tmp
+NL="
+"
for pkg in $@ ; do
-if [ -f $PKGDIR/$pkg.local ]; then
-for a in $(find $(cat $PKGDIR/$pkg.local) -type f 2>/dev/null); do
-! grep -h "[ /]${a}\$" $PKGDIR/*.sha1 >>$tmp 2>/dev/null && \
-echo " ${a}" >>$tmp
-done
-fi
+ if [ -f $PKGDIR/$pkg.local ] ; then
+ for a in $(find $(cat $PKGDIR/$pkg.local) 2>/dev/null); do
+ if [ -L $a ] ; then
+[ "$mktmpd" != "1" ] && mkdir $tmpdir && mktmpd=1
+# if symlink doesn't exist or if it changed
+if ! tar xzf $STORAGE_MEDIA/$pkg.lrp -C $tmpdir $a 2>/dev/null
|| \
+ [ "$(readlink $a)" != "$(readlink $tmpdir/$a)" ] ; then
+ s_syms=$s_syms"$a"$NL
+fi
+ elif [ -d $a ] ; then
+# Only add empty directories, non-empty dirs are always backed
up
+if ! [ "$(ls -A $a)" ] ; then
+ s_dirs=$s_dirs"$a"$NL
+fi
+ elif [ -f $a ] ; then
+! grep -h "[ /]${a}\$" $PKGDIR/*.sha1 >>$tmp 2>/dev/null && \
+ echo " ${a}" >>$tmp
+ fi
+ done
+ [ "$mktmpd" = "1" ] && rm -rf $tmpdir && mktmpd=0
+ fi
done
-
-sha1sum -c $tmp 2>/dev/null | grep FAILED | sed -e "s/: FAILED//" | \
-sort | uniq
+
+s_files=$(sha1sum -c $tmp 2>/dev/null | grep FAILED | sed -e "s/:
FAILED//")
+echo "$s_syms$s_dirs$s_files" | sort | uniq
)
rm -f $tmp
@@ -407,7 +425,7 @@
#
rundiff () {
-local backup_path vs_saved saved changed interest lrp
+local backup_path vs_saved saved changed interest lrp run_sym sav_sym
vs_saved=$1
if [ ! -f $BACKUP_PATH/$configdb.lrp ] ; then
@@ -442,15 +460,33 @@
interest=$(for file in $interest ; do echo $file ; done | sort | uniq)
for file in $interest ; do
-if [ ! -f running/$file ] ; then
-echo "*** $file has been deleted"
-elif [ -f $vs_saved/$file ] ; then
-diff -u $vs_saved/$file running/$file
-elif [ -f distribution/$file ] ; then
-diff -u distribution/$file running/$file
-else
-diff -u /dev/null running/$file
-fi
+ if [ -L running/$file ] ; then
+ run_sym=$(readlink running/$file)
+ sav_sym=$(readlink $vs_saved/$file)
+ if [ "$run_sym" != "" ] && [ "$sav_sym" = "" ]; then
+ echo "*** $file is NEW symbolic link"
+ elif [ "$run_sym" != "$sav_sym" ]; then
+ echo "*** $file is MODIFIED symbolic link"
+ fi
+ elif [ -d running/$file ] ; then
+ if ! [ -d $vs_saved/$file ] && ! [ -d distribution/$file ] ; then
+ echo "*** $file is NEW empty directory"
+ fi
+ elif [ ! -f running/$file ] ; then
+ if [ -L $vs_saved/$file ] ; then
+ echo "*** $file is DELETED symbolic link"
+ else
+ echo "*** $file has been DELETED"
+ fi
+ elif ! [ -s running/$file ] && ! [ -f $vs_saved/$file ] ; then
+ echo "*** $file is NEW empty file"
+ elif [ -f $vs_saved/$file ] ; then
+ diff -u $vs_saved/$file running/$file
+ elif [ -f distribution/$file ] ; then
+ diff -u distribution/$file running/$file
+ else
+ diff -u /dev/null running/$file
+ fi
done
}
- Original Message -
From: cpu memhd
To: "leaf-devel@lists.sourceforge.net"
Re: [leaf-devel] apkg and configdb.lrp, symlinks and empty directories
Okay, attachments aren't working. Here's the patch:
--- apkg.orig2014-09-18 23:20:09.0 -0700
+++ apkg2014-10-15 15:54:37.625305482 -0700
@@ -200,24 +200,42 @@
# figure out which files have changed from this package
package_changed_files () {
-local tmp
+local tmp tmpdir mktmpd s_syms s_dirs s_files
tmp=$TMP/$SESSIONID.sha1
+tmpdir=$TMP/$SESSIONID.tar
(
cd $PKGROOT
echo > $tmp
+NL="
+"
for pkg in $@ ; do
-if [ -f $PKGDIR/$pkg.local ]; then
-for a in $(find $(cat $PKGDIR/$pkg.local) -type f 2>/dev/null); do
-! grep -h "[ /]${a}\$" $PKGDIR/*.sha1 >>$tmp 2>/dev/null && \
-echo " ${a}" >>$tmp
-done
-fi
+ if [ -f $PKGDIR/$pkg.local ] ; then
+ for a in $(find $(cat $PKGDIR/$pkg.local) 2>/dev/null); do
+ if [ -L $a ] ; then
+[ "$mktmpd" != "1" ] && mkdir $tmpdir && mktmpd=1
+# if symlink doesn't exist or if it changed
+if ! tar xzf $STORAGE_MEDIA/$pkg.lrp -C $tmpdir $a 2>/dev/null
|| \
+ [ "$(readlink $a)" != "$(readlink $tmpdir/$a)" ] ; then
+ s_syms=$s_syms"$a"$NL
+fi
+ elif [ -d $a ] ; then
+# Only add empty directories, non-empty dirs are always backed
up
+if ! [ "$(ls -A $a)" ] ; then
+ s_dirs=$s_dirs"$a"$NL
+fi
+ elif [ -f $a ] ; then
+! grep -h "[ /]${a}\$" $PKGDIR/*.sha1 >>$tmp 2>/dev/null && \
+ echo " ${a}" >>$tmp
+ fi
+ done
+ [ "$mktmpd" = "1" ] && rm -rf $tmpdir && mktmpd=0
+ fi
done
-
-sha1sum -c $tmp 2>/dev/null | grep FAILED | sed -e "s/: FAILED//" | \
-sort | uniq
+
+s_files=$(sha1sum -c $tmp 2>/dev/null | grep FAILED | sed -e "s/:
FAILED//")
+echo "$s_syms$s_dirs$s_files" | sort | uniq
)
rm -f $tmp
@@ -407,7 +425,7 @@
#
rundiff () {
-local backup_path vs_saved saved changed interest lrp
+local backup_path vs_saved saved changed interest lrp run_sym sav_sym
vs_saved=$1
if [ ! -f $BACKUP_PATH/$configdb.lrp ] ; then
@@ -442,15 +460,33 @@
interest=$(for file in $interest ; do echo $file ; done | sort | uniq)
for file in $interest ; do
-if [ ! -f running/$file ] ; then
-echo "*** $file has been deleted"
-elif [ -f $vs_saved/$file ] ; then
-diff -u $vs_saved/$file running/$file
-elif [ -f distribution/$file ] ; then
-diff -u distribution/$file running/$file
-else
-diff -u /dev/null running/$file
-fi
+ if [ -L running/$file ] ; then
+ run_sym=$(readlink running/$file)
+ sav_sym=$(readlink $vs_saved/$file)
+ if [ "$run_sym" != "" ] && [ "$sav_sym" = "" ]; then
+ echo "*** $file is NEW symbolic link"
+ elif [ "$run_sym" != "$sav_sym" ]; then
+ echo "*** $file is MODIFIED symbolic link"
+ fi
+ elif [ -d running/$file ] ; then
+ if ! [ -d $vs_saved/$file ] && ! [ -d distribution/$file ] ; then
+ echo "*** $file is NEW empty directory"
+ fi
+ elif [ ! -f running/$file ] ; then
+ if [ -L $vs_saved/$file ] ; then
+ echo "*** $file is DELETED symbolic link"
+ else
+ echo "*** $file has been DELETED"
+ fi
+ elif [ "$(stat -c %s running/$file)" = "0" ] && ! [ -f $vs_saved/$file
] ; then
+ echo "*** $file is NEW empty file"
+ elif [ -f $vs_saved/$file ] ; then
+ diff -u $vs_saved/$file running/$file
+ elif [ -f distribution/$file ] ; then
+ diff -u distribution/$file running/$file
+ else
+ diff -u /dev/null running/$file
+ fi
done
}
- Original Message -
From: cpu memhd
To: "leaf-devel@lists.sourceforge.net"
Cc:
Sent: Wednesday, October 15, 2014 4:23 PM
Subject: Re: [leaf-devel] apkg and configdb.lrp, symlinks and empty directories
Sorry, this went private.
In the example below that is in the 'replied to' text, apkg does not backup
symlinks or empty directories and is confirmed by KP. But it's not a bug.
That's how it was designed. Symlinks are very important to me and I can't
imagine not using them.
Re: [leaf-devel] apkg and configdb.lrp, symlinks and empty directories
Sorry, this went private. In the example below that is in the 'replied to' text, apkg does not backup symlinks or empty directories and is confirmed by KP. But it's not a bug. That's how it was designed. Symlinks are very important to me and I can't imagine not using them. The only other way is to create them on the fly during bootup. Same goes for empty directories. I've attached a diff that will allow apkg to backup empty directories and symbolic links. It also has a more informative rundiff() which shows more of what changed. I've use this daily for the past week or so. Just cleaned it up a little today. If anyone is interested in the new features, please give it a try. On Wednesday, October 15, 2014 10:04 AM, kp kirchdoerfer wrote: Hi; Am Dienstag, 14. Oktober 2014, 21:43:56 schrieben Sie: > When you specify a directory in a .local file (eg., usr/local) it will > backup everything inside that directory except for empty directories or > symlinks. I have not tried a symlink specified in a .local. I don't see > that there is a way for it to work the way I think it should based on the > code. > > Here's an example: > > Add this to buildtool.cfg for the local.lrp and rebuild package: > > > >Filename= usr/local >Type= directory >Type= local >Permissions = 755 > > > > Or maybe you can just add: > > usr/local > > ...to the existing local.local file. Then... > > > cd /usr/local > ln -s /tmp . > mkdir emptydir > > > Now run a backup with: > > with_storage -r $MNT apkg -r > > Then restore the local.lrp file to your temp directory and you will find > that neither symlinks or empty directories are backed up. You are right - it does not work, neither for empty directories nor for symlinked files. What is this a real problem for you? I mean any chance we can solve this with our current logic? You're of course welcome to improve apkg, but it will require a lot of testing - but that's what betas are meant to be.-- Comprehensive Server Monitoring with Site24x7. Monitor 10 servers for $9/Month. Get alerted through email, SMS, voice calls or mobile push notifications. Take corrective actions from your mobile device. http://p.sf.net/sfu/Zoho ___ leaf-devel mailing list leaf-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/leaf-devel
Re: [leaf-devel] apkg and configdb.lrp, symlinks and empty directories
Sorry, this went private. In the example below that is in the 'replied to' text, apkg does not backup symlinks or empty directories and is confirmed by KP. But it's not a bug. That's how it was designed. Symlinks are very important to me and I can't imagine not using them. The only other way is to create them on the fly during bootup. Same goes for empty directories. I've attached a diff that will allow apkg to backup empty directories and symbolic links. It also has a more informative rundiff() which shows more of what changed. I've use this daily for the past week or so. Just cleaned it up a little today. If anyone is interested in the new features, please give it a try. On Wednesday, October 15, 2014 10:04 AM, kp kirchdoerfer wrote: Hi; Am Dienstag, 14. Oktober 2014, 21:43:56 schrieben Sie: > When you specify a directory in a .local file (eg., usr/local) it will > backup everything inside that directory except for empty directories or > symlinks. I have not tried a symlink specified in a .local. I don't see > that there is a way for it to work the way I think it should based on the > code. > > Here's an example: > > Add this to buildtool.cfg for the local.lrp and rebuild package: > > > >Filename= usr/local >Type= directory >Type= local >Permissions = 755 > > > > Or maybe you can just add: > > usr/local > > ...to the existing local.local file. Then... > > > cd /usr/local > ln -s /tmp . > mkdir emptydir > > > Now run a backup with: > > with_storage -r $MNT apkg -r > > Then restore the local.lrp file to your temp directory and you will find > that neither symlinks or empty directories are backed up. You are right - it does not work, neither for empty directories nor for symlinked files. What is this a real problem for you? I mean any chance we can solve this with our current logic? You're of course welcome to improve apkg, but it will require a lot of testing - but that's what betas are meant to be. > -jorge > > PS: Btw, do you want to keep this private or post to list? I thought I've sent my reply to list and to you personal, just because I've received your mail private and on list as well. We've had a closed list years before, but while that worked pretty well, I've learned that building a community needs to open as possible. So I do not want to keep it private. As an example (which should be kept private pls) - one of the core developers in the early years nearly burned out while working on apkg. It may have happened anyway, but at the end the "private" list, and so Bering-uClibc was near to cease, not to say, I still miss him. kp > > On Monday, October 13, 2014 6:41 AM, kp kirchdoerfer > wrote: Hi; > > I'm not shure I understood... > > If the file/symlink is part of a package, the file should be saved if given > type "local" in buildtool.cfg. The symlink should also be part of > buildtool.cfg. If this does not work, it's a bug. > > Deleted files, which are part of a package will be restored after reboot; > otherwise the package definition is wrong. > > Dynamically created files, which should be saved (e.g /root/.ssh) should be > added to local.lrp (/var/lib/lprkg/local.local) and will be saved in > configdb.lrp. > > It's similar for empty directories. > > Maybe you'll need to describe the problem you want to solve with an > example? > > kp > > Am Montag, 6. Oktober 2014, 15:28:48 schrieb cpu memhd: > > Hello, > > > > Just wondering why apkg does not backup symlinks or empty directories? > > > > I ended up modify it to backup symlinks. Not only that, but rundiff() will > > detect file and symlink deletions. And when a file is new but empty > > (apkg/rundiff() doesn't detect new/empty files, but create_local() does > > back them up): > > > > with_storage -r $MNT apkg -r | more > > > > *** usr/local/linkdir has been DELETED > > *** usr/local/newlinkdir is NEW symlink > > *** usr/local/test.file has been DELETED > > *** usr/local/test.file.2 is NEW empty > > > > It still won't backup empty directories. I think I will eventually modify > > it for this purpose, but package_changed_files() will have to be > > overhauled somewhat. > > > > Btw, this only works under: > > > > create_local () > > rundiff () > > > > I have not tested the other functions. Honestly, I don't know what they > > do/how they work. > > > > -cpu > > > > -- > > -- -- Slashdot TV. Videos for Nerds. Stuff that Matters. > > http://pubads.g.doubleclick.net/gampad/clk?id=160591471&iu=/4140/ostg.clkt > > rk > > > > ___ > > leaf-devel mailing list > > leaf-devel@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/leaf-devel -- Comprehensive Server Monitoring with Site24x7. Monitor 10 servers for $9/Month. Get alerted through email, SMS, voice calls or mobile push notific
