> > Wait ... there are still a few things to improve in ftpdir-cleanup. > > For instance if one of the out-of-date packages in $DELETEFILES is a > symlink to an "any" file, that symlink will be removed, but not the > original file. > > Also, I would prefer to have that script take care of all arches at > once, like the new db-update script does. That's the easiest way to > handle the above situation: > if an "any" package is outdated for both arches, then remove both > symlinks AND the actual file. But if it is only outdated for one arch, > then remove that symlink but keep the original file. > > I have rewritten the ftp-cleanup script this morning. I'm going to > revise it now and should submit my patches very soon. > >
Here they are. Looks like it is a lot of changes, but as I noted in the commit message, it really amounts to reshuffling the code in order to have a for loop for each ARCH, followed by a scan of the "any" directory. Hope you like it ;) F
>From 84c78a083996ba583946ce021007425d31f87b78 Mon Sep 17 00:00:00 2001 From: Francois Charette <[email protected]> Date: Mon, 20 Jul 2009 19:54:23 +0200 Subject: [PATCH] remove trailing slash from ftppath[_any] --- db-update | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/db-update b/db-update index 9d9f7eb..2c86d64 100755 --- a/db-update +++ b/db-update @@ -106,8 +106,8 @@ fi for current_arch in ${arch...@]}; do - ftppath="$FTP_BASE/$reponame/os/$current_arch/" - ftppath_any="$FTP_BASE/$reponame/os/any/" + ftppath="$FTP_BASE/$reponame/os/$current_arch" + ftppath_any="$FTP_BASE/$reponame/os/any" if [ ! -d "$ftppath" ]; then echo "FTP path for this repo ($reponame) is missing" @@ -197,7 +197,7 @@ for current_arch in ${arch...@]}; do if [ $(/bin/ls "$WORKDIR/build/"*-$current_arch$PKGEXT 2>/dev/null | wc -l) != 0 ]; then echo "Copying new files to '$ftppath'" for f in "$WORKDIR/build/"*-$current_arch$PKGEXT; do - if ! /bin/cp "$f" "$ftppath"; then + if ! /bin/cp "$f" "$ftppath/"; then die "error: failure while copying files to $ftppath" fi done -- 1.6.3.3
>From 01f4192cc3c6189c31f8b47f1f09090c0fe74076 Mon Sep 17 00:00:00 2001 From: Francois Charette <[email protected]> Date: Tue, 21 Jul 2009 11:38:09 +0200 Subject: [PATCH] Refactor ftpdir-cleanup to handle all arches This may look like a rather large patch, but the changes are mostly a reshuffling of the code to loop over all arches first, and then handle the arch-indep packages. The cronjob has been changed accordingly. Added new category DELETESYMLINKS, which are deleted instead of being moved to package-cleanup. I have also fixed the arch-specific issue with the ftppath, using parameters in config instead. --- config | 1 + cron-jobs/ftpdir-cleanup | 7 +- misc-scripts/ftpdir-cleanup | 267 ++++++++++++++++++++++++------------------- 3 files changed, 153 insertions(+), 122 deletions(-) diff --git a/config b/config index 425709a..16a71ce 100644 --- a/config +++ b/config @@ -1,5 +1,6 @@ FTP_BASE="/srv/ftp" +FTP_OS_SUFFIX="os" SVNREPO_core="file:///srv/svn-packages" SVNREPO_extra="file:///srv/svn-packages" diff --git a/cron-jobs/ftpdir-cleanup b/cron-jobs/ftpdir-cleanup index dfa968e..38c7fcd 100755 --- a/cron-jobs/ftpdir-cleanup +++ b/cron-jobs/ftpdir-cleanup @@ -1,13 +1,12 @@ #!/bin/bash repos="core extra testing community" -arches="i686 x86_64" LOCKFILE="/tmp/.ftpdircleanup.lock" cleanup () { rm -f "$LOCKFILE" - exit 0 + exit 0 } ctrl_c() { @@ -32,9 +31,7 @@ trap ctrl_c 2 . $(dirname $0)/../config for repo in $repos; do - for arch in $arches; do - $(dirname $0)/../misc-scripts/ftpdir-cleanup $repo $arch $CLEANUP_DESTDIR - done + $(dirname $0)/../misc-scripts/ftpdir-cleanup $repo $CLEANUP_DESTDIR done cleanup diff --git a/misc-scripts/ftpdir-cleanup b/misc-scripts/ftpdir-cleanup index e319b99..ee1dc2e 100755 --- a/misc-scripts/ftpdir-cleanup +++ b/misc-scripts/ftpdir-cleanup @@ -1,25 +1,22 @@ #!/bin/bash -if [ $# -ne 3 ]; then - echo "usage: $(basename $0) <reponame> <arch> <dest-dir>" +if [ $# -ne 2 ]; then + echo "usage: $(basename $0) <reponame> <dest-dir>" exit 1 fi reponame=$1 -arch=$2 -dest=$3 +dest=$2 -##### Arch specific stuff. TODO make this configurable ##### -ftppath_base="/srv/ftp/$reponame/os" ############################################################ -ftppath="$ftppath_base/$arch" - -if [ ! -d "$ftppath" ]; then - echo "FTP path '$ftppath' does not exist" +if [ ! -f "$(dirname $0)/../config" ]; then + echo "$(dirname $0)/../config not found! Aborting" exit 1 fi +. "$(dirname $0)/../config" + if [ ! -f /etc/makepkg.conf ]; then echo "/etc/makepkg.conf not found! Aborting" exit 1 @@ -28,140 +25,176 @@ fi . /etc/makepkg.conf getpkgname() { - local tmp - - tmp=${1##*/} - tmp=${tmp%$PKGEXT} - tmp=${tmp%-$arch} - echo ${tmp%-*-*} + local tmp + + tmp=${1##*/} + tmp=${tmp%$PKGEXT} + for arch in ${arch...@]}; do + tmp=${tmp%-$arch} + done + tmp=${tmp%-any} + echo ${tmp%-*-*} } -getpkgname_ver() { - local tmp +TMPDIR=$(mktemp -d /tmp/cleanup-.XXXXXX) || exit 1 - tmp=${1##*/} - tmp=${tmp%$PKGEXT} - echo ${tmp%-$arch} -} +ftppath_base="$FTP_BASE/$reponame/$FTP_OS_SUFFIX" -MISSINGFILES="" -DELETEFILES="" -EXTRAFILES="" +for arch in ${arch...@]}; do + + ftppath="$ftppath_base/$arch" + MISSINGFILES="" + DELETEFILES="" + DELETESYMLINKS="" + EXTRAFILES="" + + if [ ! -d "$ftppath" ]; then + echo "FTP path '$ftppath' does not exist" + exit 1 + fi -TMPDIR=$(mktemp -d /tmp/cleanup.XXXXXX) || exit 1 -cd "${TMPDIR}" -bsdtar xf "$ftppath/$reponame.db.tar.$DB_COMPRESSION" + cd "${TMPDIR}" + bsdtar xf "$ftppath/$reponame.db.tar.$DB_COMPRESSION" -for pkg in *; do - filename=$(grep -A1 '^%FILENAME%$' "${pkg}/desc" | tail -n1) - [ -z "${filename}" ] && filename="${pkg}${PKGEXT}" + for pkg in *; do + filename=$(grep -A1 '^%FILENAME%$' "${pkg}/desc" | tail -n1) + [ -z "${filename}" ] && filename="${pkg}${PKGEXT}" - if [ ! -e "${ftppath}/${filename}" ]; then - MISSINGFILES="${MISSINGFILES} ${filename}" - else - pkgname="$(getpkgname ${filename})" - for otherfile in ${ftppath}/${pkgname}-*; do - otherfile="$(basename ${otherfile})" - if [ "${otherfile}" != "${filename}" -a "${pkgname}" = "$(getpkgname ${otherfile})" ]; then - DELETEFILES="${DELETEFILES} ${otherfile}" + if [ ! -e "${ftppath}/${filename}" ]; then + MISSINGFILES="${MISSINGFILES} ${filename}" + else + pkgname="$(getpkgname ${filename})" + for otherfile in ${ftppath}/${pkgname}-*; do + otherfile="$(basename ${otherfile})" + if [ "${otherfile}" != "${filename}" -a "${pkgname}" = "$(getpkgname ${otherfile})" ]; then + if [ -h "${otherfile}" ]; then + DELETESYMLINKS="${DELETESYMLINKS} ${otherfile}" + else + DELETEFILES="${DELETEFILES} ${otherfile}" + fi + fi + done + fi + done + + cd "$ftppath" + for pkg in *$arch$PKGEXT; do + pkgname="$(getpkgname $pkg)" + for p in ${TMPDIR}/${pkgname}-*; do + if [ -d "${p}" -a "$(getpkgname $(basename ${p}))" = "${pkgname}" ]; then + continue 2 + fi + done + EXTRAFILES="$EXTRAFILES $pkg" + done + + cd "$ftppath" + rm -rf ${TMPDIR} + + # Do a quick check to see if a missing ARCHINDEPFILE is in the any dir + # If it is, and the file is MISSING, restore it + missfiles="$MISSINGFILES" + MISSINGFILES="" + for mf in $missfiles; do + af_satisfied=0 + if [ -e "${ftppath_base}/any/${mf}" ]; then + echo "Restoring missing 'any' symlink: ${mf}" + ln -s "../any/${mf}" "${ftppath}" + else + MISSINGFILES="${MISSINGFILES} ${mf}" fi - done + done + + echo "Scan complete for $reponame ($arch) at ${ftppath}" + + #Make sure we've done *something* before outputting anything + if [ -z "$DELETEFILES$DELETESYMLINKS$MISSINGFILES$EXTRAFILES" ]; then + continue + fi + + if [ -n "$DELETEFILES" ]; then + echo " The following files are out of date" + echo " They will be moved to '$dest'" + for f in $DELETEFILES; do + echo " $f" + done + echo "" + fi + + if [ -n "$DELETESYMLINKS" ]; then + echo " The following symlinks are out of date" + echo " They will be deleted" + for f in $DELETESYMLINKS; do + echo " $f" + done + echo "" + fi + + if [ -n "$MISSINGFILES" ]; then + echo " The following files are missing in the repo" + for f in $MISSINGFILES; do + echo " $f" + done + echo "" + fi + + if [ -n "$EXTRAFILES" ]; then + echo " The following files are in the repo but not the db" + echo " They will be moved to '$dest'" + for f in $EXTRAFILES; do + echo " $f" + done fi + + if [ -n "${DELETEFILES}" ]; then + mv ${DELETEFILES} "$dest" + echo "" + fi + + if [ -n "${DELETESYMLINKS}" ]; then + rm -f ${DELETESYMLINKS} + echo "" + fi + + if [ -n "${EXTRAFILES}" ]; then + mv ${EXTRAFILES} "$dest" + echo "" + fi + done -cd "$ftppath" -for pkg in *$arch$PKGEXT; do - pkgname="$(getpkgname $pkg)" - for p in ${TMPDIR}/${pkgname}-*; do - if [ -d "${p}" -a "$(getpkgname $(basename ${p}))" = "${pkgname}" ]; then - continue 2 - fi - done - EXTRAFILES="$EXTRAFILES $pkg" -done +ARCHINDEPFILES="" if [ -d "$ftppath_base/any" ]; then + echo "Checking arch-independent files..." cd "$ftppath_base/any" for pkg in *$PKGEXT; do [ -f "$pkg" ] || continue # in case we get a file named "*.pkg.tar.gz" + ### TODO loop over arch in ${arch...@]} ... something like this: + # teststring="" + # for arch in ${arch...@]}; do + # teststring="$teststring -a ! -h $ftppath_base/$arch/$pkg" + # done + # teststring=$(echo $teststring | sed 's/^ -a//') + # if [ eval "$teststring" ]; then if [ ! -h "$ftppath_base/i686/$pkg" -a ! -h "$ftppath_base/x86_64/$pkg" ]; then ARCHINDEPFILES="$ARCHINDEPFILES $pkg" fi done fi -cd "$ftppath" -rm -rf ${TMPDIR} - -#Make sure we've done *something* before outputting anything -if [ -z "$DELETEFILES$MISSINGFILES$EXTRAFILES$ARCHINDEPFILES" ]; then - exit 0 -fi - -# Do a quick check to see if a missing ARCHINDEPFILE is in the any dir -# If it is, and the file is MISSING, restore it -missfiles="$MISSINGFILES" -MISSINGFILES="" -for mf in $missfiles; do - af_satisfied=0 - if [ -e "${ftppath_base}/any/${mf}" ]; then - echo "Restoring missing 'any' symlink: ${mf}" - ln -s "../any/${mf}" "${ftppath}" - else - MISSINGFILES="${MISSINGFILES} ${mf}" - fi -done - -echo "Scan complete for $reponame ($arch) at ${ftppath}" - -if [ -n "$DELETEFILES" ]; then - echo " The following files are out of date" - echo " They will be moved to '$dest'" - for f in $DELETEFILES; do - echo " $f" - done - echo "" -fi - -if [ -n "$MISSINGFILES" ]; then - echo " The following files are missing in the repo" - for f in $MISSINGFILES; do - echo " $f" - done - echo "" -fi - -if [ -n "$EXTRAFILES" ]; then - echo " The following files are in the repo but not the db" - echo " They will be moved to '$dest'" - for f in $EXTRAFILES; do - echo " $f" - done -fi - if [ -n "$ARCHINDEPFILES" ]; then - echo " The following architecture independent packages" - echo " are not symlinked in the architecture repositories." - echo " They will be moved to '$dest'" + echo " The following architecture independent packages" + echo " are not symlinked in the architecture repositories." + echo " They will be moved to '$dest'" for f in $ARCHINDEPFILES; do echo " $f" done fi -if [ -n "${DELETEFILES}" ]; then - mv ${DELETEFILES} "$dest" - echo "" -fi - -if [ -n "${EXTRAFILES}" ]; then - mv ${EXTRAFILES} "$dest" - echo "" +if [ -d "$ftppath_base/any" -a -n "${ARCHINDEPFILES}" ]; then + cd "$ftppath_base/any" + mv ${ARCHINDEPFILES} "$dest" + echo "" fi - -cd "$ftppath_base/any" -if [ -n "${ARCHINDEPFILES}" ]; then - mv ${ARCHINDEPFILES} "$dest" - echo "" -fi - -- 1.6.3.3
>From 94dbd2af5dfb469baacd6ed53d3430f69ce1db12 Mon Sep 17 00:00:00 2001 From: Francois Charette <[email protected]> Date: Tue, 21 Jul 2009 13:52:21 +0200 Subject: [PATCH] Bugfix: make TMPDIR within the for loop --- misc-scripts/ftpdir-cleanup | 7 +++++-- 1 files changed, 5 insertions(+), 2 deletions(-) diff --git a/misc-scripts/ftpdir-cleanup b/misc-scripts/ftpdir-cleanup index ee1dc2e..b28ec94 100755 --- a/misc-scripts/ftpdir-cleanup +++ b/misc-scripts/ftpdir-cleanup @@ -36,12 +36,12 @@ getpkgname() { echo ${tmp%-*-*} } -TMPDIR=$(mktemp -d /tmp/cleanup-.XXXXXX) || exit 1 ftppath_base="$FTP_BASE/$reponame/$FTP_OS_SUFFIX" for arch in ${arch...@]}; do + TMPDIR=$(mktemp -d /tmp/cleanup-XXXXXX) || exit 1 ftppath="$ftppath_base/$arch" MISSINGFILES="" DELETEFILES="" @@ -53,8 +53,11 @@ for arch in ${arch...@]}; do exit 1 fi + if ! cd "${TMPDIR}" ; then + echo "Failed to cd to ${TMPDIR}" + exit 1 + fi - cd "${TMPDIR}" bsdtar xf "$ftppath/$reponame.db.tar.$DB_COMPRESSION" for pkg in *; do -- 1.6.3.3
>From f3b595598b10e615c284dbb486d4ee390cb5577a Mon Sep 17 00:00:00 2001 From: Francois Charette <[email protected]> Date: Tue, 21 Jul 2009 14:00:51 +0200 Subject: [PATCH] Very minor additions --- misc-scripts/ftpdir-cleanup | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/misc-scripts/ftpdir-cleanup b/misc-scripts/ftpdir-cleanup index b28ec94..fc99c46 100755 --- a/misc-scripts/ftpdir-cleanup +++ b/misc-scripts/ftpdir-cleanup @@ -113,6 +113,7 @@ for arch in ${arch...@]}; do #Make sure we've done *something* before outputting anything if [ -z "$DELETEFILES$DELETESYMLINKS$MISSINGFILES$EXTRAFILES" ]; then + echo "(nothing to do for $arch)" continue fi @@ -180,7 +181,7 @@ if [ -d "$ftppath_base/any" ]; then # teststring="$teststring -a ! -h $ftppath_base/$arch/$pkg" # done # teststring=$(echo $teststring | sed 's/^ -a//') - # if [ eval "$teststring" ]; then + # if [ $teststring ]; then if [ ! -h "$ftppath_base/i686/$pkg" -a ! -h "$ftppath_base/x86_64/$pkg" ]; then ARCHINDEPFILES="$ARCHINDEPFILES $pkg" fi -- 1.6.3.3
