Both url_check and url_check_ng have been touched last in 2008 and 2009, they are referenced nowhere in the repository, and have also been superseded by the 'ptxdist urlcheck' command since commit 5d39eb69ac6e116fbe0f (2016-08-20, "ptxdist: add urlcheck command").
Link: https://git.pengutronix.de/cgit/ptxdist/commit/?id=5d39eb69ac6e116fbe0f Signed-off-by: Roland Hieber <r...@pengutronix.de> --- plugins/url_check/README | 12 - plugins/url_check/main | 314 ----------------- plugins/url_check/ptxlib.bash | 351 -------------------- plugins/url_check_ng/README | 27 -- plugins/url_check_ng/main | 283 ---------------- plugins/url_check_ng/ptxlib.bash | 351 -------------------- plugins/url_check_ng/robotparser2.py.diff | 44 --- plugins/url_check_ng/robotparser2.py.readme | 1 - 8 files changed, 1383 deletions(-) delete mode 100644 plugins/url_check/README delete mode 100755 plugins/url_check/main delete mode 100644 plugins/url_check/ptxlib.bash delete mode 100644 plugins/url_check_ng/README delete mode 100755 plugins/url_check_ng/main delete mode 100644 plugins/url_check_ng/ptxlib.bash delete mode 100644 plugins/url_check_ng/robotparser2.py.diff delete mode 100644 plugins/url_check_ng/robotparser2.py.readme diff --git a/plugins/url_check/README b/plugins/url_check/README deleted file mode 100644 index 482cbe1e1284..000000000000 --- a/plugins/url_check/README +++ /dev/null @@ -1,12 +0,0 @@ -# this is a quick hack, but quite useful - -Usage: main OPTIONS - - --help, -h this help - --check-all, -a check ALL packages - -plugins/url_check/main checks the availability of all needed source packages -for the current project configuration. By default, packages -will only be checked, if the corresponding switch in -ptxdistrc is set to "y" (enabled in 'ptxdist menuconfig') - diff --git a/plugins/url_check/main b/plugins/url_check/main deleted file mode 100755 index 5789d4059bd7..000000000000 --- a/plugins/url_check/main +++ /dev/null @@ -1,314 +0,0 @@ -#!/bin/bash -# ---------------------------------------------------------- -# -# Script: PTXdist URL Checker -# Rev: 1 -# Description: -# Written by: Bj�rn B�rger <b.buer...@pengutronix.de> -# Changed: 2006-09-21 bbu -# Docs: inline -# Manpage: none -# -# ---------------------------------------------------------- -PREFIX="`basename $0` " -# ---------------------------------------------------------- -# Short Documentation / Comments -# ---------------------------------------------------------- -# -# TODO: -# build URL_CHECKER as host-tool in ptxdist. -# Problem -> uses python => big and fat -# OR find a lightweight alternative. -# -# Please note, that wget has problems w/ ftp links. -# -# - -# ---------------------------------------------------------- -# generic script settings -# ---------------------------------------------------------- -# -# The script domains - chose one or more of: -# - dumb_tool -# - tool -# - development -# - system_management -# -# PTX_SCRIPT_DOMAINS="dumb_tool" - -PTX_LIB_VERSION="2" - -# ---------------------------------------------------------- -# Default Configuration Options -# ---------------------------------------------------------- - -MKTEMP="mktemp" -WHICH="which" -GREP="grep" -EGREP="egrep" -CAT="cat" -SED="sed" -MKDIR="mkdir" - -LINKCHECKER_BIN="linkchecker" - -URL_CHECKER="$LINKCHECKER_BIN --no-warnings" -# URL_CHECKER="wget --spider" - -PTXCONFIG="selected_ptxconfig" - -logdir="`dirname $0`/log" -logfile="$logdir/url_check_log" - -# ---------------------------------------------------------- -# Load ptx shell library and generic ptx configuration -# ---------------------------------------------------------- - -PTXLIB=`dirname $0`/ptxlib.bash - -if [ -e "$PTXLIB" ] ; then - . $PTXLIB -else - echo "ERROR: ptxlib not found" - exit 1 -fi - -# ========================================================== -# Temporary files -# ========================================================== - -TMPDIR="`$MKTEMP -d /tmp/url_check_plugin.XXXXXXXXXX`" || echo "could not create TMPDIR" - -ptx_debug "TMPDIR is $TMPDIR" - -# ========================================================== -# Traps -# ========================================================== - -[ -e "$TMPDIR/on_exit_reverse.sh" ] || echo "echo" > $TMPDIR/on_exit_reverse.sh - -if [ "$DEBUG" = "true" ]; then - ptx_debug "trap function: deleting temporary files..." - trap '[ -e "$TMPDIR/on_exit_reverse.sh" ] && sh $TMPDIR/on_exit_reverse.sh ; rm -rvf /tmp/$(basename $TMPDIR)' EXIT -else - trap '[ -e "$TMPDIR/on_exit_reverse.sh" ] && sh $TMPDIR/on_exit_reverse.sh ; rm -rf /tmp/$(basename $TMPDIR)' EXIT -fi - -# ---------------------------------------------------------- -# Default Dependency check -# ---------------------------------------------------------- - -dependency_check_dirs_depends="" -dependency_check_files_depends="$PTXCONFIG" -dependency_check_tools_depends="$LINKCHECKER_BIN $MKTEMP" - -ptx_dependency_check - -# ========================================================== -# Option Parser -# ========================================================== - -Usage() { -cat <<-EOF - -Usage: `basename "$0"` OPTIONS - - --help -h this help - --check-all -a check ALL packages - --modules -m check module packages - --builtin -y check builtin packages - -$0 checks the availability of all needed source packages -for the current project configuration. By default, packages -will only be checked, if the corresponding switch in -ptxdistrc is set to "y" (enabled in 'ptxdist menuconfig') - -EOF -} - -# Parser -# ------ -# option no argument -# option: required argument -# option:: optional argument - -TEMP=`getopt --options h,a,y,m \ - --longoptions="help,check-all,modules,builtin" \ - -n "$0" -- "$@"` -if [ $? != 0 ] ; then echo "Terminating..." >&2 ; exit 1 ; fi -eval set -- "$TEMP" - -while true ; do - case "$1" in - -h|--help) - [ -z "$action" ] - action="help" ; - shift - ;; - -a|--check-all) - [ -z "$action" ] - action="check_all" ; - shift - ;; - -m|--modules) - [ -z "$action" ] - action="check_modules" ; - shift - ;; - -y|--builtin) - [ -z "$action" ] - action="check_builtin" ; - shift - ;; - --) shift ; break ;; - *) echo "Internal error!" ; exit 1 ;; - esac -done - -# ========================================================== -# Script Variables -# ========================================================== - -# none - -# ========================================================== -# Script Functions -# ========================================================== - -init(){ - # choose the right ptxdist version: - echo "PATH is $PATH" - PTXDIST_VERSION=$(echo `$GREP PTXCONF_CONFIGFILE_VERSION $PTXCONFIG` \; echo "\$PTXCONF_CONFIGFILE_VERSION" | sh) - PTXDIST=$(dirname $0)/../../bin/ptxdist - if [ "$($PTXDIST --version)" != "$PTXDIST_VERSION" ]; then - PTXDIST=ptxdist-$PTXDIST_VERSION - which $PTXDIST || PTXDIST=ptxdist - fi - PTXDIST_BIN="`$WHICH $PTXDIST`" - ptx_debug "PTXDIST is: $PTXDIST ($PTXDIST_BIN)" - if [ "$1" = "all" ]; then - # get all package labels - # (PACKAGES-y contains all activated packages) - # (PACKAGES- contains all deactivated packages) - YESPACKAGES=`$PTXDIST print PACKAGES-y` - MODPACKAGES=`$PTXDIST print PACKAGES-m` - NOPACKAGES=`$PTXDIST print PACKAGES-` - elif [ "$1" = "active" ]; then - # get only configured builtin package labels - # (PACKAGES-y contains all active builtin packages) - YESPACKAGES=`$PTXDIST print PACKAGES-y` - MODPACKAGES="" - NOPACKAGES="" - elif [ "$1" = "modules" ]; then - # get only configured MODULE package labels - # (PACKAGES-m contains all module packages) - YESPACKAGES="" - MODPACKAGES=`$PTXDIST print PACKAGES-m` - NOPACKAGES="" - else - # get all configured builtin and module package labels - # (PACKAGES-y contains all activated packages) - YESPACKAGES=`$PTXDIST print PACKAGES-y` - MODPACKAGES=`$PTXDIST print PACKAGES-m` - NOPACKAGES="" - fi - PACKAGES="$(echo $YESPACKAGES $MODPACKAGES $NOPACKAGES | tr "[a-z-]" "[A-Z_]")" -} - -create_url_list(){ - echo "creating list of download URLS for `echo $PACKAGES | wc -w` packages" - targets=`for target in $PACKAGES; do echo print-${target}_URL; done | sort -u` - $PTXDIST make -i ${targets} 2>> $TMPDIR/errors > $TMPDIR/urllist - if [ `cat "$TMPDIR/urllist" | wc -l` = "0" ]; then - echo "Fast listing failed. This may take a while!" - for target in $PACKAGES; do - $PTXDIST print ${target}_URL 2>> $TMPDIR/errors >> $TMPDIR/urllist - echo -n "." - done - fi - echo "done" -} - -test_urls(){ - egrep -v "^#|^$|^[[:space:]]" $TMPDIR/urllist \ - | while read line; do - local count_ok=0 - local count_fail=0 - rm -f $TMPDIR/errormsg - for url in $line; do - $URL_CHECKER $url >>$TMPDIR/errormsg 2>&1 - case $? in - 0) - count_ok=$[count_ok+1] - ;; - *) - count_fail=$[count_fail+1] - ;; - esac - done - if [ $count_fail -eq 0 ]; then - echo " [ OK ] $line" - elif [ $count_ok -eq 0 ]; then - echo " [ FAIL ] $line" - echo "" - echo "" - $CAT $TMPDIR/errormsg | $GREP -A 10 "Start checking at" - echo "" - else - local count=$[count_ok+$count_fail] - echo " [ $count_ok/$count ] $line" - fi - done -} - -runner(){ - $MKDIR -p $logdir - case $? in - 0) - test_urls | tee $logfile - echo "LOGFILE is --> $logfile" - ;; - *) - test_urls - ;; - esac -} - -# ========================================================== -# Script Main -# ========================================================== - -case "$action" in - help) - Usage - ;; - check_all) - init all - create_url_list >&2 - runner - ;; - check_modules) - init modules - create_url_list >&2 - runner - ;; - check_builtin) - init active - create_url_list >&2 - runner - ;; - *) - init - create_url_list >&2 - runner - ;; -esac - -# ========================================================== -# Cleanup -# ========================================================== - -# unconfigured - done by trap function -# ---------------------------------------------------------- -# End -# ---------------------------------------------------------- diff --git a/plugins/url_check/ptxlib.bash b/plugins/url_check/ptxlib.bash deleted file mode 100644 index 50249916d2b7..000000000000 --- a/plugins/url_check/ptxlib.bash +++ /dev/null @@ -1,351 +0,0 @@ -#!/bin/bash -# ----------------------------------------------------- -# Description: Basic Script Functions -# Author: Bj�rn B�rger <b.buer...@pengutronix.de> -# Date: Wed Mar 21 17:03:39 CET 2007 -# ----------------------------------------------------- - -PTXLIB="true" -MY_VERSION="2" - -if [ "$PTX_LIB_VERSION" -lt "$MY_VERSION" ] ; then - echo "WARNING: ptxlib.bash not compatible" - echo "WARNING: please audit your script" - exit 1 -fi - -FULLARGS="$@" -DEBUG=${DEBUG:="false"} - -# -# customized exit functions -# -# $1 --> Error Message -# $2 --> Exit Code -# -ptx_exit(){ - echo "$0: $1" - exit $2 -} -ptx_exit_silent(){ - ptx_debug "$0: $1" - exit $2 -} - -# -# print out error message and exit with status 1 -# -# $1: error message -# ${PREFIX}: to be printed before message -# -ptx_bailout () { - echo "${PREFIX}error: $1" >&2 - exit 1 -} - - -# -# print out warning message -# -# $1: warning message -# ${PREFIX}: to be printed before message -# -ptx_warning() { - echo "${PREFIX}warning: $1" >&2 -} - -# -# print out classic message -# -# $1: warning message -# ${PREFIX}: to be printed before message -# -ptx_message() { - echo "${PREFIX}: $1" >&1 -} - -ptx_message_n() { - if [ -n "$PREFIX" ]; then - echo -n "${PREFIX}: $1" >&1 - else - echo -n "$1" >&1 - fi -} - -# -# Debugging Output -# -ptx_debug(){ - [ "$DEBUG" = "true" ] && echo "$0: $1" >&2 -} -ptx_debug "Debugging is enabled - Turn off with DEBUG=false" - - -# -# check if a previously executed pipe returned an error -# -ptx_check_pipe_status() { - for i in "${PIPESTATUS[@]}"; do [ $i -gt 0 ] && { - echo - echo "error: a command in the pipe returned $i, bailing out" - echo - exit $i - } - done -} - -# -# present a choice [y/n] and execute command -# -# $1: question / message -# $2: command -# $3: yes message -# $4: no message -# ${PREFIX}: to be printed before message -# -# -ptx_yesno_choice(){ - ptx_debug "Message: $1" - ptx_debug "Command: $2" - dialog --yesno "$1" 0 0 - case $? in - 0) - ptx_message "${PREFIX} [YES] - $3" - $2 || ptx_error "command $2 failed" - ;; - *) - ptx_warning "${PREFIX} [NO] - $4" - ;; - esac -} - -# -# create a directory -# -ptx_check_create_dir(){ - [ -d "$1" ] || ptx_yesno_choice "$1 does not exist, create it?" "mkdir -pv $1" "[SUCCESS]" "[ABORTED]" -} - -# -# dependency check for needed files -# -# if any one of these variables is -# set, a dependency check is performed: -# -# dependency_check_files_recommends -# dependency_check_files_depends -# dependency_check_files_conflicts -# -ptx_dependency_check_files(){ - [ -z "$dependency_check_files_recommends" ] && ptx_debug "[file check] no file recommendation defined" - for file in $dependency_check_files_recommends; do - test -e $file - case $? in - 0) - ptx_debug "[file check] This Script recommends $file: OK" - ;; - *) - ptx_warning "[file check] This Script recommends $file: FILE NOT FOUND" - ;; - esac - done - [ -z "$dependency_check_files_depends" ] && ptx_debug "[file check] no file dependency defined" - for file in $dependency_check_files_depends; do - test -e $file - case $? in - 0) - ptx_debug "[file check] This Script depends on $file: OK" - ;; - *) - ptx_bailout "[file check] This Script depends on $file: FILE NOT FOUND" - ;; - esac - done - [ -z "$dependency_check_files_conflicts" ] && ptx_debug "[file check] no file conflict defined" - for file in $dependency_check_files_conflicts; do - test -e $file - case $? in - 0) - ptx_bailout "[file check] This Script conflicts with $file: FILE EXISTS" - ;; - *) - ptx_debug "[file check] This Script conflicts with $file: OK (does not exist)" - ;; - esac - done -} - -# -# dependency check for needed directories -# -# if any one of these variables is -# set, a dependency check is performed: -# -# dependency_check_dirs_recommends -# dependency_check_dirs_depends -# dependency_check_dirs_conflicts -# -ptx_dependency_check_dirs(){ - [ -z "$dependency_check_dirs_recommends" ] && ptx_debug "[dir check] no directory recommendation defined" - for directory in $dependency_check_dirs_recommends; do - test -d $directory - case $? in - 0) - ptx_debug "[dir check] This Script recommends $directory: OK" - ;; - *) - ptx_warning "[dir check] This Script recommends $directory: directory NOT FOUND" - ;; - esac - done - [ -z "$dependency_check_dirs_depends" ] && ptx_debug "[dir check] no directory dependency defined" - for directory in $dependency_check_dirs_depends; do - test -d $directory - case $? in - 0) - ptx_debug "[dir check] This Script depends on $directory: OK" - ;; - *) - ptx_bailout "[dir check] This Script depends on $directory: directory NOT FOUND" - ;; - esac - done - [ -z "$dependency_check_dirs_conflicts" ] && ptx_debug "[dir check] no directory conflict defined" - for directory in $dependency_check_dirs_conflicts; do - test -d $directory - case $? in - 0) - ptx_bailout "[dir check] This Script conflicts with $directory: directory EXISTS" - ;; - *) - ptx_debug "[dir check] This Script conflicts with $directory: OK (does not exist)" - ;; - esac - done -} - -# -# dependency check for needed tools -# -# if any one of these variables is -# set, a dependency check is performed: -# -# dependency_check_tools_recommends -# dependency_check_tools_depends -# dependency_check_tools_conflicts -# -ptx_dependency_check_tools(){ - [ -z "$dependency_check_files_recommends" ] && ptx_debug "[file check] no tool recommendation defined" - for tool in $dependency_check_tools_recommends; do - which $tool >/dev/null 2>&1 - case $? in - 0) - ptx_debug "[tool check] This Script recommends $tool: OK" - ;; - *) - ptx_warning "[tool check] This Script recommends $tool: MISSING" - ;; - esac - done - [ -z "$dependency_check_tools_depends" ] && ptx_debug "[file check] no tool dependency defined" - for tool in $dependency_check_tools_depends; do - which $tool >/dev/null 2>&1 - case $? in - 0) - ptx_debug "[tool check] This Script depends on $tool: OK" - ;; - *) - ptx_bailout "[tool check] This Script depends on $tool: MISSING" - ;; - esac - done - [ -z "$dependency_check_tools_conflicts" ] && ptx_debug "[file check] no file conflict defined" - for tool in $dependency_check_tools_conflicts; do - which $tool >/dev/null 2>&1 - case $? in - 0) - ptx_bailout "[tool check] This Script conflicts with on $tool: CONFLICT FOUND" - ;; - *) - ptx_debug "[tool check] This Script conflicts with $tool: OK" - ;; - esac - done -} - -ptx_dependency_check_hostname(){ - [ -z "$dependency_check_hostname_recommends" ] && ptx_debug "[hostname check] no hostname recommendation defined" - for hostname in $dependency_check_hostname_recommends; do - [ "`hostname`" = "$hostname" ] >/dev/null 2>&1 - case $? in - 0) - ptx_debug "[hostname check] This Script recommends host $hostname: OK" - ;; - *) - ptx_warning "[hostname check] This Script recommends host $hostname: THIS IS NOT ME" - ;; - esac - done - [ -z "$dependency_check_hostname_depends" ] && ptx_debug "[hostname check] no hostname dependency defined" - for hostname in $dependency_check_hostname_depends; do - [ "`hostname`" = "$hostname" ] >/dev/null 2>&1 - case $? in - 0) - ptx_debug "[hostname check] This Script depends on host $hostname: OK" - ;; - *) - ptx_bailout "[hostname check] This Script depends on host $hostname: THIS IS NOT ME" - ;; - esac - done - [ -z "$dependency_check_hostname_conflicts" ] && ptx_debug "[hostname check] no hostname conflict defined" - for hostname in $dependency_check_hostname_conflicts; do - [ "`hostname`" = "$hostname" ] >/dev/null 2>&1 - case $? in - 0) - ptx_bailout "[hostname check] This Script conflicts with host $hostname: CONFLICT FOUND" - ;; - *) - ptx_debug "[hostname check] This Script conflicts with host $hostname: OK" - ;; - esac - done -} - -# -# convenience wrapper for the functions above -# -# performs all available checks -# -ptx_dependency_check(){ - ptx_dependency_check_hostname - ptx_dependency_check_dirs - ptx_dependency_check_files - ptx_dependency_check_tools -} - -# service specific lib functions, based on script domains: -# -# - dumb_tool -# - tool -# - development -# - system_management -# - -if [ -n "$PTX_SCRIPT_DOMAINS" ]; then -for ptx_script_domain in $PTX_SCRIPT_DOMAINS; do - ptx_debug "loading config for domain: ${ptx_script_domain}" - SCONFS="$HOME/.ptxsd-${ptx_script_domain}.conf \ - /usr/local/etc/ptxsd-${ptx_script_domain}.conf \ - /etc/ptxsd-${ptx_script_domain}.conf \ - `dirname $0`/ptxsd-${ptx_script_domain}.conf \ - `dirname $0`/../lib/ptxsd-${ptx_script_domain}.conf" - for scriptconf in $SCONFS; do - if [ -e "$scriptconf" ] ; then - . $scriptconf && { ptx_debug "$scriptconf loaded"; break; } - fi - done - [ "$PTXSD" = "true" ] || ptx_warning " No valid Configuration for Domain: ${ptx_script_domain} found \ - [ $conf (version $MY_VERSION) ] [ caller: $0 ]" -done -fi diff --git a/plugins/url_check_ng/README b/plugins/url_check_ng/README deleted file mode 100644 index f29656fd802c..000000000000 --- a/plugins/url_check_ng/README +++ /dev/null @@ -1,27 +0,0 @@ -# new urlcheck version, which uses m2b environment -# for faster processing. This will speed up processing -# ( ~about 30 seconds to 5 minutes ). - -Usage: main OPTIONS - - --help, -h this help - --check-all, -a check ALL packages - -plugins/url_check_ng/main checks the availability of all -needed source packages for the current project configuration. -By default, packages will only be checked, if the -corresponding switch in ptxdistrc is set to "y" (enabled -in 'ptxdist menuconfig') - -Usage: - -<full_path>/plugins/url_check_ng/main [--help | --check-all ] - -OR - -user@host:YourPTXdistProject$ ptxdist make plugin-url_check_ng - -PTXdist will will try to find the plugin in -a) $PTXDIST_WORKSPACE/plugins -b) $PTXDIST_TOPDIR/plugins - diff --git a/plugins/url_check_ng/main b/plugins/url_check_ng/main deleted file mode 100755 index d896e0a866e4..000000000000 --- a/plugins/url_check_ng/main +++ /dev/null @@ -1,283 +0,0 @@ -#!/bin/bash -# ---------------------------------------------------------- -# -# Script: PTXdist URL Checker -# Rev: 1.1 -# Description: -# Written by: Bj�rn B�rger <b.buer...@pengutronix.de> -# Changed: 2006-09-21 bbu -# Docs: inline -# Manpage: none -# -# ---------------------------------------------------------- -PREFIX="`basename $0` " -# ---------------------------------------------------------- -# Short Documentation / Comments -# ---------------------------------------------------------- -# -# TODO: -# build URL_CHECKER as host-tool in ptxdist. -# Problem -> uses python => big and fat -# OR find a lightweight alternative. -# -# Please note, that wget has problems w/ ftp links. -# -# - -# ---------------------------------------------------------- -# generic script settings -# ---------------------------------------------------------- -# -# The script domains - chose one or more of: -# - dumb_tool -# - tool -# - development -# - system_management -# -# PTX_SCRIPT_DOMAINS="dumb_tool" - -PTX_LIB_VERSION="2" - -# ---------------------------------------------------------- -# Default Configuration Options -# ---------------------------------------------------------- - -MKTEMP="mktemp" -WHICH="which" -GREP="grep" -EGREP="egrep" -CAT="cat" -SED="sed" -MKDIR="mkdir" - -LINKCHECKER_BIN="linkchecker" - -# If you are using the python based linkchecker, this -# option may speed up things *a little bit* ;-) -if [ "$LINKCHECKER_BIN" == "linkchecker" ]; then - LINKCHECKER_FASTMODE=${LINKCHECKER_FASTMODE:=true} - LINKCHECKER_THREADS=${LINKCHECKER_THREADS:=50} -else - LINKCHECKER_FASTMODE=${LINKCHECKER_FASTMODE:=false} -fi - -URL_CHECKER="$LINKCHECKER_BIN --no-warnings" -# URL_CHECKER="wget --spider" - -PTXCONFIG="ptxconfig" -M2B_ENV="${PTXDIST_PLATFORMDIR}/state/environment" - -# ---------------------------------------------------------- -# Load ptx shell library and generic ptx configuration -# ---------------------------------------------------------- - -PTXLIB=`dirname $0`/ptxlib.bash - -if [ -e "$PTXLIB" ] ; then - . $PTXLIB -else - echo "ERROR: ptxlib not found" - exit 1 -fi - -get_ptxdist_var(){ - grep $1 $M2B_ENV | while read a b ; do echo $b ; done; -} - -logdir="`get_ptxdist_var PTXDIST_WORKSPACE`/log" -logfile="$logdir/url_check_ng_log" - -# ========================================================== -# Temporary files -# ========================================================== - -TMPDIR="`$MKTEMP -d /tmp/url_check_plugin.XXXXXXXXXX`" || echo "could not create TMPDIR" - -ptx_debug "TMPDIR is $TMPDIR" - -# ========================================================== -# Traps -# ========================================================== - -[ -e "$TMPDIR/on_exit_reverse.sh" ] || echo "echo" > $TMPDIR/on_exit_reverse.sh - -if [ "$DEBUG" = "true" ]; then - ptx_debug "trap function: deleting temporary files..." - trap '[ -e "$TMPDIR/on_exit_reverse.sh" ] && sh $TMPDIR/on_exit_reverse.sh ; rm -rvf /tmp/$(basename $TMPDIR)' EXIT -else - trap '[ -e "$TMPDIR/on_exit_reverse.sh" ] && sh $TMPDIR/on_exit_reverse.sh ; rm -rf /tmp/$(basename $TMPDIR)' EXIT -fi - -# ---------------------------------------------------------- -# Default Dependency check -# ---------------------------------------------------------- - -dependency_check_dirs_depends="" -dependency_check_files_depends="$PTXCONFIG" -dependency_check_tools_depends="$LINKCHECKER_BIN $MKTEMP" - -ptx_dependency_check - -# ========================================================== -# Option Parser -# ========================================================== - -Usage() { -cat <<-EOF - -Usage: `basename "$0"` OPTIONS - - --help, -h this help - --check-all, -a check ALL packages - -$0 checks the availability of all needed source packages -for the current project configuration. By default, packages -will only be checked, if the corresponding switch in -ptxdistrc is set to "y" (enabled in 'ptxdist menuconfig') - -EOF -} - -# Parser -# ------ -# option no argument -# option: required argument -# option:: optional argument - -TEMP=`getopt --options h,a \ - --longoptions="help,check-all" \ - -n "$0" -- "$@"` -if [ $? != 0 ] ; then echo "Terminating..." >&2 ; exit 1 ; fi -eval set -- "$TEMP" - -while true ; do - case "$1" in - -h|--help) - [ -z "$action" ] - action="help" ; - shift - ;; - -a|--check-all) - [ -z "$action" ] - action="check_all" ; - shift - ;; - --) shift ; break ;; - *) echo "Internal error!" ; exit 1 ;; - esac -done - -# ========================================================== -# Script Variables -# ========================================================== - -# none - -# ========================================================== -# Script Functions -# ========================================================== - -init(){ - # choose the right ptxdist version: - PTXDIST=$(echo `$GREP PTXCONF_CONFIGFILE_VERSION $PTXCONFIG` \; echo "ptxdist-\$PTXCONF_CONFIGFILE_VERSION" | sh) - PTXDIST_BIN="`$WHICH $PTXDIST || $WHICH ptxdist || ptx_bailout 'NO PTXDIST FOUND'`" - ptx_debug "PTXDIST is: $PTXDIST ($PTXDIST_BIN)" - # init package label list if it does not exist - [ -e "${M2B_ENV}" ] || $PTXDIST_BIN make dump || ptx_bailout "could not dump environment" - while read line content; do - case $line in - "PACKAGES-y"*) - # we don�t want to see kernel- and u-boot urls, so we just skip these packages - YESPACKAGES=`echo "$content" | sed -e s/kernel\ //g -e s/u-boot\ //g | tr "a-z-" "A-Z_"` - ;; - "PACKAGES-"*) - NOPACKAGES=`echo "$content" | sed -e s/kernel\ //g -e s/u-boot\ //g | tr "a-z-" "A-Z_"` - ;; - esac - done < ${M2B_ENV} - if [ "$1" = "all" ]; then - PACKAGES="$YESPACKAGES $NOPACKAGES" - else - PACKAGES="$YESPACKAGES" - fi -} - -create_url_list(){ - echo "creating list of download URLS for `echo $PACKAGES | wc -w` packages" - for target in $PACKAGES; do - grep "^${target}_URL\ " ${M2B_ENV} | while read name location; do - if [ -n "$location" ]; then - echo "$location" >> $TMPDIR/urllist - echo "<a href=\"$location\">PTXdist Target ${target}</a><br>" >> $TMPDIR/urllist.html - fi - done - echo -n "." - done - echo "done" -} - -test_urls(){ - if [ "$LINKCHECKER_FASTMODE" = "true" ] ; then - $URL_CHECKER -t${LINKCHECKER_THREADS:=10} $TMPDIR/urllist.html > $TMPDIR/errormsg - egrep -v "^URL|^Parent\ URL|^Check\ Time" $TMPDIR/errormsg | sed -n '/Start/,$p' - else - egrep -v "^#|^$|^[[:space:]]" $TMPDIR/urllist \ - | while read line; do - $URL_CHECKER $line >$TMPDIR/errormsg 2>&1 - case $? in - 0) - echo " [ OK ] $line" - ;; - *) - echo "" - echo " [ FAIL ] $line" - echo "" - $CAT $TMPDIR/errormsg | $GREP -A 10 "Start checking at" - echo "" - ;; - esac - done - fi -} - -runner(){ - $MKDIR -p $logdir - case $? in - 0) - test_urls | tee $logfile - echo "LOGFILE is --> $logfile" - ;; - *) - test_urls - ;; - esac -} - -# ========================================================== -# Script Main -# ========================================================== - -case "$action" in - help) - Usage - ;; - check_all) - init all - create_url_list >&2 - runner - ;; - *) - init - create_url_list >&2 - runner - ;; -esac - -# ========================================================== -# Cleanup -# ========================================================== - -# unconfigured - done by trap function -# ---------------------------------------------------------- -# End -# ---------------------------------------------------------- diff --git a/plugins/url_check_ng/ptxlib.bash b/plugins/url_check_ng/ptxlib.bash deleted file mode 100644 index 50249916d2b7..000000000000 --- a/plugins/url_check_ng/ptxlib.bash +++ /dev/null @@ -1,351 +0,0 @@ -#!/bin/bash -# ----------------------------------------------------- -# Description: Basic Script Functions -# Author: Bj�rn B�rger <b.buer...@pengutronix.de> -# Date: Wed Mar 21 17:03:39 CET 2007 -# ----------------------------------------------------- - -PTXLIB="true" -MY_VERSION="2" - -if [ "$PTX_LIB_VERSION" -lt "$MY_VERSION" ] ; then - echo "WARNING: ptxlib.bash not compatible" - echo "WARNING: please audit your script" - exit 1 -fi - -FULLARGS="$@" -DEBUG=${DEBUG:="false"} - -# -# customized exit functions -# -# $1 --> Error Message -# $2 --> Exit Code -# -ptx_exit(){ - echo "$0: $1" - exit $2 -} -ptx_exit_silent(){ - ptx_debug "$0: $1" - exit $2 -} - -# -# print out error message and exit with status 1 -# -# $1: error message -# ${PREFIX}: to be printed before message -# -ptx_bailout () { - echo "${PREFIX}error: $1" >&2 - exit 1 -} - - -# -# print out warning message -# -# $1: warning message -# ${PREFIX}: to be printed before message -# -ptx_warning() { - echo "${PREFIX}warning: $1" >&2 -} - -# -# print out classic message -# -# $1: warning message -# ${PREFIX}: to be printed before message -# -ptx_message() { - echo "${PREFIX}: $1" >&1 -} - -ptx_message_n() { - if [ -n "$PREFIX" ]; then - echo -n "${PREFIX}: $1" >&1 - else - echo -n "$1" >&1 - fi -} - -# -# Debugging Output -# -ptx_debug(){ - [ "$DEBUG" = "true" ] && echo "$0: $1" >&2 -} -ptx_debug "Debugging is enabled - Turn off with DEBUG=false" - - -# -# check if a previously executed pipe returned an error -# -ptx_check_pipe_status() { - for i in "${PIPESTATUS[@]}"; do [ $i -gt 0 ] && { - echo - echo "error: a command in the pipe returned $i, bailing out" - echo - exit $i - } - done -} - -# -# present a choice [y/n] and execute command -# -# $1: question / message -# $2: command -# $3: yes message -# $4: no message -# ${PREFIX}: to be printed before message -# -# -ptx_yesno_choice(){ - ptx_debug "Message: $1" - ptx_debug "Command: $2" - dialog --yesno "$1" 0 0 - case $? in - 0) - ptx_message "${PREFIX} [YES] - $3" - $2 || ptx_error "command $2 failed" - ;; - *) - ptx_warning "${PREFIX} [NO] - $4" - ;; - esac -} - -# -# create a directory -# -ptx_check_create_dir(){ - [ -d "$1" ] || ptx_yesno_choice "$1 does not exist, create it?" "mkdir -pv $1" "[SUCCESS]" "[ABORTED]" -} - -# -# dependency check for needed files -# -# if any one of these variables is -# set, a dependency check is performed: -# -# dependency_check_files_recommends -# dependency_check_files_depends -# dependency_check_files_conflicts -# -ptx_dependency_check_files(){ - [ -z "$dependency_check_files_recommends" ] && ptx_debug "[file check] no file recommendation defined" - for file in $dependency_check_files_recommends; do - test -e $file - case $? in - 0) - ptx_debug "[file check] This Script recommends $file: OK" - ;; - *) - ptx_warning "[file check] This Script recommends $file: FILE NOT FOUND" - ;; - esac - done - [ -z "$dependency_check_files_depends" ] && ptx_debug "[file check] no file dependency defined" - for file in $dependency_check_files_depends; do - test -e $file - case $? in - 0) - ptx_debug "[file check] This Script depends on $file: OK" - ;; - *) - ptx_bailout "[file check] This Script depends on $file: FILE NOT FOUND" - ;; - esac - done - [ -z "$dependency_check_files_conflicts" ] && ptx_debug "[file check] no file conflict defined" - for file in $dependency_check_files_conflicts; do - test -e $file - case $? in - 0) - ptx_bailout "[file check] This Script conflicts with $file: FILE EXISTS" - ;; - *) - ptx_debug "[file check] This Script conflicts with $file: OK (does not exist)" - ;; - esac - done -} - -# -# dependency check for needed directories -# -# if any one of these variables is -# set, a dependency check is performed: -# -# dependency_check_dirs_recommends -# dependency_check_dirs_depends -# dependency_check_dirs_conflicts -# -ptx_dependency_check_dirs(){ - [ -z "$dependency_check_dirs_recommends" ] && ptx_debug "[dir check] no directory recommendation defined" - for directory in $dependency_check_dirs_recommends; do - test -d $directory - case $? in - 0) - ptx_debug "[dir check] This Script recommends $directory: OK" - ;; - *) - ptx_warning "[dir check] This Script recommends $directory: directory NOT FOUND" - ;; - esac - done - [ -z "$dependency_check_dirs_depends" ] && ptx_debug "[dir check] no directory dependency defined" - for directory in $dependency_check_dirs_depends; do - test -d $directory - case $? in - 0) - ptx_debug "[dir check] This Script depends on $directory: OK" - ;; - *) - ptx_bailout "[dir check] This Script depends on $directory: directory NOT FOUND" - ;; - esac - done - [ -z "$dependency_check_dirs_conflicts" ] && ptx_debug "[dir check] no directory conflict defined" - for directory in $dependency_check_dirs_conflicts; do - test -d $directory - case $? in - 0) - ptx_bailout "[dir check] This Script conflicts with $directory: directory EXISTS" - ;; - *) - ptx_debug "[dir check] This Script conflicts with $directory: OK (does not exist)" - ;; - esac - done -} - -# -# dependency check for needed tools -# -# if any one of these variables is -# set, a dependency check is performed: -# -# dependency_check_tools_recommends -# dependency_check_tools_depends -# dependency_check_tools_conflicts -# -ptx_dependency_check_tools(){ - [ -z "$dependency_check_files_recommends" ] && ptx_debug "[file check] no tool recommendation defined" - for tool in $dependency_check_tools_recommends; do - which $tool >/dev/null 2>&1 - case $? in - 0) - ptx_debug "[tool check] This Script recommends $tool: OK" - ;; - *) - ptx_warning "[tool check] This Script recommends $tool: MISSING" - ;; - esac - done - [ -z "$dependency_check_tools_depends" ] && ptx_debug "[file check] no tool dependency defined" - for tool in $dependency_check_tools_depends; do - which $tool >/dev/null 2>&1 - case $? in - 0) - ptx_debug "[tool check] This Script depends on $tool: OK" - ;; - *) - ptx_bailout "[tool check] This Script depends on $tool: MISSING" - ;; - esac - done - [ -z "$dependency_check_tools_conflicts" ] && ptx_debug "[file check] no file conflict defined" - for tool in $dependency_check_tools_conflicts; do - which $tool >/dev/null 2>&1 - case $? in - 0) - ptx_bailout "[tool check] This Script conflicts with on $tool: CONFLICT FOUND" - ;; - *) - ptx_debug "[tool check] This Script conflicts with $tool: OK" - ;; - esac - done -} - -ptx_dependency_check_hostname(){ - [ -z "$dependency_check_hostname_recommends" ] && ptx_debug "[hostname check] no hostname recommendation defined" - for hostname in $dependency_check_hostname_recommends; do - [ "`hostname`" = "$hostname" ] >/dev/null 2>&1 - case $? in - 0) - ptx_debug "[hostname check] This Script recommends host $hostname: OK" - ;; - *) - ptx_warning "[hostname check] This Script recommends host $hostname: THIS IS NOT ME" - ;; - esac - done - [ -z "$dependency_check_hostname_depends" ] && ptx_debug "[hostname check] no hostname dependency defined" - for hostname in $dependency_check_hostname_depends; do - [ "`hostname`" = "$hostname" ] >/dev/null 2>&1 - case $? in - 0) - ptx_debug "[hostname check] This Script depends on host $hostname: OK" - ;; - *) - ptx_bailout "[hostname check] This Script depends on host $hostname: THIS IS NOT ME" - ;; - esac - done - [ -z "$dependency_check_hostname_conflicts" ] && ptx_debug "[hostname check] no hostname conflict defined" - for hostname in $dependency_check_hostname_conflicts; do - [ "`hostname`" = "$hostname" ] >/dev/null 2>&1 - case $? in - 0) - ptx_bailout "[hostname check] This Script conflicts with host $hostname: CONFLICT FOUND" - ;; - *) - ptx_debug "[hostname check] This Script conflicts with host $hostname: OK" - ;; - esac - done -} - -# -# convenience wrapper for the functions above -# -# performs all available checks -# -ptx_dependency_check(){ - ptx_dependency_check_hostname - ptx_dependency_check_dirs - ptx_dependency_check_files - ptx_dependency_check_tools -} - -# service specific lib functions, based on script domains: -# -# - dumb_tool -# - tool -# - development -# - system_management -# - -if [ -n "$PTX_SCRIPT_DOMAINS" ]; then -for ptx_script_domain in $PTX_SCRIPT_DOMAINS; do - ptx_debug "loading config for domain: ${ptx_script_domain}" - SCONFS="$HOME/.ptxsd-${ptx_script_domain}.conf \ - /usr/local/etc/ptxsd-${ptx_script_domain}.conf \ - /etc/ptxsd-${ptx_script_domain}.conf \ - `dirname $0`/ptxsd-${ptx_script_domain}.conf \ - `dirname $0`/../lib/ptxsd-${ptx_script_domain}.conf" - for scriptconf in $SCONFS; do - if [ -e "$scriptconf" ] ; then - . $scriptconf && { ptx_debug "$scriptconf loaded"; break; } - fi - done - [ "$PTXSD" = "true" ] || ptx_warning " No valid Configuration for Domain: ${ptx_script_domain} found \ - [ $conf (version $MY_VERSION) ] [ caller: $0 ]" -done -fi diff --git a/plugins/url_check_ng/robotparser2.py.diff b/plugins/url_check_ng/robotparser2.py.diff deleted file mode 100644 index feb8a77ba09c..000000000000 --- a/plugins/url_check_ng/robotparser2.py.diff +++ /dev/null @@ -1,44 +0,0 @@ -linkchecker patch ------------------ - -The following patch is a quick hack to prevent robots.txt -restrictions from taking effect. This is somewhat ugly, but -we need to *know exactly* if a source package is accessible -or has moved. So we explicitly don't want to honor those -restrictions. Unfortunately, the --robots-txt option is -enabled per default and you cannot disable it from the -command line. - -Since this hack is generally a bad thing, please behave -reasponsible: Don't run url_checks too often (or at -least ask the mirror maintainers for approval to do so). -A server maintainer who doesn't want spiders to index -his machine might have his reasons... ;-) - -bbu, 20071019 - ----------------------------------------------------------------------------- - ---- robotparser2.py.orig 2007-10-19 14:59:06.000000000 +0200 -+++ robotparser2.py 2007-10-19 15:03:28.000000000 +0200 -@@ -344,19 +344,9 @@ - if not isinstance(url, str): - url = url.encode("ascii", "ignore") - if self.disallow_all: -- return False -+ return True - if self.allow_all: - return True -- # search for given user agent matches -- # the first match counts -- url = urllib.quote(urlparse.urlparse(urllib.unquote(url))[2]) or "/" -- for entry in self.entries: -- if entry.applies_to(useragent): -- return entry.allowance(url) -- # try the default entry last -- if self.default_entry is not None: -- return self.default_entry.allowance(url) -- # agent not found ==> access granted - return True - - def get_crawldelay (self, useragent): diff --git a/plugins/url_check_ng/robotparser2.py.readme b/plugins/url_check_ng/robotparser2.py.readme deleted file mode 100644 index a0504219749b..000000000000 --- a/plugins/url_check_ng/robotparser2.py.readme +++ /dev/null @@ -1 +0,0 @@ -/usr/share/pycentral/linkchecker/site-packages/linkcheck -- 2.29.2
_______________________________________________ ptxdist mailing list ptxdist@pengutronix.de To unsubscribe, send a mail with subject "unsubscribe" to ptxdist-requ...@pengutronix.de
