On Thu Sep 3, 2020 at 3:36 AM HST, Adrian Schmutzler wrote: > > -----Original Message----- > > From: openwrt-devel [mailto:openwrt-devel-boun...@lists.openwrt.org] > > On Behalf Of Paul Spooren > > Sent: Freitag, 31. Juli 2020 05:15 > > To: openwrt-devel@lists.openwrt.org > > Cc: freif...@adrianschmutzler.de; Paul Spooren <m...@aparcar.org> > > Subject: [PATCH] scripts: add size_compare.sh > > > > As package size changes are a continuous topic on the mailing list this > > scripts > > helps developers to compare their local package modifications against latest > > upstream. > > > > The script downloads the latest package indexes based on env variables or > > the `.config` file. The script compares the actual installed size > > (data.tar.gz) or the IPK package size. > > Just gave this a quick try, and generally it looks good. > > However, I got 82 packages listed for " Compare packages of > ath79/generic/mips_24kc", but only 66 were in the final comparison.
The current script also interprets some options as packages, which is couldn't find a clean solution for: CONFIG_PACKAGE_MAC80211_DEBUGFS=y # CONFIG_PACKAGE_MAC80211_TRACING is not set CONFIG_PACKAGE_MAC80211_MESH=y Also, this script only checks for packages of openwrt.git, so if you have packages from other repositories installed, they'd be missing. > > Will check the code in detail later. A few comment below. > > > > > An example output is found below: > > > > ``` > > user@dawn:~/src/openwrt/openwrt$ ./scripts/size_compare.sh Compare > > packages of ath79/tiny/mips_24kc: > > dropbear busybox iw ubus > > > > Checking installed size > > > > Fetching latest package indexes... > > % Total % Received % Xferd Average Speed Time Time Time > > Current > > Dload Upload Total Spent Left > > Speed > > 100 80634 100 80634 0 0 33499 0 0:00:02 0:00:02 --:--:-- > > 33485 > > % Total % Received % Xferd Average Speed Time Time Time > > Current > > Dload Upload Total Spent Left > > Speed > > 100 54082 100 54082 0 0 24252 0 0:00:02 0:00:02 --:--:-- > > 24252 > > > > Comparing package sizes... > > Change Local Package > > 611B 208715B busybox > > 39B 5612B ubus > > -42B 34940B iw > > -14916B 89265B dropbear > > ``` > > > > I plan to integrate this script into the CI so we have a summary how sizes > > change over different architectures. > > > > Signed-off-by: Paul Spooren <m...@aparcar.org> > > --- > > scripts/size_compare.sh | 117 > > ++++++++++++++++++++++++++++++++++++++++ > > 1 file changed, 117 insertions(+) > > create mode 100755 scripts/size_compare.sh > > > > diff --git a/scripts/size_compare.sh b/scripts/size_compare.sh new file mode > > 100755 index 0000000000..b310a085a4 > > --- /dev/null > > +++ b/scripts/size_compare.sh > > @@ -0,0 +1,117 @@ > > +# SPDX-License-Identifier: GPL-2.0-or-later # # Copyright (C) 2020 Paul > > +Spooren <m...@aparcar.org> # ### ### size_compare - compare size of > > +OpenWrt packages against upstream ### ### The script compares locally > > +compiled package with the package indexes ### available upstream. This > > +way the storage impact of optimizations or ### feature modifiactions is > > +easy to see. > > +### > > +### If no environmental variables are set the scritp reads the current > > +### .config file. The evaluated env variables are the following: > > +### > > +### TARGET SUBTARGET ARCH PACKAGES BIN_DIR BASE_URL > > CHECK_INSTALLED > > +### > > +### Usage: > > +### ./scripts/size_compare.sh > > +### > > +### Options: > > +### -p --package-size Check IPK package size and not installed size > > +### -h --help This message > > + > > +CONFIG_TARGET=$(sed -n 's/^CONFIG_TARGET_BOARD="\(.*\)"$/\1/p' > > .config) > > +CONFIG_SUBTARGET=$(sed -n > > 's/^CONFIG_TARGET_SUBTARGET="\(.*\)"$/\1/p' > > +.config) CONFIG_ARCH=$(sed -n > > +'s/^CONFIG_TARGET_ARCH_PACKAGES="\(.*\)"$/\1/p' .config) > > +CONFIG_PACKAGES=$(sed -n 's/^CONFIG_PACKAGE_\(.*\)=y$/\1/p' > > .config | > > +tr '\n' ' ') CONFIG_BIN_DIR=$(sed -n > > +'s/^CONFIG_BINARY_DIR="\(.*\)"$/\1/p' .config) > > + > > +TARGET=${TARGET:-$CONFIG_TARGET} > > +SUBTARGET=${SUBTARGET:-$CONFIG_SUBTARGET} > > +ARCH=${ARCH:-$CONFIG_ARCH} > > +PACKAGES=${PACKAGES:-$CONFIG_PACKAGES} > > +BIN_DIR=${CONFIG_BIN_DIR:-./bin} > > +BASE_URL="${BASE_URL:-https://downloads.openwrt.org/snapshots}" > > +CHECK_INSTALLED="${CHECK_INSTALLED:-y}" > > + > > +TARGET_URL="$BASE_URL/targets/$TARGET/$SUBTARGET/packages/Pack > > ages.gz" > > +PACKAGES_URL="$BASE_URL/packages/$ARCH/base/Packages.gz" > > + > > +help() { > > + sed -rn 's/^### ?//;T;p' "$0" > > +} > > + > > +package_size () { > > + FOUND_PACKAGE= > > + if [ -z "$CHECK_INSTALLED" ]; then > > + SEARCH_PATTERN="Size" > > + else > > + SEARCH_PATTERN="Installed-Size" > > + fi > > + while IFS= read -r line; do > > + if [ "$line" = "Package: $2" ]; then > > + FOUND_PACKAGE=y > > + fi > > + if [ -n "$FOUND_PACKAGE" ]; then > > + case $line in > > + "$SEARCH_PATTERN"*) > > + echo "$line" | cut -d ' ' -f 2 > > + break > > + ;; > > + esac > > + fi > > + done < "$1" > > +} > > + > > +compare_sizes () { > > + for PACKAGE in $PACKAGES; do > > + if [ "$PACKAGE" = "libc" ]; then > > + continue > > + fi > > + PACKAGE_FILE=$(find "$BIN_DIR/packages/$ARCH/" > > "$BIN_DIR/targets/$TARGET/$SUBTARGET/" -name "${PACKAGE}_*.ipk") > > When I first run this, I was in x86 target where only a kernel refresh > has happened before (so, no build). > This then produces a lot of warning due to missing packages folder: > > find: ‘./bin/targets/x86/generic/’: No such file or directory > find: ‘./bin/packages/i386_pentium4/’: No such file or directory > > So, one should either suppress the message here for the find command, or > alternatively check for the folders' existence beforehand. I surpressed the error message. > > > + if [ -z "$PACKAGE_FILE" ]; then > > + continue > > + fi > > + if [ -z "$CHECK_INSTALLED" ]; then > > + SIZE_LOCAL=$(stat -c '%s' "$PACKAGE_FILE") > > + else > > + SIZE_LOCAL=$(tar tzvf "$PACKAGE_FILE" | grep > > data.tar.gz | awk '{ print $3 }') > > + fi > > + SIZE_UPSTREAM=$(package_size "$TMP_INDEX" > > "$PACKAGE") > > + SIZE_DIFF="$((SIZE_LOCAL-SIZE_UPSTREAM))" > > + echo "${SIZE_DIFF}B ${SIZE_LOCAL}B $PACKAGE" > > Personally, I think the "B" looks ugly. As we only have bytes, we could > consider to drop it. Dropped. > One could also consider to add a "+" for the first column when change is > positive. Both changes here are lowest priority. Added. I created a updated version on GitHub as I'm not the biggest fan of patchworks: https://github.com/openwrt/openwrt/pull/3599 > > + done > > +} > > + > > +if [ "$1" = "-h" ]; then > > + help > > + exit 0 > > +fi > > + > > +if [ "$1" = "-p" ]; then > > + CHECK_INSTALLED= > > +fi > > + > > +echo "Compare packages of $TARGET/$SUBTARGET/$ARCH": > > +echo "$PACKAGES" > > +echo > > + > > +if [ -z "$CHECK_INSTALLED" ]; then > > + echo "Checking IPK package size" > > +else > > + echo "Checking installed size" > > +fi > > +echo > > + > > +echo "Fetching latest package indexes..." > > +TMP_INDEX=$(mktemp /tmp/size_compare_package_index.XXXXXX) > > +curl "$TARGET_URL" | gzip -d > "$TMP_INDEX" || exit 1 curl > > +"$PACKAGES_URL" | gzip -d >> "$TMP_INDEX" || exit 1 echo > > + > > +echo "Comparing package sizes..." > > +echo "Change Local Package" > > Since we have enough space, I'd go for "Change Local Remote Package" > here. > > Best > > Adrian > > > +compare_sizes | sort -n -r > > + > > +rm "$TMP_INDEX" > > -- > > 2.25.1 > > > > > > _______________________________________________ > > openwrt-devel mailing list > > openwrt-devel@lists.openwrt.org > > https://lists.openwrt.org/mailman/listinfo/openwrt-devel _______________________________________________ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel