Some ports have huge distfiles of which we only need a tiny fraction,
see the coreboot-utils thread.
My idea was to pass a list of files to the extract phase and it works
just fine as seen in the diff below, based on this EXTRACT_FILES has no
effect for .shar or .gz distfiles, but I don't see this as an issue.
Technically, EXTRACT_FILES is a list of glob patterns read by tar(1).
Diff below has the bsd.port.mk(5) bits as well as two examples:
coreboot-utils dropping WRKDIR size from 205M to 7M and iwm-firmware
reduced from 408M to 93M - no PLIST changes.
Passing EXTRACT_FILES as is has the advantage of using sh(1)'s brace
expansion as shown in the diff, using make's :QL would have the
advantage of not preventing porters from passing bogus values, e.g.
passing unquoted or unescaped asterisks which would mistakenly be
interpreted by sh(1) instead of tar(1).
I pass it as is because brace expansion makes it easy to use and
EXTRACT_FILES wouldn't be the first variable to pass unsanitized shell
code into parts of bsd.port.mk where arbitrary code is not meant to be.
Feedback?
Index: infrastructure/mk/bsd.port.mk
===================================================================
RCS file: /cvs/ports/infrastructure/mk/bsd.port.mk,v
retrieving revision 1.1534
diff -u -p -U0 -r1.1534 bsd.port.mk
--- infrastructure/mk/bsd.port.mk 20 Apr 2020 11:49:57 -0000 1.1534
+++ infrastructure/mk/bsd.port.mk 18 May 2020 10:41:40 -0000
@@ -1344,0 +1345 @@ EXTRACT_CASES ?=
+EXTRACT_FILES ?=
@@ -1353 +1354 @@ EXTRACT_CASES += *.zip) \
- ${UNZIP} -oq ${FULLDISTDIR}/$$archive -d ${WRKDIR};;
+ ${UNZIP} -oq ${FULLDISTDIR}/$$archive -d ${WRKDIR} ${EXTRACT_FILES};;
@@ -1360 +1361 @@ EXTRACT_CASES += *.tar.xz) \
- xzdec <${FULLDISTDIR}/$$archive | ${TAR} xf -;;
+ xzdec <${FULLDISTDIR}/$$archive | ${TAR} -xf - -- ${EXTRACT_FILES};;
@@ -1362 +1363 @@ EXTRACT_CASES += *.tar.lzma) \
- lzmadec <${FULLDISTDIR}/$$archive | ${TAR} xf -;;
+ lzmadec <${FULLDISTDIR}/$$archive | ${TAR} -xf - -- ${EXTRACT_FILES};;
@@ -1368 +1369 @@ EXTRACT_CASES += *.tar.lz) \
- lunzip <${FULLDISTDIR}/$$archive | ${TAR} xf -;;
+ lunzip <${FULLDISTDIR}/$$archive | ${TAR} -xf - -- ${EXTRACT_FILES};;
@@ -1378 +1379 @@ EXTRACT_CASES += *.tar.bz2|*.tbz2|*.tbz)
- ${BZIP2} -d <${FULLDISTDIR}/$$archive | ${TAR} xf -;;
+ ${BZIP2} -d <${FULLDISTDIR}/$$archive | ${TAR} -xf - --
${EXTRACT_FILES};;
@@ -1387 +1388 @@ EXTRACT_CASES += *.tar) \
- ${TAR} xf ${FULLDISTDIR}/$$archive;;
+ ${TAR} -xf ${FULLDISTDIR}/$$archive -- ${EXTRACT_FILES};;
@@ -1393 +1394 @@ EXTRACT_CASES += *.tar.gz|*.tgz) \
- ${GZIP_CMD} -d <${FULLDISTDIR}/$$archive | ${TAR} xf -;;
+ ${GZIP_CMD} -d <${FULLDISTDIR}/$$archive | ${TAR} -xf - --
${EXTRACT_FILES};;
@@ -1397 +1398 @@ EXTRACT_CASES += *) \
- ${GZIP_CMD} -d <${FULLDISTDIR}/$$archive | ${TAR} xf -;;
+ ${GZIP_CMD} -d <${FULLDISTDIR}/$$archive | ${TAR} -xf - --
${EXTRACT_FILES};;
Index: sysutils/firmware/iwm/Makefile
===================================================================
RCS file: /cvs/ports/sysutils/firmware/iwm/Makefile,v
retrieving revision 1.13
diff -u -p -r1.13 Makefile
--- sysutils/firmware/iwm/Makefile 18 Nov 2019 11:36:56 -0000 1.13
+++ sysutils/firmware/iwm/Makefile 18 May 2020 10:41:46 -0000
@@ -5,6 +5,7 @@ FW_VER= 20191022
REVISION= 0
DISTNAME= linux-firmware-${FW_VER}
EXTRACT_SUFX= .tar.xz
+EXTRACT_FILES= ${DISTNAME}/{LICENCE.iwlwifi_firmware,iwlwifi-\*-\*.ucode}
HOMEPAGE= https://wireless.wiki.kernel.org/en/users/Drivers/iwlwifi
MASTER_SITES= https://cdn.kernel.org/pub/linux/kernel/firmware/
Index: sysutils/coreboot-utils/Makefile
===================================================================
RCS file: /cvs/ports/sysutils/coreboot-utils/Makefile,v
retrieving revision 1.3
diff -u -p -r1.3 Makefile
--- sysutils/coreboot-utils/Makefile 14 May 2020 19:20:27 -0000 1.3
+++ sysutils/coreboot-utils/Makefile 18 May 2020 10:26:38 -0000
@@ -24,6 +24,8 @@ WANTLIB = c
MASTER_SITES = https://coreboot.org/releases/
EXTRACT_SUFX = .tar.xz
+# extract only what's needed (7M) instead of everything (205M)
+EXTRACT_FILES = ${DISTNAME}/{src/commonlib,util}
USE_GMAKE = Yes
