On Mon, Aug 07, 2023 at 09:17:05PM +0100, Stuart Henderson wrote: [...]
> I think maybe I'd prefer to have some variable that could be used > *instead* of the existing GH_* variables rather than in conjunction with > them (so they can be used for all GH archive ports, rather than have > them a special case for multi-distfile ports). If that's the standard > way to do things we can have a sweep of the tree converting other > ports (or at least the ones that don't use go.port.mk ;) > > It would be kind-of helpful if it could support more than just github > too (gitlab.com, sr.ht, ..). While that could be done with different > variables (GH_xx, GL_xx, SRHT_xx etc) they're all a similar enough > layout to each other that making the site part of the variable itself > rather than the name would be simpler and easier to add more sites > (plus it covers the case where you have some port using one file from > github and one from gitlab, etc). > > Playing with syntax ideas, maybe something like this would be easy to > use for pprts not needing a rename - > > SOMEVAR+= github vim vim refs/tags/v9.0.1677 > SOMEVAR+= github vim colorschemes 22986fa2a3d2f7229efd4019fcbca411caa6afbb > > or with some auto-renaming (and specifying more of the path to avoid the > extra GH_WRKSRC which I think might not be enough in some cases anyway - > a port may have several distfiles that need to go into different base > dirs) - > > SOMEVAR+= github fortran-lang fpm refs/tags/v0.7.0 > OTHERVAR+= github toml-f toml-f e49f5523e4ee67db6628618864504448fb8c8939 > vendor/toml-f > OTHERVAR+= github urbanjost M_CLI2 > 90a1a146e19c8ad37b0469b8cbd04bc28eb67a50 vendor/M_CLI2 > > (no idea what to use as real names instead of SOMEVAR/OTHERVAR though!) > > How does that sort of thing seem to you? (i.e. using the same basic idea as > you have for submodules, but making it the standard for all gh distfiles)? I ran with your suggestion and came up with a solution that I've named distexpand. The idea is to use templates for commonly used, automatically generated and therefore predictably named, stored, and packaged dist files. 2 variables take different arguments/parts that are 'expanded' with the template to working MASTER_SITESn and DISTFILES. The current configuration in the ports Makefile is done like this, after putting distexpand.port.mk into /usr/ports/infrastructure/mk/: MODULES += distexpand DISTEXPAND += template account1 project1 id1(commithash/tag) DISTEXPANDX += template account2 project2 id2(commithash/tag) targetdir 'template' is currently set up for github, gitlab, and sourcehut. You can use multiple DISTEXPAND and DISTEXPANDX as needed. This will _not_ use up more MASTER_SITESn, as long as the template stays the same. Regarding the naming, I'm definitely open to discuss other suggestions. DISTEXPAND is what I've been able to think of that most clearly conveys the use of the fragments that are expanded to a full address for fetching the distfile. DISTEXPANDX - the last 'X' is meant to stand for 'extended' as this is the version that relocates the extracted files to a target dir. I'm slightly partial to consider naming the variables instead 'DISTEXPAND4' and 'DISTEXPAND5' which would remind the porter of the number of components for each version. For the templating, I used %account, %project, %id, %subdir as the placeholders. Those are substituted later with :S. I'm open to suggestions if there may be a more established pattern for placeholders in strings in Makefile context. This can replace GH_{ACCOUNT,PROJECT,TAGNAME,COMMIT}. Tags are detected as such, and in that case a DISTNAME will be set to $project-$tag if not otherwise set. In other scenarios, a DISTNAME or PKGNAME may need to be set. A couple of other things to note compared to before: - GH_WRKSRC is gone without replacement. Its usefulness was questionable. - It includes logic that finds the first MASTER_SITESn that isn't otherwise used, and throws an ERROR if it overruns past MASTER_SITES9. - Using tags is now by just proving '0.1.0' or 'v0.11.2' or other non-commithash string (the heuristic checks for length to determine if this is a tag or a commit hash). - It currently uses 2 longer for-loops that are almost identical, but one for DISTEXPAND, and the other one for DISTEXPANDX. Given the limitations in Makefiles, I couldn't think of a way to reuse more code there. This doesn't need to be in a module, but this way it's easy to plug in and experiment with. I'm attaching the distexpand.port.mk, as well as the patch for using it with neovim as an example. I've tested this with about 3 dozen ports that use combinations of mostly github sites, but also a gitlab and a sourcehut dist source [1]. [1] https://thfr.info/tmp/distexpand-ports.txt
Index: Makefile =================================================================== RCS file: /cvs/ports/editors/neovim/Makefile,v retrieving revision 1.37 diff -u -p -r1.37 Makefile --- Makefile 14 Jun 2023 07:47:57 -0000 1.37 +++ Makefile 9 Aug 2023 04:53:18 -0000 @@ -11,9 +11,18 @@ EMBED_LUAJIT = No COMMENT = continuation and extension of Vim -GH_ACCOUNT = neovim -GH_PROJECT = neovim -GH_TAGNAME = v0.9.1 +DISTEXPAND += github neovim neovim v0.9.1 + +# The versions listed here must match those in cmake.deps/CMakeLists.txt. +# Move static deps source code under WRKDIST so that they can be patched. +DISTEXPANDX+= github luvit luv 093a977b82077591baefe1e880d37dfa2730bd54 \ + static-deps/luv # Apache-2.0 +DISTEXPANDX+= github LuaJIT LuaJIT 505e2c03de35e2718eef0d2d3660712e06dadf1f \ + static-deps/luajit # MIT + public domain +DISTEXPANDX+= github lunarmodules lua-compat-5.3 \ + 7b783fb8efac60de8be91522d5731a9716e83d56 \ + static-deps/lua-compat-5.3 # MIT +STATIC_DEPS_WRKSRC = ${WRKDIST}/static-deps REVISION = 0 # embedded luajit @@ -23,22 +32,7 @@ CATEGORIES = editors devel HOMEPAGE = https://neovim.io MAINTAINER = Edd Barrett <e...@openbsd.org> -# The versions listed here must match those in cmake.deps/CMakeLists.txt. -LUV_VER = 093a977b82077591baefe1e880d37dfa2730bd54 -LUAJIT_VER = 505e2c03de35e2718eef0d2d3660712e06dadf1f -LUACOMPAT_VER = v0.9 - -MASTER_SITES0 = https://github.com/luvit/luv/archive/ -MASTER_SITES1 = https://github.com/LuaJIT/LuaJIT/archive/ -MASTER_SITES2 = https://github.com/keplerproject/lua-compat-5.3/archive/ -DISTFILES = ${DISTNAME}${EXTRACT_SUFX} \ - luv-{}${LUV_VER}${EXTRACT_SUFX}:0 \ - luajit-{}${LUAJIT_VER}${EXTRACT_SUFX}:1 \ - lua-compat-5.3-{}${LUACOMPAT_VER}${EXTRACT_SUFX}:2 - -# Neovim: Apache 2.0 + Vim License -# LuaJIT: MIT + public domain -# libluv: Apache 2.0 +# Apache 2.0 + Vim License PERMIT_PACKAGE = Yes DEBUG_PACKAGES = ${BUILD_PACKAGES} @@ -52,7 +46,8 @@ WANTLIB += ${MODLUA_WANTLIB} COMPILER = base-clang ports-gcc -MODULES = devel/cmake \ +MODULES = distexpand \ + devel/cmake \ lang/lua \ textproc/intltool @@ -105,15 +100,6 @@ MODPY_BUILDDEP =No TEST_DEPENDS = shells/bash \ editors/py-neovim \ editors/py-neovim${MODPY_FLAVOR} - -# Move static deps source code under WRKDIST so that they can be patched. -STATIC_DEPS_WRKSRC=${WRKDIST}/static-deps/ -post-extract: - mkdir ${STATIC_DEPS_WRKSRC} - mv ${WRKDIR}/LuaJIT-${LUAJIT_VER} ${STATIC_DEPS_WRKSRC}/luajit - mv ${WRKDIR}/luv-${LUV_VER} ${STATIC_DEPS_WRKSRC}/luv - mv ${WRKDIR}/lua-compat-5.3-${LUACOMPAT_VER:C/^v//} \ - ${STATIC_DEPS_WRKSRC}/lua-compat-5.3 # Build LuaJIT (if required) and libluv as static libraries. # Index: distinfo =================================================================== RCS file: /cvs/ports/editors/neovim/distinfo,v retrieving revision 1.20 diff -u -p -r1.20 distinfo --- distinfo 5 Jun 2023 13:22:15 -0000 1.20 +++ distinfo 9 Aug 2023 04:53:18 -0000 @@ -1,8 +1,8 @@ -SHA256 (lua-compat-5.3-v0.9.tar.gz) = rQVUDS2WpIclu3mh3vNc9mUqTi7CY3biYXyM4rqm9BY= -SHA256 (luajit-505e2c03de35e2718eef0d2d3660712e06dadf1f.tar.gz) = Z8iDmbkBoi6aI29Ld+b+Oa8A9rcUTOndb1EUHZIfEHY= -SHA256 (luv-093a977b82077591baefe1e880d37dfa2730bd54.tar.gz) = Iis4tkJfCSYhjhTn2oFIH93m+WYMH+rCWlPm+1LohuY= -SHA256 (neovim-0.9.1.tar.gz) = jbF8Kh9HdtzaAOWUieoNmLqC99Go6gMoHWQOWNijoA4= -SIZE (lua-compat-5.3-v0.9.tar.gz) = 53599 -SIZE (luajit-505e2c03de35e2718eef0d2d3660712e06dadf1f.tar.gz) = 1075264 -SIZE (luv-093a977b82077591baefe1e880d37dfa2730bd54.tar.gz) = 127241 -SIZE (neovim-0.9.1.tar.gz) = 11567358 +SHA256 (LuaJIT-LuaJIT-505e2c03de35e2718eef0d2d3660712e06dadf1f.tar.gz) = Z8iDmbkBoi6aI29Ld+b+Oa8A9rcUTOndb1EUHZIfEHY= +SHA256 (lunarmodules-lua-compat-5.3-7b783fb8efac60de8be91522d5731a9716e83d56.tar.gz) = 6JL6QFiIxNX1DS4fX3EXDw6sTmBnWdjJgLXbxep/yq8= +SHA256 (luvit-luv-093a977b82077591baefe1e880d37dfa2730bd54.tar.gz) = Iis4tkJfCSYhjhTn2oFIH93m+WYMH+rCWlPm+1LohuY= +SHA256 (neovim-neovim-v0.9.1.tar.gz) = jbF8Kh9HdtzaAOWUieoNmLqC99Go6gMoHWQOWNijoA4= +SIZE (LuaJIT-LuaJIT-505e2c03de35e2718eef0d2d3660712e06dadf1f.tar.gz) = 1075264 +SIZE (lunarmodules-lua-compat-5.3-7b783fb8efac60de8be91522d5731a9716e83d56.tar.gz) = 53632 +SIZE (luvit-luv-093a977b82077591baefe1e880d37dfa2730bd54.tar.gz) = 127241 +SIZE (neovim-neovim-v0.9.1.tar.gz) = 11567358 Index: patches/patch-static-deps_luajit_src_lj_arch_h =================================================================== RCS file: /cvs/ports/editors/neovim/patches/patch-static-deps_luajit_src_lj_arch_h,v retrieving revision 1.3 diff -u -p -r1.3 patch-static-deps_luajit_src_lj_arch_h --- patches/patch-static-deps_luajit_src_lj_arch_h 4 Jul 2022 14:49:03 -0000 1.3 +++ patches/patch-static-deps_luajit_src_lj_arch_h 9 Aug 2023 04:53:18 -0000 @@ -5,7 +5,7 @@ ignore the gcc version check ifdef hell Index: static-deps/luajit/src/lj_arch.h --- static-deps/luajit/src/lj_arch.h.orig +++ static-deps/luajit/src/lj_arch.h -@@ -432,7 +432,7 @@ +@@ -442,7 +442,7 @@ /* -- Checks for requirements --------------------------------------------- */ /* Check for minimum required compiler versions. */
# DISTEXPAND: basic form, 4 elements, doesn't move files after extraction # Syntax: # DISTEXPAND += template account project id(commit/tag) # license # # Examples: # DISTEXPAND += github vim vim v9.0.1677 # VIM License / donation-ware # DISTEXPAND += gitlab Mr_Goldberg goldberg_emulator \ # 475342f0d8b2bd7eb0d93bd7cfdd61e3ae7cda24 # LGPLv3 DISTEXPAND ?= # DISTEXPANDX: extended, 5 elements, last one is target directory to move files to # Syntax: # DISTEXPANDX += template account project id(commit/tag) targetdir # license # # Example: # DISTEXPANDX += github FNA-XNA FNA.NetStub ebff244074bb3c28aeeb8cf7b383b5a029d7e28d \ # ../FNA.NetStub # Ms-PL # # Caveats: # If DISTNAME isn't set and a tag is used for id, project-tag will be # set as DISTNAME. # If neither DISTNAME nor WRKDIST/WRKSRC are set in Makefile, this might # extract into WRKDIR. DISTEXPANDX ?= # Templates: # %account # %project # %id: commit hash/tag # GitHub MASTER_SITES_github = https://github.com/ DISTFILES_github = %account-%project-{%account/%project/archive/%subdir}%id.tar.gz HOMEPAGE_github = https://github.com/%account/%project # GitLab MASTER_SITES_gitlab = https://gitlab.com/ DISTFILES_gitlab = %account-%project-{%account/%project/-/archive/%id/}%id.tar.gz HOMEPAGE_gitlab = https://gitlab.com/%account/%project # sr.ht MASTER_SITES_sourcehut= https://git.sr.ht/ DISTFILES_sourcehut = %account-%project-{~%account/%project/archive/}%id.tar.gz HOMEPAGE_sourcehut = https://git.sr.ht/~%account/%project # needed to work with traditional MASTER_SITES + DISTNAME .if defined(DISTNAME) && defined(MASTER_SITES) DISTFILES ?= ${DISTNAME}${EXTRACT_SUFX} .endif # abort if overrun_flag _overrun_flag = # DISTEXPAND (basic template) .if !empty(DISTEXPAND) . for _template _account _project _id in ${DISTEXPAND} # check if _template is valid and no overrun so far . if empty(MASTER_SITES_${_template}) ERRORS += "Fatal: invalid choice for distexpand: ${_template}" . elif empty(_overrun_flag) _template_master_sitesn = # check which MASTER_SITESn to use . for _n in 0 1 2 3 4 5 6 7 8 9 _overrun . if ("${_n}" == "_overrun") && empty(_template_master_sitesn) ERRORS += "Fatal: too many MASTER_SITESn needed" _overrun_flag = 1 . else . if empty(_template_master_sitesn) # reuse the MASTER_SITESn where possible . if empty(MASTER_SITES${_n}) || "${MASTER_SITES${_n}}" == "${MASTER_SITES_${_template}}" _template_master_sitesn = ${_n} . endif . endif . endif . endfor # modifications based on _id characteristics # _id: tag or commit hash? heuristic using length > 39 characters _subdir = _id_length_test = ${_id:M????????????????????????????????????????} . if empty(_id_length_test) # set DISTNAME if not done by the port, based on _project + _id tag DISTNAME ?= ${_project}-${_id:S/^v//} . if "MASTER_SITES_${_template}" == "MASTER_SITES_github" _subdir = refs/tags/ . endif . endif # set the variables for bsd.port.mk . if empty(_overrun_flag) MASTER_SITES${_template_master_sitesn} = ${MASTER_SITES_${_template}} DISTFILES +:= ${DISTFILES_${_template}:S/%account/${_account}/g:S/%project/${_project}/g:S/%id/${_id}/g:S/%subdir/${_subdir}/g}:${_template_master_sitesn} . if !empty(HOMEPAGE_${_template}) HOMEPAGE ?= ${HOMEPAGE_${_template}:S/%account/${_account}/g:S/%project/${_project}/g} . endif . endif . endif . endfor .endif # DISTEXPANDX (extended template with target directory) .if empty(_overrun_flag) && !empty(DISTEXPANDX) . for _template _account _project _id _targetdir in ${DISTEXPANDX} # check if _template is valid . if empty(MASTER_SITES_${_template}) ERRORS += "Fatal: invalid choice for distexpand: ${_template}" . elif empty(_overrun_flag) _template_master_sitesn = # check which MASTER_SITESn to use . for _n in 0 1 2 3 4 5 6 7 8 9 _overrun . if "${_n}" == "_overrun" && empty(_template_master_sitesn) ERRORS += "Fatal: too many MASTER_SITESn needed" _overrun_flag = 1 . else . if empty(_template_master_sitesn) # reuse the MASTER_SITESn where possible . if empty(MASTER_SITES${_n}) || "${MASTER_SITES${_n}}" == "${MASTER_SITES_${_template}}" _template_master_sitesn = ${_n} . endif . endif . endif . endfor # modifications based on _id characteristics # _id: tag or commit hash? heuristic using length > 39 characters _subdir = _id_length_test = ${_id:M????????????????????????????????????????} . if empty(_id_length_test) # set DISTNAME if not done by the port, based on _project + _id tag DISTNAME ?= ${_project}-${_id:S/^v//} . if "MASTER_SITES_${_template}" == "MASTER_SITES_github" _subdir = refs/tags/ . endif . endif # set the variables for bsd.port.mk . if empty(_overrun_flag) MASTER_SITES${_template_master_sitesn} = ${MASTER_SITES_${_template}} DISTFILES += ${DISTFILES_${_template}:S/%account/${_account}/g:S/%project/${_project}/g:S/%id/${_id}/g:S/%subdir/${_subdir}/g}:${_template_master_sitesn} . if !empty(HOMEPAGE_${_template}) HOMEPAGE ?= ${HOMEPAGE_${_template}:S/%account/${_account}/g:S/%project/${_project}/g} . endif . endif # fix/add post-extract target MODDISTEXPAND_post-extract += \ [[ -d ${WRKSRC}/${_targetdir} ]] && rmdir ${WRKSRC}/${_targetdir} \ || mkdir -p `dirname ${WRKSRC}/${_targetdir}` ; \ mv ${WRKDIR}/${_project}-${_id:S/refs\/tags\///:S/^v//} ${WRKSRC}/${_targetdir} ; . endif . endfor .endif