Re: Converting old py3-only ports to new-style
On 2020/02/29 16:02, Stuart Henderson wrote:
> We have a number of older ports for py3-only Python modules from before
> the simpler "FLAVOR=python3" setup that we're using now.
I should add: this is for ports producing packages with the py3- prefix
(i.e. modules used by other ports). For standalone applications that
happen to be written in python I think it makes sense to keep them
unFLAVORed and with MODPY_VERSION = ${MODPY_DEFAULT_VERSION_3}.
(Later we can flip things around and make it so the py2 ones are the
special case instead).
>
> Here's an example of a conversion:
>
> - change ../Makefile to add ,python3
>
> - bump REVISION
>
> - set FLAVOR/FLAVORS = python3, remove MODPY_VERSION =
> ${MODPY_DEFAULT_VERSION_3}
>
> - add @pkgpath so that the new package is considered as a candidate to replace
> the unflavoured version
>
> - bump REVISION and add ${MODPY_FLAVOR} to all ports depending on the one you
> are switching
>
> - ...and since I'm there anyway, use https for HOMEPAGE
>
> Index: databases/Makefile
> ===
> RCS file: /cvs/ports/databases/Makefile,v
> retrieving revision 1.390
> diff -u -p -r1.390 Makefile
> --- databases/Makefile27 Feb 2020 18:10:24 - 1.390
> +++ databases/Makefile29 Feb 2020 16:01:00 -
> @@ -163,7 +163,7 @@
> SUBDIR += py-ldap,python3
> SUBDIR += py-ldap3
> SUBDIR += py-ldap3,python3
> - SUBDIR += py-minidb
> + SUBDIR += py-minidb,python3
> SUBDIR += py-mongo
> SUBDIR += py-mongo,python3
> SUBDIR += py-mysql
> Index: databases/py-minidb/Makefile
> ===
> RCS file: /cvs/ports/databases/py-minidb/Makefile,v
> retrieving revision 1.8
> diff -u -p -r1.8 Makefile
> --- databases/py-minidb/Makefile 8 Nov 2019 14:48:10 - 1.8
> +++ databases/py-minidb/Makefile 29 Feb 2020 16:01:00 -
> @@ -5,11 +5,11 @@ COMMENT = simple SQLite3 store for Pyth
> MODPY_EGG_VERSION = 2.0.2
> DISTNAME = minidb-${MODPY_EGG_VERSION}
> PKGNAME =py-minidb-${MODPY_EGG_VERSION}
> -REVISION = 0
> +REVISION = 1
>
> CATEGORIES = databases
>
> -HOMEPAGE = http://thp.io/2010/minidb/
> +HOMEPAGE = https://thp.io/2010/minidb/
>
> # BSD
> PERMIT_PACKAGE = Yes
> @@ -18,6 +18,7 @@ MODULES = lang/python
> MODPY_PI = Yes
> MODPY_PYTEST = Yes
>
> -MODPY_VERSION = ${MODPY_DEFAULT_VERSION_3}
> +FLAVORS =python3
> +FLAVOR = python3
>
> .include
> Index: databases/py-minidb/pkg/PLIST
> ===
> RCS file: /cvs/ports/databases/py-minidb/pkg/PLIST,v
> retrieving revision 1.1.1.1
> diff -u -p -r1.1.1.1 PLIST
> --- databases/py-minidb/pkg/PLIST 1 Nov 2016 13:20:10 - 1.1.1.1
> +++ databases/py-minidb/pkg/PLIST 29 Feb 2020 16:01:00 -
> @@ -1,4 +1,5 @@
> @comment $OpenBSD: PLIST,v 1.1.1.1 2016/11/01 13:20:10 rpe Exp $
> +@pkgpath databases/py-minidb
> ${MODPY_COMMENT}lib/python${MODPY_VERSION}/site-packages/${MODPY_PYCACHE}/
>
> lib/python${MODPY_VERSION}/site-packages/${MODPY_PYCACHE}minidb.${MODPY_PYC_MAGIC_TAG}pyc
>
> lib/python${MODPY_VERSION}/site-packages/minidb-${MODPY_EGG_VERSION}-py${MODPY_VERSION}.egg-info
> Index: www/urlwatch/Makefile
> ===
> RCS file: /cvs/ports/www/urlwatch/Makefile,v
> retrieving revision 1.25
> diff -u -p -r1.25 Makefile
> --- www/urlwatch/Makefile 3 Jun 2019 16:06:58 - 1.25
> +++ www/urlwatch/Makefile 29 Feb 2020 16:01:00 -
> @@ -4,7 +4,7 @@ COMMENT= monitor webpages for updates
>
> MODPY_EGG_VERSION= 2.17
> DISTNAME=urlwatch-${MODPY_EGG_VERSION}
> -REVISION=0
> +REVISION=1
>
> CATEGORIES= www
>
> @@ -23,7 +23,7 @@ MODPY_SETUPTOOLS= Yes
>
> BUILD_DEPENDS= ${RUN_DEPENDS}
> RUN_DEPENDS= converters/py-html2text${MODPY_FLAVOR} \
> - databases/py-minidb \
> + databases/py-minidb${MODPY_FLAVOR} \
> devel/py-appdirs${MODPY_FLAVOR} \
> security/py-keyring${MODPY_FLAVOR} \
> textproc/py-cssselect${MODPY_FLAVOR} \
>
Converting old py3-only ports to new-style
We have a number of older ports for py3-only Python modules from before
the simpler "FLAVOR=python3" setup that we're using now.
Here's an example of a conversion:
- change ../Makefile to add ,python3
- bump REVISION
- set FLAVOR/FLAVORS = python3, remove MODPY_VERSION =
${MODPY_DEFAULT_VERSION_3}
- add @pkgpath so that the new package is considered as a candidate to replace
the unflavoured version
- bump REVISION and add ${MODPY_FLAVOR} to all ports depending on the one you
are switching
- ...and since I'm there anyway, use https for HOMEPAGE
Index: databases/Makefile
===
RCS file: /cvs/ports/databases/Makefile,v
retrieving revision 1.390
diff -u -p -r1.390 Makefile
--- databases/Makefile 27 Feb 2020 18:10:24 - 1.390
+++ databases/Makefile 29 Feb 2020 16:01:00 -
@@ -163,7 +163,7 @@
SUBDIR += py-ldap,python3
SUBDIR += py-ldap3
SUBDIR += py-ldap3,python3
- SUBDIR += py-minidb
+ SUBDIR += py-minidb,python3
SUBDIR += py-mongo
SUBDIR += py-mongo,python3
SUBDIR += py-mysql
Index: databases/py-minidb/Makefile
===
RCS file: /cvs/ports/databases/py-minidb/Makefile,v
retrieving revision 1.8
diff -u -p -r1.8 Makefile
--- databases/py-minidb/Makefile8 Nov 2019 14:48:10 - 1.8
+++ databases/py-minidb/Makefile29 Feb 2020 16:01:00 -
@@ -5,11 +5,11 @@ COMMENT = simple SQLite3 store for Pyth
MODPY_EGG_VERSION =2.0.2
DISTNAME = minidb-${MODPY_EGG_VERSION}
PKGNAME = py-minidb-${MODPY_EGG_VERSION}
-REVISION = 0
+REVISION = 1
CATEGORIES = databases
-HOMEPAGE = http://thp.io/2010/minidb/
+HOMEPAGE = https://thp.io/2010/minidb/
# BSD
PERMIT_PACKAGE = Yes
@@ -18,6 +18,7 @@ MODULES = lang/python
MODPY_PI = Yes
MODPY_PYTEST = Yes
-MODPY_VERSION =${MODPY_DEFAULT_VERSION_3}
+FLAVORS = python3
+FLAVOR = python3
.include
Index: databases/py-minidb/pkg/PLIST
===
RCS file: /cvs/ports/databases/py-minidb/pkg/PLIST,v
retrieving revision 1.1.1.1
diff -u -p -r1.1.1.1 PLIST
--- databases/py-minidb/pkg/PLIST 1 Nov 2016 13:20:10 - 1.1.1.1
+++ databases/py-minidb/pkg/PLIST 29 Feb 2020 16:01:00 -
@@ -1,4 +1,5 @@
@comment $OpenBSD: PLIST,v 1.1.1.1 2016/11/01 13:20:10 rpe Exp $
+@pkgpath databases/py-minidb
${MODPY_COMMENT}lib/python${MODPY_VERSION}/site-packages/${MODPY_PYCACHE}/
lib/python${MODPY_VERSION}/site-packages/${MODPY_PYCACHE}minidb.${MODPY_PYC_MAGIC_TAG}pyc
lib/python${MODPY_VERSION}/site-packages/minidb-${MODPY_EGG_VERSION}-py${MODPY_VERSION}.egg-info
Index: www/urlwatch/Makefile
===
RCS file: /cvs/ports/www/urlwatch/Makefile,v
retrieving revision 1.25
diff -u -p -r1.25 Makefile
--- www/urlwatch/Makefile 3 Jun 2019 16:06:58 - 1.25
+++ www/urlwatch/Makefile 29 Feb 2020 16:01:00 -
@@ -4,7 +4,7 @@ COMMENT=monitor webpages for updates
MODPY_EGG_VERSION= 2.17
DISTNAME= urlwatch-${MODPY_EGG_VERSION}
-REVISION= 0
+REVISION= 1
CATEGORIES=www
@@ -23,7 +23,7 @@ MODPY_SETUPTOOLS= Yes
BUILD_DEPENDS= ${RUN_DEPENDS}
RUN_DEPENDS= converters/py-html2text${MODPY_FLAVOR} \
- databases/py-minidb \
+ databases/py-minidb${MODPY_FLAVOR} \
devel/py-appdirs${MODPY_FLAVOR} \
security/py-keyring${MODPY_FLAVOR} \
textproc/py-cssselect${MODPY_FLAVOR} \
Re: py3-only ports
On 2020/02/20 10:12, Kurt Mosiejczuk wrote:
> > There is a possible future direction I've been considering of moving py2
> > ports from "unflavoured" to "FLAVOR=python2" (and possibly splitting
> > python3 into multiple versions e.g. python37/python38 as we do with ruby).
> > I'm unsure if it's worthwhile at the moment, but this method of converting
> > to py3-only at least doesn't get in the way of doing that later.
>
> I like the FLAVOR=python2. I'm a little wary of the python37/python38
> thing. It will introduce a *lot* of churn each time we introduce a
> new python 3.x version to the tree or when we retire one.
Me too, but other things being equal I'd prefer not to make changes
now that would interfere with it in case we did want to change it later.
(For example if we generally have things setup where py3=unflavoured,
that *would* get in the way).
+ easier to test new python releases
- more work bumping revision in dependencies and managing category Makefiles
- brings back the MODPY_BIN_SUFFIX problem that otherwise would be helped
by moving to 3-only
On 2020/02/20 16:25, Björn Ketelaars wrote:
> Sounds like a plan...
>
> What about lines in Makefile like the one below. Does it make sense to
> keep them around? There is no need to add a suffix to scripts in the
> case of py3-only.
>
> mv ${PREFIX}/bin/$i{,${MODPY_BIN_SUFFIX}}
exactly, getting rid of these is a big win :)
For ports containing scripts which remain as py2+py3 (the most obvious
example being py-sphinx, but there are many others) I'd like to have py3
as the "unsuffixed" version, but haven't figured out how to do this yet ;)
> > Obviously there are currently a number of py3-only ports which should end
> > up using the same flavour scheme (and dependency lines to fix), if there are
> > not objections I can either review diffs or take care of those myself.
>
> I'm not sure what is needed for ports that already made the jump from
> py2/py3 to py3-only (e.g. www/jupyter-notebook). More specific: these
> ports already do some funky PLIST and quirks stuff. Do these ports need
> special attention?
jupyter-notebook is a bit of a special case as it's more "end-user software"
rather than a module (which is acknowledged by not using the py- prefix on the
package name).
In other cases (some which come to mind are flake8 and beets), this type
of port doesn't normally support using multiple python versions via
flavours, jupyter-notebook is (was) special in that regard. So I think
it's ok to keep that as-is.
Re: py3-only ports
On Thu 20/02/2020 13:55, Stuart Henderson wrote:
> Quite a few python modules are now py3-only, and more are switching
> over to that. The current ports setup where we use MODPY_VERSION=
> ${MODPY_DEFAULT_VERSION_3} and no FLAVOR isn't really very good,
> especially where dependencies are concerned with the error-prone
> combination of "py2+3 ports using ${MODPY_FLAVOR}" and "py3-only
> ports with just the port path/name".
>
> Before more are moved across I'd like to settle on a proper method.
> I've been discussing this with jasper@ (with some previous knowledge
> from adding a similar multi-version to pecl-* ports) and propose moving
> to a setup with:
>
> FLAVOR= python3
> FLAVORS= python3
>
> (and keep the ,python3 flavour in ../Makefile i.e. "SUBDIR +=
> py-Foo,python3").
> This also makes it easier to handle updates as we don't need to mess about
> with @pkgpath markers.
Sounds like a plan...
What about lines in Makefile like the one below. Does it make sense to
keep them around? There is no need to add a suffix to scripts in the
case of py3-only.
mv ${PREFIX}/bin/$i{,${MODPY_BIN_SUFFIX}}
> Obviously there are currently a number of py3-only ports which should end
> up using the same flavour scheme (and dependency lines to fix), if there are
> not objections I can either review diffs or take care of those myself.
I'm not sure what is needed for ports that already made the jump from
py2/py3 to py3-only (e.g. www/jupyter-notebook). More specific: these
ports already do some funky PLIST and quirks stuff. Do these ports need
special attention?
Re: py3-only ports
On Thu, Feb 20, 2020 at 01:55:00PM +, Stuart Henderson wrote:
> Quite a few python modules are now py3-only, and more are switching
> over to that. The current ports setup where we use MODPY_VERSION=
> ${MODPY_DEFAULT_VERSION_3} and no FLAVOR isn't really very good,
> especially where dependencies are concerned with the error-prone
> combination of "py2+3 ports using ${MODPY_FLAVOR}" and "py3-only
> ports with just the port path/name".
> Before more are moved across I'd like to settle on a proper method.
> I've been discussing this with jasper@ (with some previous knowledge
> from adding a similar multi-version to pecl-* ports) and propose moving
> to a setup with:
> FLAVOR= python3
> FLAVORS= python3
> (and keep the ,python3 flavour in ../Makefile i.e. "SUBDIR +=
> py-Foo,python3").
> This also makes it easier to handle updates as we don't need to mess about
> with @pkgpath markers.
I like this very much. Simple. It makes it obvious and feels less like
stepping on rakes like the current scheme does. (Putting the obvious
MODPY_FLAVOR after a depends and having it fil and needing to figure out
why).
(More below)
> Just merge py-foo into py3-foo automatically.
> This was done with some py3 conversions in the past:
> files/Quirks.pm:'py-nmap' => 'py3-nmap',
> files/Quirks.pm:'py-link-grammar' => 'py3-link-grammar',
> files/Quirks.pm:'py-buildbot-www' => 'py3-buildbot-www',
> files/Quirks.pm:'py-buildbot-pkg' => 'py3-buildbot-pkg',
> files/Quirks.pm:'py-buildbot-grid-view' => 'py3-buildbot-grid-view',
> files/Quirks.pm:'py-buildbot-console-view' =>
> 'py3-buildbot-console-view',
> files/Quirks.pm:'py-buildbot-waterfall-view' =>
> 'py3-buildbot-waterfall-view',
> files/Quirks.pm:'py-pafy' => 'py3-pafy',
> files/Quirks.pm:'py-zeroconf' => 'py3-zeroconf',
> files/Quirks.pm:'py-xmldiff' => 'py3-xmldiff',
Much better.
> In cases where there are bin/ scripts where the py2 version will
> have a hard conflict with the py3 version it *has* to be done this way.
> In other cases there is a possible alternative: list in
> $obsolete_reasons and have the user manually delete the py2 version if
> they don't want it. This has been done for some ports more recently
> (e.g. py-semver) but I don't think we should generally continue with it.
> Admittedly it *is* more friendly to an user writing their own py2
> scripts using modules from ports, but the common use case for packages
> is a standard package user who just has things installed as a dependency
> of something else. In that case after a pkg_add -u they get e.g.
> Obsolete package: py-semver-2.8.1p0 (python2 port superseded by python3
> version)
> : FLAVORS = python3
> : -FLAVOR ?=
> : +FLAVOR = python3
> Just hardcode the flavour here so "make" directly in the port dir works
> sanely. Note this means that MODPY_VERSION=${MODPY_DEFAULT_VERSION_3} is
> not required.
This is much friendlier to do. The existing scheme was non-obvious.
> : diff -u -p -r1.8 PLIST
> : --- net/py-netmiko/pkg/PLIST13 Nov 2019 20:13:39 - 1.8
> : +++ net/py-netmiko/pkg/PLIST12 Feb 2020 19:14:28 -
> : @@ -1,4 +1,6 @@
> : @comment $OpenBSD: PLIST,v 1.8 2019/11/13 20:13:39 jasper Exp $
> : +@conflict py-netmiko-*
> : +@pkgpath net/py-netmiko
> Works with the quirks changes above to allow automatically replacing
> py-netmiko with py3-netmiko.
> In this particular case there aren't really conflicting files but I think
> it makes sense to do that as standard (in cases where there are scripts in
> bin/ without a suffix then there would be a conflict - easier to just
> always include it so people looking for examples to crib from have an
> easier time).
Yes. I agree.
> Obviously there are currently a number of py3-only ports which should end
> up using the same flavour scheme (and dependency lines to fix), if there are
> not objections I can either review diffs or take care of those myself.
No objections here.
> There is a possible future direction I've been considering of moving py2
> ports from "unflavoured" to "FLAVOR=python2" (and possibly splitting
> python3 into multiple versions e.g. python37/python38 as we do with ruby).
> I'm unsure if it's worthwhile at the moment, but this method of converting
> to py3-only at least doesn't get in the way of doing that later.
I like the FLAVOR=python2. I'm a little wary of the python37/python38
thing. It will introduce a *lot* of churn each time we introduce a
new python 3.x version to the tree or when we retire one.
--Kurt
Re: py3-only ports
All you're saying makes sense to me.
py3-only ports
Quite a few python modules are now py3-only, and more are switching
over to that. The current ports setup where we use MODPY_VERSION=
${MODPY_DEFAULT_VERSION_3} and no FLAVOR isn't really very good,
especially where dependencies are concerned with the error-prone
combination of "py2+3 ports using ${MODPY_FLAVOR}" and "py3-only
ports with just the port path/name".
Before more are moved across I'd like to settle on a proper method.
I've been discussing this with jasper@ (with some previous knowledge
from adding a similar multi-version to pecl-* ports) and propose moving
to a setup with:
FLAVOR= python3
FLAVORS=python3
(and keep the ,python3 flavour in ../Makefile i.e. "SUBDIR += py-Foo,python3").
This also makes it easier to handle updates as we don't need to mess about
with @pkgpath markers.
To demonstrate here's a diff that Jasper put together for py-napalm and
deps. The new version of py-netmiko is now py3-only, py-napalm depends on
this so needs to also move to py3-only, and py-IOSXR doesn't absolutely
need to move but it's not used for anything else in-tree so might as well
come along for the ride. I'll add some commentary inline.
: Index: net/Makefile
: ===
: RCS file: /cvs/ports/net/Makefile,v
: retrieving revision 1.1167
: diff -u -p -r1.1167 Makefile
: --- net/Makefile 31 Jan 2020 23:34:42 - 1.1167
: +++ net/Makefile 12 Feb 2020 19:14:28 -
: @@ -545,7 +545,6 @@
: SUBDIR += pure-ftpd,virtual_chroot
: SUBDIR += purple-rocketchat
: SUBDIR += putty
: - SUBDIR += py-IOSXR
: SUBDIR += py-IOSXR,python3
: SUBDIR += py-IP
: SUBDIR += py-IP,python3
: @@ -604,7 +603,6 @@
: SUBDIR += py-netaddr,python3
: SUBDIR += py-netifaces
: SUBDIR += py-netifaces,python3
: - SUBDIR += py-netmiko
: SUBDIR += py-netmiko,python3
: SUBDIR += py-nmap
: SUBDIR += py-nxos
obvious, drop py2 builds
: Index: devel/quirks/Makefile
: ===
: RCS file: /cvs/ports/devel/quirks/Makefile,v
: retrieving revision 1.841
: diff -u -p -r1.841 Makefile
: --- devel/quirks/Makefile 12 Feb 2020 14:34:05 - 1.841
: +++ devel/quirks/Makefile 12 Feb 2020 19:14:28 -
: @@ -5,7 +5,7 @@ CATEGORIES = devel databases
: DISTFILES =
:
: # API.rev
: -PKGNAME =quirks-3.232
: +PKGNAME =quirks-3.233
: PKG_ARCH = *
: MAINTAINER = Marc Espie
:
: Index: devel/quirks/files/Quirks.pm
: ===
: RCS file: /cvs/ports/devel/quirks/files/Quirks.pm,v
: retrieving revision 1.859
: diff -u -p -r1.859 Quirks.pm
: --- devel/quirks/files/Quirks.pm 12 Feb 2020 14:34:05 - 1.859
: +++ devel/quirks/files/Quirks.pm 12 Feb 2020 19:14:28 -
: @@ -339,6 +339,9 @@ my $stem_extensions = {
: 'ilmbase' => 'OpenEXR',
: 'openexr-viewers' => 'OpenEXR-tools',
: 'libvirt-python' => 'py-libvirt',
: + 'py-netmiko' => 'py3-netmiko',
: + 'py-napalm' => 'py3-napalm',
: + 'py-IOSXR' => 'py3-IOSXR',
: };
:
: my $obsolete_reason = {
Just merge py-foo into py3-foo automatically.
This was done with some py3 conversions in the past:
files/Quirks.pm:'py-nmap' => 'py3-nmap',
files/Quirks.pm:'py-link-grammar' => 'py3-link-grammar',
files/Quirks.pm:'py-buildbot-www' => 'py3-buildbot-www',
files/Quirks.pm:'py-buildbot-pkg' => 'py3-buildbot-pkg',
files/Quirks.pm:'py-buildbot-grid-view' => 'py3-buildbot-grid-view',
files/Quirks.pm:'py-buildbot-console-view' =>
'py3-buildbot-console-view',
files/Quirks.pm:'py-buildbot-waterfall-view' =>
'py3-buildbot-waterfall-view',
files/Quirks.pm:'py-pafy' => 'py3-pafy',
files/Quirks.pm:'py-zeroconf' => 'py3-zeroconf',
files/Quirks.pm:'py-xmldiff' => 'py3-xmldiff',
In cases where there are bin/ scripts where the py2 version will
have a hard conflict with the py3 version it *has* to be done this way.
In other cases there is a possible alternative: list in
$obsolete_reasons and have the user manually delete the py2 version if
they don't want it. This has been done for some ports more recently
(e.g. py-semver) but I don't think we should generally continue with it.
Admittedly it *is* more friendly to an user writing their own py2
scripts using modules from ports, but the common use case for packages
is a standard package user who just has things insta
