Over this last weekend I went to PyCascades and after Python conferences they have "Sprints", which are like hackathons, but lots of different Python projects get together to do stuff as well as use it as a time to attract new contributors. The last few conferences I attended, I took that time write some perl while to learning more about how the Python Package Index (PyPI) works and finally this time made some progress on adding dependency support for PyPI to portgen(1). I think with this support it works well enough to merge that recent patch mentioning the support on in the manpage.
Of note, there is some weird new thing they are doing with "Wheels" and my test runs wanted to download these things into a .eggs directory for running tests even though the packages were installed. I don't know what needs to happen to fix that, but it seems unhelpful. https://pythonwheels.com/ I would appreciate tests and reviews of this, especially from folks who know about Python packaging or who want to tell me to simplify my perl. There are three semi-unrelated patches here, as the python port I used for testing (pwned) is Python 3 only and its license is "GPL-3.0" 1. Add gpl_3_0 as a recognized variant for "GPLv3" 2. Better detect what MODPY_VERSIONs the module supports 3. Find dependencies OK for the license detection? Index: infrastructure/lib/OpenBSD/PortGen/License.pm =================================================================== RCS file: /cvs/ports/infrastructure/lib/OpenBSD/PortGen/License.pm,v retrieving revision 1.2 diff -u -p -u -r1.2 License.pm --- infrastructure/lib/OpenBSD/PortGen/License.pm 27 Apr 2016 09:58:35 -0000 1.2 +++ infrastructure/lib/OpenBSD/PortGen/License.pm 27 Feb 2019 19:32:29 -0000 @@ -38,6 +38,7 @@ my %good_licenses = ( gpl_2 => 'GPLv2', gpl_2_0 => 'GPLv2', gpl_3 => 'GPLv3', + gpl_3_0 => 'GPLv3', lgpl => 'LGPL', lgpl_2_1 => 'LGPL v2.1', 'lgpl_2_1+' => 'LGPL v2.1', OK for the MODPY_VERSION detection? Do I need to do something else so that these Python 3 only ports work with ${MODPY_FLAVOR}? Index: infrastructure/lib/OpenBSD/PortGen/Port/PyPI.pm =================================================================== RCS file: /cvs/ports/infrastructure/lib/OpenBSD/PortGen/Port/PyPI.pm,v retrieving revision 1.1.1.1 diff -u -p -u -r1.1.1.1 PyPI.pm --- infrastructure/lib/OpenBSD/PortGen/Port/PyPI.pm 18 Jan 2016 18:08:20 -0000 1.1.1.1 +++ infrastructure/lib/OpenBSD/PortGen/Port/PyPI.pm 27 Feb 2019 19:32:29 -0000 @@ -73,12 +74,22 @@ sub fill_in_makefile $self->set_license( $di->{info}{license} ); $self->set_descr( $di->{info}{summary} ); - for ( @{ $di->{info}{classifiers} } ) { - if (/^Programming Language :: Python :: 3/) { - $self->set_other( 'FLAVORS', 'python3' ); - $self->set_other( 'FLAVOR', '' ); - last; - } + my @versions = do { + my %seen; + sort { $a <=> $b } grep { !$seen{$_}++ } map { + /^Programming Language :: Python :: (\d+)/ ? $1 : () + } @{ $di->{info}{classifiers} } + }; + + if ( @versions > 1 ) { + shift @versions; # remove default, lowest + $self->set_other( 'FLAVORS', "python$_" ) for @versions; + $self->set_other( 'FLAVOR', '' ); + } + elsif ( @versions == 1 && $versions[0] != 2 ) { + $self->set_other( + MODPY_VERSION => "\${MODPY_DEFAULT_VERSION_$_}" ) + for @versions; } } I actually first wrote a version of this that extracts the egg_info and looks through the "requirements" that generates. Then I accidentally dumped out the JSON that PyPI returns and it turns out it now figures out those dependencies somehow and lists them for me! So, after parsing their very odd dependency strings, that is probably subtly wrong, I now have something that seems to work. One thing I wasn't sure of was whether the "*_DEPENDS" need a ${MODPY_FLAVOR} attached? I also let it add the "extra" dependencies I didn't recognize as "RUN_DEPENDS" although they are probably optional. Would it be preferred to instead skip those items? OK for the get_deps change? (I assume not yet as I am fairly sure there are some needed tweaks) Index: infrastructure/lib/OpenBSD/PortGen/Port/PyPI.pm =================================================================== RCS file: /cvs/ports/infrastructure/lib/OpenBSD/PortGen/Port/PyPI.pm,v retrieving revision 1.1.1.1 diff -u -p -u -r1.1.1.1 PyPI.pm --- infrastructure/lib/OpenBSD/PortGen/Port/PyPI.pm 18 Jan 2016 18:08:20 -0000 1.1.1.1 +++ infrastructure/lib/OpenBSD/PortGen/Port/PyPI.pm 27 Feb 2019 19:32:29 -0000 @@ -22,6 +22,7 @@ use warnings; use parent 'OpenBSD::PortGen::Port'; use OpenBSD::PortGen::Dependency; +use OpenBSD::PortGen::Utils qw( module_in_ports ); sub ecosystem_prefix { @@ -51,7 +52,7 @@ sub name_new_port { my ( $self, $di ) = @_; - my $name = $di->{info}{name}; + my $name = ref $di ? $di->{info}{name} : $di; $name = "py-$name" unless $name =~ /^py-/; return "pypi/$name"; @@ -73,12 +74,22 @@ sub fill_in_makefile $self->set_license( $di->{info}{license} ); $self->set_descr( $di->{info}{summary} ); - for ( @{ $di->{info}{classifiers} } ) { - if (/^Programming Language :: Python :: 3/) { - $self->set_other( 'FLAVORS', 'python3' ); - $self->set_other( 'FLAVOR', '' ); - last; - } + my @versions = do { + my %seen; + sort { $a <=> $b } grep { !$seen{$_}++ } map { + /^Programming Language :: Python :: (\d+)/ ? $1 : () + } @{ $di->{info}{classifiers} } + }; + + if ( @versions > 1 ) { + shift @versions; # remove default, lowest + $self->set_other( 'FLAVORS', "python$_" ) for @versions; + $self->set_other( 'FLAVOR', '' ); + } + elsif ( @versions && $versions[0] != 2 ) { + $self->set_other( + MODPY_VERSION => "\${MODPY_DEFAULT_VERSION_$_}" ) + for @versions; } } @@ -94,6 +105,71 @@ sub postextract sub get_deps { + my ( $self, $di, $wrksrc ) = @_; + my $deps = OpenBSD::PortGen::Dependency->new(); + + # This, especially the os detection, may need additional + # work to be reliable, but I don't really know what PyPI returns. + # https://www.python.org/dev/peps/pep-0508/ + foreach ( @{ $di->{info}->{requires_dist} || [] } ) { + my $phase = 'run'; + + #https://packaging.python.org/specifications/core-metadata/#name + my $name; + if ( s/^([A-Z0-9]|[A-Z0-9][A-Z0-9._-]*[A-Z0-9])\b//i ) { + $name = $1; + } + next unless $name; + + # remove comma separated "extra" inside [] + if ( s/^ \s* \[ ( [^]]* ) \] \s*//x ) { + $phase = $1; + } + + my ( $req, $meta ) = split /\s*;\s*/; + $req =~ s/^\s+//; + $req ||= ">=0"; + $req =~ s/^.*( [(] ) (.*?) (?(1) [)] ).*/$2/x; + + $meta ||= ''; + while ( $meta =~ /\bextra \s* == \s* (['"])? (.+?) \1 /gx ) { + $phase = $2; + } + + my @plat; + while ( $meta =~ / + (?:sys_platform|os_name) \s* == \s* (['"]) (.+?) \1 + /gx ) { + push @plat, $2; + } + + next if @plat and join( " ", @plat ) !~ /OpenBSD/i; + + my $port = module_in_ports( $name, 'py-' ) + || $self->name_new_port($name); + + if ( $phase eq 'build' ) { + $deps->add_build( $port, $req ); + } elsif ( $phase eq 'test' ) { + $deps->add_test( $port, $req ); + } elsif ( $phase eq 'dev' ) { + # switch this to "ne 'run'" to avoid optional deps + warn "Not adding '$phase' dep on $port\n"; + next; + } else { + warn "Adding '$phase' dep on $port as run dep\n" + unless $phase eq 'run'; + $deps->add_run( $port, $req ); + } + + # don't have it in tree, port it + if ( $port =~ m{^pypi/} ) { + my $o = OpenBSD::PortGen::Port::PyPI->new(); + $o->port($name); + } + } + + return $deps->format; } sub get_config_style
