Re: [Distutils] multiple backports of ipaddress and a world of pain

[Chris Withers]

  
  
Hi Nick,

On 18/02/2016 13:32, Nick Coghlan
  wrote:


  

  On 17 February 2016 at 04:37, Chris
Withers 
wrote:

  So, RHEL7, for worse or worse, ships with Python 2.7.5.


It's 2.7.5 + important security backports, so any
  package that relies on PEP 466 features like
  ssl.create_default_context() should be fine in 7.2+. (You
  can also switch on default certificate verification if you
  want it: https://access.redhat.com/articles/2039753
  )

  

  

The company I'm with at the moment is one of the more aggressive
operating system release followers I've worked with or for, and even
we're not on 7.2 yet!
 

  

  
That means to keep pip
  happy, you need to do these dances in all the virtualenvs
  you create:
  
  http://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning
  http://urllib3.readthedocs.org/en/latest/security.html#pyopenssl


If urllib3 is actually using version detection rather
  than feature detection as recommended in PEP 466 (https://www.python.org/dev/peps/pep-0466/#backwards-compatibility-considerations),
  then that's a missing bug report against urllib3

  

  

I guess we'll see when we start testing. We currently have a
python-virtualenv package that pulls in a kitchen sink's work of
dependencies, as recommended at those links...


  

  
  
  It looks like you found a resolution to this part of the
  problem, but those dependencies should only be needed on 7.0
  and 7.1
  

Unfortunately, I missed this use case
  when PEP 508 was being defined, so there's currently no
  capability for Python level dependencies to be conditional on
  the presence or absence of particular attributes in other
  modules :(

  

Not sure that's such a biggie here, I'd more like to see pip at
least notice that it's trying to install two files into the same
location.

cheers,

Chris
  

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 516 and pypa.json

[Ethan Furman]

On 02/18/2016 12:00 PM, Daniel Holth wrote:
> On Thu, Feb 18, 2016 at 9:06 AM, Ethan Furman wrote:

>> I saw PEP-0516 go through check-ins, and had a question about the
>> pypa.json portion of the proposal -- namely, why are we using a
>> .json file?
>>
>> I presume this is a file that will be created by hand, and while
>> json is not as ugly as xml, it's certainly not pretty.


I'm still pulling for RFC 822 format :)


You know what?  xml is okay.*  ;)

--
~Ethan~

* Not really!  Just kidding!
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 516 and pypa.json

[Daniel Holth]

I'm still pulling for RFC 822 format :)

On Thu, Feb 18, 2016, 14:33 Nathaniel Smith  wrote:

> On Thu, Feb 18, 2016 at 9:06 AM, Ethan Furman  wrote:
> > Greetings!
> >
> > I saw PEP-0516 go through check-ins, and had a question about the
> pypa.json
> > portion of the proposal -- namely, why are we using a .json file?
> >
> > I presume this is a file that will be created by hand, and while json is
> not
> > as ugly as xml, it's certainly not pretty.
> >
> > Can we not use an .ini file, or a white-space significant format?  The
> > latter should be familiar, if not comfortable, to any Python programmer.
>
> These are still draft proposals, and the actual file format is the
> least interesting question to discuss. Think of the JSON thing as
> being a placeholder for now :-).
>
> There is no obvious solution, because .ini files are extremely
> underspecified, nothing else is in the stdlib, yaml contains its own
> share of gibbering horrors, toml is not widely used and is associated
> with an extremely divisive figure, etc. etc. Don't worry, though:
> there will be bikeshedding. Ideally *after* the more substantive
> issues are settled :-)
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 516 and pypa.json

[Nathaniel Smith]

On Thu, Feb 18, 2016 at 9:06 AM, Ethan Furman  wrote:
> Greetings!
>
> I saw PEP-0516 go through check-ins, and had a question about the pypa.json
> portion of the proposal -- namely, why are we using a .json file?
>
> I presume this is a file that will be created by hand, and while json is not
> as ugly as xml, it's certainly not pretty.
>
> Can we not use an .ini file, or a white-space significant format?  The
> latter should be familiar, if not comfortable, to any Python programmer.

These are still draft proposals, and the actual file format is the
least interesting question to discuss. Think of the JSON thing as
being a placeholder for now :-).

There is no obvious solution, because .ini files are extremely
underspecified, nothing else is in the stdlib, yaml contains its own
share of gibbering horrors, toml is not widely used and is associated
with an extremely divisive figure, etc. etc. Don't worry, though:
there will be bikeshedding. Ideally *after* the more substantive
issues are settled :-)

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] PEP 516 and pypa.json

[Ethan Furman]

Greetings!

I saw PEP-0516 go through check-ins, and had a question about the 
pypa.json portion of the proposal -- namely, why are we using a .json file?


I presume this is a file that will be created by hand, and while json is 
not as ugly as xml, it's certainly not pretty.


Can we not use an .ini file, or a white-space significant format?  The 
latter should be familiar, if not comfortable, to any Python programmer.


--
~Ethan~
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] abstract build system PEP update

[Paul Moore]

On 18 February 2016 at 13:48, Nick Coghlan  wrote:
>> Yes, twine is a good, concrete example.
>
> I believe tools like pyp2rpm, conda skeleton, py2dsc and fpm also rely on
> that static sdist metadata (I'm not 100% sure on that, but it would make
> sense for them to do so).

Yeah, I'm sure there are a lot of tools out there that use the sdist
metadata. In a lot of ways, pip is an extremely atypical example of a
sdist consumer.

> We just spend so much time worrying about the dependency management problems
> that PKG-INFO *doesn't* handle that we forget the simpler descriptive
> metadata that it covers accurately :)

Indeed - and by focusing solely on pip's requirements in many ways
we're designing for an edge case (admittedly a hugely significant and
by far the most used edge case, but even so...) :-)

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] abstract build system PEP update

[Daniel Holth]

Yes PKG-INFO is reasonably useful and easy to hand-edit. It would be easy
to maintain. If you were doing it manually the only thing you would need to
update frequently would be the version number.

On Thu, Feb 18, 2016 at 8:48 AM Nick Coghlan  wrote:

> On 18 February 2016 at 07:30, Paul Moore  wrote:
>
>> On 17 February 2016 at 21:01, Nathaniel Smith  wrote:
>> > On Feb 17, 2016 4:44 AM, "Donald Stufft"  wrote:
>> >>
>> > [...]
>> >> You could say that using twine to handle the uploading is a thing
>> people
>> >> should
>> >> do (and I agree!) but that currently relies on having static metadata
>> >> inside of
>> >> the sdist that twine can parse, static metadata that isn't going to
>> exist
>> >> if
>> >> you just simply tarball up a directory on disk.
>> >
>> > Ah-ha, this is useful. The reason this hasn't been considered, at least
>> in
>> > my proposal, is that I think this is the first I've heard that there is
>> > anything that cares about what's in an sdist besides setup.py :-).
>>
>> Yes, twine is a good, concrete example.
>>
>
> I believe tools like pyp2rpm, conda skeleton, py2dsc and fpm also rely on
> that static sdist metadata (I'm not 100% sure on that, but it would make
> sense for them to do so).
>
> We just spend so much time worrying about the dependency management
> problems that PKG-INFO *doesn't* handle that we forget the simpler
> descriptive metadata that it covers accurately :)
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] abstract build system PEP update

[Nick Coghlan]

On 18 February 2016 at 07:30, Paul Moore  wrote:

> On 17 February 2016 at 21:01, Nathaniel Smith  wrote:
> > On Feb 17, 2016 4:44 AM, "Donald Stufft"  wrote:
> >>
> > [...]
> >> You could say that using twine to handle the uploading is a thing people
> >> should
> >> do (and I agree!) but that currently relies on having static metadata
> >> inside of
> >> the sdist that twine can parse, static metadata that isn't going to
> exist
> >> if
> >> you just simply tarball up a directory on disk.
> >
> > Ah-ha, this is useful. The reason this hasn't been considered, at least
> in
> > my proposal, is that I think this is the first I've heard that there is
> > anything that cares about what's in an sdist besides setup.py :-).
>
> Yes, twine is a good, concrete example.
>

I believe tools like pyp2rpm, conda skeleton, py2dsc and fpm also rely on
that static sdist metadata (I'm not 100% sure on that, but it would make
sense for them to do so).

We just spend so much time worrying about the dependency management
problems that PKG-INFO *doesn't* handle that we forget the simpler
descriptive metadata that it covers accurately :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] multiple backports of ipaddress and a world of pain

[Nick Coghlan]

On 17 February 2016 at 04:37, Chris Withers  wrote:

> Hi All,
>
> (Apologies for copying in the maintainers of the two backports and
> django-netfields directly, I'm not sure you're on this distutils list...)
>
> This is painful and horrible, and I wish pip would prevent
> modules/packages with the same name being installed by different
> distributions at the same time, but even if it did, that would just force
> something to happen rather than this:
>
> So, RHEL7, for worse or worse, ships with Python 2.7.5.


It's 2.7.5 + important security backports, so any package that relies on
PEP 466 features like ssl.create_default_context() should be fine in 7.2+.
(You can also switch on default certificate verification if you want it:
https://access.redhat.com/articles/2039753 )


> That means to keep pip happy, you need to do these dances in all the
> virtualenvs you create:
>
>
> http://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning
> http://urllib3.readthedocs.org/en/latest/security.html#pyopenssl


If urllib3 is actually using version detection rather than feature
detection as recommended in PEP 466 (
https://www.python.org/dev/peps/pep-0466/#backwards-compatibility-considerations),
then that's a missing bug report against urllib3


> One of those extra packages drags in this backport:
>
> https://pypi.python.org/pypi/ipaddress
>
> Yay! Now we have a happy pip talking to both PyPI and our internal DevPI
> server!
>
> Right, so in a Django project I need to use
> https://pypi.python.org/pypi/django-netfields. This, however, chooses
> this backport instead:
>
> https://pypi.python.org/pypi/py2-ipaddress
>
> So, now we have two packages installing ipaddress.py, except they're two
> very different versions and make different assumptions about what to do
> with Python 2 strings.
>
> What should happen here? (other than me crying a lot...)
>

It looks like you found a resolution to this part of the problem, but those
dependencies should only be needed on 7.0 and 7.1

Unfortunately, I missed this use case when PEP 508 was being defined, so
there's currently no capability for Python level dependencies to be
conditional on the presence or absence of particular attributes in other
modules :(

Regards,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Olivier Grisel]

Strong +1 for all Donald's proposals.

-- 
Olivier
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] abstract build system PEP update

[Donald Stufft]

> On Feb 18, 2016, at 5:26 AM, Paul Moore  wrote:
> 
> [1] One downsides of managing PEPs as PRs against the packaging specs
> is that there is no easy quotable term to refer to them by - we could,
> and probably should, use the PR number. And as a consequence, it's
> hard to quickly find the text because there's no simple form for the
> URL (again mitigated by using the PR number).

I've merged the two PRs and given them numbers, Robert's PEP is PEP 516 and
Nathaniel's is PEP 517.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] abstract build system PEP update

[Paul Moore]

On 17 February 2016 at 22:58, Robert Collins  wrote:
>> The latter option is far easier (we're basically almost there) but it
>> will *not* suit build tools that can't (or won't) generate a sdist
>> that conforms to PEP 314 (i.e., contains PKG-INFO).
>
> I still don't understand how *making* an sdist is tied into this PEP?
> AIUI flit's only concern with sdists was that they didn't want a
> setup.py with all the stuff we've discussed about that. Perhaps a flit
> author can weigh in - generating good PEP 314 PKG-INFO shouldn't be a
> problem for flit, has no Python2/3 implication, and doesn't require a
> setup.py.

That's why I didn't specifically refer to flit. In the context of this
point, if projects produce sdists (i.e., they contain PKG-INFO) then
we're find. AFAIK, there's no PEP saying a sdist has to contain a
setup.py.

Backward compatibility issues as noted in your PEP may imply that a
minimal "shim" setup.py should be included. I'd have to reread what
you said to confirm if you mandated a shim or not[1] but that's a side
issue for now.

> PEP 241 and 314 do not require a setup.py to be present AFAICT.

Correct (as far as *I* can tell). And that's fine.

> I think all we need to do is make sure that its crystal clear that
> we're not changing any specified part of the contract of sdists: you
> should still make them, they still need to meet PEP 314 setup.py
> was never PEP'd as part of the contract.

Exactly. This was what I was proposing, so I agree 100%

> We are changing the defacto contract, and so we should make that clear
> in the prose.

And yet, this is where we have to be careful, as pip isn't the only
consumer of that contract (that's the point of Donald bringing up
twine). You say in another mail:

> How about:"""
> The existing
> PEP-314 [#pep314] requirements for sdists still apply, and distutils
> or setuptools
> users can use ``setup.py sdist`` to create an sdist. Other tools should create
> sdists compatible with PEP-314 [#pep314]. Note that pip itself does not 
> require
> PEP-314 compatibility - it does not use any of the metadata from sdists - they
> are treated like source trees from disk or version control."""

While this is accurate as regards pip not using the PEP 314 metadata,
I think it's important to point out that in practical terms pip *does*
require PEP 314 compatible sdists, in that it *locates* source by
querying PyPI for a sdist, and so it inherits PyPI's rules for what is
hosted there. I don't know if PyPI checks for PEP 314 metadata, or if
Warehouse will in future, but that is something we need to take into
consideration. And even if they don't, we recommend using twine for
uploads, and twine *does* require PEP 314.

As worded, the note about pip will[2] increase pressure for a means to
host the sources for that tool on PyPI, which in turn will increase
pressure for PyPI/Warehouse to *not* enforce PEP 314, and in turn for
twine to not enforce that PEP. We aren't ready to offer an alternative
to PEP 314 yet, though.

What I don't understand is why it matters so much to you that the
proposal *doesn't* simply say "sdists should conform to PEP 314", and
you keep pushing to leave the door open for people to omit the
metadata. (That's in *sdists* - as you point out, pip doesn't use or
need metadata in the source tree, but that's already clear in the
proposal). All I can imagine is that it's because you want to allow
flit users the extra flexibility.

My questions for flit (and any other build tool that expects to use
these proposals):

1. Can you produce PEP 314 compatible sdists?
2. If not, what are your reasons (and does this proposal change
anything in that regard)?
3. Assuming you don't intend to produce PEP 314 compatible sdists,
what is your recommendation to your users on how they publish source,
should they wish to?
4. If your users were unable to publish source on PyPI because of your
lack of a sdist, would that change your view on whether you support
sdists?
5. What changes would you need to the sdist format to make it acceptable to you?

Unless those questions expose a significant roadblock that flit simply
cannot overcome with PEP 314 compatibility, it seems to me that
there's no harm in mandating PEP 314 for sdists without qualification,
and leaving further discussion about de facto behaviour, possible
changes, etc, to the "sdist 2.0" discussion that we're all trying to
defer.

By the way - I should point out that it's pretty straightforward for a
project author to *manually* create a PKG-INFO file in their flit
project, bundle it up with all of the rest of the sources, name the
archive correctly, and have a PEP 314 compatible sdist. Managing that
PKG-INFO file in parallel with flit's own copy of the metadata is
likely to be a PITA, but how is that not just a usability issue to be
dealt with between the users and authors of flit?

Paul

[1] One downsides of managing PEPs as PRs against the packaging specs
is that there is no