Re: [Twisted-Python] Twisted trunk and klein?

[Craig Rodrigues]

On Thu, Feb 11, 2021 at 2:29 PM Glyph  wrote:
>
>
> I filed https://github.com/pypa/pip/issues/9601 for this since it seems like 
> it's well outside of our control.  If anyone else can add more information 
> that would be great.
>


Thanks for doing the analysis and filing that bug.  I didn't want to
jump on anything, and wanted to give some breathing room for
folks to look at that and chime in.

Wilfredo has this PR to klein where he is also looking at these
issues: https://github.com/twisted/klein/pull/450

So I think that I will not hold up the Twisted core release based on
this issue.  Hopefully this issue can be resolved outside of Twisted
core.

--
Craig

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted trunk and klein?

[Glyph]

> On Feb 11, 2021, at 2:09 PM, Glyph  wrote:
> 
> 
> 
>> On Feb 11, 2021, at 1:04 PM, Glyph > > wrote:
>> 
>> 
>>> On Feb 11, 2021, at 11:41 AM, Wilfredo Sánchez Vega >> > wrote:
>>> 
  My concern here is that Twisted may have added a dependency on requests, 
 and that dependency means that if you want IDNA==3.1, as Klein’s tox.ini 
 does, that you aren’t allowed to.  That seems weak; I’d say a bug.  No?
>>> 
>>>  Note this is only cause when installing treq, so the issue seems to be 
>>> there, though for some reason, it’s only a problem with Twisted trunk, 
>>> which kind of confuses me, but I haven’t dug into it beyond trying to 
>>> figure out a bit about what fails to install.
>>> 
>>>  So I agree probably it’s not a blocker for Twisted, and it’s only annoying 
>>> to Klein for its own testing, not it’s clients, but if I were a client of 
>>> treq (and Klein is), I’d think this is a bug.
>> 
>> Attempting to reproduce this locally was a real adventure for me.  I had a 
>> bunch of failure-to-terminate cases from the new pip resolver, until I gave 
>> up, download the newest security update of each release of python, upgraded 
>> pip, tox, setuptools, and virtualenv for every python version, fully cleaned 
>> my git checkout, and re-ran.
>> 
>> Now everything passes except for coverage-py39-twtrunk, which still fails to 
>> terminate.
>> 
>> Under -, it spends a very long time printing messages like these over 
>> and over again:
>> 
>> INFO: pip is looking at multiple versions of setuptools to determine which 
>> version is compatible with other requirements. This could take a while.
>> INFO: This is taking longer than usual. You might need to provide the 
>> dependency resolver with stricter constraints to reduce runtime. If you want 
>> to abort this run, you can press Ctrl + C to do so. To improve how pip 
>> performs, tell us what happened here: 
>> https://pip.pypa.io/surveys/backtracking 
>> 
>> 
>> as it installs every version of setuptools and six that has ever existed.
>> 
>> It seems like Klein has backed itself into a very weird and complex corner 
>> with the new dependency resolver, but it doesn't appear unique to Twisted 
>> trunk, just perhaps tickled ever so slightly worse.
> 
> I've now been running `tox - -r -e coverage-py39-twtrunk` for well over 
> an hour, so I think we may have some pip resolver bugs to report.

I filed https://github.com/pypa/pip/issues/9601 
 for this since it seems like it's 
well outside of our control.  If anyone else can add more information that 
would be great.

-g

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted trunk and klein?

[Glyph]

> On Feb 11, 2021, at 1:04 PM, Glyph  wrote:
> 
> 
>> On Feb 11, 2021, at 11:41 AM, Wilfredo Sánchez Vega > > wrote:
>> 
>>>  My concern here is that Twisted may have added a dependency on requests, 
>>> and that dependency means that if you want IDNA==3.1, as Klein’s tox.ini 
>>> does, that you aren’t allowed to.  That seems weak; I’d say a bug.  No?
>> 
>>  Note this is only cause when installing treq, so the issue seems to be 
>> there, though for some reason, it’s only a problem with Twisted trunk, which 
>> kind of confuses me, but I haven’t dug into it beyond trying to figure out a 
>> bit about what fails to install.
>> 
>>  So I agree probably it’s not a blocker for Twisted, and it’s only annoying 
>> to Klein for its own testing, not it’s clients, but if I were a client of 
>> treq (and Klein is), I’d think this is a bug.
> 
> Attempting to reproduce this locally was a real adventure for me.  I had a 
> bunch of failure-to-terminate cases from the new pip resolver, until I gave 
> up, download the newest security update of each release of python, upgraded 
> pip, tox, setuptools, and virtualenv for every python version, fully cleaned 
> my git checkout, and re-ran.
> 
> Now everything passes except for coverage-py39-twtrunk, which still fails to 
> terminate.
> 
> Under -, it spends a very long time printing messages like these over and 
> over again:
> 
> INFO: pip is looking at multiple versions of setuptools to determine which 
> version is compatible with other requirements. This could take a while.
> INFO: This is taking longer than usual. You might need to provide the 
> dependency resolver with stricter constraints to reduce runtime. If you want 
> to abort this run, you can press Ctrl + C to do so. To improve how pip 
> performs, tell us what happened here: 
> https://pip.pypa.io/surveys/backtracking 
> 
> 
> as it installs every version of setuptools and six that has ever existed.
> 
> It seems like Klein has backed itself into a very weird and complex corner 
> with the new dependency resolver, but it doesn't appear unique to Twisted 
> trunk, just perhaps tickled ever so slightly worse.

I've now been running `tox - -r -e coverage-py39-twtrunk` for well over an 
hour, so I think we may have some pip resolver bugs to report.

-g

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted trunk and klein?

[Glyph]

> On Feb 11, 2021, at 1:15 PM, Adi Roiban  wrote:
> 
> I guess that we can remove idna from setup.cfg TLS section in Twisted
> 

Nope; we use it directly, in 
https://github.com/twisted/twisted/blob/7cf6c8bc320ac5fd96b4784f6feb932ea819856d/src/twisted/internet/_idna.py
 
.

In fact, if anything, we should bump it up, since we use it even in non-tls 
configurations: 
https://github.com/twisted/twisted/blob/cd97222df2ca7032bbff2fe9a8793d7b42dee8b7/src/twisted/internet/_resolver.py#L208
 

 .

-g___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted trunk and klein?

[Adi Roiban]

On Thu, 11 Feb 2021 at 19:26, Wilfredo Sánchez Vega 
wrote:

> On Feb 10, 2021, at 3:44 PM, Glyph  wrote:
>
>
> This isn't even a bug in Klein, it's an issue with a version pin in its
> tox.ini:
> https://github.com/twisted/klein/blob/6e7b37158dea2fe73180809803a872ed98143c6d/tox.ini#L36
>
>
> The constraints from requests (<3,>=2.5) and hyperlink (>=2.5) are
> perfectly compatible; one's just a subset of the other.
>
>
>   Sure, they are compatible, but Klein doesn’t use requests directly, and
> this only looks to be failing in trunk.
>
>   My concern here is that Twisted may have added a dependency on requests,
> and that dependency means that if you want IDNA==3.1, as Klein’s tox.ini
> does, that you aren’t allowed to.  That seems weak; I’d say a bug.  No?
>
>
I have created a new virtual env and executed `pip install .` on trunk.

It installed the following packages:

constantly, zope.interface, attrs, incremental, six, Automat, idna,
hyperlink, Twisted



`requests` was not installed.

I wasn't expecting to see `idna` as it is listed as TLS only deps but
it looks like hyperlink depends on idna.

Here is the dep tree

$ pipdeptree -fl
Twisted @ file:///home/adi/dev/twisted
  attrs==20.3.0
  Automat==20.2.0
attrs==20.3.0
six==1.15.0
  constantly==15.1.0
  hyperlink==21.0.0
idna==3.1
  incremental==17.5.0
  zope.interface==5.2.0
setuptools==44.0.0
wheel==0.36.2



I guess that we can remove idna from setup.cfg TLS section in Twisted


tls =
pyopenssl >= 16.0.0
# service_identity 18.1.0 added support for validating IP addresses in
# certificate subjectAltNames
service_identity >= 18.1.0
idna >= 2.4

Cheers

-- 
Adi Roiban
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted trunk and klein?

[Glyph]

> On Feb 11, 2021, at 11:41 AM, Wilfredo Sánchez Vega  
> wrote:
> 
>>  My concern here is that Twisted may have added a dependency on requests, 
>> and that dependency means that if you want IDNA==3.1, as Klein’s tox.ini 
>> does, that you aren’t allowed to.  That seems weak; I’d say a bug.  No?
> 
>  Note this is only cause when installing treq, so the issue seems to be 
> there, though for some reason, it’s only a problem with Twisted trunk, which 
> kind of confuses me, but I haven’t dug into it beyond trying to figure out a 
> bit about what fails to install.
> 
>  So I agree probably it’s not a blocker for Twisted, and it’s only annoying 
> to Klein for its own testing, not it’s clients, but if I were a client of 
> treq (and Klein is), I’d think this is a bug.

Attempting to reproduce this locally was a real adventure for me.  I had a 
bunch of failure-to-terminate cases from the new pip resolver, until I gave up, 
download the newest security update of each release of python, upgraded pip, 
tox, setuptools, and virtualenv for every python version, fully cleaned my git 
checkout, and re-ran.

Now everything passes except for coverage-py39-twtrunk, which still fails to 
terminate.

Under -, it spends a very long time printing messages like these over and 
over again:

INFO: pip is looking at multiple versions of setuptools to determine which 
version is compatible with other requirements. This could take a while.
INFO: This is taking longer than usual. You might need to provide the 
dependency resolver with stricter constraints to reduce runtime. If you want to 
abort this run, you can press Ctrl + C to do so. To improve how pip performs, 
tell us what happened here: https://pip.pypa.io/surveys/backtracking 


as it installs every version of setuptools and six that has ever existed.

It seems like Klein has backed itself into a very weird and complex corner with 
the new dependency resolver, but it doesn't appear unique to Twisted trunk, 
just perhaps tickled ever so slightly worse.

-g___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted trunk and klein?

[Wilfredo Sánchez Vega]

>   My concern here is that Twisted may have added a dependency on requests, 
> and that dependency means that if you want IDNA==3.1, as Klein’s tox.ini 
> does, that you aren’t allowed to.  That seems weak; I’d say a bug.  No?

  Note this is only cause when installing treq, so the issue seems to be there, 
though for some reason, it’s only a problem with Twisted trunk, which kind of 
confuses me, but I haven’t dug into it beyond trying to figure out a bit about 
what fails to install.

  So I agree probably it’s not a blocker for Twisted, and it’s only annoying to 
Klein for its own testing, not it’s clients, but if I were a client of treq 
(and Klein is), I’d think this is a bug.

-wsv

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted trunk and klein?

[Wilfredo Sánchez Vega]

On Feb 10, 2021, at 3:44 PM, Glyph  wrote:
> 
> This isn't even a bug in Klein, it's an issue with a version pin in its 
> tox.ini:  
> https://github.com/twisted/klein/blob/6e7b37158dea2fe73180809803a872ed98143c6d/tox.ini#L36
>  
> 
>  
> 
> The constraints from requests (<3,>=2.5) and hyperlink (>=2.5) are perfectly 
> compatible; one's just a subset of the other.

  Sure, they are compatible, but Klein doesn’t use requests directly, and this 
only looks to be failing in trunk.

  My concern here is that Twisted may have added a dependency on requests, and 
that dependency means that if you want IDNA==3.1, as Klein’s tox.ini does, that 
you aren’t allowed to.  That seems weak; I’d say a bug.  No?

-wsv

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted trunk and klein?

[Glyph]

> On Feb 10, 2021, at 3:18 PM, Craig Rodrigues  wrote:
> 
> Hi,
> 
> I quickly asked Wilfredo if Twisted trunk worked with Klein, and he mentioned
> that he saw this dependency issue on the Twisted klein side when used
> with Twisted trunk:
> 
> The conflict is caused by:
>The user requested idna==3.1
>hyperlink 21.0.0 depends on idna>=2.5
>requests 2.25.1 depends on idna<3 and >=2.5
> 
> 
> Is that problem purely for klein to deal with, or is there
> any issue that should be fixed in Twisted trunk?

This isn't even a bug in Klein, it's an issue with a version pin in its 
tox.ini:  
https://github.com/twisted/klein/blob/6e7b37158dea2fe73180809803a872ed98143c6d/tox.ini#L36
 

 

The constraints from requests (<3,>=2.5) and hyperlink (>=2.5) are perfectly 
compatible; one's just a subset of the other.

So it might be nice to submit a Klein PR at some point soon, but it's not a 
release blocker.

-g___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Twisted trunk and klein?

[Craig Rodrigues]

Hi,

I quickly asked Wilfredo if Twisted trunk worked with Klein, and he mentioned
that he saw this dependency issue on the Twisted klein side when used
with Twisted trunk:

The conflict is caused by:
The user requested idna==3.1
hyperlink 21.0.0 depends on idna>=2.5
requests 2.25.1 depends on idna<3 and >=2.5


Is that problem purely for klein to deal with, or is there
any issue that should be fixed in Twisted trunk?

--
Craig

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python