Re: [Distutils] Accessing tests_require and setup_requires from the setup.py

[Sebastien Awwad]

This ties into what I've been working on to fix the package dependency conflict 
resolution problem for pip , actually:

You may be able to use a tool I wrote to automatically extract requirements 
from setup.py, without installing (knowing that setup.py is arbitrary code and 
that dependencies are not strictly static). I opted to go with an admittedly 
drastic method of patching pip 8 to extract dependency data from each source 
distribution it touches in download mode when called by my dependency scraper. 
I decided that in the absence of static requirements for source distributions, 
the best I could really do in practice was to parse requirements exactly the 
way pip does. If you want, you can run the scraper from my project, which is 
here (project itself still a WIP). In particular, if you install it and run 
'python depresolve/scrape_deps_and_detect_conflicts.py 
"some-package-name(1.0.0)"', it'll spit out the dependencies to a json file for 
the sdist for version 1.0.0 of some-package-name (more instructions here 
 - it can 
also operate with local sdists or indexes).

In my case, for pypa/pip:issue988, I needed to harvest mass dependency info to 
test a few different dependency conflict resolvers on. I'm working on writing 
up some of what I've learned and will probably end up recommending a basic 
integrated backtracking resolver within pip - probably an updated version of 
rbtcollins' backtracking resolver pip patches 
 (which I'd be happy to rework and send 
a PR to pip on, if Robert doesn't have the bandwidth for it).

Sebastien

> On Jun 3, 2016, at 09:58, Daniel Holth  wrote:
> 
> Here is how you can write setup_requires and test_requires to a file, by 
> adding a plugin to egg_info.writers in setuptools.
> 
> https://gist.github.com/dholth/59e4c8a0c0d963b019d81e18bf0a89e3 
> 
> 
> On Fri, Jun 3, 2016 at 9:29 AM Paul Moore  > wrote:
> On 3 June 2016 at 14:24, Christopher Baines  > wrote:
> > On 03/06/16 14:19, Paul Moore wrote:
> >> On 3 June 2016 at 13:20, Christopher Baines  >> > wrote:
> >>> I'm trying to write a script to get information about a source
> >>> distributions requirements (from the source distribution), but I'm not
> >>> sure how to access the tests_require and setup_requires that can
> >>> sometimes be found in the setup.py?
> >>>
> >>> Apologies if this is really simple, and I've just missed the answer, but
> >>> I've searched for it a few times now, and not come up with anything.
> >>
> >> If I understand what you're trying to achieve, the only way of getting
> >> the "final" information (i.e, what will actually get used to install)
> >> is by running the setup.py script. That's basically the key issue with
> >> the executable setup.py format - there's no way to know the
> >> information without running the script.
> >>
> >> You may be able to get the information without doing a full install by
> >> using the "setup.py egg_info" subcommand provided by setuptools.
> >> That's what pip uses, for example (but pip doesn't look at
> >> tests_require or setup_requires, so you'd have to check if that
> >> information was available by that route).
> >
> > As far as I can see (I checked setuptools and flake8), neither
> > tests_require or setup_requires are present in the egg_info metadata
> > directory.
> >
> > Is there no way of getting setuptools to write the data out to a file?
> 
> Maybe you could write your own command class? Or monkeypatch
> setuptools.setup() to write its arguments to a file?
> 
> I don't know of any non-ugly way, though, sorry...
> Paul
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org 
> 
> https://mail.python.org/mailman/listinfo/distutils-sig 
> 
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Accessing tests_require and setup_requires from the setup.py

[Sebastien Awwad]

- I'm not at all familiar with composer.
- For DNF - I assume you mean CNF, logical expression format generally used
in SAT solvers (not DNF as in I did-not-finish the blog post I'm working on
on this yet).

If the question is why backtracking rather than SAT solving, I start to get
into that in background here
.
I'm hesitant to link to this scribbling. I'll have a broader summary with
data later - I've run some comparisons using enthought/depsolver's SAT
solver, along with some backtrackers (the pip patch previously linked to
and a currently-flawed one of my own).

On Fri, Jun 3, 2016 at 12:46 PM Daniel Holth  wrote:

> Tell me what you know about SAT solvers, dnf and composer.
>
> On Fri, Jun 3, 2016, 12:28 Sebastien Awwad 
> wrote:
>
>> This ties into what I've been working on to fix the package dependency
>> conflict resolution problem for pip
>> , actually:
>>
>> You may be able to use a tool I wrote to automatically extract
>> requirements from setup.py, without installing (knowing that setup.py is
>> arbitrary code and that dependencies are not strictly static). I opted
>> to go with an admittedly drastic method of patching pip 8 to extract
>> dependency data from each source distribution it touches in download mode
>> when called by my dependency scraper. I decided that in the absence of
>> static requirements for source distributions, the best I could really do in
>> practice was to parse requirements exactly the way pip does. If you want,
>> you can run the scraper from my project, which is here (project itself
>> still a WIP). In particular, if you install it and run 'python 
>> depresolve/scrape_deps_and_detect_conflicts.py
>> "some-package-name(1.0.0)"', it'll spit out the dependencies to a json
>> file for the sdist for version 1.0.0 of some-package-name (more
>> instructions here
>>  - it
>> can also operate with local sdists or indexes).
>>
>> In my case, for pypa/pip:issue988, I needed to harvest mass dependency
>> info to test a few different dependency conflict resolvers on. I'm working
>> on writing up some of what I've learned and will probably end up
>> recommending a basic integrated backtracking resolver within pip - probably
>> an updated version of rbtcollins' backtracking resolver pip patches
>>  (which I'd be happy to rework
>> and send a PR to pip on, if Robert doesn't have the bandwidth for it).
>>
>> Sebastien
>>
> On Jun 3, 2016, at 09:58, Daniel Holth  wrote:
>>
>> Here is how you can write setup_requires and test_requires to a file, by
>> adding a plugin to egg_info.writers in setuptools.
>>
>> https://gist.github.com/dholth/59e4c8a0c0d963b019d81e18bf0a89e3
>>
>> On Fri, Jun 3, 2016 at 9:29 AM Paul Moore  wrote:
>>
>>> On 3 June 2016 at 14:24, Christopher Baines  wrote:
>>> > On 03/06/16 14:19, Paul Moore wrote:
>>> >> On 3 June 2016 at 13:20, Christopher Baines  wrote:
>>> >>> I'm trying to write a script to get information about a source
>>> >>> distributions requirements (from the source distribution), but I'm
>>> not
>>> >>> sure how to access the tests_require and setup_requires that can
>>> >>> sometimes be found in the setup.py?
>>> >>>
>>> >>> Apologies if this is really simple, and I've just missed the answer,
>>> but
>>> >>> I've searched for it a few times now, and not come up with anything.
>>> >>
>>> >> If I understand what you're trying to achieve, the only way of getting
>>> >> the "final" information (i.e, what will actually get used to install)
>>> >> is by running the setup.py script. That's basically the key issue with
>>> >> the executable setup.py format - there's no way to know the
>>> >> information without running the script.
>>> >>
>>> >> You may be able to get the information without doing a full install by
>>> >> using the "setup.py egg_info" subcommand provided by setuptools.
>>> >> That's what pip uses, for example (but pip doesn't look at
>>> >> tests_require or setup_requires, so you'd have to check if that
>>> >> information was available by that route).
>>> >
>>> > As far as I can see (I checked setuptools and flake8), neither
>>> > tests_require or setup_requires are present in the egg_info metadata
>>> > directory.
>>> >
>>> > Is there no way of getting setuptools to write the data out to a file?
>>>
>>> Maybe you could write your own command class? Or monkeypatch
>>> setuptools.setup() to write its arguments to a file?
>>>
>>> I don't know of any non-ugly way, though, sorry...
>>> Paul
>>> ___
>>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>>
>> 

Re: [Distutils] Accessing tests_require and setup_requires from the setup.py

[Christopher Baines]

On 03/06/16 14:19, Paul Moore wrote:
> On 3 June 2016 at 13:20, Christopher Baines  wrote:
>> I'm trying to write a script to get information about a source
>> distributions requirements (from the source distribution), but I'm not
>> sure how to access the tests_require and setup_requires that can
>> sometimes be found in the setup.py?
>>
>> Apologies if this is really simple, and I've just missed the answer, but
>> I've searched for it a few times now, and not come up with anything.
> 
> If I understand what you're trying to achieve, the only way of getting
> the "final" information (i.e, what will actually get used to install)
> is by running the setup.py script. That's basically the key issue with
> the executable setup.py format - there's no way to know the
> information without running the script.
> 
> You may be able to get the information without doing a full install by
> using the "setup.py egg_info" subcommand provided by setuptools.
> That's what pip uses, for example (but pip doesn't look at
> tests_require or setup_requires, so you'd have to check if that
> information was available by that route).

As far as I can see (I checked setuptools and flake8), neither
tests_require or setup_requires are present in the egg_info metadata
directory.

Is there no way of getting setuptools to write the data out to a file?

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Accessing tests_require and setup_requires from the setup.py

[Tres Seaver]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 06/03/2016 09:58 AM, Daniel Holth wrote:
> Here is how you can write setup_requires and test_requires to a file,
> by adding a plugin to egg_info.writers in setuptools.

FWIW, eggtestinfo is an 'egg_info.writers' plugin which dumps
test-related metadata to a text file:

 https://pypi.python.org/pypi/eggtestinfo


Tres.
- -- 
===
Tres Seaver  +1 540-429-0999  tsea...@palladion.com
Palladion Software   "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iQIcBAEBAgAGBQJXUdRpAAoJEPKpaDSJE9HYQhsP/jrMq4q1BgCCMCaazvo9B4vX
uS9uzHj+T2noHwMcluSQB9zdSvCufNbKSoXQ62MUkbPEKwa1TarKzm+XZci/tBba
eOxwEPilHHRBF7no44ZTgo6kq27oOC9Z0em7F9awADxWmJ0mcgOksQTXjzczuP7h
BPoj1v/vxSLc87Q61qXYWMFTpiY4SjHn7+0xu7txQ13GJSaNjqNK/geQI+18Tpw2
76G0I1TDbXel+2JlgYEmwHkFPpFuhVaJXyJi8ZjvjdH/jN47oT6wsFwyxRzlVOXx
oFp//hdb05K9i2R9rdLznKzkrQCrlYUTEBXEIpMCacfnoHgKrn+MKHCdI4FXKXGz
Rm8LUh+oS0VU5IA8Rt4od08Qd68HE9CQERvx03JOQ7mKJIjxEODeC80JduzVGZxj
7YpA5sRxbsDTbnHD+8uodpFDDU0h9NMWLbxdDJ42Q1j2iNNr7ZZDR2tJZ8Y76EMl
55Mf5WZK5W+kzkUVodpK1yTxLR8+swadfjno2QnVhjA9h3vW1ZV11AREPTLauHbE
S+eoyN92ga4eyhZSwaQdhz6KsZgFpUcmhwvlrsl0dIlkkHs1mFjPL/J4PvR2SVKl
ug1Wmm0EvwPUQ0cDcRBeJtPF4git5SeMYLvGcMbsIpHdZloK+xaFVyMjgrTGA1o5
NTZokg2p47XvwuCA4XlH
=d2oJ
-END PGP SIGNATURE-

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Accessing tests_require and setup_requires from the setup.py

[Daniel Holth]

Tell me what you know about SAT solvers, dnf and composer.

On Fri, Jun 3, 2016, 12:28 Sebastien Awwad  wrote:

> This ties into what I've been working on to fix the package dependency
> conflict resolution problem for pip
> , actually:
>
> You may be able to use a tool I wrote to automatically extract
> requirements from setup.py, without installing (knowing that setup.py is
> arbitrary code and that dependencies are not strictly static). I opted to
> go with an admittedly drastic method of patching pip 8 to extract
> dependency data from each source distribution it touches in download mode
> when called by my dependency scraper. I decided that in the absence of
> static requirements for source distributions, the best I could really do in
> practice was to parse requirements exactly the way pip does. If you want,
> you can run the scraper from my project, which is here (project itself
> still a WIP). In particular, if you install it and run 'python 
> depresolve/scrape_deps_and_detect_conflicts.py
> "some-package-name(1.0.0)"', it'll spit out the dependencies to a json
> file for the sdist for version 1.0.0 of some-package-name (more
> instructions here
>  - it
> can also operate with local sdists or indexes).
>
> In my case, for pypa/pip:issue988, I needed to harvest mass dependency
> info to test a few different dependency conflict resolvers on. I'm working
> on writing up some of what I've learned and will probably end up
> recommending a basic integrated backtracking resolver within pip - probably
> an updated version of rbtcollins' backtracking resolver pip patches
>  (which I'd be happy to rework and
> send a PR to pip on, if Robert doesn't have the bandwidth for it).
>
> Sebastien
>
> On Jun 3, 2016, at 09:58, Daniel Holth  wrote:
>
> Here is how you can write setup_requires and test_requires to a file, by
> adding a plugin to egg_info.writers in setuptools.
>
> https://gist.github.com/dholth/59e4c8a0c0d963b019d81e18bf0a89e3
>
> On Fri, Jun 3, 2016 at 9:29 AM Paul Moore  wrote:
>
>> On 3 June 2016 at 14:24, Christopher Baines  wrote:
>> > On 03/06/16 14:19, Paul Moore wrote:
>> >> On 3 June 2016 at 13:20, Christopher Baines  wrote:
>> >>> I'm trying to write a script to get information about a source
>> >>> distributions requirements (from the source distribution), but I'm not
>> >>> sure how to access the tests_require and setup_requires that can
>> >>> sometimes be found in the setup.py?
>> >>>
>> >>> Apologies if this is really simple, and I've just missed the answer,
>> but
>> >>> I've searched for it a few times now, and not come up with anything.
>> >>
>> >> If I understand what you're trying to achieve, the only way of getting
>> >> the "final" information (i.e, what will actually get used to install)
>> >> is by running the setup.py script. That's basically the key issue with
>> >> the executable setup.py format - there's no way to know the
>> >> information without running the script.
>> >>
>> >> You may be able to get the information without doing a full install by
>> >> using the "setup.py egg_info" subcommand provided by setuptools.
>> >> That's what pip uses, for example (but pip doesn't look at
>> >> tests_require or setup_requires, so you'd have to check if that
>> >> information was available by that route).
>> >
>> > As far as I can see (I checked setuptools and flake8), neither
>> > tests_require or setup_requires are present in the egg_info metadata
>> > directory.
>> >
>> > Is there no way of getting setuptools to write the data out to a file?
>>
>> Maybe you could write your own command class? Or monkeypatch
>> setuptools.setup() to write its arguments to a file?
>>
>> I don't know of any non-ugly way, though, sorry...
>> Paul
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Accessing tests_require and setup_requires from the setup.py

[Daniel Holth]

Here is how you can write setup_requires and test_requires to a file, by
adding a plugin to egg_info.writers in setuptools.

https://gist.github.com/dholth/59e4c8a0c0d963b019d81e18bf0a89e3

On Fri, Jun 3, 2016 at 9:29 AM Paul Moore  wrote:

> On 3 June 2016 at 14:24, Christopher Baines  wrote:
> > On 03/06/16 14:19, Paul Moore wrote:
> >> On 3 June 2016 at 13:20, Christopher Baines  wrote:
> >>> I'm trying to write a script to get information about a source
> >>> distributions requirements (from the source distribution), but I'm not
> >>> sure how to access the tests_require and setup_requires that can
> >>> sometimes be found in the setup.py?
> >>>
> >>> Apologies if this is really simple, and I've just missed the answer,
> but
> >>> I've searched for it a few times now, and not come up with anything.
> >>
> >> If I understand what you're trying to achieve, the only way of getting
> >> the "final" information (i.e, what will actually get used to install)
> >> is by running the setup.py script. That's basically the key issue with
> >> the executable setup.py format - there's no way to know the
> >> information without running the script.
> >>
> >> You may be able to get the information without doing a full install by
> >> using the "setup.py egg_info" subcommand provided by setuptools.
> >> That's what pip uses, for example (but pip doesn't look at
> >> tests_require or setup_requires, so you'd have to check if that
> >> information was available by that route).
> >
> > As far as I can see (I checked setuptools and flake8), neither
> > tests_require or setup_requires are present in the egg_info metadata
> > directory.
> >
> > Is there no way of getting setuptools to write the data out to a file?
>
> Maybe you could write your own command class? Or monkeypatch
> setuptools.setup() to write its arguments to a file?
>
> I don't know of any non-ugly way, though, sorry...
> Paul
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Accessing tests_require and setup_requires from the setup.py

[Paul Moore]

On 3 June 2016 at 14:24, Christopher Baines  wrote:
> On 03/06/16 14:19, Paul Moore wrote:
>> On 3 June 2016 at 13:20, Christopher Baines  wrote:
>>> I'm trying to write a script to get information about a source
>>> distributions requirements (from the source distribution), but I'm not
>>> sure how to access the tests_require and setup_requires that can
>>> sometimes be found in the setup.py?
>>>
>>> Apologies if this is really simple, and I've just missed the answer, but
>>> I've searched for it a few times now, and not come up with anything.
>>
>> If I understand what you're trying to achieve, the only way of getting
>> the "final" information (i.e, what will actually get used to install)
>> is by running the setup.py script. That's basically the key issue with
>> the executable setup.py format - there's no way to know the
>> information without running the script.
>>
>> You may be able to get the information without doing a full install by
>> using the "setup.py egg_info" subcommand provided by setuptools.
>> That's what pip uses, for example (but pip doesn't look at
>> tests_require or setup_requires, so you'd have to check if that
>> information was available by that route).
>
> As far as I can see (I checked setuptools and flake8), neither
> tests_require or setup_requires are present in the egg_info metadata
> directory.
>
> Is there no way of getting setuptools to write the data out to a file?

Maybe you could write your own command class? Or monkeypatch
setuptools.setup() to write its arguments to a file?

I don't know of any non-ugly way, though, sorry...
Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Accessing tests_require and setup_requires from the setup.py

[Paul Moore]

On 3 June 2016 at 13:20, Christopher Baines  wrote:
> I'm trying to write a script to get information about a source
> distributions requirements (from the source distribution), but I'm not
> sure how to access the tests_require and setup_requires that can
> sometimes be found in the setup.py?
>
> Apologies if this is really simple, and I've just missed the answer, but
> I've searched for it a few times now, and not come up with anything.

If I understand what you're trying to achieve, the only way of getting
the "final" information (i.e, what will actually get used to install)
is by running the setup.py script. That's basically the key issue with
the executable setup.py format - there's no way to know the
information without running the script.

You may be able to get the information without doing a full install by
using the "setup.py egg_info" subcommand provided by setuptools.
That's what pip uses, for example (but pip doesn't look at
tests_require or setup_requires, so you'd have to check if that
information was available by that route).

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Accessing tests_require and setup_requires from the setup.py

[Christopher Baines]

I'm trying to write a script to get information about a source
distributions requirements (from the source distribution), but I'm not
sure how to access the tests_require and setup_requires that can
sometimes be found in the setup.py?

Apologies if this is really simple, and I've just missed the answer, but
I've searched for it a few times now, and not come up with anything.

Thanks,

Chris
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig