Re: [Distutils] Parsing requirements, pip has no API ...

[Marc Abramowitz]

Agree 100% with dstufft and have long been pointing people to his blog post at 
caremad.io. Incidentally his post maps closely to (and references) a blog post 
from the Ruby world about Bundler and Gemfile vs. gemspecs, because this is a 
general concept that applies to all platforms (not really unique to Python) and 
this is backed up by the .deb vs. Chef example. 

In short, dependencies for building 
reusable blocks (libraries/packages) are different than dependencies for 
building non-reusable apps/systems/environments. The former emphasizes 
reusability and flexibility and minimal pinning. The latter instead emphasizes 
repeatability and thus constraining tightly. 

I don't think requirements.txt is an anti-pattern. I do feel like things that 
suck the contents of a requirements.txt into the setup.py (like pbr) are an 
anti-pattern. 
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Parsing requirements, pip has no API ...

[Thomas Güttler]

Am 14.02.2015 um 00:11 schrieb Donald Stufft:
...

> requirements.txt and setup.py serve different purposes, requirements.txt is 
> for
> an environment, setup.py is for a package. It doesn't make sense for a 
> setup.py
> to read from a requirement.txt just like it wouldn't make sense for a deb
> package to read from a Chef cookbook.
> 
> Often the reason people do this is they want to support people installing 
> their
> thing with ``pip install -r requirements.txt`` from within a check out without
> needing to list their dependencies twice. That's a reasonable thing to want
> which is why the requirements file format has a construct that enables it,
> simply make a requirements.txt file that contains "." or "-e ." and pip will
> automatically install the project and all of it's dependencies.
> 
> For more information, see https://caremad.io/2013/07/setup-vs-requirement/.

thank you very much for your answer. I understand the topic better. The
comparison to a chef cookbook was good.




-- 
http://www.thomas-guettler.de/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Parsing requirements, pip has no API ...

[Donald Stufft]

> On Feb 13, 2015, at 4:44 PM, Ian Cordasco  wrote:
> 
> pip is a command-line tool. The fact that you can import it doesn't
> make it a library. If it documents no public API then it has none and
> you shouldn't be relying on things that you can import from pip. You
> can import requests from pip but you shouldn't do that either.
> 
> There seems to be a need for a separate library for this use case but
> there isn't at the moment. In general, requirements.txt seems to be an
> anti-pattern. You either have to use likely to break tooling or you'll
> have to reinvent that from scratch. You're better off putting it
> directly in setup.py and using setup.py to install dependencies in a
> virtualenv instead of requirements.txt
> 
> On Fri, Feb 13, 2015 at 3:27 PM, Thomas Güttler
>  wrote:
>> I was told:
>> 
>> {{{
>> Pip does not have a public API and because of that there is no backwards 
>> compatibility contract. It's impossible to fully parse every type of 
>> requirements.txt without a session so either parse_requirements needs to 
>> create one if it doesn't (which means if we forget to pass in a session 
>> somewhere it'll use the wrong one) or it needs one passed in.
>> }}}
>> From https://github.com/pypa/pip/issues/2422#issuecomment-74271718
>> 
>> 
>> Up to now we used parse_requirements() of pip, but in new versions you need 
>> to pass in a
>> session.
>> 
>> If I see changes like this:
>> 
>> setup.py
>> - install_requires=[str(req.req) for req in 
>> parse_requirements("requirements.txt")],
>> + install_requires=[str(req.req) for req in 
>> parse_requirements("requirements.txt", session=uuid.uuid1())],
>> 
>> ... I think something is wrong.
>> 
>> 
>> I am not an expert in python packaging details. I just want it to work.
>> 
>> What is wrong here?
>> 
>>  - You should not use parse_requirements() in setup.py
>>  - pip should not change its API.
>>  - you should not use pip at all, you should use ...?
>> 
>> Regards,
>>  Thomas
>> 
>> 
>> --
>> http://www.thomas-guettler.de/
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

requirements.txt and setup.py serve different purposes, requirements.txt is for
an environment, setup.py is for a package. It doesn't make sense for a setup.py
to read from a requirement.txt just like it wouldn't make sense for a deb
package to read from a Chef cookbook.

Often the reason people do this is they want to support people installing their
thing with ``pip install -r requirements.txt`` from within a check out without
needing to list their dependencies twice. That's a reasonable thing to want
which is why the requirements file format has a construct that enables it,
simply make a requirements.txt file that contains "." or "-e ." and pip will
automatically install the project and all of it's dependencies.

For more information, see https://caremad.io/2013/07/setup-vs-requirement/.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Parsing requirements, pip has no API ...

[Marcus Smith]

In 90% of the cases I see, requirements.txt are used to define the

> requirements for the project to function which typically are the exact
> same requirements necessary when installing the project. People also
> will then write a test-requirements.txt (or dev-requirements.txt) file
> to have a complete development environment. In this case, Thomas seems
> to be using requirements.txt to define the requirements necessary when
> installing the software they're developing.
>
> If requirements.txt were used solely for a development environment,
> they would look more like
>
> | .
> | dev-requirement-1>=0.1
> | # etc.
>
> Instead they seem to be used more to define the same requirements
> someone would define in install_requires. This is the anti-pattern I'm
> talking about.
>

understood, I hear you, but for a breakdown of some valid use cases, the
pip docs covers 4 good use cases

https://pip.pypa.io/en/latest/user_guide.html#requirements-files
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Parsing requirements, pip has no API ...

[Ian Cordasco]

On Fri, Feb 13, 2015 at 3:59 PM, Marcus Smith  wrote:
>
>> In general, requirements.txt seems to be an
>> anti-pattern. You either have to use likely to break tooling or you'll
>> have to reinvent that from scratch. You're better off putting it
>> directly in setup.py and using setup.py to install dependencies in a
>> virtualenv instead of requirements.txt
>
>
> I don't know your context for calling it an anti-pattern, but there are
> valid use cases for requirements.txt vs install_requires.
> here's what the "Python Packaging User Guide" has on the distinction
>
> https://packaging.python.org/en/latest/requirements.html
>
> skipping to the distinctions, it lists four:
>
> Whereas install_requires defines the dependencies for a single project,
> Requirements Files are often used to define the requirements for a complete
> python environment.
>
> Whereas install_requires requirements are minimal, requirements files often
> contain an exhaustive listing of pinned versions for the purpose of
> achieving repeatable installations of a complete environment.
>
> Whereas install_requires requirements are “Abstract”, requirements files
> often contain pip options like --index-url or --find-links to make
> requirements “Concrete”. [1]
>
> Whereas install_requires metadata is automatically analyzed by pip during an
> install, requirements files are not, and only are used when a user
> specifically installs them using pip install -r.


In 90% of the cases I see, requirements.txt are used to define the
requirements for the project to function which typically are the exact
same requirements necessary when installing the project. People also
will then write a test-requirements.txt (or dev-requirements.txt) file
to have a complete development environment. In this case, Thomas seems
to be using requirements.txt to define the requirements necessary when
installing the software they're developing.

If requirements.txt were used solely for a development environment,
they would look more like

| .
| dev-requirement-1>=0.1
| # etc.

Instead they seem to be used more to define the same requirements
someone would define in install_requires. This is the anti-pattern I'm
talking about.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Parsing requirements, pip has no API ...

[Marcus Smith]

> In general, requirements.txt seems to be an
> anti-pattern. You either have to use likely to break tooling or you'll
> have to reinvent that from scratch. You're better off putting it
> directly in setup.py and using setup.py to install dependencies in a
> virtualenv instead of requirements.txt
>

I don't know your context for calling it an anti-pattern, but there are
valid use cases for requirements.txt vs install_requires.
here's what the "Python Packaging User Guide" has on the distinction

https://packaging.python.org/en/latest/requirements.html

skipping to the distinctions, it lists four:

Whereas install_requires defines the dependencies for a single
project, *Requirements
Files* 
are often used to define the requirements for a complete python environment.
Whereas install_requires requirements are minimal, requirements files often
contain an exhaustive listing of pinned versions for the purpose of
achieving *repeatable installations*
 of a complete
environment.

Whereas install_requires requirements are “Abstract”, requirements files
often contain pip options like --index-url or --find-links to make
requirements “Concrete”. [1]


Whereas install_requires metadata is automatically analyzed by pip during
an install, requirements files are not, and only are used when a user
specifically installs them using pip install -r.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Parsing requirements, pip has no API ...

[Paul Moore]

On 13 February 2015 at 21:27, Thomas Güttler
 wrote:
>   - You should not use parse_requirements() in setup.py

This one, basically. Ian provided the details.
Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Parsing requirements, pip has no API ...

[Ian Cordasco]

pip is a command-line tool. The fact that you can import it doesn't
make it a library. If it documents no public API then it has none and
you shouldn't be relying on things that you can import from pip. You
can import requests from pip but you shouldn't do that either.

There seems to be a need for a separate library for this use case but
there isn't at the moment. In general, requirements.txt seems to be an
anti-pattern. You either have to use likely to break tooling or you'll
have to reinvent that from scratch. You're better off putting it
directly in setup.py and using setup.py to install dependencies in a
virtualenv instead of requirements.txt

On Fri, Feb 13, 2015 at 3:27 PM, Thomas Güttler
 wrote:
> I was told:
>
> {{{
> Pip does not have a public API and because of that there is no backwards 
> compatibility contract. It's impossible to fully parse every type of 
> requirements.txt without a session so either parse_requirements needs to 
> create one if it doesn't (which means if we forget to pass in a session 
> somewhere it'll use the wrong one) or it needs one passed in.
> }}}
> From https://github.com/pypa/pip/issues/2422#issuecomment-74271718
>
>
> Up to now we used parse_requirements() of pip, but in new versions you need 
> to pass in a
> session.
>
> If I see changes like this:
>
> setup.py
> - install_requires=[str(req.req) for req in 
> parse_requirements("requirements.txt")],
> + install_requires=[str(req.req) for req in 
> parse_requirements("requirements.txt", session=uuid.uuid1())],
>
>  ... I think something is wrong.
>
>
> I am not an expert in python packaging details. I just want it to work.
>
> What is wrong here?
>
>   - You should not use parse_requirements() in setup.py
>   - pip should not change its API.
>   - you should not use pip at all, you should use ...?
>
> Regards,
>   Thomas
>
>
> --
> http://www.thomas-guettler.de/
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Parsing requirements, pip has no API ...

[Thomas Güttler]

I was told:

{{{
Pip does not have a public API and because of that there is no backwards 
compatibility contract. It's impossible to fully parse every type of 
requirements.txt without a session so either parse_requirements needs to create 
one if it doesn't (which means if we forget to pass in a session somewhere 
it'll use the wrong one) or it needs one passed in.
}}}
>From https://github.com/pypa/pip/issues/2422#issuecomment-74271718


Up to now we used parse_requirements() of pip, but in new versions you need to 
pass in a 
session.

If I see changes like this:

setup.py
- install_requires=[str(req.req) for req in 
parse_requirements("requirements.txt")],
+ install_requires=[str(req.req) for req in 
parse_requirements("requirements.txt", session=uuid.uuid1())],

 ... I think something is wrong.


I am not an expert in python packaging details. I just want it to work.

What is wrong here?

  - You should not use parse_requirements() in setup.py
  - pip should not change its API.
  - you should not use pip at all, you should use ...?

Regards,
  Thomas


-- 
http://www.thomas-guettler.de/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig