Re: [Distutils] deprecating pip install --target

[Mark Niedzielski]

A new, important, use of --target is in creating installation bundles
for AWS Lambda functions using a Python runtime.  AWS expects a zip file
containing the primary source file and all dependent packages.  'pip
install -t' creates exactly the structure needed.  Further, because a
'pip install -t .' has the same behavior as a 'npm install' we can
easily use the same packaging code for Python and Node Lambda function
bundles.

Thanks.

-Mark

This e-mail message and all attachments transmitted with it may contain 
privileged and/or confidential information intended solely for the use of the 
addressee(s). If the reader of this message is not the intended recipient, you 
are hereby notified that any reading, dissemination, distribution, copying, 
forwarding or other use of this message or its attachments is strictly 
prohibited. If you have received this message in error, please notify the 
sender immediately and delete this message, all attachments and all copies and 
backups thereof.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] deprecating pip install --target

[Jon Parrott via Distutils-SIG]

Currently, Google App Engine recommends that users use "--target" to vendor
packages:
https://cloud.google.com/appengine/docs/python/tools/libraries27?hl=en#vendoring

Ostensibly we also support virtualenvs with only pure-python packages.

> Alternatively, it should be possible to *detect* the problem cases, so
> why not do that, and reject them? Effectively, reduce the scope of
> --target to pure-python wheels only.

This would be acceptable for the App Engine use case.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] deprecating pip install --target

[Wolfgang Maier]

On 17.02.2016 11:09, Paul Moore wrote:

On 16 February 2016 at 22:52, Robert Collins  wrote:

An alternative would be great, though I can probably fake things somehow for
my purposes.


Sounds similar to Daniel's need - and again, --prefix + setting PATH
and PYTHONPATH would be better.


Note that if I read the help for --prefix correctly, "pip install
--target x foo" puts foo in x, whereas "pip install --prefix x foo"
puts foo in x/lib. So how would setting prefix allow me to put foo in
x, and not in a subdirectory? That is specifically my requirement (and
the vendoring requirement in general).


I just discovered a way that lets you do just that, but before using it 
for my own code I'd like to know whether you would consider it a weird 
hack that will probably not work in the future or something reasonable 
that could be used in production code? Here's the trick:


write a temporary distutils setup.cfg file in the current working 
directory with the content:


[install]
install-lib=abspath/to/target_dir

then run pip from that directory like so:

pip install packagexy --prefix abspath/to/target_dir

Of note, combining a local setup.cfg file and --prefix like this isn't 
documented and to me it wasn't even clear whether the file would be 
expected in the current working directory that pip gets run in or in the 
downloaded package.


What is more, the local setup.cfg file can also be used to specify a 
complete installation scheme for use without the --prefix option.


As I said I'm really interested in your opinions,
Wolfgang

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] deprecating pip install --target

[Steve Dower]

Another alternative is making "pip the library" as has been discussed in the 
past. 

Certainly my needs would be satisfied by a library that can get me as far as 
wheel files given package name/s and version spec (and index URL I guess). 
Unpacking wheels isn't especially difficult and in my case I know there are no 
existing installs to worry about.

Top-posted from my Windows Phone

-Original Message-
From: "Paul Moore" <p.f.mo...@gmail.com>
Sent: ‎2/‎17/‎2016 2:10
To: "Robert Collins" <robe...@robertcollins.net>
Cc: "Steve Dower" <pyt...@stevedower.id.au>; "Python Distutils SIG" 
<Distutils-Sig@python.org>
Subject: Re: [Distutils] deprecating pip install --target

On 16 February 2016 at 22:52, Robert Collins <robe...@robertcollins.net> wrote:
>> An alternative would be great, though I can probably fake things somehow for
>> my purposes.
>
> Sounds similar to Daniel's need - and again, --prefix + setting PATH
> and PYTHONPATH would be better.

Note that if I read the help for --prefix correctly, "pip install
--target x foo" puts foo in x, whereas "pip install --prefix x foo"
puts foo in x/lib. So how would setting prefix allow me to put foo in
x, and not in a subdirectory? That is specifically my requirement (and
the vendoring requirement in general). I *know* that means there's no
obvious place to put data files or extensions or whatever, and that's
fine by me.

It seems that if we want to go down this route, we need to include the
full set of --install-purelib, --install-platlib, --install-scripts
etc arguments to pip. But that's probably the wrong solution - if we
want to start playing with the various install location parameters to
pip install (--target, --prefix, --root) we should probably do a
"proper" job and just find a way to allow user-defined schemes.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] deprecating pip install --target

[Paul Moore]

On 16 February 2016 at 22:52, Robert Collins  wrote:
>> An alternative would be great, though I can probably fake things somehow for
>> my purposes.
>
> Sounds similar to Daniel's need - and again, --prefix + setting PATH
> and PYTHONPATH would be better.

Note that if I read the help for --prefix correctly, "pip install
--target x foo" puts foo in x, whereas "pip install --prefix x foo"
puts foo in x/lib. So how would setting prefix allow me to put foo in
x, and not in a subdirectory? That is specifically my requirement (and
the vendoring requirement in general). I *know* that means there's no
obvious place to put data files or extensions or whatever, and that's
fine by me.

It seems that if we want to go down this route, we need to include the
full set of --install-purelib, --install-platlib, --install-scripts
etc arguments to pip. But that's probably the wrong solution - if we
want to start playing with the various install location parameters to
pip install (--target, --prefix, --root) we should probably do a
"proper" job and just find a way to allow user-defined schemes.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] deprecating pip install --target

[Robert Collins]

On 13 February 2016 at 03:54, Steve Dower  wrote:
> I was also planning to use it in an upcoming project that has to "do its
> own" package management. The aim was to install different versions of
> packages in different directories and use sys.path modifications to resolve
> them at runtime (kind of like what setuptools did in the older days).
>
> An alternative would be great, though I can probably fake things somehow for
> my purposes.

Sounds similar to Daniel's need - and again, --prefix + setting PATH
and PYTHONPATH would be better.

-Rob



-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] deprecating pip install --target

[Robert Collins]

On 13 February 2016 at 04:12, Daniel Holth  wrote:
> My setup-requires wrapper, adding pre-setup.py dependency installation to
> setup.py, relies on this feature. It needs to install something in a
> directory that is only added to PYTHONPATH during the installation and does
> not interfere with the normal environment.

Python packages that create (and need) scripts, or use data_files,
won't work with --target. Using a --prefix and setting PYTHONPATH
*and* PATH appropriately would be better.

-Rob


-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] deprecating pip install --target

[Steve Dower]

I was also planning to use it in an upcoming project that has to "do its own" 
package management. The aim was to install different versions of packages in 
different directories and use sys.path modifications to resolve them at runtime 
(kind of like what setuptools did in the older days).

An alternative would be great, though I can probably fake things somehow for my 
purposes.

Cheers,
Steve

Top-posted from my Windows Phone

-Original Message-
From: "Vinay Sajip" <vinay_sa...@yahoo.co.uk>
Sent: ‎2/‎11/‎2016 15:18
To: "Distutils-Sig@Python.Org" <Distutils-Sig@Python.Org>
Subject: Re: [Distutils] deprecating pip install --target

Robert Collins  robertcollins.net> writes:

> 
> This is fairly broken - it doesn't handle platlib vs purelib (see pip
> PR 3450), doesn't handle data files, or any other layout. Donald says
> pip uses it to maintain the _vendor subtrees only, which doesn't seem
> like a particularly strong use case.
> 
> Certainly the help description for it is misleading - since what we're
> actually doing is copying only a subset of what the package installed
> - so at a minimum we need to be much clearer about what it does.
> 
> But, I think it would be better to deprecate it and remove it... so
> I'm pinging here to see if anyone can explain a sensible use case for
> it in the context of pip :)

I use it in pretty much the same way as Paul mentioned - I wouldn't like it 
to go unless something equivalent is available. Updating the help / 
documentation for it to better reflect what it does would be uncontroversial 
for me, but I see no strong reason for deprecation and removal. As Paul 
suggests, it can get stricter about what it'll handle.

Regards,

Vinay Sajip

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] deprecating pip install --target

[Daniel Holth]

My setup-requires wrapper, adding pre-setup.py dependency installation to
setup.py, relies on this feature. It needs to install something in a
directory that is only added to PYTHONPATH during the installation and does
not interfere with the normal environment.

On Fri, Feb 12, 2016 at 9:55 AM Steve Dower <pyt...@stevedower.id.au> wrote:

> I was also planning to use it in an upcoming project that has to "do its
> own" package management. The aim was to install different versions of
> packages in different directories and use sys.path modifications to resolve
> them at runtime (kind of like what setuptools did in the older days).
>
> An alternative would be great, though I can probably fake things somehow
> for my purposes.
>
> Cheers,
> Steve
>
> Top-posted from my Windows Phone
> --
> From: Vinay Sajip <vinay_sa...@yahoo.co.uk>
> Sent: ‎2/‎11/‎2016 15:18
> To: Distutils-Sig@Python.Org
> Subject: Re: [Distutils] deprecating pip install --target
>
> Robert Collins  robertcollins.net> writes:
>
> >
> > This is fairly broken - it doesn't handle platlib vs purelib (see pip
> > PR 3450), doesn't handle data files, or any other layout. Donald says
> > pip uses it to maintain the _vendor subtrees only, which doesn't seem
> > like a particularly strong use case.
> >
> > Certainly the help description for it is misleading - since what we're
> > actually doing is copying only a subset of what the package installed
> > - so at a minimum we need to be much clearer about what it does.
> >
> > But, I think it would be better to deprecate it and remove it... so
> > I'm pinging here to see if anyone can explain a sensible use case for
> > it in the context of pip :)
>
> I use it in pretty much the same way as Paul mentioned - I wouldn't like
> it
> to go unless something equivalent is available. Updating the help /
> documentation for it to better reflect what it does would be
> uncontroversial
> for me, but I see no strong reason for deprecation and removal. As Paul
> suggests, it can get stricter about what it'll handle.
>
> Regards,
>
> Vinay Sajip
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] deprecating pip install --target

[Robert Collins]

This is fairly broken - it doesn't handle platlib vs purelib (see pip
PR 3450), doesn't handle data files, or any other layout. Donald says
pip uses it to maintain the _vendor subtrees only, which doesn't seem
like a particularly strong use case.

Certainly the help description for it is misleading - since what we're
actually doing is copying only a subset of what the package installed
- so at a minimum we need to be much clearer about what it does.

But, I think it would be better to deprecate it and remove it... so
I'm pinging here to see if anyone can explain a sensible use case for
it in the context of pip :)

-Rob



-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] deprecating pip install --target

[Paul Moore]

On 11 February 2016 at 19:07, Robert Collins  wrote:
> But, I think it would be better to deprecate it and remove it... so
> I'm pinging here to see if anyone can explain a sensible use case for
> it in the context of pip :)

I have used it in the past to bundle libraries in an application I'm
building using zipapp. So I'd write my __main__.py, create a "libs"
subdirectory, and "pip install --target libs" any dependencies I have.
Because it's a zipped application, I'm only including pure-Python
libraries so as long as they don't have data files they are fine.
That's basically the same use case as vendoring.

It's not something I do a lot, but I'd like there to be at a minimum,
a documented replacement approach (that's roughly as convenient -
"install your dependencies somewhere and copy them" or "download a
wheel and unzip it manually" aren't really what I have in mind). I do
think that creating zipped applications *should* be a reasonable use
case - I think it's a woefully under-used deployment option, and I'd
hate to see more obstacles put in the way of people using it.

Alternatively, it should be possible to *detect* the problem cases, so
why not do that, and reject them? Effectively, reduce the scope of
--target to pure-python wheels only.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] deprecating pip install --target

[Vinay Sajip]

Robert Collins  robertcollins.net> writes:

> 
> This is fairly broken - it doesn't handle platlib vs purelib (see pip
> PR 3450), doesn't handle data files, or any other layout. Donald says
> pip uses it to maintain the _vendor subtrees only, which doesn't seem
> like a particularly strong use case.
> 
> Certainly the help description for it is misleading - since what we're
> actually doing is copying only a subset of what the package installed
> - so at a minimum we need to be much clearer about what it does.
> 
> But, I think it would be better to deprecate it and remove it... so
> I'm pinging here to see if anyone can explain a sensible use case for
> it in the context of pip :)

I use it in pretty much the same way as Paul mentioned - I wouldn't like it 
to go unless something equivalent is available. Updating the help / 
documentation for it to better reflect what it does would be uncontroversial 
for me, but I see no strong reason for deprecation and removal. As Paul 
suggests, it can get stricter about what it'll handle.

Regards,

Vinay Sajip

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig