Re: [Distutils] Distributing packages to offline machines

[Nick Coghlan]

On 7 April 2018 at 12:21, Wes Turner  wrote:
>
>
> On Friday, April 6, 2018, Ben Finney  wrote:
>>
>> Nick Coghlan  writes:
>>
>> > Keep a requirements.txt file or `Pipfile` in source control, then run
>> > CI jobs based on that repo […]
>>
>> What is a “Pipfile”?
>
> https://docs.pipenv.org/
>
> https://docs.pipenv.org/basics/#example-pipfile-pipfile-lock

And some additional info is available in PyPUG's application
dependency management tutorial:
https://packaging.python.org/tutorials/managing-dependencies/

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Distributing packages to offline machines

[Wes Turner]

On Friday, April 6, 2018, Ben Finney  wrote:

> Nick Coghlan  writes:
>
> > Keep a requirements.txt file or `Pipfile` in source control, then run
> > CI jobs based on that repo […]
>
> What is a “Pipfile”?


https://docs.pipenv.org/

https://docs.pipenv.org/basics/#example-pipfile-pipfile-lock


>
> --
>  \  “Software patents provide one more means of controlling access |
>   `\  to information. They are the tool of choice for the internet |
> _o__) highwayman.” —Anthony Taylor |
> Ben Finney
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Distributing packages to offline machines

[Ben Finney]

Nick Coghlan  writes:

> Keep a requirements.txt file or `Pipfile` in source control, then run
> CI jobs based on that repo […]

What is a “Pipfile”?

-- 
 \  “Software patents provide one more means of controlling access |
  `\  to information. They are the tool of choice for the internet |
_o__) highwayman.” —Anthony Taylor |
Ben Finney

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Distributing packages to offline machines

[Nick Coghlan]

On 5 April 2018 at 11:11, Eric Gorr  wrote:
> I had a question about distributing python packages to offline machines when
> the offline machine is running a different OS then a machine with an
> internet connection. The packages I am concerned with are third party upon
> which mine depend.
>
> Based on what I have learned so far, there are three solutions.
>
> (a) Use a CI to run a fleet of machines for each OS one needs to target to
> obtain the OS specific wheels.
>
> The issue I see with (a) is the shear amount work it would take to setup and
> maintain such a system.

Depending on just how many packages and how many operating system
versions are involved, this option may be less work that you
anticipate, thanks to the excellent wagon project from Cloudify:
https://github.com/cloudify-cosmo/wagon

Keep a requirements.txt file or `Pipfile` in source control, then run
CI jobs based on that repo to emit wagon files with all the required
wheels embedded in them (either sourced from PyPI, or built locally in
CI as needed).

Cheers,
Nick.

P.S. See 
https://github.com/cloudify-cosmo/wagon/issues/30#issuecomment-374096806
for the technique of supplying a dummy setup.py file that gives wagon
the metadata it wants, without actually needing to define a full top
level "application" package.

P.P.S Depending on how strict the requirement is for "offline wheel
files" vs "offline prebuilt binary dependencies", it may also be worth
your while to explore
https://www.paypal-engineering.com/2016/09/07/python-packaging-at-paypal/
(regarding the use of conda as a Python platform manager) and
https://conda.io/docs/user-guide/tasks/create-custom-channels.html
(regarding the creation of tar-friendly custom conda channels)

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Distributing packages to offline machines

[Thomas Kluyver]

On Thu, Apr 5, 2018, at 2:11 AM, Eric Gorr wrote:
> (c) use https://warehouse.pypa.io/api-reference/json to look for
> distributed wheels for the target OS and python version and
> download them directly. (This may make for a nice flag to add to
> pip somewhere...the ability to specify what wheel one wants when
> it isn’t for the machine pip is running on)
If it's useful, Pynsist includes some code to find and download wheels
for Windows, regardless of the platform it runs on:
https://github.com/takluyver/pynsist/blob/master/nsist/pypi.py

Pynsist is a tool to make Windows installers. The tool can also run on
Linux and Mac, but it always builds Windows installers. I didn't know
about "pip download --platform" when I wrote this - it may be redundant.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Distributing packages to offline machines

[Wes Turner]

Have you already tried `pip download --platform`?

https://pip.pypa.io/en/stable/reference/pip_download/#cmdoption-platform


It may be worth setting up devpi (maybe in a container) and caching the
packages; particularly for CI:

https://packaging.python.org/guides/index-mirrors-and-caches/

AFAIU, there's no way to specify a limited set of packages and platforms to
mirror with bandersnatch?

On Wednesday, April 4, 2018, Eric Gorr  wrote:

> I had a question about distributing python packages to offline machines
> when the offline machine is running a different OS then a machine with an
> internet connection. The packages I am concerned with are third party upon
> which mine depend.
>
> Based on what I have learned so far, there are three solutions.
>
> (a) Use a CI to run a fleet of machines for each OS one needs to target to
> obtain the OS specific wheels.
>
> (b) 'pip download  --no-binary :all:' -- the intention here
> is to grab the source distribution without any OS specific code included.
>
> (c) use https://warehouse.pypa.io/api-reference/json to look for
> distributed wheels for the target OS and python version and download them
> directly. (This may make for a nice flag to add to pip somewhere...the
> ability to specify what wheel one wants when it isn’t for the machine pip
> is running on)
>
> The issue I see with (a) is the shear amount work it would take to setup
> and maintain such a system.
>
> The issue I see with (b) is that it is not 100% reliable as some packages
> are tricky to install and may not work well with 'pip download’.
>
> I have not played around with (c) yet, so I do not know how well it will
> work, but it seems like a viable solution.
>
> I was just wondering if anyone had any comments on this as I think though
> the ways to solve this problem.
>
> Thank you.
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Distributing packages to offline machines

[Eric Gorr]

I had a question about distributing python packages to offline machines
when the offline machine is running a different OS then a machine with an
internet connection. The packages I am concerned with are third party upon
which mine depend.

Based on what I have learned so far, there are three solutions.

(a) Use a CI to run a fleet of machines for each OS one needs to target to
obtain the OS specific wheels.

(b) 'pip download  --no-binary :all:' -- the intention here
is to grab the source distribution without any OS specific code included.

(c) use https://warehouse.pypa.io/api-reference/json to look for
distributed wheels for the target OS and python version and download them
directly. (This may make for a nice flag to add to pip somewhere...the
ability to specify what wheel one wants when it isn’t for the machine pip
is running on)

The issue I see with (a) is the shear amount work it would take to setup
and maintain such a system.

The issue I see with (b) is that it is not 100% reliable as some packages
are tricky to install and may not work well with 'pip download’.

I have not played around with (c) yet, so I do not know how well it will
work, but it seems like a viable solution.

I was just wondering if anyone had any comments on this as I think though
the ways to solve this problem.

Thank you.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig