[Distutils] Re: Packaging Advice for EFF's Certbot

[Daniel Shaulov]

On Tue, 24 Jul 2018, 02:47 Brad Warren,  wrote:

> It was suggested to me that I post to this list to see if anyone was
> interested in offering advice for how we should approach this. Of course,
> Certbot is written entirely in Python.
>

A few days ago, someone posted here about XAR, might interest you:

https://github.com/facebookincubator/xar/blob/master/README.md
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/LWMT5YZ3URSG5TVFQ2CVXDODLJGP3P4B/

[Distutils] Re: Packaging Advice for EFF's Certbot

[Alex Walters]

> -Original Message-
> From: Nathaniel Smith 
> Sent: Monday, July 23, 2018 10:31 PM
> To: Brad Warren 
> Cc: distutils-sig 
> Subject: [Distutils] Re: Packaging Advice for EFF's Certbot
> 
> 
> Reading the problem description at the top of your document, my first
> thought was that this seemed like exactly what conda is designed for:
> a "real" package manager designed to be portable across platforms and
> work in isolation from the system package manager.
> 
> You should also look at Nix and Guix, which are the other systems I
> see people mention in this space.
> 
> I'm not an expert in conda at all -- if you want to go down this path
> you should probably have a chat with Anaconda and also conda-forge
> (which is a very impressive community-run packaging and build effort).
> I have some idea about some of the questions you raised though :-):
> 

As a user of certbot, docker, conda, nix, and guix are non-starters.  I'm not 
depending on those tools for my production server (and while docker may be a 
dependency for some people, that is hardly universal).  Adding heavyweight 
technical dependencies are problematic if your goal is to get everyone using 
your software.  You're better off with cx_freeze or pyinstaller binaries 
downloaded from a website or a PPA-like-system to add to system package 
managers, which are not perfect solutions either.

> > How will separately distributed plugins work?
> 
> Conda has a system they call "channels" to let third-parties
> distribute extra conda packages, and existing systems for
> using/hosting/maintaining them. (Sort of similar to Ubuntu PPAs, if
> you know those.)
> 
> > How should the user invoke Certbot (and maybe conda) if we don’t want
> to put another Python in the user’s PATH to avoid breaking other Python
> code on their system?
> 
> A little shell script to set the PATH and then exec the right binary
> should work. Or just setting up the #! line in your main script
> properly.
> 
> > What should we do for systems not using 32-bit or 64-bit x86?
> 
> I know the conda folks have some stuff for ARM, though I don't know the
> details.
> 
> > If we didn’t want to trust any binaries built by someone else or proprietary
> code, how much work would that be?
> 
> This is where you want to talk to conda-forge – one of the original
> motivations was to make a community alternative to Anaconda Inc's
> official packages (which were not originally open-source, and do still
> contain proprietary code). Nowadays everyone's on better terms, but
> having once rebuilt the whole distro from the ground up means they can
> probably share some experience with you here.
> 
> -n
> 
> --
> Nathaniel J. Smith -- https://vorpus.org
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
> Message archived at https://mail.python.org/mm3/archives/list/distutils-
> s...@python.org/message/SFKA346UB3UQHZWNKONC63CT5VSKUTHB/
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/F2MIDHEX2BX4TTQLWPIADPSHMKYVRQ6T/

[Distutils] Re: Packaging Advice for EFF's Certbot

[Michael Sarahan]

Thanks Nathaniel for the mention of conda.

> How will separately distributed plugins work?

Conda is based on a unix-like filesystem layout.  There's a notion of a
"prefix", which on linux systems is something like /usr or /usr/local.
This can exist anywhere with conda, but there's a lib, bin, and include
subfolder, for example.  Windows is special, but much the same.  Having
that layout makes plugins easy, because the paths are both standardized and
relocatable (assume relative paths most of the time).

> How should the user invoke Certbot (and maybe conda) if we don’t want to
put another Python in the user’s PATH to avoid breaking other Python code
on their system?

Conda has this "activate" functionality.  It can stay off of PATH until you
need it, modify it temporarily, and also set environment variables as part
of activation if you need that.  There are ways to make shortcuts that run
executables with a conda environment activated, too.

> What should we do for systems not using 32-bit or 64-bit x86?

Cross compiling is quite possible, though still pretty young.  We have
pseudo cross compilers based on crosstool-ng right now, and extending the
concept to true cross compilers would be reasonably easy (but still would
require elbow grease on your end.)

> If we didn’t want to trust any binaries built by someone else or
proprietary code, how much work would that be?

That depends very much on how much of the stack you feel like you need to
rebuild.  Just python?  That's maybe a day's work at most, though more
platforms means more time, and if you need to cross compile, that may mean
more troubleshooting.  The recipes are out there, you'd just need to either
run them to build your own packages, or tweak them to your heart's
content.  Conda-forge is considered the authoritative source, and Anaconda
forks their recipes and modifies them to implement different functionality,
more optimization, or newer conda packaging features.  The eventual goal is
for Conda-forge and Anaconda to be in sync.  Obviously forking further from
Conda-forge or Anaconda implies more work for you.

I work for Anaconda, and am also a core member of Conda-forge.  If you have
any questions or just want to talk about how conda/conda-forge may or may
not meet your needs, please feel free to reach out.

HTH,
Michael

On Mon, Jul 23, 2018 at 9:33 PM Nathaniel Smith  wrote:

> On Mon, Jul 23, 2018 at 4:31 PM, Brad Warren  wrote:
> > Hi!
> >
> > I work at the Electronic Frontier Foundation on Certbot which is the most
> > popular end user application for obtaining and installing SSL/TLS
> > certificates from Let’s Encrypt. Over the past few years, distributing
> > Certbot has been one of our development team's biggest challenges and
> we’re
> > currently rethinking how we do so.
> >
> > It was suggested to me that I post to this list to see if anyone was
> > interested in offering advice for how we should approach this. Of course,
> > Certbot is written entirely in Python.
> >
> > If you’re interested, I wrote up a bit of background and what we’re
> > currently thinking at
> >
> https://docs.google.com/document/d/1y2tc65yWnGuYsun9wsXu7ZRLCVT9eih9eu0pj7Ado60/edit?usp=sharing
> .
> > Feel free to reach out to me on or off list or on IRC at bmw on Freenode.
>
> Reading the problem description at the top of your document, my first
> thought was that this seemed like exactly what conda is designed for:
> a "real" package manager designed to be portable across platforms and
> work in isolation from the system package manager.
>
> You should also look at Nix and Guix, which are the other systems I
> see people mention in this space.
>
> I'm not an expert in conda at all -- if you want to go down this path
> you should probably have a chat with Anaconda and also conda-forge
> (which is a very impressive community-run packaging and build effort).
> I have some idea about some of the questions you raised though :-):
>
> > How will separately distributed plugins work?
>
> Conda has a system they call "channels" to let third-parties
> distribute extra conda packages, and existing systems for
> using/hosting/maintaining them. (Sort of similar to Ubuntu PPAs, if
> you know those.)
>
> > How should the user invoke Certbot (and maybe conda) if we don’t want to
> put another Python in the user’s PATH to avoid breaking other Python code
> on their system?
>
> A little shell script to set the PATH and then exec the right binary
> should work. Or just setting up the #! line in your main script
> properly.
>
> > What should we do for systems not using 32-bit or 64-bit x86?
>
> I know the conda folks have some stuff for ARM, though I don't know the
> details.
>
> > If we didn’t want to trust any binaries built by someone else or
> proprietary code, how much work would that be?
>
> This is where you want to talk to conda-forge – one of the original
> motivations was to make a community alternative to Anaconda Inc's
> official packages (which were not originally 

[Distutils] Re: Packaging Advice for EFF's Certbot

[Nathaniel Smith]

On Mon, Jul 23, 2018 at 4:31 PM, Brad Warren  wrote:
> Hi!
>
> I work at the Electronic Frontier Foundation on Certbot which is the most
> popular end user application for obtaining and installing SSL/TLS
> certificates from Let’s Encrypt. Over the past few years, distributing
> Certbot has been one of our development team's biggest challenges and we’re
> currently rethinking how we do so.
>
> It was suggested to me that I post to this list to see if anyone was
> interested in offering advice for how we should approach this. Of course,
> Certbot is written entirely in Python.
>
> If you’re interested, I wrote up a bit of background and what we’re
> currently thinking at
> https://docs.google.com/document/d/1y2tc65yWnGuYsun9wsXu7ZRLCVT9eih9eu0pj7Ado60/edit?usp=sharing.
> Feel free to reach out to me on or off list or on IRC at bmw on Freenode.

Reading the problem description at the top of your document, my first
thought was that this seemed like exactly what conda is designed for:
a "real" package manager designed to be portable across platforms and
work in isolation from the system package manager.

You should also look at Nix and Guix, which are the other systems I
see people mention in this space.

I'm not an expert in conda at all -- if you want to go down this path
you should probably have a chat with Anaconda and also conda-forge
(which is a very impressive community-run packaging and build effort).
I have some idea about some of the questions you raised though :-):

> How will separately distributed plugins work?

Conda has a system they call "channels" to let third-parties
distribute extra conda packages, and existing systems for
using/hosting/maintaining them. (Sort of similar to Ubuntu PPAs, if
you know those.)

> How should the user invoke Certbot (and maybe conda) if we don’t want to put 
> another Python in the user’s PATH to avoid breaking other Python code on 
> their system?

A little shell script to set the PATH and then exec the right binary
should work. Or just setting up the #! line in your main script
properly.

> What should we do for systems not using 32-bit or 64-bit x86?

I know the conda folks have some stuff for ARM, though I don't know the details.

> If we didn’t want to trust any binaries built by someone else or proprietary 
> code, how much work would that be?

This is where you want to talk to conda-forge – one of the original
motivations was to make a community alternative to Anaconda Inc's
official packages (which were not originally open-source, and do still
contain proprietary code). Nowadays everyone's on better terms, but
having once rebuilt the whole distro from the ground up means they can
probably share some experience with you here.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/SFKA346UB3UQHZWNKONC63CT5VSKUTHB/

[Distutils] Re: Packaging Advice for EFF's Certbot

[Eli Ribble via Distutils-SIG]

I've read through your document. The question is interesting. You have
exactly the same bootstrapping problem that pip has. If I were in your
shoes here's how I would architect a solution:

 1 - If you can shift your dependencies to be pure python, do it. If you
can't, stop distributing certbot-auto outside of a package manager so you
can avoid having to build a badly implemented package manager inside
certbot-auto.
 2 - Move to requiring Python 3.4+ so you can rely on the presence of venv
and pip.
 3 - Continue using virtualenvs to bootstrap to the latest version of
certbot
 4 - Use pip to install plugins for certbot.

You've already identified why things like Docker or Snaps won't work for
your use case. You could rearchitect to use system package managers and
download files at runtime to update your logic without overwriting what the
system package manager installs. This would be a lot of work for little
gain. Therefore you've got to use your own update mechanism. Since you're
written in python you might as well use pip and virtualenv to solve some of
your problems. Especially since that's what you're using now.

End goal would be something like this:

sudo apt-get install certbot
 - now you have the non-python dependencies and an (old) implementation of
certbot that can update itself
sudo certbot 
 - certbot creates a virtualenv
 - certbot pip installs itself inside that virtualenv
 - certbot calls certbot inside virtualenv to do real work
 - certbot notices the system is using nginx, asks user if they want the
plugin for nginx, user says 'yes'
 - certbot uses pip to install certbot-nginx plugin
 - certbot reloads itself to enable plugin

I think I'd separate out certbot installed by the package manager which is
just a simple bootstrapper and the certbot installed in the virtualenv
which does real work. The bootstrapper updates rarely and does so via
package managers. The virtualenv gets checked for updates and updated every
time the bootstrapper runs, including in cron jobs.

Take all that with a huge helping of salt, I've got zero reputation around
here.

-Eli Ribble

On Mon, Jul 23, 2018 at 4:48 PM Brad Warren  wrote:

> Hi!
>
> I work at the Electronic Frontier Foundation on Certbot which is the most
> popular end user application for obtaining and installing SSL/TLS
> certificates from Let’s Encrypt. Over the past few years, distributing
> Certbot has been one of our development team's biggest challenges and we’re
> currently rethinking how we do so.
>
> It was suggested to me that I post to this list to see if anyone was
> interested in offering advice for how we should approach this. Of course,
> Certbot is written entirely in Python.
>
> If you’re interested, I wrote up a bit of background and what we’re
> currently thinking at
> https://docs.google.com/document/d/1y2tc65yWnGuYsun9wsXu7ZRLCVT9eih9eu0pj7Ado60/edit?usp=sharing.
> Feel free to reach out to me on or off list or on IRC at bmw on Freenode.
>
> Thanks so much for any help.
>
> Best,
> Brad Warren
> Senior Staff Technologist
> Electronic Frontier Foundation
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
> Message archived at
> https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/QMTQCDVFSBABR55EFCMMHNSORXYG5NPE/
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/DO4OTGLIDC5VSD4PCL6WWTQECU3IGUZU/

[Distutils] Re: Packaging Advice for EFF's Certbot

[Chris Jerdonek]

Hi,

Interesting write-up! A couple comments / questions:

1) Can you list in the document the non-Python dependencies and what
they're used for to give people an idea?

2) I don't know how Certbot is architected, but would it be possible
to put the "meat" of Certbot inside a Docker container (ideally
including most of the non-Python dependencies), and then have a much
lighter-weight Python application (with fewer dependencies) running
outside of Docker, and that calls the application inside the Docker
container when it needs to do stuff? It seems like that would give you
a lot more freedom and control with respect to the non-Python
dependencies.

The outer application could be used for the parts of the application
that can't be run inside a Docker container (e.g. what you referenced
here: "Many of Certbot’s features require it to have direct access to
the user’s web server configuration so we can modify its configuration
and restart the server.")

--Chris


On Mon, Jul 23, 2018 at 4:31 PM, Brad Warren  wrote:
> Hi!
>
> I work at the Electronic Frontier Foundation on Certbot which is the most
> popular end user application for obtaining and installing SSL/TLS
> certificates from Let’s Encrypt. Over the past few years, distributing
> Certbot has been one of our development team's biggest challenges and we’re
> currently rethinking how we do so.
>
> It was suggested to me that I post to this list to see if anyone was
> interested in offering advice for how we should approach this. Of course,
> Certbot is written entirely in Python.
>
> If you’re interested, I wrote up a bit of background and what we’re
> currently thinking at
> https://docs.google.com/document/d/1y2tc65yWnGuYsun9wsXu7ZRLCVT9eih9eu0pj7Ado60/edit?usp=sharing.
> Feel free to reach out to me on or off list or on IRC at bmw on Freenode.
>
> Thanks so much for any help.
>
> Best,
> Brad Warren
> Senior Staff Technologist
> Electronic Frontier Foundation
>
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
> Message archived at
> https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/QMTQCDVFSBABR55EFCMMHNSORXYG5NPE/
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/I6PNMFKNTCMSEX5JEXQAJ234MY7DCM5I/

[Distutils] Packaging Advice for EFF's Certbot

[Brad Warren]

Hi!

I work at the Electronic Frontier Foundation on Certbot which is the most 
popular end user application for obtaining and installing SSL/TLS certificates 
from Let’s Encrypt. Over the past few years, distributing Certbot has been one 
of our development team's biggest challenges and we’re currently rethinking how 
we do so.

It was suggested to me that I post to this list to see if anyone was interested 
in offering advice for how we should approach this. Of course, Certbot is 
written entirely in Python.

If you’re interested, I wrote up a bit of background and what we’re currently 
thinking at 
https://docs.google.com/document/d/1y2tc65yWnGuYsun9wsXu7ZRLCVT9eih9eu0pj7Ado60/edit?usp=sharing
 
.
 Feel free to reach out to me on or off list or on IRC at bmw on Freenode.

Thanks so much for any help.

Best,
Brad Warren
Senior Staff Technologist
Electronic Frontier Foundation--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/QMTQCDVFSBABR55EFCMMHNSORXYG5NPE/

[Distutils] Re: Make an ordered list of sdists to be installed?

[Tzu-ping Chung]

I just happened to be looking into this area, and may be able to offer some 
help.

Some background information (optional reading): 
https://github.com/pypa/pipenv/issues/2596#issuecomment-405656806

Sébastien (@sdispater) is the author of Poetry, including the resolver, called 
Mixology. As he mentioned in the issue, he recently rewrote the resolver, and 
had yet to decide when and how to extract if from other inner workings of 
Poetry.

The old version of Mixology (before the rewrite), however, is still available 
on PyPI, and is good enough for this particular use with some additional work.

I made a quick, very inefficient implementation, available as a GitHub Gist:
https://gist.github.com/uranusjr/a7a9f20c6e43810bd19f0c73e9617182

Hopefully this would help with your need 

TP


From: distutils-sig-requ...@python.org
Sent: 24 July 2018 00:00
To: distutils-sig@python.org

I don’t know the details, but I did read that Poetry has a sophisticated 
dependency resolver. 

https://github.com/sdispater/poetry

I don’t know if there is a way to access the resolver independently of the 
tool, but perhaps it would provide a handy reference. 
2018년 7월 23일 (월) 오전 5:49, Thomas Kluyver 님이 작성:
Hi all,

Do we know of any tool that can, given the name of one or more packages, follow 
dependency chains and produce a list of packages in the order they need to be 
installed, assuming every package needed will be built from source?

Running "pip download --no-binary :all: ipython" gets me a set of sdists to be 
installed, but I lose any information about the order. I assume some packages 
will fail to build if their dependencies are not installed first, so the order 
is significant.

Pip appears to keep track of the ordering internally: if I run "pip install 
--no-binary :all: ipython", all the dependencies are downloaded, and then the 
collected packages are installed starting from those with no dependencies and 
finishing with the package I requested. But I don't know of any way to get this 
information out of pip. Is there an option that I'm overlooking? Or some other 
tool that can do this?

The use case I'm thinking about is to automatically generate instructions for a 
build system which separates the downloading and installing steps, so for each 
step it expects one or more URLs to download, along with instructions for how 
to install that piece. The installation steps shouldn't download further data. 
I could work around the issue by telling it to download all the sdists in a 
single step and then install in one shot with --no-index and --find-links. But 
it's more elegant - and better for caching - if we can install each package as 
a single step.

Thanks,
Thomas
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/LGTH3IYBMVKBS4PYGFJ6A7N5GW5ZKFUY/
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/VX734J3S4EBKDSZR5BIRQPOH5LVHTHHE/

[Distutils] Re: Make an ordered list of sdists to be installed?

[Pradyun Gedam]

On Mon, Jul 23, 2018 at 3:19 PM Thomas Kluyver  wrote:
>
> Hi all,
>
> Do we know of any tool that can, given the name of one or more packages, 
> follow dependency chains and produce a list of packages in the order they 
> need to be installed, assuming every package needed will be built from source?
>
> Running "pip download --no-binary :all: ipython" gets me a set of sdists to 
> be installed, but I lose any information about the order. I assume some 
> packages will fail to build if their dependencies are not installed first, so 
> the order is significant.
>
> Pip appears to keep track of the ordering internally: if I run "pip install 
> --no-binary :all: ipython", all the dependencies are downloaded, and then the 
> collected packages are installed starting from those with no dependencies and 
> finishing with the package I requested. But I don't know of any way to get 
> this information out of pip. Is there an option that I'm overlooking? Or some 
> other tool that can do this?
>
> The use case I'm thinking about is to automatically generate instructions for 
> a build system which separates the downloading and installing steps, so for 
> each step it expects one or more URLs to download, along with instructions 
> for how to install that piece. The installation steps shouldn't download 
> further data. I could work around the issue by telling it to download all the 
> sdists in a single step and then install in one shot with --no-index and 
> --find-links. But it's more elegant - and better for caching - if we can 
> install each package as a single step.
>
> Thanks,
> Thomas
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
> Message archived at 
> https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/LGTH3IYBMVKBS4PYGFJ6A7N5GW5ZKFUY/

Hey!

This is actually related to a really old feature request on pip. [1]
Essentially, the request is to allow the user to get the order and
versions of packages that pip would choose to install. Some of the
preparatory refactoring that's been done for bringing in a newer
implementation of the dependency resolver to pip [2] means that this
should be much easier to implement today than it would have been in
the past.

I should point out that these don't include build-dependencies for PEP 518.

Cheers,
Pradyun

[1]: https://github.com/pypa/pip/issues/53
[2]: https://github.com/pradyunsg/zazo
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/MJL5LSSHNFV76UJHS372EHZ72IC7CPFS/

[Distutils] Re: Make an ordered list of sdists to be installed?

[Donald Stufft]

The things you get from pip download should be order independent. It won’t 
include build dependencies though.

> On Jul 23, 2018, at 5:48 AM, Thomas Kluyver  wrote:
> 
> Hi all,
> 
> Do we know of any tool that can, given the name of one or more packages, 
> follow dependency chains and produce a list of packages in the order they 
> need to be installed, assuming every package needed will be built from source?
> 
> Running "pip download --no-binary :all: ipython" gets me a set of sdists to 
> be installed, but I lose any information about the order. I assume some 
> packages will fail to build if their dependencies are not installed first, so 
> the order is significant.
> 
> Pip appears to keep track of the ordering internally: if I run "pip install 
> --no-binary :all: ipython", all the dependencies are downloaded, and then the 
> collected packages are installed starting from those with no dependencies and 
> finishing with the package I requested. But I don't know of any way to get 
> this information out of pip. Is there an option that I'm overlooking? Or some 
> other tool that can do this?
> 
> The use case I'm thinking about is to automatically generate instructions for 
> a build system which separates the downloading and installing steps, so for 
> each step it expects one or more URLs to download, along with instructions 
> for how to install that piece. The installation steps shouldn't download 
> further data. I could work around the issue by telling it to download all the 
> sdists in a single step and then install in one shot with --no-index and 
> --find-links. But it's more elegant - and better for caching - if we can 
> install each package as a single step.
> 
> Thanks,
> Thomas
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
> Message archived at 
> https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/LGTH3IYBMVKBS4PYGFJ6A7N5GW5ZKFUY/
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/FSPNQDWKZW4FBIWYJECW67PMCJFCE4NH/

[Distutils] Re: Make an ordered list of sdists to be installed?

[Nicholas Chammas]

I don’t know the details, but I did read that Poetry has a sophisticated
dependency resolver.

https://github.com/sdispater/poetry

I don’t know if there is a way to access the resolver independently of the
tool, but perhaps it would provide a handy reference.
2018년 7월 23일 (월) 오전 5:49, Thomas Kluyver 님이 작성:

> Hi all,
>
> Do we know of any tool that can, given the name of one or more packages,
> follow dependency chains and produce a list of packages in the order they
> need to be installed, assuming every package needed will be built from
> source?
>
> Running "pip download --no-binary :all: ipython" gets me a set of sdists
> to be installed, but I lose any information about the order. I assume some
> packages will fail to build if their dependencies are not installed first,
> so the order is significant.
>
> Pip appears to keep track of the ordering internally: if I run "pip
> install --no-binary :all: ipython", all the dependencies are downloaded,
> and then the collected packages are installed starting from those with no
> dependencies and finishing with the package I requested. But I don't know
> of any way to get this information out of pip. Is there an option that I'm
> overlooking? Or some other tool that can do this?
>
> The use case I'm thinking about is to automatically generate instructions
> for a build system which separates the downloading and installing steps, so
> for each step it expects one or more URLs to download, along with
> instructions for how to install that piece. The installation steps
> shouldn't download further data. I could work around the issue by telling
> it to download all the sdists in a single step and then install in one shot
> with --no-index and --find-links. But it's more elegant - and better for
> caching - if we can install each package as a single step.
>
> Thanks,
> Thomas
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
> Message archived at
> https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/LGTH3IYBMVKBS4PYGFJ6A7N5GW5ZKFUY/
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/BSODWBIYPSOGHUWJRPZ4Y2SZSSU4ZDGG/

[Distutils] Make an ordered list of sdists to be installed?

[Thomas Kluyver]

Hi all,

Do we know of any tool that can, given the name of one or more packages, follow 
dependency chains and produce a list of packages in the order they need to be 
installed, assuming every package needed will be built from source?

Running "pip download --no-binary :all: ipython" gets me a set of sdists to be 
installed, but I lose any information about the order. I assume some packages 
will fail to build if their dependencies are not installed first, so the order 
is significant.

Pip appears to keep track of the ordering internally: if I run "pip install 
--no-binary :all: ipython", all the dependencies are downloaded, and then the 
collected packages are installed starting from those with no dependencies and 
finishing with the package I requested. But I don't know of any way to get this 
information out of pip. Is there an option that I'm overlooking? Or some other 
tool that can do this?

The use case I'm thinking about is to automatically generate instructions for a 
build system which separates the downloading and installing steps, so for each 
step it expects one or more URLs to download, along with instructions for how 
to install that piece. The installation steps shouldn't download further data. 
I could work around the issue by telling it to download all the sdists in a 
single step and then install in one shot with --no-index and --find-links. But 
it's more elegant - and better for caching - if we can install each package as 
a single step.

Thanks,
Thomas
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/LGTH3IYBMVKBS4PYGFJ6A7N5GW5ZKFUY/