Yusuke, I know it's been some time since you posted this; it looks like
it was a complicated question and volunteers said "oh I need to think
about that and get to it later" and then no one did.
We're about to close down this mailing list to reduce the number of
places where discussion fragments; if you're still dealing with
difficulties using/building Python packages at Zillow, I encourage you
to forward your note to https://discuss.python.org/c/packaging or
https://mail.python.org/mailman3/lists/distutils-sig.python.org/ to
discuss further.
Sorry again!
-Sumana
--
Sumana Harihareswara
Changeset Consulting
https://changeset.nyc
On 2/16/19 5:45 PM, Yusuke Tsutsumi wrote:
Hi pypa-dev,
This is a rather involved post, so I appreciate the time. I've been
shepparding a lot of the way we do Python at a company called Zillow. We
use Python quite a bit, and have a huge need to re-use tools and keep them
stable among the 200+ Python repositories we have.
Internally, we heavily use a project I wrote called uranium
(https://uranium.readthedocs.io/) that is effectively a python-based
version of Make that uses Pip and Virtualenv under the hood. I have two
main open questions:
1. Does pypa have a plan for additional, arbitrary build steps for
applications?
The reason uranium is so freeform is to handle aspects of building an
application that are outside the scope of downloading and installing python
packages:
* simplifying the configuration and startup of local dev builds of web
servers
* packaging the final application as a tarball, in a propietary format
* bootstrapping test configuration (e.g. linting rules)
* pulling in the above using a common dependency
* picking and choosing what to install from source vs from a wheel (we
sometimes need to link python packages to compiled c shared objects, but we
use wheels when we can)
I know that pyproject.toml was designed to allow custom configuration
values for various systems (e.g. black rules). Is there a story for how
this type of stuff would be done?
Now that I'm typing this... I think I would probably write a python package
that encapsulates a lot of the configuration we have today, and then expose
ways to manipulate that through the pyproject.toml. What I would end up
with is a mega-plugin that facilitates everything that was described above.
2. Support for "platform versioning"
This next one may be a little weird.. so bear with me :)
Internally, we have found a lot of value in maintaining a blessed version
set for packages that do not have a local override. For example, this has
allowed us to pin back backwards-incompatible dependencies that were not
expressed properly from the package maintainer via semver.
In other words, we have a centralized system that keeps track of blessed
versions, kind of like a giant requirements.txt:
requests == 2.21.0
aiohttp == 3.5.4
Locally, if you do not specify a specific version, then the blessed version
will be inserted in it's place. So locally if you had something like:
requests == 2.7.0
And you wanted aiohttp, requests, the final result would be:
requests == 2.7.0
aiohttp = 3.5.4
regardless of what the latest version in the package repository actually is.
The pros and cons of a system like this are probably worth a debate on it's
own :) But aside from that: is there any plans to support something like
this?
Currently Uranium extends the pip package resolution process to ensure this
works for dependencies as well. constraints.txt is almost perfect but IIRC
it can conflict with a requirements.txt or a setup.py specification, rather
than omit itself.
Summary:
my hope is to migrate over to more common open source tooling, and I think
the work done around pyproject.toml and other projects like poetry are a
huge step. I'm just trying to get a read on whether there are thoughts
around this I should focus effort on, or figure out a solution outside of
that.
Thanks!
-Yusuke
--
You received this message because you are subscribed to the Google Groups
"pypa-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to pypa-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/pypa-dev/b2fc0149-2ce8-8b23-40bd-818b8cca36e0%40changeset.nyc.
