I read and mostly agree with Chris and Paul as we operate in similar spaces and probably have similar experiences with trying to unify packaging related tooling (its hard, we are all currently trying to undo this).
Without getting too focused on the details, despite the technical and implementation challenges, I _also_ am super interested in the user experience (this is why I am working on pipenv in the first place). Pipenv does have a number of issues currently, many of them are technical and a lot of them are being worked through (mostly by breaking things apart into smaller libraries that are not actually part of pipenv). Since you mentioned it specifically and there were a few minor misconceptions I just figured I'll speak to those for now: > Not ambitious enough. This is a fuzzy sort of thing, but perception matters, > and it's right there in the name: it's a tool to use pip, to manage an > environment. If we're reconceiving this as the grand unified entryway to all > of Python, then the name starts to feel pretty weird. Well not sure why the name is weird, pip (current packaging tooling) + env (environments) = pipenv. It doesn't build libraries for you and we've been pretty adamant about that point because we feel it's really not a good design to intermingle the construction of apps with the construction of libraries. I'm all for making users lives better; most users don't have a starting experience of wanting to package something up and ship it to pypi but nearly all users make environments and install packages all the time. It seems like it might be a mistake to start people off by confusing them about which things are applications and which things are libraries, how they work, etc > Uses Pipfile as a project marker instead of pyproject.toml. See above. pyproject.toml wasn't standardized yet when pipenv was released (and still isn't, beyond that it is a file that could exist and store information). Pipfile was intended to replace requirements.txt per some previous thread on the topic, and pipenv was an experimental implementation of the separation between the two different ways that people currently use requirements.txt in the wild -- one as a kind of abstract, unpinned dependency list (Pipfile), and the other as a transitive closure (Pipfile.lock). Since neither is standardized _for applications_, I'm not totally sure this is an actual sticking point. In either case, this seems super minor... > Not shipped with Python. (Obviously not pipenv's fault, but nonetheless.) (Not sure it should be) > Environments should be stored in project directory, not off in $HOME > somewhere. (Not sure what this is about, but some of the folks present were > quite insistent.) They used to be by default stored in $PROJECT/.venv but user feedback led us to use $WORKON_HOME by default. This is configurable by environment variable ($PIPENV_VENV_IN_PROJECT) or if you simply have a virtualenv in the .venv folder in your project directory. > Environments should be relocatable. And that will be possible whenever we can use venv across platforms and python versions. Currently that isn't possible, and we are forced to use virtualenv for compatibility. > Hardcoded to only support "default" and "dev" environments, which is > insufficient. Why? I mean, if you are planning to replace setuptools / flit / other build systems with pipenv and use pipfile as your new specification for declaring extras, I guess, but that's not how it's designed currently. Beyond that, I think we need some actual information on this one -- adding more complexity to any tool (including this kind of complexity) is going to ask more of the user in terms of frontloaded knowledge. This constraint limits the space a bit and for applications, I've very rarely seen actual limitations of this setup (but am interested, we have obviously had this feedback before but are not eager to add knobs in this specific area). > No mechanism for sharing prespecified commands like "run tests" or "reformat". There is, but the documentation on the topic is not very thorough: https://pipenv.readthedocs.io/en/latest/advanced/#custom-script-shortcuts See also: https://github.com/sarugaku/requirementslib/blob/master/Pipfile#L26 For an example for the specific cases you mentioned, the Pipfile entry in that project looks like this: [scripts] black = 'black src/requirementslib/ --exclude "/(\.git|\.hg|\.mypy_cache|\.tox|\.venv|_build|buck-out|build|dist)/"' tests = "pytest -v --ignore=src/requirementslib/_vendor/ tests" > Can't install Python. (There's... really no reason we *couldn't* distribute > pre-built Python interpreters on PyPI? between the python.org installers and > the manylinux image, we're already building redistributable run-anywhere > binaries for the most popular platforms on every Python release; we just > aren't zipping them up and putting them on PyPI.) Erm, well actually you can install python currently via pyenv on linux, and Tzu-ping is the maintainer of a pyenv-clone on windows which we've just never really got around to integrating fully. I've spoken to some of the folks over at Anaconda and I know they are interested in this as well especially given that it's pretty straightforward. It hasn't been a primary focus lately, but the tooling does exist (I don't think I've ever used it personally though) Anyway, this is all a good discussion to have and I really appreciate you kicking it off. I've been following the __pypackages__ conversation a bit since pycon and I honestly don't have much opinion about where we want to put stuff, but I'm not sure that the impact of the folder is going to be as great to the user as people might imagine -- the tooling is already being built, so maybe it's just a matter of agreeing on that as the place to put stuff, which schema to follow, and honestly working with some new users. I do this quite a bit but I haven't done any formal information gathering. Anecdotally I'll always tell you I'm right, but if we had some user data on specific pain points / usability issues I'd definitely be prepared to change my mind. Dan Ryan gh: @techalchemy // e: d...@danryan.co > -----Original Message----- > From: Nathaniel Smith [mailto:n...@pobox.com] > Sent: Sunday, September 30, 2018 6:42 AM > To: distutils-sig > Subject: [Distutils] Notes from python core sprint on workflow tooling > > Now that the basic wheels/pip/PyPI infrastructure is mostly > functional, there's been a lot of interest in improving higher-level > project workflow. We have a lot of powerful tools for this – > virtualenv, pyenv, conda, tox, pipenv, poetry, ... – and more in > development, like PEP 582 [1], which adds a support for project-local > packages directories (`__pypackages__/`) directly to the interpreter. > > But to me it feels like right now, Python workflow tools are like the > blind men and the elephant [2]. Each group sees one part of the > problem, and so we end up with one set of people building legs, > another a trunk, a third some ears... and there's no overall plan for > how they can fit together. > > For example, PEP 582 is trying to solve the problem that virtualenv is > really hard to use for beginners just starting out [3]. This is a > serious problem! But I don't want a solution that *only* works for > beginners starting out, so that once they get a little more > sophisticated they have to throw it out and learn something new from > scratch. > > So I think now might be a time for a bit of top-down design. **I want > a picture of the elephant.** If we had that, maybe we could see how > all these different ideas could be put together into a coherent whole. > So at the Python core sprint a few weeks ago, I dragged some > interested parties [4] into a room with a whiteboard [5], and we made > a start at it. And now I'm writing it up to share with you all. > > This is very much a draft, intended as a seed for discussion, not a > conclusion. > > [1] https://www.python.org/dev/peps/pep-0582/ > [2] https://en.wikipedia.org/wiki/Blind_men_and_an_elephant > [3] https://www.python.org/dev/peps/pep-0582/#motivation > [4] I won't try to list names, because I know I'll forget someone, and > I don't know if everyone would agree with everything I wrote there. > But thank you all! > [5] https://photos.app.goo.gl/4HfY8P3ESPNi9oLMA, including special > guest appearance by Kushal's elbow > > > # The idealized lifecycle of a Python project > > ## 1. Beginner > > Everyone starts out as a rank beginner. This may be the first time > they have programmed at all. At this stage, users want to: > > - install *one* thing to get started (e.g. python itself) > - write and run simple scripts (standalone .py files) > - run a REPL > - install and use PyPI packages like requests or numpy > - install and use tools like jupyter > - their IDE should also be able to find these packages/tools > > Over time, they'll probably end up with multiple scripts, and maybe > want to organize them into subdirectories. The above should all work > from subdirectories. > > ## 2. Sharing with others > > Now we have a neat little script. Or maybe we've made a pretty jupyter > notebook that computes some crucial business analytics. We want to > share it with our friends or coworkers. We still need the features > above; and now we also care about: > > - version control > - some way for our friend to reconstruct, on their computer: > - the same PyPI packages that we were using > - the same tools that we were using > - the ways we invoked those tools > > This last point is important: as projects grow in complexity, and are > used by a wider audience, they often end up with fairly complex tool > specifications that have to be shared among a team. For example: > > - to run tests: in an environment that has pytest, pytest-cov, and > pytest-trio installed, and with our project working directory on > PYTHONPATH, run `pytest -Werror --cov ...` > - to format code: in an environment using python 3.6 or later, that > has black installed, run `black -l 79 *.py my-util-directory/*.py` > > This kind of tool specification also puts us in a good position to set > up CI when we reach that point. > > At this point our project can grow in a few different directions. > > > ## 3a. Deployable webapp > > This adds the requirement to "deploy". I think this is mostly covered > by the set-up-an-environment-to-run-a-command functionality already > described? I'm not super familiar with this, but it's pipenv's core > target, and pipenv doesn't have much more than that, so I assume > that's about right... > > ## 3b. Reusable library > > For this we also need to: > > - Build sdists and wheels > - Which means: pyproject.toml, and some way to invoke it > - Install our library into our environments > - Including dependency locking (best practice is to not pin > dependencies in wheel metadata, but to pin all dependencies in CI; so > there needs to be some way to track those separately, but integrated > enough that it's not a huge ceremony to add or change a dependency) > > ## 3c. Reusable standalone app > > I think this is pretty much like the "Reusable library", except that > it'd be nice to have better tools to build/distribute standalone > applications. But if we had them, we could invoke them the same way as > we invoke other build systems? > > > # How do existing tools/proposals fit into this picture? > > pyenv, virtualenv, and conda all solve parts of the "create an > environment" problem, but consider the other aspects out-of-scope. > > tox solves the problem of keeping a shared record of how to run a > bunch of different tools in the appropriate environments, but doesn't > handle pinning or procuring appropriate python versions, and requires > a separate bootstrapping step to install tox. > > `__pypackages__` (if implemented) makes it very easy for beginners to > use PyPI packages in their own scripts and from the REPL; in > particular, it would be part of python, so it meets the "install *one* > thing" criterion. But, it doesn't provide any way to run tools. > (There's no way to put `__pypackages__/bin` on PATH.) It doesn't allow > scripts to be organized into subdirectories. (For security reasons, we > can't have the python interpreter going off walking the filesystem > looking for `__pypackages__/`, so the PEP specifies that > `__pypackages__/` has to be in the same directory as the script that > uses it.) There's no way to share your `__pypackages__` environment > with a friend. So... it seems like a something that people would > outgrow very quickly. > > pipenv and poetry are interesting. Their basic strategy is to say, > there is a top-level command that acts as your entry point to > performing workflow actions on on a python project (`pipenv` or > `poetry`, respectively). And this strategy at least in principle can > solve the problems that `__pypackages__/` runs into. In particular, it > doesn't rely on `$PATH`, so it can run tools; and because it's a > dedicated project management tool, it can go looking for the project > marker file. > > > # A fantastic elephant > > So if our idealized user had an idealized tool, what would that look like? > > They'll be interacting with Python through a dedicated tool, similar > to pipenv or poetry. In my little fantasy here I'll call it `pyp`, > because (a) I want to be neutral, (b) 6 characters is too long. > > To get this tool, either they install Python (via python.org download, > apt, homebrew, whatever), and the tool is automatically included. Or > else, they install the tool directly, and it has the ability to > install Python interpreters when needed. > > Once they have the tool, they start by making a new directory for > their project (this way they're ready to switch to version control > later). > > Then they somehow mark this directory as being a "python project > root". I guess the UI would be something like `pyp new <name>` and it > just does it for you, but we have to figure out what this creates on > disk. We need some sort of marker file. Files that currently serve > this kind of role include tox.ini, Pipfile, pyproject.toml, > __pypackages__, ... But only one of these is a standard thing we're > already committed to sticking with, so, pyproject.toml it is. Let's > make it the marker for any python project, not just redistributable > libraries. (And if we do grow up into a redistributable library, then > we're already prepared.) > > In the initial default configuration, there's a single default > environment. You can install things with `pyp install ...` or `pyp > uninstall ...`, and it tracks the requested packages in some > standardized way in pyproject.toml, and also pins specific versions > somewhere (could be pyproject.toml again I guess, or poetry's > pyproject.lock would work too). This way when we decide to share our > project later, our friends can recreate our environment on their > system. > > However, there's also the capability to configure multiple custom > execution environments, including python version and installed > packages. And the capability to configure new aliases like `pyp test` > or `pyp reformat`, which run some specified command in a specified > environment. > > Since the install/locking metadata is all standardized, you can even > switch between competing tools, and integrate with third-party tools > like pyup.io. > > For redistributable libraries, we also need some way to get the wheel > metadata and the workflow metadata to play nicely together. Maybe this > means that we need a standardized install-requires field in > pyproject.toml, so that build backends and workflow tools have a > shared source of truth? > > > # What's wrong with pipenv? > > Since pipenv is the tool that those of us in the room were most > familiar with, that comes closest to matching this vision, we > brainstormed a list of complaints about it. Some of these are more > reasonable than others. > > - Not ambitious enough. This is a fuzzy sort of thing, but perception > matters, and it's right there in the name: it's a tool to use pip, to > manage an environment. If we're reconceiving this as the grand unified > entryway to all of Python, then the name starts to feel pretty weird. > The whole thing where it's only intended to work for webapp-style > projects would have to change. > > - Uses Pipfile as a project marker instead of pyproject.toml. > > - Not shipped with Python. (Obviously not pipenv's fault, but nonetheless.) > > - Environments should be stored in project directory, not off in $HOME > somewhere. (Not sure what this is about, but some of the folks present > were quite insistent.) > > - Environments should be relocatable. > > - Hardcoded to only support "default" and "dev" environments, which is > insufficient. > > - No mechanism for sharing prespecified commands like "run tests" or > "reformat". > > - Can't install Python. (There's... really no reason we *couldn't* > distribute pre-built Python interpreters on PyPI? between the > python.org installers and the manylinux image, we're already building > redistributable run-anywhere binaries for the most popular platforms > on every Python release; we just aren't zipping them up and putting > them on PyPI.) > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > -- > Distutils-SIG mailing list -- distutils-sig@python.org > To unsubscribe send an email to distutils-sig-le...@python.org > https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ > Message archived at https://mail.python.org/mm3/archives/list/distutils- > s...@python.org/message/YFJITQB37MZOPOFJJF3OAQOY4TOAFXYM/ -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/O4NFRKAQGOYE222P7WFR37C5R4OXKOJ3/
