Thanks Chris, these are excellent suggestions. Can you provide a reference to a PR you’re working or have worked on so we can jumpstart the process on our end?
Dan Ryan // pipenv maintainer gh: @techalchemy > On Sep 25, 2018, at 5:47 PM, Chris Jerdonek <chris.jerdo...@gmail.com> wrote: > >> On Tue, Sep 25, 2018 at 6:46 AM, Dan Ryan <d...@danryan.co> wrote: >> As far as the pipenv roadmap, we are rewriting a good chunk of it but it >> essentially the atomic things we do or would like to do with regards to >> shared functionality at a high level: >> >> - parse user input/requirements >> - retrieve package metadata >> - resolve dependencies (not in pip right now obviously) — may involve >> building wheels/sdists >> - download / build packages >> - install packages (into a virtualenv sometimes) >> - uninstall packages (from a virtualenv sometimes) >> - list packages etc (in a virtualenv sometimes) >> >> We know how these are accomplished in pip and we know how we are handling >> them both in pipenv and in our alternative implementations, which I believe >> still have some dependency on pip. For us the roadmap is basically to figure >> out >> >> - where do the abstractions live >> - how do we start >> - ??? > > Thanks, Dan. I think Donald answered this in an email much earlier in > this thread -- the one he sent on Sept. 19 beginning: > > """ > My general recommendation if you want a Python > implementation/interface for something pip does, is: > - Open an issue on the pip repository to document your intent and to > make sure that there is nobody there who is against having that > functionality split out. ... > ... > """ > > In addition, to ensure you're taking the approach of splitting out > parts of pip instead of creating yet another implementation separate > from pip (leading to duplicated work as I said), I would recommend > also doing the following: > > File PR's in pip to refactor out various functionality that makes > sense into stand-alone bits with their own tests (e.g. unit tests, > etc). This would live in pip but be disentangled from the rest of pip > from a code dependency perspective. It could even be in a separate > "future libraries" subdirectory if the pip committers are okay with > it. (Optional: since you're already vendoring pip, you could perhaps > use this factored out code at your own risk, like you're already doing > with some of pip's code base anyway.) > > The above is something I started doing at a small scale -- not to > break it out into a library, but for its own sake because it makes it > easier to maintain and work on pip as a code base, fixing bugs, etc. > > Some of the nice things about this approach are that-- > > 1. It can be done even before the steps that Donald outlined (like I'm > doing), e.g. so that you don't need to wait for a PEP to be finished. > And it could even help in the later creation of such PEPs. > 2. By doing so, as a side effect of the PR process, it automatically > causes communication with pip maintainers and works towards the > "splitting out parts of pip" goal -- all without duplicating any work > because both pip and pipenv developers are aware and collaborating on > these changes. > 3. It is incremental as you go, ensuring that it is compatible with > pip and that pip is using it, etc. > 4. It improves pip in the process (as well as pipenv, if you're still > vendoring in some form). In other words, the work is useful from the > beginning, whether or not it's ever broken out into a separately > maintained library. > 5. It is forward progress towards the goal of a separate library. > > --Chris > > > >> >> Dan Ryan // pipenv maintainer >> gh: @techalchemy >> >>>> On Sep 25, 2018, at 8:11 AM, Paul Moore <p.f.mo...@gmail.com> wrote: >>>> >>>>> On Tue, 25 Sep 2018 at 12:53, Chris Jerdonek <chris.jerdo...@gmail.com> >>>>> wrote: >>>>> >>>>> On Tue, Sep 25, 2018 at 3:47 AM, Tzu-ping Chung <uranu...@gmail.com> >>>>> wrote: >>>>> We are using pip internals for things pip wasn’t implemented for. >>>>> Specifically, >>>>> Pipenv uses pip’s package-fetching functions to implement its >>>>> platform-agnostic >>>>> resolver. pip does not have this, so there’s no functional overlap here. >>>>> Those >>>>> utilities are used to build something that doesn’t exist in pip, so >>>>> there’s no >>>>> duplicated efforts. >>>>> >>>>> My recent focus on making sense of packaging implementations and >>>>> splitting out >>>>> parts of pip is exactly to prevent potential duplicated efforts. If we >>>>> can’t use pip >>>>> internals, let’s make things we want to use *not* internal! >>>> >>>> I don't see this practice of "splitting out parts of pip" actually >>>> being followed so far though, and I want to tell you how it's leading >>>> to duplicated efforts right now -- even for the PR's I happen to be >>>> working on today. >>> >>> [...] >>> >>>> So by creating implementations of functions separate from pip instead >>>> of splitting them out of pip, it's leading to duplicated work for >>>> people working on pip's code base. A process of splitting something >>>> out of pip would I think involve a process more like what I'm >>>> following -- namely to refactor functionality in pip *first* so it can >>>> be used separately, and then pull that out so you only have one >>>> implementation at any point in time. Otherwise, you wind up creating >>>> two versions of similar functionality that not only need to be created >>>> twice, but also later reconciled if the plan is merge them back >>>> together. >>> >>> This is a very good point, and thanks for the concrete example. For >>> the record, I'm also a little annoyed that pipenv have been doing all >>> this work in isolation. There's been very little communication between >>> the pip and pipenv projects, and I think that's contributed to the >>> problem. I, for one, wasn't even aware for a long time that they were >>> embedding their own copy of pip. When we made pip's internals in >>> private in pip 10, we got essentially no feedback from anyone saying >>> that it would be a problem, and finding out that it had a sufficiently >>> major impact on pipenv that they had to embed the old version of pip >>> *after* the release was (to put it politely) suboptimal :-( >>> >>> Having said all this, this thread is the start of an attempt to work >>> better together and I think we should be glad that it's happening and >>> try to make it work. But I don't think we've got off to a particularly >>> good start and it will need work. In particular, I'd like to see an >>> attempt from *both* projects to clarify our goals - I've been speaking >>> solely for myself so far, it would be good to get the other pip >>> developers' views and set out a sort of roadmap, and for pipenv to do >>> the same. >>> >>> Paul >>> -- >>> Distutils-SIG mailing list -- distutils-sig@python.org >>> To unsubscribe send an email to distutils-sig-le...@python.org >>> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ >>> Message archived at >>> https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/OGXUNDRBLTNPOWLLFBVF45SW4SRGYSPP/ -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/YPZD2426DXFAETF6AXVMWQGK7MZZMEZH/
