On Thursday, December 8, 2016, Wes Turner <wes.tur...@gmail.com> wrote:
> > > On Thursday, December 1, 2016, Freddy Rietdijk <freddyrietd...@fridh.nl > <javascript:_e(%7B%7D,'cvml','freddyrietd...@fridh.nl');>> wrote: > >> Hi, >> >> I would like to propose that, as a community, we jointly maintain a >> curated set of Python packages that are known to work together. These >> packages would receive security updates for some time and every couple of >> months a new major release of the curated set comes available. The idea of >> this is inspired by Haskell LTS, so maybe we should call this PyPI LTS? >> >> So why a PyPI LTS? >> >> PyPI makes available all versions of packages that were uploaded, and by >> default installers like pip will try to use the latest available versions >> of packages, unless told otherwise. With a requirements.txt file (or a >> future pipfile.lock) and setup.py we can pin as much as we like our >> requirements of respectively the environment and package requirements, >> thereby making a more reproducible environment possible and also fixing the >> API for developers. Pinning requirements is often a manual job, although >> one could use pip freeze or other tools. >> > > https://github.com/nvie/pip-tools : > > - requirements.in -> pip-compile -> requirements.txt (~pipfile.lock) > - I can't remember whether pip-compile includes the checksum in the > compiled requirements.txt > > https://pip.pypa.io/en/stable/reference/pip_install/#hash-checking-mode > > - Current recommendation: sha256 > - (These are obviously different for platform-specific wheels and bdists) > > https://github.com/rbanffy/pip-chill > > - Unlike pip freeze, pip-chill includes only top-level deps > > > >> >> >> >> A common problem is when two packages in a certain environment require >> different versions of a package. Having a curated set of packages, >> developers could be encouraged to test against the latest stable and >> nightly of the curated package set, thereby increasing compatibility >> between different packages, something I think we all want. >> >> Having a compatible set of packages is not only interesting for >> developers, but also for downstream distributions. All distributions try to >> find a set of packages that are working together and release them. This is >> a lot of work, and I think it would be in everyone's benefit if we try to >> solve this issue together. >> > > I think conda has already been mentioned. > > - environment.yml : http://conda.pydata.org/docs/using/envs.html#use- > environment-from-file > > https://conda-forge.github.io > > - "A community led collection of recipes, build infrastructure and > distributions for the conda package manager." > - "AppVeyor, CircleCI and TravisCI" > > >> >> A possible solution >> >> Downstream, that is developers and distributions, will need a set of >> packages that are known to work together. At minimum this would consist of, >> per package, the name of the package and its version, but for >> reproducibility I would propose adding the filename and hash as well. >> Because there isn't any reliable method to extract the requirements of a >> package, I propose also including `setup_requires`, install_requires`, and >> `tests_require` explicitly. That way, distributions can automatically build >> recipes for the packages (although non-Python dependencies would still have >> to be resolved by the distribution). >> >> The package set would be released as lts-YYYY-MM-REVISION, and developers >> can choose to track a specific revision, but would typically be asked to >> track only lts-YYYY-MM which would resolve to the latest REVISION. >> >> >> Because dependencies vary per Python language version, interpreter, and >> operating system, we would have to have these sets for each combination and >> therefore I propose having a source which evaluates to say a TOML/JSON file >> per version/interpreter/OS. >> How this source file should be written I don't know; while I think the >> Nix expression language is an excellent choice for this, it is not possible >> for everyone to use and therefore likely not an option. >> > > YAML: environment.yml, meta.yaml > > - http://conda.pydata.org/docs/building/meta-yaml.html# > requirements-section > - http://conda.pydata.org/docs/building/meta-yaml.html#test-section > > Could/would there be a package with an integration test suite in tests/? > > Practically, a developer would want a subset of the given known-good-set > (and then additional packages), so: > > - fork/copy requirements-YYYY-MM-REV--<OSNAMEVER>.txt > - #comment out unused deps > - add '-r addl-requirements.txt' > > >> >> >> Open questions >> >> There are still plenty of open questions. >> >> - Who decides when a package is updated that would break dependents? This >> is an issue all distributions face, so maybe we should involve them. >> > > IDK if e.g. https://requires.io can post to a mailing list? > > - "Stop wasting your time by manually keeping track of changelogs. > Requires.io keeps your python projects secure by monitoring their > dependencies." > - Source: https://github.com/requires > - https://wiki.jenkins-ci.org/display/JENKINS/ShiningPanda+Plugin > > > > >> - How would this be integrated with pip / virtualenv / pipfile.lock / >> requirements.txt / setup.py? See e.g. https://github.com/pypa/p >> ipfile/issues/10#issuecomment-262229620 >> > > - https://tox.readthedocs.io/en/latest/ > - tox.ini > - http://doc.devpi.net/latest/ > - http://doc.devpi.net/latest/quickstart-releaseprocess. > html#devpi-test-testing-an-uploaded-package > - Such a known-good-set could be hosted as a package index w/ devpi http://docs.openstack.org/developer/pbr/ - pbr does away with setup.py and install_requires in favor of just requirements.txt > > > >> >> References to Haskell LTS >> >> Here are several links to some interesting documents on how Haskell LTS >> works. >> - A blog post describing what Haskell LTS is: https://www.fpcomplete.com >> /blog/2014/12/backporting-bug-fixes >> - Rules regarding uploading and breaking packages: https://github.com/f >> pco/stackage/blob/master/MAINTAINERS.md#adding-a-package >> - The actual LTS files https://github.com/fpco/lts-haskell >> > Haskell ... pandoc ... https://sites.google.com/site/pydatalog/ > >> >> What do you think of this proposal? Would you be interested in this as >> developer, or packager? >> > > Given time to work on this, I'd probably spend it on developing a new or > existing (?) comprehensive integration test suite for an already-maintained > package set: > > https://docs.continuum.io/anaconda/pkg-docs > > https://www.enthought.com/products/canopy/package-index/ > > https://github.com/audreyr/cookiecutter-pypackage/blob/ > master/%7B%7Bcookiecutter.project_slug%7D%7D/tests/test_ > %7B%7Bcookiecutter.project_slug%7D%7D.py > > >> >> Freddy >> > >
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig