> For Java, we run an instance of Archiva: https://archiva.wikimedia.org/ > It's not a perfect approach but I think we can and should move in that direction with all our other ecosystems
Gitlab package registries may help us here! On Mon, May 8, 2023 at 8:59 AM Andrew Otto <o...@wikimedia.org> wrote: > > Tangent: is it worthwhile to establish a consensus for best practices > with package pinning and package management for Python projects in the > Wikimedia ecosystem? > Yes! That would be awesome. I have spent a lot of time floundering in this > area trying to make decisions; it'd be nice if we had a good guideline > established. > > > I'm working on an "Essential Tools for Managing Python Development > Environments > <https://docs.google.com/document/d/1pl6QCDWGebGjRrHixgQNES8qRf-8PCzwLSsFj7jI1HY/edit#heading=h.i5ou2ywgmb09>" > tutorial > Awesome! Did you consider conda envs? FWIW, we rely on conda envs > <https://gitlab.wikimedia.org/repos/data-engineering/workflow_utils#building-project-conda-distribution-environments> > in the Data Engineering world to work around the lack of ability to > securely run docker images in production. We didn't try pyenv, mainly > because conda gets us more than just python :) > > > Poetry is a modern lockfile-based packaging and dependency management > tool worth looking into > Poetry was nice, but I found that it wasn't as comprehensive (yet?) as ye > old setuptools based stuff. I can't quite recall what, but I think it was > around support for datafiles and scripts? But, this was 1.5 years ago so > maybe things are better now. > > Thank you! > > On Fri, May 5, 2023 at 11:32 AM Slavina Stefanova < > sstefan...@wikimedia.org> wrote: > >> Tangent: is it worthwhile to establish a consensus for best practices >>> with package pinning and package management for Python projects in the >>> Wikimedia ecosystem? When I last worked on a python project ( >>> https://wikitech.wikimedia.org/wiki/Add_Link) I found it confusing that >>> we have so many different tools and approaches for doing these things, and >>> seems like we'd benefit from having a standard, supported way. (Or maybe >>> that already exists and I haven't found it?) >> >> >> I'm working on an "Essential Tools for Managing Python Development >> Environments >> <https://docs.google.com/document/d/1pl6QCDWGebGjRrHixgQNES8qRf-8PCzwLSsFj7jI1HY/edit#heading=h.i5ou2ywgmb09>" >> tutorial that will be published to the wikis when ready. Maybe that could >> be expanded upon? In my experience though, it can be hard to get people to >> agree on following a standard, especially when there are so many different >> options and many folks already have their favorite tools and workflows. But >> it would be nice to have a set of recommendations to reduce the cognitive >> load. >> >> -- >> Slavina Stefanova (she/her) >> Software Engineer - Technical Engagement >> >> Wikimedia Foundation >> >> >> On Fri, May 5, 2023 at 5:18 PM Kosta Harlan <khar...@wikimedia.org> >> wrote: >> >>> Tangent: is it worthwhile to establish a consensus for best practices >>> with package pinning and package management for Python projects in the >>> Wikimedia ecosystem? When I last worked on a python project ( >>> https://wikitech.wikimedia.org/wiki/Add_Link) I found it confusing that >>> we have so many different tools and approaches for doing these things, and >>> seems like we'd benefit from having a standard, supported way. (Or maybe >>> that already exists and I haven't found it?) >>> >>> Kosta >>> >>> On 5. May 2023, at 13:51, Slavina Stefanova <sstefan...@wikimedia.org> >>> wrote: >>> >>> Poetry is a modern lockfile-based packaging and dependency management >>> tool worth looking into. It also supports exporting dependencies into a >>> requirements.txt file, should you need that (nice if you want to >>> containerize an app without bloating the image with Poetry, for instance). >>> >>> https://python-poetry.org/ <https://python-poetry.org/> >>> >>> -- >>> Slavina Stefanova (she/her) >>> Software Engineer - Technical Engagement >>> >>> Wikimedia Foundation >>> >>> >>> On Fri, May 5, 2023 at 1:38 PM Sebastian Berlin < >>> sebastian.ber...@wikimedia.se> wrote: >>> >>>> A word of warning: using `pip freeze` to populate requirements.txt can >>>> result in a hard to read (very long) file and other issues: >>>> https://medium.com/@tomagee/pip-freeze-requirements-txt-considered-harmful-f0bce66cf895 >>>> . >>>> >>>> *Sebastian Berlin* >>>> Utvecklare/*Developer* >>>> Wikimedia Sverige (WMSE) >>>> >>>> E-post/*E-Mail*: sebastian.ber...@wikimedia.se >>>> Telefon/*Phone*: (+46) 0707 - 92 03 84 >>>> >>>> >>>> On Fri, 5 May 2023 at 13:17, Amir Sarabadani <ladsgr...@gmail.com> >>>> wrote: >>>> >>>>> You can also create an empty virtual env, install all requirements and >>>>> then do >>>>> pip freeze > requirements.txt >>>>> >>>>> That should take care of pinning >>>>> >>>>> Am Fr., 5. Mai 2023 um 13:11 Uhr schrieb Lucas Werkmeister < >>>>> lucas.werkmeis...@wikimedia.de>: >>>>> >>>>>> For the general case of Python projects, I’d argue that a better >>>>>> solution is to adopt the lockfile pattern (package-lock.json, >>>>>> composer.lock, Cargo.lock, etc.) and pin *all* dependencies, and >>>>>> only update them when the new versions have been tested and are known to >>>>>> work. pip-tools <https://pypi.org/project/pip-tools/> can help with >>>>>> that, for example (requirements.in specifies “loose” dependencies; >>>>>> pip-compile creates a pinned requirements.txt; pip-sync installs it; >>>>>> pip-compile >>>>>> -U upgrades requirements.txt later; you check both requirements.in >>>>>> and requirements.txt into version control.) But I don’t know if that >>>>>> applies in your integration/config case. >>>>>> >>>>>> Am Do., 4. Mai 2023 um 18:08 Uhr schrieb Antoine Musso < >>>>>> has...@free.fr>: >>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> This is for python projects. >>>>>>> >>>>>>> Today, May 4th, urllib3 <https://pypi.org/project/urllib3/#history> >>>>>>> has released a new major version 2.0.2 which breaks the extremely >>>>>>> popular >>>>>>> requests <https://pypi.org/project/requests/> library. >>>>>>> >>>>>>> The fix is to pin urllib3<2 to prevent the new major version from >>>>>>> being installed (example >>>>>>> <https://gerrit.wikimedia.org/r/c/integration/config/+/915736/1/tox.ini> >>>>>>> ). >>>>>>> >>>>>>> https://phabricator.wikimedia.org/T335977 >>>>>>> >>>>>>> Upstream issue: https://github.com/psf/requests/issues/6432 >>>>>>> >>>>>>> >>>>>>> Antoine "hashar" Musso >>>>>>> Wikimedia Release Engineering >>>>>>> _______________________________________________ >>>>>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org >>>>>>> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org >>>>>>> >>>>>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/ >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Lucas Werkmeister (he/er) >>>>>> Software Engineer >>>>>> >>>>>> Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin >>>>>> Phone: +49 (0)30-577 11 62-0 >>>>>> https://wikimedia.de >>>>>> >>>>>> Imagine a world in which every single human being can freely share in >>>>>> the sum of all knowledge. Help us to achieve our vision! >>>>>> https://spenden.wikimedia.de >>>>>> >>>>>> Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. >>>>>> V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg >>>>>> unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt >>>>>> für Körperschaften I Berlin, Steuernummer 27/029/42207. >>>>>> _______________________________________________ >>>>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org >>>>>> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org >>>>>> >>>>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/ >>>>> >>>>> >>>>> >>>>> -- >>>>> Amir (he/him) >>>>> >>>>> _______________________________________________ >>>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org >>>>> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org >>>>> >>>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/ >>>> >>>> _______________________________________________ >>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org >>>> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org >>>> >>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/ >>> >>> _______________________________________________ >>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org >>> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org >>> >>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/ >>> >>> >>> _______________________________________________ >>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org >>> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org >>> >>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/ >> >> _______________________________________________ >> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org >> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org >> >> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/ > >
_______________________________________________ Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
