Indeed. Poetry is what we wanted to try already for quite a while and it seems it has pretty much what we need including extras support and .lock file - quite similar to npm approach + nice set of command line tools to manage deps.
J. On Mon, Jun 24, 2019 at 11:24 PM Ash Berlin-Taylor <a...@apache.org> wrote: > Hi Max, > > Did you take a look at Poetry (https://poetry.eustace.io/ < > https://poetry.eustace.io/>)? How does this PR compare? > > -ash > > > > On 24 Jun 2019, at 22:02, Maxime Beauchemin <maximebeauche...@gmail.com> > wrote: > > > > This Superset PR may be relevant and useful for Airflow too (still WiP): > > https://github.com/apache/incubator-superset/pull/7762 > > > > Basically it allows your project to define a "requirements/" folder with > a > > "requirements.json" as a single source of truth for all deps, including > the > > details of what goes in the "extra_requires", and compiles/pins all sets > of > > dependencies in a set of useful files > > > > Now: > > * this requirements.json is parsed and used directly as is in `setup.py` > > (same as before) > > * running "compile_requirements.py" effectively creates a set of > > "pip-compiled" full tree and pinned requirements for each set of > > requirements. > > * these pinned files can be used in CI / docker / builds to avoid bad > > surprises like it appears you did. The package (setup.py) is still > > permissive and allows people to pick their versions, but CI can be pinned > > and rock solid. > > > > The folder includes something like: > > requirements/requirements-hive.in > > requirements/requirements-hive.txt > > requirements/requirements-postgres.in > > requirements/requirements-postgres.txt > > # ... (2 files for each extra_requires defined in the JSON file) > > > > There may be some subtleties around py2/py3 reqs diverging, but you could > > probably bake that into that logic if you wanted to (for Superset it's a > > non-issue since we only support py36). > > > > Max > > > > On Mon, Jun 24, 2019 at 1:16 AM Kaxil Naik <kaxiln...@gmail.com> wrote: > > > >> It works :) - The CI passed for the commit on v1-10-test, so your PR > should > >> pass as well. > >> > >> On Mon, Jun 24, 2019 at 1:40 PM Jarek Potiuk <jarek.pot...@polidea.com> > >> wrote: > >> > >>> I pinned auto-api to below 1.1.0 and re-run the test - Let's hope it > >> works > >>> for now . > >>> > >>> On Mon, Jun 24, 2019 at 10:05 AM Jarek Potiuk < > jarek.pot...@polidea.com> > >>> wrote: > >>> > >>>> OK. After some checks it turns out it's not only Sphinx really but > also > >>>> sphinx-autoapi (v 1.1.0 released yesterday). > >>>> > >>>> J. > >>>> > >>>> > >>>> On Mon, Jun 24, 2019 at 9:29 AM Jarek Potiuk < > jarek.pot...@polidea.com > >>> > >>>> wrote: > >>>> > >>>>> Kamil prepared a fix https://github.com/apache/airflow/pull/5468 > >>> (thanks!) and > >>>>> when I tested it on a clean pip-install with latest sphinx it seems > to > >>> work > >>>>> fine (Running Travis CI build > >>>>> <https://travis-ci.org/apache/airflow/builds/549600100>now). Let's > >> see > >>>>> if it helps. > >>>>> > >>>>> BTW. The AIP-10 Docker image last step (pending Kubernetes fix) > should > >>>>> solve most of the dependency issues, however I think this particular > >>>>> problem could be missed (it's an interesting one). > >>>>> > >>>>> I will soon start another thread (after dockerfile merging) about > >>> finally > >>>>> looking into making dependencies more manageable and less > >> transient-deps > >>>>> problems. I already have some ideas how we can approach the problem > >> that > >>>>> Airflow is both: library (no-dependency pinning recommended) and > >>>>> end-product (strong pinning recommended). > >>>>> > >>>>> J. > >>>>> > >>>>> On Mon, Jun 24, 2019 at 8:48 AM Ash Berlin-Taylor <a...@apache.org> > >>> wrote: > >>>>> > >>>>>> Any cyclic imports we have aren't actually a problem and are handled > >>>>>> fine by python 3 (obviously, else it wouldn't work) so I think this > >> is > >>> a > >>>>>> big in sphinx with it being over zealous. > >>>>>> > >>>>>> This only affects tests and development, not our users in this case. > >>>>>> > >>>>>> If it's difficult to fix I would go with pinning - I want to get a > >>>>>> 1.10.4 beta 1 out today. > >>>>>> > >>>>>> -ash > >>>>>> > >>>>>> On 24 June 2019 00:37:57 BST, Jarek Potiuk < > jarek.pot...@polidea.com > >>> > >>>>>> wrote: > >>>>>>> > >>>>>>> Hello Everyone, > >>>>>>> > >>>>>>> We have yet another case where relesing a package (Sphinx 2.1.2 in > >>> this > >>>>>>> case) - started to cause master to fail. > >>>>>>> > >>>>>>> Should we pin Sphinx to an earlier version or maybe someone can > >> help > >>> and > >>>>>>> figure out and fix it quickly (seems that there are some cyclic > >>> imports > >>>>>>> that probably need solving anyway if Sphinx is right about them) ? > >>> @Kamil > >>>>>>> BreguĊa <kamil.breg...@polidea.com> @Fokko Driesprong > >>> <fo...@driesprong.frl> > >>>>>>> ? WDYT ? > >>>>>>> > >>>>>>> I created an issue for that > >>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-4841 > >>>>>>> > >>>>>>> J. > >>>>>>> > >>>>>>> > >>>>> > >>>>> -- > >>>>> > >>>>> Jarek Potiuk > >>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer > >>>>> > >>>>> M: +48 660 796 129 <+48660796129> > >>>>> [image: Polidea] <https://www.polidea.com/> > >>>>> > >>>>> > >>>> > >>>> -- > >>>> > >>>> Jarek Potiuk > >>>> Polidea <https://www.polidea.com/> | Principal Software Engineer > >>>> > >>>> M: +48 660 796 129 <+48660796129> > >>>> [image: Polidea] <https://www.polidea.com/> > >>>> > >>>> > >>> > >>> -- > >>> > >>> Jarek Potiuk > >>> Polidea <https://www.polidea.com/> | Principal Software Engineer > >>> > >>> M: +48 660 796 129 <+48660796129> > >>> [image: Polidea] <https://www.polidea.com/> > >>> > >> > >> > >> -- > >> *Kaxil Naik* > >> *Big Data Consultant | DevOps Data Engineer* > >> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark & > Neo4j > >> Developer > >> *LinkedIn*: https://www.linkedin.com/in/kaxil > >> > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
