Oh nice, I had never looked into poetry before. I might just use that on the Superset side too then!
Max On Mon, Jun 24, 2019 at 2:31 PM Jarek Potiuk <jarek.pot...@polidea.com> wrote: > Indeed. Poetry is what we wanted to try already for quite a while and it > seems it has pretty much what we need including extras support and .lock > file - quite similar to npm approach + nice set of command line tools to > manage deps. > > J. > > On Mon, Jun 24, 2019 at 11:24 PM Ash Berlin-Taylor <a...@apache.org> wrote: > > > Hi Max, > > > > Did you take a look at Poetry (https://poetry.eustace.io/ < > > https://poetry.eustace.io/>)? How does this PR compare? > > > > -ash > > > > > > > On 24 Jun 2019, at 22:02, Maxime Beauchemin < > maximebeauche...@gmail.com> > > wrote: > > > > > > This Superset PR may be relevant and useful for Airflow too (still > WiP): > > > https://github.com/apache/incubator-superset/pull/7762 > > > > > > Basically it allows your project to define a "requirements/" folder > with > > a > > > "requirements.json" as a single source of truth for all deps, including > > the > > > details of what goes in the "extra_requires", and compiles/pins all > sets > > of > > > dependencies in a set of useful files > > > > > > Now: > > > * this requirements.json is parsed and used directly as is in > `setup.py` > > > (same as before) > > > * running "compile_requirements.py" effectively creates a set of > > > "pip-compiled" full tree and pinned requirements for each set of > > > requirements. > > > * these pinned files can be used in CI / docker / builds to avoid bad > > > surprises like it appears you did. The package (setup.py) is still > > > permissive and allows people to pick their versions, but CI can be > pinned > > > and rock solid. > > > > > > The folder includes something like: > > > requirements/requirements-hive.in > > > requirements/requirements-hive.txt > > > requirements/requirements-postgres.in > > > requirements/requirements-postgres.txt > > > # ... (2 files for each extra_requires defined in the JSON file) > > > > > > There may be some subtleties around py2/py3 reqs diverging, but you > could > > > probably bake that into that logic if you wanted to (for Superset it's > a > > > non-issue since we only support py36). > > > > > > Max > > > > > > On Mon, Jun 24, 2019 at 1:16 AM Kaxil Naik <kaxiln...@gmail.com> > wrote: > > > > > >> It works :) - The CI passed for the commit on v1-10-test, so your PR > > should > > >> pass as well. > > >> > > >> On Mon, Jun 24, 2019 at 1:40 PM Jarek Potiuk < > jarek.pot...@polidea.com> > > >> wrote: > > >> > > >>> I pinned auto-api to below 1.1.0 and re-run the test - Let's hope it > > >> works > > >>> for now . > > >>> > > >>> On Mon, Jun 24, 2019 at 10:05 AM Jarek Potiuk < > > jarek.pot...@polidea.com> > > >>> wrote: > > >>> > > >>>> OK. After some checks it turns out it's not only Sphinx really but > > also > > >>>> sphinx-autoapi (v 1.1.0 released yesterday). > > >>>> > > >>>> J. > > >>>> > > >>>> > > >>>> On Mon, Jun 24, 2019 at 9:29 AM Jarek Potiuk < > > jarek.pot...@polidea.com > > >>> > > >>>> wrote: > > >>>> > > >>>>> Kamil prepared a fix https://github.com/apache/airflow/pull/5468 > > >>> (thanks!) and > > >>>>> when I tested it on a clean pip-install with latest sphinx it seems > > to > > >>> work > > >>>>> fine (Running Travis CI build > > >>>>> <https://travis-ci.org/apache/airflow/builds/549600100>now). Let's > > >> see > > >>>>> if it helps. > > >>>>> > > >>>>> BTW. The AIP-10 Docker image last step (pending Kubernetes fix) > > should > > >>>>> solve most of the dependency issues, however I think this > particular > > >>>>> problem could be missed (it's an interesting one). > > >>>>> > > >>>>> I will soon start another thread (after dockerfile merging) about > > >>> finally > > >>>>> looking into making dependencies more manageable and less > > >> transient-deps > > >>>>> problems. I already have some ideas how we can approach the problem > > >> that > > >>>>> Airflow is both: library (no-dependency pinning recommended) and > > >>>>> end-product (strong pinning recommended). > > >>>>> > > >>>>> J. > > >>>>> > > >>>>> On Mon, Jun 24, 2019 at 8:48 AM Ash Berlin-Taylor <a...@apache.org> > > >>> wrote: > > >>>>> > > >>>>>> Any cyclic imports we have aren't actually a problem and are > handled > > >>>>>> fine by python 3 (obviously, else it wouldn't work) so I think > this > > >> is > > >>> a > > >>>>>> big in sphinx with it being over zealous. > > >>>>>> > > >>>>>> This only affects tests and development, not our users in this > case. > > >>>>>> > > >>>>>> If it's difficult to fix I would go with pinning - I want to get a > > >>>>>> 1.10.4 beta 1 out today. > > >>>>>> > > >>>>>> -ash > > >>>>>> > > >>>>>> On 24 June 2019 00:37:57 BST, Jarek Potiuk < > > jarek.pot...@polidea.com > > >>> > > >>>>>> wrote: > > >>>>>>> > > >>>>>>> Hello Everyone, > > >>>>>>> > > >>>>>>> We have yet another case where relesing a package (Sphinx 2.1.2 > in > > >>> this > > >>>>>>> case) - started to cause master to fail. > > >>>>>>> > > >>>>>>> Should we pin Sphinx to an earlier version or maybe someone can > > >> help > > >>> and > > >>>>>>> figure out and fix it quickly (seems that there are some cyclic > > >>> imports > > >>>>>>> that probably need solving anyway if Sphinx is right about them) > ? > > >>> @Kamil > > >>>>>>> BreguĊa <kamil.breg...@polidea.com> @Fokko Driesprong > > >>> <fo...@driesprong.frl> > > >>>>>>> ? WDYT ? > > >>>>>>> > > >>>>>>> I created an issue for that > > >>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-4841 > > >>>>>>> > > >>>>>>> J. > > >>>>>>> > > >>>>>>> > > >>>>> > > >>>>> -- > > >>>>> > > >>>>> Jarek Potiuk > > >>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer > > >>>>> > > >>>>> M: +48 660 796 129 <+48660796129> > > >>>>> [image: Polidea] <https://www.polidea.com/> > > >>>>> > > >>>>> > > >>>> > > >>>> -- > > >>>> > > >>>> Jarek Potiuk > > >>>> Polidea <https://www.polidea.com/> | Principal Software Engineer > > >>>> > > >>>> M: +48 660 796 129 <+48660796129> > > >>>> [image: Polidea] <https://www.polidea.com/> > > >>>> > > >>>> > > >>> > > >>> -- > > >>> > > >>> Jarek Potiuk > > >>> Polidea <https://www.polidea.com/> | Principal Software Engineer > > >>> > > >>> M: +48 660 796 129 <+48660796129> > > >>> [image: Polidea] <https://www.polidea.com/> > > >>> > > >> > > >> > > >> -- > > >> *Kaxil Naik* > > >> *Big Data Consultant | DevOps Data Engineer* > > >> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark & > > Neo4j > > >> Developer > > >> *LinkedIn*: https://www.linkedin.com/in/kaxil > > >> > > > > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> >
