This Superset PR may be relevant and useful for Airflow too (still WiP): https://github.com/apache/incubator-superset/pull/7762
Basically it allows your project to define a "requirements/" folder with a "requirements.json" as a single source of truth for all deps, including the details of what goes in the "extra_requires", and compiles/pins all sets of dependencies in a set of useful files Now: * this requirements.json is parsed and used directly as is in `setup.py` (same as before) * running "compile_requirements.py" effectively creates a set of "pip-compiled" full tree and pinned requirements for each set of requirements. * these pinned files can be used in CI / docker / builds to avoid bad surprises like it appears you did. The package (setup.py) is still permissive and allows people to pick their versions, but CI can be pinned and rock solid. The folder includes something like: requirements/requirements-hive.in requirements/requirements-hive.txt requirements/requirements-postgres.in requirements/requirements-postgres.txt # ... (2 files for each extra_requires defined in the JSON file) There may be some subtleties around py2/py3 reqs diverging, but you could probably bake that into that logic if you wanted to (for Superset it's a non-issue since we only support py36). Max On Mon, Jun 24, 2019 at 1:16 AM Kaxil Naik <kaxiln...@gmail.com> wrote: > It works :) - The CI passed for the commit on v1-10-test, so your PR should > pass as well. > > On Mon, Jun 24, 2019 at 1:40 PM Jarek Potiuk <jarek.pot...@polidea.com> > wrote: > > > I pinned auto-api to below 1.1.0 and re-run the test - Let's hope it > works > > for now . > > > > On Mon, Jun 24, 2019 at 10:05 AM Jarek Potiuk <jarek.pot...@polidea.com> > > wrote: > > > > > OK. After some checks it turns out it's not only Sphinx really but also > > > sphinx-autoapi (v 1.1.0 released yesterday). > > > > > > J. > > > > > > > > > On Mon, Jun 24, 2019 at 9:29 AM Jarek Potiuk <jarek.pot...@polidea.com > > > > > wrote: > > > > > >> Kamil prepared a fix https://github.com/apache/airflow/pull/5468 > > (thanks!) and > > >> when I tested it on a clean pip-install with latest sphinx it seems to > > work > > >> fine (Running Travis CI build > > >> <https://travis-ci.org/apache/airflow/builds/549600100>now). Let's > see > > >> if it helps. > > >> > > >> BTW. The AIP-10 Docker image last step (pending Kubernetes fix) should > > >> solve most of the dependency issues, however I think this particular > > >> problem could be missed (it's an interesting one). > > >> > > >> I will soon start another thread (after dockerfile merging) about > > finally > > >> looking into making dependencies more manageable and less > transient-deps > > >> problems. I already have some ideas how we can approach the problem > that > > >> Airflow is both: library (no-dependency pinning recommended) and > > >> end-product (strong pinning recommended). > > >> > > >> J. > > >> > > >> On Mon, Jun 24, 2019 at 8:48 AM Ash Berlin-Taylor <a...@apache.org> > > wrote: > > >> > > >>> Any cyclic imports we have aren't actually a problem and are handled > > >>> fine by python 3 (obviously, else it wouldn't work) so I think this > is > > a > > >>> big in sphinx with it being over zealous. > > >>> > > >>> This only affects tests and development, not our users in this case. > > >>> > > >>> If it's difficult to fix I would go with pinning - I want to get a > > >>> 1.10.4 beta 1 out today. > > >>> > > >>> -ash > > >>> > > >>> On 24 June 2019 00:37:57 BST, Jarek Potiuk <jarek.pot...@polidea.com > > > > >>> wrote: > > >>>> > > >>>> Hello Everyone, > > >>>> > > >>>> We have yet another case where relesing a package (Sphinx 2.1.2 in > > this > > >>>> case) - started to cause master to fail. > > >>>> > > >>>> Should we pin Sphinx to an earlier version or maybe someone can > help > > and > > >>>> figure out and fix it quickly (seems that there are some cyclic > > imports > > >>>> that probably need solving anyway if Sphinx is right about them) ? > > @Kamil > > >>>> BreguĊa <kamil.breg...@polidea.com> @Fokko Driesprong > > <fo...@driesprong.frl> > > >>>> ? WDYT ? > > >>>> > > >>>> I created an issue for that > > >>>> https://issues.apache.org/jira/browse/AIRFLOW-4841 > > >>>> > > >>>> J. > > >>>> > > >>>> > > >> > > >> -- > > >> > > >> Jarek Potiuk > > >> Polidea <https://www.polidea.com/> | Principal Software Engineer > > >> > > >> M: +48 660 796 129 <+48660796129> > > >> [image: Polidea] <https://www.polidea.com/> > > >> > > >> > > > > > > -- > > > > > > Jarek Potiuk > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > M: +48 660 796 129 <+48660796129> > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > > > -- > > > > Jarek Potiuk > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > M: +48 660 796 129 <+48660796129> > > [image: Polidea] <https://www.polidea.com/> > > > > > -- > *Kaxil Naik* > *Big Data Consultant | DevOps Data Engineer* > *Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j > Developer > *LinkedIn*: https://www.linkedin.com/in/kaxil >
