Thanks for pointing it out Jakob.
I am still very fresh in the ASF community and learning the ropes and
etiquette and code of conduct. Apologies for my ignorance.
I re-read the conduct and FAQ now again - with more understanding and will
pay more attention to wording in the future. As you mentioned it's more the
wording than intentions, but since it was in TL;DR; it has stronger
consequences.
BTW. Thanks for actually following the code of conduct and pointing it out
in respectful manner. I really appreciate it.
J.
Principal Software Engineer
Phone: +48660796129
On Thu, 4 Oct 2018, 20:41 Jakob Homan, <jgho...@gmail.com> wrote:
> > TL;DR; A change is coming in the way how dependencies/requirements are
> > specified for Apache Airflow - they will be fixed rather than flexible
> (==
> > rather than >=).
>
> > This is follow up after Slack discussion we had with Ash and Kaxil -
> > summarising what we propose we'll do.
>
> Hey all. It's great that we're moving this discussion back from Slack
> to the mailing list. But I've gotta point out that the wording needs
> a small but critical fix up:
>
> "A change *is* coming... they *will* be fixed"
>
> needs to be
>
> "We'd like to propose a change... We would like to make them fixed."
>
> The first says that this decision has been made and the result of the
> decision, which was made on Slack, is being reported back to the
> mailing list. The second is more accurate to the rest of the
> discussion ('what we propose...'). And again, since it's axiomatic in
> ASF that if it didn't happen on a list, it didn't happen[1], we gotta
> make sure there's no confusion about where the community is on the
> decision-making process.
>
> Thanks,
> Jakob
>
> [1]
> https://community.apache.org/newbiefaq.html#NewbieFAQ-IsthereaCodeofConductforApacheprojects
> ?
On Thu, Oct 4, 2018 at 9:56 AM Alex Guziel
> <alex.guz...@airbnb.com.invalid> wrote:
> >
> > You should run `pip check` to ensure no conflicts. Pip does not do this
> on
> > its own.
> >
> > On Thu, Oct 4, 2018 at 9:20 AM Jarek Potiuk <jarek.pot...@polidea.com>
> > wrote:
> >
> > > Great that this discussion already happened :). Lots of useful things
> in
> > > it. And yes - it means pinning in requirement.txt - this is how
> pip-tools
> > > work.
> > >
> > > J.
> > >
> > > Principal Software Engineer
> > > Phone: +48660796129
> > >
> > > On Thu, 4 Oct 2018, 18:14 Arthur Wiedmer, <arthur.wied...@gmail.com>
> > > wrote:
> > >
> > > > Hi Jarek,
> > > >
> > > > I will +1 the discussion Dan is referring to and George's advice.
> > > >
> > > > I just want to double check we are talking about pinning in
> > > > requirements.txt only.
> > > >
> > > > This offers the ability to
> > > > pip install -r requirements.txt
> > > > pip install --no-deps airflow
> > > > For a guaranteed install which works.
> > > >
> > > > Several different requirement files can be provided for specific use
> > > cases,
> > > > like a stable dev one for instance for people wanting to work on
> > > operators
> > > > and non-core functions.
> > > >
> > > > However, I think we should proactively test in CI against unpinned
> > > > dependencies (though it might be a separate case in the matrix) , so
> that
> > > > we get advance warning if possible that things will break.
> > > > CI downtime is not a bad thing here, it actually caught a problem :)
> > > >
> > > > We should unpin as possible in setup.py to only maintain minimum
> required
> > > > compatibility. The process of pinning in setup.py is extremely
> > > detrimental
> > > > when you have a large number of python libraries installed with
> different
> > > > pinned versions.
> > > >
> > > > Best,
> > > > Arthur
> > > >
> > > > On Thu, Oct 4, 2018 at 8:36 AM Dan Davydov
> <ddavy...@twitter.com.invalid
> > > >
> > > > wrote:
> > > >
> > > > > Relevant discussion about this:
> > > > >
> > > > >
> > > >
> > >
> https://github.com/apache/incubator-airflow/pull/1809#issuecomment-257502174
> > > > >
> > > > > On Thu, Oct 4, 2018 at 11:25 AM Jarek Potiuk <
> jarek.pot...@polidea.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > TL;DR; A change is coming in the way how
> dependencies/requirements
> > > are
> > > > > > specified for Apache Airflow - they will be fixed rather than
> > > flexible
> > > > > (==
> > > > > > rather than >=).
> > > > > >
> > > > > > This is follow up after Slack discussion we had with Ash and
> Kaxil -
> > > > > > summarising what we propose we'll do.
> > > > > >
> > > > > > *Problem:*
> > > > > > During last few weeks we experienced quite a few downtimes of
> > > TravisCI
> > > > > > builds (for all PRs/branches including master) as some of the
> > > > transitive
> > > > > > dependencies were automatically upgraded. This because in a
> number of
> > > > > > dependencies we have >= rather than == dependencies.
> > > > > >
> > > > > > Whenever there is a new release of such dependency, it might
> cause
> > > > chain
> > > > > > reaction with upgrade of transitive dependencies which might get
> into
> > > > > > conflict.
> > > > > >
> > > > > > An example was Flask-AppBuilder vs flask-login transitive
> dependency
> > > > with
> > > > > > click. They started to conflict once AppBuilder has released
> version
> > > > > > 1.12.0.
> > > > > >
> > > > > > *Diagnosis:*
> > > > > > Transitive dependencies with "flexible" versions (where >= is
> used
> > > > > instead
> > > > > > of ==) is a reason for "dependency hell". We will sooner or
> later hit
> > > > > other
> > > > > > cases where not fixed dependencies cause similar problems with
> other
> > > > > > transitive dependencies. We need to fix-pin them. This causes
> > > problems
> > > > > for
> > > > > > both - released versions (cause they stop to work!) and for
> > > development
> > > > > > (cause they break master builds in TravisCI and prevent people
> from
> > > > > > installing development environment from the scratch.
> > > > > >
> > > > > > *Solution:*
> > > > > >
> > > > > > - Following the old-but-good post
> > > > > > https://nvie.com/posts/pin-your-packages/ we are going to
> fix the
> > > > > > pinned
> > > > > > dependencies to specific versions (so basically all
> dependencies
> > > are
> > > > > > "fixed").
> > > > > > - We will introduce mechanism to be able to upgrade
> dependencies
> > > > with
> > > > > > pip-tools (https://github.com/jazzband/pip-tools). We might
> also
> > > > > take a
> > > > > > look at pipenv: https://pipenv.readthedocs.io/en/latest/
> > > > > > - People who would like to upgrade some dependencies for
> their PRs
> > > > > will
> > > > > > still be able to do it - but such upgrades will be in their PR
> > > thus
> > > > > they
> > > > > > will go through TravisCI tests and they will also have to be
> > > > specified
> > > > > > with
> > > > > > pinned fixed versions (==). This should be part of review
> process
> > > to
> > > > > > make
> > > > > > sure new/changed requirements are pinned.
> > > > > > - In release process there will be a point where an upgrade
> will
> > > be
> > > > > > attempted for all requirements (using pip-tools) so that we
> are
> > > not
> > > > > > stuck
> > > > > > with older releases. This will be in controlled PR environment
> > > where
> > > > > > there
> > > > > > will be time to fix all dependencies without impacting others
> and
> > > > > likely
> > > > > > enough time to "vet" such changes (this can be done for
> alpha/beta
> > > > > > releases
> > > > > > for example).
> > > > > > - As a side effect dependencies specification will become far
> > > > simpler
> > > > > > and straightforward.
> > > > > >
> > > > > > Happy to hear community comments to the proposal. I am happy to
> take
> > > a
> > > > > lead
> > > > > > on that, open JIRA issue and implement if this is something
> community
> > > > is
> > > > > > happy with.
> > > > > >
> > > > > > J.
> > > > > >
> > > > > > --
> > > > > >
> > > > > > *Jarek Potiuk, Principal Software Engineer*
> > > > > > Mobile: +48 660 796 129
> > > > > >
> > > > >
> > > >
> > >
>
