Awesome work Feng. Well done.
On 17/07/2018, 13:27, "James Meickle" <jmeic...@quantopian.com.INVALID> wrote:
Really excited for this one - we have a lot of internal access controls and
this will help us implement them properly. It's going to be great being
able to give everyone access to see the overall state of DAG progress
without seeing its parameters or logs!
On Tue, Jul 17, 2018 at 12:48 AM, Ruiqin Yang <yrql...@gmail.com> wrote:
> Congratulations! Extraordinary work! Thank you very much! This has been a
> highly desired feature for us for quite a while.
>
> Cheers,
> Kevin Yang
>
> Tao Feng <fengta...@gmail.com>于2018年7月16日 周一下午9:30写道：
>
> > Hi,
> >
> > Just want to give an update that Airflow DAG level access just checked
in
> > today(https://github.com/apache/incubator-airflow/pull/3197). Thanks a
> lot
> > for Max and Joy's review which helps me improving the pr. I create the
> > following three tickets as a follow up:
> >
> > https://issues.apache.org/jira/browse/AIRFLOW-2694 # Allow parsing
> access
> > control in DAG file
> > https://issues.apache.org/jira/browse/AIRFLOW-2693 # Implement sync_perm
> > model view endpoint
> > https://issues.apache.org/jira/browse/AIRFLOW-2695 # Support assign
> groups
> > of dag permission to a role
> >
> > I will start working on them in Q3.
> >
> > Thanks a lot,
> > -Tao
> >
> > On Sun, Apr 8, 2018 at 11:26 AM, Tao Feng <fengta...@gmail.com> wrote:
> >
> > > Hi Max, Joy and Everyone,
> > >
> > > Based on the discussion, I put up a work in progress pr (
> > > https://github.com/apache/incubator-airflow/pull/3197/) with a doc(
> > > https://docs.google.com/document/d/1qs26lE9kAuCY0Qa0ga-8
> > > 0EQ7d7m4s-590lhjtMBjmxw/edit#) for DAG level access. I would like to
> get
> > > some early feedbacks and see if I miss anything or am in the wrong
> > > direction as it touches the core part.
> > >
> > > In the meantime, I will still continue improving the pr for couples of
> > > todos.
> > >
> > > Looking forward to your feedback.
> > >
> > > Thanks,
> > > -Tao
> > >
> > > On Mon, Apr 2, 2018 at 2:44 PM, Tao Feng <fengta...@gmail.com> wrote:
> > >
> > >> Hi everyone,
> > >>
> > >> Thanks a lot for all the great discussions. To summarize in brief,
> here
> > >> are the few approaches we discussed so far:
> > >>
> > >> 1. One permission per DAG. The user has homogenous rights on the dag.
> > >> The concerns:
> > >>
> > >> - not flexible to certain use cases(e.g the user has view only
> access
> > >> on certain dags and write access on certain other dags );
> > >> - difficult to implement as it breaks the existing FAB security
> > model.
> > >>
> > >> 2. Extend current model(ab_permission_view_role) with an additional
> > >> optional column (dag_id).
> > >> The concerns:
> > >>
> > >> - introduce a 3rd column to existing permission_view table.
> > >> - It requires us to handle db migration carefully from view only
> > >> access UI to dag level access UI.
> > >>
> > >> 3. Define a set of pre-defined dag-level permissions. Reuse the
> current
> > >> existing permission_view model and consider each individual dag as a
> new
> > >> view.
> > >>
> > >> It seems that the 3rd approach is a preferable approach which makes
> the
> > >> security model easy to extend without introducing additional
> > complexity. If
> > >> no other concern, I will work with Max to create an initial proposal
/
> > PR
> > >> based on 3rd approach for the work(https://issues.apache.org
> > >> /jira/browse/AIRFLOW-2267).
> > >>
> > >> Thanks,
> > >> -Tao
> > >>
> > >> On Sat, Mar 31, 2018 at 12:09 AM, Joy Gao <j...@wepay.com> wrote:
> > >>
> > >>> +1!
> > >>>
> > >>> I was originally tempted to re-use existing perms and views for
> > dag-level
> > >>> access control since dag-level perm/view is a subset of view-level
> > >>> perm/view, but your proposal of defining new dag-level perms/views
> > >>> independent from view-level perms/views is interesting. This
actually
> > >>> makes
> > >>> a lot of sense, since even in the existing models, views can also be
> > menu
> > >>> options, so we are simply extending the concept of views to include
> > dags
> > >>> as
> > >>> well.
> > >>>
> > >>> On Fri, Mar 30, 2018 at 5:24 PM Maxime Beauchemin <
> > >>> maximebeauche...@gmail.com> wrote:
> > >>>
> > >>> > I'd suggest something else that avoids having to add a 3rd column.
> I
> > >>> think
> > >>> > we can fit our use case into the existing structure nicely.
> > >>> >
> > >>> > My idea is to mimic what FAB does with its own Models.
> > >>> >
> > >>> > When you create a Model and ModelView in FAB (say DagRun for
> > example),
> > >>> it
> > >>> > creates a new view_menu (DagRun) and matches it with existing
> > >>> permission
> > >>> > (can_read, can_delete, can_edit, can_show, ...) to create a new
set
> > of
> > >>> > "permission_views" which are combinations of permission and
> view_menu
> > >>> as in
> > >>> > "DagRun - can_read", "DagRun - can_edit", ... It's not a cartesian
> > >>> product
> > >>> > of all perms and view_menus, it's a predetermined list of
> > >>> model-specific
> > >>> > perms that get combined with DagRun here.
> > >>> >
> > >>> > Similarly, when Airflow would detect a new DAG (example "my_dag"),
> it
> > >>> > would create a new view_menu `my_dag`, and match it with
> permissions
> > >>> that
> > >>> > are identified as "combinable" with DAG. To avoid potential
> conflicts
> > >>> I'd
> > >>> > create new permissions that are DAG-specific like "dag_clear",
> > >>> "dag_run",
> > >>> > "dag_view".
> > >>> >
> > >>> > I'm arguing about the how to use the FAB model here specifically.
I
> > >>> think
> > >>> > this allows for all the flexibility we need without changing the
> > >>> model. I
> > >>> > care less about what exactly the atomicity of the per DAG perms
> > should
> > >>> look
> > >>> > like. As far as I'm concerned, per-DAG read and write is probably
> > >>> granular
> > >>> > enough
> > >>> >
> > >>> > Also note that:
> > >>> > * we need an "_all_dags" logical DAG, meaning we'd have extra
> > >>> > permission_views "_all_dags - dag_view", "_all_dags - dag_run",
> > >>> "all_dags -
> > >>> > dag_clear"
> > >>> > * we probably want to derive FAB's SecurityManger and have an
> > >>> > AirflowSecurityManager that has an extra method
> "can_dag_action(user,
> > >>> > dag_id, action)". The SecurityManger class is neat because people
> can
> > >>> > provide their own if they want and override methods. That means
> that
> > >>> you
> > >>> > can defer any of the security-related checks to another system if
> you
> > >>> want
> > >>> > to. Many companies have some sort of company RBAC system and that
> can
> > >>> be
> > >>> > used to integrate with it.
> > >>> >
> > >>> > Max
> > >>> >
> > >>> >
> > >>> >
> > >>> > On Fri, Mar 30, 2018 at 4:54 PM, Joy Gao <j...@wepay.com> wrote:
> > >>> >
> > >>> >> Hi all,
> > >>> >>
> > >>> >> I also agree that having view-only access to some dags while
write
> > >>> access
> > >>> >> to other dags is useful, so I prefer option 2. Although option 2
> is
> > >>> more
> > >>> >> difficult to manage, it is cleaner and more consistent with the
> > >>> current
> > >>> >> security model. (On the other hand, even though option 1 may be
> may
> > be
> > >>> >> easier to manage, it might be trickier to implement: with one
perm
> > per
> > >>> >> dag, it breaks the existing FAB security model since the
existing,
> > >>> more
> > >>> >> granular permissions will have to be grouped for each dag).
> > >>> >>
> > >>> >> What Brian suggested in the other thread makes sense to me...
> > >>> >>
> > >>> >> The current security model in FAB looks like the following:
> > >>> >>
> > >>> >> ---
> > >>> permissions
> > >>> >> user --- role --- permission_view ---|
> > >>> >> --- views
> > >>> >>
> > >>> >> FAB generates a few relationship tables to manage the mappings,
> one
> > of
> > >>> >> them is *ab_permission_view_role*, which has all the
> > >>> >> role-to-permission_view mapping. We can add a dag_id column to
> this
> > >>> >> relationship that expresses dag-level granularity, so the
security
> > >>> model
> > >>> >> becomes:
> > >>> >>
> > >>> >>
> > ---
> > >>> >> permissions
> > >>> >> |
> > >>> >> user --- role --- dag_permission_view --- --- views
> > >>> >> |
> > >>> >>
> ---
> > >>> dags
> > >>> >>
> > >>> >> We can make the dag_id field an optional field in the
> relationship,
> > >>> and
> > >>> >> only lazily add new dag-level mappings for DAGs that specify
> 'access
> > >>> >> control'. This way we don't have to create new permissions or
> views.
> > >>> (We
> > >>> >> will still have to introduce new dag-level roles on top of the 5
> > >>> generic
> > >>> >> roles (public/viewer/user/op/admin)).
> > >>> >>
> > >>> >> (I think this is similar to what Arthur suggested earlier, but
not
> > >>> sure
> > >>> >> if I interpreted correctly).
> > >>> >>
> > >>> >> Joy
> > >>> >>
> > >>> >> On Thu, Mar 29, 2018 at 10:01 AM, Arthur Wiedmer <
> > >>> >> arthur.wied...@gmail.com> wrote:
> > >>> >>
> > >>> >>> (Creating a new thread)
> > >>> >>>
> > >>> >>> Hi Max,
> > >>> >>>
> > >>> >>> I was just wondering about this. There are definite use cases
for
> > >>> people
> > >>> >>> having only view access to some DAGs, mostly for monitoring. I
> want
> > >>> to know
> > >>> >>> what the upstream DAGs are doing, but maybe I don't need
> clear/run
> > >>> access.
> > >>> >>>
> > >>> >>> I feel like the granular operation permissions will start to
> become
> > >>> >>> unwieldy fast. It grows as (# users * # DAGs * # operations) but
> > with
> > >>> >>> hopefully a small constant factor in front: most people should
> only
> > >>> have a
> > >>> >>> small number of DAGs they care about. The Ops team has a need to
> > have
> > >>> >>> access to all on the other hand.
> > >>> >>>
> > >>> >>> I was wondering we could get by with something slightly more
> > complex
> > >>> but
> > >>> >>> easier on the size of that permissions table :
> > >>> >>> 1) One toggle on the user level for broad access (ALL:ALL,
> > >>> >>> RUN/CLEAR:ALL, VIEW:ALL) default NULL
> > >>> >>> 2) More granular permissions at the DAG level.
> > >>> >>>
> > >>> >>> So in order, check the user's broad level permission first, then
> > DAG
> > >>> >>> level. For large amounts of DAGs, this should help shave a
little
> > >>> bit from
> > >>> >>> that table.
> > >>> >>>
> > >>> >>> Best,
> > >>> >>> Arthur
> > >>> >>>
> > >>> >>>
> > >>> >>> On Thu, Mar 29, 2018 at 8:27 AM, Maxime Beauchemin <
> > >>> >>> maximebeauche...@gmail.com> wrote:
> > >>> >>>
> > >>> >>>> Hijacking the thread further here, any thoughts on how to
> > breakdown
> > >>> per
> > >>> >>>> DAG
> > >>> >>>> access?
> > >>> >>>>
> > >>> >>>> Tao & I are talking about introducing per-DAG permissions and
> one
> > >>> big
> > >>> >>>> question is whether we'll need to support different
> > operation-types
> > >>> at a
> > >>> >>>> per-DAG level, which changes the way we need to model the
perms.
> > >>> >>>>
> > >>> >>>> First [simpler] option is to introduce one perm per DAG. If you
> > have
> > >>> >>>> access
> > >>> >>>> to 5 DAGs, and you have `can_clear` and `can_run`, you'll have
> > >>> >>>> homogenous
> > >>> >>>> rights on the DAGs you have access to.
> > >>> >>>>
> > >>> >>>> Second option is to have a breakdown per DAG. Meaning for each
> DAG
> > >>> we
> > >>> >>>> create a set of perms ({dag_id}_can_view, {dag_id}_can_modify,
> > >>> ...). So
> > >>> >>>> one
> > >>> >>>> user could have modify on some DAGs, view on others, and other
> > DAGs
> > >>> >>>> would
> > >>> >>>> be invisible. This could be broken down further
> > ({dag_id}_can_clear,
> > >>> >>>> ...)
> > >>> >>>> but it gets hard to manage.
> > >>> >>>>
> > >>> >>>> Thoughts?
> > >>> >>>>
> > >>> >>>> Max
> > >>> >>>>
> > >>> >>>> On Wed, Mar 28, 2018 at 10:02 PM, Tao Feng <fengta...@gmail.com
> >
> > >>> wrote:
> > >>> >>>>
> > >>> >>>> > Great work Joy. This is awesome! I am interested in helping
> out
> > >>> the
> > >>> >>>> per dag
> > >>> >>>> > level access. Just created a ticket to check(AIRFLOW-2267).
> Let
> > >>> me
> > >>> >>>> know if
> > >>> >>>> > you have any suggestions. I will share my proposal once I am
> > >>> ready.
> > >>> >>>> >
> > >>> >>>> > On Fri, Mar 23, 2018 at 6:45 PM, Joy Gao <j...@wepay.com>
> > wrote:
> > >>> >>>> >
> > >>> >>>> > > Hey guys!
> > >>> >>>> > >
> > >>> >>>> > > The RBAC UI <https://github.com/apache/inc
> > >>> ubator-airflow/pull/3015>
> > >>> >>>> has
> > >>> >>>> > > been merged to master. I'm looking forward to early
> adopters'
> > >>> >>>> feedback
> > >>> >>>> > and
> > >>> >>>> > > bug reports. I also hope to have more folks helping out
with
> > the
> > >>> >>>> RBAC UI,
> > >>> >>>> > > especially with introducing DAG-Level access control, which
> > is a
> > >>> >>>> feature
> > >>> >>>> > > that a lot of people have been asking. If you are
interested
> > in
> > >>> >>>> helping
> > >>> >>>> > out
> > >>> >>>> > > with this effort, let's talk more!
> > >>> >>>> > >
> > >>> >>>> > > This commit will be in the 1.10.0 release, and we are going
> to
> > >>> >>>> maintain
> > >>> >>>> > > both UIs simultaneously for a short period of time. Once
> RBAC
> > >>> UI is
> > >>> >>>> > stable
> > >>> >>>> > > and battle-tested, we will deprecate the old UI and
> eventually
> > >>> >>>> remove it
> > >>> >>>> > > from the repo (around Airflow 2.0.0 or 2.1.0 release). This
> is
> > >>> to
> > >>> >>>> prevent
> > >>> >>>> > > two UIs from forking into separate paths, as that would
> become
> > >>> very
> > >>> >>>> > > difficult to maintain.
> > >>> >>>> > >
> > >>> >>>> > > Going forward while both UIs are up, if you are making a
> > change
> > >>> to
> > >>> >>>> any
> > >>> >>>> > > files in airflow/www/ (old UI), where applicable, please
> also
> > >>> make
> > >>> >>>> the
> > >>> >>>> > > change to the airflow/www_rbac/ (new UI). If you rather not
> > make
> > >>> >>>> changes
> > >>> >>>> > in
> > >>> >>>> > > both UIs, it is recommended that you only make the changes
> to
> > >>> the
> > >>> >>>> RBAC
> > >>> >>>> > UI,
> > >>> >>>> > > since that is the one we are maintaining in the long term.
> > >>> >>>> > >
> > >>> >>>> > > I'm excited that the RBAC UI will be able to bring
> additional
> > >>> >>>> security to
> > >>> >>>> > > Airflow, and with FAB framework in place we can look into
> > >>> >>>> leveraging it
> > >>> >>>> > for
> > >>> >>>> > > a unified set of APIs used by both UI and CLI.
> > >>> >>>> > >
> > >>> >>>> > > Joy
> > >>> >>>> > >
> > >>> >>>> > >
> > >>> >>>> > >
> > >>> >>>> > > On Thu, Feb 8, 2018 at 11:31 AM, Joy Gao <j...@wepay.com>
> > >>> wrote:
> > >>> >>>> > >
> > >>> >>>> > > > Hi folks,
> > >>> >>>> > > >
> > >>> >>>> > > > I have a PR <https://github.com/apache/
> > >>> >>>> incubator-airflow/pull/3015>
> > >>> >>>> > out
> > >>> >>>> > > > for the new UI. I've included instructions on how to test
> it
> > >>> out
> > >>> >>>> in the
> > >>> >>>> > > PR
> > >>> >>>> > > > description. Looking forward to your feedbacks.
> > >>> >>>> > > >
> > >>> >>>> > > > Cheers,
> > >>> >>>> > > > Joy
> > >>> >>>> > > >
> > >>> >>>> > > > On Fri, Dec 1, 2017 at 6:18 PM, Joy Gao <j...@wepay.com>
> > >>> wrote:
> > >>> >>>> > > >
> > >>> >>>> > > >> Thanks for the background info. Would be really awesome
> for
> > >>> you
> > >>> >>>> to
> > >>> >>>> > have
> > >>> >>>> > > >> PyPi access :D I'll make the change to have Airflow
> > >>> Webserver's
> > >>> >>>> FAB
> > >>> >>>> > > >> dependency pointing to my fork for the mean time.
> > >>> >>>> > > >>
> > >>> >>>> > > >> For folks who are interested in RBAC, I will be giving a
> > >>> >>>> talk/demo at
> > >>> >>>> > > the Airflow
> > >>> >>>> > > >> Meet-Up
> > >>> >>>> > > >> <
> > https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-
> > >>> >>>> > > Meetup/events/244525050/>
> > >>> >>>> > > >> next Monday. Happy to chat afterwards about it as well
:)
> > >>> >>>> > > >>
> > >>> >>>> > > >> On Thu, Nov 30, 2017 at 8:36 AM, Maxime Beauchemin <
> > >>> >>>> > > >> maximebeauche...@gmail.com> wrote:
> > >>> >>>> > > >>
> > >>> >>>> > > >>> A bit of related history here:
> > >>> >>>> > > >>> https://github.com/dpgaspar/Flask-AppBuilder/issues/399
> > >>> >>>> > > >>>
> > >>> >>>> > > >>> On Thu, Nov 30, 2017 at 8:33 AM, Maxime Beauchemin <
> > >>> >>>> > > >>> maximebeauche...@gmail.com> wrote:
> > >>> >>>> > > >>>
> > >>> >>>> > > >>> > Given I have merge rights on FAB I could probably do
> > >>> another
> > >>> >>>> round
> > >>> >>>> > of
> > >>> >>>> > > >>> > review and get your PRs through. I would really like
> to
> > >>> get
> > >>> >>>> the
> > >>> >>>> > main
> > >>> >>>> > > >>> > maintainer's input on things that touch the core
> > >>> >>>> (composite-key
> > >>> >>>> > > >>> support) as
> > >>> >>>> > > >>> > he might have concerns/intuitions that we can't know
> > >>> about.
> > >>> >>>> > > >>> >
> > >>> >>>> > > >>> > I do not have Pypi access though so I cannot push new
> > >>> releases
> > >>> >>>> > out. I
> > >>> >>>> > > >>> > could ask for that.
> > >>> >>>> > > >>> >
> > >>> >>>> > > >>> > I've threatened to fork the project before, that's
> > always
> > >>> an
> > >>> >>>> > option.
> > >>> >>>> > > >>> I've
> > >>> >>>> > > >>> > noticed his involvement is sporadic and comes in
> bursts.
> > >>> >>>> > > >>> >
> > >>> >>>> > > >>> > In the meantime, you can have the dependency in
> Airflow
> > >>> >>>> Webserver
> > >>> >>>> > > >>> pointing
> > >>> >>>> > > >>> > straight to your fork.
> > >>> >>>> > > >>> >
> > >>> >>>> > > >>> > Max
> > >>> >>>> > > >>> >
> > >>> >>>> > > >>> > On Wed, Nov 29, 2017 at 7:02 PM, Joy Gao <
> > j...@wepay.com>
> > >>> >>>> wrote:
> > >>> >>>> > > >>> >
> > >>> >>>> > > >>> >> I just created a new webserver instance if you
> haven't
> > >>> >>>> gotten a
> > >>> >>>> > > >>> chance to
> > >>> >>>> > > >>> >> fiddle around with the new web UI and the RBAC
> > >>> configurations
> > >>> >>>> > > (thanks
> > >>> >>>> > > >>> >> Maxime for getting started with this earlier!):
> > >>> >>>> > > >>> >>
> > >>> >>>> > > >>> >> http://104.209.38.171:8080/
> > >>> >>>> > > >>> >>
> > >>> >>>> > > >>> >> Admin Account
> > >>> >>>> > > >>> >> username: admin
> > >>> >>>> > > >>> >> password: admin
> > >>> >>>> > > >>> >>
> > >>> >>>> > > >>> >> Read-Only Account
> > >>> >>>> > > >>> >> username: viewer
> > >>> >>>> > > >>> >> password: password
> > >>> >>>> > > >>> >>
> > >>> >>>> > > >>> >>
> > >>> >>>> > > >>> >> On Wed, Nov 29, 2017 at 2:58 PM, Joy Gao <
> > j...@wepay.com
> > >>> >
> > >>> >>>> wrote:
> > >>> >>>> > > >>> >>
> > >>> >>>> > > >>> >> > Hi folks,
> > >>> >>>> > > >>> >> >
> > >>> >>>> > > >>> >> > Thanks for all the feedback regarding to the new
> > >>> Airflow
> > >>> >>>> > Webserver
> > >>> >>>> > > >>> UI
> > >>> >>>> > > >>> >> > <https://github.com/wepay/airflow-webserver/>!
> I've
> > >>> been
> > >>> >>>> > actively
> > >>> >>>> > > >>> >> > addressing all the bugs that were raised on
Github.
> > So
> > >>> I
> > >>> >>>> want to
> > >>> >>>> > > >>> take
> > >>> >>>> > > >>> >> this
> > >>> >>>> > > >>> >> > opportunity to discuss two issues coming up:
> > >>> >>>> > > >>> >> >
> > >>> >>>> > > >>> >> > The first issue is unaddressed PRs in FAB. If
these
> > PRs
> > >>> >>>> continue
> > >>> >>>> > > to
> > >>> >>>> > > >>> stay
> > >>> >>>> > > >>> >> > unaddressed, RBAC is blocked from making further
> > >>> progress.
> > >>> >>>> If
> > >>> >>>> > this
> > >>> >>>> > > >>> >> continue
> > >>> >>>> > > >>> >> > to be an issue, I'm inclined to fork FAB, even
> though
> > >>> it's
> > >>> >>>> not
> > >>> >>>> > > >>> >> idealistic.
> > >>> >>>> > > >>> >> >
> > >>> >>>> > > >>> >> >
> > >>> >>>> > > >>> >> > - PR/631 <https://github.com/dpgaspar/F
> > >>> >>>> > > lask-AppBuilder/pull/631>
> > >>> >>>> > > >>> >> Binary
> > >>> >>>> > > >>> >> > column support (merged, unreleased)
> > >>> >>>> > > >>> >> > <https://github.com/dpgaspar/F
> > >>> lask-AppBuilder/pull/631>
> > >>> >>>> > > >>> >> > - PR/639 <https://github.com/dpgaspar/F
> > >>> >>>> > > lask-AppBuilder/pull/639>
> > >>> >>>> > > >>> >> Composite
> > >>> >>>> > > >>> >> > primary key support (unmerged)
> > >>> >>>> > > >>> >> > - PR/655 <https://github.com/dpgaspar/F
> > >>> >>>> > > lask-AppBuilder/pull/655>
> > >>> >>>> > > >>> >> Form
> > >>> >>>> > > >>> >> > prefill support (unmerged)
> > >>> >>>> > > >>> >> >
> > >>> >>>> > > >>> >> >
> > >>> >>>> > > >>> >> > The second issue is an open question about the
next
> > >>> step of
> > >>> >>>> > > Airflow
> > >>> >>>> > > >>> >> > Webserver itself. Here are the 3 potential
> directions
> > >>> we
> > >>> >>>> could
> > >>> >>>> > > >>> take, and
> > >>> >>>> > > >>> >> > I've added my thought on each.
> > >>> >>>> > > >>> >> >
> > >>> >>>> > > >>> >> > 1. Permanently keep Airflow Webserver as a
> separated
> > >>> >>>> package
> > >>> >>>> > from
> > >>> >>>> > > >>> >> Airflow,
> > >>> >>>> > > >>> >> > and treat it as another UI option. Keep `www` in
> > >>> Airflow.
> > >>> >>>> Allow
> > >>> >>>> > > >>> >> development
> > >>> >>>> > > >>> >> > on both UIs.
> > >>> >>>> > > >>> >> > *I'm not a fan of this. When there is an existing
> UI
> > in
> > >>> >>>> Airflow,
> > >>> >>>> > > >>> most
> > >>> >>>> > > >>> >> > contributors would prefer to maintain the official
> > >>> version
> > >>> >>>> that
> > >>> >>>> > is
> > >>> >>>> > > >>> >> > installed out-of-the-box. **Having a second UI
> > outside
> > >>> of
> > >>> >>>> > Airflow
> > >>> >>>> > > >>> will
> > >>> >>>> > > >>> >> > make maintaining it very difficult, leading to an
> > >>> eventual
> > >>> >>>> death
> > >>> >>>> > > of
> > >>> >>>> > > >>> the
> > >>> >>>> > > >>> >> new
> > >>> >>>> > > >>> >> > UI :(*
> > >>> >>>> > > >>> >> >
> > >>> >>>> > > >>> >> > 2. Permanently keep Airflow Webserver as a
> separated
> > >>> >>>> package
> > >>> >>>> > from
> > >>> >>>> > > >>> >> Airflow,
> > >>> >>>> > > >>> >> > but freeze all development on `www` and direct
all
> > >>> future
> > >>> >>>> UI
> > >>> >>>> > > >>> >> development
> > >>> >>>> > > >>> >> > to Airflow Webserver, eventually removing `www`
> > >>> completely
> > >>> >>>> when
> > >>> >>>> > > >>> Airflow
> > >>> >>>> > > >>> >> > Webserver is stable.
> > >>> >>>> > > >>> >> > *I'm not a fan of this either. First of all, the
> > views
> > >>> and
> > >>> >>>> > models
> > >>> >>>> > > >>> are
> > >>> >>>> > > >>> >> > tightly coupled in both old and new UI; until we
> > have a
> > >>> >>>> > > full-fledged
> > >>> >>>> > > >>> >> REST
> > >>> >>>> > > >>> >> > API to build the UI (and cli) on top of it,
> > separating
> > >>> >>>> them to a
> > >>> >>>> > > >>> >> separate
> > >>> >>>> > > >>> >> > package now will potentially cause dependency
> issues
> > >>> and
> > >>> >>>> add
> > >>> >>>> > > >>> >> complication
> > >>> >>>> > > >>> >> > to our release cycle. **Secondly, **majority of
> > Airflow
> > >>> >>>> users
> > >>> >>>> > run
> > >>> >>>> > > >>> >> Airflow
> > >>> >>>> > > >>> >> > with the UI; it's one of Airflow's best features.
> > >>> >>>> Separating UI
> > >>> >>>> > > out
> > >>> >>>> > > >>> of
> > >>> >>>> > > >>> >> > Airflow core will complicate setup and
> configuration,
> > >>> while
> > >>> >>>> > making
> > >>> >>>> > > >>> >> Airflow
> > >>> >>>> > > >>> >> > core less complete.*
> > >>> >>>> > > >>> >> >
> > >>> >>>> > > >>> >> > 3. Merge Airflow Webserver back into Airflow as
> > `www2`,
> > >>> >>>> freeze
> > >>> >>>> > all
> > >>> >>>> > > >>> >> > development on `www`, eventually removing `www`
> > >>> completely
> > >>> >>>> when
> > >>> >>>> > > >>> `www2`
> > >>> >>>> > > >>> >> is
> > >>> >>>> > > >>> >> > stable.
> > >>> >>>> > > >>> >> > *This makes the most sense to me. Airflow
Webserver
> > is
> > >>> >>>> developed
> > >>> >>>> > > >>> with
> > >>> >>>> > > >>> >> the
> > >>> >>>> > > >>> >> > goal of feature parity to the current UI, plus
> > >>> additional
> > >>> >>>> RBAC
> > >>> >>>> > > >>> >> capability,
> > >>> >>>> > > >>> >> > in hope to replace the old UI completely. Yes,
this
> > >>> means
> > >>> >>>> there
> > >>> >>>> > > >>> will be
> > >>> >>>> > > >>> >> a
> > >>> >>>> > > >>> >> > short period of having to maintain two UIs, but
> once
> > we
> > >>> >>>> freeze
> > >>> >>>> > > >>> >> development
> > >>> >>>> > > >>> >> > on www, it shouldn't be a concern for long.*
> > >>> >>>> > > >>> >> >
> > >>> >>>> > > >>> >> > I'd love to hear everyone's thoughts on this! I'm
> > >>> excited
> > >>> >>>> about
> > >>> >>>> > > >>> bringing
> > >>> >>>> > > >>> >> > RBAC to airflow and I hope it's something others
> will
> > >>> find
> > >>> >>>> > useful
> > >>> >>>> > > as
> > >>> >>>> > > >>> >> well!
> > >>> >>>> > > >>> >> >
> > >>> >>>> > > >>> >> > Cheers,
> > >>> >>>> > > >>> >> > Joy
> > >>> >>>> > > >>> >> >
> > >>> >>>> > > >>> >> > On Mon, Nov 20, 2017 at 11:24 AM, Joy Gao <
> > >>> j...@wepay.com>
> > >>> >>>> > wrote:
> > >>> >>>> > > >>> >> >
> > >>> >>>> > > >>> >> >> Thank you everyone for the active feedback so
far,
> > and
> > >>> >>>> thanks
> > >>> >>>> > for
> > >>> >>>> > > >>> >> setting
> > >>> >>>> > > >>> >> >> up the demo Maxime!
> > >>> >>>> > > >>> >> >>
> > >>> >>>> > > >>> >> >> Going to work on pruning through the issues in
the
> > >>> >>>> upcoming
> > >>> >>>> > days.
> > >>> >>>> > > >>> >> >>
> > >>> >>>> > > >>> >> >> Fokko/Maxime, do you recall the SQLAlchemy
> Exception
> > >>> >>>> message
> > >>> >>>> > so I
> > >>> >>>> > > >>> can
> > >>> >>>> > > >>> >> >> look into it? Otherwise I'll wait until it's down
> > >>> again =P
> > >>> >>>> > > >>> >> >>
> > >>> >>>> > > >>> >> >> Cheers,
> > >>> >>>> > > >>> >> >>
> > >>> >>>> > > >>> >> >> Joy
> > >>> >>>> > > >>> >> >>
> > >>> >>>> > > >>> >> >> On Mon, Nov 20, 2017 at 9:35 AM, Maxime
> Beauchemin <
> > >>> >>>> > > >>> >> >> maximebeauche...@gmail.com> wrote:
> > >>> >>>> > > >>> >> >>
> > >>> >>>> > > >>> >> >>> I just restarted it, not sure how long it will
> take
> > >>> to
> > >>> >>>> get in
> > >>> >>>> > a
> > >>> >>>> > > >>> bad
> > >>> >>>> > > >>> >> state
> > >>> >>>> > > >>> >> >>> again...
> > >>> >>>> > > >>> >> >>>
> > >>> >>>> > > >>> >> >>> Max
> > >>> >>>> > > >>> >> >>>
> > >>> >>>> > > >>> >> >>> On Sun, Nov 19, 2017 at 11:55 PM, Driesprong,
> Fokko
> > >>> >>>> > > >>> >> <fo...@driesprong.frl
> > >>> >>>> > > >>> >> >>> >
> > >>> >>>> > > >>> >> >>> wrote:
> > >>> >>>> > > >>> >> >>>
> > >>> >>>> > > >>> >> >>> > Good morning,
> > >>> >>>> > > >>> >> >>> >
> > >>> >>>> > > >>> >> >>> > The demo provided by Max is down, it throws a
> > >>> >>>> > > >>> SQLAlchemyexception
> > >>> >>>> > > >>> >> :'(
> > >>> >>>> > > >>> >> >>> >
> > >>> >>>> > > >>> >> >>> > Cheers, Fokko
> > >>> >>>> > > >>> >> >>> >
> > >>> >>>> > > >>> >> >>> > 2017-11-18 19:14 GMT+01:00 Chris Riccomini <
> > >>> >>>> > > >>> criccom...@apache.org>:
> > >>> >>>> > > >>> >> >>> >
> > >>> >>>> > > >>> >> >>> > > @bolke, open issues on the Github repo,
> please.
> > >>> >>>> > > >>> >> >>> > >
> > >>> >>>> > > >>> >> >>> > > On Sat, Nov 18, 2017 at 10:13 AM, Bolke de
> > Bruin
> > >>> <
> > >>> >>>> > > >>> >> bdbr...@gmail.com>
> > >>> >>>> > > >>> >> >>> > > wrote:
> > >>> >>>> > > >>> >> >>> > >
> > >>> >>>> > > >>> >> >>> > > > Chris,
> > >>> >>>> > > >>> >> >>> > > >
> > >>> >>>> > > >>> >> >>> > > > Do you want us to report bugs somewhere (I
> > have
> > >>> >>>> > > encountered
> > >>> >>>> > > >>> a
> > >>> >>>> > > >>> >> >>> few)? Or
> > >>> >>>> > > >>> >> >>> > > > just generic user experiences posted here?
> > >>> >>>> > > >>> >> >>> > > >
> > >>> >>>> > > >>> >> >>> > > > Cheers
> > >>> >>>> > > >>> >> >>> > > > Bolke
> > >>> >>>> > > >>> >> >>> > > >
> > >>> >>>> > > >>> >> >>> > > > >
Kaxil Naik
Data Reply
2nd Floor, Nova South
160 Victoria Street, Westminster
London SW1E 5LB - UK
phone: +44 (0)20 7730 6000
k.n...@reply.com
www.reply.com
On 18 Nov 2017, at 00:47, Chris
> Riccomini <
> > >>> >>>> > > >>> >> criccom...@apache.org
> > >>> >>>> > > >>> >> >>> >
> > >>> >>>> > > >>> >> >>> > > wrote:
> > >>> >>>> > > >>> >> >>> > > > >
> > >>> >>>> > > >>> >> >>> > > > > Hey all,
> > >>> >>>> > > >>> >> >>> > > > >
> > >>> >>>> > > >>> >> >>> > > > > I know the weekend is coming up, and for
> > >>> those
> > >>> >>>> of us
> > >>> >>>> > in
> > >>> >>>> > > >>> the
> > >>> >>>> > > >>> >> US,
> > >>> >>>> > > >>> >> >>> next
> > >>> >>>> > > >>> >> >>> > > week
> > >>> >>>> > > >>> >> >>> > > > > is a bit of a slow holiday week. Would
> love
> > >>> to
> > >>> >>>> get
> > >>> >>>> > some
> > >>> >>>> > > >>> >> feedback
> > >>> >>>> > > >>> >> >>> from
> > >>> >>>> > > >>> >> >>> > > > > everyone on this. The goal would ideally
> to
> > >>> be to
> > >>> >>>> > > >>> converge on
> > >>> >>>> > > >>> >> >>> this
> > >>> >>>> > > >>> >> >>> > and
> > >>> >>>> > > >>> >> >>> > > > > eventually replace the existing Airflow
> UI
> > >>> with
> > >>> >>>> this
> > >>> >>>> > > one.
> > >>> >>>> > > >>> >> >>> > > > >
> > >>> >>>> > > >>> >> >>> > > > > Cheers,
> > >>> >>>> > > >>> >> >>> > > > > Chris
> > >>> >>>> > > >>> >> >>> > > > >
> > >>> >>>> > > >>> >> >>> > > > > On Fri, Nov 17, 2017 at 1:44 PM, Joy Gao
> <
> > >>> >>>> > > j...@wepay.com>
> > >>> >>>> > > >>> >> wrote:
> > >>> >>>> > > >>> >> >>> > > > >
> > >>> >>>> > > >>> >> >>> > > > >> Hi guys.
> > >>> >>>> > > >>> >> >>> > > > >>
> > >>> >>>> > > >>> >> >>> > > > >> I've been working on moving airflow
from
> > >>> >>>> Flask-Admin
> > >>> >>>> > to
> > >>> >>>> > > >>> >> >>> > > Flask-AppBuilder
> > >>> >>>> > > >>> >> >>> > > > >> for RBAC
> > >>> >>>> > > >>> >> >>> > > > >> <https://cwiki.apache.org/
> > >>> >>>> > confluence/display/AIRFLOW/
> > >>> >>>> > > >>> >> >>> > > > Airflow+RBAC+proposal
> > >>> >>>> > > >>> >> >>> > > > >>> ,
> > >>> >>>> > > >>> >> >>> > > > >> check it out at
> > >>> https://github.com/wepay/airfl
> > >>> >>>> > > >>> ow-webserver.
> > >>> >>>> > > >>> >> >>> > > > >>
> > >>> >>>> > > >>> >> >>> > > > >> It's still a work-in-progress, but most
> > >>> >>>> features you
> > >>> >>>> > > see
> > >>> >>>> > > >>> in
> > >>> >>>> > > >>> >> the
> > >>> >>>> > > >>> >> >>> > > > webserver
> > >>> >>>> > > >>> >> >>> > > > >> UI today is available there. For those
> who
> > >>> are
> > >>> >>>> > > >>> interested in
> > >>> >>>> > > >>> >> >>> RBAC,
> > >>> >>>> > > >>> >> >>> > I'd
> > >>> >>>> > > >>> >> >>> > > > love
> > >>> >>>> > > >>> >> >>> > > > >> to get some early feedback in terms of
> the
> > >>> >>>> following:
> > >>> >>>> > > >>> >> >>> > > > >>
> > >>> >>>> > > >>> >> >>> > > > >> - New Flask-AppBuilder UI (any
> > >>> bugs/regressions)
> > >>> >>>> > > >>> >> >>> > > > >> - Setup issues
> > >>> >>>> > > >>> >> >>> > > > >> - Ease of integration with third party
> > auth
> > >>> >>>> (i.e.
> > >>> >>>> > LDAP,
> > >>> >>>> > > >>> AD,
> > >>> >>>> > > >>> >> >>> OAuth,
> > >>> >>>> > > >>> >> >>> > > etc.)
> > >>> >>>> > > >>> >> >>> > > > >> - Any other thoughts/concerns
> > >>> >>>> > > >>> >> >>> > > > >>
> > >>> >>>> > > >>> >> >>> > > > >> Thanks a lot!
> > >>> >>>> > > >>> >> >>> > > > >>
> > >>> >>>> > > >>> >> >>> > > > >> Cheers,
> > >>> >>>> > > >>> >> >>> > > > >> Joy
> > >>> >>>> > > >>> >> >>> > > > >>
> > >>> >>>> > > >>> >> >>> > > >
> > >>> >>>> > > >>> >> >>> > > >
> > >>> >>>> > > >>> >> >>> > >
> > >>> >>>> > > >>> >> >>> >
> > >>> >>>> > > >>> >> >>>
> > >>> >>>> > > >>> >> >>
> > >>> >>>> > > >>> >> >>
> > >>> >>>> > > >>> >> >
> > >>> >>>> > > >>> >>
> > >>> >>>> > > >>> >
> > >>> >>>> > > >>> >
> > >>> >>>> > > >>>
> > >>> >>>> > > >>
> > >>> >>>> > > >>
> > >>> >>>> > > >
> > >>> >>>> > >
> > >>> >>>> >
> > >>> >>>>
> > >>> >>>
> > >>> >>>
> > >>> >>
> > >>> >
> > >>>
> > >>
> > >>
> > >
> >
>
