[jira] [Closed] (AIRFLOW-3546) Typo in jobs.py logs
[ https://issues.apache.org/jira/browse/AIRFLOW-3546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng closed AIRFLOW-3546. - Resolution: Fixed > Typo in jobs.py logs > > > Key: AIRFLOW-3546 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3546 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Reporter: Stan Kudrow >Assignee: Stan Kudrow >Priority: Trivial > > PR: https://github.com/apache/incubator-airflow/pull/4346 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3542) next_ds semantics broken for manually triggered runs
[
https://issues.apache.org/jira/browse/AIRFLOW-3542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16725231#comment-16725231
]
Tao Feng commented on AIRFLOW-3542:
---
lgtm. cc [~milton0825]
> next_ds semantics broken for manually triggered runs
>
>
> Key: AIRFLOW-3542
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3542
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler
>Affects Versions: 1.10.2
>Reporter: Dan Davydov
>Assignee: Dan Davydov
>Priority: Major
>
> {color:#22}next_ds{color}{color:#22} is useful when you need
> cron-style scheduling, e.g. a task that runs for date "X" uses that date for
> its logic, e.g. send an email to users saying the run that was supposed to
> run for date "X" has completed. The problem is it doesn't behave as expected
> when it comes to manually triggered runs as illustrated by the diagrams
> below.{color}
>
> Using execution_date in a task
> *Scheduled Run (works as expected)*
> execution_date1 start_date1
> \/ \/
> *|-|*
> /\ /\
> \_/
> scheduling_interval
>
> *Manual Run* *(works as expected)*
> triggered_date + execution_date + start_date
> \/
> *|*
>
> Using next_ds in a Task
> *Scheduled Run (works as expected)*
> next_ds1 + start_date1 next_ds2 + start_date2
> \/ \/
> *||*
> /\ /\
> \/
> scheduling_interval
>
> *Manual Run* *(next_ds1 is expected to match triggered_date as in the case
> for the manually triggered run that uses the regular execution_date above)*
> triggered_date next_ds1 + start_date
> \/ \/
> *|-|*
> /\ /\
> \/
> 0 to scheduling_interval (depending on when the next execution
> date is)
> Proposal
> Have next_ds always set to execution_date for manually triggered runs instead
> of the next schedule-interval aligned execution date.
>
> This _might_ break backwards compatibility for some users but it can be
> argued that the current functionality is a bug. If it's really desired we can
> create new aliases that behave logically although I am against this.
>
> prev_ds should probably also be made consistent with this logic.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (AIRFLOW-3506) Fail to query log from elasticsearch
[ https://issues.apache.org/jira/browse/AIRFLOW-3506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng closed AIRFLOW-3506. - Resolution: Fixed > Fail to query log from elasticsearch > > > Key: AIRFLOW-3506 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3506 > Project: Apache Airflow > Issue Type: Bug > Components: logging >Reporter: Ping Zhang >Assignee: Ping Zhang >Priority: Major > Labels: easyfix, newbie > > We have noticed that the elasticsearch_logging_backend use match to query the > log_id from elasticsearch, which ends up getting more results from > elasticsearch. It should use match_phrase to query the log_id as a phrase. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3166) RBAC DAG Level Access Control - Admin can't see DAGS on the UI
[ https://issues.apache.org/jira/browse/AIRFLOW-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16722053#comment-16722053 ] Tao Feng commented on AIRFLOW-3166: --- hey [~sid88in] , could you give a try with the latest master branch? I am able to see all dags with admin while only a specific dag with a given predefined role. > RBAC DAG Level Access Control - Admin can't see DAGS on the UI > --- > > Key: AIRFLOW-3166 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3166 > Project: Apache Airflow > Issue Type: Bug >Reporter: Siddharth >Assignee: Tao Feng >Priority: Major > Attachments: Screen Shot 2018-10-05 at 8.53.17 AM.png > > > This issue is related to code in master branch, > In the image below: I have created 2 users (1. admin, 2. non-admin) and then > created a docker image with master branch code (future Airflow 2.0 version). > Now, I can see DAG level perms (can_dag_edit, can_dag_read) in base perms > which is good. But admin or user can't see any dags even if admin has the > perms to see/edit all dags. > > !Screen Shot 2018-10-05 at 8.53.17 AM.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (AIRFLOW-3520) RBAC UI seems to have bug in master branch
[ https://issues.apache.org/jira/browse/AIRFLOW-3520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng closed AIRFLOW-3520. - Resolution: Not A Problem > RBAC UI seems to have bug in master branch > -- > > Key: AIRFLOW-3520 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3520 > Project: Apache Airflow > Issue Type: Bug >Reporter: Tao Feng >Priority: Major > Attachments: Screen Shot 2018-12-14 at 10.58.07 PM.png > > > !Screen Shot 2018-12-14 at 10.58.07 PM.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3520) RBAC UI seems to have bug in master branch
[ https://issues.apache.org/jira/browse/AIRFLOW-3520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16722047#comment-16722047 ] Tao Feng commented on AIRFLOW-3520: --- after rerun npm install, doesn't seem to be the case. > RBAC UI seems to have bug in master branch > -- > > Key: AIRFLOW-3520 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3520 > Project: Apache Airflow > Issue Type: Bug >Reporter: Tao Feng >Priority: Major > Attachments: Screen Shot 2018-12-14 at 10.58.07 PM.png > > > !Screen Shot 2018-12-14 at 10.58.07 PM.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3520) RBAC UI seems to have bug in master branch
Tao Feng created AIRFLOW-3520: - Summary: RBAC UI seems to have bug in master branch Key: AIRFLOW-3520 URL: https://issues.apache.org/jira/browse/AIRFLOW-3520 Project: Apache Airflow Issue Type: Bug Reporter: Tao Feng Attachments: Screen Shot 2018-12-14 at 10.58.07 PM.png !Screen Shot 2018-12-14 at 10.58.07 PM.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (AIRFLOW-3452) Cannot view dags at /home page
[
https://issues.apache.org/jira/browse/AIRFLOW-3452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Feng closed AIRFLOW-3452.
-
Resolution: Fixed
> Cannot view dags at /home page
> --
>
> Key: AIRFLOW-3452
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3452
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Jinhui Zhang
>Priority: Blocker
>
> I checked out the latest master branch(commit
> {{[9dce1f0|https://github.com/apache/incubator-airflow/commit/9dce1f0740f69af0ee86709a1a34a002b245aa3e]}})
> and restarted my Airflow webserver. But I cannot view any dag at the home
> page. I inspected the frontend code and found there's a
> {{style="display:none;"}} on the \{{main-content}}, and the source code says
> so at
> [https://github.com/apache/incubator-airflow/blob/master/airflow/www_rbac/templates/airflow/dags.html#L31]
> . Is this a known issue? How should I fix it?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (AIRFLOW-1919) Add option to query for DAG runs given a DAG ID
[
https://issues.apache.org/jira/browse/AIRFLOW-1919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Feng closed AIRFLOW-1919.
-
> Add option to query for DAG runs given a DAG ID
> ---
>
> Key: AIRFLOW-1919
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1919
> Project: Apache Airflow
> Issue Type: Improvement
> Components: cli
>Affects Versions: 1.8.0
>Reporter: Steen Manniche
>Assignee: Tao Feng
>Priority: Trivial
> Fix For: 2.0.0
>
>
> Having a way to list all DAG runs for a given DAG identifier would be useful
> when trying to get a programmatic overview of running DAGs. Something along
> the lines of
> {code}
> airflow list_runs $DAG_ID
> {code}
> Which would return the running DAGs for {{$DAG_ID}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1919) Add option to query for DAG runs given a DAG ID
[
https://issues.apache.org/jira/browse/AIRFLOW-1919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16720421#comment-16720421
]
Tao Feng commented on AIRFLOW-1919:
---
[~villasv] , the code has been checked in.
> Add option to query for DAG runs given a DAG ID
> ---
>
> Key: AIRFLOW-1919
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1919
> Project: Apache Airflow
> Issue Type: Improvement
> Components: cli
>Affects Versions: 1.8.0
>Reporter: Steen Manniche
>Assignee: Tao Feng
>Priority: Trivial
> Fix For: 2.0.0
>
>
> Having a way to list all DAG runs for a given DAG identifier would be useful
> when trying to get a programmatic overview of running DAGs. Something along
> the lines of
> {code}
> airflow list_runs $DAG_ID
> {code}
> Which would return the running DAGs for {{$DAG_ID}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3507) Fix Airflow k8s CI
[ https://issues.apache.org/jira/browse/AIRFLOW-3507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng updated AIRFLOW-3507: -- Summary: Fix Airflow k8s CI (was: Fix k8s ci) > Fix Airflow k8s CI > -- > > Key: AIRFLOW-3507 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3507 > Project: Apache Airflow > Issue Type: Bug >Reporter: Tao Feng >Assignee: Tao Feng >Priority: Major > > The k8s ci is failed with two > tests([https://travis-ci.org/apache/incubator-airflow/jobs/467343912).] Fix > the tests to unblock airflow k8s ci. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-3507) Fix Airflow k8s CI
[ https://issues.apache.org/jira/browse/AIRFLOW-3507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng reassigned AIRFLOW-3507: - Assignee: (was: Tao Feng) > Fix Airflow k8s CI > -- > > Key: AIRFLOW-3507 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3507 > Project: Apache Airflow > Issue Type: Bug >Reporter: Tao Feng >Priority: Major > > The k8s ci is failed with two > tests([https://travis-ci.org/apache/incubator-airflow/jobs/467343912).] Fix > the tests to unblock airflow k8s ci. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3507) Fix k8s ci
Tao Feng created AIRFLOW-3507: - Summary: Fix k8s ci Key: AIRFLOW-3507 URL: https://issues.apache.org/jira/browse/AIRFLOW-3507 Project: Apache Airflow Issue Type: Bug Reporter: Tao Feng Assignee: Tao Feng The k8s ci is failed with two tests([https://travis-ci.org/apache/incubator-airflow/jobs/467343912).] Fix the tests to unblock airflow k8s ci. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Issue Comment Deleted] (AIRFLOW-3223) RBAC with GitHub Authentication
[ https://issues.apache.org/jira/browse/AIRFLOW-3223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng updated AIRFLOW-3223: -- Comment: was deleted (was: Would it be suitable to fix / add the feature in flask-appbuilder?) > RBAC with GitHub Authentication > > > Key: AIRFLOW-3223 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3223 > Project: Apache Airflow > Issue Type: Wish > Components: authentication >Affects Versions: 1.10.0 >Reporter: Vikram Fugro >Assignee: Sai Phanindhra >Priority: Major > > With airflow 1.10 released having RBAC support, I was wondering how I do > configure GitHub Auth with airflow's RBAC. In which case, I believe we don't > have to create any users using airflow. Are there any notes on this? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3223) RBAC with GitHub Authentication
[ https://issues.apache.org/jira/browse/AIRFLOW-3223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16719757#comment-16719757 ] Tao Feng commented on AIRFLOW-3223: --- Would it be suitable to fix / add the feature in flask-appbuilder? > RBAC with GitHub Authentication > > > Key: AIRFLOW-3223 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3223 > Project: Apache Airflow > Issue Type: Wish > Components: authentication >Affects Versions: 1.10.0 >Reporter: Vikram Fugro >Assignee: Sai Phanindhra >Priority: Major > > With airflow 1.10 released having RBAC support, I was wondering how I do > configure GitHub Auth with airflow's RBAC. In which case, I believe we don't > have to create any users using airflow. Are there any notes on this? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (AIRFLOW-2770) kubernetes: add support for dag folder in the docker image
[ https://issues.apache.org/jira/browse/AIRFLOW-2770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng closed AIRFLOW-2770. - Resolution: Fixed > kubernetes: add support for dag folder in the docker image > -- > > Key: AIRFLOW-2770 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2770 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Rurui Ye >Assignee: Rurui Ye >Priority: Critical > > currently the kube executor need to provider dag_volume_chain or git repo in > the config file, but if the user has build dag into their docker image, they > doesn't need to provider these two options, and they can manager their dag > version by manager the docker image version. > So I suppose we can add the a new configuration as > kube.config.dag_folder_path along with dag_volume_chain and git repo. with > this config, we can run the worker just from the dags in docker image. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (AIRFLOW-3502) Add config option to control celery pool used
[ https://issues.apache.org/jira/browse/AIRFLOW-3502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng closed AIRFLOW-3502. - Resolution: Fixed Assignee: Gabriel Silk > Add config option to control celery pool used > - > > Key: AIRFLOW-3502 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3502 > Project: Apache Airflow > Issue Type: Improvement > Components: celery >Reporter: Gabriel Silk >Assignee: Gabriel Silk >Priority: Minor > > This adds a config option for the "pool" to allow uses to specify "prefork" > vs "solo" etc. This is particularly useful with infrastructures that don't > play nicely with celery's default prefork multi-processing model. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (AIRFLOW-2546) Set logging level correctly to avoid no section error
[ https://issues.apache.org/jira/browse/AIRFLOW-2546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng closed AIRFLOW-2546. - Resolution: Fixed > Set logging level correctly to avoid no section error > - > > Key: AIRFLOW-2546 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2546 > Project: Apache Airflow > Issue Type: Bug >Reporter: Tao Feng >Assignee: Tao Feng >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-390) [AIRFLOW-Don't load example dags by default]
[ https://issues.apache.org/jira/browse/AIRFLOW-390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16689831#comment-16689831 ] Tao Feng commented on AIRFLOW-390: -- We discuss this previously. The conclusion is to keep load_example as default behaviour for n00b users. We should close this ticket. > [AIRFLOW-Don't load example dags by default] > > > Key: AIRFLOW-390 > URL: https://issues.apache.org/jira/browse/AIRFLOW-390 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Sunny Sun >Priority: Trivial > Labels: easyfix > Original Estimate: 24h > Remaining Estimate: 24h > > Load examples should by default be set to False, so they are not > automatically deployed into production environments. This is especially heavy > because the twitter example dag requires Hive, which users may or may not use > in their own deployments. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3153) send dag last_run to statsd
Tao Feng created AIRFLOW-3153: - Summary: send dag last_run to statsd Key: AIRFLOW-3153 URL: https://issues.apache.org/jira/browse/AIRFLOW-3153 Project: Apache Airflow Issue Type: Bug Reporter: Tao Feng Assignee: Tao Feng -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-3004) Add configuration option to disable schedules
[ https://issues.apache.org/jira/browse/AIRFLOW-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng reassigned AIRFLOW-3004: - Assignee: Tao Feng (was: Jacob Greenfield) > Add configuration option to disable schedules > - > > Key: AIRFLOW-3004 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3004 > Project: Apache Airflow > Issue Type: Improvement > Components: configuration, scheduler >Reporter: Jacob Greenfield >Assignee: Tao Feng >Priority: Minor > > We have a particular use case where we'd like there to be a configuration > option that controls the scheduler and prevents use of the cron schedules > globally for all DAGs, while still allowing manual submission (trigger_dag) > and task instances being scheduled. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3004) Add configuration option to disable schedules
[ https://issues.apache.org/jira/browse/AIRFLOW-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16637562#comment-16637562 ] Tao Feng commented on AIRFLOW-3004: --- PR merged. Please close the ticket. > Add configuration option to disable schedules > - > > Key: AIRFLOW-3004 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3004 > Project: Apache Airflow > Issue Type: Improvement > Components: configuration, scheduler >Reporter: Jacob Greenfield >Assignee: Jacob Greenfield >Priority: Minor > > We have a particular use case where we'd like there to be a configuration > option that controls the scheduler and prevents use of the cron schedules > globally for all DAGs, while still allowing manual submission (trigger_dag) > and task instances being scheduled. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-3004) Add configuration option to disable schedules
[ https://issues.apache.org/jira/browse/AIRFLOW-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng reassigned AIRFLOW-3004: - Assignee: Jacob Greenfield (was: Tao Feng) > Add configuration option to disable schedules > - > > Key: AIRFLOW-3004 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3004 > Project: Apache Airflow > Issue Type: Improvement > Components: configuration, scheduler >Reporter: Jacob Greenfield >Assignee: Jacob Greenfield >Priority: Minor > > We have a particular use case where we'd like there to be a configuration > option that controls the scheduler and prevents use of the cron schedules > globally for all DAGs, while still allowing manual submission (trigger_dag) > and task instances being scheduled. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3111) Confusing comments and instructions for log templates in UPDATING.md and default_airflow.cfg
[
https://issues.apache.org/jira/browse/AIRFLOW-3111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16628014#comment-16628014
]
Tao Feng commented on AIRFLOW-3111:
---
I don't have permission to close the ticket. Could you close the ticket given
it is merged?
> Confusing comments and instructions for log templates in UPDATING.md and
> default_airflow.cfg
>
>
> Key: AIRFLOW-3111
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3111
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Eric Chang
>Assignee: Eric Chang
>Priority: Minor
>
> The new 1.10 release includes instructions on updating the *airflow.cfg*
> options core.log_filename_template and
> elasticsearch.elasticsearch_log_id_template that are technically incorrect:
>
> {code:java}
> // UPDATING.md
> ...the following defaults need to be added.
> [core]
> log_filename_template = ti.dag_id / ti.task_id / ts
> / try_number .log
> [elasticsearch]
> elasticsearch_log_id_template =
> {{dag_id}}-{{task_id}}-{{execution_date}}-{{try_number}}{code}
>
> Inserting the above options into an existing *airflow.cfg* will result in a
> jinja parse failure because `` is not a valid template string. The reason
> the extra braces are necessary is because the default *airflow.cfg* copied to
> AIRFLOW_HOME is a rendered version of *default_airflow.cfg* (rendered by
> *airflow.configuration.parameterized_config*).
> The confusion is compounded by several comments copied over to *airflow.cfg*
> stating that
>
> {code:java}
> # we need to escape the curly braces by adding an additional curly brace
> {code}
> But that's is only true for *default_airflow.cfg* and doesn't apply to the
> actual *airflow.cfg*.
> I'll submit a PR that corrects this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2983) Add prev_ds_nodash and next_ds_nodash macro
[
https://issues.apache.org/jira/browse/AIRFLOW-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Feng updated AIRFLOW-2983:
--
Description:
Similar to [#3418|https://github.com/apache/incubator-airflow/pull/3418], but
introduce two nodash macros:
{\{ prev_ds_nodash }}: the previous execution date as \{{ MMDD }}
{\{ next_ds_nodash }}: the next execution date as \{{ MMDD }}
We introduced prev_ds and next_ds in
[#3418|https://github.com/apache/incubator-airflow/pull/3418] which is
beneficial for backfill. By using these two macros, we could override the
schedule_internal from daily to certain days(e.g 7 days) in backfill.
> Add prev_ds_nodash and next_ds_nodash macro
>
>
> Key: AIRFLOW-2983
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2983
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Tao Feng
>Assignee: Tao Feng
>Priority: Minor
>
> Similar to [#3418|https://github.com/apache/incubator-airflow/pull/3418], but
> introduce two nodash macros:
> {\{ prev_ds_nodash }}: the previous execution date as \{{ MMDD }}
> {\{ next_ds_nodash }}: the next execution date as \{{ MMDD }}
> We introduced prev_ds and next_ds in
> [#3418|https://github.com/apache/incubator-airflow/pull/3418] which is
> beneficial for backfill. By using these two macros, we could override the
> schedule_internal from daily to certain days(e.g 7 days) in backfill.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-2983) Add prev_ds_nodash and next_ds_nodash macro
Tao Feng created AIRFLOW-2983: - Summary: Add prev_ds_nodash and next_ds_nodash macro Key: AIRFLOW-2983 URL: https://issues.apache.org/jira/browse/AIRFLOW-2983 Project: Apache Airflow Issue Type: Bug Reporter: Tao Feng Assignee: Tao Feng -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2909) Remove deprecated airflow.operators.sensors module
[ https://issues.apache.org/jira/browse/AIRFLOW-2909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589300#comment-16589300 ] Tao Feng commented on AIRFLOW-2909: --- It seems that I don't have the permission to close the ticket. [~tedmiston] , could you close this one given the pr is checked in? > Remove deprecated airflow.operators.sensors module > -- > > Key: AIRFLOW-2909 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2909 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Affects Versions: 2.0.0 >Reporter: Taylor Edmiston >Assignee: Taylor Edmiston >Priority: Minor > > This PR removes the now deprecated for 2.0 airflow.operators.sensors module > and updates all references to its sensors to use the new > airflow.sensors.foo_sensor layout. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2866) Missing CSRF Token Error on Web UI Create/Update Operations
[
https://issues.apache.org/jira/browse/AIRFLOW-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16576793#comment-16576793
]
Tao Feng commented on AIRFLOW-2866:
---
Thanks [~craigf] . Let me comment that back to ticket / pr.
> Missing CSRF Token Error on Web UI Create/Update Operations
> ---
>
> Key: AIRFLOW-2866
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2866
> Project: Apache Airflow
> Issue Type: Bug
> Components: webapp
>Reporter: Jasper Kahn
>Priority: Major
>
> Attempting to modify or delete many resources (such as Connections or Users)
> results in a 400 from the webserver:
> {quote}{{Bad Request}}
> {{The CSRF session token is missing.}}{quote}
> Logs report:
> {quote}{{[2018-08-07 18:45:15,771] \{csrf.py:251} INFO - The CSRF session
> token is missing.}}
> {{192.168.9.1 - - [07/Aug/2018:18:45:15 +] "POST
> /admin/connection/delete/ HTTP/1.1" 400 150
> "http://localhost:8081/admin/connection/"; "Mozilla/5.0 (X11; Linux x86_64)
> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.84
> Safari/537.36"}}{quote}
> Chrome dev tools show the CSRF token is present in the request payload.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2866) Missing CSRF Token Error on Web UI Create/Update Operations
[
https://issues.apache.org/jira/browse/AIRFLOW-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16576674#comment-16576674
]
Tao Feng commented on AIRFLOW-2866:
---
[~craigf] , I don't have the setup in my mac. Could you apply this
pr([https://github.com/apache/incubator-airflow/pull/3738)] and see if it
solves the issue?
> Missing CSRF Token Error on Web UI Create/Update Operations
> ---
>
> Key: AIRFLOW-2866
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2866
> Project: Apache Airflow
> Issue Type: Bug
> Components: webapp
>Reporter: Jasper Kahn
>Priority: Major
>
> Attempting to modify or delete many resources (such as Connections or Users)
> results in a 400 from the webserver:
> {quote}{{Bad Request}}
> {{The CSRF session token is missing.}}{quote}
> Logs report:
> {quote}{{[2018-08-07 18:45:15,771] \{csrf.py:251} INFO - The CSRF session
> token is missing.}}
> {{192.168.9.1 - - [07/Aug/2018:18:45:15 +] "POST
> /admin/connection/delete/ HTTP/1.1" 400 150
> "http://localhost:8081/admin/connection/"; "Mozilla/5.0 (X11; Linux x86_64)
> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.84
> Safari/537.36"}}{quote}
> Chrome dev tools show the CSRF token is present in the request payload.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2866) Missing CSRF Token Error on Web UI Create/Update Operations
[
https://issues.apache.org/jira/browse/AIRFLOW-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16576627#comment-16576627
]
Tao Feng commented on AIRFLOW-2866:
---
which version did you run? Could it be related to AIRFLOW-2809?
> Missing CSRF Token Error on Web UI Create/Update Operations
> ---
>
> Key: AIRFLOW-2866
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2866
> Project: Apache Airflow
> Issue Type: Bug
> Components: webapp
>Reporter: Jasper Kahn
>Priority: Major
>
> Attempting to modify or delete many resources (such as Connections or Users)
> results in a 400 from the webserver:
> {quote}{{Bad Request}}
> {{The CSRF session token is missing.}}{quote}
> Logs report:
> {quote}{{[2018-08-07 18:45:15,771] \{csrf.py:251} INFO - The CSRF session
> token is missing.}}
> {{192.168.9.1 - - [07/Aug/2018:18:45:15 +] "POST
> /admin/connection/delete/ HTTP/1.1" 400 150
> "http://localhost:8081/admin/connection/"; "Mozilla/5.0 (X11; Linux x86_64)
> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.84
> Safari/537.36"}}{quote}
> Chrome dev tools show the CSRF token is present in the request payload.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2883) Not search dag owner if owners are missing
[ https://issues.apache.org/jira/browse/AIRFLOW-2883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng updated AIRFLOW-2883: -- Attachment: screen_shot_2018-08-08_at_6.08.29_pm.png > Not search dag owner if owners are missing > -- > > Key: AIRFLOW-2883 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2883 > Project: Apache Airflow > Issue Type: Bug >Reporter: Tao Feng >Assignee: Tao Feng >Priority: Minor > > If the owners field are missing for a dag in airflow metadata, we will > encounter "NoneType" object has no attribute 'lower' when we search. We > should skip searching in this case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2883) Not search dag owner if owners are missing
[ https://issues.apache.org/jira/browse/AIRFLOW-2883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng updated AIRFLOW-2883: -- Attachment: (was: screen_shot_2018-08-08_at_6.08.29_pm.png) > Not search dag owner if owners are missing > -- > > Key: AIRFLOW-2883 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2883 > Project: Apache Airflow > Issue Type: Bug >Reporter: Tao Feng >Assignee: Tao Feng >Priority: Minor > > If the owners field are missing for a dag in airflow metadata, we will > encounter "NoneType" object has no attribute 'lower' when we search. We > should skip searching in this case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2883) Not search dag owner if owners are missing
Tao Feng created AIRFLOW-2883: - Summary: Not search dag owner if owners are missing Key: AIRFLOW-2883 URL: https://issues.apache.org/jira/browse/AIRFLOW-2883 Project: Apache Airflow Issue Type: Bug Reporter: Tao Feng Assignee: Tao Feng If the owners field are missing for a dag in airflow metadata, we will encounter "NoneType" object has no attribute 'lower' when we search. We should skip searching in this case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-2787) Airflow scheduler dies on DAGs with NULL DagRun run_id
[
https://issues.apache.org/jira/browse/AIRFLOW-2787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Feng reassigned AIRFLOW-2787:
-
Assignee: George Leslie-Waksman
> Airflow scheduler dies on DAGs with NULL DagRun run_id
> --
>
> Key: AIRFLOW-2787
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2787
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: George Leslie-Waksman
>Assignee: George Leslie-Waksman
>Priority: Critical
>
> When a DagRun is created with NULL run_id, the scheduler subprocess will
> crash when checking `is_backfill`:
> {noformat}
> Got an exception! Propagating...
> Traceback (most recent call last):
> File "/usr/local/lib/python3.6/site-packages/airflow/jobs.py", line 347, in
> helper
> pickle_dags)
> File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 53,
> in wrapper
> result = func(*args, **kwargs)
> File "/usr/local/lib/python3.6/site-packages/airflow/jobs.py", line 1583,
> in process_file
> self._process_dags(dagbag, dags, ti_keys_to_schedule)
> File "/usr/local/lib/python3.6/site-packages/airflow/jobs.py", line 1175,
> in _process_dags
> self._process_task_instances(dag, tis_out)
> File "/usr/local/lib/python3.6/site-packages/airflow/jobs.py", line 873, in
> _process_task_instances
> if run.is_backfill:
> File "/usr/local/lib/python3.6/site-packages/airflow/models.py", line 4257,
> in is_backfill
> if "backfill" in self.run_id:
> TypeError: argument of type 'NoneType' is not iterable
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-351) Failed to clear downstream tasks
[
https://issues.apache.org/jira/browse/AIRFLOW-351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16551001#comment-16551001
]
Tao Feng commented on AIRFLOW-351:
--
Does the deepcopy issue happen on backfill? One thing could also do is to add
something which fails to deepcopy to shallow_copy_attrs of the
operator(introduced in
[https://github.com/apache/incubator-airflow/pull/3528/files).] We found
protobuf failed to deepcopy in python 2 in our env.
> Failed to clear downstream tasks
>
>
> Key: AIRFLOW-351
> URL: https://issues.apache.org/jira/browse/AIRFLOW-351
> Project: Apache Airflow
> Issue Type: Bug
> Components: models, subdag, webserver
>Affects Versions: Airflow 1.7.1.3
>Reporter: Adinata
>Assignee: Cameron Moberg
>Priority: Major
> Attachments: dag_error.py, error.log, error_on_clear_dag.txt,
> ubuntu-14-packages.log, ubuntu-16-oops.log, ubuntu-16-packages.log
>
>
> {code}
> / ( () ) \___
> /( ( ( ) _)) ) )\
>(( ( )() ) ( ) )
> ((/ ( _( ) ( _) ) ( () ) )
> ( ( ( (_) ((( ) .((_ ) . )_
>( ( )( ( )) ) . ) ( )
> ( ( ( ( ) ( _ ( _) ). ) . ) ) ( )
> ( ( ( ) ( ) ( )) ) _)( ) ) )
> ( ( ( \ ) ((_ ( ) ( ) ) ) ) )) ( )
> ( ( ( ( (_ ( ) ( _) ) ( ) ) )
> ( ( ( ( ( ) (_ ) ) ) _) ) _( ( )
> (( ( )(( _) _) _(_ ( (_ )
>(_((__(_(__(( ( ( | ) ) ) )_))__))_)___)
>((__)\\||lll|l||/// \_))
> ( /(/ ( ) ) )\ )
> (( ( ( | | ) ) )\ )
>( /(| / ( )) ) ) )) )
> ( ( _(|)_) )
> ( ||\(|(|)|/|| )
> (|(||(||))
> ( //|/l|||)|\\ \ )
> (/ / // /|//\\ \ \ \ _)
> ---
> Node: 9889a7c79e9b
> ---
> Traceback (most recent call last):
> File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1817, in
> wsgi_app
> response = self.full_dispatch_request()
> File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1477, in
> full_dispatch_request
> rv = self.handle_user_exception(e)
> File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1381, in
> handle_user_exception
> reraise(exc_type, exc_value, tb)
> File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1475, in
> full_dispatch_request
> rv = self.dispatch_request()
> File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1461, in
> dispatch_request
> return self.view_functions[rule.endpoint](**req.view_args)
> File "/usr/local/lib/python2.7/dist-packages/flask_admin/base.py", line 68,
> in inner
> return self._run_view(f, *args, **kwargs)
> File "/usr/local/lib/python2.7/dist-packages/flask_admin/base.py", line
> 367, in _run_view
> return fn(self, *args, **kwargs)
> File "/usr/local/lib/python2.7/dist-packages/flask_login.py", line 755, in
> decorated_view
> return func(*args, **kwargs)
> File "/usr/local/lib/python2.7/dist-packages/airflow/www/utils.py", line
> 118, in wrapper
> return f(*args, **kwargs)
> File "/usr/local/lib/python2.7/dist-packages/airflow/www/utils.py", line
> 167, in wrapper
> return f(*args, **kwargs)
> File "/usr/local/lib/python2.7/dist-packages/airflow/www/views.py", line
> 1017, in clear
> include_upstream=upstream)
> File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 2870,
> in sub_dag
> dag = copy.deepcopy(self)
> File "/usr/lib/python2.7/copy.py", line 174, in deepcopy
> y = copier(memo)
> File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 2856,
> in __deepcopy__
> setattr(result, k, copy.deepcopy(v, memo))
> File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
> y = copier(x, memo)
> File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
> y[deepcopy(key, memo)] = deepcopy(value, memo)
> File "/usr/lib/python2.7/copy.py", line 174, in deepcopy
> y = copier(memo)
> File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 1974,
> in __deepcopy__
> setattr(result, k, copy.deepcopy(v, memo))
> File "/usr/lib/python2.7/copy.py"
[jira] [Commented] (AIRFLOW-2768) `airflow clear --upstream --only_failed` doesn't clear tasks upstream
[ https://issues.apache.org/jira/browse/AIRFLOW-2768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16548310#comment-16548310 ] Tao Feng commented on AIRFLOW-2768: --- Hey [~7yl4r] , have you tried `airflow backfill --rerun_failed_tasks` ? By doing so, airflow will auto rerun only the failed task during backfill. > `airflow clear --upstream --only_failed` doesn't clear tasks upstream > - > > Key: AIRFLOW-2768 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2768 > Project: Apache Airflow > Issue Type: Bug > Components: cli >Affects Versions: 1.9.0 > Environment: airflow v1.9.0 >Reporter: Tylar Murray >Priority: Major > Labels: cli > > Should `airflow clear --upstream --only_failed mydagname` clear non-failed > tasks upstream of a failed task? > I believe so, yes. If not, then `airflow clear --upstream --only_failed` and > `airflow clear --only_failed` are equivalent. > Upstream tasks are not being cleared. The same is true for `–downstream`. > > My use-case is: I want to clear all tasks all DagRuns that contain a failed > task. In other words: `airflow clear mydagname`, but skip DagRuns that have > completed sucessfully. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2718) Backfill reset_dagrun option allows user to specify certain tasks
Tao Feng created AIRFLOW-2718: - Summary: Backfill reset_dagrun option allows user to specify certain tasks Key: AIRFLOW-2718 URL: https://issues.apache.org/jira/browse/AIRFLOW-2718 Project: Apache Airflow Issue Type: Bug Reporter: Tao Feng Assignee: Tao Feng -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1406) ignore_depends_on_past property ignored by backfill
[
https://issues.apache.org/jira/browse/AIRFLOW-1406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532047#comment-16532047
]
Tao Feng commented on AIRFLOW-1406:
---
have you tried --ignore_first_depends_on_past
([https://github.com/apache/incubator-airflow/blob/master/airflow/bin/cli.py#L1425)]
option ?
> ignore_depends_on_past property ignored by backfill
> ---
>
> Key: AIRFLOW-1406
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1406
> Project: Apache Airflow
> Issue Type: Bug
> Components: backfill
>Affects Versions: Airflow 1.8
>Reporter: Tobias Kaymak
>Assignee: Tao Feng
>Priority: Major
>
> Trying to run the following command does not ignore the first depends_on_past
> dependency of the DAG that has depends_on_past set to true:
> {code}airflow backfill -t search_log_sensor -s 2017-07-10 -e 2017-07-11 -I
> google_pipelines{code}
> neither does this one:
> {code}airflow backfill -t search_log_sensor -s 2017-07-10 -e 2017-07-11
> --ignore_depends_on_past google_pipelines {code}
> both result in:
> {code}
> BackfillJob is deadlocked.Some of the deadlocked tasks were unable to run
> because of "depends_on_past" relationships. Try running the backfill with the
> option "ignore_first_depends_on_past=True" or passing "-I" at the command
> line.
> {code}
> Trying to run with
> {code}
> --ignore_depends_on_past=True
> {code}
> yields:
> {code}
> airflow backfill: error: argument -I/--ignore_first_depends_on_past: ignored
> explicit argument 'True'
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1406) ignore_depends_on_past property ignored by backfill
[
https://issues.apache.org/jira/browse/AIRFLOW-1406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532049#comment-16532049
]
Tao Feng commented on AIRFLOW-1406:
---
Based on the souce code read, I don't think --ignore_depends_on_past is
supported by backfill command.
> ignore_depends_on_past property ignored by backfill
> ---
>
> Key: AIRFLOW-1406
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1406
> Project: Apache Airflow
> Issue Type: Bug
> Components: backfill
>Affects Versions: Airflow 1.8
>Reporter: Tobias Kaymak
>Assignee: Tao Feng
>Priority: Major
>
> Trying to run the following command does not ignore the first depends_on_past
> dependency of the DAG that has depends_on_past set to true:
> {code}airflow backfill -t search_log_sensor -s 2017-07-10 -e 2017-07-11 -I
> google_pipelines{code}
> neither does this one:
> {code}airflow backfill -t search_log_sensor -s 2017-07-10 -e 2017-07-11
> --ignore_depends_on_past google_pipelines {code}
> both result in:
> {code}
> BackfillJob is deadlocked.Some of the deadlocked tasks were unable to run
> because of "depends_on_past" relationships. Try running the backfill with the
> option "ignore_first_depends_on_past=True" or passing "-I" at the command
> line.
> {code}
> Trying to run with
> {code}
> --ignore_depends_on_past=True
> {code}
> yields:
> {code}
> airflow backfill: error: argument -I/--ignore_first_depends_on_past: ignored
> explicit argument 'True'
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-1406) ignore_depends_on_past property ignored by backfill
[
https://issues.apache.org/jira/browse/AIRFLOW-1406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Feng reassigned AIRFLOW-1406:
-
Assignee: Tao Feng
> ignore_depends_on_past property ignored by backfill
> ---
>
> Key: AIRFLOW-1406
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1406
> Project: Apache Airflow
> Issue Type: Bug
> Components: backfill
>Affects Versions: Airflow 1.8
>Reporter: Tobias Kaymak
>Assignee: Tao Feng
>Priority: Major
>
> Trying to run the following command does not ignore the first depends_on_past
> dependency of the DAG that has depends_on_past set to true:
> {code}airflow backfill -t search_log_sensor -s 2017-07-10 -e 2017-07-11 -I
> google_pipelines{code}
> neither does this one:
> {code}airflow backfill -t search_log_sensor -s 2017-07-10 -e 2017-07-11
> --ignore_depends_on_past google_pipelines {code}
> both result in:
> {code}
> BackfillJob is deadlocked.Some of the deadlocked tasks were unable to run
> because of "depends_on_past" relationships. Try running the backfill with the
> option "ignore_first_depends_on_past=True" or passing "-I" at the command
> line.
> {code}
> Trying to run with
> {code}
> --ignore_depends_on_past=True
> {code}
> yields:
> {code}
> airflow backfill: error: argument -I/--ignore_first_depends_on_past: ignored
> explicit argument 'True'
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-2701) Clean up dangling backfill dagrun
Tao Feng created AIRFLOW-2701: - Summary: Clean up dangling backfill dagrun Key: AIRFLOW-2701 URL: https://issues.apache.org/jira/browse/AIRFLOW-2701 Project: Apache Airflow Issue Type: Bug Reporter: Tao Feng Assignee: Tao Feng When user tries to backfill and hit ctrol+9, the backfill dagrun will stay as running state. We should set it to failed if it has unfinished tasks. In our production, we see lots of these dangling backfill dagrun which will cause as one active dagrun in the next backfill. This may prevent user from backfilling if the max_active_run is reached. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2695) Support permission of groups of dag with role
[ https://issues.apache.org/jira/browse/AIRFLOW-2695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng updated AIRFLOW-2695: -- Description: Follow up pr after airflow-2267, this will be done either by enhancing the logic in FAB or by providing a cli command. The approach is still TBD. (was: Follow up pr after airflow-2267) > Support permission of groups of dag with role > - > > Key: AIRFLOW-2695 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2695 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Tao Feng >Assignee: Tao Feng >Priority: Major > > Follow up pr after airflow-2267, this will be done either by enhancing the > logic in FAB or by providing a cli command. The approach is still TBD. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2695) Support assign groups of dag permission to a role
[ https://issues.apache.org/jira/browse/AIRFLOW-2695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng updated AIRFLOW-2695: -- Summary: Support assign groups of dag permission to a role (was: Support permission of groups of dag with role) > Support assign groups of dag permission to a role > - > > Key: AIRFLOW-2695 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2695 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Tao Feng >Assignee: Tao Feng >Priority: Major > > Follow up pr after airflow-2267, this will be done either by enhancing the > logic in FAB or by providing a cli command. The approach is still TBD. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2694) Allow parsing access control in DAG file
[
https://issues.apache.org/jira/browse/AIRFLOW-2694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Feng updated AIRFLOW-2694:
--
Issue Type: New Feature (was: Bug)
> Allow parsing access control in DAG file
>
>
> Key: AIRFLOW-2694
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2694
> Project: Apache Airflow
> Issue Type: New Feature
>Reporter: Tao Feng
>Assignee: Tao Feng
>Priority: Major
>
> After AIRFLOW-2267 is checkedin, modify DAG parsing logic to allow user
> passing permission logic through DAG file like the following example:
>
> dag = DAG('mydag', access_control=\{`role`: [ 'custom_role1'], `perm`:
> `can_dag_read` }, ...)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2693) Implement sync_perm model view endpoint
[ https://issues.apache.org/jira/browse/AIRFLOW-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng updated AIRFLOW-2693: -- Issue Type: New Feature (was: Bug) > Implement sync_perm model view endpoint > --- > > Key: AIRFLOW-2693 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2693 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Tao Feng >Assignee: Tao Feng >Priority: Major > > After AIRFLOW-2267 is checked-in, implement sync_perm model view endpoint > which allows airflow scheduler to call upon to sync permission with roles -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2695) Support permission of groups of dag with role
Tao Feng created AIRFLOW-2695: - Summary: Support permission of groups of dag with role Key: AIRFLOW-2695 URL: https://issues.apache.org/jira/browse/AIRFLOW-2695 Project: Apache Airflow Issue Type: New Feature Reporter: Tao Feng Assignee: Tao Feng Follow up pr after airflow-2267 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2694) Allow parsing access control in DAG file
Tao Feng created AIRFLOW-2694:
-
Summary: Allow parsing access control in DAG file
Key: AIRFLOW-2694
URL: https://issues.apache.org/jira/browse/AIRFLOW-2694
Project: Apache Airflow
Issue Type: Bug
Reporter: Tao Feng
Assignee: Tao Feng
After AIRFLOW-2267 is checkedin, modify DAG parsing logic to allow user passing
permission logic through DAG file like the following example:
dag = DAG('mydag', access_control=\{`role`: [ 'custom_role1'], `perm`:
`can_dag_read` }, ...)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-2693) Implement sync_perm model view endpoint
Tao Feng created AIRFLOW-2693: - Summary: Implement sync_perm model view endpoint Key: AIRFLOW-2693 URL: https://issues.apache.org/jira/browse/AIRFLOW-2693 Project: Apache Airflow Issue Type: Bug Reporter: Tao Feng Assignee: Tao Feng After AIRFLOW-2267 is checked-in, implement sync_perm model view endpoint which allows airflow scheduler to call upon to sync permission with roles -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2685) Excessive logging about missing config sections
[ https://issues.apache.org/jira/browse/AIRFLOW-2685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525482#comment-16525482 ] Tao Feng commented on AIRFLOW-2685: --- I have a pr to fix(https://github.com/apache/incubator-airflow/pull/3443). > Excessive logging about missing config sections > --- > > Key: AIRFLOW-2685 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2685 > Project: Apache Airflow > Issue Type: Bug > Components: worker >Affects Versions: 1.10 >Reporter: James Meickle >Priority: Minor > Attachments: log.txt > > > Log attached. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2678) Fix db scheme unit test to remove checking fab models
Tao Feng created AIRFLOW-2678: - Summary: Fix db scheme unit test to remove checking fab models Key: AIRFLOW-2678 URL: https://issues.apache.org/jira/browse/AIRFLOW-2678 Project: Apache Airflow Issue Type: Bug Reporter: Tao Feng Assignee: Tao Feng Currently airflow doesn't have FAB models as well migration script for the models. We should ignore checking those models in the unit test. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (AIRFLOW-2621) Remove unnecessary return statement in TaskConcurrencyDep
[ https://issues.apache.org/jira/browse/AIRFLOW-2621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng closed AIRFLOW-2621. - Resolution: Won't Fix > Remove unnecessary return statement in TaskConcurrencyDep > - > > Key: AIRFLOW-2621 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2621 > Project: Apache Airflow > Issue Type: Bug >Reporter: Tao Feng >Assignee: Tao Feng >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (AIRFLOW-2652) implement / enhance pythonOperator deepcopy
[ https://issues.apache.org/jira/browse/AIRFLOW-2652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng closed AIRFLOW-2652. - Resolution: Fixed > implement / enhance pythonOperator deepcopy > --- > > Key: AIRFLOW-2652 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2652 > Project: Apache Airflow > Issue Type: Bug >Reporter: Tao Feng >Assignee: Tao Feng >Priority: Major > > Currently if we backfill an airflow task which is pythonOperator, when > airflow does a deepcopy of callable from the operator we may encounter > unexpected issue. We should skip the callable deepcopy as it will not get > mutated. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2652) implement / enhance pythonOperator deepcopy
Tao Feng created AIRFLOW-2652: - Summary: implement / enhance pythonOperator deepcopy Key: AIRFLOW-2652 URL: https://issues.apache.org/jira/browse/AIRFLOW-2652 Project: Apache Airflow Issue Type: Bug Reporter: Tao Feng Assignee: Tao Feng Currently if we backfill an airflow task which is pythonOperator, when airflow does a deepcopy of callable from the operator we may encounter unexpected issue. We should skip the callable deepcopy as it will not get mutated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2538) update faq doc on how to reduce airflow scheduler latency
[ https://issues.apache.org/jira/browse/AIRFLOW-2538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-2538. --- Resolution: Fixed > update faq doc on how to reduce airflow scheduler latency > - > > Key: AIRFLOW-2538 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2538 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Tao Feng >Assignee: Tao Feng >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-1919) Add option to query for DAG runs given a DAG ID
[
https://issues.apache.org/jira/browse/AIRFLOW-1919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Feng reassigned AIRFLOW-1919:
-
Assignee: Tao Feng
> Add option to query for DAG runs given a DAG ID
> ---
>
> Key: AIRFLOW-1919
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1919
> Project: Apache Airflow
> Issue Type: Improvement
> Components: cli
>Affects Versions: Airflow 1.8
>Reporter: Steen Manniche
>Assignee: Tao Feng
>Priority: Trivial
>
> Having a way to list all DAG runs for a given DAG identifier would be useful
> when trying to get a programmatic overview of running DAGs. Something along
> the lines of
> {code}
> airflow list_runs $DAG_ID
> {code}
> Which would return the running DAGs for {{$DAG_ID}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-2621) Remove unnecessary return statement in TaskConcurrencyDep
Tao Feng created AIRFLOW-2621: - Summary: Remove unnecessary return statement in TaskConcurrencyDep Key: AIRFLOW-2621 URL: https://issues.apache.org/jira/browse/AIRFLOW-2621 Project: Apache Airflow Issue Type: Bug Reporter: Tao Feng Assignee: Tao Feng -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1104) Concurrency check in scheduler should count queued tasks as well as running
[ https://issues.apache.org/jira/browse/AIRFLOW-1104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16510224#comment-16510224 ] Tao Feng commented on AIRFLOW-1104: --- hey [~dfowler] , looking at the history of the change. I think the change is easy, but it is hard to ignore or reproduce the issue(task instance could associated with a fail dag run when origin pr is created). I think it would be good if [~saguziel] could comment in current state of art whether this is still an issue. > Concurrency check in scheduler should count queued tasks as well as running > --- > > Key: AIRFLOW-1104 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1104 > Project: Apache Airflow > Issue Type: Bug > Environment: see https://github.com/apache/incubator-airflow/pull/2221 > "Tasks with the QUEUED state should also be counted below, but for now we > cannot count them. This is because there is no guarantee that queued tasks in > failed dagruns will or will not eventually run and queued tasks that will > never run will consume slots and can stall a DAG. Once we can guarantee that > all queued tasks in failed dagruns will never run (e.g. make sure that all > running/newly queued TIs have running dagruns), then we can include QUEUED > tasks here, with the constraint that they are in running dagruns." >Reporter: Alex Guziel >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1104) Concurrency check in scheduler should count queued tasks as well as running
[ https://issues.apache.org/jira/browse/AIRFLOW-1104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16509004#comment-16509004 ] Tao Feng commented on AIRFLOW-1104: --- let me take a look later today or tomorrow. > Concurrency check in scheduler should count queued tasks as well as running > --- > > Key: AIRFLOW-1104 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1104 > Project: Apache Airflow > Issue Type: Bug > Environment: see https://github.com/apache/incubator-airflow/pull/2221 > "Tasks with the QUEUED state should also be counted below, but for now we > cannot count them. This is because there is no guarantee that queued tasks in > failed dagruns will or will not eventually run and queued tasks that will > never run will consume slots and can stall a DAG. Once we can guarantee that > all queued tasks in failed dagruns will never run (e.g. make sure that all > running/newly queued TIs have running dagruns), then we can include QUEUED > tasks here, with the constraint that they are in running dagruns." >Reporter: Alex Guziel >Assignee: Tao Feng >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2583) Enhance ExternalTaskSensor
Tao Feng created AIRFLOW-2583: - Summary: Enhance ExternalTaskSensor Key: AIRFLOW-2583 URL: https://issues.apache.org/jira/browse/AIRFLOW-2583 Project: Apache Airflow Issue Type: Bug Reporter: Tao Feng Assignee: Tao Feng Details will follow in the pr. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2566) Change backfill to rerun failed tasks instead of throwing exception
Tao Feng created AIRFLOW-2566: - Summary: Change backfill to rerun failed tasks instead of throwing exception Key: AIRFLOW-2566 URL: https://issues.apache.org/jira/browse/AIRFLOW-2566 Project: Apache Airflow Issue Type: Bug Reporter: Tao Feng Assignee: Tao Feng -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2558) Clear TASK/DAG is clearing all executions
[
https://issues.apache.org/jira/browse/AIRFLOW-2558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16501313#comment-16501313
]
Tao Feng commented on AIRFLOW-2558:
---
pr is created.
> Clear TASK/DAG is clearing all executions
> -
>
> Key: AIRFLOW-2558
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2558
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: Airflow 2.0
>Reporter: Marcos Bernardelli
>Assignee: Tao Feng
>Priority: Major
>
> When I try to clear a DAG/TASK specific execution, the Airflow try to execute
> all the past executions:
> [Animeted
> GIF|https://gist.githubusercontent.com/bern4rdelli/34c1e57acd53c8c67417604202f3e0e6/raw/4bcb3d3c23f2a3bb7f7bfb3e977d935e5bb9f0ee/clear.gif]
> (I failed miserable trying to attache the animated GIF :()
>
> This behavior was changed here:
> [https://github.com/apache/incubator-airflow/pull/3444]
> The old version looks like this:
> {code:python}
> drs = session.query(DagModel).filter_by(dag_id=self.dag_id).all()
> {code}
> Then it's changed to:
> {code:python}
> drs = session.query(DagRun).filter_by(dag_id=self.dag_id).all()
> {code}
> This new query (using DagRun) get all the past executions, even when the
> "Past" button is not checked.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2558) Clear TASK/DAG is clearing all executions
[
https://issues.apache.org/jira/browse/AIRFLOW-2558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16499602#comment-16499602
]
Tao Feng commented on AIRFLOW-2558:
---
Will look into it this week. thanks for reporting.
> Clear TASK/DAG is clearing all executions
> -
>
> Key: AIRFLOW-2558
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2558
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: Airflow 2.0
>Reporter: Marcos Bernardelli
>Assignee: Tao Feng
>Priority: Major
>
> When I try to clear a DAG/TASK specific execution, the Airflow try to execute
> all the past executions:
> [Animeted
> GIF|https://gist.githubusercontent.com/bern4rdelli/34c1e57acd53c8c67417604202f3e0e6/raw/4bcb3d3c23f2a3bb7f7bfb3e977d935e5bb9f0ee/clear.gif]
> (I failed miserable trying to attache the animated GIF :()
>
> This behavior was changed here:
> [https://github.com/apache/incubator-airflow/pull/3444]
> The old version looks like this:
> {code:python}
> drs = session.query(DagModel).filter_by(dag_id=self.dag_id).all()
> {code}
> Then it's changed to:
> {code:python}
> drs = session.query(DagRun).filter_by(dag_id=self.dag_id).all()
> {code}
> This new query (using DagRun) get all the past executions, even when the
> "Past" button is not checked.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-2546) Set logging level correctly to avoid no section error
Tao Feng created AIRFLOW-2546: - Summary: Set logging level correctly to avoid no section error Key: AIRFLOW-2546 URL: https://issues.apache.org/jira/browse/AIRFLOW-2546 Project: Apache Airflow Issue Type: Bug Reporter: Tao Feng Assignee: Tao Feng -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2538) update faq doc on how to reduce airflow scheduler latency
Tao Feng created AIRFLOW-2538: - Summary: update faq doc on how to reduce airflow scheduler latency Key: AIRFLOW-2538 URL: https://issues.apache.org/jira/browse/AIRFLOW-2538 Project: Apache Airflow Issue Type: Improvement Reporter: Tao Feng Assignee: Tao Feng -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (AIRFLOW-2527) Be able to clear Dag Runs + Tasks from command line
[ https://issues.apache.org/jira/browse/AIRFLOW-2527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng closed AIRFLOW-2527. - Resolution: Not A Problem > Be able to clear Dag Runs + Tasks from command line > --- > > Key: AIRFLOW-2527 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2527 > Project: Apache Airflow > Issue Type: Bug >Reporter: Tao Feng >Assignee: Tao Feng >Priority: Major > > Make sure airflow clear could clear both dag runs and tasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (AIRFLOW-2172) Cast template_fields into a list if the operator's template_fields has only one element
[
https://issues.apache.org/jira/browse/AIRFLOW-2172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Feng closed AIRFLOW-2172.
-
Resolution: Won't Fix
> Cast template_fields into a list if the operator's template_fields has only
> one element
> ---
>
> Key: AIRFLOW-2172
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2172
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Tao Feng
>Assignee: Tao Feng
>Priority: Minor
>
> Normally user defined operator is derived from BaseOperator. And a jinjafied
> tuple is created for template_fields. If only one field is provided in the
> operator's template_fields as following for example:
> template_fields = ('hql')
> the resolve_template_files method in BaseOperator actually will throw an
> exception, as the template_fields is now considered as a string instead of a
> list/tuple. In the example, 'h', 'q', 'l' will all consider template field
> attribute.
> The workaround is to cast template_frields into a list when its actual type
> is a string. Not sure it is a real bug, but it takes me some time to figure
> out this issue...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-1823) API get_task_info is incompatible with manual runs created by UI
[ https://issues.apache.org/jira/browse/AIRFLOW-1823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng reassigned AIRFLOW-1823: - Assignee: Tao Feng > API get_task_info is incompatible with manual runs created by UI > > > Key: AIRFLOW-1823 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1823 > Project: Apache Airflow > Issue Type: Bug > Components: api >Affects Versions: Airflow 2.0 > Environment: ubuntu > Airflow 1.9rc02 > commit: > https://github.com/apache/incubator-airflow/blob/master/airflow/www/api/experimental/endpoints.py#L126 >Reporter: Jeremy Lewi >Assignee: Tao Feng >Priority: Minor > > The API method > [task_instance_info|https://github.com/apache/incubator-airflow/blob/master/airflow/www/api/experimental/endpoints.py#L126] > doesn't work with manual runs created by the UI. > The UI creates dag runs with ids with sub second precision in the name. An > example of a run created by the UI is > 2017-11-16T20:23:32.045330 > The endpoint for > [task_instance_info|https://github.com/apache/incubator-airflow/blob/master/airflow/www/api/experimental/endpoints.py#L126] > however assumes the dag run id is of the form '%Y-%m-%dT%H:%M:%S'. > Runs triggered via the CLI generate run ids with the form expected by the API. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (AIRFLOW-2529) Graph View DAG Run dropdown
[ https://issues.apache.org/jira/browse/AIRFLOW-2529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16492176#comment-16492176 ] Tao Feng edited comment on AIRFLOW-2529 at 5/27/18 10:43 PM: - Not sure if it is related, but we introduce a new config(default_dag_run_display_number) in [https://github.com/apache/incubator-airflow/pull/3279] which controls the numbers of dag-run to load in UI. was (Author: taofeng): Not sure if it is related, but we introduce a new config in [https://github.com/apache/incubator-airflow/pull/3279] which controls the numbers of dag-run to load in UI. > Graph View DAG Run dropdown > --- > > Key: AIRFLOW-2529 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2529 > Project: Apache Airflow > Issue Type: Improvement > Components: webapp >Affects Versions: 1.9.0 >Reporter: Stefan Seelmann >Priority: Major > > The "Graph View" has a dropdown which contains all DAG run IDs. If there are > many (thousands) of DAG runs the page gets barely usable. It takes multiple > seconds to load the page because all DAG runs must be fetched from DB, are > processed, and a long option list is rendered in the browser. It is also not > very useful because in such a long list it is hard to find a particular DAG > run. > A simple fix to address the load time would be to just limit the number of > shown DAG runs. For example only the latest N are shown, N could be > "page_size" from airflow.cfg which is also used in other views. If the DAG > run that should be shown (via query parameters execution_date or run_id) is > not included in the N lastest list it can still be added by a 2nd SQL query. > A more complex change to improve usability would require a different way to > select a DAG run. For example a popup to search for DAG runs with pagination > etc. But such functionality already exits in the /dagrun UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2529) Graph View DAG Run dropdown
[ https://issues.apache.org/jira/browse/AIRFLOW-2529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16492176#comment-16492176 ] Tao Feng commented on AIRFLOW-2529: --- Not sure if it is related, but we introduce a new config in [https://github.com/apache/incubator-airflow/pull/3279] which controls the numbers of dag-run to load in UI. > Graph View DAG Run dropdown > --- > > Key: AIRFLOW-2529 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2529 > Project: Apache Airflow > Issue Type: Improvement > Components: webapp >Affects Versions: 1.9.0 >Reporter: Stefan Seelmann >Priority: Major > > The "Graph View" has a dropdown which contains all DAG run IDs. If there are > many (thousands) of DAG runs the page gets barely usable. It takes multiple > seconds to load the page because all DAG runs must be fetched from DB, are > processed, and a long option list is rendered in the browser. It is also not > very useful because in such a long list it is hard to find a particular DAG > run. > A simple fix to address the load time would be to just limit the number of > shown DAG runs. For example only the latest N are shown, N could be > "page_size" from airflow.cfg which is also used in other views. If the DAG > run that should be shown (via query parameters execution_date or run_id) is > not included in the N lastest list it can still be added by a 2nd SQL query. > A more complex change to improve usability would require a different way to > select a DAG run. For example a popup to search for DAG runs with pagination > etc. But such functionality already exits in the /dagrun UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2527) Be able to clear Dag Runs + Tasks from command line
Tao Feng created AIRFLOW-2527: - Summary: Be able to clear Dag Runs + Tasks from command line Key: AIRFLOW-2527 URL: https://issues.apache.org/jira/browse/AIRFLOW-2527 Project: Apache Airflow Issue Type: Bug Reporter: Tao Feng Assignee: Tao Feng Make sure airflow clear could clear both dag runs and tasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-1022) Subdag can't receive templated fields
[ https://issues.apache.org/jira/browse/AIRFLOW-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng reassigned AIRFLOW-1022: - Assignee: (was: Tao Feng) > Subdag can't receive templated fields > - > > Key: AIRFLOW-1022 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1022 > Project: Apache Airflow > Issue Type: Improvement > Components: subdag >Reporter: Marcos Takahashi >Priority: Major > Labels: easyfix > > Subdag's can't receive any templated fields as the Operator is setted as > tuple() > (https://github.com/apache/incubator-airflow/blob/master/airflow/operators/subdag_operator.py#L24) > instead of any other templated dict like on PythonOperator > (https://github.com/apache/incubator-airflow/blob/master/airflow/operators/python_operator.py#L52). > That makes impossible on getting some important values like execution_date. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-1022) Subdag can't receive templated fields
[ https://issues.apache.org/jira/browse/AIRFLOW-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng reassigned AIRFLOW-1022: - Assignee: Tao Feng > Subdag can't receive templated fields > - > > Key: AIRFLOW-1022 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1022 > Project: Apache Airflow > Issue Type: Improvement > Components: subdag >Reporter: Marcos Takahashi >Assignee: Tao Feng >Priority: Major > Labels: easyfix > > Subdag's can't receive any templated fields as the Operator is setted as > tuple() > (https://github.com/apache/incubator-airflow/blob/master/airflow/operators/subdag_operator.py#L24) > instead of any other templated dict like on PythonOperator > (https://github.com/apache/incubator-airflow/blob/master/airflow/operators/python_operator.py#L52). > That makes impossible on getting some important values like execution_date. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2492) Fix test_dag_catchup_option test
Tao Feng created AIRFLOW-2492: - Summary: Fix test_dag_catchup_option test Key: AIRFLOW-2492 URL: https://issues.apache.org/jira/browse/AIRFLOW-2492 Project: Apache Airflow Issue Type: Bug Reporter: Tao Feng There seems to be some sort of race condition happens to the test which causes the CI fails sometimes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2487) Allow user to set connection type for druid ingestion hook
Tao Feng created AIRFLOW-2487: - Summary: Allow user to set connection type for druid ingestion hook Key: AIRFLOW-2487 URL: https://issues.apache.org/jira/browse/AIRFLOW-2487 Project: Apache Airflow Issue Type: Bug Reporter: Tao Feng Assignee: Tao Feng -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-1737) set_task_instance_state fails because of strptime
[ https://issues.apache.org/jira/browse/AIRFLOW-1737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng reassigned AIRFLOW-1737: - Assignee: Tao Feng > set_task_instance_state fails because of strptime > - > > Key: AIRFLOW-1737 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1737 > Project: Apache Airflow > Issue Type: Bug > Components: webapp >Reporter: Andre Boechat >Assignee: Tao Feng >Priority: Minor > Attachments: Screenshot_2017-10-18_15-58-29.png > > > Context: > * DAG run triggered manually > * Using the web application to change the state of a task > When trying to set the state of a task, an exception is thrown: *ValueError: > unconverted data remains: ..372649* (look at the attached screenshot). > I think the problem comes from the "execution date" created by manually > triggered DAGs, since the date-time includes a fractional part. In my > database, I see scheduled DAGs with execution dates like "10-18T15:00:00", > while manually triggered ones with dates like "09-21T16:36:16.170988". If we > look at the method *set_task_instance_state* in *airflow.www.views*, we see > that the format string used with *strptime* doesn't consider any fractional > part. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2479) Create doc for RBAC UI
Tao Feng created AIRFLOW-2479: - Summary: Create doc for RBAC UI Key: AIRFLOW-2479 URL: https://issues.apache.org/jira/browse/AIRFLOW-2479 Project: Apache Airflow Issue Type: Improvement Reporter: Tao Feng Assignee: Tao Feng -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2478) Improve doc FAQ section
Tao Feng created AIRFLOW-2478: - Summary: Improve doc FAQ section Key: AIRFLOW-2478 URL: https://issues.apache.org/jira/browse/AIRFLOW-2478 Project: Apache Airflow Issue Type: Improvement Reporter: Tao Feng Assignee: Tao Feng -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2477) Improve time units for task duration and landing times charts for RBAC UI
Tao Feng created AIRFLOW-2477: - Summary: Improve time units for task duration and landing times charts for RBAC UI Key: AIRFLOW-2477 URL: https://issues.apache.org/jira/browse/AIRFLOW-2477 Project: Apache Airflow Issue Type: Bug Reporter: Tao Feng Assignee: Tao Feng -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2376) backports.configparser.NoSectionError: No section: u'hive'
[
https://issues.apache.org/jira/browse/AIRFLOW-2376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16471197#comment-16471197
]
Tao Feng commented on AIRFLOW-2376:
---
I reproduce it by running airflow backfill example_http_operator -s 2018-02-01
-e 2018-02-02 -i . I believe this happens to the run command which the
reporter stack trace shows [~sanand]
> backports.configparser.NoSectionError: No section: u'hive'
> --
>
> Key: AIRFLOW-2376
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2376
> Project: Apache Airflow
> Issue Type: Improvement
> Components: configuration, core
>Affects Versions: 1.10.0, 2.0.0
>Reporter: Ruslan Dautkhanov
>Assignee: Tao Feng
>Priority: Major
>
> Airflow master throws
> {noformat}
> [2018-04-25 11:42:42,452] {models.py:1509} INFO - Executing
> on 2018-04-25T10:28:11.172589-06:00
> [2018-04-25 11:42:42,452] {base_task_runner.py:123} INFO - Running: ['bash',
> '-c', u'airflow run DISCOVER-Oracle-Load-Mar2017-v1 start
> 2018-04-25T10:28:11.172589-06:00 --job_id 545 --pool DISCOVER-Prod --raw -sd
> DAGS_FOLDER/discover/discover-ora-load-2.py --cfg_path /tmp/tmpyI6FvX']
> [2018-04-25 11:42:43,241] {base_task_runner.py:106} INFO - Job 545: Subtask
> start
> /opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/psycopg2/__init__.py:144:
> UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in
> order to keep installing from binary please use "pip install psycopg2-binary"
> instead. For details see:
> .
> [2018-04-25 11:42:43,242] {base_task_runner.py:106} INFO - Job 545: Subtask
> start """)
> [2018-04-25 11:42:43,355] {base_task_runner.py:106} INFO - Job 545: Subtask
> start [2018-04-25 11:42:43,353] {__init__.py:50} INFO - Using executor
> LocalExecutor
> [2018-04-25 11:42:43,546] {base_task_runner.py:106} INFO - Job 545: Subtask
> start Traceback (most recent call last):
> [2018-04-25 11:42:43,546] {base_task_runner.py:106} INFO - Job 545: Subtask
> start File "/opt/cloudera/parcels/Anaconda/bin/airflow", line 6, in
> [2018-04-25 11:42:43,546] {base_task_runner.py:106} INFO - Job 545: Subtask
> start exec(compile(open(__file__).read(), __file__, 'exec'))
> [2018-04-25 11:42:43,546] {base_task_runner.py:106} INFO - Job 545: Subtask
> start File
> "/opt/airflow/airflow-20180420/src/apache-airflow/airflow/bin/airflow", line
> 32, in
> [2018-04-25 11:42:43,547] {base_task_runner.py:106} INFO - Job 545: Subtask
> start args.func(args)
> [2018-04-25 11:42:43,547] {base_task_runner.py:106} INFO - Job 545: Subtask
> start File
> "/opt/airflow/airflow-20180420/src/apache-airflow/airflow/utils/cli.py", line
> 77, in wrapper
> [2018-04-25 11:42:43,547] {base_task_runner.py:106} INFO - Job 545: Subtask
> start raise e
> [2018-04-25 11:42:43,547] {base_task_runner.py:106} INFO - Job 545: Subtask
> start backports.configparser.NoSectionError: No section: u'hive'
> [2018-04-25 11:42:47,415] {logging_mixin.py:95} INFO - [2018-04-25
> 11:42:47,415] {jobs.py:2548} INFO - Task exited with return code 1
> {noformat}
> Adding empty "hive" section into airflow.cfg works around this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-2376) backports.configparser.NoSectionError: No section: u'hive'
[
https://issues.apache.org/jira/browse/AIRFLOW-2376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Feng reassigned AIRFLOW-2376:
-
Assignee: Tao Feng
> backports.configparser.NoSectionError: No section: u'hive'
> --
>
> Key: AIRFLOW-2376
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2376
> Project: Apache Airflow
> Issue Type: Improvement
> Components: configuration, core
>Affects Versions: 1.10.0, 2.0.0
>Reporter: Ruslan Dautkhanov
>Assignee: Tao Feng
>Priority: Major
>
> Airflow master throws
> {noformat}
> [2018-04-25 11:42:42,452] {models.py:1509} INFO - Executing
> on 2018-04-25T10:28:11.172589-06:00
> [2018-04-25 11:42:42,452] {base_task_runner.py:123} INFO - Running: ['bash',
> '-c', u'airflow run DISCOVER-Oracle-Load-Mar2017-v1 start
> 2018-04-25T10:28:11.172589-06:00 --job_id 545 --pool DISCOVER-Prod --raw -sd
> DAGS_FOLDER/discover/discover-ora-load-2.py --cfg_path /tmp/tmpyI6FvX']
> [2018-04-25 11:42:43,241] {base_task_runner.py:106} INFO - Job 545: Subtask
> start
> /opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/psycopg2/__init__.py:144:
> UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in
> order to keep installing from binary please use "pip install psycopg2-binary"
> instead. For details see:
> .
> [2018-04-25 11:42:43,242] {base_task_runner.py:106} INFO - Job 545: Subtask
> start """)
> [2018-04-25 11:42:43,355] {base_task_runner.py:106} INFO - Job 545: Subtask
> start [2018-04-25 11:42:43,353] {__init__.py:50} INFO - Using executor
> LocalExecutor
> [2018-04-25 11:42:43,546] {base_task_runner.py:106} INFO - Job 545: Subtask
> start Traceback (most recent call last):
> [2018-04-25 11:42:43,546] {base_task_runner.py:106} INFO - Job 545: Subtask
> start File "/opt/cloudera/parcels/Anaconda/bin/airflow", line 6, in
> [2018-04-25 11:42:43,546] {base_task_runner.py:106} INFO - Job 545: Subtask
> start exec(compile(open(__file__).read(), __file__, 'exec'))
> [2018-04-25 11:42:43,546] {base_task_runner.py:106} INFO - Job 545: Subtask
> start File
> "/opt/airflow/airflow-20180420/src/apache-airflow/airflow/bin/airflow", line
> 32, in
> [2018-04-25 11:42:43,547] {base_task_runner.py:106} INFO - Job 545: Subtask
> start args.func(args)
> [2018-04-25 11:42:43,547] {base_task_runner.py:106} INFO - Job 545: Subtask
> start File
> "/opt/airflow/airflow-20180420/src/apache-airflow/airflow/utils/cli.py", line
> 77, in wrapper
> [2018-04-25 11:42:43,547] {base_task_runner.py:106} INFO - Job 545: Subtask
> start raise e
> [2018-04-25 11:42:43,547] {base_task_runner.py:106} INFO - Job 545: Subtask
> start backports.configparser.NoSectionError: No section: u'hive'
> [2018-04-25 11:42:47,415] {logging_mixin.py:95} INFO - [2018-04-25
> 11:42:47,415] {jobs.py:2548} INFO - Task exited with return code 1
> {noformat}
> Adding empty "hive" section into airflow.cfg works around this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-2429) Make Airflow Flake8 compliant
[ https://issues.apache.org/jira/browse/AIRFLOW-2429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng reassigned AIRFLOW-2429: - Assignee: Tao Feng > Make Airflow Flake8 compliant > -- > > Key: AIRFLOW-2429 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2429 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Fokko Driesprong >Assignee: Tao Feng >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2444) Remove unused option(include_adhoc) in cli backfill command
Tao Feng created AIRFLOW-2444: - Summary: Remove unused option(include_adhoc) in cli backfill command Key: AIRFLOW-2444 URL: https://issues.apache.org/jira/browse/AIRFLOW-2444 Project: Apache Airflow Issue Type: Bug Reporter: Tao Feng Assignee: Tao Feng Include_adhoc is removed in this pr([https://github.com/apache/incubator-airflow/pull/1667/files).] This option currently doesn't do anything. Just remove it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-2407) Undefined names in Python code
[
https://issues.apache.org/jira/browse/AIRFLOW-2407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Feng reassigned AIRFLOW-2407:
-
Assignee: (was: Tao Feng)
> Undefined names in Python code
> --
>
> Key: AIRFLOW-2407
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2407
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: cclauss
>Priority: Minor
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> flake8 testing of https://github.com/apache/incubator-airflow on Python 3.6.3
> $ *flake8 . --count --select=E901,E999,F821,F822,F823 --show-source
> --statistics*
> {noformat}
> ./airflow/contrib/auth/backends/kerberos_auth.py:67:13: F821 undefined name
> 'logging'
> logging.error('Password validation for principal %s failed %s',
> user_principal, e)
> ^
> ./airflow/contrib/hooks/aws_hook.py:75:13: F821 undefined name 'logging'
> logging.warning("Option Error in parsing s3 config file")
> ^
> ./airflow/contrib/operators/datastore_export_operator.py:105:19: F821
> undefined name 'AirflowException'
> raise AirflowException('Operation failed:
> result={}'.format(result))
> ^
> ./airflow/contrib/operators/datastore_import_operator.py:94:19: F821
> undefined name 'AirflowException'
> raise AirflowException('Operation failed:
> result={}'.format(result))
> ^
> ./airflow/contrib/sensors/qubole_sensor.py:62:9: F821 undefined name 'this'
> this.log.info('Poking: %s', self.data)
> ^
> ./airflow/contrib/sensors/qubole_sensor.py:68:13: F821 undefined name
> 'logging'
> logging.exception(e)
> ^
> ./airflow/contrib/sensors/qubole_sensor.py:71:9: F821 undefined name 'this'
> this.log.info('Status of this Poke: %s', status)
> ^
> ./airflow/www/app.py:148:17: F821 undefined name 'reload'
> reload(e)
> ^
> ./tests/operators/hive_operator.py:178:27: F821 undefined name 'cursor_mock'
> __enter__=cursor_mock,
> ^
> ./tests/operators/hive_operator.py:184:27: F821 undefined name 'get_conn_mock'
> __enter__=get_conn_mock,
> ^
> ./tests/operators/test_virtualenv_operator.py:166:19: F821 undefined name
> 'virtualenv_string_args'
> print(virtualenv_string_args)
> ^
> ./tests/operators/test_virtualenv_operator.py:167:16: F821 undefined name
> 'virtualenv_string_args'
> if virtualenv_string_args[0] != virtualenv_string_args[2]:
>^
> ./tests/operators/test_virtualenv_operator.py:167:45: F821 undefined name
> 'virtualenv_string_args'
> if virtualenv_string_args[0] != virtualenv_string_args[2]:
> ^
> 13F821 undefined name 'logging'
> 13
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-2407) Undefined names in Python code
[
https://issues.apache.org/jira/browse/AIRFLOW-2407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Feng reassigned AIRFLOW-2407:
-
Assignee: Tao Feng
> Undefined names in Python code
> --
>
> Key: AIRFLOW-2407
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2407
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: cclauss
>Assignee: Tao Feng
>Priority: Minor
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> flake8 testing of https://github.com/apache/incubator-airflow on Python 3.6.3
> $ *flake8 . --count --select=E901,E999,F821,F822,F823 --show-source
> --statistics*
> {noformat}
> ./airflow/contrib/auth/backends/kerberos_auth.py:67:13: F821 undefined name
> 'logging'
> logging.error('Password validation for principal %s failed %s',
> user_principal, e)
> ^
> ./airflow/contrib/hooks/aws_hook.py:75:13: F821 undefined name 'logging'
> logging.warning("Option Error in parsing s3 config file")
> ^
> ./airflow/contrib/operators/datastore_export_operator.py:105:19: F821
> undefined name 'AirflowException'
> raise AirflowException('Operation failed:
> result={}'.format(result))
> ^
> ./airflow/contrib/operators/datastore_import_operator.py:94:19: F821
> undefined name 'AirflowException'
> raise AirflowException('Operation failed:
> result={}'.format(result))
> ^
> ./airflow/contrib/sensors/qubole_sensor.py:62:9: F821 undefined name 'this'
> this.log.info('Poking: %s', self.data)
> ^
> ./airflow/contrib/sensors/qubole_sensor.py:68:13: F821 undefined name
> 'logging'
> logging.exception(e)
> ^
> ./airflow/contrib/sensors/qubole_sensor.py:71:9: F821 undefined name 'this'
> this.log.info('Status of this Poke: %s', status)
> ^
> ./airflow/www/app.py:148:17: F821 undefined name 'reload'
> reload(e)
> ^
> ./tests/operators/hive_operator.py:178:27: F821 undefined name 'cursor_mock'
> __enter__=cursor_mock,
> ^
> ./tests/operators/hive_operator.py:184:27: F821 undefined name 'get_conn_mock'
> __enter__=get_conn_mock,
> ^
> ./tests/operators/test_virtualenv_operator.py:166:19: F821 undefined name
> 'virtualenv_string_args'
> print(virtualenv_string_args)
> ^
> ./tests/operators/test_virtualenv_operator.py:167:16: F821 undefined name
> 'virtualenv_string_args'
> if virtualenv_string_args[0] != virtualenv_string_args[2]:
>^
> ./tests/operators/test_virtualenv_operator.py:167:45: F821 undefined name
> 'virtualenv_string_args'
> if virtualenv_string_args[0] != virtualenv_string_args[2]:
> ^
> 13F821 undefined name 'logging'
> 13
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-2389) Create a pinot db api hook
Tao Feng created AIRFLOW-2389: - Summary: Create a pinot db api hook Key: AIRFLOW-2389 URL: https://issues.apache.org/jira/browse/AIRFLOW-2389 Project: Apache Airflow Issue Type: Improvement Reporter: Tao Feng Assignee: Tao Feng create a db api hook for pinot(https://github.com/linkedin/pinot) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2371) SubdagOperator ignores pooling
[ https://issues.apache.org/jira/browse/AIRFLOW-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16451482#comment-16451482 ] Tao Feng commented on AIRFLOW-2371: --- Thanks Xiao Zhu. I wonder if some airflow experts could help to comment on how much effort it is to make subdag first class in airflow as it would be super useful. cc @[mistercrunch|https://github.com/mistercrunch] . > SubdagOperator ignores pooling > -- > > Key: AIRFLOW-2371 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2371 > Project: Apache Airflow > Issue Type: Bug > Components: subdag >Affects Versions: Airflow 1.9.0 > Environment: Airflow 1.9.0 >Reporter: Xiao Zhu >Priority: Major > > Basically we need a long term solution for > https://issues.apache.org/jira/browse/AIRFLOW-74 . Using SequentialExecutor > is a workaround, however it slows down execution of subdags. > Looking for a better solution. > I also see that when running subdags, db connection pools are not shared > either (since they are separate processes), which causes db disconnection > issues like https://issues.apache.org/jira/browse/AIRFLOW-2340 on our db > because there can be too many connections. I hope it can be fixed too when > subdag operator works with pooling.. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-74) SubdagOperators can consume all celeryd worker processes
[
https://issues.apache.org/jira/browse/AIRFLOW-74?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16450411#comment-16450411
]
Tao Feng commented on AIRFLOW-74:
-
[~xzhu] , just to be clear, you could still use celeryExectuor etc for your dag
level execution. But this only limits to use sequentialExector as default
within the subdag.
> SubdagOperators can consume all celeryd worker processes
>
>
> Key: AIRFLOW-74
> URL: https://issues.apache.org/jira/browse/AIRFLOW-74
> Project: Apache Airflow
> Issue Type: Bug
> Components: celery
>Affects Versions: Airflow 1.7.1, Airflow 1.7.0, Airflow 1.6.2
> Environment: Airflow 1.7.1rc3 with CeleryExecutor
> 1 webserver
> 1 scheduler
> 2 workers
>Reporter: Steven Yvinec-Kruyk
>Assignee: zgl
>Priority: Major
> Fix For: 1.10.0
>
>
> If the amount of concurrent ```SubdagOperator``` running >= the no. of celery
> worker processes tasks are unable to work. All SDOs come to a complete halt.
> Futhermore performance of a DAG is drastically reduced even before full
> saturation of the workers as less workers are gradually available for actual
> tasks. A workaround for this is to specify ```SequentialExecutor``` be used
> by the ```SubdagOperator```
> ```
> from datetime import timedelta, datetime
> from airflow.models import DAG, Pool
> from airflow.operators import BashOperator, SubDagOperator, DummyOperator
> from airflow.executors import SequentialExecutor
> import airflow
> # -\
> # DEFINE THE POOLS
> # -/
> session = airflow.settings.Session()
> for p in ['test_pool_1', 'test_pool_2', 'test_pool_3']:
> pool = (
> session.query(Pool)
> .filter(Pool.pool == p)
> .first())
> if not pool:
> session.add(Pool(pool=p, slots=8))
> session.commit()
> # -\
> # DEFINE THE DAG
> # -/
> # Define the Dag Name. This must be unique.
> dag_name = 'hanging_subdags_n16_sqe'
> # Default args are passed to each task
> default_args = {
> 'owner': 'Airflow',
> 'depends_on_past': False,
> 'start_date': datetime(2016, 04, 10),
> 'retries': 0,
> 'retry_interval': timedelta(minutes=5),
> 'email': ['y...@email.com'],
> 'email_on_failure': True,
> 'email_on_retry': True,
> 'wait_for_downstream': False,
> }
> # Create the dag object
> dag = DAG(dag_name,
> default_args=default_args,
> schedule_interval='0 0 * * *'
> )
> # -\
> # DEFINE THE TASKS
> # -/
> def get_subdag(dag, sd_id, pool=None):
> subdag = DAG(
> dag_id='{parent_dag}.{sd_id}'.format(
> parent_dag=dag.dag_id,
> sd_id=sd_id),
> params=dag.params,
> default_args=dag.default_args,
> template_searchpath=dag.template_searchpath,
> user_defined_macros=dag.user_defined_macros,
> )
> t1 = BashOperator(
> task_id='{sd_id}_step_1'.format(
> sd_id=sd_id
> ),
> bash_command='echo "hello" && sleep 60',
> dag=subdag,
> pool=pool,
> executor=SequentialExecutor
> )
> t2 = BashOperator(
> task_id='{sd_id}_step_two'.format(
> sd_id=sd_id
> ),
> bash_command='echo "hello" && sleep 15',
> dag=subdag,
> pool=pool,
> executor=SequentialExecutor
> )
> t2.set_upstream(t1)
> sdo = SubDagOperator(
> task_id=sd_id,
> subdag=subdag,
> retries=0,
> retry_delay=timedelta(seconds=5),
> dag=dag,
> depends_on_past=True,
> )
> return sdo
> start_task = DummyOperator(
> task_id='start',
> dag=dag
> )
> for n in range(1, 17):
> sd_i = get_subdag(dag=dag, sd_id='level_1_{n}'.format(n=n),
> pool='test_pool_1')
> sd_ii = get_subdag(dag=dag, sd_id='level_2_{n}'.format(n=n),
> pool='test_pool_2')
> sd_iii = get_subdag(dag=dag, sd_id='level_3_{n}'.format(n=n),
> pool='test_pool_3')
> sd_i.set_upstream(start_task)
> sd_ii.set_upstream(sd_i)
> sd_iii.set_upstream(sd_ii)
> ```
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-74) SubdagOperators can consume all celeryd worker processes
[
https://issues.apache.org/jira/browse/AIRFLOW-74?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16450406#comment-16450406
]
Tao Feng commented on AIRFLOW-74:
-
[~xzhu] , There are some issues for subdag to use with other executors(like
celery). So here is to change the default executor for subdag to sequential.
You could always override the executor to something else as long as you are
aware of the risk. Ideally we should fix and make subdag as first class. But
that is a long term fix.
> SubdagOperators can consume all celeryd worker processes
>
>
> Key: AIRFLOW-74
> URL: https://issues.apache.org/jira/browse/AIRFLOW-74
> Project: Apache Airflow
> Issue Type: Bug
> Components: celery
>Affects Versions: Airflow 1.7.1, Airflow 1.7.0, Airflow 1.6.2
> Environment: Airflow 1.7.1rc3 with CeleryExecutor
> 1 webserver
> 1 scheduler
> 2 workers
>Reporter: Steven Yvinec-Kruyk
>Assignee: zgl
>Priority: Major
> Fix For: 1.10.0
>
>
> If the amount of concurrent ```SubdagOperator``` running >= the no. of celery
> worker processes tasks are unable to work. All SDOs come to a complete halt.
> Futhermore performance of a DAG is drastically reduced even before full
> saturation of the workers as less workers are gradually available for actual
> tasks. A workaround for this is to specify ```SequentialExecutor``` be used
> by the ```SubdagOperator```
> ```
> from datetime import timedelta, datetime
> from airflow.models import DAG, Pool
> from airflow.operators import BashOperator, SubDagOperator, DummyOperator
> from airflow.executors import SequentialExecutor
> import airflow
> # -\
> # DEFINE THE POOLS
> # -/
> session = airflow.settings.Session()
> for p in ['test_pool_1', 'test_pool_2', 'test_pool_3']:
> pool = (
> session.query(Pool)
> .filter(Pool.pool == p)
> .first())
> if not pool:
> session.add(Pool(pool=p, slots=8))
> session.commit()
> # -\
> # DEFINE THE DAG
> # -/
> # Define the Dag Name. This must be unique.
> dag_name = 'hanging_subdags_n16_sqe'
> # Default args are passed to each task
> default_args = {
> 'owner': 'Airflow',
> 'depends_on_past': False,
> 'start_date': datetime(2016, 04, 10),
> 'retries': 0,
> 'retry_interval': timedelta(minutes=5),
> 'email': ['y...@email.com'],
> 'email_on_failure': True,
> 'email_on_retry': True,
> 'wait_for_downstream': False,
> }
> # Create the dag object
> dag = DAG(dag_name,
> default_args=default_args,
> schedule_interval='0 0 * * *'
> )
> # -\
> # DEFINE THE TASKS
> # -/
> def get_subdag(dag, sd_id, pool=None):
> subdag = DAG(
> dag_id='{parent_dag}.{sd_id}'.format(
> parent_dag=dag.dag_id,
> sd_id=sd_id),
> params=dag.params,
> default_args=dag.default_args,
> template_searchpath=dag.template_searchpath,
> user_defined_macros=dag.user_defined_macros,
> )
> t1 = BashOperator(
> task_id='{sd_id}_step_1'.format(
> sd_id=sd_id
> ),
> bash_command='echo "hello" && sleep 60',
> dag=subdag,
> pool=pool,
> executor=SequentialExecutor
> )
> t2 = BashOperator(
> task_id='{sd_id}_step_two'.format(
> sd_id=sd_id
> ),
> bash_command='echo "hello" && sleep 15',
> dag=subdag,
> pool=pool,
> executor=SequentialExecutor
> )
> t2.set_upstream(t1)
> sdo = SubDagOperator(
> task_id=sd_id,
> subdag=subdag,
> retries=0,
> retry_delay=timedelta(seconds=5),
> dag=dag,
> depends_on_past=True,
> )
> return sdo
> start_task = DummyOperator(
> task_id='start',
> dag=dag
> )
> for n in range(1, 17):
> sd_i = get_subdag(dag=dag, sd_id='level_1_{n}'.format(n=n),
> pool='test_pool_1')
> sd_ii = get_subdag(dag=dag, sd_id='level_2_{n}'.format(n=n),
> pool='test_pool_2')
> sd_iii = get_subdag(dag=dag, sd_id='level_3_{n}'.format(n=n),
> pool='test_pool_3')
> sd_i.set_upstream(start_task)
> sd_ii.set_upstream(sd_i)
> sd_iii.set_upstream(sd_ii)
> ```
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (AIRFLOW-2255) Add alembic migration script for AIRFLOW-2059
[ https://issues.apache.org/jira/browse/AIRFLOW-2255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng closed AIRFLOW-2255. - Resolution: Duplicate > Add alembic migration script for AIRFLOW-2059 > - > > Key: AIRFLOW-2255 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2255 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Tao Feng >Assignee: Tao Feng >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-390) [AIRFLOW-Don't load example dags by default]
[ https://issues.apache.org/jira/browse/AIRFLOW-390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng reassigned AIRFLOW-390: Assignee: (was: Tao Feng) > [AIRFLOW-Don't load example dags by default] > > > Key: AIRFLOW-390 > URL: https://issues.apache.org/jira/browse/AIRFLOW-390 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Sunny Sun >Priority: Trivial > Labels: easyfix > Original Estimate: 24h > Remaining Estimate: 24h > > Load examples should by default be set to False, so they are not > automatically deployed into production environments. This is especially heavy > because the twitter example dag requires Hive, which users may or may not use > in their own deployments. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2257) ValueError from flask-admin causing Charts feature to break
[
https://issues.apache.org/jira/browse/AIRFLOW-2257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438463#comment-16438463
]
Tao Feng commented on AIRFLOW-2257:
---
can't reproduce with the master branch...
> ValueError from flask-admin causing Charts feature to break
> ---
>
> Key: AIRFLOW-2257
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2257
> Project: Apache Airflow
> Issue Type: Bug
> Components: dependencies, ui
>Affects Versions: 1.9.0
>Reporter: Brian Sloan
>Priority: Major
>
> Using Airflow 1.9 and RDS PostgreSQL for the metadata database.
> This bug may be related to this flask-admin / sqlalchemy issue:
> https://github.com/flask-admin/flask-admin/issues/1588
> To reproduce, go to "Data Profiling –> Charts" and click the Edit icon to
> edit any chart. I get an HTTP 500 with this traceback:
> {code:java}
> Traceback (most recent call last):
> File "/var/airflow/env/local/lib/python2.7/site-packages/flask/app.py",
> line 1988, in wsgi_app
> response = self.full_dispatch_request()
> File "/var/airflow/env/local/lib/python2.7/site-packages/flask/app.py",
> line 1641, in full_dispatch_request
> rv = self.handle_user_exception(e)
> File "/var/airflow/env/local/lib/python2.7/site-packages/flask/app.py",
> line 1544, in handle_user_exception
> reraise(exc_type, exc_value, tb)
> File "/var/airflow/env/local/lib/python2.7/site-packages/flask/app.py",
> line 1639, in full_dispatch_request
> rv = self.dispatch_request()
> File "/var/airflow/env/local/lib/python2.7/site-packages/flask/app.py",
> line 1625, in dispatch_request
> return self.view_functions[rule.endpoint](**req.view_args)
> File
> "/var/airflow/env/local/lib/python2.7/site-packages/flask_admin/base.py",
> line 69, in inner
> return self._run_view(f, *args, **kwargs)
> File
> "/var/airflow/env/local/lib/python2.7/site-packages/flask_admin/base.py",
> line 368, in _run_view
> return fn(self, *args, **kwargs)
> File
> "/var/airflow/env/local/lib/python2.7/site-packages/flask_admin/model/base.py",
> line 1999, in edit_view
> return_url=return_url)
> File
> "/var/airflow/env/local/lib/python2.7/site-packages/flask_admin/base.py",
> line 308, in render
> return render_template(template, **kwargs)
> File
> "/var/airflow/env/local/lib/python2.7/site-packages/flask/templating.py",
> line 134, in render_template
> context, ctx.app)
> File
> "/var/airflow/env/local/lib/python2.7/site-packages/flask/templating.py",
> line 116, in _render
> rv = template.render(context)
> File
> "/var/airflow/env/local/lib/python2.7/site-packages/jinja2/environment.py",
> line 989, in render
> return self.environment.handle_exception(exc_info, True)
> File
> "/var/airflow/env/local/lib/python2.7/site-packages/jinja2/environment.py",
> line 754, in handle_exception
> reraise(exc_type, exc_value, tb)
> File
> "/var/airflow/env/local/lib/python2.7/site-packages/airflow/www/templates/airflow/chart/edit.html",
> line 18, in top-level template code
> {% extends 'admin/model/edit.html' %}
> File
> "/var/airflow/env/local/lib/python2.7/site-packages/flask_admin/templates/bootstrap3/admin/model/edit.html",
> line 3, in top-level template code
> {% from 'admin/lib.html' import extra with context %} {# backward
> compatible #}
> File
> "/var/airflow/env/local/lib/python2.7/site-packages/airflow/www/templates/admin/master.html",
> line 18, in top-level template code
> {% extends 'admin/base.html' %}
> File
> "/var/airflow/env/local/lib/python2.7/site-packages/flask_admin/templates/bootstrap3/admin/base.html",
> line 30, in top-level template code
> {% block page_body %}
> File
> "/var/airflow/env/local/lib/python2.7/site-packages/airflow/www/templates/admin/master.html",
> line 104, in block "page_body"
> {% block body %}
> File
> "/var/airflow/env/local/lib/python2.7/site-packages/flask_admin/templates/bootstrap3/admin/model/edit.html",
> line 32, in block "body"
> {% block edit_form %}
> File
> "/var/airflow/env/local/lib/python2.7/site-packages/flask_admin/templates/bootstrap3/admin/model/edit.html",
> line 33, in block "edit_form"
> {{ lib.render_form(form, return_url, extra(), form_opts) }}
> File
> "/var/airflow/env/local/lib/python2.7/site-packages/flask_admin/templates/bootstrap3/admin/lib.html",
> line 202, in template
> {% call form_tag(action=action) %}
> File
> "/var/airflow/env/local/lib/python2.7/site-packages/flask_admin/templates/bootstrap3/admin/lib.html",
> line 182, in template
> {{ caller() }}
> File
> "/var/airflow/env/local/lib/python2.7/site-packages/flask_admin/templates/bootstrap3/admin/lib.html",
> line 203, in template
> {{ render_form_fields(for
[jira] [Assigned] (AIRFLOW-390) [AIRFLOW-Don't load example dags by default]
[ https://issues.apache.org/jira/browse/AIRFLOW-390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng reassigned AIRFLOW-390: Assignee: Tao Feng > [AIRFLOW-Don't load example dags by default] > > > Key: AIRFLOW-390 > URL: https://issues.apache.org/jira/browse/AIRFLOW-390 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Sunny Sun >Assignee: Tao Feng >Priority: Trivial > Labels: easyfix > Original Estimate: 24h > Remaining Estimate: 24h > > Load examples should by default be set to False, so they are not > automatically deployed into production environments. This is especially heavy > because the twitter example dag requires Hive, which users may or may not use > in their own deployments. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-390) [AIRFLOW-Don't load example dags by default]
[ https://issues.apache.org/jira/browse/AIRFLOW-390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng reassigned AIRFLOW-390: Assignee: (was: Tao Feng) > [AIRFLOW-Don't load example dags by default] > > > Key: AIRFLOW-390 > URL: https://issues.apache.org/jira/browse/AIRFLOW-390 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Sunny Sun >Priority: Trivial > Labels: easyfix > Original Estimate: 24h > Remaining Estimate: 24h > > Load examples should by default be set to False, so they are not > automatically deployed into production environments. This is especially heavy > because the twitter example dag requires Hive, which users may or may not use > in their own deployments. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-390) [AIRFLOW-Don't load example dags by default]
[ https://issues.apache.org/jira/browse/AIRFLOW-390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng reassigned AIRFLOW-390: Assignee: Tao Feng > [AIRFLOW-Don't load example dags by default] > > > Key: AIRFLOW-390 > URL: https://issues.apache.org/jira/browse/AIRFLOW-390 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Sunny Sun >Assignee: Tao Feng >Priority: Trivial > Labels: easyfix > Original Estimate: 24h > Remaining Estimate: 24h > > Load examples should by default be set to False, so they are not > automatically deployed into production environments. This is especially heavy > because the twitter example dag requires Hive, which users may or may not use > in their own deployments. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-2304) Quick start documentation is not quick: should mention scheduler part
[ https://issues.apache.org/jira/browse/AIRFLOW-2304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng reassigned AIRFLOW-2304: - Assignee: Tao Feng > Quick start documentation is not quick: should mention scheduler part > - > > Key: AIRFLOW-2304 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2304 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Shawn Xu >Assignee: Tao Feng >Priority: Major > > The [Quick start documentation|https://airflow.apache.org/start.html] part > should offer user a push button style launch to pilot stuff quickly. > > However right now it only mentions web server so none of the DAGs ended up > running. And this is particularly confusing to new users if they don't know > what to expect (flow will be in running state but tasks have no states). > > There are 2 things to get DAGs scheduled: > # Start a scheduler even if the default SequentialExecutor is used > # The DAG must be enabled. > They should be documented in quick start doc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-1633) docker_operator needs a way to set shm_size
[
https://issues.apache.org/jira/browse/AIRFLOW-1633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Feng reassigned AIRFLOW-1633:
-
Assignee: Tao Feng
> docker_operator needs a way to set shm_size
> ---
>
> Key: AIRFLOW-1633
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1633
> Project: Apache Airflow
> Issue Type: Improvement
>Reporter: Gregory Benison
>Assignee: Tao Feng
>Priority: Major
>
> The "docker run" command line client has an option "--shm-size" that allows
> setting the size of /dev/shm in the container. The "docker" package - on
> which docker_operator is based - does provide an interface for setting this
> property; docker.utils.create_host_config() has a "shm_size" argument that
> defaults to None. However docker_operator calls that method in a way that
> does not provide a mechanism for accessing that argument:
> {noformat}
> self.container = self.cli.create_container(
> command=self.get_command(),
> cpu_shares=cpu_shares,
> environment=self.environment,
>
> host_config=self.cli.create_host_config(binds=self.volumes,
>
> network_mode=self.network_mode),
> image=image,
> mem_limit=self.mem_limit,
> user=self.user
> )
> {noformat}
> It would be useful to provide a way to pass the "shm_size" option to the call
> to create_host_config(). Actually create_host_config() has a large number of
> options that can't be controlled through docker_operator - perhaps it would
> make sense to find a way to expose all or most of them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2285) Need to be able to stop a running DAG
[ https://issues.apache.org/jira/browse/AIRFLOW-2285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16427225#comment-16427225 ] Tao Feng commented on AIRFLOW-2285: --- Hey Gregory, have you tried in the uI(Browse->DAG Runs)? You could find out your dag and mark the running dag state to failed / finished. It should stop the running dag. > Need to be able to stop a running DAG > - > > Key: AIRFLOW-2285 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2285 > Project: Apache Airflow > Issue Type: New Feature > Components: DAG >Affects Versions: 1.9.0 >Reporter: Gregory Dougherty >Priority: Major > Labels: features > > There is no way to stop a running DAG. As DAGs can have serious bugs (for > example, uploading data to the wrong location, when being charged for every > MB uploaded), need a way to force a DAG to stop running -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2278) Add possibility to edit DAGs in webapp
[
https://issues.apache.org/jira/browse/AIRFLOW-2278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16424265#comment-16424265
]
Tao Feng commented on AIRFLOW-2278:
---
hey Mykola, not sure if you are aware of the ongoing work, but
https://issues.apache.org/jira/browse/AIRFLOW-2267 should help with this
request.
> Add possibility to edit DAGs in webapp
> --
>
> Key: AIRFLOW-2278
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2278
> Project: Apache Airflow
> Issue Type: Improvement
> Components: webapp
>Reporter: Mykola Mykhalov
>Assignee: Mykola Mykhalov
>Priority: Minor
>
> When you need to make some minor changes in your DAG would be nice to have
> possibility to edit it from webapp.
> To protect DAG from editing by everyone should be set up arg inside DAG
> 'editable_by'. Where '*' means everyone, and ['user1', 'user2'] means
> specific users.
> Example when DAG can be editable in webapp by user1 and user2 only:
> args = {
> 'owner': 'airflow',
> 'editable_by': ['user1', 'user2'],
> 'start_date': airflow.utils.dates.days_ago(2)
> }
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (AIRFLOW-2267) Implement DAG-level access for airflow
[ https://issues.apache.org/jira/browse/AIRFLOW-2267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-2267 started by Tao Feng. - > Implement DAG-level access for airflow > -- > > Key: AIRFLOW-2267 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2267 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Tao Feng >Assignee: Tao Feng >Priority: Major > > Now we have a new airflow UI based on RBAC, we would like to implement per > DAG-level access on top of it. I have started some initial discussions with > Max @mistercrunch, and will come up with a proposal shortly to gather more > feedback from the community. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2267) Implement DAG-level access for airflow
[ https://issues.apache.org/jira/browse/AIRFLOW-2267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng updated AIRFLOW-2267: -- Summary: Implement DAG-level access for airflow (was: Implement Per DAG-level access for airflow) > Implement DAG-level access for airflow > -- > > Key: AIRFLOW-2267 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2267 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Tao Feng >Assignee: Tao Feng >Priority: Major > > Now we have a new airflow UI based on RBAC, we would like to implement per > DAG-level access on top of it. I have started some initial discussions with > Max @mistercrunch, and will come up with a proposal shortly to gather more > feedback from the community. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
