[jira] [Commented] (AIRFLOW-7063) dag.clear() slowness caused by multiple UNION statements and tis.count()

2020-03-14 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-7063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17059319#comment-17059319 ] Qian Yu commented on AIRFLOW-7063: -- Here's a self-contained test to reproduce this slo

[jira] [Updated] (AIRFLOW-7063) dag.clear() slowness caused by multiple UNION statements and tis.count()

2020-03-14 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-7063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu updated AIRFLOW-7063: - Description: When multiple {{ExternalTaskMarker}} are used, {{dag.clear()}} becomes very slow when cle

[jira] [Created] (AIRFLOW-7063) dag.clear() slowness caused by multiple UNION statements and tis.count()

2020-03-14 Thread Qian Yu (Jira)
Qian Yu created AIRFLOW-7063: Summary: dag.clear() slowness caused by multiple UNION statements and tis.count() Key: AIRFLOW-7063 URL: https://issues.apache.org/jira/browse/AIRFLOW-7063 Project: Apache Ai

[jira] [Commented] (AIRFLOW-1230) Upstream_failed tasks are not executed when the DAG is restarted after failure

2020-02-28 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-1230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17048223#comment-17048223 ] Qian Yu commented on AIRFLOW-1230: -- The problem is seems to be caused by what you are

[jira] [Updated] (AIRFLOW-6834) Fix flaky test test_scheduler_job.py::TestDagFileProcessor::test_dag_file_processor_process_task_instances_depends_on_past

2020-02-23 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu updated AIRFLOW-6834: - Description: test_scheduler_job.py has a few flaky tests. Some are marked with pytest.mark.xfail, but

[jira] [Assigned] (AIRFLOW-6834) Fix flaky test test_scheduler_job.py::TestDagFileProcessor::test_dag_file_processor_process_task_instances_depends_on_past

2020-02-19 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu reassigned AIRFLOW-6834: Assignee: Qian Yu > Fix flaky test > test_scheduler_job.py::TestDagFileProcessor::test_dag_file

[jira] [Commented] (AIRFLOW-6840) Bump up version of future to 0.18.2 in v-10-testing to get closer to python 3.8

2020-02-19 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17039793#comment-17039793 ] Qian Yu commented on AIRFLOW-6840: -- Tagging some people who might know the answer: [~

[jira] [Updated] (AIRFLOW-6840) Bump up version of future to 0.18.2 in v-10-testing to get closer to python 3.8

2020-02-19 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu updated AIRFLOW-6840: - Summary: Bump up version of future to 0.18.2 in v-10-testing to get closer to python 3.8 (was: Bump up

[jira] [Created] (AIRFLOW-6840) Bump up version of future to 0.18.2 in v-10-testing

2020-02-19 Thread Qian Yu (Jira)
Qian Yu created AIRFLOW-6840: Summary: Bump up version of future to 0.18.2 in v-10-testing Key: AIRFLOW-6840 URL: https://issues.apache.org/jira/browse/AIRFLOW-6840 Project: Apache Airflow Issue

[jira] [Created] (AIRFLOW-6834) Fix flaky test test_scheduler_job.py::TestDagFileProcessor::test_dag_file_processor_process_task_instances_depends_on_past

2020-02-18 Thread Qian Yu (Jira)
Qian Yu created AIRFLOW-6834: Summary: Fix flaky test test_scheduler_job.py::TestDagFileProcessor::test_dag_file_processor_process_task_instances_depends_on_past Key: AIRFLOW-6834 URL: https://issues.apache.org/jira/b

[jira] [Closed] (AIRFLOW-5444) action_logging missing important fields such as Dag Id and Task Id for POST actions

2020-02-15 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu closed AIRFLOW-5444. > action_logging missing important fields such as Dag Id and Task Id for POST > actions > --

[jira] [Created] (AIRFLOW-6800) SSHHook: Close file object when reading ssh config

2020-02-13 Thread Qian Yu (Jira)
Qian Yu created AIRFLOW-6800: Summary: SSHHook: Close file object when reading ssh config Key: AIRFLOW-6800 URL: https://issues.apache.org/jira/browse/AIRFLOW-6800 Project: Apache Airflow Issue T

[jira] [Created] (AIRFLOW-6704) TaskInstance.operator is not set when task is marked success or failed in the Web UI

2020-02-01 Thread Qian Yu (Jira)
Qian Yu created AIRFLOW-6704: Summary: TaskInstance.operator is not set when task is marked success or failed in the Web UI Key: AIRFLOW-6704 URL: https://issues.apache.org/jira/browse/AIRFLOW-6704 Projec

[jira] [Assigned] (AIRFLOW-6704) TaskInstance.operator is not set when task is marked success or failed in the Web UI

2020-02-01 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu reassigned AIRFLOW-6704: Assignee: Qian Yu > TaskInstance.operator is not set when task is marked success or failed in th

[jira] [Updated] (AIRFLOW-6704) TaskInstance.operator is not set when task is marked success or failed in the Web UI

2020-02-01 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu updated AIRFLOW-6704: - Issue Type: Bug (was: Improvement) > TaskInstance.operator is not set when task is marked success or f

[jira] [Commented] (AIRFLOW-6657) Deprecate BaseBranchOperator

2020-02-01 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17028041#comment-17028041 ] Qian Yu commented on AIRFLOW-6657: -- I renamed the JIRA to be "Deprecate BaseBranchOper

[jira] [Updated] (AIRFLOW-6657) Deprecate BaseBranchOperator

2020-02-01 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu updated AIRFLOW-6657: - Description: This PR added a base class {{BaseBranchOperator}} that is intended to let people implemen

[jira] [Updated] (AIRFLOW-6657) Deprecate BaseBranchOperator

2020-02-01 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu updated AIRFLOW-6657: - Summary: Deprecate BaseBranchOperator (was: Implement BranchPythonOperator as BaseBranchOperator) > D

[jira] [Assigned] (AIRFLOW-5391) Clearing a task skipped by BranchPythonOperator will cause the task to execute

2020-01-28 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu reassigned AIRFLOW-5391: Assignee: Qian Yu > Clearing a task skipped by BranchPythonOperator will cause the task to execu

[jira] [Commented] (AIRFLOW-6657) Implement BranchPythonOperator as BaseBranchOperator

2020-01-28 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17024976#comment-17024976 ] Qian Yu commented on AIRFLOW-6657: -- Okay nvm it does look very awkward to implement {{

[jira] [Commented] (AIRFLOW-6657) Implement BranchPythonOperator as BaseBranchOperator

2020-01-28 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17024954#comment-17024954 ] Qian Yu commented on AIRFLOW-6657: -- [~kaxilnaik] [~ash] [~chronitis]  It is not clear

[jira] [Created] (AIRFLOW-6657) Implement BranchPythonOperator as BaseBranchOperator

2020-01-28 Thread Qian Yu (Jira)
Qian Yu created AIRFLOW-6657: Summary: Implement BranchPythonOperator as BaseBranchOperator Key: AIRFLOW-6657 URL: https://issues.apache.org/jira/browse/AIRFLOW-6657 Project: Apache Airflow Issue

[jira] [Commented] (AIRFLOW-6527) Error sending Celery task:Timeout in send_task_to_executor

2020-01-12 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17013670#comment-17013670 ] Qian Yu commented on AIRFLOW-6527: -- [~kaxilnaik] [~kamil.bregula] I put up a PR here t

[jira] [Commented] (AIRFLOW-6527) Error sending Celery task:Timeout in send_task_to_executor

2020-01-09 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17012447#comment-17012447 ] Qian Yu commented on AIRFLOW-6527: -- Tagging some people who might know: [~kamil.bregul

[jira] [Updated] (AIRFLOW-6527) Error sending Celery task:Timeout in send_task_to_executor

2020-01-09 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu updated AIRFLOW-6527: - Description: We use Airflow with CeleryExecutor and redis broker. Our airflow scheduler often encounte

[jira] [Created] (AIRFLOW-6527) Error sending Celery task:Timeout in send_task_to_executor

2020-01-09 Thread Qian Yu (Jira)
Qian Yu created AIRFLOW-6527: Summary: Error sending Celery task:Timeout in send_task_to_executor Key: AIRFLOW-6527 URL: https://issues.apache.org/jira/browse/AIRFLOW-6527 Project: Apache Airflow

[jira] [Commented] (AIRFLOW-6497) Scheduler creates DagBag in the same process with outdated info

2020-01-07 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17010303#comment-17010303 ] Qian Yu commented on AIRFLOW-6497: -- [~kamil.bregula] Do you want to create a PR? Or if

[jira] [Assigned] (AIRFLOW-2279) Clearing Tasks Across DAGs

2020-01-07 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu reassigned AIRFLOW-2279: Assignee: Qian Yu > Clearing Tasks Across DAGs > -- > >

[jira] [Commented] (AIRFLOW-6497) Scheduler creates DagBag in the same process with outdated info

2020-01-06 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17009401#comment-17009401 ] Qian Yu commented on AIRFLOW-6497: -- [~bolke] [~potiuk] [~kamil.bregula] any idea / sug

[jira] [Created] (AIRFLOW-6497) Scheduler creates DagBag in the same process with outdated info

2020-01-06 Thread Qian Yu (Jira)
Qian Yu created AIRFLOW-6497: Summary: Scheduler creates DagBag in the same process with outdated info Key: AIRFLOW-6497 URL: https://issues.apache.org/jira/browse/AIRFLOW-6497 Project: Apache Airflow

[jira] [Assigned] (AIRFLOW-6250) on_failure_callback does not know the task_id when handle_failure() is called without passing context

2019-12-12 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu reassigned AIRFLOW-6250: Assignee: Qian Yu > on_failure_callback does not know the task_id when handle_failure() is calle

[jira] [Created] (AIRFLOW-6250) on_failure_callback does not know the task_id when handle_failure() is called without passing context

2019-12-12 Thread Qian Yu (Jira)
Qian Yu created AIRFLOW-6250: Summary: on_failure_callback does not know the task_id when handle_failure() is called without passing context Key: AIRFLOW-6250 URL: https://issues.apache.org/jira/browse/AIRFLOW-6250

[jira] [Closed] (AIRFLOW-5648) Add ClearTaskOperator to allow clearing/re-running tasks from within a DAG

2019-11-22 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu closed AIRFLOW-5648. Resolution: Won't Fix See comments on the closed PR: [https://github.com/apache/airflow/pull/6392]   Th

[jira] [Created] (AIRFLOW-5840) Add a link to the external task that ExternalTaskSensor is waiting for

2019-11-03 Thread Qian Yu (Jira)
Qian Yu created AIRFLOW-5840: Summary: Add a link to the external task that ExternalTaskSensor is waiting for Key: AIRFLOW-5840 URL: https://issues.apache.org/jira/browse/AIRFLOW-5840 Project: Apache Airf

[jira] [Commented] (AIRFLOW-2279) Clearing Tasks Across DAGs

2019-11-03 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16965639#comment-16965639 ] Qian Yu commented on AIRFLOW-2279: -- I'm in the process of working on something I call

[jira] [Assigned] (AIRFLOW-5648) Add ClearTaskOperator to allow clearing/re-running tasks from within a DAG

2019-10-17 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu reassigned AIRFLOW-5648: Assignee: Qian Yu > Add ClearTaskOperator to allow clearing/re-running tasks from within a DAG >

[jira] [Updated] (AIRFLOW-5648) Add ClearTaskOperator to allow clearing/re-running tasks from within a DAG

2019-10-14 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu updated AIRFLOW-5648: - Description: There are use cases where some external conditions have changed and a section of the DAG

[jira] [Updated] (AIRFLOW-5648) Add ClearTaskOperator to allow clearing/re-running tasks from within a DAG

2019-10-14 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu updated AIRFLOW-5648: - Description: There are use cases where some external conditions have changed and a section of the DAG

[jira] [Updated] (AIRFLOW-5648) Add ClearTaskOperator to allow clearing/re-running tasks from within a DAG

2019-10-13 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu updated AIRFLOW-5648: - Labels: airflow clear duplicate operator re-run rerun task (was: clear duplicate operator re-run rerun

[jira] [Updated] (AIRFLOW-5648) Add ClearTaskOperator to allow clearing/re-running tasks from within a DAG

2019-10-13 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu updated AIRFLOW-5648: - Labels: ClearTaskOperator airflow clear duplicate operator re-run rerun task (was: airflow clear dupli

[jira] [Updated] (AIRFLOW-5648) Add ClearTaskOperator to allow clearing/re-running tasks from within a DAG

2019-10-13 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu updated AIRFLOW-5648: - Description: There are use cases where some external conditions have changed and a section of the DAG

[jira] [Created] (AIRFLOW-5648) Add ClearTaskOperator to allow clearing/re-running tasks from within a DAG

2019-10-13 Thread Qian Yu (Jira)
Qian Yu created AIRFLOW-5648: Summary: Add ClearTaskOperator to allow clearing/re-running tasks from within a DAG Key: AIRFLOW-5648 URL: https://issues.apache.org/jira/browse/AIRFLOW-5648 Project: Apache

[jira] [Commented] (AIRFLOW-2219) Race condition to DagRun.verify_integrity between Scheduler and Webserver

2019-09-23 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16936370#comment-16936370 ] Qian Yu commented on AIRFLOW-2219: -- Also this looks related or the same as:  https://i

[jira] [Commented] (AIRFLOW-2219) Race condition to DagRun.verify_integrity between Scheduler and Webserver

2019-09-23 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16936368#comment-16936368 ] Qian Yu commented on AIRFLOW-2219: -- Just saw this error today too.   This JIRA is ma

[jira] [Updated] (AIRFLOW-5462) Google Authentication redirection page crashes with KeyError: 'login'

2019-09-18 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu updated AIRFLOW-5462: - Priority: Minor (was: Major) > Google Authentication redirection page crashes with KeyError: 'login' >

[jira] [Updated] (AIRFLOW-5410) AirflowConfigParser no longer find conf entries with capitalized letters

2019-09-18 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu updated AIRFLOW-5410: - Priority: Minor (was: Major) > AirflowConfigParser no longer find conf entries with capitalized letter

[jira] [Created] (AIRFLOW-5462) Google Authentication redirection page crashes with KeyError: 'login'

2019-09-11 Thread Qian Yu (Jira)
Qian Yu created AIRFLOW-5462: Summary: Google Authentication redirection page crashes with KeyError: 'login' Key: AIRFLOW-5462 URL: https://issues.apache.org/jira/browse/AIRFLOW-5462 Project: Apache Airfl

[jira] [Created] (AIRFLOW-5444) action_logging missing important fields such as Dag Id and Task Id for POST actions

2019-09-08 Thread Qian Yu (Jira)
Qian Yu created AIRFLOW-5444: Summary: action_logging missing important fields such as Dag Id and Task Id for POST actions Key: AIRFLOW-5444 URL: https://issues.apache.org/jira/browse/AIRFLOW-5444 Project

[jira] [Commented] (AIRFLOW-4052) To allow filtering using "event" and "owner" in "Log" view

2019-09-08 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925288#comment-16925288 ] Qian Yu commented on AIRFLOW-4052: -- The "Fix Version" here is shown as 1.10.3, but cle

[jira] [Created] (AIRFLOW-5410) AirflowConfigParser no longer find conf entries with capitalized letters

2019-09-05 Thread Qian Yu (Jira)
Qian Yu created AIRFLOW-5410: Summary: AirflowConfigParser no longer find conf entries with capitalized letters Key: AIRFLOW-5410 URL: https://issues.apache.org/jira/browse/AIRFLOW-5410 Project: Apache Ai

[jira] [Updated] (AIRFLOW-5391) Clearing a task skipped by BranchPythonOperator will cause the task to execute

2019-09-03 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu updated AIRFLOW-5391: - Description: I tried this on 1.10.3 and 1.10.4, both have this issue:  E.g. in this example from the d

[jira] [Created] (AIRFLOW-5391) Clearing a task skipped by BranchPythonOperator will cause the task to execute

2019-09-03 Thread Qian Yu (Jira)
Qian Yu created AIRFLOW-5391: Summary: Clearing a task skipped by BranchPythonOperator will cause the task to execute Key: AIRFLOW-5391 URL: https://issues.apache.org/jira/browse/AIRFLOW-5391 Project: Apa

[jira] [Assigned] (AIRFLOW-4013) Mark success/failed is picking all execution date

2019-08-09 Thread Qian Yu (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-4013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu reassigned AIRFLOW-4013: Assignee: Qian Yu > Mark success/failed is picking all execution date >