[GitHub] [airflow] tooptoop4 commented on a change in pull request #6954: [AIRFLOW-4355] removed task should not lead to dagrun success

2020-01-06 Thread GitBox
tooptoop4 commented on a change in pull request #6954: [AIRFLOW-4355] removed task should not lead to dagrun success URL: https://github.com/apache/airflow/pull/6954#discussion_r363625074 ## File path: airflow/models/dagrun.py ## @@ -311,32 +313,51 @@ def

[GitHub] [airflow] tooptoop4 commented on a change in pull request #7038: [AIRFLOW-4495] allow externally triggered dags to run for future exec dates

2020-01-06 Thread GitBox
tooptoop4 commented on a change in pull request #7038: [AIRFLOW-4495] allow externally triggered dags to run for future exec dates URL: https://github.com/apache/airflow/pull/7038#discussion_r363623656 ## File path: airflow/ti_deps/deps/runnable_exec_date_dep.py ## @@

[jira] [Updated] (AIRFLOW-6419) dag_processor_manager/webserver/scheduler logs should be created under date folder

2020-01-06 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated AIRFLOW-6419: -- Description: dag level logs are written under separate date folders. This is great because the old dates

[GitHub] [airflow] albertusk95 commented on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection

2020-01-06 Thread GitBox
albertusk95 commented on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection URL: https://github.com/apache/airflow/pull/7075#issuecomment-571470394 @tooptoop4 I think you might want to try this sample DAG to reproduce the issue. a) create an environment var for spark

[GitHub] [airflow] tooptoop4 commented on a change in pull request #6342: [AIRFLOW-5662] fix incorrect naming for scheduler used slot metric

2020-01-06 Thread GitBox
tooptoop4 commented on a change in pull request #6342: [AIRFLOW-5662] fix incorrect naming for scheduler used slot metric URL: https://github.com/apache/airflow/pull/6342#discussion_r363619884 ## File path: airflow/jobs/scheduler_job.py ## @@ -1568,15 +1567,18 @@ def

[GitHub] [airflow] codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop

2020-01-06 Thread GitBox
codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571443074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=h1) Report > Merging

[GitHub] [airflow] albertusk95 commented on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection

2020-01-06 Thread GitBox
albertusk95 commented on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection URL: https://github.com/apache/airflow/pull/7075#issuecomment-571468228 > > @tooptoop4 if we don't remove the spark check on line 177, how to use this hook to track driver status deployed on yarn,

[GitHub] [airflow] tooptoop4 edited a comment on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection

2020-01-06 Thread GitBox
tooptoop4 edited a comment on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection URL: https://github.com/apache/airflow/pull/7075#issuecomment-571462616 > @tooptoop4 if we don't remove the spark check on line 177, how to use this hook to track driver status deployed on

[GitHub] [airflow] tooptoop4 edited a comment on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection

2020-01-06 Thread GitBox
tooptoop4 edited a comment on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection URL: https://github.com/apache/airflow/pull/7075#issuecomment-571462616 > @tooptoop4 if we don't remove the spark check on line 177, how to use this hook to track driver status deployed on

[GitHub] [airflow] albertusk95 commented on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection

2020-01-06 Thread GitBox
albertusk95 commented on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection URL: https://github.com/apache/airflow/pull/7075#issuecomment-571462963 > @tooptoop4 the tests pass only for the case when the connection information (host, port, conn_type, etc.) are stored in

[GitHub] [airflow] tooptoop4 commented on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection

2020-01-06 Thread GitBox
tooptoop4 commented on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection URL: https://github.com/apache/airflow/pull/7075#issuecomment-571462616 spark hook works for me and tests pass without this pr. can u send the add connection cli command u create to hit the issue?

[jira] [Commented] (AIRFLOW-6497) Scheduler creates DagBag in the same process with outdated info

2020-01-06 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009401#comment-17009401 ] Qian Yu commented on AIRFLOW-6497: -- [~bolke] [~potiuk] [~kamil.bregula] any idea / suggestions? >

[jira] [Created] (AIRFLOW-6497) Scheduler creates DagBag in the same process with outdated info

2020-01-06 Thread Qian Yu (Jira)
Qian Yu created AIRFLOW-6497: Summary: Scheduler creates DagBag in the same process with outdated info Key: AIRFLOW-6497 URL: https://issues.apache.org/jira/browse/AIRFLOW-6497 Project: Apache Airflow

[jira] [Updated] (AIRFLOW-6487) Romove unnecessary for loop

2020-01-06 Thread Wenqiang Li (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenqiang Li updated AIRFLOW-6487: - Description: Romove unnecessary for loop: In DagFileProcessorManager, dag files are processed

[jira] [Assigned] (AIRFLOW-6405) Bigquery Update Table Properties Operator

2020-01-06 Thread Jithin Sukumar (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jithin Sukumar reassigned AIRFLOW-6405: --- Assignee: Jithin Sukumar > Bigquery Update Table Properties Operator >

[GitHub] [airflow] Gabriel39 removed a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop

2020-01-06 Thread GitBox
Gabriel39 removed a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571433348 Why it failed? Can someone help me? thx This is an automated

[GitHub] [airflow] codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop

2020-01-06 Thread GitBox
codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571443074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=h1) Report > Merging

[GitHub] [airflow] codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop

2020-01-06 Thread GitBox
codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571443074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=h1) Report > Merging

[GitHub] [airflow] codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop

2020-01-06 Thread GitBox
codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571443074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=h1) Report > Merging

[GitHub] [airflow] codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop

2020-01-06 Thread GitBox
codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571443074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=h1) Report > Merging

[GitHub] [airflow] codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop

2020-01-06 Thread GitBox
codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571443074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=h1) Report > Merging

[GitHub] [airflow] codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop

2020-01-06 Thread GitBox
codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571443074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=h1) Report > Merging

[GitHub] [airflow] codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop

2020-01-06 Thread GitBox
codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571443074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=h1) Report > Merging

[GitHub] [airflow] codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop

2020-01-06 Thread GitBox
codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571443074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=h1) Report > Merging

[GitHub] [airflow] codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop

2020-01-06 Thread GitBox
codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571443074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=h1) Report > Merging

[GitHub] [airflow] codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop

2020-01-06 Thread GitBox
codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571443074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=h1) Report > Merging

[GitHub] [airflow] codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop

2020-01-06 Thread GitBox
codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571443074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=h1) Report > Merging

[GitHub] [airflow] codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop

2020-01-06 Thread GitBox
codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571443074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=h1) Report > Merging

[GitHub] [airflow] codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop

2020-01-06 Thread GitBox
codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571443074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=h1) Report > Merging

[GitHub] [airflow] codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop

2020-01-06 Thread GitBox
codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571443074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=h1) Report > Merging

[GitHub] [airflow] codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop

2020-01-06 Thread GitBox
codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571443074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=h1) Report > Merging

[GitHub] [airflow] codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop

2020-01-06 Thread GitBox
codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571443074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=h1) Report > Merging

[GitHub] [airflow] codecov-io commented on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop

2020-01-06 Thread GitBox
codecov-io commented on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571443074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=h1) Report > Merging

[GitHub] [airflow] codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop

2020-01-06 Thread GitBox
codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571443074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=h1) Report > Merging

[GitHub] [airflow] codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop

2020-01-06 Thread GitBox
codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571443074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=h1) Report > Merging

[GitHub] [airflow] codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop

2020-01-06 Thread GitBox
codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571443074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=h1) Report > Merging

[GitHub] [airflow] codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop

2020-01-06 Thread GitBox
codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571443074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=h1) Report > Merging

[GitHub] [airflow] kaxil opened a new pull request #7092: [AIRFLOW-6347] BugFix: Can't get task logs when serialization is enabled

2020-01-06 Thread GitBox
kaxil opened a new pull request #7092: [AIRFLOW-6347] BugFix: Can't get task logs when serialization is enabled URL: https://github.com/apache/airflow/pull/7092 Quoting from Jira: **Problem**: When I set next config options in `airflow.cfg`: ```ini [core]

[jira] [Commented] (AIRFLOW-6347) Can't get task logs in webUI when DAG serialization is enabled

2020-01-06 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009374#comment-17009374 ] ASF GitHub Bot commented on AIRFLOW-6347: - kaxil commented on pull request #7092:

[jira] [Commented] (AIRFLOW-6489) Add BATS unit testing for bash scripts

2020-01-06 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009372#comment-17009372 ] ASF GitHub Bot commented on AIRFLOW-6489: - potiuk commented on pull request #7091:

[GitHub] [airflow] potiuk opened a new pull request #7091: [AIRFLOW-6489] Separate integrations [WIP]

2020-01-06 Thread GitBox
potiuk opened a new pull request #7091: [AIRFLOW-6489] Separate integrations [WIP] URL: https://github.com/apache/airflow/pull/7091 --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) - [x] Description above provides context of the change

[GitHub] [airflow] Gabriel39 commented on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop

2020-01-06 Thread GitBox
Gabriel39 commented on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571433348 Why it failed? Can someone help me? thx This is an automated message from

[GitHub] [airflow] albertusk95 edited a comment on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection

2020-01-06 Thread GitBox
albertusk95 edited a comment on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection URL: https://github.com/apache/airflow/pull/7075#issuecomment-571424197 @tooptoop4 if we don't remove the spark check on line 177, how to use this hook to track driver status deployed on yarn,

[GitHub] [airflow] albertusk95 edited a comment on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection

2020-01-06 Thread GitBox
albertusk95 edited a comment on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection URL: https://github.com/apache/airflow/pull/7075#issuecomment-571429121 @tooptoop4 the tests pass only for the case when the connection information (host, port, conn_type, etc.) are stored in

[jira] [Created] (AIRFLOW-6496) Choosing integration at Breeze start

2020-01-06 Thread Jarek Potiuk (Jira)
Jarek Potiuk created AIRFLOW-6496: - Summary: Choosing integration at Breeze start Key: AIRFLOW-6496 URL: https://issues.apache.org/jira/browse/AIRFLOW-6496 Project: Apache Airflow Issue

[GitHub] [airflow] albertusk95 edited a comment on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection

2020-01-06 Thread GitBox
albertusk95 edited a comment on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection URL: https://github.com/apache/airflow/pull/7075#issuecomment-571429121 @tooptoop4 the tests pass only for the case when the connection information (host, port, conn_type, etc.) are stored in

[GitHub] [airflow] albertusk95 edited a comment on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection

2020-01-06 Thread GitBox
albertusk95 edited a comment on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection URL: https://github.com/apache/airflow/pull/7075#issuecomment-571429121 @tooptoop4 the tests pass only for the case when the connection information (host, port, conn_type, etc.) are stored in

[GitHub] [airflow] albertusk95 commented on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection

2020-01-06 Thread GitBox
albertusk95 commented on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection URL: https://github.com/apache/airflow/pull/7075#issuecomment-571429121 @tooptoop4 the tests pass only for the case when the connection information (host, port, conn_type, etc.) are stored in database.

[GitHub] [airflow] codecov-io edited a comment on issue #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner

2020-01-06 Thread GitBox
codecov-io edited a comment on issue #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner URL: https://github.com/apache/airflow/pull/7090#issuecomment-571428318 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=h1) Report > Merging

[GitHub] [airflow] codecov-io edited a comment on issue #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner

2020-01-06 Thread GitBox
codecov-io edited a comment on issue #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner URL: https://github.com/apache/airflow/pull/7090#issuecomment-571428318 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=h1) Report > Merging

[GitHub] [airflow] codecov-io edited a comment on issue #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner

2020-01-06 Thread GitBox
codecov-io edited a comment on issue #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner URL: https://github.com/apache/airflow/pull/7090#issuecomment-571428318 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=h1) Report > Merging

[GitHub] [airflow] codecov-io edited a comment on issue #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner

2020-01-06 Thread GitBox
codecov-io edited a comment on issue #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner URL: https://github.com/apache/airflow/pull/7090#issuecomment-571428318 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=h1) Report > Merging

[GitHub] [airflow] codecov-io edited a comment on issue #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner

2020-01-06 Thread GitBox
codecov-io edited a comment on issue #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner URL: https://github.com/apache/airflow/pull/7090#issuecomment-571428318 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=h1) Report > Merging

[GitHub] [airflow] codecov-io edited a comment on issue #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner

2020-01-06 Thread GitBox
codecov-io edited a comment on issue #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner URL: https://github.com/apache/airflow/pull/7090#issuecomment-571428318 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=h1) Report > Merging

[GitHub] [airflow] codecov-io edited a comment on issue #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner

2020-01-06 Thread GitBox
codecov-io edited a comment on issue #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner URL: https://github.com/apache/airflow/pull/7090#issuecomment-571428318 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=h1) Report > Merging

[GitHub] [airflow] codecov-io edited a comment on issue #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner

2020-01-06 Thread GitBox
codecov-io edited a comment on issue #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner URL: https://github.com/apache/airflow/pull/7090#issuecomment-571428318 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=h1) Report > Merging

[GitHub] [airflow] codecov-io edited a comment on issue #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner

2020-01-06 Thread GitBox
codecov-io edited a comment on issue #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner URL: https://github.com/apache/airflow/pull/7090#issuecomment-571428318 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=h1) Report > Merging

[GitHub] [airflow] codecov-io commented on issue #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner

2020-01-06 Thread GitBox
codecov-io commented on issue #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner URL: https://github.com/apache/airflow/pull/7090#issuecomment-571428318 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=h1) Report > Merging

[GitHub] [airflow] albertusk95 edited a comment on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection

2020-01-06 Thread GitBox
albertusk95 edited a comment on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection URL: https://github.com/apache/airflow/pull/7075#issuecomment-571424197 @tooptoop4 if we don't remove the spark check on line 177, how to use this hook to track driver status deployed on yarn,

[GitHub] [airflow] albertusk95 commented on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection

2020-01-06 Thread GitBox
albertusk95 commented on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection URL: https://github.com/apache/airflow/pull/7075#issuecomment-571424197 @tooptoop4 if we don't remove the spark check on line 177, how to use this hook to track driver status deployed on yarn, mesos,

[jira] [Commented] (AIRFLOW-6495) Load DAG only once when running a task using StandardTaskRunner

2020-01-06 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009323#comment-17009323 ] ASF GitHub Bot commented on AIRFLOW-6495: - mik-laj commented on pull request #7090:

[GitHub] [airflow] mik-laj opened a new pull request #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner

2020-01-06 Thread GitBox
mik-laj opened a new pull request #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner URL: https://github.com/apache/airflow/pull/7090 If the process is created using a fork, it will share all loaded modules that were available before forking. This allows

[jira] [Created] (AIRFLOW-6495) Load DAG only once when running a task using StandardTaskRunner

2020-01-06 Thread Kamil Bregula (Jira)
Kamil Bregula created AIRFLOW-6495: -- Summary: Load DAG only once when running a task using StandardTaskRunner Key: AIRFLOW-6495 URL: https://issues.apache.org/jira/browse/AIRFLOW-6495 Project:

[jira] [Commented] (AIRFLOW-6381) DAG name changes style of some columns but not all

2020-01-06 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009315#comment-17009315 ] ASF GitHub Bot commented on AIRFLOW-6381: - kaxil commented on pull request #6985:

[jira] [Updated] (AIRFLOW-6381) DAG name changes style of some columns but not all

2020-01-06 Thread Kaxil Naik (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-6381: Fix Version/s: (was: 1.10.8) 2.0.0 > DAG name changes style of some

[jira] [Updated] (AIRFLOW-6381) DAG name changes style of some columns but not all

2020-01-06 Thread Kaxil Naik (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-6381: Fix Version/s: (was: 2.0.0) 1.10.8 > DAG name changes style of some

[jira] [Commented] (AIRFLOW-6381) DAG name changes style of some columns but not all

2020-01-06 Thread ASF subversion and git services (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009316#comment-17009316 ] ASF subversion and git services commented on AIRFLOW-6381: -- Commit

[GitHub] [airflow] kaxil merged pull request #6985: [AIRFLOW-6381] Remove styling based on DAG id from DAGs page

2020-01-06 Thread GitBox
kaxil merged pull request #6985: [AIRFLOW-6381] Remove styling based on DAG id from DAGs page URL: https://github.com/apache/airflow/pull/6985 This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] boring-cyborg[bot] commented on issue #6985: [AIRFLOW-6381] Remove styling based on DAG id from DAGs page

2020-01-06 Thread GitBox
boring-cyborg[bot] commented on issue #6985: [AIRFLOW-6381] Remove styling based on DAG id from DAGs page URL: https://github.com/apache/airflow/pull/6985#issuecomment-571409212 Awesome work, congrats on your first merged pull request!

[GitHub] [airflow] mik-laj commented on a change in pull request #6993: [AIRFLOW-4502] new cli command - task_states_for_dag_run

2020-01-06 Thread GitBox
mik-laj commented on a change in pull request #6993: [AIRFLOW-4502] new cli command - task_states_for_dag_run URL: https://github.com/apache/airflow/pull/6993#discussion_r363566924 ## File path: tests/cli/commands/test_task_command.py ## @@ -184,6 +184,20 @@ def

[GitHub] [airflow] yuqian90 commented on a change in pull request #7038: [AIRFLOW-4495] allow externally triggered dags to run for future exec dates

2020-01-06 Thread GitBox
yuqian90 commented on a change in pull request #7038: [AIRFLOW-4495] allow externally triggered dags to run for future exec dates URL: https://github.com/apache/airflow/pull/7038#discussion_r363565941 ## File path: airflow/jobs/scheduler_job.py ## @@ -629,13 +629,18 @@

[GitHub] [airflow] yuqian90 commented on a change in pull request #7038: [AIRFLOW-4495] allow externally triggered dags to run for future exec dates

2020-01-06 Thread GitBox
yuqian90 commented on a change in pull request #7038: [AIRFLOW-4495] allow externally triggered dags to run for future exec dates URL: https://github.com/apache/airflow/pull/7038#discussion_r363566046 ## File path: airflow/ti_deps/deps/runnable_exec_date_dep.py ## @@

[jira] [Commented] (AIRFLOW-2516) Deadlock found when trying to update task_instance table

2020-01-06 Thread Jarek Potiuk (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009301#comment-17009301 ] Jarek Potiuk commented on AIRFLOW-2516: --- Cool!. Keep my fingers crossed.  What's so confusing in

[GitHub] [airflow] mik-laj commented on a change in pull request #6985: [AIRFLOW-6381] Remove styling based on DAG id from DAGs page

2020-01-06 Thread GitBox
mik-laj commented on a change in pull request #6985: [AIRFLOW-6381] Remove styling based on DAG id from DAGs page URL: https://github.com/apache/airflow/pull/6985#discussion_r363566077 ## File path: airflow/www/templates/airflow/dags.html ## @@ -88,7 +88,7 @@ DAGs

[GitHub] [airflow] mik-laj commented on a change in pull request #7086: [AIRFLOW-XXXX] Fix typos and broken links in development docs

2020-01-06 Thread GitBox
mik-laj commented on a change in pull request #7086: [AIRFLOW-] Fix typos and broken links in development docs URL: https://github.com/apache/airflow/pull/7086#discussion_r363565780 ## File path: TESTING.rst ## @@ -278,7 +278,7 @@ Local and Remote Debugging

[jira] [Reopened] (AIRFLOW-4355) Externally triggered DAG is marked as 'success' even if a task has been 'removed'!

2020-01-06 Thread Jarek Potiuk (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk reopened AIRFLOW-4355: --- Resolved by mistake. > Externally triggered DAG is marked as 'success' even if a task has

[GitHub] [airflow] mik-laj commented on a change in pull request #6954: [AIRFLOW-4355] removed task should not lead to dagrun success

2020-01-06 Thread GitBox
mik-laj commented on a change in pull request #6954: [AIRFLOW-4355] removed task should not lead to dagrun success URL: https://github.com/apache/airflow/pull/6954#discussion_r363565513 ## File path: airflow/models/dagrun.py ## @@ -311,32 +313,51 @@ def update_state(self,

[GitHub] [airflow] mik-laj commented on issue #6954: [AIRFLOW-4355] removed task should not lead to dagrun success

2020-01-06 Thread GitBox
mik-laj commented on issue #6954: [AIRFLOW-4355] removed task should not lead to dagrun success URL: https://github.com/apache/airflow/pull/6954#issuecomment-571398463 It works for me. ``` @conf_vars({ ('scheduler', 'removed_tasks_lead_to_dagrun_failure'): 'True'

[GitHub] [airflow] mik-laj edited a comment on issue #6954: [AIRFLOW-4355] removed task should not lead to dagrun success

2020-01-06 Thread GitBox
mik-laj edited a comment on issue #6954: [AIRFLOW-4355] removed task should not lead to dagrun success URL: https://github.com/apache/airflow/pull/6954#issuecomment-571398463 It works for me. ```python @conf_vars({ ('scheduler', 'removed_tasks_lead_to_dagrun_failure'):

[GitHub] [airflow] codecov-io edited a comment on issue #6954: [AIRFLOW-4355] removed task should not lead to dagrun success

2020-01-06 Thread GitBox
codecov-io edited a comment on issue #6954: [AIRFLOW-4355] removed task should not lead to dagrun success URL: https://github.com/apache/airflow/pull/6954#issuecomment-569545898 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6954?src=pr=h1) Report > Merging

[GitHub] [airflow] codecov-io edited a comment on issue #6954: [AIRFLOW-4355] removed task should not lead to dagrun success

2020-01-06 Thread GitBox
codecov-io edited a comment on issue #6954: [AIRFLOW-4355] removed task should not lead to dagrun success URL: https://github.com/apache/airflow/pull/6954#issuecomment-569545898 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6954?src=pr=h1) Report > Merging

[GitHub] [airflow] codecov-io edited a comment on issue #6954: [AIRFLOW-4355] removed task should not lead to dagrun success

2020-01-06 Thread GitBox
codecov-io edited a comment on issue #6954: [AIRFLOW-4355] removed task should not lead to dagrun success URL: https://github.com/apache/airflow/pull/6954#issuecomment-569545898 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6954?src=pr=h1) Report > Merging

[GitHub] [airflow] codecov-io edited a comment on issue #7038: [AIRFLOW-4495] allow externally triggered dags to run for future exec dates

2020-01-06 Thread GitBox
codecov-io edited a comment on issue #7038: [AIRFLOW-4495] allow externally triggered dags to run for future exec dates URL: https://github.com/apache/airflow/pull/7038#issuecomment-570935774 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=h1) Report > Merging

[GitHub] [airflow] codecov-io edited a comment on issue #7038: [AIRFLOW-4495] allow externally triggered dags to run for future exec dates

2020-01-06 Thread GitBox
codecov-io edited a comment on issue #7038: [AIRFLOW-4495] allow externally triggered dags to run for future exec dates URL: https://github.com/apache/airflow/pull/7038#issuecomment-570935774 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=h1) Report > Merging

[GitHub] [airflow] codecov-io edited a comment on issue #7038: [AIRFLOW-4495] allow externally triggered dags to run for future exec dates

2020-01-06 Thread GitBox
codecov-io edited a comment on issue #7038: [AIRFLOW-4495] allow externally triggered dags to run for future exec dates URL: https://github.com/apache/airflow/pull/7038#issuecomment-570935774 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=h1) Report > Merging

[GitHub] [airflow] codecov-io edited a comment on issue #7038: [AIRFLOW-4495] allow externally triggered dags to run for future exec dates

2020-01-06 Thread GitBox
codecov-io edited a comment on issue #7038: [AIRFLOW-4495] allow externally triggered dags to run for future exec dates URL: https://github.com/apache/airflow/pull/7038#issuecomment-570935774 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=h1) Report > Merging

[jira] [Commented] (AIRFLOW-2516) Deadlock found when trying to update task_instance table

2020-01-06 Thread Zhidong (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009281#comment-17009281 ] Zhidong commented on AIRFLOW-2516: -- Hi [~potiuk], no new deadlock observed since last update (4days

[GitHub] [airflow] codecov-io edited a comment on issue #6954: [AIRFLOW-4355] removed task should not lead to dagrun success

2020-01-06 Thread GitBox
codecov-io edited a comment on issue #6954: [AIRFLOW-4355] removed task should not lead to dagrun success URL: https://github.com/apache/airflow/pull/6954#issuecomment-569545898 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6954?src=pr=h1) Report > Merging

[GitHub] [airflow] codecov-io edited a comment on issue #7038: [AIRFLOW-4495] allow externally triggered dags to run for future exec dates

2020-01-06 Thread GitBox
codecov-io edited a comment on issue #7038: [AIRFLOW-4495] allow externally triggered dags to run for future exec dates URL: https://github.com/apache/airflow/pull/7038#issuecomment-570935774 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=h1) Report > :exclamation:

[GitHub] [airflow] zhongjiajie commented on issue #6881: [AIRFLOW-6326] Sort cli commands and arg

2020-01-06 Thread GitBox
zhongjiajie commented on issue #6881: [AIRFLOW-6326] Sort cli commands and arg URL: https://github.com/apache/airflow/pull/6881#issuecomment-571391602 Code now sort by long option and CI green, the behavior like below ```sh usage: airflow tasks run [-h] [--cfg_path CFG_PATH] [-f]

[GitHub] [airflow] tooptoop4 commented on issue #7089: [AIRFLOW-6452] scheduler_job.py - remove excess sleep/log/duration calls

2020-01-06 Thread GitBox
tooptoop4 commented on issue #7089: [AIRFLOW-6452] scheduler_job.py - remove excess sleep/log/duration calls URL: https://github.com/apache/airflow/pull/7089#issuecomment-571391385 before this PR it had 2 sleeps (no need!), it seems the one I removed was there before the

[jira] [Updated] (AIRFLOW-6451) self._print_stat() in dag_processing.py should be skippable by config option

2020-01-06 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated AIRFLOW-6451: -- Description: perf benefit clean up extra poll, logs, typos was:perf benefit > self._print_stat() in

[jira] [Work started] (AIRFLOW-6452) scheduler_job.py - remove excess sleep/log/duration calls

2020-01-06 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-6452 started by t oo. - > scheduler_job.py - remove excess sleep/log/duration calls >

[jira] [Assigned] (AIRFLOW-2516) Deadlock found when trying to update task_instance table

2020-01-06 Thread Zhidong (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhidong reassigned AIRFLOW-2516: Assignee: Jarek Potiuk (was: Zhidong) > Deadlock found when trying to update task_instance

[jira] [Commented] (AIRFLOW-6452) scheduler_job.py - remove excess sleep/log/duration calls

2020-01-06 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009275#comment-17009275 ] ASF GitHub Bot commented on AIRFLOW-6452: - tooptoop4 commented on pull request #7089:

[jira] [Assigned] (AIRFLOW-2516) Deadlock found when trying to update task_instance table

2020-01-06 Thread Zhidong (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhidong reassigned AIRFLOW-2516: Assignee: Zhidong > Deadlock found when trying to update task_instance table >

[GitHub] [airflow] tooptoop4 opened a new pull request #7089: [AIRFLOW-6452] scheduler_job.py - remove excess sleep/log/duration calls

2020-01-06 Thread GitBox
tooptoop4 opened a new pull request #7089: [AIRFLOW-6452] scheduler_job.py - remove excess sleep/log/duration calls URL: https://github.com/apache/airflow/pull/7089 --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) - [ ] Description

[GitHub] [airflow] tooptoop4 edited a comment on issue #6954: [AIRFLOW-4355] removed task should not lead to dagrun success

2020-01-06 Thread GitBox
tooptoop4 edited a comment on issue #6954: [AIRFLOW-4355] removed task should not lead to dagrun success URL: https://github.com/apache/airflow/pull/6954#issuecomment-571383906 > Can you add some tests? it is tough to reproduce the scenario. I tried below but it always says

[GitHub] [airflow] tooptoop4 commented on issue #6954: [AIRFLOW-4355] removed task should not lead to dagrun success

2020-01-06 Thread GitBox
tooptoop4 commented on issue #6954: [AIRFLOW-4355] removed task should not lead to dagrun success URL: https://github.com/apache/airflow/pull/6954#issuecomment-571383906 > Can you add some tests? it is tough to reproduce the scenario. I tried below but it always says DAGRun is

[GitHub] [airflow] tooptoop4 commented on a change in pull request #6954: [AIRFLOW-4355] removed task should not lead to dagrun success

2020-01-06 Thread GitBox
tooptoop4 commented on a change in pull request #6954: [AIRFLOW-4355] removed task should not lead to dagrun success URL: https://github.com/apache/airflow/pull/6954#discussion_r363549714 ## File path: airflow/models/dagrun.py ## @@ -311,32 +313,42 @@ def

  1   2   3   4   5   >