[GitHub] [airflow] ephraimbuddy opened a new pull request #8778: Add separate example DAGs and system tests for google cloud speech

2020-05-07 Thread GitBox
ephraimbuddy opened a new pull request #8778: URL: https://github.com/apache/airflow/pull/8778 --- This PR fixes some of the issues in #8280 - Separated the speech system tests into different modules - Separated the guides into different files - Added different examples for

[airflow] branch master updated (6e4f5fa -> b7566e1)

2020-05-07 Thread kamilbregula
This is an automated email from the ASF dual-hosted git repository. kamilbregula pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from 6e4f5fa [AIRFLOW-4568]The ExternalTaskSensor should be configurable to raise an Airflow Exception in case

[GitHub] [airflow] TeddyHartanto commented on a change in pull request #7735: [AIRFLOW-4549] Allow skipped tasks to satisfy wait_for_downstream

2020-05-07 Thread GitBox
TeddyHartanto commented on a change in pull request #7735: URL: https://github.com/apache/airflow/pull/7735#discussion_r421923733 ## File path: tests/dags/test_issue_1225.py ## @@ -47,12 +47,7 @@ def fail(): dag=dag1, pool='test_backfill_pooled_task_pool',) -# DAG

[jira] [Commented] (AIRFLOW-4549) wait_for_downstream does not respect skipped tasks

2020-05-07 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102228#comment-17102228 ] ASF GitHub Bot commented on AIRFLOW-4549: - TeddyHartanto commented on a change in pull request

[jira] [Commented] (AIRFLOW-4549) wait_for_downstream does not respect skipped tasks

2020-05-07 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102227#comment-17102227 ] ASF GitHub Bot commented on AIRFLOW-4549: - TeddyHartanto commented on a change in pull request

[jira] [Commented] (AIRFLOW-4549) wait_for_downstream does not respect skipped tasks

2020-05-07 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102226#comment-17102226 ] ASF GitHub Bot commented on AIRFLOW-4549: - TeddyHartanto commented on a change in pull request

[GitHub] [airflow] TeddyHartanto commented on a change in pull request #7735: [AIRFLOW-4549] Allow skipped tasks to satisfy wait_for_downstream

2020-05-07 Thread GitBox
TeddyHartanto commented on a change in pull request #7735: URL: https://github.com/apache/airflow/pull/7735#discussion_r421923733 ## File path: tests/dags/test_issue_1225.py ## @@ -47,12 +47,7 @@ def fail(): dag=dag1, pool='test_backfill_pooled_task_pool',) -# DAG

[GitHub] [airflow] TeddyHartanto commented on a change in pull request #7735: [AIRFLOW-4549] Allow skipped tasks to satisfy wait_for_downstream

2020-05-07 Thread GitBox
TeddyHartanto commented on a change in pull request #7735: URL: https://github.com/apache/airflow/pull/7735#discussion_r421923688 ## File path: tests/models/test_dagrun.py ## @@ -552,3 +561,131 @@ def with_all_tasks_removed(dag): dagrun.verify_integrity()

[GitHub] [airflow] schnie opened a new pull request #8777: Add helm chart

2020-05-07 Thread GitBox
schnie opened a new pull request #8777: URL: https://github.com/apache/airflow/pull/8777 This PR adds a default helm chart for Airflow. This chart is based on the one we use at Astronomer to manage hundreds of production deployments. The chart has been cleaned up to remove any

[airflow] tag nightly-master updated (bd29ee3 -> 6e4f5fa)

2020-05-07 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to tag nightly-master in repository https://gitbox.apache.org/repos/asf/airflow.git. *** WARNING: tag nightly-master was modified! *** from bd29ee3 (commit) to 6e4f5fa (commit) from

[airflow] tag nightly-master updated (bd29ee3 -> 6e4f5fa)

2020-05-07 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to tag nightly-master in repository https://gitbox.apache.org/repos/asf/airflow.git. *** WARNING: tag nightly-master was modified! *** from bd29ee3 (commit) to 6e4f5fa (commit) from

[jira] [Commented] (AIRFLOW-4549) wait_for_downstream does not respect skipped tasks

2020-05-07 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102169#comment-17102169 ] ASF GitHub Bot commented on AIRFLOW-4549: - kaxil commented on a change in pull request #7735:

[jira] [Commented] (AIRFLOW-4549) wait_for_downstream does not respect skipped tasks

2020-05-07 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102165#comment-17102165 ] ASF GitHub Bot commented on AIRFLOW-4549: - kaxil commented on a change in pull request #7735:

[GitHub] [airflow] kaxil commented on a change in pull request #7735: [AIRFLOW-4549] Allow skipped tasks to satisfy wait_for_downstream

2020-05-07 Thread GitBox
kaxil commented on a change in pull request #7735: URL: https://github.com/apache/airflow/pull/7735#discussion_r421894092 ## File path: tests/models/test_dagrun.py ## @@ -552,3 +561,131 @@ def with_all_tasks_removed(dag): dagrun.verify_integrity()

[GitHub] [airflow] kaxil commented on a change in pull request #7735: [AIRFLOW-4549] Allow skipped tasks to satisfy wait_for_downstream

2020-05-07 Thread GitBox
kaxil commented on a change in pull request #7735: URL: https://github.com/apache/airflow/pull/7735#discussion_r421893773 ## File path: tests/models/test_dagrun.py ## @@ -552,3 +561,131 @@ def with_all_tasks_removed(dag): dagrun.verify_integrity()

[jira] [Commented] (AIRFLOW-4549) wait_for_downstream does not respect skipped tasks

2020-05-07 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102164#comment-17102164 ] ASF GitHub Bot commented on AIRFLOW-4549: - kaxil commented on a change in pull request #7735:

[GitHub] [airflow] kaxil commented on a change in pull request #7735: [AIRFLOW-4549] Allow skipped tasks to satisfy wait_for_downstream

2020-05-07 Thread GitBox
kaxil commented on a change in pull request #7735: URL: https://github.com/apache/airflow/pull/7735#discussion_r421893536 ## File path: tests/dags/test_issue_1225.py ## @@ -47,12 +47,7 @@ def fail(): dag=dag1, pool='test_backfill_pooled_task_pool',) -# DAG tests

[jira] [Commented] (AIRFLOW-4543) Update slack operator to support slackclient v2

2020-05-07 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-4543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102162#comment-17102162 ] ASF GitHub Bot commented on AIRFLOW-4543: - kaxil commented on a change in pull request #5519:

[GitHub] [airflow] kaxil commented on a change in pull request #5519: [AIRFLOW-4543] Update slack operator to support slackclient v2

2020-05-07 Thread GitBox
kaxil commented on a change in pull request #5519: URL: https://github.com/apache/airflow/pull/5519#discussion_r421892844 ## File path: tests/providers/slack/hooks/test_slack.py ## @@ -19,82 +19,154 @@ import unittest import mock +from slack.errors import SlackApiError

[GitHub] [airflow] kaxil commented on a change in pull request #5519: [AIRFLOW-4543] Update slack operator to support slackclient v2

2020-05-07 Thread GitBox
kaxil commented on a change in pull request #5519: URL: https://github.com/apache/airflow/pull/5519#discussion_r421892088 ## File path: airflow/providers/slack/hooks/slack.py ## @@ -27,40 +28,88 @@ # noinspection PyAbstractClass class SlackHook(BaseHook): """ +

[jira] [Commented] (AIRFLOW-4543) Update slack operator to support slackclient v2

2020-05-07 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-4543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102157#comment-17102157 ] ASF GitHub Bot commented on AIRFLOW-4543: - kaxil commented on a change in pull request #5519:

[jira] [Commented] (AIRFLOW-4543) Update slack operator to support slackclient v2

2020-05-07 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-4543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102156#comment-17102156 ] ASF GitHub Bot commented on AIRFLOW-4543: - kaxil commented on a change in pull request #5519:

[GitHub] [airflow] kaxil commented on a change in pull request #5519: [AIRFLOW-4543] Update slack operator to support slackclient v2

2020-05-07 Thread GitBox
kaxil commented on a change in pull request #5519: URL: https://github.com/apache/airflow/pull/5519#discussion_r421892012 ## File path: tests/providers/slack/hooks/test_slack.py ## @@ -19,82 +19,154 @@ import unittest import mock +from slack.errors import SlackApiError

[GitHub] [airflow] jaketf commented on issue #8673: Data Fusion Hook Start pipeline will succeed before pipeline is in RUNNING state

2020-05-07 Thread GitBox
jaketf commented on issue #8673: URL: https://github.com/apache/airflow/issues/8673#issuecomment-625581362 I will reach out to my team and see if there are folks interested in joining airflow community by patching this This

[GitHub] [airflow] mik-laj commented on issue #8673: Data Fusion Hook Start pipeline will succeed before pipeline is in RUNNING state

2020-05-07 Thread GitBox
mik-laj commented on issue #8673: URL: https://github.com/apache/airflow/issues/8673#issuecomment-625579505 We also currently have priorities set for other tasks. If it is very, we can change our plans. If it is very important, we can change our plans, but we would prefer to avoid it.

[jira] [Work started] (AIRFLOW-3369) Un-pausing a DAG with catchup =False creates an extra DAG run (1.10)

2020-05-07 Thread Kaxil Naik (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-3369 started by Kaxil Naik. --- > Un-pausing a DAG with catchup =False creates an extra DAG run (1.10) >

[jira] [Assigned] (AIRFLOW-3369) Un-pausing a DAG with catchup =False creates an extra DAG run (1.10)

2020-05-07 Thread Kaxil Naik (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik reassigned AIRFLOW-3369: --- Assignee: Kaxil Naik > Un-pausing a DAG with catchup =False creates an extra DAG run

[jira] [Commented] (AIRFLOW-3369) Un-pausing a DAG with catchup =False creates an extra DAG run (1.10)

2020-05-07 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102146#comment-17102146 ] ASF GitHub Bot commented on AIRFLOW-3369: - kaxil opened a new pull request #8776: URL:

[GitHub] [airflow] kaxil opened a new pull request #8776: WIP: [AIRFLOW-3369] BugFix: Unpausing a DAG with catchup=False creates an extra DAG run

2020-05-07 Thread GitBox
kaxil opened a new pull request #8776: URL: https://github.com/apache/airflow/pull/8776 https://issues.apache.org/jira/browse/AIRFLOW-3369 --- Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Unit

[GitHub] [airflow] jaketf commented on issue #8673: Data Fusion Hook Start pipeline will succeed before pipeline is in RUNNING state

2020-05-07 Thread GitBox
jaketf commented on issue #8673: URL: https://github.com/apache/airflow/issues/8673#issuecomment-625578250 CDAP / Data Fusion is not a priority for me right now. I will not be able to find cycles for this in the next few weeks. Perhaps you could find a user who cares about this

[jira] [Commented] (AIRFLOW-7104) Add Secret backend for GCP Secrets Manager

2020-05-07 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-7104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102098#comment-17102098 ] ASF GitHub Bot commented on AIRFLOW-7104: - sethvargo commented on a change in pull request

[GitHub] [airflow] sethvargo commented on a change in pull request #7795: [AIRFLOW-7104] Add Secret backend for GCP Secrets Manager

2020-05-07 Thread GitBox
sethvargo commented on a change in pull request #7795: URL: https://github.com/apache/airflow/pull/7795#discussion_r421850039 ## File path: airflow/providers/google/cloud/secrets/secrets_manager.py ## @@ -0,0 +1,132 @@ +# Licensed to the Apache Software Foundation (ASF) under

[GitHub] [airflow] ashb commented on a change in pull request #8739: Test that DagFileProcessor can operator against on a Serialized DAG

2020-05-07 Thread GitBox
ashb commented on a change in pull request #8739: URL: https://github.com/apache/airflow/pull/8739#discussion_r421844181 ## File path: tests/jobs/test_scheduler_job.py ## @@ -2253,7 +2307,17 @@ def evaluate_dagrun( self.null_exec.mock_task_fail(dag_id, tid,

[GitHub] [airflow] ashb commented on a change in pull request #8775: Correctly restore upstream_task_ids when deserializing Operators

2020-05-07 Thread GitBox
ashb commented on a change in pull request #8775: URL: https://github.com/apache/airflow/pull/8775#discussion_r421843524 ## File path: airflow/providers/google/cloud/example_dags/example_gcs.py ## @@ -126,8 +126,8 @@ ) # [START howto_operator_gcs_delete_bucket] -

[GitHub] [airflow] ashb opened a new pull request #8775: Correctly restore upstream_task_ids when deserializing Operators

2020-05-07 Thread GitBox
ashb opened a new pull request #8775: URL: https://github.com/apache/airflow/pull/8775 This test exposed a bug in one of the example dags, that wasn't caught by #6549. That will be a fixed in a separate issue, but it caused the round-trip tests to fail here Should fix #8720, (but

[GitHub] [airflow] ashb commented on issue #8703: Support for set in XCom serialization

2020-05-07 Thread GitBox
ashb commented on issue #8703: URL: https://github.com/apache/airflow/issues/8703#issuecomment-625537935 XCom used to be pickle by default (and still is on 1.10?), that'll be why it has gone un-noticed for so long. This is

[GitHub] [airflow] ashb commented on a change in pull request #8754: Add SQL query tracking for pytest

2020-05-07 Thread GitBox
ashb commented on a change in pull request #8754: URL: https://github.com/apache/airflow/pull/8754#discussion_r421835369 ## File path: scripts/perf/perf_kit/sqlalchemy.py ## @@ -15,31 +15,59 @@ # specific language governing permissions and limitations # under the License.

[GitHub] [airflow] turbaszek edited a comment on pull request #8652: [AIP-31] Implement XComArg model to functionally pass output from one operator to the next

2020-05-07 Thread GitBox
turbaszek edited a comment on pull request #8652: URL: https://github.com/apache/airflow/pull/8652#issuecomment-625532608 @jonathanshir it seems that some of the tests are flaky or something: ``` File "/opt/airflow/tests/models/test_xcom_arg.py", line 130 in test_xcom_pass_to_op

[GitHub] [airflow] turbaszek commented on pull request #8652: [AIP-31] Implement XComArg model to functionally pass output from one operator to the next

2020-05-07 Thread GitBox
turbaszek commented on pull request #8652: URL: https://github.com/apache/airflow/pull/8652#issuecomment-625532608 It seems that some of the test are flaky or something: ``` File "/opt/airflow/tests/models/test_xcom_arg.py", line 130 in test_xcom_pass_to_op ``` In general, we

[GitHub] [airflow] ashb commented on issue #8774: Move singularity out of main CI tests and into separate docker image/system test

2020-05-07 Thread GitBox
ashb commented on issue #8774: URL: https://github.com/apache/airflow/issues/8774#issuecomment-625532382 Wait, I've just looked at the singularity operator tests - it already seems to mock it, so does anyone know why we include this in our CI image? @potiuk ?

[GitHub] [airflow] ashb opened a new issue #8774: Move singularity out of main CI tests and into separate docker image/system test

2020-05-07 Thread GitBox
ashb opened a new issue #8774: URL: https://github.com/apache/airflow/issues/8774 A CI build failed because we couldn't download a release from https://github.com/sylabs/singularity -- which while it was a random network blib, there is no _need_ for the tests for the singularity operators

[GitHub] [airflow] ashb commented on pull request #8728: Show Deprecation warning on duplicate Task ids

2020-05-07 Thread GitBox
ashb commented on pull request #8728: URL: https://github.com/apache/airflow/pull/8728#issuecomment-625528936 Only cos it's related to this PR, not that it's a bug with this PR And I think I've just found a problem in our example dags caused by us using `!=` in master:

[GitHub] [airflow] ashb commented on pull request #8718: Set store_serialized_dags from config in UI

2020-05-07 Thread GitBox
ashb commented on pull request #8718: URL: https://github.com/apache/airflow/pull/8718#issuecomment-625526747 No problem @anitakar, it's good to check these things, and it's often easier to talk in code! (And sorry for being a bit curt!)

[GitHub] [airflow] houqp opened a new pull request #8773: fix typing errors reported by dmypy

2020-05-07 Thread GitBox
houqp opened a new pull request #8773: URL: https://github.com/apache/airflow/pull/8773 fixes 2 new errors reported by dmypy. --- Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Unit tests coverage

[GitHub] [airflow] ashb commented on a change in pull request #8772: Correctly store non-default Nones in serialized tasks/dags

2020-05-07 Thread GitBox
ashb commented on a change in pull request #8772: URL: https://github.com/apache/airflow/pull/8772#discussion_r421826962 ## File path: airflow/serialization/serialized_objects.py ## @@ -395,6 +411,59 @@ def deserialize_operator(cls, encoded_op: Dict[str, Any]) ->

[GitHub] [airflow] ashb commented on a change in pull request #8772: Correctly store non-default Nones in serialized tasks/dags

2020-05-07 Thread GitBox
ashb commented on a change in pull request #8772: URL: https://github.com/apache/airflow/pull/8772#discussion_r421826729 ## File path: airflow/serialization/serialized_objects.py ## @@ -395,6 +411,59 @@ def deserialize_operator(cls, encoded_op: Dict[str, Any]) ->

[GitHub] [airflow] ashb opened a new pull request #8772: Correctly store non-default Nones in serialized tasks/dags

2020-05-07 Thread GitBox
ashb opened a new pull request #8772: URL: https://github.com/apache/airflow/pull/8772 The default schedule_interval for a DAG is `@daily`, so `schedule_interval=None` is actually not the default, but we were not storing _any_ null attributes previously. This meant that upon

[jira] [Commented] (AIRFLOW-7104) Add Secret backend for GCP Secrets Manager

2020-05-07 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-7104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102064#comment-17102064 ] ASF GitHub Bot commented on AIRFLOW-7104: - kaxil commented on a change in pull request #7795:

[GitHub] [airflow] kaxil commented on a change in pull request #7795: [AIRFLOW-7104] Add Secret backend for GCP Secrets Manager

2020-05-07 Thread GitBox
kaxil commented on a change in pull request #7795: URL: https://github.com/apache/airflow/pull/7795#discussion_r421825954 ## File path: airflow/providers/google/cloud/secrets/secrets_manager.py ## @@ -0,0 +1,132 @@ +# Licensed to the Apache Software Foundation (ASF) under one

[GitHub] [airflow] Acehaidrey commented on pull request #8702: Add context to execution_date_fn in ExternalTaskSensor

2020-05-07 Thread GitBox
Acehaidrey commented on pull request #8702: URL: https://github.com/apache/airflow/pull/8702#issuecomment-625516604 @jhtimmins @dimberman can you please comment when you have a minute? This is an automated message from

[GitHub] [airflow] mik-laj commented on a change in pull request #7795: [AIRFLOW-7104] Add Secret backend for GCP Secrets Manager

2020-05-07 Thread GitBox
mik-laj commented on a change in pull request #7795: URL: https://github.com/apache/airflow/pull/7795#discussion_r421816660 ## File path: airflow/providers/google/cloud/secrets/secrets_manager.py ## @@ -0,0 +1,132 @@ +# Licensed to the Apache Software Foundation (ASF) under

[jira] [Commented] (AIRFLOW-7104) Add Secret backend for GCP Secrets Manager

2020-05-07 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-7104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102054#comment-17102054 ] ASF GitHub Bot commented on AIRFLOW-7104: - mik-laj commented on a change in pull request #7795:

[jira] [Commented] (AIRFLOW-7104) Add Secret backend for GCP Secrets Manager

2020-05-07 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-7104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102055#comment-17102055 ] ASF GitHub Bot commented on AIRFLOW-7104: - mik-laj commented on a change in pull request #7795:

[jira] [Commented] (AIRFLOW-7104) Add Secret backend for GCP Secrets Manager

2020-05-07 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-7104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102053#comment-17102053 ] ASF GitHub Bot commented on AIRFLOW-7104: - mik-laj commented on a change in pull request #7795:

[GitHub] [airflow] mik-laj commented on a change in pull request #7795: [AIRFLOW-7104] Add Secret backend for GCP Secrets Manager

2020-05-07 Thread GitBox
mik-laj commented on a change in pull request #7795: URL: https://github.com/apache/airflow/pull/7795#discussion_r421816660 ## File path: airflow/providers/google/cloud/secrets/secrets_manager.py ## @@ -0,0 +1,132 @@ +# Licensed to the Apache Software Foundation (ASF) under

[GitHub] [airflow] mik-laj commented on a change in pull request #7795: [AIRFLOW-7104] Add Secret backend for GCP Secrets Manager

2020-05-07 Thread GitBox
mik-laj commented on a change in pull request #7795: URL: https://github.com/apache/airflow/pull/7795#discussion_r421816660 ## File path: airflow/providers/google/cloud/secrets/secrets_manager.py ## @@ -0,0 +1,132 @@ +# Licensed to the Apache Software Foundation (ASF) under

[GitHub] [airflow] Acehaidrey commented on pull request #8680: Add metric for start/end task run

2020-05-07 Thread GitBox
Acehaidrey commented on pull request #8680: URL: https://github.com/apache/airflow/pull/8680#issuecomment-625514762 @jhtimmins when you get a chance This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] houqp opened a new pull request #8771: add metric for monitoring email notification failure

2020-05-07 Thread GitBox
houqp opened a new pull request #8771: URL: https://github.com/apache/airflow/pull/8771 This should help catch unexpected email notification failures. We need to monitor the monitoring system :) --- Make sure to mark the boxes below before creating PR: [x] - [x]

[jira] [Commented] (AIRFLOW-7104) Add Secret backend for GCP Secrets Manager

2020-05-07 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-7104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102034#comment-17102034 ] ASF GitHub Bot commented on AIRFLOW-7104: - sethvargo commented on a change in pull request

[GitHub] [airflow] sethvargo commented on a change in pull request #7795: [AIRFLOW-7104] Add Secret backend for GCP Secrets Manager

2020-05-07 Thread GitBox
sethvargo commented on a change in pull request #7795: URL: https://github.com/apache/airflow/pull/7795#discussion_r421795771 ## File path: airflow/providers/google/cloud/secrets/secrets_manager.py ## @@ -0,0 +1,132 @@ +# Licensed to the Apache Software Foundation (ASF) under

[GitHub] [airflow] casassg commented on a change in pull request #8652: [AIP-31] Implement XComArg model to functionally pass output from one operator to the next

2020-05-07 Thread GitBox
casassg commented on a change in pull request #8652: URL: https://github.com/apache/airflow/pull/8652#discussion_r421782519 ## File path: airflow/models/xcom_arg.py ## @@ -0,0 +1,147 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor

[GitHub] [airflow] kaxil commented on issue #8760: Webserver becomes unusable after 100,000 tasks were completed

2020-05-07 Thread GitBox
kaxil commented on issue #8760: URL: https://github.com/apache/airflow/issues/8760#issuecomment-625474431 Is there a particular endpoint that is slow in the Webserver ? This is an automated message from the Apache Git

[GitHub] [airflow] mik-laj commented on pull request #8754: Add SQL query tracking for pytest

2020-05-07 Thread GitBox
mik-laj commented on pull request #8754: URL: https://github.com/apache/airflow/pull/8754#issuecomment-625468574 @kaxil @ashb and our pytest expert - @turbaszek Can I ask for look? All non-quarantined check are green.

[GitHub] [airflow] mik-laj commented on a change in pull request #8754: Add SQL query tracking for pytest

2020-05-07 Thread GitBox
mik-laj commented on a change in pull request #8754: URL: https://github.com/apache/airflow/pull/8754#discussion_r421760872 ## File path: TESTING.rst ## @@ -902,6 +902,36 @@ You should also consider running it with ``restart`` command when you change the This will clean-up

[GitHub] [airflow] mik-laj commented on a change in pull request #8754: Add SQL query tracking for pytest

2020-05-07 Thread GitBox
mik-laj commented on a change in pull request #8754: URL: https://github.com/apache/airflow/pull/8754#discussion_r421760814 ## File path: scripts/perf/perf_kit/sqlalchemy.py ## @@ -52,28 +80,40 @@ def after_cursor_execute(conn, cursor, statement, parameters, context,

[jira] [Commented] (AIRFLOW-249) Refactor the SLA mechanism

2020-05-07 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101971#comment-17101971 ] ASF GitHub Bot commented on AIRFLOW-249: houqp commented on pull request #8545: URL:

[GitHub] [airflow] houqp commented on pull request #8545: [AIRFLOW-249] Refactor the SLA mechanism (Continuation from #3584 )

2020-05-07 Thread GitBox
houqp commented on pull request #8545: URL: https://github.com/apache/airflow/pull/8545#issuecomment-625434121 > My conclusion is that option 1 is a better trade-off, because one has to go through all TIs in a DagRun to determine if a DR can be free from further checking (e.g., if a DR

[GitHub] [airflow] anitakar commented on pull request #8718: Set store_serialized_dags from config in UI

2020-05-07 Thread GitBox
anitakar commented on pull request #8718: URL: https://github.com/apache/airflow/pull/8718#issuecomment-625428708 Closing the issue as only difference seems to be calling set_paused and its implementation has changed. Sorry for taking your time without checking everything first.

[jira] [Commented] (AIRFLOW-249) Refactor the SLA mechanism

2020-05-07 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101952#comment-17101952 ] ASF GitHub Bot commented on AIRFLOW-249: houqp commented on a change in pull request #8545: URL:

[GitHub] [airflow] houqp commented on a change in pull request #8545: [AIRFLOW-249] Refactor the SLA mechanism (Continuation from #3584 )

2020-05-07 Thread GitBox
houqp commented on a change in pull request #8545: URL: https://github.com/apache/airflow/pull/8545#discussion_r421708159 ## File path: airflow/models/baseoperator.py ## @@ -392,10 +442,79 @@ def __init__( % (self.task_id, dag.dag_id))

[GitHub] [airflow] boring-cyborg[bot] commented on issue #8770: Airflow 1.10.7 logs from S3 won't load, just hanging

2020-05-07 Thread GitBox
boring-cyborg[bot] commented on issue #8770: URL: https://github.com/apache/airflow/issues/8770#issuecomment-625422513 Thanks for opening your first issue here! Be sure to follow the issue template! This is an automated

[GitHub] [airflow] aarrtteemmuuss opened a new issue #8770: Airflow 1.10.7 logs from S3 won't load, just hanging

2020-05-07 Thread GitBox
aarrtteemmuuss opened a new issue #8770: URL: https://github.com/apache/airflow/issues/8770 **Apache Airflow version**: 1.10.7 **Environment**: python 3.7, running locally **What happened**: I have following configuration in airflow.cfg: `remote_logging = True`

[GitHub] [airflow] Sinsin1367 commented on pull request #8734: Added optional logging for pod container statuses once a pod fails. T…

2020-05-07 Thread GitBox
Sinsin1367 commented on pull request #8734: URL: https://github.com/apache/airflow/pull/8734#issuecomment-625412537 @ashb @chrismclennon I would appreciate if you guys take a look and share your thoughts. This is an

[GitHub] [airflow] anitakar opened a new pull request #8769: Set store_serialized_dags from config instead of defaulting it to false

2020-05-07 Thread GitBox
anitakar opened a new pull request #8769: URL: https://github.com/apache/airflow/pull/8769 Fix running dag from non-RBAC (old, deprecated) UI This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] anitakar commented on a change in pull request #8764: Store dags cleanup 1 10

2020-05-07 Thread GitBox
anitakar commented on a change in pull request #8764: URL: https://github.com/apache/airflow/pull/8764#discussion_r421691736 ## File path: airflow/www_rbac/views.py ## @@ -564,7 +568,8 @@ def dag_details(self, session=None): dag_id = request.args.get('dag_id')

[GitHub] [airflow] anitakar commented on a change in pull request #8764: Store dags cleanup 1 10

2020-05-07 Thread GitBox
anitakar commented on a change in pull request #8764: URL: https://github.com/apache/airflow/pull/8764#discussion_r421691284 ## File path: airflow/www/views.py ## @@ -94,7 +94,16 @@ UTF8_READER = codecs.getreader('utf-8') -dagbag = models.DagBag(settings.DAGS_FOLDER,

[jira] [Commented] (AIRFLOW-2310) Enable AWS Glue Job Integration

2020-05-07 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101913#comment-17101913 ] ASF GitHub Bot commented on AIRFLOW-2310: - paulsyl commented on pull request #6007: URL:

[GitHub] [airflow] paulsyl commented on pull request #6007: [AIRFLOW-2310] Enable AWS Glue Job Integration

2020-05-07 Thread GitBox
paulsyl commented on pull request #6007: URL: https://github.com/apache/airflow/pull/6007#issuecomment-625403415 > @paulsyl could you possibly share your local Glue implementation (custom operator, hook, etc) as a Gist or otherwise that we could run in the meantime? If so I'd be happy to

[GitHub] [airflow] XD-DENG commented on pull request #8742: Avoid color info in response of /dag_stats & /task_stats

2020-05-07 Thread GitBox
XD-DENG commented on pull request #8742: URL: https://github.com/apache/airflow/pull/8742#issuecomment-625392001 Rebased to the latest master This is an automated message from the Apache Git Service. To respond to the

[jira] [Commented] (AIRFLOW-2310) Enable AWS Glue Job Integration

2020-05-07 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101899#comment-17101899 ] ASF GitHub Bot commented on AIRFLOW-2310: - pdeardorff-r7 commented on pull request #6007: URL:

[GitHub] [airflow] pdeardorff-r7 commented on pull request #6007: [AIRFLOW-2310] Enable AWS Glue Job Integration

2020-05-07 Thread GitBox
pdeardorff-r7 commented on pull request #6007: URL: https://github.com/apache/airflow/pull/6007#issuecomment-625389437 @paulsyl could you possibly share your local Glue implementation (custom operator, hook, etc) as a Gist or otherwise that we could run in the meantime? If so I'd be

[airflow] branch v1-10-test updated: Show Deprecation warning on duplicate Task ids (#8728)

2020-05-07 Thread kaxilnaik
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/v1-10-test by this push: new 1ef5f13 Show Deprecation warning

[GitHub] [airflow] mik-laj opened a new pull request #8768: [POC] Mark keywords-only arguments in method signatures

2020-05-07 Thread GitBox
mik-laj opened a new pull request #8768: URL: https://github.com/apache/airflow/pull/8768 Hello, I am not sure if this feature is supported by all our tools, especially flake8, so I run the CI task to check it. Best regards, Kamil --- Make sure to mark the boxes

[GitHub] [airflow] ipeluffo edited a comment on issue #8311: Task Duration Missing from the Graph View Tool-tip for Running Tasks

2020-05-07 Thread GitBox
ipeluffo edited a comment on issue #8311: URL: https://github.com/apache/airflow/issues/8311#issuecomment-625382511 I noticed this issue also in (some) tasks on finished DAGs. Has anyone else noticed this? ![Screenshot 2020-05-07 at 18 10

[GitHub] [airflow] ipeluffo commented on issue #8311: Task Duration Missing from the Graph View Tool-tip for Running Tasks

2020-05-07 Thread GitBox
ipeluffo commented on issue #8311: URL: https://github.com/apache/airflow/issues/8311#issuecomment-625382511 I noticed this issue also in (some) tasks on finished DAGs. Has anyone else noticed this? This is an automated

[GitHub] [airflow] anitakar commented on pull request #8718: Set store_serialized_dags from config in UI

2020-05-07 Thread GitBox
anitakar commented on pull request #8718: URL: https://github.com/apache/airflow/pull/8718#issuecomment-625364951 > > Hi, > > I will collectively answer the questions here. > > Many of the changes that I have made have been a result of setting store_serialized_dags from conf in

[GitHub] [airflow] kaxil commented on a change in pull request #8746: Remove old airflow logger causing side effects in tests

2020-05-07 Thread GitBox
kaxil commented on a change in pull request #8746: URL: https://github.com/apache/airflow/pull/8746#discussion_r421625121 ## File path: tests/test_logging_config.py ## @@ -97,6 +93,33 @@ SETTINGS_DEFAULT_NAME = 'custom_airflow_local_settings' +def reset_logging(): +

[GitHub] [airflow] ashb commented on a change in pull request #8746: Remove old airflow logger causing side effects in tests

2020-05-07 Thread GitBox
ashb commented on a change in pull request #8746: URL: https://github.com/apache/airflow/pull/8746#discussion_r421621414 ## File path: tests/test_logging_config.py ## @@ -97,6 +93,33 @@ SETTINGS_DEFAULT_NAME = 'custom_airflow_local_settings' +def reset_logging(): +

[GitHub] [airflow] turbaszek commented on a change in pull request #8651: [AIP-31] [AIRFLOW-8058] Retrieve current execution context without altering function's signature

2020-05-07 Thread GitBox
turbaszek commented on a change in pull request #8651: URL: https://github.com/apache/airflow/pull/8651#discussion_r421604020 ## File path: docs/howto/use-additional-execute-contextmanager.rst ## @@ -0,0 +1,47 @@ + .. Licensed to the Apache Software Foundation (ASF) under one

[GitHub] [airflow] turbaszek commented on a change in pull request #8651: [AIP-31] [AIRFLOW-8058] Retrieve current execution context without altering function's signature

2020-05-07 Thread GitBox
turbaszek commented on a change in pull request #8651: URL: https://github.com/apache/airflow/pull/8651#discussion_r421602145 ## File path: airflow/models/taskinstance.py ## @@ -1110,6 +1116,27 @@ def signal_handler(signum, frame): session.merge(self)

[GitHub] [airflow] potiuk opened a new pull request #8767: Backport packages are renamed to include backport in their name

2020-05-07 Thread GitBox
potiuk opened a new pull request #8767: URL: https://github.com/apache/airflow/pull/8767 --- Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Unit tests coverage for changes (not needed for documentation

[GitHub] [airflow] jonathanshir commented on a change in pull request #8651: [AIP-31] [AIRFLOW-8058] Retrieve current execution context without altering function's signature

2020-05-07 Thread GitBox
jonathanshir commented on a change in pull request #8651: URL: https://github.com/apache/airflow/pull/8651#discussion_r421592500 ## File path: airflow/config_templates/config.yml ## @@ -366,6 +366,17 @@ type: string example: "path.to.CustomXCom" default:

[jira] [Commented] (AIRFLOW-4543) Update slack operator to support slackclient v2

2020-05-07 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-4543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101727#comment-17101727 ] ASF GitHub Bot commented on AIRFLOW-4543: - serkef commented on a change in pull request #5519:

[GitHub] [airflow] serkef commented on a change in pull request #5519: [AIRFLOW-4543] Update slack operator to support slackclient v2

2020-05-07 Thread GitBox
serkef commented on a change in pull request #5519: URL: https://github.com/apache/airflow/pull/5519#discussion_r421559066 ## File path: tests/providers/slack/hooks/test_slack.py ## @@ -19,82 +19,154 @@ import unittest import mock +from slack.errors import SlackApiError

[jira] [Commented] (AIRFLOW-4549) wait_for_downstream does not respect skipped tasks

2020-05-07 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101679#comment-17101679 ] ASF GitHub Bot commented on AIRFLOW-4549: - TeddyHartanto commented on a change in pull request

[GitHub] [airflow] TeddyHartanto commented on a change in pull request #7735: [AIRFLOW-4549] Allow skipped tasks to satisfy wait_for_downstream

2020-05-07 Thread GitBox
TeddyHartanto commented on a change in pull request #7735: URL: https://github.com/apache/airflow/pull/7735#discussion_r421504185 ## File path: tests/dags/test_issue_1225.py ## @@ -47,12 +47,7 @@ def fail(): dag=dag1, pool='test_backfill_pooled_task_pool',) -# DAG

[jira] [Commented] (AIRFLOW-4549) wait_for_downstream does not respect skipped tasks

2020-05-07 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101675#comment-17101675 ] ASF GitHub Bot commented on AIRFLOW-4549: - TeddyHartanto commented on a change in pull request

[jira] [Commented] (AIRFLOW-4549) wait_for_downstream does not respect skipped tasks

2020-05-07 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101672#comment-17101672 ] ASF GitHub Bot commented on AIRFLOW-4549: - TeddyHartanto commented on a change in pull request

[GitHub] [airflow] TeddyHartanto commented on a change in pull request #7735: [AIRFLOW-4549] Allow skipped tasks to satisfy wait_for_downstream

2020-05-07 Thread GitBox
TeddyHartanto commented on a change in pull request #7735: URL: https://github.com/apache/airflow/pull/7735#discussion_r421499807 ## File path: tests/dags/test_issue_1225.py ## @@ -47,12 +47,7 @@ def fail(): dag=dag1, pool='test_backfill_pooled_task_pool',) -# DAG

[GitHub] [airflow] TeddyHartanto commented on a change in pull request #7735: [AIRFLOW-4549] Allow skipped tasks to satisfy wait_for_downstream

2020-05-07 Thread GitBox
TeddyHartanto commented on a change in pull request #7735: URL: https://github.com/apache/airflow/pull/7735#discussion_r421499807 ## File path: tests/dags/test_issue_1225.py ## @@ -47,12 +47,7 @@ def fail(): dag=dag1, pool='test_backfill_pooled_task_pool',) -# DAG

  1   2   >