[GitHub] [airflow] mik-laj commented on pull request #8227: Add run_type to DagRun

2020-05-10 Thread GitBox
mik-laj commented on pull request #8227: URL: https://github.com/apache/airflow/pull/8227#issuecomment-626477268 @turbaszek It seems to me we should use MANUAL run type here. This is an automated message from the Apache Git

[GitHub] [airflow] mik-laj commented on a change in pull request #8631: Refactor BigQuery hook methods to use python library

2020-05-10 Thread GitBox
mik-laj commented on a change in pull request #8631: URL: https://github.com/apache/airflow/pull/8631#discussion_r422786834 ## File path: airflow/providers/google/cloud/hooks/bigquery.py ## @@ -1100,42 +1157,28 @@ def run_table_upsert(self, dataset_id: str, table_resource:

[GitHub] [airflow] mik-laj commented on a change in pull request #8631: Refactor BigQuery hook methods to use python library

2020-05-10 Thread GitBox
mik-laj commented on a change in pull request #8631: URL: https://github.com/apache/airflow/pull/8631#discussion_r422786200 ## File path: airflow/providers/google/cloud/hooks/bigquery.py ## @@ -1036,58 +1089,62 @@ def run_grant_dataset_view_access(self, :type

[GitHub] [airflow] mik-laj commented on a change in pull request #8631: Refactor BigQuery hook methods to use python library

2020-05-10 Thread GitBox
mik-laj commented on a change in pull request #8631: URL: https://github.com/apache/airflow/pull/8631#discussion_r422785678 ## File path: airflow/providers/google/cloud/hooks/bigquery.py ## @@ -512,84 +555,40 @@ def create_external_table(self, # pylint:

[GitHub] [airflow] mik-laj commented on a change in pull request #8631: Refactor BigQuery hook methods to use python library

2020-05-10 Thread GitBox
mik-laj commented on a change in pull request #8631: URL: https://github.com/apache/airflow/pull/8631#discussion_r422785678 ## File path: airflow/providers/google/cloud/hooks/bigquery.py ## @@ -512,84 +555,40 @@ def create_external_table(self, # pylint:

[GitHub] [airflow] mik-laj commented on a change in pull request #8631: Refactor BigQuery hook methods to use python library

2020-05-10 Thread GitBox
mik-laj commented on a change in pull request #8631: URL: https://github.com/apache/airflow/pull/8631#discussion_r422785227 ## File path: airflow/providers/google/cloud/hooks/bigquery.py ## @@ -598,33 +597,86 @@ def create_external_table(self, # pylint:

[GitHub] [airflow] mik-laj edited a comment on pull request #8768: [POC] Mark keywords-only arguments in hook method signatures

2020-05-10 Thread GitBox
mik-laj edited a comment on pull request #8768: URL: https://github.com/apache/airflow/pull/8768#issuecomment-625714537 @ashb What do you mean? These methods now must use keyword arguments only.

[GitHub] [airflow] mik-laj commented on a change in pull request #8807: Added automated release notes generation for backport operators

2020-05-10 Thread GitBox
mik-laj commented on a change in pull request #8807: URL: https://github.com/apache/airflow/pull/8807#discussion_r422776607 ## File path: backport_packages/setup_backport_packages.py ## @@ -422,13 +371,376 @@ def usage(): print(text) print() print("You can

[GitHub] [airflow] mik-laj commented on a change in pull request #8807: Added automated release notes generation for backport operators

2020-05-10 Thread GitBox
mik-laj commented on a change in pull request #8807: URL: https://github.com/apache/airflow/pull/8807#discussion_r422776394 ## File path: backport_packages/setup_backport_packages.py ## @@ -422,13 +371,376 @@ def usage(): print(text) print() print("You can

[GitHub] [airflow] mik-laj commented on a change in pull request #8807: Added automated release notes generation for backport operators

2020-05-10 Thread GitBox
mik-laj commented on a change in pull request #8807: URL: https://github.com/apache/airflow/pull/8807#discussion_r422774281 ## File path: backport_packages/setup_backport_packages.py ## @@ -422,13 +371,376 @@ def usage(): print(text) print() print("You can

[GitHub] [airflow] mik-laj commented on a change in pull request #8807: Added automated release notes generation for backport operators

2020-05-10 Thread GitBox
mik-laj commented on a change in pull request #8807: URL: https://github.com/apache/airflow/pull/8807#discussion_r422772250 ## File path: backport_packages/setup_backport_packages.py ## @@ -422,13 +371,376 @@ def usage(): print(text) print() print("You can

[GitHub] [airflow] mik-laj commented on a change in pull request #8807: Added automated release notes generation for backport operators

2020-05-10 Thread GitBox
mik-laj commented on a change in pull request #8807: URL: https://github.com/apache/airflow/pull/8807#discussion_r422772250 ## File path: backport_packages/setup_backport_packages.py ## @@ -422,13 +371,376 @@ def usage(): print(text) print() print("You can

[GitHub] [airflow] mik-laj commented on a change in pull request #8807: Added automated release notes generation for backport operators

2020-05-10 Thread GitBox
mik-laj commented on a change in pull request #8807: URL: https://github.com/apache/airflow/pull/8807#discussion_r422771411 ## File path: dev/BACKPORT_PACKAGES.md ## @@ -0,0 +1,119 @@ + + +# Backport packages + +# Overview + +This document describes process of preparing

[GitHub] [airflow] mik-laj commented on a change in pull request #8807: Added automated release notes generation for backport operators

2020-05-10 Thread GitBox
mik-laj commented on a change in pull request #8807: URL: https://github.com/apache/airflow/pull/8807#discussion_r422771325 ## File path: dev/BACKPORT_PACKAGES.md ## @@ -0,0 +1,119 @@ + + +# Backport packages + +# Overview + +This document describes process of preparing

[GitHub] [airflow] mik-laj commented on a change in pull request #8807: Added automated release notes generation for backport operators

2020-05-10 Thread GitBox
mik-laj commented on a change in pull request #8807: URL: https://github.com/apache/airflow/pull/8807#discussion_r422771208 ## File path: dev/BACKPORT_PACKAGES.md ## @@ -0,0 +1,119 @@ + + +# Backport packages + +# Overview + +This document describes process of preparing

[GitHub] [airflow] mik-laj commented on a change in pull request #8807: Added automated release notes generation for backport operators

2020-05-10 Thread GitBox
mik-laj commented on a change in pull request #8807: URL: https://github.com/apache/airflow/pull/8807#discussion_r422771071 ## File path: dev/BACKPORT_PACKAGES.md ## @@ -0,0 +1,119 @@ + + +# Backport packages + +# Overview + +This document describes process of preparing

[GitHub] [airflow] mik-laj commented on a change in pull request #8807: Added automated release notes generation for backport operators

2020-05-10 Thread GitBox
mik-laj commented on a change in pull request #8807: URL: https://github.com/apache/airflow/pull/8807#discussion_r422770823 ## File path: dev/BACKPORT_PACKAGES.md ## @@ -0,0 +1,119 @@ + + +# Backport packages + +# Overview + +This document describes process of preparing

[GitHub] [airflow] mik-laj commented on a change in pull request #8807: Added automated release notes generation for backport operators

2020-05-10 Thread GitBox
mik-laj commented on a change in pull request #8807: URL: https://github.com/apache/airflow/pull/8807#discussion_r422770693 ## File path: dev/BACKPORT_PACKAGES.md ## @@ -0,0 +1,119 @@ + + +# Backport packages + +# Overview + +This document describes process of preparing

[GitHub] [airflow] mik-laj commented on a change in pull request #8807: Added automated release notes generation for backport operators

2020-05-10 Thread GitBox
mik-laj commented on a change in pull request #8807: URL: https://github.com/apache/airflow/pull/8807#discussion_r422770320 ## File path: requirements/requirements-python3.7.txt ## @@ -38,6 +38,7 @@ alembic==1.4.2 amqp==2.5.2 analytics-python==1.2.9 ansiwrap==0.8.4

[GitHub] [airflow] mik-laj commented on a change in pull request #8807: Added automated release notes generation for backport operators

2020-05-10 Thread GitBox
mik-laj commented on a change in pull request #8807: URL: https://github.com/apache/airflow/pull/8807#discussion_r422770469 ## File path: dev/BACKPORT_PACKAGES.md ## @@ -0,0 +1,119 @@ + + +# Backport packages + +# Overview + +This document describes process of preparing

[GitHub] [airflow] mik-laj commented on a change in pull request #8807: Added automated release notes generation for backport operators

2020-05-10 Thread GitBox
mik-laj commented on a change in pull request #8807: URL: https://github.com/apache/airflow/pull/8807#discussion_r422768830 ## File path: setup.py ## @@ -510,9 +514,9 @@ def write_version(filename: str = os.path.join(*[my_dir, "airflow", "git_version 'elasticsearch':

[GitHub] [airflow] mik-laj commented on a change in pull request #8807: Added automated release notes generation for backport operators

2020-05-10 Thread GitBox
mik-laj commented on a change in pull request #8807: URL: https://github.com/apache/airflow/pull/8807#discussion_r422768168 ## File path: setup.py ## @@ -249,8 +249,9 @@ def write_version(filename: str = os.path.join(*[my_dir, "airflow", "git_version

[GitHub] [airflow] mik-laj commented on a change in pull request #8807: Added automated release notes generation for backport operators

2020-05-10 Thread GitBox
mik-laj commented on a change in pull request #8807: URL: https://github.com/apache/airflow/pull/8807#discussion_r422767019 ## File path: setup.py ## @@ -545,6 +549,7 @@ def write_version(filename: str = os.path.join(*[my_dir, "airflow", "git_version 'singularity':

[GitHub] [airflow] mik-laj commented on a change in pull request #8807: Added automated release notes generation for backport operators

2020-05-10 Thread GitBox
mik-laj commented on a change in pull request #8807: URL: https://github.com/apache/airflow/pull/8807#discussion_r422766615 ## File path: tests/test_core_to_contrib.py ## @@ -1499,14 +1574,14 @@ "airflow.providers.amazon.aws.sensors.sqs.SQSSensor",

[GitHub] [airflow] mik-laj commented on a change in pull request #8807: Added automated release notes generation for backport operators

2020-05-10 Thread GitBox
mik-laj commented on a change in pull request #8807: URL: https://github.com/apache/airflow/pull/8807#discussion_r422766304 ## File path: setup.py ## @@ -475,10 +479,10 @@ def write_version(filename: str = os.path.join(*[my_dir, "airflow", "git_version devel_hadoop =

[GitHub] [airflow] mik-laj commented on a change in pull request #8807: Added automated release notes generation for backport operators

2020-05-10 Thread GitBox
mik-laj commented on a change in pull request #8807: URL: https://github.com/apache/airflow/pull/8807#discussion_r422766428 ## File path: tests/test_core_to_contrib.py ## @@ -184,14 +188,14 @@ 'airflow.providers.apache.druid.hooks.druid.DruidDbApiHook',

[GitHub] [airflow] mik-laj commented on a change in pull request #8807: Added automated release notes generation for backport operators

2020-05-10 Thread GitBox
mik-laj commented on a change in pull request #8807: URL: https://github.com/apache/airflow/pull/8807#discussion_r422763613 ## File path: backport_packages/setup_backport_packages.py ## @@ -193,17 +166,16 @@ def _remover(node: LN, capture: Capture, filename: Filename) ->

[GitHub] [airflow] mik-laj commented on a change in pull request #8807: Added automated release notes generation for backport operators

2020-05-10 Thread GitBox
mik-laj commented on a change in pull request #8807: URL: https://github.com/apache/airflow/pull/8807#discussion_r422761479 ## File path: INSTALL ## @@ -45,11 +45,11 @@ pip install . --constraint requirements/requirements-python3.7.txt # START EXTRAS HERE all, all_dbs,

[GitHub] [airflow] mik-laj commented on pull request #8808: Support YAML input for CloudBuildCreateOperator

2020-05-10 Thread GitBox
mik-laj commented on pull request #8808: URL: https://github.com/apache/airflow/pull/8808#issuecomment-626451844 I tried to implement this support based on template_ext. What do you think? ```diff 05:31 $ git diff --cached diff --git

[GitHub] [airflow] mik-laj commented on a change in pull request #8808: Support YAML input for CloudBuildCreateOperator

2020-05-10 Thread GitBox
mik-laj commented on a change in pull request #8808: URL: https://github.com/apache/airflow/pull/8808#discussion_r422760527 ## File path: airflow/providers/google/cloud/example_dags/example_cloud_build.yaml ## @@ -0,0 +1,20 @@ +# Licensed to the Apache Software Foundation

[GitHub] [airflow] mik-laj commented on a change in pull request #8808: Support YAML input for CloudBuildCreateOperator

2020-05-10 Thread GitBox
mik-laj commented on a change in pull request #8808: URL: https://github.com/apache/airflow/pull/8808#discussion_r422757972 ## File path: airflow/providers/google/cloud/example_dags/example_cloud_build.py ## @@ -99,6 +99,12 @@ task_id="create_build_from_repo_result",

[GitHub] [airflow] mik-laj commented on a change in pull request #8808: Support YAML input for CloudBuildCreateOperator

2020-05-10 Thread GitBox
mik-laj commented on a change in pull request #8808: URL: https://github.com/apache/airflow/pull/8808#discussion_r422757760 ## File path: airflow/providers/google/cloud/example_dags/example_cloud_build.yaml ## @@ -0,0 +1,20 @@ +# Licensed to the Apache Software Foundation

[GitHub] [airflow] mik-laj commented on a change in pull request #8808: Support YAML input for CloudBuildCreateOperator

2020-05-10 Thread GitBox
mik-laj commented on a change in pull request #8808: URL: https://github.com/apache/airflow/pull/8808#discussion_r422757560 ## File path: airflow/providers/google/cloud/example_dags/example_cloud_build.py ## @@ -99,6 +99,12 @@ task_id="create_build_from_repo_result",

[GitHub] [airflow] mik-laj commented on a change in pull request #8808: Support YAML input for CloudBuildCreateOperator

2020-05-10 Thread GitBox
mik-laj commented on a change in pull request #8808: URL: https://github.com/apache/airflow/pull/8808#discussion_r422757468 ## File path: airflow/providers/google/cloud/operators/cloud_build.py ## @@ -83,13 +85,29 @@ def _reformat_storage_source(self):

[GitHub] [airflow] mik-laj commented on a change in pull request #8808: Support YAML input for CloudBuildCreateOperator

2020-05-10 Thread GitBox
mik-laj commented on a change in pull request #8808: URL: https://github.com/apache/airflow/pull/8808#discussion_r422757102 ## File path: airflow/providers/google/cloud/operators/cloud_build.py ## @@ -83,13 +85,29 @@ def _reformat_storage_source(self):

[GitHub] [airflow] mik-laj commented on a change in pull request #8809: [AIRFLOW-6294] Create guide for Dataflow operators

2020-05-10 Thread GitBox
mik-laj commented on a change in pull request #8809: URL: https://github.com/apache/airflow/pull/8809#discussion_r422753200 ## File path: docs/howto/operator/gcp/dataflow.rst ## @@ -0,0 +1,120 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or more

[jira] [Commented] (AIRFLOW-6294) Create guide for Dataflow service operator

2020-05-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17104022#comment-17104022 ] ASF GitHub Bot commented on AIRFLOW-6294: - mik-laj commented on a change in pull request #8809:

[airflow] tag nightly-master updated (c7788a6 -> 493b685)

2020-05-10 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to tag nightly-master in repository https://gitbox.apache.org/repos/asf/airflow.git. *** WARNING: tag nightly-master was modified! *** from c7788a6 (commit) to 493b685 (commit) from

[GitHub] [airflow] kinderyj commented on issue #8799: Install airflow to kubernetes failed

2020-05-10 Thread GitBox
kinderyj commented on issue #8799: URL: https://github.com/apache/airflow/issues/8799#issuecomment-626443601 > Note that this was Dockerfile that was only used for testing. The location of scripts and way of installation changed since then - you can see it here:

[GitHub] [airflow] mik-laj commented on a change in pull request #8809: [AIRFLOW-6294] Create guide for Dataflow operators

2020-05-10 Thread GitBox
mik-laj commented on a change in pull request #8809: URL: https://github.com/apache/airflow/pull/8809#discussion_r422752902 ## File path: docs/howto/operator/gcp/dataflow.rst ## @@ -0,0 +1,120 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or more

[jira] [Commented] (AIRFLOW-6294) Create guide for Dataflow service operator

2020-05-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17104021#comment-17104021 ] ASF GitHub Bot commented on AIRFLOW-6294: - mik-laj commented on a change in pull request #8809:

[jira] [Commented] (AIRFLOW-6294) Create guide for Dataflow service operator

2020-05-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17104020#comment-17104020 ] ASF GitHub Bot commented on AIRFLOW-6294: - mik-laj commented on a change in pull request #8809:

[GitHub] [airflow] mik-laj commented on a change in pull request #8809: [AIRFLOW-6294] Create guide for Dataflow operators

2020-05-10 Thread GitBox
mik-laj commented on a change in pull request #8809: URL: https://github.com/apache/airflow/pull/8809#discussion_r422751986 ## File path: docs/howto/operator/gcp/dataflow.rst ## @@ -0,0 +1,120 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or more

[jira] [Commented] (AIRFLOW-6294) Create guide for Dataflow service operator

2020-05-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17104017#comment-17104017 ] ASF GitHub Bot commented on AIRFLOW-6294: - mik-laj commented on a change in pull request #8809:

[GitHub] [airflow] mik-laj commented on a change in pull request #8809: [AIRFLOW-6294] Create guide for Dataflow operators

2020-05-10 Thread GitBox
mik-laj commented on a change in pull request #8809: URL: https://github.com/apache/airflow/pull/8809#discussion_r422751334 ## File path: airflow/providers/google/cloud/example_dags/example_dataflow.py ## @@ -108,9 +114,17 @@ ) # [END howto_operator_start_python_job]

[jira] [Commented] (AIRFLOW-6294) Create guide for Dataflow service operator

2020-05-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17104015#comment-17104015 ] ASF GitHub Bot commented on AIRFLOW-6294: - mik-laj commented on pull request #8809: URL:

[GitHub] [airflow] mik-laj commented on pull request #8809: [AIRFLOW-6294] Create guide for Dataflow operators

2020-05-10 Thread GitBox
mik-laj commented on pull request #8809: URL: https://github.com/apache/airflow/pull/8809#issuecomment-626441074 In my opinion, we need to add some information. * new section which will describe the ways to run a pipeline. Currently, the pipeline can be started using a local running

[GitHub] [airflow] zhongjiajie commented on a change in pull request #8796: Better help on parameterized breeze commands

2020-05-10 Thread GitBox
zhongjiajie commented on a change in pull request #8796: URL: https://github.com/apache/airflow/pull/8796#discussion_r422739813 ## File path: breeze ## @@ -535,7 +535,7 @@ function prepare_command_files() { # Prepare script for "run test" prepare_command_file

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #8809: Airflow 6294 create guide for dataflow operators

2020-05-10 Thread GitBox
boring-cyborg[bot] commented on pull request #8809: URL: https://github.com/apache/airflow/pull/8809#issuecomment-626408155 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our

[GitHub] [airflow] tanjinP opened a new pull request #8809: Airflow 6294 create guide for dataflow operators

2020-05-10 Thread GitBox
tanjinP opened a new pull request #8809: URL: https://github.com/apache/airflow/pull/8809 Closes [AIRFLOW-6294](https://issues.apache.org/jira/browse/AIRFLOW-6294) / issue #8202 . Contains a guide that is appropriately linked to on how to use the various operators in Dataflow.

[jira] [Commented] (AIRFLOW-249) Refactor the SLA mechanism

2020-05-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103970#comment-17103970 ] ASF GitHub Bot commented on AIRFLOW-249: houqp edited a comment on pull request #8545: URL:

[GitHub] [airflow] houqp edited a comment on pull request #8545: [AIRFLOW-249] Refactor the SLA mechanism (Continuation from #3584 )

2020-05-10 Thread GitBox
houqp edited a comment on pull request #8545: URL: https://github.com/apache/airflow/pull/8545#issuecomment-626404118 If we can add `.filter(DR.state != State.SUCCESS)` to the query filter list, then I am also in favor of option 1 (without 100 fixed limit of course). It's simpler than 2

[GitHub] [airflow] houqp edited a comment on pull request #8545: [AIRFLOW-249] Refactor the SLA mechanism (Continuation from #3584 )

2020-05-10 Thread GitBox
houqp edited a comment on pull request #8545: URL: https://github.com/apache/airflow/pull/8545#issuecomment-626404118 If we can add `.filter(DR.state != State.SUCCESS)` to the query filter list, then I am also in favor of option 1 (without fixed 100 limit of course). It's simpler than 2

[jira] [Commented] (AIRFLOW-249) Refactor the SLA mechanism

2020-05-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103969#comment-17103969 ] ASF GitHub Bot commented on AIRFLOW-249: houqp edited a comment on pull request #8545: URL:

[jira] [Commented] (AIRFLOW-249) Refactor the SLA mechanism

2020-05-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103968#comment-17103968 ] ASF GitHub Bot commented on AIRFLOW-249: houqp commented on pull request #8545: URL:

[GitHub] [airflow] houqp commented on pull request #8545: [AIRFLOW-249] Refactor the SLA mechanism (Continuation from #3584 )

2020-05-10 Thread GitBox
houqp commented on pull request #8545: URL: https://github.com/apache/airflow/pull/8545#issuecomment-626404118 If we can add `.filter(DR.state != State.SUCCESS)` to the query filter list, then I am also in favor of option 1. It's simpler than 2 and shouldn't run into performance issue for

[jira] [Commented] (AIRFLOW-5873) KubernetesPodOperator fixes and test

2020-05-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103956#comment-17103956 ] ASF GitHub Bot commented on AIRFLOW-5873: - dsaiztc commented on a change in pull request #6523:

[GitHub] [airflow] dsaiztc commented on a change in pull request #6523: [AIRFLOW-5873] KubernetesPodOperator fixes and test

2020-05-10 Thread GitBox
dsaiztc commented on a change in pull request #6523: URL: https://github.com/apache/airflow/pull/6523#discussion_r422708666 ## File path: airflow/contrib/operators/kubernetes_pod_operator.py ## @@ -158,17 +167,17 @@ def execute(self, context): raise

[jira] [Commented] (AIRFLOW-249) Refactor the SLA mechanism

2020-05-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103954#comment-17103954 ] ASF GitHub Bot commented on AIRFLOW-249: seanxwzhang commented on pull request #8545: URL:

[GitHub] [airflow] seanxwzhang commented on pull request #8545: [AIRFLOW-249] Refactor the SLA mechanism (Continuation from #3584 )

2020-05-10 Thread GitBox
seanxwzhang commented on pull request #8545: URL: https://github.com/apache/airflow/pull/8545#issuecomment-626397934 > Option 1 doesn't guarantee correctness right? i.e. if there are more dagruns that need to be checked than the preset limit, some of them will be ignored? True. I

[GitHub] [airflow] stale[bot] commented on pull request #7806: Decouple DAG from DagBag and DAG from SerializedDagModel - remove cyclic imports

2020-05-10 Thread GitBox
stale[bot] commented on pull request #7806: URL: https://github.com/apache/airflow/pull/7806#issuecomment-626394083 This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[GitHub] [airflow] joppevos opened a new pull request #8808: Support YAML input for CloudBuildCreateOperator

2020-05-10 Thread GitBox
joppevos opened a new pull request #8808: URL: https://github.com/apache/airflow/pull/8808 addresses the (following)[https://github.com/apache/airflow/issues/8567#issuecomment-620212297] issue. --- Make sure to mark the boxes below before creating PR: [x] - [x]

[jira] [Commented] (AIRFLOW-2310) Enable AWS Glue Job Integration

2020-05-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103936#comment-17103936 ] ASF GitHub Bot commented on AIRFLOW-2310: - abdulbasitds commented on pull request #6007: URL:

[GitHub] [airflow] abdulbasitds commented on pull request #6007: [AIRFLOW-2310] Enable AWS Glue Job Integration

2020-05-10 Thread GitBox
abdulbasitds commented on pull request #6007: URL: https://github.com/apache/airflow/pull/6007#issuecomment-626382841 > Hello @kaxil and @feluelle do you have idea about the current failing test? it says something like the tests are not listed It was due to naming of the tests.

[GitHub] [airflow] evgenyshulman commented on a change in pull request #8805: Resolve upstream tasks when template field is XComArg

2020-05-10 Thread GitBox
evgenyshulman commented on a change in pull request #8805: URL: https://github.com/apache/airflow/pull/8805#discussion_r422693890 ## File path: airflow/models/baseoperator.py ## @@ -672,10 +672,20 @@ def _set_xcomargs_dependencies(self) -> None: """ from

[airflow] branch master updated (9bb91ef -> 493b685)

2020-05-10 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from 9bb91ef Add comments to breeze scripts (#8797) add 493b685 Add separate example DAGs and system tests for

[GitHub] [airflow] potiuk commented on pull request #8778: Add separate example DAGs and system tests for google cloud speech

2020-05-10 Thread GitBox
potiuk commented on pull request #8778: URL: https://github.com/apache/airflow/pull/8778#issuecomment-626382141 Thanks @ephraimbuddy ! This is an automated message from the Apache Git Service. To respond to the message,

[jira] [Commented] (AIRFLOW-7068) Create EC2 Hook, Operator and Sensor

2020-05-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-7068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103935#comment-17103935 ] ASF GitHub Bot commented on AIRFLOW-7068: - mustafagok commented on pull request #7731: URL:

[GitHub] [airflow] evgenyshulman commented on a change in pull request #8805: Resolve upstream tasks when template field is XComArg

2020-05-10 Thread GitBox
evgenyshulman commented on a change in pull request #8805: URL: https://github.com/apache/airflow/pull/8805#discussion_r422693309 ## File path: airflow/models/baseoperator.py ## @@ -60,9 +60,25 @@ ScheduleInterval = Union[str, timedelta, relativedelta] +class

[GitHub] [airflow] mustafagok commented on pull request #7731: [AIRFLOW-7068] Create EC2 Hook, Operator and Sensor

2020-05-10 Thread GitBox
mustafagok commented on pull request #7731: URL: https://github.com/apache/airflow/pull/7731#issuecomment-626382139 So, it's ready for merge I think @feluelle This is an automated message from the Apache Git Service. To

[GitHub] [airflow] abdulbasitds commented on a change in pull request #6007: [AIRFLOW-2310] Enable AWS Glue Job Integration

2020-05-10 Thread GitBox
abdulbasitds commented on a change in pull request #6007: URL: https://github.com/apache/airflow/pull/6007#discussion_r422661617 ## File path: airflow/providers/amazon/aws/hooks/glue.py ## @@ -0,0 +1,180 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or

[jira] [Commented] (AIRFLOW-2310) Enable AWS Glue Job Integration

2020-05-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103920#comment-17103920 ] ASF GitHub Bot commented on AIRFLOW-2310: - abdulbasitds commented on a change in pull request

[GitHub] [airflow] potiuk commented on pull request #8807: Added automated release notes generation for backport operators

2020-05-10 Thread GitBox
potiuk commented on pull request #8807: URL: https://github.com/apache/airflow/pull/8807#issuecomment-626365997 I also added `dev\BACKPORT_PACKAGES.md` file where I described the process of preparing the packages to release. The next steps (releasing) will follow shortly

[airflow] branch master updated: Add comments to breeze scripts (#8797)

2020-05-10 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/master by this push: new 9bb91ef Add comments to breeze scripts

[GitHub] [airflow] potiuk commented on a change in pull request #8796: Better help on parameterized breeze commands

2020-05-10 Thread GitBox
potiuk commented on a change in pull request #8796: URL: https://github.com/apache/airflow/pull/8796#discussion_r422677889 ## File path: BREEZE.rst ## @@ -1046,15 +1046,21 @@ This is the current syntax for `./breeze <./breeze>`_: Detailed usage for command: static-check

[GitHub] [airflow] XD-DENG commented on pull request #8742: Avoid color info in response of /dag_stats & /task_stats

2020-05-10 Thread GitBox
XD-DENG commented on pull request #8742: URL: https://github.com/apache/airflow/pull/8742#issuecomment-626361211 Hi guys @mik-laj @kaxil @ashb , a gentle ping. Mind taking another look? This is an automated message from the

[GitHub] [airflow] potiuk commented on pull request #8807: Added automated release notes generation for backport operators

2020-05-10 Thread GitBox
potiuk commented on pull request #8807: URL: https://github.com/apache/airflow/pull/8807#issuecomment-626359395 Hey (mostly @ashb and @kaxil - but also @mik-laj and @turbaszek mostly. This is one monstrous PR but it has to be so, because it has all the Backport packages readme generated

[GitHub] [airflow] boring-cyborg[bot] commented on issue #8806: DataProcSparkOperator getting reattached to the previous run - as this feature is not required every time, hence we need a flag in the t

2020-05-10 Thread GitBox
boring-cyborg[bot] commented on issue #8806: URL: https://github.com/apache/airflow/issues/8806#issuecomment-626358015 Thanks for opening your first issue here! Be sure to follow the issue template! This is an automated

[GitHub] [airflow] potiuk opened a new pull request #8807: Added automated release notes generation for backport operators

2020-05-10 Thread GitBox
potiuk opened a new pull request #8807: URL: https://github.com/apache/airflow/pull/8807 --- Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Unit tests coverage for changes (not needed for documentation

[GitHub] [airflow] mdnawed2010 opened a new issue #8806: DataProcSparkOperator getting reattached to the previous run - as this feature is not required every time, hence we need a flag in the task its

2020-05-10 Thread GitBox
mdnawed2010 opened a new issue #8806: URL: https://github.com/apache/airflow/issues/8806 **Apache Airflow version**: 1.10.6 (looks like this problem is for all ver >=1.10.4) **Environment**: Google cloud composer(composer-1.10.2-airflow-1.10.6) Issue : This is

[GitHub] [airflow] kaxil commented on pull request #8560: Allow to define custom XCom class

2020-05-10 Thread GitBox
kaxil commented on pull request #8560: URL: https://github.com/apache/airflow/pull/8560#issuecomment-626357585 Tentatively added it to 1.10.11 milestone, will double-check when cherry-picking though to see if it for sure doesn't break backward-compatibility.

[GitHub] [airflow] webster-chainalysis commented on issue #8212: Can't read S3 remote logs using Airflow 1.10.9

2020-05-10 Thread GitBox
webster-chainalysis commented on issue #8212: URL: https://github.com/apache/airflow/issues/8212#issuecomment-626347626 @dimon222 - That makes sense. I'm currently using gevent behind an Amazon elastic load balancer. When using the sync worker class with the ELB I had unacceptable

[GitHub] [airflow] abdulbasitds commented on a change in pull request #6007: [AIRFLOW-2310] Enable AWS Glue Job Integration

2020-05-10 Thread GitBox
abdulbasitds commented on a change in pull request #6007: URL: https://github.com/apache/airflow/pull/6007#discussion_r422661617 ## File path: airflow/providers/amazon/aws/hooks/glue.py ## @@ -0,0 +1,180 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or

[jira] [Commented] (AIRFLOW-2310) Enable AWS Glue Job Integration

2020-05-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103854#comment-17103854 ] ASF GitHub Bot commented on AIRFLOW-2310: - abdulbasitds commented on a change in pull request

[jira] [Commented] (AIRFLOW-2310) Enable AWS Glue Job Integration

2020-05-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103845#comment-17103845 ] ASF GitHub Bot commented on AIRFLOW-2310: - abdulbasitds commented on pull request #6007: URL:

[GitHub] [airflow] abdulbasitds commented on pull request #6007: [AIRFLOW-2310] Enable AWS Glue Job Integration

2020-05-10 Thread GitBox
abdulbasitds commented on pull request #6007: URL: https://github.com/apache/airflow/pull/6007#issuecomment-626345860 Hello @kaxil and @feluelle do you have idea about the current failing test? it says something like the tests are not listed

[GitHub] [airflow] turbaszek commented on a change in pull request #8651: [AIP-31] [AIRFLOW-8058] Retrieve current execution context without altering function's signature

2020-05-10 Thread GitBox
turbaszek commented on a change in pull request #8651: URL: https://github.com/apache/airflow/pull/8651#discussion_r422653688 ## File path: airflow/models/taskinstance.py ## @@ -1110,6 +1116,27 @@ def signal_handler(signum, frame): session.merge(self)

[GitHub] [airflow] mik-laj commented on pull request #7731: [AIRFLOW-7068] Create EC2 Hook, Operator and Sensor

2020-05-10 Thread GitBox
mik-laj commented on pull request #7731: URL: https://github.com/apache/airflow/pull/7731#issuecomment-626335689 Emr and Ec2 are not PEP compliant. I think we should use EMR and EC2. I don't require it. I only point to the conventions we use for GCP and which is recommended by PEP

[jira] [Commented] (AIRFLOW-7068) Create EC2 Hook, Operator and Sensor

2020-05-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-7068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103817#comment-17103817 ] ASF GitHub Bot commented on AIRFLOW-7068: - mik-laj commented on pull request #7731: URL:

[GitHub] [airflow] turbaszek commented on a change in pull request #8805: Resolve upstream tasks when template field is XComArg

2020-05-10 Thread GitBox
turbaszek commented on a change in pull request #8805: URL: https://github.com/apache/airflow/pull/8805#discussion_r422638768 ## File path: airflow/models/baseoperator.py ## @@ -60,9 +60,25 @@ ScheduleInterval = Union[str, timedelta, relativedelta] +class

[GitHub] [airflow] turbaszek commented on a change in pull request #8805: Resolve upstream tasks when template field is XComArg

2020-05-10 Thread GitBox
turbaszek commented on a change in pull request #8805: URL: https://github.com/apache/airflow/pull/8805#discussion_r422651378 ## File path: airflow/models/baseoperator.py ## @@ -60,9 +60,25 @@ ScheduleInterval = Union[str, timedelta, relativedelta] +class

[GitHub] [airflow] turbaszek commented on a change in pull request #8805: Resolve upstream tasks when template field is XComArg

2020-05-10 Thread GitBox
turbaszek commented on a change in pull request #8805: URL: https://github.com/apache/airflow/pull/8805#discussion_r422651378 ## File path: airflow/models/baseoperator.py ## @@ -60,9 +60,25 @@ ScheduleInterval = Union[str, timedelta, relativedelta] +class

[GitHub] [airflow] turbaszek commented on a change in pull request #8805: Resolve upstream tasks when template field is XComArg

2020-05-10 Thread GitBox
turbaszek commented on a change in pull request #8805: URL: https://github.com/apache/airflow/pull/8805#discussion_r422651378 ## File path: airflow/models/baseoperator.py ## @@ -60,9 +60,25 @@ ScheduleInterval = Union[str, timedelta, relativedelta] +class

[GitHub] [airflow] feluelle commented on pull request #7731: [AIRFLOW-7068] Create EC2 Hook, Operator and Sensor

2020-05-10 Thread GitBox
feluelle commented on pull request #7731: URL: https://github.com/apache/airflow/pull/7731#issuecomment-626332522 @mik-laj So `Emr` works but `Ec2` doesn't? What is the difference? This is an automated message from the

[jira] [Commented] (AIRFLOW-7068) Create EC2 Hook, Operator and Sensor

2020-05-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-7068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103809#comment-17103809 ] ASF GitHub Bot commented on AIRFLOW-7068: - feluelle commented on pull request #7731: URL:

[GitHub] [airflow] KimchaC commented on issue #8605: Add Production-ready docker compose for the production image

2020-05-10 Thread GitBox
KimchaC commented on issue #8605: URL: https://github.com/apache/airflow/issues/8605#issuecomment-626331675 Here's my docker-compose config using LocalExecutor... ### docker-compose.airflow.yml: ```yml version: '2.1' services: airflow: # image:

[GitHub] [airflow] mik-laj commented on pull request #7731: [AIRFLOW-7068] Create EC2 Hook, Operator and Sensor

2020-05-10 Thread GitBox
mik-laj commented on pull request #7731: URL: https://github.com/apache/airflow/pull/7731#issuecomment-626331536 @feluelle it's not PEP 'compliant. > Note: When using acronyms in CapWords, capitalize all the letters of the acronym. Thus HTTPServerError is better than

[jira] [Commented] (AIRFLOW-7068) Create EC2 Hook, Operator and Sensor

2020-05-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-7068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103806#comment-17103806 ] ASF GitHub Bot commented on AIRFLOW-7068: - mik-laj commented on pull request #7731: URL:

[GitHub] [airflow] turbaszek commented on a change in pull request #8805: Resolve upstream tasks when template field is XComArg

2020-05-10 Thread GitBox
turbaszek commented on a change in pull request #8805: URL: https://github.com/apache/airflow/pull/8805#discussion_r422648227 ## File path: tests/models/test_baseoperator.py ## @@ -262,6 +263,33 @@ def test_email_on_actions(self): assert test_task.email_on_retry is

[GitHub] [airflow] turbaszek commented on a change in pull request #8805: Resolve upstream tasks when template field is XComArg

2020-05-10 Thread GitBox
turbaszek commented on a change in pull request #8805: URL: https://github.com/apache/airflow/pull/8805#discussion_r422643152 ## File path: airflow/models/baseoperator.py ## @@ -60,9 +60,25 @@ ScheduleInterval = Union[str, timedelta, relativedelta] +class

  1   2   >