[GitHub] [airflow] ausiddiqui commented on pull request #11195: 2.0 UI Overhaul/Refresh

2020-09-30 Thread GitBox
ausiddiqui commented on pull request #11195: URL: https://github.com/apache/airflow/pull/11195#issuecomment-701203026 This looks great! Had a few requests: Is it possible to add a legend in the main _DAGs View (Home)_ for the Recent Tasks? There are many columns and colors and hard t

[GitHub] [airflow] gvazharov commented on issue #10860: Timeouts in Airflow scheduler logs on AKS

2020-09-30 Thread GitBox
gvazharov commented on issue #10860: URL: https://github.com/apache/airflow/issues/10860#issuecomment-701226788 Hi I am experiencing the same problem and don't know how to fix it, is there any kind of update? This is an auto

[GitHub] [airflow] michalslowikowski00 commented on issue #9949: Create Operators for Cloud Dataprep

2020-09-30 Thread GitBox
michalslowikowski00 commented on issue #9949: URL: https://github.com/apache/airflow/issues/9949#issuecomment-701227053 Yes. This is an automated message from the Apache Git Service. To respond to the message, please log on t

[GitHub] [airflow] mik-laj closed issue #9949: Create Operators for Cloud Dataprep

2020-09-30 Thread GitBox
mik-laj closed issue #9949: URL: https://github.com/apache/airflow/issues/9949 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

[GitHub] [airflow] nalepae opened a new issue #11200: Sometimes, all the steps of a DAG are in state success, but the DAG itself in is state failed.

2020-09-30 Thread GitBox
nalepae opened a new issue #11200: URL: https://github.com/apache/airflow/issues/11200 **Apache Airflow version**: 1.10.3 on Google Cloud Composer (so 1.10.3-composer) **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): (Managed by Google Cloud Composer

[GitHub] [airflow] turbaszek commented on issue #9300: Create DatafusionPipelineStateSensor

2020-09-30 Thread GitBox
turbaszek commented on issue #9300: URL: https://github.com/apache/airflow/issues/9300#issuecomment-701238917 @mik-laj what is the reason to close this issue? Was this sensors implemented? This is an automated message from t

[GitHub] [airflow] turbaszek opened a new issue #9300: Create DatafusionPipelineStateSensor

2020-09-30 Thread GitBox
turbaszek opened a new issue #9300: URL: https://github.com/apache/airflow/issues/9300 **Description** Create a sensor that will wait for Cloud Datafusion pipeline to be in expected state. **Use case / motivation** _Originally posted by @jaketf in https://github.com/ap

[GitHub] [airflow] mik-laj commented on issue #9300: Create DatafusionPipelineStateSensor

2020-09-30 Thread GitBox
mik-laj commented on issue #9300: URL: https://github.com/apache/airflow/issues/9300#issuecomment-701239866 it felt like it was finished. I saw the related PR that is closed. This is an automated message from the Apache Git

[GitHub] [airflow] WSHoekstra commented on pull request #11195: 2.0 UI Overhaul/Refresh

2020-09-30 Thread GitBox
WSHoekstra commented on pull request #11195: URL: https://github.com/apache/airflow/pull/11195#issuecomment-701242248 I love most of it, but there are a few things I'd like to see differently. - I'll use the 'Search Dags' bar more than the 'Search by tags' search bar, so I'd like to see

[GitHub] [airflow] turbaszek commented on a change in pull request #10947: Add amazon glacier to GCS transfer operator

2020-09-30 Thread GitBox
turbaszek commented on a change in pull request #10947: URL: https://github.com/apache/airflow/pull/10947#discussion_r497332142 ## File path: docs/howto/operator/amazon/aws/glacier.rst ## @@ -0,0 +1,90 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or m

[GitHub] [airflow] turbaszek commented on a change in pull request #10947: Add amazon glacier to GCS transfer operator

2020-09-30 Thread GitBox
turbaszek commented on a change in pull request #10947: URL: https://github.com/apache/airflow/pull/10947#discussion_r497333442 ## File path: airflow/providers/amazon/aws/transfers/glacier_to_gcs.py ## @@ -0,0 +1,122 @@ +# +# Licensed to the Apache Software Foundation (ASF) und

[GitHub] [airflow] turbaszek commented on issue #9300: Create DatafusionPipelineStateSensor

2020-09-30 Thread GitBox
turbaszek commented on issue #9300: URL: https://github.com/apache/airflow/issues/9300#issuecomment-701247182 The PR fixed the operator that works in a synchronous manner. This ticket is about creating a sensors and allowing async mode in `DatafusionCreatePipelineOperator` similar to #1067

[GitHub] [airflow] turbaszek commented on a change in pull request #11132: Add loggingconfig rule

2020-09-30 Thread GitBox
turbaszek commented on a change in pull request #11132: URL: https://github.com/apache/airflow/pull/11132#discussion_r497340967 ## File path: airflow/upgrade/rules/logging_configuration.py ## @@ -0,0 +1,83 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or m

[GitHub] [airflow] turbaszek commented on a change in pull request #11132: Add loggingconfig rule

2020-09-30 Thread GitBox
turbaszek commented on a change in pull request #11132: URL: https://github.com/apache/airflow/pull/11132#discussion_r497341275 ## File path: airflow/upgrade/rules/logging_configuration.py ## @@ -0,0 +1,83 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or m

[GitHub] [airflow] turbaszek commented on a change in pull request #11132: Add loggingconfig rule

2020-09-30 Thread GitBox
turbaszek commented on a change in pull request #11132: URL: https://github.com/apache/airflow/pull/11132#discussion_r497341878 ## File path: airflow/upgrade/rules/logging_configuration.py ## @@ -0,0 +1,83 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or m

[GitHub] [airflow] turbaszek commented on a change in pull request #10991: add azure files to gcs transfer operator

2020-09-30 Thread GitBox
turbaszek commented on a change in pull request #10991: URL: https://github.com/apache/airflow/pull/10991#discussion_r497343222 ## File path: airflow/models/baseoperator.py ## @@ -89,7 +89,7 @@ class BaseOperator(Operator, LoggingMixin, metaclass=BaseOperatorMeta): Abstra

[GitHub] [airflow] TobKed commented on a change in pull request #8550: Add DataflowStartFlexTemplateOperator

2020-09-30 Thread GitBox
TobKed commented on a change in pull request #8550: URL: https://github.com/apache/airflow/pull/8550#discussion_r497343799 ## File path: tests/providers/google/cloud/operators/test_dataflow_system.py ## @@ -26,4 +37,191 @@ class CloudDataflowExampleDagsSystemTest(GoogleSystemT

[GitHub] [airflow] turbaszek commented on a change in pull request #10991: add azure files to gcs transfer operator

2020-09-30 Thread GitBox
turbaszek commented on a change in pull request #10991: URL: https://github.com/apache/airflow/pull/10991#discussion_r497344161 ## File path: airflow/providers/google/cloud/transfers/azure_fileshare_to_gcs.py ## @@ -0,0 +1,191 @@ +# +# Licensed to the Apache Software Foundation

[GitHub] [airflow] turbaszek commented on a change in pull request #10991: add azure files to gcs transfer operator

2020-09-30 Thread GitBox
turbaszek commented on a change in pull request #10991: URL: https://github.com/apache/airflow/pull/10991#discussion_r497344394 ## File path: airflow/providers/google/cloud/transfers/azure_fileshare_to_gcs.py ## @@ -0,0 +1,191 @@ +# +# Licensed to the Apache Software Foundation

[GitHub] [airflow] turbaszek commented on a change in pull request #10991: add azure files to gcs transfer operator

2020-09-30 Thread GitBox
turbaszek commented on a change in pull request #10991: URL: https://github.com/apache/airflow/pull/10991#discussion_r497345352 ## File path: airflow/providers/google/cloud/transfers/azure_fileshare_to_gcs.py ## @@ -0,0 +1,191 @@ +# +# Licensed to the Apache Software Foundation

[GitHub] [airflow] turbaszek commented on a change in pull request #10991: add azure files to gcs transfer operator

2020-09-30 Thread GitBox
turbaszek commented on a change in pull request #10991: URL: https://github.com/apache/airflow/pull/10991#discussion_r497346816 ## File path: airflow/providers/google/cloud/transfers/azure_fileshare_to_gcs.py ## @@ -0,0 +1,191 @@ +# +# Licensed to the Apache Software Foundation

[GitHub] [airflow] turbaszek commented on a change in pull request #10991: add azure files to gcs transfer operator

2020-09-30 Thread GitBox
turbaszek commented on a change in pull request #10991: URL: https://github.com/apache/airflow/pull/10991#discussion_r497347352 ## File path: airflow/providers/google/cloud/transfers/azure_fileshare_to_gcs.py ## @@ -0,0 +1,191 @@ +# +# Licensed to the Apache Software Foundation

[GitHub] [airflow] turbaszek commented on a change in pull request #10991: add azure files to gcs transfer operator

2020-09-30 Thread GitBox
turbaszek commented on a change in pull request #10991: URL: https://github.com/apache/airflow/pull/10991#discussion_r497348834 ## File path: airflow/providers/google/cloud/transfers/azure_fileshare_to_gcs.py ## @@ -0,0 +1,191 @@ +# +# Licensed to the Apache Software Foundation

[GitHub] [airflow] yamrzou opened a new issue #11201: Timeouts in KubernetesPodOperator

2020-09-30 Thread GitBox
yamrzou opened a new issue #11201: URL: https://github.com/apache/airflow/issues/11201 I have a number of long running KubernetesPodOperator tasks, which fail after around two hours with the following error. ``` [2020-09-30 05:44:27,065] {pod_launcher.py:173} INFO - Event: run-pr

[GitHub] [airflow] feluelle commented on a change in pull request #10947: Add amazon glacier to GCS transfer operator

2020-09-30 Thread GitBox
feluelle commented on a change in pull request #10947: URL: https://github.com/apache/airflow/pull/10947#discussion_r497360648 ## File path: airflow/providers/amazon/aws/transfers/glacier_to_gcs.py ## @@ -0,0 +1,103 @@ +# +# Licensed to the Apache Software Foundation (ASF) unde

[GitHub] [airflow] mik-laj commented on a change in pull request #11197: Add tips and tricks oracle shops page to production-deployment.rst

2020-09-30 Thread GitBox
mik-laj commented on a change in pull request #11197: URL: https://github.com/apache/airflow/pull/11197#discussion_r497360455 ## File path: docs/production-deployment.rst ## @@ -556,3 +556,36 @@ Keytab secret and both containers in the same Pod share the volume, where tempor

[GitHub] [airflow] feluelle commented on a change in pull request #10947: Add amazon glacier to GCS transfer operator

2020-09-30 Thread GitBox
feluelle commented on a change in pull request #10947: URL: https://github.com/apache/airflow/pull/10947#discussion_r497360648 ## File path: airflow/providers/amazon/aws/transfers/glacier_to_gcs.py ## @@ -0,0 +1,103 @@ +# +# Licensed to the Apache Software Foundation (ASF) unde

[GitHub] [airflow] mik-laj commented on issue #7997: Having a REST layer for all command line interface of Airflow

2020-09-30 Thread GitBox
mik-laj commented on issue #7997: URL: https://github.com/apache/airflow/issues/7997#issuecomment-701270166 Related: https://github.com/apache/airflow/issues/10552 This is an automated message from the Apache Git Service. To

[GitHub] [airflow] mik-laj commented on issue #7967: Add dag_run to the task_instance table or create new taskuuid column and use this to uniquely identify a task

2020-09-30 Thread GitBox
mik-laj commented on issue #7967: URL: https://github.com/apache/airflow/issues/7967#issuecomment-701275781 It is possible to increase the gradation. You need to set "replace_microseconds" to "false" (boolean as string) in request body or use experimental API: More: https://airflow.

[GitHub] [airflow] michalslowikowski00 commented on a change in pull request #10947: Add amazon glacier to GCS transfer operator

2020-09-30 Thread GitBox
michalslowikowski00 commented on a change in pull request #10947: URL: https://github.com/apache/airflow/pull/10947#discussion_r497376687 ## File path: docs/howto/operator/amazon/aws/glacier.rst ## @@ -0,0 +1,90 @@ + .. Licensed to the Apache Software Foundation (ASF) under one

[GitHub] [airflow] michalslowikowski00 commented on a change in pull request #10947: Add amazon glacier to GCS transfer operator

2020-09-30 Thread GitBox
michalslowikowski00 commented on a change in pull request #10947: URL: https://github.com/apache/airflow/pull/10947#discussion_r497379355 ## File path: airflow/providers/amazon/aws/transfers/glacier_to_gcs.py ## @@ -0,0 +1,122 @@ +# +# Licensed to the Apache Software Foundation

[jira] [Commented] (AIRFLOW-6585) Timestamp bug in RefreshKubeConfigLoader

2020-09-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17204613#comment-17204613 ] ASF GitHub Bot commented on AIRFLOW-6585: - ramsesrm commented on pull request #

[GitHub] [airflow] ramsesrm commented on pull request #7153: [AIRFLOW-6585] Fixed Timestamp bug in RefreshKubeConfigLoader

2020-09-30 Thread GitBox
ramsesrm commented on pull request #7153: URL: https://github.com/apache/airflow/pull/7153#issuecomment-701286290 Hi, this is still an issue on Airflow v1.10.12. Can we reopen the PR? This is an automated message from the Apa

[GitHub] [airflow] michalslowikowski00 commented on a change in pull request #10947: Add amazon glacier to GCS transfer operator

2020-09-30 Thread GitBox
michalslowikowski00 commented on a change in pull request #10947: URL: https://github.com/apache/airflow/pull/10947#discussion_r497384382 ## File path: airflow/providers/amazon/aws/transfers/glacier_to_gcs.py ## @@ -0,0 +1,110 @@ +# +# Licensed to the Apache Software Foundation

[GitHub] [airflow] arsalan993 opened a new issue #11202: Airflow flower: Connection reset

2020-09-30 Thread GitBox
arsalan993 opened a new issue #11202: URL: https://github.com/apache/airflow/issues/11202 I am using airflow using celery executor and monitoring it using airflow dashboard and airflow flower. After couple of minutes later i get following error on terminal running flower `[E 200

[GitHub] [airflow] boring-cyborg[bot] commented on issue #11202: Airflow flower: Connection reset

2020-09-30 Thread GitBox
boring-cyborg[bot] commented on issue #11202: URL: https://github.com/apache/airflow/issues/11202#issuecomment-701293867 Thanks for opening your first issue here! Be sure to follow the issue template! This is an automate

[GitHub] [airflow] FHoffmannCode commented on a change in pull request #10991: add azure files to gcs transfer operator

2020-09-30 Thread GitBox
FHoffmannCode commented on a change in pull request #10991: URL: https://github.com/apache/airflow/pull/10991#discussion_r497400757 ## File path: airflow/models/baseoperator.py ## @@ -89,7 +89,7 @@ class BaseOperator(Operator, LoggingMixin, metaclass=BaseOperatorMeta): Ab

[GitHub] [airflow] yuqian90 commented on issue #9557: Improvements to the "Executor reports task instance finished (failed) although the task says its queued" exception messages

2020-09-30 Thread GitBox
yuqian90 commented on issue #9557: URL: https://github.com/apache/airflow/issues/9557#issuecomment-701302315 There's an open issue discussing several different causes that can lead to this error. I would suggest we keep the error message the same for the time being until the issue is corre

[GitHub] [airflow] boring-cyborg[bot] commented on issue #11203: Big Query Location Not working

2020-09-30 Thread GitBox
boring-cyborg[bot] commented on issue #11203: URL: https://github.com/apache/airflow/issues/11203#issuecomment-701303009 Thanks for opening your first issue here! Be sure to follow the issue template! This is an automate

[GitHub] [airflow] Iskz opened a new issue #11203: Big Query Location Not working

2020-09-30 Thread GitBox
Iskz opened a new issue #11203: URL: https://github.com/apache/airflow/issues/11203 We are having issue when trying to use BigQueryHook to retrieve a dataset which is NOT IN US ``` from airflow.contrib.hooks.bigquery_hook import BigQueryHook bq_hook = BigQueryHook(bigquery_conn

[GitHub] [airflow] turbaszek commented on a change in pull request #10947: Add amazon glacier to GCS transfer operator

2020-09-30 Thread GitBox
turbaszek commented on a change in pull request #10947: URL: https://github.com/apache/airflow/pull/10947#discussion_r497404001 ## File path: airflow/providers/amazon/aws/transfers/glacier_to_gcs.py ## @@ -0,0 +1,122 @@ +# +# Licensed to the Apache Software Foundation (ASF) und

[GitHub] [airflow] turbaszek commented on a change in pull request #10947: Add amazon glacier to GCS transfer operator

2020-09-30 Thread GitBox
turbaszek commented on a change in pull request #10947: URL: https://github.com/apache/airflow/pull/10947#discussion_r497404001 ## File path: airflow/providers/amazon/aws/transfers/glacier_to_gcs.py ## @@ -0,0 +1,122 @@ +# +# Licensed to the Apache Software Foundation (ASF) und

[GitHub] [airflow] FHoffmannCode commented on a change in pull request #10991: add azure files to gcs transfer operator

2020-09-30 Thread GitBox
FHoffmannCode commented on a change in pull request #10991: URL: https://github.com/apache/airflow/pull/10991#discussion_r497414640 ## File path: airflow/providers/google/cloud/transfers/azure_fileshare_to_gcs.py ## @@ -0,0 +1,191 @@ +# +# Licensed to the Apache Software Founda

[GitHub] [airflow] FHoffmannCode commented on a change in pull request #10991: add azure files to gcs transfer operator

2020-09-30 Thread GitBox
FHoffmannCode commented on a change in pull request #10991: URL: https://github.com/apache/airflow/pull/10991#discussion_r497414850 ## File path: airflow/providers/google/cloud/transfers/azure_fileshare_to_gcs.py ## @@ -0,0 +1,191 @@ +# +# Licensed to the Apache Software Founda

[GitHub] [airflow] FHoffmannCode commented on a change in pull request #10991: add azure files to gcs transfer operator

2020-09-30 Thread GitBox
FHoffmannCode commented on a change in pull request #10991: URL: https://github.com/apache/airflow/pull/10991#discussion_r497416770 ## File path: airflow/providers/google/cloud/transfers/azure_fileshare_to_gcs.py ## @@ -0,0 +1,191 @@ +# +# Licensed to the Apache Software Founda

[GitHub] [airflow] KIRY4 commented on issue #10860: Timeouts in Airflow scheduler logs on AKS

2020-09-30 Thread GitBox
KIRY4 commented on issue #10860: URL: https://github.com/apache/airflow/issues/10860#issuecomment-701314516 > Hi I am experiencing the same problem and don't know how to fix it, is there any kind of update? Yeah I'm still experiencing it too. Also I'm struggling from following messa

[GitHub] [airflow] FHoffmannCode commented on a change in pull request #10991: add azure files to gcs transfer operator

2020-09-30 Thread GitBox
FHoffmannCode commented on a change in pull request #10991: URL: https://github.com/apache/airflow/pull/10991#discussion_r497418763 ## File path: airflow/providers/google/cloud/transfers/azure_fileshare_to_gcs.py ## @@ -0,0 +1,191 @@ +# +# Licensed to the Apache Software Founda

[GitHub] [airflow] michalslowikowski00 commented on a change in pull request #10947: Add amazon glacier to GCS transfer operator

2020-09-30 Thread GitBox
michalslowikowski00 commented on a change in pull request #10947: URL: https://github.com/apache/airflow/pull/10947#discussion_r497427101 ## File path: airflow/providers/amazon/aws/transfers/glacier_to_gcs.py ## @@ -0,0 +1,122 @@ +# +# Licensed to the Apache Software Foundation

[GitHub] [airflow] TobKed commented on issue #11203: Big Query Location Not working

2020-09-30 Thread GitBox
TobKed commented on issue #11203: URL: https://github.com/apache/airflow/issues/11203#issuecomment-701323961 You can try to use `airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.run_query` method. However `bq_hook.get_records` should work anyway. Which version of airflow d

[GitHub] [airflow] mik-laj commented on issue #11073: Test that the xcom API endpoint handles joins properly

2020-09-30 Thread GitBox
mik-laj commented on issue #11073: URL: https://github.com/apache/airflow/issues/11073#issuecomment-701325378 CC: @randr97 @ephraimbuddy This is an automated message from the Apache Git Service. To respond to the message, pl

[GitHub] [airflow] mik-laj commented on issue #8666: K8S Pods labels have task_id and dag_id which then gets warned

2020-09-30 Thread GitBox
mik-laj commented on issue #8666: URL: https://github.com/apache/airflow/issues/8666#issuecomment-701329274 @lihan The problem is that you have your DAG_ID or Task ID contains illegal characters. In that case, a safe path is started which is very slow. See: https://github.com/apache/airflo

[GitHub] [airflow] mik-laj commented on issue #8803: Impersonate service accounts while running GCP Operators without key material (if airflow is running on GCP)

2020-09-30 Thread GitBox
mik-laj commented on issue #8803: URL: https://github.com/apache/airflow/issues/8803#issuecomment-701334321 @olchas Is is done? Is there anything else to do that is not described in #10596? This is an automated message from

[GitHub] [airflow] Iskz commented on issue #11203: Big Query Location Not working

2020-09-30 Thread GitBox
Iskz commented on issue #11203: URL: https://github.com/apache/airflow/issues/11203#issuecomment-701335246 currently i'm on version 1.10.12 for airflow.providers.google.cloud.hooks.bigquery.BigQueryHook package are you referring to https://pypi.org/project/apache-airflow-backport-pro

[GitHub] [airflow] anirudhbagri commented on issue #8160: SimpleHttpOperator aborts connection after 5 minutes

2020-09-30 Thread GitBox
anirudhbagri commented on issue #8160: URL: https://github.com/apache/airflow/issues/8160#issuecomment-701345257 I faced a similar issue, for me it was a bad request that was sent using `http.hook`. Fixed it by correcting the request body. --

[GitHub] [airflow] turbaszek merged pull request #10947: Add amazon glacier to GCS transfer operator

2020-09-30 Thread GitBox
turbaszek merged pull request #10947: URL: https://github.com/apache/airflow/pull/10947 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [airflow] FHoffmannCode commented on a change in pull request #10991: add azure files to gcs transfer operator

2020-09-30 Thread GitBox
FHoffmannCode commented on a change in pull request #10991: URL: https://github.com/apache/airflow/pull/10991#discussion_r497488779 ## File path: airflow/providers/google/cloud/transfers/azure_fileshare_to_gcs.py ## @@ -0,0 +1,191 @@ +# +# Licensed to the Apache Software Founda

[airflow] branch master updated (9860719 -> 00ffedb)

2020-09-30 Thread turbaszek
This is an automated email from the ASF dual-hosted git repository. turbaszek pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from 9860719 [AIRFLOW-5545] Fixes recursion in DAG cycle testing (#6175) add 00ffedb Add amazon glacier to GCS

[airflow] branch master updated (9860719 -> 00ffedb)

2020-09-30 Thread turbaszek
This is an automated email from the ASF dual-hosted git repository. turbaszek pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from 9860719 [AIRFLOW-5545] Fixes recursion in DAG cycle testing (#6175) add 00ffedb Add amazon glacier to GCS

[GitHub] [airflow] ryanahamilton commented on a change in pull request #11195: 2.0 UI Overhaul/Refresh

2020-09-30 Thread GitBox
ryanahamilton commented on a change in pull request #11195: URL: https://github.com/apache/airflow/pull/11195#discussion_r497499654 ## File path: airflow/www/templates/airflow/code.html ## @@ -22,22 +22,22 @@ {% block page_title %}{{ dag.dag_id }} - Code - Airflow{% endblock %

[GitHub] [airflow] tooptoop4 opened a new issue #11204: another state for dagrun - skipped

2020-09-30 Thread GitBox
tooptoop4 opened a new issue #11204: URL: https://github.com/apache/airflow/issues/11204 right now dagrun returns state of success/failed. I would like a 3rd state - skipped. use case: dag polls a website looking for a change, if website has not changed I would like the dagrun (not

[airflow] branch constraints-master updated: Updating constraints. Build id:280455594

2020-09-30 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-master in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-master by this push: new e5c5c18 Updating

[airflow] branch constraints-master updated: Updating constraints. Build id:280455594

2020-09-30 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-master in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-master by this push: new e5c5c18 Updating

[GitHub] [airflow] TobKed commented on a change in pull request #8550: Add DataflowStartFlexTemplateOperator

2020-09-30 Thread GitBox
TobKed commented on a change in pull request #8550: URL: https://github.com/apache/airflow/pull/8550#discussion_r497534203 ## File path: airflow/providers/google/cloud/operators/dataflow.py ## @@ -406,6 +406,71 @@ def on_kill(self) -> None: self.hook.cancel_job(job

[GitHub] [airflow] boring-cyborg[bot] commented on issue #11205: Simplify DataflowJobStatus states

2020-09-30 Thread GitBox
boring-cyborg[bot] commented on issue #11205: URL: https://github.com/apache/airflow/issues/11205#issuecomment-701423351 Thanks for opening your first issue here! Be sure to follow the issue template! This is an automate

[GitHub] [airflow] TobKed opened a new issue #11205: Simplify DataflowJobStatus states

2020-09-30 Thread GitBox
TobKed opened a new issue #11205: URL: https://github.com/apache/airflow/issues/11205 **Description** `JOB_STATE_ prefix` is not necessary in `airflow.providers.google.cloud.hooks.dataflow.DataflowJobStatus` states names and would be nice to simplify this. https://git

[GitHub] [airflow] TobKed commented on issue #11205: Simplify DataflowJobStatus states

2020-09-30 Thread GitBox
TobKed commented on issue #11205: URL: https://github.com/apache/airflow/issues/11205#issuecomment-701423402 cc @jaketf This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

[GitHub] [airflow] TobKed commented on a change in pull request #8550: Add DataflowStartFlexTemplateOperator

2020-09-30 Thread GitBox
TobKed commented on a change in pull request #8550: URL: https://github.com/apache/airflow/pull/8550#discussion_r497550006 ## File path: tests/providers/google/cloud/hooks/test_dataflow.py ## @@ -984,15 +1034,17 @@ def test_dataflow_job_is_job_running_with_no_job(self):

[GitHub] [airflow] TobKed commented on a change in pull request #8553: Add DataflowStartSQLQuery operator

2020-09-30 Thread GitBox
TobKed commented on a change in pull request #8553: URL: https://github.com/apache/airflow/pull/8553#discussion_r497550417 ## File path: airflow/providers/google/cloud/hooks/dataflow.py ## @@ -93,15 +94,30 @@ def inner_wrapper(self: "DataflowHook", *args, **kwargs) -> RT: cla

[GitHub] [airflow] ashmeet13 commented on issue #11144: Create UndefinedJinjaVariablesRule to ease upgrade to 2.0

2020-09-30 Thread GitBox
ashmeet13 commented on issue #11144: URL: https://github.com/apache/airflow/issues/11144#issuecomment-701439517 Some progress that I have made. Thanks @ashb for guiding towards `TaskInstance` for getting the complete context. Also towards `BaseOperator.render_template` With this

[GitHub] [airflow] avirlrma commented on issue #11177: Add template fields renderers to operators

2020-09-30 Thread GitBox
avirlrma commented on issue #11177: URL: https://github.com/apache/airflow/issues/11177#issuecomment-701453271 Hey, Can I take this up? This is an automated message from the Apache Git Service. To respond to the message, p

[GitHub] [airflow] ashb commented on issue #10279: Tooltips not show up in Safari

2020-09-30 Thread GitBox
ashb commented on issue #10279: URL: https://github.com/apache/airflow/issues/10279#issuecomment-701454963 If this only applies to non-RBAC UI then this is less important - the old UI isn't under active development and only receiving critical bug fixes only. PR to fix this should tar

[GitHub] [airflow] TobKed commented on issue #11203: Big Query Location Not working

2020-09-30 Thread GitBox
TobKed commented on issue #11203: URL: https://github.com/apache/airflow/issues/11203#issuecomment-701461602 I referred to the latest state on the master. I will investigate how backport packages works, I am not familiar with them. I found PR related to the location support https://gith

[GitHub] [airflow] boring-cyborg[bot] commented on issue #11206: Systemwide installed airflow won't run scheduler using non-root user depending on HOME user location

2020-09-30 Thread GitBox
boring-cyborg[bot] commented on issue #11206: URL: https://github.com/apache/airflow/issues/11206#issuecomment-701463570 Thanks for opening your first issue here! Be sure to follow the issue template! This is an automate

[GitHub] [airflow] Dubrzr opened a new issue #11206: Systemwide installed airflow won't run scheduler using non-root user depending on HOME user location

2020-09-30 Thread GitBox
Dubrzr opened a new issue #11206: URL: https://github.com/apache/airflow/issues/11206 **Apache Airflow version**: 1.10.12 **Environment**: docker image running ubuntu 20.04 - **OS** (e.g. from /etc/os-release): Ubuntu 20.04 - **Kernel** (e.g. `uname -a`): 3.

[GitHub] [airflow] francescomucio opened a new pull request #11207: Breeze start-airflow command wasn't able to initialize the db in 1.10.x

2020-09-30 Thread GitBox
francescomucio opened a new pull request #11207: URL: https://github.com/apache/airflow/pull/11207 The database initialization needs a different cli command (the old `initdb`). Also it is not necessary to create a dummy user in version 1.10.x to allow people to login in the Web UI.

[GitHub] [airflow] Sangarshanan commented on a change in pull request #11132: Add loggingconfig rule

2020-09-30 Thread GitBox
Sangarshanan commented on a change in pull request #11132: URL: https://github.com/apache/airflow/pull/11132#discussion_r497606455 ## File path: airflow/upgrade/rules/logging_configuration.py ## @@ -0,0 +1,83 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# o

[GitHub] [airflow] potiuk commented on a change in pull request #11207: Breeze start-airflow command wasn't able to initialize the db in 1.10.x

2020-09-30 Thread GitBox
potiuk commented on a change in pull request #11207: URL: https://github.com/apache/airflow/pull/11207#discussion_r497612001 ## File path: scripts/in_container/check_environment.sh ## @@ -107,8 +107,12 @@ function startairflow_if_requested() { . "$( dirname "${BASH_SOU

[GitHub] [airflow] sergioesca4 commented on a change in pull request #11197: Add tips and tricks oracle shops page to production-deployment.rst

2020-09-30 Thread GitBox
sergioesca4 commented on a change in pull request #11197: URL: https://github.com/apache/airflow/pull/11197#discussion_r497615185 ## File path: docs/production-deployment.rst ## @@ -556,3 +556,36 @@ Keytab secret and both containers in the same Pod share the volume, where temp

[GitHub] [airflow] sergioesca4 commented on a change in pull request #11197: Add tips and tricks oracle shops page to production-deployment.rst

2020-09-30 Thread GitBox
sergioesca4 commented on a change in pull request #11197: URL: https://github.com/apache/airflow/pull/11197#discussion_r497615558 ## File path: docs/production-deployment.rst ## @@ -556,3 +556,36 @@ Keytab secret and both containers in the same Pod share the volume, where temp

[GitHub] [airflow] mik-laj commented on a change in pull request #11197: Add tips and tricks oracle shops page to production-deployment.rst

2020-09-30 Thread GitBox
mik-laj commented on a change in pull request #11197: URL: https://github.com/apache/airflow/pull/11197#discussion_r497616651 ## File path: docs/production-deployment.rst ## @@ -556,3 +556,36 @@ Keytab secret and both containers in the same Pod share the volume, where tempor

[GitHub] [airflow] francescomucio commented on a change in pull request #11207: Breeze start-airflow command wasn't able to initialize the db in 1.10.x

2020-09-30 Thread GitBox
francescomucio commented on a change in pull request #11207: URL: https://github.com/apache/airflow/pull/11207#discussion_r497617047 ## File path: scripts/in_container/check_environment.sh ## @@ -107,8 +107,12 @@ function startairflow_if_requested() { . "$( dirname "${

[GitHub] [airflow] boring-cyborg[bot] commented on issue #11208: Beginners documentation for data pipelines

2020-09-30 Thread GitBox
boring-cyborg[bot] commented on issue #11208: URL: https://github.com/apache/airflow/issues/11208#issuecomment-701487548 Thanks for opening your first issue here! Be sure to follow the issue template! This is an automate

[GitHub] [airflow] datapythonista opened a new issue #11208: Beginners documentation for data pipelines

2020-09-30 Thread GitBox
datapythonista opened a new issue #11208: URL: https://github.com/apache/airflow/issues/11208 **Description** I'm new to Airflow, and it was very easy to get started creating a simple DAG. Thanks for the tutorial and other beginner docs. But since getting the "hello world" kind of D

[GitHub] [airflow] eladkal commented on a change in pull request #11197: Add tips and tricks oracle shops page to production-deployment.rst

2020-09-30 Thread GitBox
eladkal commented on a change in pull request #11197: URL: https://github.com/apache/airflow/pull/11197#discussion_r497657297 ## File path: docs/production-deployment.rst ## @@ -556,3 +556,36 @@ Keytab secret and both containers in the same Pod share the volume, where tempor

[GitHub] [airflow] eladkal commented on issue #11204: another state for dagrun - skipped

2020-09-30 Thread GitBox
eladkal commented on issue #11204: URL: https://github.com/apache/airflow/issues/11204#issuecomment-701517505 Correct me if I'm wrong here but in your suggestion the DAG must run in order to check the website status. So you expect the DAG to run but eventually mark itself as SKIPPED? -

[GitHub] [airflow] tooptoop4 commented on issue #11204: another state for dagrun - skipped

2020-09-30 Thread GitBox
tooptoop4 commented on issue #11204: URL: https://github.com/apache/airflow/issues/11204#issuecomment-701525801 yes, name could be RAN_WITH_SKIPPED_RESULT or whatever This is an automated message from the Apache Git Service.

[GitHub] [airflow] XD-DENG commented on issue #10624: 发送钉钉webhook添加其他参数 如:代理

2020-09-30 Thread GitBox
XD-DENG commented on issue #10624: URL: https://github.com/apache/airflow/issues/10624#issuecomment-701530691 Hi @mik-laj , I have requested for an English description earlier, while no response from the issue author. I suggest to leave it, @zhongjiajie have assigned this issue to himself

[GitHub] [airflow] sergioesca4 commented on a change in pull request #11197: Add tips and tricks oracle shops page to production-deployment.rst

2020-09-30 Thread GitBox
sergioesca4 commented on a change in pull request #11197: URL: https://github.com/apache/airflow/pull/11197#discussion_r497680790 ## File path: docs/production-deployment.rst ## @@ -556,3 +556,36 @@ Keytab secret and both containers in the same Pod share the volume, where temp

[GitHub] [airflow] eladkal opened a new pull request #11209: Aws dynamodb

2020-09-30 Thread GitBox
eladkal opened a new pull request #11209: URL: https://github.com/apache/airflow/pull/11209 Removing `aws_` prefix from `aws_dynamodb.py` to allign with the rest of [AWS hook files](https://github.com/apache/airflow/tree/master/airflow/providers/amazon/aws/hooks). --

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #11210: Change job submission template from underscores to dashes to conform with k8s naming convention for Datraproc on GKE

2020-09-30 Thread GitBox
boring-cyborg[bot] commented on pull request #11210: URL: https://github.com/apache/airflow/pull/11210#issuecomment-701582358 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Con

[GitHub] [airflow] omarismail94 opened a new pull request #11210: Change job submission template from underscores to dashes to conform with k8s naming convention for Datraproc on GKE

2020-09-30 Thread GitBox
omarismail94 opened a new pull request #11210: URL: https://github.com/apache/airflow/pull/11210 Change job submission template from underscores to dashes to conform with k8s naming convention for Datraproc on GKE [1] Rn, if you submit a job with the `DataProcPySparkOperator` to a Da

[GitHub] [airflow] turbaszek opened a new pull request #11211: Add Github Code Scanning

2020-09-30 Thread GitBox
turbaszek opened a new pull request #11211: URL: https://github.com/apache/airflow/pull/11211 Github just released Github Code Scanning: https://github.blog/2020-09-30-code-scanning-is-now-available/ --- **^ Add meaningful description above** Read the **[Pull Requ

[GitHub] [airflow] iadi7ya opened a new pull request #11212: Create hdfs.rst

2020-09-30 Thread GitBox
iadi7ya opened a new pull request #11212: URL: https://github.com/apache/airflow/pull/11212 Guide for HDFS operators. Created initial guide for HDFS operators explaining the Sensor Classes for hdfs available in existing code. --- **^ Add meaningful description above**

[GitHub] [airflow] turbaszek commented on issue #11203: Big Query Location Not working

2020-09-30 Thread GitBox
turbaszek commented on issue #11203: URL: https://github.com/apache/airflow/issues/11203#issuecomment-701640605 The `BigQueryHook` from providers package has no `get_records` method. Please see the information about backport packages: https://airflow.readthedocs.io/en/latest/backport-pro

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #11212: Create hdfs.rst

2020-09-30 Thread GitBox
boring-cyborg[bot] commented on pull request #11212: URL: https://github.com/apache/airflow/pull/11212#issuecomment-701640537 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Con

[GitHub] [airflow] turbaszek commented on issue #11177: Add template fields renderers to operators

2020-09-30 Thread GitBox
turbaszek commented on issue #11177: URL: https://github.com/apache/airflow/issues/11177#issuecomment-701642129 @avirlrma please feel free to open a PR referencing this issue. You don't have to change all operators at once 👍 ---

[GitHub] [airflow] turbaszek commented on issue #11208: Beginners documentation for data pipelines

2020-09-30 Thread GitBox
turbaszek commented on issue #11208: URL: https://github.com/apache/airflow/issues/11208#issuecomment-701644859 @datapythonista would you like to contribute such documentation? I think it would be awesome 🚀 This is an autom

[GitHub] [airflow] turbaszek commented on issue #11202: Airflow flower: Connection reset

2020-09-30 Thread GitBox
turbaszek commented on issue #11202: URL: https://github.com/apache/airflow/issues/11202#issuecomment-701645566 @arsalan993 is there any problem with your Airflow deployment or you only see log messages like this one (which I think comes from postgres?)? --

[GitHub] [airflow] turbaszek edited a comment on issue #11202: Airflow flower: Connection reset

2020-09-30 Thread GitBox
turbaszek edited a comment on issue #11202: URL: https://github.com/apache/airflow/issues/11202#issuecomment-701645566 @arsalan993 is there any problem with your Airflow deployment or you only see log messages like this one?

[GitHub] [airflow] datapythonista commented on issue #11208: Beginners documentation for data pipelines

2020-09-30 Thread GitBox
datapythonista commented on issue #11208: URL: https://github.com/apache/airflow/issues/11208#issuecomment-701648028 > @datapythonista would you like to contribute such documentation? I think it would be awesome Absolutely, as soon as I confirm this makes sense, and I understand myse

  1   2   >