[GitHub] [airflow] j-y-matsubara commented on a change in pull request #8757: Fix the incorrect description of pod_mutation_hook in kubernetes.rst

2020-05-08 Thread GitBox
j-y-matsubara commented on a change in pull request #8757: URL: https://github.com/apache/airflow/pull/8757#discussion_r422455491 ## File path: docs/kubernetes.rst ## @@ -40,11 +40,44 @@ that has the ability to mutate pod objects before sending them to the Kubernetes for

[GitHub] [airflow] j-y-matsubara commented on a change in pull request #8757: Fix the incorrect description of pod_mutation_hook in kubernetes.rst

2020-05-08 Thread GitBox
j-y-matsubara commented on a change in pull request #8757: URL: https://github.com/apache/airflow/pull/8757#discussion_r422455718 ## File path: docs/kubernetes.rst ## @@ -40,11 +40,44 @@ that has the ability to mutate pod objects before sending them to the Kubernetes for

[GitHub] [airflow] j-y-matsubara commented on a change in pull request #8757: Fix the incorrect description of pod_mutation_hook in kubernetes.rst

2020-05-08 Thread GitBox
j-y-matsubara commented on a change in pull request #8757: URL: https://github.com/apache/airflow/pull/8757#discussion_r422455491 ## File path: docs/kubernetes.rst ## @@ -40,11 +40,44 @@ that has the ability to mutate pod objects before sending them to the Kubernetes for

[airflow] tag nightly-master updated (6e4f5fa -> 42c5975)

2020-05-08 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to tag nightly-master in repository https://gitbox.apache.org/repos/asf/airflow.git. *** WARNING: tag nightly-master was modified! *** from 6e4f5fa (commit) to 42c5975 (commit) from

[GitHub] [airflow] zsmeijin edited a comment on issue #8676: Task exited with return code 1 without any warning/error message after reboot server and restart service

2020-05-08 Thread GitBox
zsmeijin edited a comment on issue #8676: URL: https://github.com/apache/airflow/issues/8676#issuecomment-626074212 I finally figure out how to reproduce this bug. When you config email in airflow.cfg and your dag contains email operator or use smtp serivce, if your smtp password

[GitHub] [airflow] zsmeijin commented on issue #8676: Task exited with return code 1 without any warning/error message after reboot server and restart service

2020-05-08 Thread GitBox
zsmeijin commented on issue #8676: URL: https://github.com/apache/airflow/issues/8676#issuecomment-626074212 I finally figure out how to reproduce this bug. If you config email in airflow.cfg and your dag contains email operator or use smtp serivce, and there's something wrong with

[GitHub] [airflow] jhtimmins commented on issue #8674: Fix problems with, and test agsinst, Python 3.8

2020-05-08 Thread GitBox
jhtimmins commented on issue #8674: URL: https://github.com/apache/airflow/issues/8674#issuecomment-626068865 PR #8794 is part of this issue This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] jhtimmins opened a new pull request #8794: Use fork when test relies on mock.patch in parent process.

2020-05-08 Thread GitBox
jhtimmins opened a new pull request #8794: URL: https://github.com/apache/airflow/pull/8794 The `test_scheduler_reschedule` method fails when creating processes in `spawn` mode, which is the default mode for Python 3.8 on MacOS. `test_scheduler_reschedule` depends on `mock.patch()`

[airflow] branch master updated (a091c1f -> 42c5975)

2020-05-08 Thread kamilbregula
This is an automated email from the ASF dual-hosted git repository. kamilbregula pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from a091c1f fix typing errors reported by dmypy (#8773) add 42c5975 Update example SingularityOperator DAG

[GitHub] [airflow] jvstein opened a new issue #8792: KubernetesPodOperator does not return XCOM on pod failure

2020-05-08 Thread GitBox
jvstein opened a new issue #8792: URL: https://github.com/apache/airflow/issues/8792 **Apache Airflow version**: 1.10.9 **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): 1.14.9 **Environment**: - **Cloud provider or hardware

[GitHub] [airflow] boring-cyborg[bot] commented on issue #8792: KubernetesPodOperator does not return XCOM on pod failure

2020-05-08 Thread GitBox
boring-cyborg[bot] commented on issue #8792: URL: https://github.com/apache/airflow/issues/8792#issuecomment-626055393 Thanks for opening your first issue here! Be sure to follow the issue template! This is an automated

[GitHub] [airflow] ashb commented on a change in pull request #8701: Adding ElastiCache Hook for creating, describing and deleting replication groups

2020-05-08 Thread GitBox
ashb commented on a change in pull request #8701: URL: https://github.com/apache/airflow/pull/8701#discussion_r422410165 ## File path: airflow/providers/amazon/aws/hooks/elasticache.py ## @@ -0,0 +1,238 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or

[GitHub] [airflow] peter-volkov opened a new pull request #8791: [8650] Add Yandex.Cloud custom connection to 1.10 for backported provider

2020-05-08 Thread GitBox
peter-volkov opened a new pull request #8791: URL: https://github.com/apache/airflow/pull/8791 --- Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [ ] Unit tests coverage for changes (not needed for

[GitHub] [airflow] ashb commented on a change in pull request #8757: Fix the incorrect description of pod_mutation_hook in kubernetes.rst

2020-05-08 Thread GitBox
ashb commented on a change in pull request #8757: URL: https://github.com/apache/airflow/pull/8757#discussion_r422377934 ## File path: docs/kubernetes.rst ## @@ -40,11 +40,44 @@ that has the ability to mutate pod objects before sending them to the Kubernetes for scheduling.

[GitHub] [airflow] ashb commented on pull request #8769: Fix running DAG from UI when dag serialization is on

2020-05-08 Thread GitBox
ashb commented on pull request #8769: URL: https://github.com/apache/airflow/pull/8769#issuecomment-626022797 What are the steps to reproduce this error with serialization turned on please? This is an automated message from

[GitHub] [airflow] ashb commented on a change in pull request #8757: Fix the incorrect description of pod_mutation_hook in kubernetes.rst

2020-05-08 Thread GitBox
ashb commented on a change in pull request #8757: URL: https://github.com/apache/airflow/pull/8757#discussion_r422377856 ## File path: docs/kubernetes.rst ## @@ -40,11 +40,44 @@ that has the ability to mutate pod objects before sending them to the Kubernetes for scheduling.

[airflow] branch master updated: fix typing errors reported by dmypy (#8773)

2020-05-08 Thread ash
This is an automated email from the ASF dual-hosted git repository. ash pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/master by this push: new a091c1f fix typing errors reported by dmypy

[GitHub] [airflow] ashb commented on a change in pull request #8772: Correctly store non-default Nones in serialized tasks/dags

2020-05-08 Thread GitBox
ashb commented on a change in pull request #8772: URL: https://github.com/apache/airflow/pull/8772#discussion_r422375825 ## File path: airflow/serialization/serialized_objects.py ## @@ -395,6 +411,59 @@ def deserialize_operator(cls, encoded_op: Dict[str, Any]) ->

[GitHub] [airflow] ashb opened a new pull request #8790: Update example SingularityOperator dag

2020-05-08 Thread GitBox
ashb opened a new pull request #8790: URL: https://github.com/apache/airflow/pull/8790 The main thing I was fixing here was `start_date=utcnow()` which is always going to be wrong (discovered via a test in #8772). While I was updating the DAG I updated it to use context manager,

[GitHub] [airflow] ashb commented on pull request #8772: Correctly store non-default Nones in serialized tasks/dags

2020-05-08 Thread GitBox
ashb commented on pull request #8772: URL: https://github.com/apache/airflow/pull/8772#issuecomment-626013593 > Oh, cos it's parsing in a subprocess. H! > > I don't think we need to parse _all_ dags in a subprocess, just one is enough. Changed it to only parse

[GitHub] [airflow] dimon222 commented on issue #8212: Can't read S3 remote logs using Airflow 1.10.9

2020-05-08 Thread GitBox
dimon222 commented on issue #8212: URL: https://github.com/apache/airflow/issues/8212#issuecomment-625964491 @webster-chainalysis Worker class switch in airflow.cfg from gevent/eventlet to "sync" seem to resolve the issue for me.

[GitHub] [airflow] tooptoop4 opened a new issue #8789: Ability to throttle scheduler bursts on single pool

2020-05-08 Thread GitBox
tooptoop4 opened a new issue #8789: URL: https://github.com/apache/airflow/issues/8789 Current issue is I have pools configured on same type of task where I can have up to max 100 pool slots of that task type running in parallel BUT I want ability to say don't START running more than 5 at

[GitHub] [airflow] ddelange commented on a change in pull request #6523: [AIRFLOW-5873] KubernetesPodOperator fixes and test

2020-05-08 Thread GitBox
ddelange commented on a change in pull request #6523: URL: https://github.com/apache/airflow/pull/6523#discussion_r422295222 ## File path: airflow/contrib/operators/kubernetes_pod_operator.py ## @@ -158,17 +167,17 @@ def execute(self, context): raise

[jira] [Commented] (AIRFLOW-5873) KubernetesPodOperator fixes and test

2020-05-08 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102820#comment-17102820 ] ASF GitHub Bot commented on AIRFLOW-5873: - ddelange commented on a change in pull request

[jira] [Commented] (AIRFLOW-5873) KubernetesPodOperator fixes and test

2020-05-08 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102812#comment-17102812 ] ASF GitHub Bot commented on AIRFLOW-5873: - ddelange commented on a change in pull request

[GitHub] [airflow] ddelange commented on a change in pull request #6523: [AIRFLOW-5873] KubernetesPodOperator fixes and test

2020-05-08 Thread GitBox
ddelange commented on a change in pull request #6523: URL: https://github.com/apache/airflow/pull/6523#discussion_r422295222 ## File path: airflow/contrib/operators/kubernetes_pod_operator.py ## @@ -158,17 +167,17 @@ def execute(self, context): raise

[GitHub] [airflow] dsaiztc commented on a change in pull request #6523: [AIRFLOW-5873] KubernetesPodOperator fixes and test

2020-05-08 Thread GitBox
dsaiztc commented on a change in pull request #6523: URL: https://github.com/apache/airflow/pull/6523#discussion_r422286551 ## File path: airflow/contrib/operators/kubernetes_pod_operator.py ## @@ -158,17 +167,17 @@ def execute(self, context): raise

[jira] [Commented] (AIRFLOW-5873) KubernetesPodOperator fixes and test

2020-05-08 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102801#comment-17102801 ] ASF GitHub Bot commented on AIRFLOW-5873: - dsaiztc commented on a change in pull request #6523:

[jira] [Commented] (AIRFLOW-5873) KubernetesPodOperator fixes and test

2020-05-08 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102797#comment-17102797 ] ASF GitHub Bot commented on AIRFLOW-5873: - dsaiztc commented on a change in pull request #6523:

[GitHub] [airflow] dsaiztc commented on a change in pull request #6523: [AIRFLOW-5873] KubernetesPodOperator fixes and test

2020-05-08 Thread GitBox
dsaiztc commented on a change in pull request #6523: URL: https://github.com/apache/airflow/pull/6523#discussion_r422286551 ## File path: airflow/contrib/operators/kubernetes_pod_operator.py ## @@ -158,17 +167,17 @@ def execute(self, context): raise

[jira] [Commented] (AIRFLOW-6786) Adding KafkaConsumerHook, KafkaProducerHook, and KafkaSensor

2020-05-08 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102715#comment-17102715 ] ASF GitHub Bot commented on AIRFLOW-6786: - dferguson992 commented on a change in pull request

[GitHub] [airflow] turbaszek commented on pull request #8610: Added Upload Multiple Entity Read Files to specified big query dataset

2020-05-08 Thread GitBox
turbaszek commented on pull request #8610: URL: https://github.com/apache/airflow/pull/8610#issuecomment-625898833 @michalslowikowski00 can you rebase please? This is an automated message from the Apache Git Service. To

[GitHub] [airflow] dferguson992 commented on a change in pull request #7407: [AIRFLOW-6786] Add KafkaConsumerHook, KafkaProduerHook and KafkaSensor

2020-05-08 Thread GitBox
dferguson992 commented on a change in pull request #7407: URL: https://github.com/apache/airflow/pull/7407#discussion_r422240631 ## File path: airflow/providers/apache/kafka/sensors/kafka_sensor.py ## @@ -0,0 +1,76 @@ +# +# Licensed to the Apache Software Foundation (ASF)

[jira] [Commented] (AIRFLOW-6786) Adding KafkaConsumerHook, KafkaProducerHook, and KafkaSensor

2020-05-08 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102707#comment-17102707 ] ASF GitHub Bot commented on AIRFLOW-6786: - dferguson992 commented on a change in pull request

[GitHub] [airflow] dferguson992 commented on a change in pull request #7407: [AIRFLOW-6786] Add KafkaConsumerHook, KafkaProduerHook and KafkaSensor

2020-05-08 Thread GitBox
dferguson992 commented on a change in pull request #7407: URL: https://github.com/apache/airflow/pull/7407#discussion_r422236286 ## File path: airflow/providers/apache/kafka/hooks/kafka_consumer_hook.py ## @@ -0,0 +1,80 @@ +# +# Licensed to the Apache Software Foundation (ASF)

[GitHub] [airflow] webster-chainalysis commented on issue #8212: Can't read S3 remote logs using Airflow 1.10.9

2020-05-08 Thread GitBox
webster-chainalysis commented on issue #8212: URL: https://github.com/apache/airflow/issues/8212#issuecomment-625893851 @dimon222 - Yes. This is an automated message from the Apache Git Service. To respond to the message,

[airflow] branch master updated: Add WorldRemit as Airflow user (#8786)

2020-05-08 Thread turbaszek
This is an automated email from the ASF dual-hosted git repository. turbaszek pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/master by this push: new 09770e4 Add WorldRemit as Airflow user

[GitHub] [airflow] anitakar commented on pull request #8769: Fix running DAG from UI when dag serialization is on

2020-05-08 Thread GitBox
anitakar commented on pull request #8769: URL: https://github.com/apache/airflow/pull/8769#issuecomment-625848154 @mik-laj Do you want to take a look? This is an automated message from the Apache Git Service. To respond to

[GitHub] [airflow] Sinsin1367 commented on a change in pull request #8734: Added optional logging for pod container statuses once a pod fails. T…

2020-05-08 Thread GitBox
Sinsin1367 commented on a change in pull request #8734: URL: https://github.com/apache/airflow/pull/8734#discussion_r422181509 ## File path: airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py ## @@ -177,6 +181,7 @@ def __init__(self, # pylint:

[GitHub] [airflow] Sinsin1367 commented on a change in pull request #8734: Added optional logging for pod container statuses once a pod fails. T…

2020-05-08 Thread GitBox
Sinsin1367 commented on a change in pull request #8734: URL: https://github.com/apache/airflow/pull/8734#discussion_r422179683 ## File path: airflow/kubernetes/pod_launcher.py ## @@ -183,6 +183,23 @@ def read_pod_logs(self, pod: V1Pod, tail_lines: int = 10):

[GitHub] [airflow] aneesh-joseph commented on a change in pull request #8777: Add Production Helm chart support

2020-05-08 Thread GitBox
aneesh-joseph commented on a change in pull request #8777: URL: https://github.com/apache/airflow/pull/8777#discussion_r422158586 ## File path: chart/templates/configmap.yaml ## @@ -0,0 +1,108 @@ + +## Airflow ConfigMap

[GitHub] [airflow] aneesh-joseph commented on a change in pull request #8777: Add Production Helm chart support

2020-05-08 Thread GitBox
aneesh-joseph commented on a change in pull request #8777: URL: https://github.com/apache/airflow/pull/8777#discussion_r422158586 ## File path: chart/templates/configmap.yaml ## @@ -0,0 +1,108 @@ + +## Airflow ConfigMap

[GitHub] [airflow] aneesh-joseph commented on a change in pull request #8777: Add Production Helm chart support

2020-05-08 Thread GitBox
aneesh-joseph commented on a change in pull request #8777: URL: https://github.com/apache/airflow/pull/8777#discussion_r422158586 ## File path: chart/templates/configmap.yaml ## @@ -0,0 +1,108 @@ + +## Airflow ConfigMap

[GitHub] [airflow] aneesh-joseph commented on a change in pull request #8777: Add Production Helm chart support

2020-05-08 Thread GitBox
aneesh-joseph commented on a change in pull request #8777: URL: https://github.com/apache/airflow/pull/8777#discussion_r422158586 ## File path: chart/templates/configmap.yaml ## @@ -0,0 +1,108 @@ + +## Airflow ConfigMap

[GitHub] [airflow] aneesh-joseph commented on a change in pull request #8777: Add Production Helm chart support

2020-05-08 Thread GitBox
aneesh-joseph commented on a change in pull request #8777: URL: https://github.com/apache/airflow/pull/8777#discussion_r422158586 ## File path: chart/templates/configmap.yaml ## @@ -0,0 +1,108 @@ + +## Airflow ConfigMap

[GitHub] [airflow] kaxil commented on a change in pull request #8695: Make celery worker_prefetch_multiplier configurable

2020-05-08 Thread GitBox
kaxil commented on a change in pull request #8695: URL: https://github.com/apache/airflow/pull/8695#discussion_r422158345 ## File path: airflow/config_templates/default_airflow.cfg ## @@ -550,6 +550,15 @@ worker_concurrency = 8 # Example: worker_autoscale = 16,12 #

[GitHub] [airflow] aneesh-joseph commented on a change in pull request #8777: Add Production Helm chart support

2020-05-08 Thread GitBox
aneesh-joseph commented on a change in pull request #8777: URL: https://github.com/apache/airflow/pull/8777#discussion_r422158586 ## File path: chart/templates/configmap.yaml ## @@ -0,0 +1,108 @@ + +## Airflow ConfigMap

[GitHub] [airflow] nadflinn commented on a change in pull request #8695: Make celery worker_prefetch_multiplier configurable

2020-05-08 Thread GitBox
nadflinn commented on a change in pull request #8695: URL: https://github.com/apache/airflow/pull/8695#discussion_r422156337 ## File path: airflow/config_templates/default_airflow.cfg ## @@ -550,6 +550,15 @@ worker_concurrency = 8 # Example: worker_autoscale = 16,12 #

[GitHub] [airflow] nadflinn commented on a change in pull request #8695: Make celery worker_prefetch_multiplier configurable

2020-05-08 Thread GitBox
nadflinn commented on a change in pull request #8695: URL: https://github.com/apache/airflow/pull/8695#discussion_r422154400 ## File path: airflow/config_templates/default_airflow.cfg ## @@ -550,6 +550,15 @@ worker_concurrency = 8 # Example: worker_autoscale = 16,12 #

[GitHub] [airflow] nadflinn commented on a change in pull request #8695: Make celery worker_prefetch_multiplier configurable

2020-05-08 Thread GitBox
nadflinn commented on a change in pull request #8695: URL: https://github.com/apache/airflow/pull/8695#discussion_r422152685 ## File path: airflow/config_templates/config.yml ## @@ -1130,6 +1130,19 @@ type: string example: 16,12 default: ~ +- name:

[GitHub] [airflow] ashb commented on pull request #8772: Correctly store non-default Nones in serialized tasks/dags

2020-05-08 Thread GitBox
ashb commented on pull request #8772: URL: https://github.com/apache/airflow/pull/8772#issuecomment-625814879 Oh, cos it's parsing in a subprocess. H! I don't think we need to parse _all_ dags in a subprocess, just one is enough.

[GitHub] [airflow] ashb commented on pull request #8772: Correctly store non-default Nones in serialized tasks/dags

2020-05-08 Thread GitBox
ashb commented on pull request #8772: URL: https://github.com/apache/airflow/pull/8772#issuecomment-625814458 Curious, the DAG with the problem has `'start_date': datetime.utcnow()` -- that's "wrong" but I wouldn't have expected this test to fail. I guess we are loading the dag twice,

[jira] [Commented] (AIRFLOW-3369) Un-pausing a DAG with catchup =False creates an extra DAG run (1.10)

2020-05-08 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102581#comment-17102581 ] ASF GitHub Bot commented on AIRFLOW-3369: - ashb commented on pull request #8776: URL:

[GitHub] [airflow] ashb commented on pull request #8776: WIP: [AIRFLOW-3369] BugFix: Unpausing a DAG with catchup=False creates an extra DAG run

2020-05-08 Thread GitBox
ashb commented on pull request #8776: URL: https://github.com/apache/airflow/pull/8776#issuecomment-625811055 Lol, one line fixes are the Best* Tests please :) (I'm sure you'd get around to it) \* The worst.

[airflow] branch v1-10-stable updated: Add note extra links only render on when using RBAC webserver (#8788)

2020-05-08 Thread ash
This is an automated email from the ASF dual-hosted git repository. ash pushed a commit to branch v1-10-stable in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/v1-10-stable by this push: new b099571 Add note extra links only

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #8788: Add note that the RBAC UI is required for extra links

2020-05-08 Thread GitBox
boring-cyborg[bot] commented on pull request #8788: URL: https://github.com/apache/airflow/pull/8788#issuecomment-625808396 Awesome work, congrats on your first merged pull request! This is an automated message from the

[GitHub] [airflow] kaxil commented on a change in pull request #8772: Correctly store non-default Nones in serialized tasks/dags

2020-05-08 Thread GitBox
kaxil commented on a change in pull request #8772: URL: https://github.com/apache/airflow/pull/8772#discussion_r422132180 ## File path: airflow/serialization/serialized_objects.py ## @@ -395,6 +411,59 @@ def deserialize_operator(cls, encoded_op: Dict[str, Any]) ->

[GitHub] [airflow] kaxil commented on pull request #8772: Correctly store non-default Nones in serialized tasks/dags

2020-05-08 Thread GitBox
kaxil commented on pull request #8772: URL: https://github.com/apache/airflow/pull/8772#issuecomment-625803983 > Curious failure: > > ``` > E AssertionError: default_args[start_date] matches > E assert == > ``` > > I wonder why I

[GitHub] [airflow] ashb commented on pull request #8772: Correctly store non-default Nones in serialized tasks/dags

2020-05-08 Thread GitBox
ashb commented on pull request #8772: URL: https://github.com/apache/airflow/pull/8772#issuecomment-625803075 Curious failure: ``` E AssertionError: default_args[start_date] matches E assert == ``` I wonder why I didn't see that

[GitHub] [airflow] kaxil commented on a change in pull request #8772: Correctly store non-default Nones in serialized tasks/dags

2020-05-08 Thread GitBox
kaxil commented on a change in pull request #8772: URL: https://github.com/apache/airflow/pull/8772#discussion_r422127805 ## File path: tests/serialization/test_dag_serialization.py ## @@ -273,58 +271,70 @@ def test_deserialization(self): for dag_id in

[GitHub] [airflow] Blacksmoke16 commented on pull request #8737: Add note that the RBAC UI is required for extra links

2020-05-08 Thread GitBox
Blacksmoke16 commented on pull request #8737: URL: https://github.com/apache/airflow/pull/8737#issuecomment-625802179 Oops, I borked the merge @ashb. PR is recreated as https://github.com/apache/airflow/pull/8788. This is

[GitHub] [airflow] Blacksmoke16 opened a new pull request #8788: Add note that the RBAC UI is required for extra links

2020-05-08 Thread GitBox
Blacksmoke16 opened a new pull request #8788: URL: https://github.com/apache/airflow/pull/8788 After trying unsuccessfully to add extra links based on the [guide](https://airflow.apache.org/docs/stable/howto/define_extra_link.html), I came across some stackoverflow posts,

[jira] [Commented] (AIRFLOW-4549) wait_for_downstream does not respect skipped tasks

2020-05-08 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102549#comment-17102549 ] ASF GitHub Bot commented on AIRFLOW-4549: - TeddyHartanto commented on a change in pull request

[GitHub] [airflow] TeddyHartanto commented on a change in pull request #7735: [AIRFLOW-4549] Allow skipped tasks to satisfy wait_for_downstream

2020-05-08 Thread GitBox
TeddyHartanto commented on a change in pull request #7735: URL: https://github.com/apache/airflow/pull/7735#discussion_r422125930 ## File path: tests/dags/test_issue_1225.py ## @@ -47,12 +47,7 @@ def fail(): dag=dag1, pool='test_backfill_pooled_task_pool',) -# DAG

[GitHub] [airflow] Rcharriol commented on pull request #7777: [AIRFLOW-Date_range compatible with schedule_interval]

2020-05-08 Thread GitBox
Rcharriol commented on pull request #: URL: https://github.com/apache/airflow/pull/#issuecomment-625800899 @mik-laj Im having troubles to pass the tests, and got no clue why is failing, can i get some help please?

[jira] [Commented] (AIRFLOW-2310) Enable AWS Glue Job Integration

2020-05-08 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102546#comment-17102546 ] ASF GitHub Bot commented on AIRFLOW-2310: - kaxil commented on a change in pull request #6007:

[GitHub] [airflow] kaxil commented on a change in pull request #6007: [AIRFLOW-2310] Enable AWS Glue Job Integration

2020-05-08 Thread GitBox
kaxil commented on a change in pull request #6007: URL: https://github.com/apache/airflow/pull/6007#discussion_r422125128 ## File path: tests/providers/amazon/aws/hooks/test_aws_glue_job_hook.py ## @@ -0,0 +1,107 @@ +# +# Licensed to the Apache Software Foundation (ASF) under

[jira] [Commented] (AIRFLOW-2310) Enable AWS Glue Job Integration

2020-05-08 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102545#comment-17102545 ] ASF GitHub Bot commented on AIRFLOW-2310: - kaxil commented on a change in pull request #6007:

[GitHub] [airflow] kaxil commented on a change in pull request #6007: [AIRFLOW-2310] Enable AWS Glue Job Integration

2020-05-08 Thread GitBox
kaxil commented on a change in pull request #6007: URL: https://github.com/apache/airflow/pull/6007#discussion_r422124800 ## File path: tests/providers/amazon/aws/hooks/test_aws_glue_job_hook.py ## @@ -0,0 +1,107 @@ +# +# Licensed to the Apache Software Foundation (ASF) under

[GitHub] [airflow] kaxil commented on pull request #6007: [AIRFLOW-2310] Enable AWS Glue Job Integration

2020-05-08 Thread GitBox
kaxil commented on pull request #6007: URL: https://github.com/apache/airflow/pull/6007#issuecomment-625799603 Another failed test ``` * Module airflow.providers.amazon.aws.hooks.glue airflow/providers/amazon/aws/hooks/glue.py:70:8: C0103: Attribute name "S3_GLUE_LOGS"

[jira] [Commented] (AIRFLOW-2310) Enable AWS Glue Job Integration

2020-05-08 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102544#comment-17102544 ] ASF GitHub Bot commented on AIRFLOW-2310: - kaxil commented on pull request #6007: URL:

[GitHub] [airflow] kaxil commented on a change in pull request #8772: Correctly store non-default Nones in serialized tasks/dags

2020-05-08 Thread GitBox
kaxil commented on a change in pull request #8772: URL: https://github.com/apache/airflow/pull/8772#discussion_r422121186 ## File path: airflow/serialization/serialized_objects.py ## @@ -285,10 +297,14 @@ class SerializedBaseOperator(BaseOperator, BaseSerialization):

[GitHub] [airflow] kaxil commented on a change in pull request #8772: Correctly store non-default Nones in serialized tasks/dags

2020-05-08 Thread GitBox
kaxil commented on a change in pull request #8772: URL: https://github.com/apache/airflow/pull/8772#discussion_r422119825 ## File path: airflow/serialization/serialized_objects.py ## @@ -119,11 +120,16 @@ def _is_primitive(cls, var: Any) -> bool: @classmethod def

[jira] [Commented] (AIRFLOW-2310) Enable AWS Glue Job Integration

2020-05-08 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102538#comment-17102538 ] ASF GitHub Bot commented on AIRFLOW-2310: - kaxil commented on pull request #6007: URL:

[GitHub] [airflow] kaxil commented on pull request #6007: [AIRFLOW-2310] Enable AWS Glue Job Integration

2020-05-08 Thread GitBox
kaxil commented on pull request #6007: URL: https://github.com/apache/airflow/pull/6007#issuecomment-625794242 I have added 2 commits to fix some of the issues ! This is an automated message from the Apache Git Service. To

[GitHub] [airflow] mshalak-nix edited a comment on issue #8760: Webserver becomes unusable after 100,000 tasks were completed

2020-05-08 Thread GitBox
mshalak-nix edited a comment on issue #8760: URL: https://github.com/apache/airflow/issues/8760#issuecomment-625792761 Looks like I figured it out. The issue was with prometheus exporter I used, here is the fix: https://github.com/robinhood/airflow-prometheus-exporter/pull/25 It

[GitHub] [airflow] mshalak-nix commented on issue #8760: Webserver becomes unusable after 100,000 tasks were completed

2020-05-08 Thread GitBox
mshalak-nix commented on issue #8760: URL: https://github.com/apache/airflow/issues/8760#issuecomment-625792761 Looks like I figured it out. The issue was with prometheus exporter I used, here is the fix: https://github.com/robinhood/airflow-prometheus-exporter/pull/25 It generated

[GitHub] [airflow] kaxil commented on a change in pull request #6007: [AIRFLOW-2310] Enable AWS Glue Job Integration

2020-05-08 Thread GitBox
kaxil commented on a change in pull request #6007: URL: https://github.com/apache/airflow/pull/6007#discussion_r422115507 ## File path: docs/operators-and-hooks-ref.rst ## @@ -329,6 +329,12 @@ These integrations allow you to perform various operations within the Amazon Web

[jira] [Commented] (AIRFLOW-2310) Enable AWS Glue Job Integration

2020-05-08 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102534#comment-17102534 ] ASF GitHub Bot commented on AIRFLOW-2310: - kaxil commented on a change in pull request #6007:

[GitHub] [airflow] boittega opened a new pull request #8787: [8723] Spark JDBC Hook conf default parameter

2020-05-08 Thread GitBox
boittega opened a new pull request #8787: URL: https://github.com/apache/airflow/pull/8787 When spark_conf parameter is not set, the SparkJdbcHook raises an error because it is not iterable. Following the same behaviour than SparkSubmitHook. Resolves #8723 --- Make sure to

[GitHub] [airflow] kaxil commented on pull request #6007: [AIRFLOW-2310] Enable AWS Glue Job Integration

2020-05-08 Thread GitBox
kaxil commented on pull request #6007: URL: https://github.com/apache/airflow/pull/6007#issuecomment-625791176 Static check are failing: ``` diff --git a/docs/operators-and-hooks-ref.rst b/docs/operators-and-hooks-ref.rst index 0fcf11b..0b024ff 100644 ---

[jira] [Commented] (AIRFLOW-2310) Enable AWS Glue Job Integration

2020-05-08 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102530#comment-17102530 ] ASF GitHub Bot commented on AIRFLOW-2310: - kaxil commented on pull request #6007: URL:

[GitHub] [airflow] mik-laj commented on pull request #8721: Add OpenAPI specification (II)

2020-05-08 Thread GitBox
mik-laj commented on pull request #8721: URL: https://github.com/apache/airflow/pull/8721#issuecomment-625789285 Preview: https://editor.swagger.io/?url=https://raw.githubusercontent.com/PolideaInternal/airflow/openapi-spec/openapi.yaml

[GitHub] [airflow] potiuk commented on pull request #8768: [POC] Mark keywords-only arguments in hook method signatures

2020-05-08 Thread GitBox
potiuk commented on pull request #8768: URL: https://github.com/apache/airflow/pull/8768#issuecomment-625787883 However static checks are failing as our annotation is confusing pylint :( This is an automated message from

[jira] [Commented] (AIRFLOW-2310) Enable AWS Glue Job Integration

2020-05-08 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102518#comment-17102518 ] ASF GitHub Bot commented on AIRFLOW-2310: - kaxil commented on pull request #6007: URL:

[jira] [Commented] (AIRFLOW-2310) Enable AWS Glue Job Integration

2020-05-08 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102519#comment-17102519 ] ASF GitHub Bot commented on AIRFLOW-2310: - kaxil commented on a change in pull request #6007:

[GitHub] [airflow] kaxil commented on a change in pull request #6007: [AIRFLOW-2310] Enable AWS Glue Job Integration

2020-05-08 Thread GitBox
kaxil commented on a change in pull request #6007: URL: https://github.com/apache/airflow/pull/6007#discussion_r422105893 ## File path: airflow/providers/amazon/aws/operators/glue.py ## @@ -0,0 +1,114 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or

[GitHub] [airflow] kaxil commented on pull request #6007: [AIRFLOW-2310] Enable AWS Glue Job Integration

2020-05-08 Thread GitBox
kaxil commented on pull request #6007: URL: https://github.com/apache/airflow/pull/6007#issuecomment-625783624 Docs are failing: ``` The HTML pages are in _build/html. Error 1 WARNING: Field list ends without a blank line; unexpected

[GitHub] [airflow] boittega opened a new pull request #8786: Add WorldRemit as Airflow user

2020-05-08 Thread GitBox
boittega opened a new pull request #8786: URL: https://github.com/apache/airflow/pull/8786 Adding WorldRemit as Airflow user. --- Make sure to mark the boxes below before creating PR: [x] - [ ] Description above provides context of the change - [ ] Unit tests coverage

[jira] [Commented] (AIRFLOW-7068) Create EC2 Hook, Operator and Sensor

2020-05-08 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-7068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102485#comment-17102485 ] ASF GitHub Bot commented on AIRFLOW-7068: - mustafagok commented on pull request #7731: URL:

[GitHub] [airflow] mustafagok commented on pull request #7731: [AIRFLOW-7068] Create EC2 Hook, Operator and Sensor

2020-05-08 Thread GitBox
mustafagok commented on pull request #7731: URL: https://github.com/apache/airflow/pull/7731#issuecomment-625770673 > I restarted the failed travis check. Lets wait until it succeds and then I can merge it. @mustafagok can you add example dag + system test in another PR? That would be

[GitHub] [airflow] potiuk commented on issue #8774: Move singularity out of main CI tests and into separate docker image/system test

2020-05-08 Thread GitBox
potiuk commented on issue #8774: URL: https://github.com/apache/airflow/issues/8774#issuecomment-625770630 Not sure if it is still needed. Maybe @vsoch can tell more about it whether it is actually needed in the image. I think we had a least few "integration" kind of tests where we run

[GitHub] [airflow] RyanSiu1995 commented on issue #8780: Missing logs if the pod was killed by Kubernetes

2020-05-08 Thread GitBox
RyanSiu1995 commented on issue #8780: URL: https://github.com/apache/airflow/issues/8780#issuecomment-625769729 Yes, that's exactly what happen with my case. This is an automated message from the Apache Git Service. To

[GitHub] [airflow] msumit commented on issue #8780: Missing logs if the pod was killed by Kubernetes

2020-05-08 Thread GitBox
msumit commented on issue #8780: URL: https://github.com/apache/airflow/issues/8780#issuecomment-625768894 @RyanSiu1995 so it means that the worker pod is trying to upload the logs to GCS when being killed, but failing due to some issue in getting DB session?

[GitHub] [airflow] potiuk opened a new issue #8785: Move out all unnecessary binaries installed in CI image

2020-05-08 Thread GitBox
potiuk opened a new issue #8785: URL: https://github.com/apache/airflow/issues/8785 **Description** We have a lot of unnecessary binaries/libraries baked into the CI image. And we can move it out in the form of integration images downloaded separately by docker-compose.

[GitHub] [airflow] potiuk opened a new issue #8784: Move Minicluster out of the main CI image

2020-05-08 Thread GitBox
potiuk opened a new issue #8784: URL: https://github.com/apache/airflow/issues/8784 **Description** Minicluster is unnecessarily part of the CI image. It should be moved out and used separately **Use case / motivation** Download and install Mincluster image only when

[GitHub] [airflow] potiuk opened a new issue #8783: Move Hadoop distro out of CI image

2020-05-08 Thread GitBox
potiuk opened a new issue #8783: URL: https://github.com/apache/airflow/issues/8783 **Description** Hadoop distro is unnecessarily part of the CI image. It should be moved out and used separately **Use case / motivation** Download and install Hadoop image only when

[GitHub] [airflow] potiuk opened a new issue #8782: Move Kubernetes tests and kind setup out of the container to host

2020-05-08 Thread GitBox
potiuk opened a new issue #8782: URL: https://github.com/apache/airflow/issues/8782 **Description** We should move kind setup out of the container for CI to the host environment **Use case / motivation** Makes CI image smaller and setup easier for local testing.

[GitHub] [airflow] RyanSiu1995 commented on issue #8780: Missing logs if the pod was killed by Kubernetes

2020-05-08 Thread GitBox
RyanSiu1995 commented on issue #8780: URL: https://github.com/apache/airflow/issues/8780#issuecomment-625761275 But, to be honest, it is pretty odd that the function `create_session()` will not actually have any action of creation. It just tries to return the session object in

[GitHub] [airflow] turbaszek commented on pull request #8227: Add run_type to DagRun

2020-05-08 Thread GitBox
turbaszek commented on pull request #8227: URL: https://github.com/apache/airflow/pull/8227#issuecomment-625757619 > https://github.com/apache/airflow/blob/master/airflow/operators/dagrun_operator.py#L65-L71 @mik-laj do you think we should add new run type like `TRIGGERED`?

  1   2   >