[GitHub] [airflow] mik-laj commented on issue #8202: Create guide for Dataflow operators

2020-05-09 Thread GitBox
mik-laj commented on issue #8202: URL: https://github.com/apache/airflow/issues/8202#issuecomment-626277797 @tanjinP I would be happy if you added information that asynchronous execution is recommended. https://cloud.google.com/dataflow/docs/guides/specifying-exec-params#python_8 I

[GitHub] [airflow] mik-laj commented on issue #8202: Create guide for Dataflow operators

2020-05-09 Thread GitBox
mik-laj commented on issue #8202: URL: https://github.com/apache/airflow/issues/8202#issuecomment-626277095 @tanjinP Fantastic. I am working on this integration now with the Dataflow team, so this guide would be very helpful. I saw that you applied for 3 services, but then this service is

[GitHub] [airflow] mik-laj edited a comment on issue #8202: Create guide for Dataflow operators

2020-05-09 Thread GitBox
mik-laj edited a comment on issue #8202: URL: https://github.com/apache/airflow/issues/8202#issuecomment-626277095 @tanjinP Fantastic. I am working on this integration now with the Dataflow team, so this guide would be very helpful. I saw that you applied for 3 services, but then this

[GitHub] [airflow] gdevanla commented on issue #8696: Skip task itself instead of all downstream tasks

2020-05-09 Thread GitBox
gdevanla commented on issue #8696: URL: https://github.com/apache/airflow/issues/8696#issuecomment-626276892 Nope, you can go ahead and work on it. I hit the same issue with in our internal project and was looking through the code to understand what was happening. Then I came across this

[GitHub] [airflow] gdevanla edited a comment on issue #8696: Skip task itself instead of all downstream tasks

2020-05-09 Thread GitBox
gdevanla edited a comment on issue #8696: URL: https://github.com/apache/airflow/issues/8696#issuecomment-626276892 Nope, you can go ahead and work on it. I hit the same issue with our internal project and was looking through the code to understand what was happening. Then I came across

[GitHub] [airflow] j-y-matsubara edited a comment on issue #8696: Skip task itself instead of all downstream tasks

2020-05-09 Thread GitBox
j-y-matsubara edited a comment on issue #8696: URL: https://github.com/apache/airflow/issues/8696#issuecomment-626276776 @gdevanla Sorry for cutting in. I agree with your idea. I was having trouble with this, so I am trying to fix it. Are you planning to fix it?

[GitHub] [airflow] j-y-matsubara commented on issue #8696: Skip task itself instead of all downstream tasks

2020-05-09 Thread GitBox
j-y-matsubara commented on issue #8696: URL: https://github.com/apache/airflow/issues/8696#issuecomment-626276776 @gdevanla Sorry for cutting in, and I agree with your idea. I was having trouble with this, so I am trying to fix it. Are you planning to fix it?

[GitHub] [airflow] tanjinP commented on issue #8205: Create guide for Cloud Key Management Service (KMS) operators

2020-05-09 Thread GitBox
tanjinP commented on issue #8205: URL: https://github.com/apache/airflow/issues/8205#issuecomment-626276202 I can take this on - this is a service I've worked heavily in. This is an automated message from the Apache Git

[GitHub] [airflow] tanjinP commented on issue #8202: Create guide for Dataflow operators

2020-05-09 Thread GitBox
tanjinP commented on issue #8202: URL: https://github.com/apache/airflow/issues/8202#issuecomment-626276187 I can take this on - this is a service I've worked heavily in. This is an automated message from the Apache Git

[GitHub] [airflow] j-y-matsubara commented on a change in pull request #8757: Fix the incorrect description of pod_mutation_hook in kubernetes.rst

2020-05-09 Thread GitBox
j-y-matsubara commented on a change in pull request #8757: URL: https://github.com/apache/airflow/pull/8757#discussion_r422455491 ## File path: docs/kubernetes.rst ## @@ -40,11 +40,44 @@ that has the ability to mutate pod objects before sending them to the Kubernetes for

[GitHub] [airflow] j-y-matsubara commented on a change in pull request #8757: Fix the incorrect description of pod_mutation_hook in kubernetes.rst

2020-05-09 Thread GitBox
j-y-matsubara commented on a change in pull request #8757: URL: https://github.com/apache/airflow/pull/8757#discussion_r422455491 ## File path: docs/kubernetes.rst ## @@ -40,11 +40,44 @@ that has the ability to mutate pod objects before sending them to the Kubernetes for

[GitHub] [airflow] j-y-matsubara commented on a change in pull request #8757: Fix the incorrect description of pod_mutation_hook in kubernetes.rst

2020-05-09 Thread GitBox
j-y-matsubara commented on a change in pull request #8757: URL: https://github.com/apache/airflow/pull/8757#discussion_r422455491 ## File path: docs/kubernetes.rst ## @@ -40,11 +40,44 @@ that has the ability to mutate pod objects before sending them to the Kubernetes for

[GitHub] [airflow] maganaluis commented on pull request #8256: updated _write_args on PythonVirtualenvOperator

2020-05-09 Thread GitBox
maganaluis commented on pull request #8256: URL: https://github.com/apache/airflow/pull/8256#issuecomment-626267914 > Small issues, apart from that; LGTM Thank you @Fokko, I switched those to follow a create_session approach with the with statement.

[GitHub] [airflow] kaxil edited a comment on pull request #8776: [AIRFLOW-1156] BugFix: Unpausing a DAG with catchup=False creates an extra DAG run

2020-05-09 Thread GitBox
kaxil edited a comment on pull request #8776: URL: https://github.com/apache/airflow/pull/8776#issuecomment-626267001 > Lol, one line fixes are the Best* > > Tests please :) (I'm sure you'd get around to it) > > * The worst.  Yeah - Looks like this bug was for a long

[jira] [Commented] (AIRFLOW-1156) Using a timedelta object as a Schedule Interval with catchup=False causes the start_date to no longer be honored.

2020-05-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-1156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103625#comment-17103625 ] ASF GitHub Bot commented on AIRFLOW-1156: - kaxil edited a comment on pull request #8776: URL:

[jira] [Work started] (AIRFLOW-1156) Using a timedelta object as a Schedule Interval with catchup=False causes the start_date to no longer be honored.

2020-05-09 Thread Kaxil Naik (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-1156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-1156 started by Kaxil Naik. --- > Using a timedelta object as a Schedule Interval with catchup=False causes the

[jira] [Closed] (AIRFLOW-3369) Un-pausing a DAG with catchup =False creates an extra DAG run (1.10)

2020-05-09 Thread Kaxil Naik (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik closed AIRFLOW-3369. --- Resolution: Duplicate PR to fix this: https://github.com/apache/airflow/pull/8776 Closing the

[jira] [Commented] (AIRFLOW-6577) DAG Backfill with timedelta runs twice

2020-05-09 Thread Kaxil Naik (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103622#comment-17103622 ] Kaxil Naik commented on AIRFLOW-6577: - PR to fix this: https://github.com/apache/airflow/pull/8776

[jira] [Commented] (AIRFLOW-1056) Single dag run triggered when un-pausing job with catchup=False

2020-05-09 Thread Kaxil Naik (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103621#comment-17103621 ] Kaxil Naik commented on AIRFLOW-1056: - PR to fix this: https://github.com/apache/airflow/pull/8776

[jira] [Closed] (AIRFLOW-6577) DAG Backfill with timedelta runs twice

2020-05-09 Thread Kaxil Naik (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik closed AIRFLOW-6577. --- Resolution: Duplicate > DAG Backfill with timedelta runs twice >

[jira] [Closed] (AIRFLOW-1056) Single dag run triggered when un-pausing job with catchup=False

2020-05-09 Thread Kaxil Naik (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik closed AIRFLOW-1056. --- Resolution: Duplicate > Single dag run triggered when un-pausing job with catchup=False >

[jira] [Commented] (AIRFLOW-1156) Using a timedelta object as a Schedule Interval with catchup=False causes the start_date to no longer be honored.

2020-05-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-1156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103619#comment-17103619 ] ASF GitHub Bot commented on AIRFLOW-1156: - kaxil commented on pull request #8776: URL:

[GitHub] [airflow] kaxil commented on pull request #8776: [AIRFLOW-1156] BugFix: Unpausing a DAG with catchup=False creates an extra DAG run

2020-05-09 Thread GitBox
kaxil commented on pull request #8776: URL: https://github.com/apache/airflow/pull/8776#issuecomment-626267001 > Lol, one line fixes are the Best* > > Tests please :) (I'm sure you'd get around to it) > > * The worst. :D Yeah - Looks like this bug was for a long time

[GitHub] [airflow] amithmathew opened a new issue #8803: Impersonate service accounts while running GCP Operators

2020-05-09 Thread GitBox
amithmathew opened a new issue #8803: URL: https://github.com/apache/airflow/issues/8803 **Description** Allow running Google Cloud operators using Service Accounts, without having to provide key material while running on GCP. If the Compute instance Service Accounts on which Airflow

[GitHub] [airflow] boring-cyborg[bot] commented on issue #8803: Impersonate service accounts while running GCP Operators

2020-05-09 Thread GitBox
boring-cyborg[bot] commented on issue #8803: URL: https://github.com/apache/airflow/issues/8803#issuecomment-626266912 Thanks for opening your first issue here! Be sure to follow the issue template! This is an automated

[airflow] tag nightly-master updated (42c5975 -> c7788a6)

2020-05-09 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to tag nightly-master in repository https://gitbox.apache.org/repos/asf/airflow.git. *** WARNING: tag nightly-master was modified! *** from 42c5975 (commit) to c7788a6 (commit) from

[airflow] branch master updated: Add imap_attachment_to_s3 example dag and system test (#8669)

2020-05-09 Thread kaxilnaik
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/master by this push: new c7788a6 Add imap_attachment_to_s3 example

[jira] [Commented] (AIRFLOW-2310) Enable AWS Glue Job Integration

2020-05-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103575#comment-17103575 ] ASF GitHub Bot commented on AIRFLOW-2310: - abdulbasitds edited a comment on pull request #6007:

[jira] [Commented] (AIRFLOW-2310) Enable AWS Glue Job Integration

2020-05-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103576#comment-17103576 ] ASF GitHub Bot commented on AIRFLOW-2310: - abdulbasitds commented on pull request #6007: URL:

[jira] [Commented] (AIRFLOW-5906) Add authenticator parameter to snowflake_hook

2020-05-09 Thread ASF subversion and git services (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103574#comment-17103574 ] ASF subversion and git services commented on AIRFLOW-5906: -- Commit

[jira] [Commented] (AIRFLOW-5906) Add authenticator parameter to snowflake_hook

2020-05-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103573#comment-17103573 ] ASF GitHub Bot commented on AIRFLOW-5906: - boring-cyborg[bot] commented on pull request #8642:

[jira] [Commented] (AIRFLOW-2310) Enable AWS Glue Job Integration

2020-05-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103572#comment-17103572 ] ASF GitHub Bot commented on AIRFLOW-2310: - abdulbasitds edited a comment on pull request #6007:

[airflow] branch master updated (21cc7d7 -> cd635dd)

2020-05-09 Thread kaxilnaik
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from 21cc7d7 Document default timeout value for SSHOperator (#8744) add cd635dd [AIRFLOW-5906] Add

[GitHub] [airflow] abdulbasitds commented on pull request #6007: [AIRFLOW-2310] Enable AWS Glue Job Integration

2020-05-09 Thread GitBox
abdulbasitds commented on pull request #6007: URL: https://github.com/apache/airflow/pull/6007#issuecomment-626256604 > > > Appearenlty, some unittest are failing. I did a pytest in morning and reworked, every files was succeeding all test. havent changed anything but now in base aws

[GitHub] [airflow] abdulbasitds edited a comment on pull request #6007: [AIRFLOW-2310] Enable AWS Glue Job Integration

2020-05-09 Thread GitBox
abdulbasitds edited a comment on pull request #6007: URL: https://github.com/apache/airflow/pull/6007#issuecomment-626255328 > > Appearenlty, some unittest are failing. I did a pytest in morning and reworked, every files was succeeding all test. havent changed anything but now in base aws

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #8642: [AIRFLOW-5906] Add authenticator parameter to snowflake_hook

2020-05-09 Thread GitBox
boring-cyborg[bot] commented on pull request #8642: URL: https://github.com/apache/airflow/pull/8642#issuecomment-626256549 Awesome work, congrats on your first merged pull request! This is an automated message from the

[jira] [Commented] (AIRFLOW-5906) Add authenticator parameter to snowflake_hook

2020-05-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103569#comment-17103569 ] ASF GitHub Bot commented on AIRFLOW-5906: - kaxil commented on a change in pull request #8642:

[GitHub] [airflow] abdulbasitds edited a comment on pull request #6007: [AIRFLOW-2310] Enable AWS Glue Job Integration

2020-05-09 Thread GitBox
abdulbasitds edited a comment on pull request #6007: URL: https://github.com/apache/airflow/pull/6007#issuecomment-626255328 > > Appearenlty, some unittest are failing. I did a pytest in morning and reworked, every files was succeeding all test. havent changed anything but now in base aws

[GitHub] [airflow] kaxil commented on a change in pull request #8642: [AIRFLOW-5906] Add authenticator parameter to snowflake_hook

2020-05-09 Thread GitBox
kaxil commented on a change in pull request #8642: URL: https://github.com/apache/airflow/pull/8642#discussion_r422566889 ## File path: tests/providers/snowflake/hooks/test_snowflake.py ## @@ -134,3 +136,15 @@ def test_key_pair_auth_not_encrypted(self):

[GitHub] [airflow] kaxil commented on a change in pull request #8642: [AIRFLOW-5906] Add authenticator parameter to snowflake_hook

2020-05-09 Thread GitBox
kaxil commented on a change in pull request #8642: URL: https://github.com/apache/airflow/pull/8642#discussion_r422566593 ## File path: tests/providers/snowflake/hooks/test_snowflake.py ## @@ -134,3 +136,15 @@ def test_key_pair_auth_not_encrypted(self):

[jira] [Commented] (AIRFLOW-5906) Add authenticator parameter to snowflake_hook

2020-05-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103568#comment-17103568 ] ASF GitHub Bot commented on AIRFLOW-5906: - kaxil commented on a change in pull request #8642:

[jira] [Commented] (AIRFLOW-2310) Enable AWS Glue Job Integration

2020-05-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103564#comment-17103564 ] ASF GitHub Bot commented on AIRFLOW-2310: - abdulbasitds edited a comment on pull request #6007:

[GitHub] [airflow] abdulbasitds edited a comment on pull request #6007: [AIRFLOW-2310] Enable AWS Glue Job Integration

2020-05-09 Thread GitBox
abdulbasitds edited a comment on pull request #6007: URL: https://github.com/apache/airflow/pull/6007#issuecomment-626255328 > > Appearenlty, some unittest are failing. I did a pytest in morning and reworked, every files was succeeding all test. havent changed anything but now in base aws

[GitHub] [airflow] abdulbasitds edited a comment on pull request #6007: [AIRFLOW-2310] Enable AWS Glue Job Integration

2020-05-09 Thread GitBox
abdulbasitds edited a comment on pull request #6007: URL: https://github.com/apache/airflow/pull/6007#issuecomment-626255328 > > Appearenlty, some unittest are failing. I did a pytest in morning and reworked, every files was succeeding all test. havent changed anything but now in base aws

[jira] [Commented] (AIRFLOW-2310) Enable AWS Glue Job Integration

2020-05-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103562#comment-17103562 ] ASF GitHub Bot commented on AIRFLOW-2310: - abdulbasitds edited a comment on pull request #6007:

[jira] [Commented] (AIRFLOW-2310) Enable AWS Glue Job Integration

2020-05-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103561#comment-17103561 ] ASF GitHub Bot commented on AIRFLOW-2310: - abdulbasitds commented on pull request #6007: URL:

[GitHub] [airflow] abdulbasitds commented on pull request #6007: [AIRFLOW-2310] Enable AWS Glue Job Integration

2020-05-09 Thread GitBox
abdulbasitds commented on pull request #6007: URL: https://github.com/apache/airflow/pull/6007#issuecomment-626255328 > > Appearenlty, some unittest are failing. I did a pytest in morning and reworked, every files was succeeding all test. havent changed anything but now in base aws class

[GitHub] [airflow] kaxil commented on pull request #8749: add AWS StepFunctions integrations to the aws provider

2020-05-09 Thread GitBox
kaxil commented on pull request #8749: URL: https://github.com/apache/airflow/pull/8749#issuecomment-626255210 cc @baolsen This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #8744: Document default timeout value for SSHOperator

2020-05-09 Thread GitBox
boring-cyborg[bot] commented on pull request #8744: URL: https://github.com/apache/airflow/pull/8744#issuecomment-626254623 Awesome work, congrats on your first merged pull request! This is an automated message from the

[airflow] branch master updated: Document default timeout value for SSHOperator (#8744)

2020-05-09 Thread kaxilnaik
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/master by this push: new 21cc7d7 Document default timeout value for

[airflow] branch master updated: Fix docs on creating CustomOperator (#8678)

2020-05-09 Thread kaxilnaik
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/master by this push: new 5e1c33a Fix docs on creating

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #8678: Updated docs so demo code runs

2020-05-09 Thread GitBox
boring-cyborg[bot] commented on pull request #8678: URL: https://github.com/apache/airflow/pull/8678#issuecomment-626254500 Awesome work, congrats on your first merged pull request! This is an automated message from the

[GitHub] [airflow] kaxil commented on pull request #6007: [AIRFLOW-2310] Enable AWS Glue Job Integration

2020-05-09 Thread GitBox
kaxil commented on pull request #6007: URL: https://github.com/apache/airflow/pull/6007#issuecomment-626254185 > Appearenlty, some unittest are failing. I did a pytest in morning and reworked, every files was succeeding all test. havent changed anything but now in base aws class there is

[jira] [Commented] (AIRFLOW-2310) Enable AWS Glue Job Integration

2020-05-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103556#comment-17103556 ] ASF GitHub Bot commented on AIRFLOW-2310: - kaxil commented on pull request #6007: URL:

[GitHub] [airflow] kaxil opened a new pull request #8802: Allow passing secrets backend_kwargs to AWS SSM client

2020-05-09 Thread GitBox
kaxil opened a new pull request #8802: URL: https://github.com/apache/airflow/pull/8802 Allow passing secrets backend_kwargs to AWS SSM client We were already doing this for all other Secret backends. --- Make sure to mark the boxes below before creating PR: [x] -

[jira] [Commented] (AIRFLOW-2310) Enable AWS Glue Job Integration

2020-05-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103552#comment-17103552 ] ASF GitHub Bot commented on AIRFLOW-2310: - abdulbasitds commented on pull request #6007: URL:

[GitHub] [airflow] abdulbasitds commented on pull request #6007: [AIRFLOW-2310] Enable AWS Glue Job Integration

2020-05-09 Thread GitBox
abdulbasitds commented on pull request #6007: URL: https://github.com/apache/airflow/pull/6007#issuecomment-626253136 Appearenlty, some unittest are failing. I did a pytest in morning and reworked, every files was succeeding all test. havent changed anything but now in base aws class

[GitHub] [airflow] gdevanla commented on issue #8696: Skip task itself instead of all downstream tasks

2020-05-09 Thread GitBox
gdevanla commented on issue #8696: URL: https://github.com/apache/airflow/issues/8696#issuecomment-626252453 @yuqian90 I believe you are pointing to this line of code which should not be performing this task here.

[jira] [Commented] (AIRFLOW-2310) Enable AWS Glue Job Integration

2020-05-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103533#comment-17103533 ] ASF GitHub Bot commented on AIRFLOW-2310: - abdulbasitds commented on pull request #6007: URL:

[GitHub] [airflow] abdulbasitds commented on pull request #6007: [AIRFLOW-2310] Enable AWS Glue Job Integration

2020-05-09 Thread GitBox
abdulbasitds commented on pull request #6007: URL: https://github.com/apache/airflow/pull/6007#issuecomment-626248933 > > > doesn't conform to snake_case naming style > > > > > > @kaxil I have solved other issues (havent pushed yet) , but for the last one(reducing number of

[jira] [Commented] (AIRFLOW-2310) Enable AWS Glue Job Integration

2020-05-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103530#comment-17103530 ] ASF GitHub Bot commented on AIRFLOW-2310: - kaxil edited a comment on pull request #6007: URL:

[GitHub] [airflow] kaxil commented on pull request #6007: [AIRFLOW-2310] Enable AWS Glue Job Integration

2020-05-09 Thread GitBox
kaxil commented on pull request #6007: URL: https://github.com/apache/airflow/pull/6007#issuecomment-626248652 > > doesn't conform to snake_case naming style > > @kaxil I have solved other issues (havent pushed yet) , but for the last one(reducing number of arguments) I will have to

[jira] [Commented] (AIRFLOW-2310) Enable AWS Glue Job Integration

2020-05-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103531#comment-17103531 ] ASF GitHub Bot commented on AIRFLOW-2310: - kaxil edited a comment on pull request #6007: URL:

[GitHub] [airflow] kaxil edited a comment on pull request #6007: [AIRFLOW-2310] Enable AWS Glue Job Integration

2020-05-09 Thread GitBox
kaxil edited a comment on pull request #6007: URL: https://github.com/apache/airflow/pull/6007#issuecomment-626248652 > > doesn't conform to snake_case naming style > > @kaxil I have solved other issues (havent pushed yet) , but for the last one(reducing number of arguments) I will

[GitHub] [airflow] kaxil edited a comment on pull request #6007: [AIRFLOW-2310] Enable AWS Glue Job Integration

2020-05-09 Thread GitBox
kaxil edited a comment on pull request #6007: URL: https://github.com/apache/airflow/pull/6007#issuecomment-626248652 > > doesn't conform to snake_case naming style > > @kaxil I have solved other issues (havent pushed yet) , but for the last one(reducing number of arguments) I will

[jira] [Commented] (AIRFLOW-2310) Enable AWS Glue Job Integration

2020-05-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103529#comment-17103529 ] ASF GitHub Bot commented on AIRFLOW-2310: - kaxil commented on pull request #6007: URL:

[jira] [Commented] (AIRFLOW-2310) Enable AWS Glue Job Integration

2020-05-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103528#comment-17103528 ] ASF GitHub Bot commented on AIRFLOW-2310: - abdulbasitds removed a comment on pull request

[GitHub] [airflow] abdulbasitds removed a comment on pull request #6007: [AIRFLOW-2310] Enable AWS Glue Job Integration

2020-05-09 Thread GitBox
abdulbasitds removed a comment on pull request #6007: URL: https://github.com/apache/airflow/pull/6007#issuecomment-626245782 > doesn't conform to snake_case naming style @kaxil I have solved other issues (havent pushed yet) , but for the last one(reducing number of arguments) I

[airflow] branch master updated (bc19778 -> 7506c73)

2020-05-09 Thread ash
This is an automated email from the ASF dual-hosted git repository. ash pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from bc19778 [AIP-31] Implement XComArg to pass output from one operator to the next (#8652) add 7506c73 Add

[airflow] branch master updated (db1b51d -> bc19778)

2020-05-09 Thread kaxilnaik
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from db1b51d Make celery worker_prefetch_multiplier configurable (#8695) add bc19778 [AIP-31] Implement

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #8652: [AIP-31] Implement XComArg model to functionally pass output from one operator to the next

2020-05-09 Thread GitBox
boring-cyborg[bot] commented on pull request #8652: URL: https://github.com/apache/airflow/pull/8652#issuecomment-626245939 Awesome work, congrats on your first merged pull request! This is an automated message from the

[GitHub] [airflow] kaxil commented on a change in pull request #8652: [AIP-31] Implement XComArg model to functionally pass output from one operator to the next

2020-05-09 Thread GitBox
kaxil commented on a change in pull request #8652: URL: https://github.com/apache/airflow/pull/8652#discussion_r422556125 ## File path: airflow/models/baseoperator.py ## @@ -1121,6 +1139,12 @@ def set_upstream(self, task_or_task_list: Union['BaseOperator', List['BaseOperat

[jira] [Commented] (AIRFLOW-2310) Enable AWS Glue Job Integration

2020-05-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103515#comment-17103515 ] ASF GitHub Bot commented on AIRFLOW-2310: - abdulbasitds commented on pull request #6007: URL:

[GitHub] [airflow] abdulbasitds commented on pull request #6007: [AIRFLOW-2310] Enable AWS Glue Job Integration

2020-05-09 Thread GitBox
abdulbasitds commented on pull request #6007: URL: https://github.com/apache/airflow/pull/6007#issuecomment-626245782 > doesn't conform to snake_case naming style @kaxil I have solved other issues (havent pushed yet) , but for the last one(reducing number of arguments) I will have

[GitHub] [airflow] kaxil commented on a change in pull request #8652: [AIP-31] Implement XComArg model to functionally pass output from one operator to the next

2020-05-09 Thread GitBox
kaxil commented on a change in pull request #8652: URL: https://github.com/apache/airflow/pull/8652#discussion_r422555955 ## File path: airflow/models/baseoperator.py ## @@ -1121,6 +1139,12 @@ def set_upstream(self, task_or_task_list: Union['BaseOperator', List['BaseOperat

[GitHub] [airflow] kaxil commented on a change in pull request #8652: [AIP-31] Implement XComArg model to functionally pass output from one operator to the next

2020-05-09 Thread GitBox
kaxil commented on a change in pull request #8652: URL: https://github.com/apache/airflow/pull/8652#discussion_r422555955 ## File path: airflow/models/baseoperator.py ## @@ -1121,6 +1139,12 @@ def set_upstream(self, task_or_task_list: Union['BaseOperator', List['BaseOperat

[GitHub] [airflow] kaxil commented on a change in pull request #8652: [AIP-31] Implement XComArg model to functionally pass output from one operator to the next

2020-05-09 Thread GitBox
kaxil commented on a change in pull request #8652: URL: https://github.com/apache/airflow/pull/8652#discussion_r422555817 ## File path: airflow/models/baseoperator.py ## @@ -1121,6 +1139,12 @@ def set_upstream(self, task_or_task_list: Union['BaseOperator', List['BaseOperat

[jira] [Commented] (AIRFLOW-6786) Adding KafkaConsumerHook, KafkaProducerHook, and KafkaSensor

2020-05-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103513#comment-17103513 ] ASF GitHub Bot commented on AIRFLOW-6786: - kaxil commented on a change in pull request #7407:

[jira] [Commented] (AIRFLOW-6786) Adding KafkaConsumerHook, KafkaProducerHook, and KafkaSensor

2020-05-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103512#comment-17103512 ] ASF GitHub Bot commented on AIRFLOW-6786: - kaxil commented on a change in pull request #7407:

[GitHub] [airflow] kaxil commented on a change in pull request #7407: [AIRFLOW-6786] Add KafkaConsumerHook, KafkaProduerHook and KafkaSensor

2020-05-09 Thread GitBox
kaxil commented on a change in pull request #7407: URL: https://github.com/apache/airflow/pull/7407#discussion_r422555361 ## File path: airflow/providers/apache/kafka/sensors/kafka_sensor.py ## @@ -0,0 +1,81 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one

[jira] [Commented] (AIRFLOW-6786) Adding KafkaConsumerHook, KafkaProducerHook, and KafkaSensor

2020-05-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103511#comment-17103511 ] ASF GitHub Bot commented on AIRFLOW-6786: - kaxil commented on a change in pull request #7407:

[GitHub] [airflow] kaxil commented on a change in pull request #7407: [AIRFLOW-6786] Add KafkaConsumerHook, KafkaProduerHook and KafkaSensor

2020-05-09 Thread GitBox
kaxil commented on a change in pull request #7407: URL: https://github.com/apache/airflow/pull/7407#discussion_r422555319 ## File path: airflow/providers/apache/kafka/sensors/kafka_sensor.py ## @@ -0,0 +1,81 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one

[GitHub] [airflow] kaxil commented on a change in pull request #7407: [AIRFLOW-6786] Add KafkaConsumerHook, KafkaProduerHook and KafkaSensor

2020-05-09 Thread GitBox
kaxil commented on a change in pull request #7407: URL: https://github.com/apache/airflow/pull/7407#discussion_r422555228 ## File path: airflow/providers/apache/kafka/sensors/kafka_sensor.py ## @@ -0,0 +1,84 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one

[GitHub] [airflow] ashb commented on pull request #8772: Correctly store non-default Nones in serialized tasks/dags

2020-05-09 Thread GitBox
ashb commented on pull request #8772: URL: https://github.com/apache/airflow/pull/8772#issuecomment-626244825 (We've got that code in this diff comment, we can bring it back when we want it, but that will need more changes elsewhere to serialize more fields

[GitHub] [airflow] ashb commented on pull request #8772: Correctly store non-default Nones in serialized tasks/dags

2020-05-09 Thread GitBox
ashb commented on pull request #8772: URL: https://github.com/apache/airflow/pull/8772#issuecomment-626244699 I was wrong about where that fn was used, so I've removed that complex code an replace it with this in the test instead: ```python if serialized_task.resources is

[jira] [Commented] (AIRFLOW-2310) Enable AWS Glue Job Integration

2020-05-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103510#comment-17103510 ] ASF GitHub Bot commented on AIRFLOW-2310: - kaxil edited a comment on pull request #6007: URL:

[jira] [Commented] (AIRFLOW-2310) Enable AWS Glue Job Integration

2020-05-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103509#comment-17103509 ] ASF GitHub Bot commented on AIRFLOW-2310: - kaxil commented on pull request #6007: URL:

[GitHub] [airflow] kaxil edited a comment on pull request #6007: [AIRFLOW-2310] Enable AWS Glue Job Integration

2020-05-09 Thread GitBox
kaxil edited a comment on pull request #6007: URL: https://github.com/apache/airflow/pull/6007#issuecomment-626244214 Please fix the following tests too: ``` * Module airflow.providers.amazon.aws.hooks.glue airflow/providers/amazon/aws/hooks/glue.py:70:8: C0103:

[GitHub] [airflow] kaxil commented on pull request #6007: [AIRFLOW-2310] Enable AWS Glue Job Integration

2020-05-09 Thread GitBox
kaxil commented on pull request #6007: URL: https://github.com/apache/airflow/pull/6007#issuecomment-626244214 Please fix the following tests too: ``` Permissions already fixed * Module airflow.providers.amazon.aws.hooks.glue

[GitHub] [airflow] gdevanla edited a comment on pull request #8699: Scheduler just checks for task instances in 'running' state in execution.

2020-05-09 Thread GitBox
gdevanla edited a comment on pull request #8699: URL: https://github.com/apache/airflow/pull/8699#issuecomment-626243867 @ashb My first thought was also to handle this at the executor level. But, I believe, we need to allow the scheduler to reset the state of the task to `scheduled`, if

[GitHub] [airflow] gdevanla edited a comment on pull request #8699: Scheduler just checks for task instances in 'running' state in execution.

2020-05-09 Thread GitBox
gdevanla edited a comment on pull request #8699: URL: https://github.com/apache/airflow/pull/8699#issuecomment-626243867 @ashb My first thought was also to handle this at the executor level. But, I believe, we need to allow the scheduler to reset the state of the task to `scheduled`, if

[GitHub] [airflow] gdevanla commented on pull request #8699: Scheduler just checks for task instances in 'running' state in execution.

2020-05-09 Thread GitBox
gdevanla commented on pull request #8699: URL: https://github.com/apache/airflow/pull/8699#issuecomment-626243867 @ashb My first thought was also to handle this at the executor level. But, I believe, we need to allow the scheduler to reset the state of the task to `scheduled`, if it was

[GitHub] [airflow] gdevanla commented on a change in pull request #8699: Scheduler just checks for task instances in 'running' state in execution.

2020-05-09 Thread GitBox
gdevanla commented on a change in pull request #8699: URL: https://github.com/apache/airflow/pull/8699#discussion_r422554030 ## File path: airflow/jobs/scheduler_job.py ## @@ -1275,7 +1275,7 @@ def _find_executable_task_instances(self, simple_dag_bag, session=None):

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #8801: Add optional network profile configuration parameter

2020-05-09 Thread GitBox
boring-cyborg[bot] commented on pull request #8801: URL: https://github.com/apache/airflow/pull/8801#issuecomment-626241147 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our

[GitHub] [airflow] lutzkuen opened a new pull request #8801: Add optional network profile configuration parameter

2020-05-09 Thread GitBox
lutzkuen opened a new pull request #8801: URL: https://github.com/apache/airflow/pull/8801 This PR adds optional parameters for the AzureContainerInstanceGroupOperator to forward IP Address cconfiguration, Network Profile and Restart policy to the Azure Container group. This is useful if

[jira] [Commented] (AIRFLOW-2310) Enable AWS Glue Job Integration

2020-05-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103500#comment-17103500 ] ASF GitHub Bot commented on AIRFLOW-2310: - kaxil commented on a change in pull request #6007:

[GitHub] [airflow] kaxil commented on a change in pull request #6007: [AIRFLOW-2310] Enable AWS Glue Job Integration

2020-05-09 Thread GitBox
kaxil commented on a change in pull request #6007: URL: https://github.com/apache/airflow/pull/6007#discussion_r422551368 ## File path: docs/operators-and-hooks-ref.rst ## @@ -387,6 +387,14 @@ These integrations allow you to perform various operations within the Amazon Web

[GitHub] [airflow] nadflinn commented on issue #8480: Celery autoscaling overrides normal worker_concurrency setting

2020-05-09 Thread GitBox
nadflinn commented on issue #8480: URL: https://github.com/apache/airflow/issues/8480#issuecomment-626240649 @turbaszek I think this can be closed, correct? This is an automated message from the Apache Git Service. To

[GitHub] [airflow] kaxil commented on pull request #8772: Correctly store non-default Nones in serialized tasks/dags

2020-05-09 Thread GitBox
kaxil commented on pull request #8772: URL: https://github.com/apache/airflow/pull/8772#issuecomment-626240186 Feel free to merge once we add a comment in the doc on why we kept the change. This is an automated message from

[jira] [Commented] (AIRFLOW-6829) Auto apply apply_default decorator to all sub-classess

2020-05-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103491#comment-17103491 ] ASF GitHub Bot commented on AIRFLOW-6829: - stale[bot] commented on pull request #7450: URL:

  1   2   3   >