[GitHub] [airflow] timhealz commented on pull request #8849: Add multiple file upload functionality to GCS hook

2020-05-13 Thread GitBox
timhealz commented on pull request #8849: URL: https://github.com/apache/airflow/pull/8849#issuecomment-628400819 @mik-laj @turbaszek how about instead of creating a new `upload_multiple` method in the hook, we have the `local_to_gcs.py` operator call the `GCSHook.upload` method one or

[GitHub] [airflow] pranjalmittal commented on pull request #8644: Added SalesforceHook missing method to return only dataframe (#8565)

2020-05-13 Thread GitBox
pranjalmittal commented on pull request #8644: URL: https://github.com/apache/airflow/pull/8644#issuecomment-628389253 @feluelle, I have fixed the comment, let me know if any other changes are required. This is an automated

[GitHub] [airflow] pranjalmittal removed a comment on pull request #8644: Added SalesforceHook missing method to return only dataframe (#8565)

2020-05-13 Thread GitBox
pranjalmittal removed a comment on pull request #8644: URL: https://github.com/apache/airflow/pull/8644#issuecomment-628388867 > Besides one comment LGTM  This is an automated message from the Apache Git Service. To

[GitHub] [airflow] pranjalmittal commented on pull request #8644: Added SalesforceHook missing method to return only dataframe (#8565)

2020-05-13 Thread GitBox
pranjalmittal commented on pull request #8644: URL: https://github.com/apache/airflow/pull/8644#issuecomment-628388867 > Besides one comment LGTM  This is an automated message from the Apache Git Service. To respond

[GitHub] [airflow] pranjalmittal closed pull request #8644: Added SalesforceHook missing method to return only dataframe (#8565)

2020-05-13 Thread GitBox
pranjalmittal closed pull request #8644: URL: https://github.com/apache/airflow/pull/8644 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [airflow] pranjalmittal commented on a change in pull request #8644: Added SalesforceHook missing method to return only dataframe (#8565)

2020-05-13 Thread GitBox
pranjalmittal commented on a change in pull request #8644: URL: https://github.com/apache/airflow/pull/8644#discussion_r424872252 ## File path: airflow/providers/salesforce/hooks/salesforce.py ## @@ -236,6 +236,61 @@ def write_object_to_file(self, if fmt not in

[GitHub] [airflow] auvipy commented on pull request #8855: Speed up CeleryExecutor when sending only a single task to Celery

2020-05-13 Thread GitBox
auvipy commented on pull request #8855: URL: https://github.com/apache/airflow/pull/8855#issuecomment-628385105 in the next release of celery 4.4.3 you can optionally use cythonize version of py-amqp in some part which should improve some performence.

[GitHub] [airflow] rahulj51 commented on issue #8731: Logging module (custom) no longer working

2020-05-13 Thread GitBox
rahulj51 commented on issue #8731: URL: https://github.com/apache/airflow/issues/8731#issuecomment-628384741 This looks like it's related to #8484 This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] houqp opened a new pull request #8865: fix task and dag stats on home page

2020-05-13 Thread GitBox
houqp opened a new pull request #8865: URL: https://github.com/apache/airflow/pull/8865 d.dag_id is not a valid attribute. in order to use dag_id variable in a closure callback, it needs to be passed in as a fuction so the right value can be captured for each for loop. ---

[GitHub] [airflow] mik-laj commented on issue #8861: HttpHook.run method doesn't convert dict in 'data' parameter to json object

2020-05-13 Thread GitBox
mik-laj commented on issue #8861: URL: https://github.com/apache/airflow/issues/8861#issuecomment-628375097 Interesting. Do you want to work on it? Can I assign you to this ticket? Here is contributor guide: https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst

[GitHub] [airflow] MichaelChirico opened a new pull request #8864: [AIRFLOW-XXXX] remove vestigial sentence fragment in changelog

2020-05-13 Thread GitBox
MichaelChirico opened a new pull request #8864: URL: https://github.com/apache/airflow/pull/8864 Please update this further if you know what the intended remainder of this sentence was... PS!! The `Suggest a change on this page` link at

[GitHub] [airflow] mik-laj commented on pull request #8751: [AIRFLOW-6856] Bulk fetch paused_dag_ids (v1-10-test)

2020-05-13 Thread GitBox
mik-laj commented on pull request #8751: URL: https://github.com/apache/airflow/pull/8751#issuecomment-628373546 Zrobię to jako osobne commit.Zrobiłem to jako jeden PR z dwóch powodów: - cała zmiane odtworzyłem od nowa, ponieważ jest zbyt dużo różnic, -

[airflow] branch master updated: Do not create a separate process for one task in CeleryExecutor (#8855)

2020-05-13 Thread kamilbregula
This is an automated email from the ASF dual-hosted git repository. kamilbregula pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/master by this push: new fc862a3 Do not create a separate

[GitHub] [airflow] mik-laj merged pull request #8855: Speed up CeleryExecutor when sending only a single task to Celery

2020-05-13 Thread GitBox
mik-laj merged pull request #8855: URL: https://github.com/apache/airflow/pull/8855 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [airflow] mik-laj commented on pull request #8855: Speed up CeleryExecutor when sending only a single task to Celery

2020-05-13 Thread GitBox
mik-laj commented on pull request #8855: URL: https://github.com/apache/airflow/pull/8855#issuecomment-628371106 Breeze on MacOS ``` Before: Loop time: 153.950 ms Loop time: 130.430 ms Loop time: 138.379 ms Loop time: 136.401 ms Loop time: 129.030 ms Loop time:

[jira] [Commented] (AIRFLOW-6878) AIRFLOW__KUBERNETES__KUBE_CLIENT_REQUEST_ARGS default value misconfigured

2020-05-13 Thread Teddy Hartanto (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17106824#comment-17106824 ] Teddy Hartanto commented on AIRFLOW-6878: - Hi [~kaxilnaik], I think the root cause of the issue

[airflow] tag nightly-master updated (7d69987 -> e61b9bb)

2020-05-13 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to tag nightly-master in repository https://gitbox.apache.org/repos/asf/airflow.git. *** WARNING: tag nightly-master was modified! *** from 7d69987 (commit) to e61b9bb (commit) from

[GitHub] [airflow] xinbinhuang commented on pull request #8863: Add EMR operators howto docs

2020-05-13 Thread GitBox
xinbinhuang commented on pull request #8863: URL: https://github.com/apache/airflow/pull/8863#issuecomment-628321923 @feluelle I forgot to add docs for the EMR operators in #8618. This PR adds it back This is an automated

[GitHub] [airflow] xinbinhuang opened a new pull request #8863: Add EMR operators howto docs

2020-05-13 Thread GitBox
xinbinhuang opened a new pull request #8863: URL: https://github.com/apache/airflow/pull/8863 --- Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Unit tests coverage for changes (not needed for documentation

[GitHub] [airflow] xinbinhuang closed pull request #8862: Add EMR Operator howto docs

2020-05-13 Thread GitBox
xinbinhuang closed pull request #8862: URL: https://github.com/apache/airflow/pull/8862 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [airflow] xinbinhuang opened a new pull request #8862: Add EMR Operator howto docs

2020-05-13 Thread GitBox
xinbinhuang opened a new pull request #8862: URL: https://github.com/apache/airflow/pull/8862 --- Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Unit tests coverage for changes (not needed for documentation

[GitHub] [airflow] boring-cyborg[bot] commented on issue #8861: Misleading HttpHook.run method signature

2020-05-13 Thread GitBox
boring-cyborg[bot] commented on issue #8861: URL: https://github.com/apache/airflow/issues/8861#issuecomment-628294761 Thanks for opening your first issue here! Be sure to follow the issue template! This is an automated

[GitHub] [airflow] ToxaZ opened a new issue #8861: Misleading HttpHook.run method signature

2020-05-13 Thread GitBox
ToxaZ opened a new issue #8861: URL: https://github.com/apache/airflow/issues/8861 **Apache Airflow version**: 1.10.10 **What happened**: Execution of HttpHook leads to error (on the API serverside): ``` {"Error":"Could not read JSON: Unrecognized token 'format': was

[GitHub] [airflow] jhtimmins edited a comment on issue #8674: Fix problems with, and test agsinst, Python 3.8

2020-05-13 Thread GitBox
jhtimmins edited a comment on issue #8674: URL: https://github.com/apache/airflow/issues/8674#issuecomment-628275845 Should Airflow support process spawning? Tl;Dr - Spawning creates the possibility of hard-to-detect bugs in child processes. It’s the default on MacOS, the sole

[GitHub] [airflow] jhtimmins edited a comment on issue #8674: Fix problems with, and test agsinst, Python 3.8

2020-05-13 Thread GitBox
jhtimmins edited a comment on issue #8674: URL: https://github.com/apache/airflow/issues/8674#issuecomment-628275845 Should Airflow support process spawning? Tl;Dr - Spawning creates the possibility of hard-to-detect bugs in child processes. It’s the default on MacOS, the sole

[GitHub] [airflow] jhtimmins commented on issue #8674: Fix problems with, and test agsinst, Python 3.8

2020-05-13 Thread GitBox
jhtimmins commented on issue #8674: URL: https://github.com/apache/airflow/issues/8674#issuecomment-628275845 Should Airflow support process spawning? Tl;Dr - Spawning creates the possibility of hard-to-detect bugs in child processes. It’s the default on MacOS, the sole option on

[GitHub] [airflow] ashb commented on pull request #8855: Speed up CeleryExecutor when sending only a single task to Celery

2020-05-13 Thread GitBox
ashb commented on pull request #8855: URL: https://github.com/apache/airflow/pull/8855#issuecomment-628245996 (Somewhat surprised to find creating a process takes 100ms. It should be a fork which should) Change makes sense though

[GitHub] [airflow] pujaji commented on issue #8525: SQLBranchOperator

2020-05-13 Thread GitBox
pujaji commented on issue #8525: URL: https://github.com/apache/airflow/issues/8525#issuecomment-628240078 No regrets! Happy coding-contributing!! :dagger: This is an automated message from the Apache Git Service. To

[GitHub] [airflow] pujaji edited a comment on issue #8525: SQLBranchOperator

2020-05-13 Thread GitBox
pujaji edited a comment on issue #8525: URL: https://github.com/apache/airflow/issues/8525#issuecomment-628240078 No regrets! Happy coding-contributing!! :+1: This is an automated message from the Apache Git Service. To

[GitHub] [airflow] samuelkhtu commented on issue #8525: SQLBranchOperator

2020-05-13 Thread GitBox
samuelkhtu commented on issue #8525: URL: https://github.com/apache/airflow/issues/8525#issuecomment-628239493 Hi @pujaji , I am sorry I misunderstood you. Sounds good. I will give this a try! This is an automated message

[GitHub] [airflow] pujaji commented on issue #8525: SQLBranchOperator

2020-05-13 Thread GitBox
pujaji commented on issue #8525: URL: https://github.com/apache/airflow/issues/8525#issuecomment-628239217 @samuelkhtu No Samuel You have got me wrong. I have stopped working on this. I urge you to accomplish this ans present this to the community

[GitHub] [airflow] kaxil commented on issue #8844: [SQSSensor] max_messages template field issue

2020-05-13 Thread GitBox
kaxil commented on issue #8844: URL: https://github.com/apache/airflow/issues/8844#issuecomment-628239210 Unfortunately, any templated_field can only be a str or an Iterable of string or recurse class that has template_field that is string. It can't be **int**

[GitHub] [airflow] kaxil edited a comment on issue #8859: DuplicateTaskIdFound missing from exceptions.py in pypi package of Apr 9, 2020

2020-05-13 Thread GitBox
kaxil edited a comment on issue #8859: URL: https://github.com/apache/airflow/issues/8859#issuecomment-628237525 here is the code from 1.10.10 for the file that is shown in the stacktrace.

[GitHub] [airflow] kaxil commented on issue #8859: DuplicateTaskIdFound missing from exceptions.py in pypi package of Apr 9, 2020

2020-05-13 Thread GitBox
kaxil commented on issue #8859: URL: https://github.com/apache/airflow/issues/8859#issuecomment-628237525 here is the code from 1.10.10 for the file that is show in the stacktrace.

[GitHub] [airflow] kaxil closed issue #8859: DuplicateTaskIdFound missing from exceptions.py in pypi package of Apr 9, 2020

2020-05-13 Thread GitBox
kaxil closed issue #8859: URL: https://github.com/apache/airflow/issues/8859 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] kaxil commented on issue #8859: DuplicateTaskIdFound missing from exceptions.py in pypi package of Apr 9, 2020

2020-05-13 Thread GitBox
kaxil commented on issue #8859: URL: https://github.com/apache/airflow/issues/8859#issuecomment-628236928 I can't reproduce this error. `DuplicateTaskIdFound` is not a part of 1.10.10 This is an automated message from the

[GitHub] [airflow] samuelkhtu commented on issue #8525: SQLBranchOperator

2020-05-13 Thread GitBox
samuelkhtu commented on issue #8525: URL: https://github.com/apache/airflow/issues/8525#issuecomment-628236594 Thanks @pujaji , sounds like you've got this. I will circle back in a few days. Can't wait to see this new operator!

[GitHub] [airflow] feluelle commented on a change in pull request #8749: add AWS StepFunctions integrations to the aws provider

2020-05-13 Thread GitBox
feluelle commented on a change in pull request #8749: URL: https://github.com/apache/airflow/pull/8749#discussion_r424711158 ## File path: airflow/providers/amazon/aws/operators/step_function_get_execution_output.py ## @@ -0,0 +1,61 @@ +# Licensed to the Apache Software

[GitHub] [airflow] feluelle commented on a change in pull request #8749: add AWS StepFunctions integrations to the aws provider

2020-05-13 Thread GitBox
feluelle commented on a change in pull request #8749: URL: https://github.com/apache/airflow/pull/8749#discussion_r424711244 ## File path: airflow/providers/amazon/aws/operators/step_function_start_execution.py ## @@ -0,0 +1,74 @@ +# Licensed to the Apache Software Foundation

[jira] [Commented] (AIRFLOW-7102) Add try_number to airflow context

2020-05-13 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-7102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17106643#comment-17106643 ] ASF GitHub Bot commented on AIRFLOW-7102: - cong-zhu commented on pull request #7792: URL:

[GitHub] [airflow] cong-zhu commented on pull request #7792: [AIRFLOW-7102] Add try_number to airflow context

2020-05-13 Thread GitBox
cong-zhu commented on pull request #7792: URL: https://github.com/apache/airflow/pull/7792#issuecomment-628226539 @KevinYang21 can we merge this change? This is an automated message from the Apache Git Service. To respond

[GitHub] [airflow] potiuk commented on pull request #8618: Add AWS EMR System tests

2020-05-13 Thread GitBox
potiuk commented on pull request #8618: URL: https://github.com/apache/airflow/pull/8618#issuecomment-628199089 Fantastics! Thanks @feluelle ! This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] samuelkhtu commented on issue #8525: SQLBranchOperator

2020-05-13 Thread GitBox
samuelkhtu commented on issue #8525: URL: https://github.com/apache/airflow/issues/8525#issuecomment-628197562 Thank you both! This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [airflow] samuelkhtu removed a comment on issue #8525: SQLBranchOperator

2020-05-13 Thread GitBox
samuelkhtu removed a comment on issue #8525: URL: https://github.com/apache/airflow/issues/8525#issuecomment-628196134 Thanks @jeffolsi & @potiuk Since there are many SQL favors supported within Airflow, do you have a specific backend in mind? SQLite? Postgres? MySQL?

[GitHub] [airflow] samuelkhtu commented on issue #8525: SQLBranchOperator

2020-05-13 Thread GitBox
samuelkhtu commented on issue #8525: URL: https://github.com/apache/airflow/issues/8525#issuecomment-628196134 Thanks @jeffolsi & @potiuk Since there are many SQL favors supported within Airflow, do you have a specific backend in mind? SQLite? Postgres? MySQL? Maybe you

[GitHub] [airflow] potiuk commented on issue #8525: SQLBranchOperator

2020-05-13 Thread GitBox
potiuk commented on issue #8525: URL: https://github.com/apache/airflow/issues/8525#issuecomment-628195754 Yep. Thanks @jeffolsi for the explanation. Yep. I think this is about right. This is an automated message from the

[airflow] branch master updated (2878f17 -> e61b9bb)

2020-05-13 Thread feluelle
This is an automated email from the ASF dual-hosted git repository. feluelle pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from 2878f17 Relax Flask-Appbuilder version to ~=2.3.4 (#8857) add e61b9bb Add AWS EMR System tests (#8618) No

[airflow] branch master updated (81fb9d6 -> 2878f17)

2020-05-13 Thread kaxilnaik
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from 81fb9d6 Add metric for monitoring email notification failures (#8771) add 2878f17 Relax Flask-Appbuilder

[airflow] branch master updated (c3af681 -> 81fb9d6)

2020-05-13 Thread kaxilnaik
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from c3af681 Convert tests/jobs/test_base_job.py to pytest (#8856) add 81fb9d6 Add metric for monitoring email

[airflow] branch master updated (f1dc2e0 -> c3af681)

2020-05-13 Thread ash
This is an automated email from the ASF dual-hosted git repository. ash pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from f1dc2e0 The librabbitmq library stopped installing for python3.7 (#8853) add c3af681 Convert

[airflow] branch master updated (ed3f513 -> f1dc2e0)

2020-05-13 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from ed3f513 Correctly pass sleep time from AWSAthenaOperator down to the hook. (#8845) add f1dc2e0 The

[airflow] branch master updated (8a94d18 -> ed3f513)

2020-05-13 Thread ash
This is an automated email from the ASF dual-hosted git repository. ash pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from 8a94d18 Fix Environment Variable in perf/scheduler_dag_execution_timing.py (#8847) add ed3f513 Correctly pass

[airflow] branch master updated (e1e833b -> 8a94d18)

2020-05-13 Thread ash
This is an automated email from the ASF dual-hosted git repository. ash pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from e1e833b Update GoogleBaseHook to not follow 308 and use 60s timeout (#8816) add 8a94d18 Fix Environment