[GitHub] [airflow] pukaleashish commented on issue #17566: Airflow with latest slim-buster(3.9) along with latest stable debian release.

2021-08-11 Thread GitBox
pukaleashish commented on issue #17566: URL: https://github.com/apache/airflow/issues/17566#issuecomment-897364873 PFA XLS sheet for detailed vulnerability issues [Airflow_vulneribility.xlsx](https://github.com/apache/airflow/files/6973091/Airflow_vulneribility.xlsx) -- This is

[GitHub] [airflow] yevsh commented on issue #17395: DAG is executed twice on submit

2021-08-11 Thread GitBox
yevsh commented on issue #17395: URL: https://github.com/apache/airflow/issues/17395#issuecomment-897362563 @ephraimbuddy , any ideas? thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] uranusjr commented on issue #17566: Airflow with latest slim-buster(3.9) along with latest stable debian release.

2021-08-11 Thread GitBox
uranusjr commented on issue #17566: URL: https://github.com/apache/airflow/issues/17566#issuecomment-897361025 Could you describe what vulnerability issues you are referring to? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [airflow] uranusjr commented on pull request #16352: AIP-39: ``DagRun.data_interval_start`` and ``data_interval_end``

2021-08-11 Thread GitBox
uranusjr commented on pull request #16352: URL: https://github.com/apache/airflow/pull/16352#issuecomment-897360425 Hmm, it’s still failing, but now for a different version-db combination. I’m going to say it’s flaky and not related… -- This is an automated message from the Apache Git

[GitHub] [airflow] malthe commented on pull request #17545: Airflow exceptions pass-thru during "on_execute_callback"

2021-08-11 Thread GitBox
malthe commented on pull request #17545: URL: https://github.com/apache/airflow/pull/17545#issuecomment-897360098 @potiuk but it's only relevant to check these skip conditions during the creation of a new DAG run. I don't think the tight loop argument applies in this case. And checking a

[GitHub] [airflow] pukaleashish opened a new issue #17566: Airflow with latest slim-buster(3.9) along with latest stable debian release.

2021-08-11 Thread GitBox
pukaleashish opened a new issue #17566: URL: https://github.com/apache/airflow/issues/17566 The current 3.6 slim-buster version has vulnerabilities issues. Could you please update to 3.9 slim-buster with stable debian release. ETA for doing this activity. -- This is an automated

[GitHub] [airflow] boring-cyborg[bot] commented on issue #17566: Airflow with latest slim-buster(3.9) along with latest stable debian release.

2021-08-11 Thread GitBox
boring-cyborg[bot] commented on issue #17566: URL: https://github.com/apache/airflow/issues/17566#issuecomment-897358906 Thanks for opening your first issue here! Be sure to follow the issue template! -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] uranusjr commented on a change in pull request #16866: Remove default_args pattern + added get_current_context() use for Core Airflow example DAGs

2021-08-11 Thread GitBox
uranusjr commented on a change in pull request #16866: URL: https://github.com/apache/airflow/pull/16866#discussion_r687396592 ## File path: airflow/example_dags/example_xcom.py ## @@ -78,6 +104,14 @@ def puller(**kwargs): pull = PythonOperator( task_id='puller',

[GitHub] [airflow] uranusjr commented on pull request #16161: Ensure datasetReference key exists in BigQueryHook.create_empty_dataset

2021-08-11 Thread GitBox
uranusjr commented on pull request #16161: URL: https://github.com/apache/airflow/pull/16161#issuecomment-897350839 @kurtqq feel free to comment on the original issue, we can assign you there if you want to submit a pull request. -- This is an automated message from the Apache Git

[GitHub] [airflow] uranusjr opened a new pull request #16352: AIP-39: ``DagRun.data_interval_start`` and ``data_interval_end``

2021-08-11 Thread GitBox
uranusjr opened a new pull request #16352: URL: https://github.com/apache/airflow/pull/16352 Close #16304. Todo: * [x] Add those fields to the actual `DagRun` model. * [x] Populate the fields when a `DagRun` instance is created (when `DagRun.execution_date` is populated). I think

[GitHub] [airflow] uranusjr commented on pull request #16352: AIP-39: ``DagRun.data_interval_start`` and ``data_interval_end``

2021-08-11 Thread GitBox
uranusjr commented on pull request #16352: URL: https://github.com/apache/airflow/pull/16352#issuecomment-897340148 Doesn’t repro locally. Let’s try again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [airflow] uranusjr closed pull request #16352: AIP-39: ``DagRun.data_interval_start`` and ``data_interval_end``

2021-08-11 Thread GitBox
uranusjr closed pull request #16352: URL: https://github.com/apache/airflow/pull/16352 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[airflow] branch v2-1-test updated: Add State types for tasks and DAGs (#15285)

2021-08-11 Thread jhtimmins
This is an automated email from the ASF dual-hosted git repository. jhtimmins pushed a commit to branch v2-1-test in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/v2-1-test by this push: new c37de28 Add State types for tasks

[GitHub] [airflow] uranusjr commented on pull request #16352: AIP-39: ``DagRun.data_interval_start`` and ``data_interval_end``

2021-08-11 Thread GitBox
uranusjr commented on pull request #16352: URL: https://github.com/apache/airflow/pull/16352#issuecomment-897326965 `TestSchedulerJob.test_verify_integrity_if_dag_not_changed` Huh that’s new, need to investigate a bit. -- This is an automated message from the Apache Git Service. To

[GitHub] [airflow] kurtqq commented on pull request #16161: Ensure datasetReference key exists in BigQueryHook.create_empty_dataset

2021-08-11 Thread GitBox
kurtqq commented on pull request #16161: URL: https://github.com/apache/airflow/pull/16161#issuecomment-897323556 i think this is worth fixing -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [airflow] josh-fell commented on issue #9415: [AIP-31] Update DAG examples to use Functional DAG style

2021-08-11 Thread GitBox
josh-fell commented on issue #9415: URL: https://github.com/apache/airflow/issues/9415#issuecomment-897309570 FYI - Just waiting on 1 PR to get merged for core example DAG updates then I'll be good to go taking this on shortly thereafter. -- This is an automated message from the Apache

[GitHub] [airflow] josh-fell commented on a change in pull request #16866: Remove default_args pattern + added get_current_context() use for Core Airflow example DAGs

2021-08-11 Thread GitBox
josh-fell commented on a change in pull request #16866: URL: https://github.com/apache/airflow/pull/16866#discussion_r687325009 ## File path: airflow/example_dags/example_xcom.py ## @@ -78,6 +104,14 @@ def puller(**kwargs): pull = PythonOperator(

[GitHub] [airflow] josh-fell commented on a change in pull request #16866: Remove default_args pattern + added get_current_context() use for Core Airflow example DAGs

2021-08-11 Thread GitBox
josh-fell commented on a change in pull request #16866: URL: https://github.com/apache/airflow/pull/16866#discussion_r687325009 ## File path: airflow/example_dags/example_xcom.py ## @@ -78,6 +104,14 @@ def puller(**kwargs): pull = PythonOperator(

[GitHub] [airflow] LionelZhao28 commented on pull request #17502: localize the run_id in dagrun

2021-08-11 Thread GitBox
LionelZhao28 commented on pull request #17502: URL: https://github.com/apache/airflow/pull/17502#issuecomment-897303309 @ashb @kaxil @XD-DENG The PR is ready for review. I think this change is quite useful for us who are not using UTC time. Please review it when you are free.

[GitHub] [airflow] aa1371 commented on pull request #16829: Queue support for DaskExecutor using Dask Worker Resources

2021-08-11 Thread GitBox
aa1371 commented on pull request #16829: URL: https://github.com/apache/airflow/pull/16829#issuecomment-897294117 updated -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[airflow] 02/03: Fix external elasticsearch logs link (#16357)

2021-08-11 Thread jhtimmins
This is an automated email from the ASF dual-hosted git repository. jhtimmins pushed a commit to branch v2-1-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 5ea02ccf460dd3fcd0f85cd2d51e83d451f87a04 Author: Jed Cunningham <66968678+jedcunning...@users.noreply.github.com>

[airflow] 01/03: Fixed broken json_client (#17529)

2021-08-11 Thread jhtimmins
This is an automated email from the ASF dual-hosted git repository. jhtimmins pushed a commit to branch v2-1-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 3d6db6d98522788be32c9b1a231d0e9a6941e50c Author: sw1010 AuthorDate: Tue Aug 10 13:26:54 2021 +0200 Fixed

[airflow] branch v2-1-test updated (fe70111 -> 8f96385)

2021-08-11 Thread jhtimmins
This is an automated email from the ASF dual-hosted git repository. jhtimmins pushed a change to branch v2-1-test in repository https://gitbox.apache.org/repos/asf/airflow.git. from fe70111 Add 'queued' to DagRunState (#16854) new 3d6db6d Fixed broken json_client (#17529) new

[airflow] 03/03: Add State types for tasks and DAGs (#15285)

2021-08-11 Thread jhtimmins
This is an automated email from the ASF dual-hosted git repository. jhtimmins pushed a commit to branch v2-1-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 8f963850be85aecd530b281a034468e375661c4c Author: Andrew Godwin AuthorDate: Tue Jul 6 17:36:01 2021 -0600

[airflow] 01/03: Fix bug that log can't be shown when task runs failed (#16768)

2021-08-11 Thread jhtimmins
This is an automated email from the ASF dual-hosted git repository. jhtimmins pushed a commit to branch v2-1-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 26e8d2d77aeb51b8dbfb758c477417ccd0c45063 Author: Zhanfeng Huo AuthorDate: Fri Jul 23 06:13:40 2021 +0800

[airflow] 02/03: Fix task retries when they receive sigkill and have retries and properly handle sigterm (#16301)

2021-08-11 Thread jhtimmins
This is an automated email from the ASF dual-hosted git repository. jhtimmins pushed a commit to branch v2-1-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 1be2003b69b6f0cdd58d0743640bdb797acdfac4 Author: Ephraim Anierobi AuthorDate: Wed Jul 28 15:57:35 2021 +0100

[airflow] branch v2-1-test updated (20ed40b -> fe70111)

2021-08-11 Thread jhtimmins
This is an automated email from the ASF dual-hosted git repository. jhtimmins pushed a change to branch v2-1-test in repository https://gitbox.apache.org/repos/asf/airflow.git. from 20ed40b Better diagnostics and self-healing of docker-compose (#17484) new 26e8d2d Fix bug that log

[airflow] 03/03: Add 'queued' to DagRunState (#16854)

2021-08-11 Thread jhtimmins
This is an automated email from the ASF dual-hosted git repository. jhtimmins pushed a commit to branch v2-1-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit fe7011100e4c2e789d2ca185e2733373414bd3ae Author: Ephraim Anierobi AuthorDate: Wed Jul 7 21:37:22 2021 +0100

[GitHub] [airflow] github-actions[bot] commented on issue #13391: MySqlToHiveOperator , ”invalid path“ while loding the extracted csv to hive

2021-08-11 Thread GitBox
github-actions[bot] commented on issue #13391: URL: https://github.com/apache/airflow/issues/13391#issuecomment-897241641 This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further

[GitHub] [airflow] github-actions[bot] closed pull request #16446: Use resource and action names in test_security.py.

2021-08-11 Thread GitBox
github-actions[bot] closed pull request #16446: URL: https://github.com/apache/airflow/pull/16446 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] github-actions[bot] commented on pull request #14176: Simplify CONTRIBUTING.rst

2021-08-11 Thread GitBox
github-actions[bot] commented on pull request #14176: URL: https://github.com/apache/airflow/pull/14176#issuecomment-897241593 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank

[GitHub] [airflow] repocho commented on pull request #17537: Extend init_containers defined in pod_override

2021-08-11 Thread GitBox
repocho commented on pull request #17537: URL: https://github.com/apache/airflow/pull/17537#issuecomment-897230204 Thanks for the time reviewing this PR @jedcunningham ! I have added a test for this. Let me know if you need anything more. Thanks again. -- This is an automated

[GitHub] [airflow] andrewgodwin opened a new pull request #17565: Add Triggerer warning banner in UI

2021-08-11 Thread GitBox
andrewgodwin opened a new pull request #17565: URL: https://github.com/apache/airflow/pull/17565 This adds a "triggerer is not running" banner in the Web UI, similar to the one that shows when the scheduler is not running. As well as checking for a TriggererJob, though, it also

[airflow] 10/11: Install providers from sources in prod image only on main (#17458)

2021-08-11 Thread jhtimmins
This is an automated email from the ASF dual-hosted git repository. jhtimmins pushed a commit to branch v2-1-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit bc79182ba24142c57c4b95da26cd59219f6a03da Author: Jarek Potiuk AuthorDate: Sat Aug 7 22:47:48 2021 +0200

[airflow] 09/11: Attempt to reduce flakiness of PythonVirtualeEnv test_airflow_context (#17486)

2021-08-11 Thread jhtimmins
This is an automated email from the ASF dual-hosted git repository. jhtimmins pushed a commit to branch v2-1-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit dcfc23ecd5fd8d175d9d46097f6f05ce46575a8a Author: Jarek Potiuk AuthorDate: Sat Aug 7 22:47:15 2021 +0200

[airflow] 08/11: Disable Helm tests when branch is not main

2021-08-11 Thread jhtimmins
This is an automated email from the ASF dual-hosted git repository. jhtimmins pushed a commit to branch v2-1-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 005a5d0afc5879a9db1a4d1be6b778101bbf235b Author: Jarek Potiuk AuthorDate: Fri Aug 6 10:19:26 2021 +0200

[airflow] 06/11: Switches to "/" convention in ghcr.io images (#17356)

2021-08-11 Thread jhtimmins
This is an automated email from the ASF dual-hosted git repository. jhtimmins pushed a commit to branch v2-1-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit d5f0b1ad9b96d51c64d1c33b783528a7d4946c0e Author: Jarek Potiuk AuthorDate: Thu Aug 5 18:39:43 2021 +0200

[airflow] 11/11: Better diagnostics and self-healing of docker-compose (#17484)

2021-08-11 Thread jhtimmins
This is an automated email from the ASF dual-hosted git repository. jhtimmins pushed a commit to branch v2-1-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 20ed40b4ba2f754c205a2d193319cd99d8b144f7 Author: Jarek Potiuk AuthorDate: Mon Aug 9 10:41:57 2021 +0200

[airflow] 05/11: Fix failing static checks in main (#17424)

2021-08-11 Thread jhtimmins
This is an automated email from the ASF dual-hosted git repository. jhtimmins pushed a commit to branch v2-1-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 9d2ad2acb996b34c491dacbf911cc99670c21df9 Author: Jarek Potiuk AuthorDate: Wed Aug 4 23:33:29 2021 +0200 Fix

[airflow] 04/11: Add timeout when asking whether to rebuild image (#17412)

2021-08-11 Thread jhtimmins
This is an automated email from the ASF dual-hosted git repository. jhtimmins pushed a commit to branch v2-1-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 2763e8e9708d46de54749577f6762c7fa6cb82e8 Author: Jarek Potiuk AuthorDate: Wed Aug 4 15:44:06 2021 +0200 Add

[airflow] 07/11: Fixes some static check errors (#17432)

2021-08-11 Thread jhtimmins
This is an automated email from the ASF dual-hosted git repository. jhtimmins pushed a commit to branch v2-1-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 31005cb5b6869469ce51ad33950a1afbaaa18adc Author: Jarek Potiuk AuthorDate: Thu Aug 5 08:43:51 2021 +0200

[airflow] 01/11: Enhancement to bash scripts (#17098)

2021-08-11 Thread jhtimmins
This is an automated email from the ASF dual-hosted git repository. jhtimmins pushed a commit to branch v2-1-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 403a087a3a089b4f7e92023f9336bf8acdd972f2 Author: Shraman Basyal <49772843+bshra...@users.noreply.github.com>

[airflow] 02/11: Improve image building documentation for new users (#17409)

2021-08-11 Thread jhtimmins
This is an automated email from the ASF dual-hosted git repository. jhtimmins pushed a commit to branch v2-1-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit f2da764a3d50ec7fe5ea2e3497b21dce5f0219a0 Author: Jarek Potiuk AuthorDate: Wed Aug 4 23:07:22 2021 +0200

[airflow] 03/11: Optimizes structure of the Dockerfiles and use latest tools (#17418)

2021-08-11 Thread jhtimmins
This is an automated email from the ASF dual-hosted git repository. jhtimmins pushed a commit to branch v2-1-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 70629cad94ff26ace19f9a817884e171e626898b Author: Jarek Potiuk AuthorDate: Wed Aug 4 23:32:12 2021 +0200

[airflow] branch v2-1-test updated (205d6d4 -> 20ed40b)

2021-08-11 Thread jhtimmins
This is an automated email from the ASF dual-hosted git repository. jhtimmins pushed a change to branch v2-1-test in repository https://gitbox.apache.org/repos/asf/airflow.git. from 205d6d4 Run mini scheduler in LocalTaskJob during task exit (#16289) new 403a087 Enhancement to bash

[GitHub] [airflow] andrewgodwin opened a new pull request #17564: Fixed long delays never returning in the Temporal trigger

2021-08-11 Thread GitBox
andrewgodwin opened a new pull request #17564: URL: https://github.com/apache/airflow/pull/17564 This trigger was never actually working for delays over two hours, as it was comparing a Pendulum.Period to a datetime.datetime, in a way that always returned `True`. This commit fixes

[GitHub] [airflow] rchangj commented on issue #17558: New job fail immediately after start, no logs

2021-08-11 Thread GitBox
rchangj commented on issue #17558: URL: https://github.com/apache/airflow/issues/17558#issuecomment-897213964 The SSL certs are just self signed certs created as following. openssl genrsa -out private.pem 2048 openssl req -new -x509 -key private.pem -out cacert.pem -days 1095

[GitHub] [airflow] potiuk commented on pull request #17562: Chart: Support extraContainers in k8s workers

2021-08-11 Thread GitBox
potiuk commented on pull request #17562: URL: https://github.com/apache/airflow/pull/17562#issuecomment-897207982 OH NO.. I see that Sidecar container feature has not only dropped from 1.18, but also did not make it to 1.19 and 1.20 and has been finally dropped

[airflow] branch main updated (a1834ce -> 944dc32)

2021-08-11 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git. from a1834ce Use dag_maker fixture in test_processor.py (#17506) add 944dc32 Fix Invalid log order in

[GitHub] [airflow] potiuk merged pull request #17551: Fix Invalid log order in ElasticsearchTaskHandler

2021-08-11 Thread GitBox
potiuk merged pull request #17551: URL: https://github.com/apache/airflow/pull/17551 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #17551: Fix Invalid log order in ElasticsearchTaskHandler

2021-08-11 Thread GitBox
boring-cyborg[bot] commented on pull request #17551: URL: https://github.com/apache/airflow/pull/17551#issuecomment-897204423 Awesome work, congrats on your first merged pull request! -- This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [airflow] snolan-amount opened a new pull request #17563: Add Spark to the EMR cluster for the job flow examples

2021-08-11 Thread GitBox
snolan-amount opened a new pull request #17563: URL: https://github.com/apache/airflow/pull/17563 The example step fails if Spark is not included in the applications in the cluster. --- **^ Add meaningful description above** Read the **[Pull Request

[GitHub] [airflow] potiuk edited a comment on pull request #17562: Chart: Support extraContainers in k8s workers

2021-08-11 Thread GitBox
potiuk edited a comment on pull request #17562: URL: https://github.com/apache/airflow/pull/17562#issuecomment-897197886 Should not we use (and possibly even enforce) the "lifecycle: Sidecar" for those extraContainers ? It is available from K8S 1.18 and seems to handle nicely not only

[GitHub] [airflow] potiuk commented on pull request #17562: Chart: Support extraContainers in k8s workers

2021-08-11 Thread GitBox
potiuk commented on pull request #17562: URL: https://github.com/apache/airflow/pull/17562#issuecomment-897197886 Should not we use (and possibly even enforce) the "lifecycle: Sidecar" ? It is available from K8S 1.18 and seems to handle nicely not only automated termination when all

[airflow] branch main updated: Use dag_maker fixture in test_processor.py (#17506)

2021-08-11 Thread ephraimanierobi
This is an automated email from the ASF dual-hosted git repository. ephraimanierobi pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new a1834ce Use dag_maker fixture in

[GitHub] [airflow] ephraimbuddy merged pull request #17506: Use `dag_maker` fixture in test_processor.py

2021-08-11 Thread GitBox
ephraimbuddy merged pull request #17506: URL: https://github.com/apache/airflow/pull/17506 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] github-actions[bot] commented on pull request #17562: Chart: Support extraContainers in k8s workers

2021-08-11 Thread GitBox
github-actions[bot] commented on pull request #17562: URL: https://github.com/apache/airflow/pull/17562#issuecomment-897194550 The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not

[airflow] branch main updated (ec5b4fe -> 3773224)

2021-08-11 Thread jedcunningham
This is an automated email from the ASF dual-hosted git repository. jedcunningham pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git. from ec5b4fe AIP-40: Add Deferrable "Async" Tasks (#15389) add 3773224 Chart: Allow podTemplate to be

[GitHub] [airflow] jedcunningham merged pull request #17560: Chart: Allow podTemplate to be templated

2021-08-11 Thread GitBox
jedcunningham merged pull request #17560: URL: https://github.com/apache/airflow/pull/17560 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] jedcunningham commented on pull request #17562: Chart: Support extraContainers in k8s workers

2021-08-11 Thread GitBox
jedcunningham commented on pull request #17562: URL: https://github.com/apache/airflow/pull/17562#issuecomment-897191987 I'm tempted to also add a warning in `NOTES.txt` (maybe with a way to hide the warning so folks aren't annoyed). Thoughts? (I just foresee folks running into

[GitHub] [airflow] github-actions[bot] commented on pull request #17551: Fix Invalid log order in ElasticsearchTaskHandler

2021-08-11 Thread GitBox
github-actions[bot] commented on pull request #17551: URL: https://github.com/apache/airflow/pull/17551#issuecomment-897191897 The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not

[GitHub] [airflow] jedcunningham opened a new pull request #17562: Chart: Support extraContainers in k8s workers

2021-08-11 Thread GitBox
jedcunningham opened a new pull request #17562: URL: https://github.com/apache/airflow/pull/17562 This allows extraContainers to be provided for k8s workers, however one must keep in mind they are responsible for signaling any sidecars to exit. -- This is an automated message from

[airflow] 03/03: Run mini scheduler in LocalTaskJob during task exit (#16289)

2021-08-11 Thread jhtimmins
This is an automated email from the ASF dual-hosted git repository. jhtimmins pushed a commit to branch v2-1-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 205d6d4debe111d053b8674ff2ea991b95203be6 Author: Ephraim Anierobi AuthorDate: Thu Jun 10 14:29:30 2021 +0100

[airflow] 02/03: Fix race condition with dagrun callbacks (#16741)

2021-08-11 Thread jhtimmins
This is an automated email from the ASF dual-hosted git repository. jhtimmins pushed a commit to branch v2-1-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 38d4bab47fe268dd0f9ae3760e87ec143b56184e Author: Jed Cunningham <66968678+jedcunning...@users.noreply.github.com>

[airflow] 01/03: Add 'queued' state to DagRun (#16401)

2021-08-11 Thread jhtimmins
This is an automated email from the ASF dual-hosted git repository. jhtimmins pushed a commit to branch v2-1-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit f7bece3da14879712a895e897dad339944170ce3 Author: Ephraim Anierobi AuthorDate: Tue Jul 6 15:03:27 2021 +0100

[airflow] branch v2-1-test updated (0bd9558 -> 205d6d4)

2021-08-11 Thread jhtimmins
This is an automated email from the ASF dual-hosted git repository. jhtimmins pushed a change to branch v2-1-test in repository https://gitbox.apache.org/repos/asf/airflow.git. from 0bd9558 Proper warning message when recorded PID is different from current PID (#17411) new f7bece3

[GitHub] [airflow] potiuk commented on issue #17558: New job fail immediately after start, no logs

2021-08-11 Thread GitBox
potiuk commented on issue #17558: URL: https://github.com/apache/airflow/issues/17558#issuecomment-897172899 It's really, really hard to say without more logs, You might want to try to find more logs. Also It would be great if you explain what kind of SSL configuration you've made (why?)

[GitHub] [airflow] github-actions[bot] commented on pull request #17560: Chart: Allow podTemplate to be templated

2021-08-11 Thread GitBox
github-actions[bot] commented on pull request #17560: URL: https://github.com/apache/airflow/pull/17560#issuecomment-897171100 The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not

[GitHub] [airflow] potiuk opened a new pull request #17561: Simplify bug report template

2021-08-11 Thread GitBox
potiuk opened a new pull request #17561: URL: https://github.com/apache/airflow/pull/17561 By looking at several tens of issues recently I think I got a bit better understanding on what people are actually reporting and which information is really important - and I think the report

[GitHub] [airflow] jedcunningham opened a new pull request #17560: Chart: Allow podTemplate to be templated

2021-08-11 Thread GitBox
jedcunningham opened a new pull request #17560: URL: https://github.com/apache/airflow/pull/17560 We should allow the user to use a templated string for `podTemplate`, as it allows one to start with the template file and just modifying as required instead of needing to grab the rendered

[GitHub] [airflow] potiuk edited a comment on pull request #17545: Airflow exceptions pass-thru during "on_execute_callback"

2021-08-11 Thread GitBox
potiuk edited a comment on pull request #17545: URL: https://github.com/apache/airflow/pull/17545#issuecomment-897142870 > And what do you call your branching operator ... "check-if-saturday-task123". It doesn't seem particularly elegant to me. I agree with @kaxil that this case is

[GitHub] [airflow] potiuk commented on pull request #17545: Airflow exceptions pass-thru during "on_execute_callback"

2021-08-11 Thread GitBox
potiuk commented on pull request #17545: URL: https://github.com/apache/airflow/pull/17545#issuecomment-897142870 > And what do you call your branching operator ... "check-if-saturday-task123". It doesn't seem particularly elegant to me. I agree with @kaxil that this case is not

[GitHub] [airflow] malthe edited a comment on pull request #17545: Airflow exceptions pass-thru during "on_execute_callback"

2021-08-11 Thread GitBox
malthe edited a comment on pull request #17545: URL: https://github.com/apache/airflow/pull/17545#issuecomment-897133662 @kaxil it's an option of course – as is subclassing and other ways to code around the problem. But I think there's a usability gain here that's worth considering

[GitHub] [airflow] malthe commented on pull request #17545: Airflow exceptions pass-thru during "on_execute_callback"

2021-08-11 Thread GitBox
malthe commented on pull request #17545: URL: https://github.com/apache/airflow/pull/17545#issuecomment-897133662 @kaxil it's an option of course – as is subclassing and other ways to code around the problem. But I think there's a usability gain here that's worth considering because it's

[GitHub] [airflow] kaxil commented on a change in pull request #17557: Make EMR cluster visible to all users note

2021-08-11 Thread GitBox
kaxil commented on a change in pull request #17557: URL: https://github.com/apache/airflow/pull/17557#discussion_r687169792 ## File path: docs/apache-airflow-providers-amazon/operators/emr.rst ## @@ -73,6 +73,10 @@ To create a job flow at EMR, you need to specify the

[GitHub] [airflow] kaxil commented on pull request #17545: Airflow exceptions pass-thru during "on_execute_callback"

2021-08-11 Thread GitBox
kaxil commented on pull request #17545: URL: https://github.com/apache/airflow/pull/17545#issuecomment-897130778 > @kaxil in our business flows, it's often the case that some branches are only applicable to certain days – but the DAG as a whole runs every day. Why not use

[GitHub] [airflow] kaxil commented on pull request #17545: Airflow exceptions pass-thru during "on_execute_callback"

2021-08-11 Thread GitBox
kaxil commented on pull request #17545: URL: https://github.com/apache/airflow/pull/17545#issuecomment-897130143 We have Trigger rules, Branch Operators and cron expression for this use case. While I agree with @uranusjr partially, I feel this is a wrong use-case. -- This is an

[GitHub] [airflow] malthe commented on pull request #17545: Airflow exceptions pass-thru during "on_execute_callback"

2021-08-11 Thread GitBox
malthe commented on pull request #17545: URL: https://github.com/apache/airflow/pull/17545#issuecomment-897129361 @kaxil in our business flows, it's often the case that some branches are only applicable to certain days – but the DAG as a whole runs every day. -- This is an automated

[GitHub] [airflow] malthe commented on pull request #17545: Airflow exceptions pass-thru during "on_execute_callback"

2021-08-11 Thread GitBox
malthe commented on pull request #17545: URL: https://github.com/apache/airflow/pull/17545#issuecomment-897128837 @potiuk for UI purposes etc I think there is a value in having a mechanism to specifically determine whether or not a particular DAG run will execute a task or not.

[GitHub] [airflow] kaxil commented on pull request #17545: Airflow exceptions pass-thru during "on_execute_callback"

2021-08-11 Thread GitBox
kaxil commented on pull request #17545: URL: https://github.com/apache/airflow/pull/17545#issuecomment-897128657 Sounds like a wrong usecase, if you want to skip running on Saturday's, just exclude Saturday from schedule_interval via cron expression: https://crontab.guru/#0_0_*_*_0-5 --

[GitHub] [airflow] potiuk commented on pull request #17545: Airflow exceptions pass-thru during "on_execute_callback"

2021-08-11 Thread GitBox
potiuk commented on pull request #17545: URL: https://github.com/apache/airflow/pull/17545#issuecomment-897123893 > @kaxil so the issue is how @uranusjr put it, that subclassing isn't necessarily a natural thing in this situation – why would I subclass an `OracleOperator` if I just want

[GitHub] [airflow] ferruzzi commented on pull request #17557: Make EMR cluster visible to all users note

2021-08-11 Thread GitBox
ferruzzi commented on pull request #17557: URL: https://github.com/apache/airflow/pull/17557#issuecomment-897121944 Nice clarification. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow-ci-infra] potiuk closed pull request #19: Add gcp cloud init

2021-08-11 Thread GitBox
potiuk closed pull request #19: URL: https://github.com/apache/airflow-ci-infra/pull/19 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] malthe commented on pull request #17545: Airflow exceptions pass-thru during "on_execute_callback"

2021-08-11 Thread GitBox
malthe commented on pull request #17545: URL: https://github.com/apache/airflow/pull/17545#issuecomment-897116510 @kaxil so the issue is how @uranusjr put it, that subclassing isn't necessarily a natural thing in this situation – why would I subclass an `OracleOperator` if I just want to

[airflow-ci-infra] branch upgrade-to-latest-runner created (now 063db81)

2021-08-11 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch upgrade-to-latest-runner in repository https://gitbox.apache.org/repos/asf/airflow-ci-infra.git. at 063db81 Upgrade to latest released runner version This branch includes the following

[airflow-ci-infra] 01/01: Upgrade to latest released runner version

2021-08-11 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch upgrade-to-latest-runner in repository https://gitbox.apache.org/repos/asf/airflow-ci-infra.git commit 063db818d59f022634e47c5406282bda5acc7f28 Author: Jarek Potiuk AuthorDate: Wed Aug 11

[airflow-ci-infra] branch disable-packer-role created (now 2939550)

2021-08-11 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch disable-packer-role in repository https://gitbox.apache.org/repos/asf/airflow-ci-infra.git. at 2939550 Remove unnecessary packer role setting This branch includes the following new

[airflow-ci-infra] 01/01: Remove unnecessary packer role setting

2021-08-11 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch disable-packer-role in repository https://gitbox.apache.org/repos/asf/airflow-ci-infra.git commit 293955018d5f95e8192305d3710749af89ff5605 Author: Jarek Potiuk AuthorDate: Wed Aug 11 22:05:57

[GitHub] [airflow] kaxil commented on pull request #17545: Airflow exceptions pass-thru during "on_execute_callback"

2021-08-11 Thread GitBox
kaxil commented on pull request #17545: URL: https://github.com/apache/airflow/pull/17545#issuecomment-897105720 @malthe We already have some exceptions for that: check https://airflow.apache.org/docs/apache-airflow/stable/concepts/tasks.html#special-exceptions -- This is an automated

[GitHub] [airflow] kaxil commented on a change in pull request #17557: Make EMR cluster visible to all users note

2021-08-11 Thread GitBox
kaxil commented on a change in pull request #17557: URL: https://github.com/apache/airflow/pull/17557#discussion_r687118707 ## File path: docs/apache-airflow-providers-amazon/operators/emr.rst ## @@ -73,6 +73,10 @@ To create a job flow at EMR, you need to specify the

[GitHub] [airflow] kaxil commented on pull request #15389: AIP-40: Deferrable Tasks

2021-08-11 Thread GitBox
kaxil commented on pull request #15389: URL: https://github.com/apache/airflow/pull/15389#issuecomment-897096018 Looks good to me -- I think we can follow up if there are more changes -- This is an automated message from the Apache Git Service. To respond to the message, please

[airflow] branch main updated (b06d528 -> ec5b4fe)

2021-08-11 Thread kaxilnaik
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git. from b06d528 Don't cache Google Secret Manager client (#17539) add ec5b4fe AIP-40: Add Deferrable "Async" Tasks

[GitHub] [airflow] kaxil merged pull request #15389: AIP-40: Deferrable Tasks

2021-08-11 Thread GitBox
kaxil merged pull request #15389: URL: https://github.com/apache/airflow/pull/15389 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] github-actions[bot] commented on pull request #15389: AIP-40: Deferrable Tasks

2021-08-11 Thread GitBox
github-actions[bot] commented on pull request #15389: URL: https://github.com/apache/airflow/pull/15389#issuecomment-897095313 The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and

[GitHub] [airflow] jedcunningham commented on pull request #17537: Extend init_containers defined in pod_override

2021-08-11 Thread GitBox
jedcunningham commented on pull request #17537: URL: https://github.com/apache/airflow/pull/17537#issuecomment-897093972 Thanks @repocho! Can you add a test for this in

[GitHub] [airflow] jedcunningham commented on a change in pull request #15389: AIP-40: Deferrable Tasks

2021-08-11 Thread GitBox
jedcunningham commented on a change in pull request #15389: URL: https://github.com/apache/airflow/pull/15389#discussion_r687106610 ## File path: airflow/jobs/triggerer_job.py ## @@ -0,0 +1,380 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more

[GitHub] [airflow] jbkc85 commented on issue #13542: Task stuck in "scheduled" or "queued" state, pool has all slots queued, nothing is executing

2021-08-11 Thread GitBox
jbkc85 commented on issue #13542: URL: https://github.com/apache/airflow/issues/13542#issuecomment-897078856 > hi @jbkc85 thank you for the suggestion! May I know how to create different pools in Airflow ? many thanks! Pools are made through the Admin interface. Then you can

[GitHub] [airflow] collinmcnulty opened a new issue #17559: Dag Import Error Hidden for existing dags leading to nondescriptive UI errors

2021-08-11 Thread GitBox
collinmcnulty opened a new issue #17559: URL: https://github.com/apache/airflow/issues/17559 Python version: 3.7.10 **Apache Airflow version**: 2.1.1 **Apache Airflow Provider versions** (please include all providers that are relevant to your bug): None **Kubernetes

[GitHub] [airflow] boring-cyborg[bot] commented on issue #17558: New job fail immediately after start, no logs

2021-08-11 Thread GitBox
boring-cyborg[bot] commented on issue #17558: URL: https://github.com/apache/airflow/issues/17558#issuecomment-897065208 Thanks for opening your first issue here! Be sure to follow the issue template! -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] rchangj opened a new issue #17558: New job fail immediately after start, no logs

2021-08-11 Thread GitBox
rchangj opened a new issue #17558: URL: https://github.com/apache/airflow/issues/17558 **Apache Airflow version**:1.10.12 **Apache Airflow Provider versions** (please include all providers that are relevant to your bug): No other providers **Kubernetes version (if you are

  1   2   3   >