Re: [PR] Send explicit task logs when marking tasks stuck in queued as failed [airflow]

2023-11-25 Thread via GitHub
ephraimbuddy commented on code in PR #35857: URL: https://github.com/apache/airflow/pull/35857#discussion_r1405348580 ## airflow/jobs/scheduler_job_runner.py: ## @@ -1577,15 +1577,18 @@ def _fail_tasks_stuck_in_queued(self, session: Session = NEW_SESSION) -> None:

(airflow) branch v2-8-test updated (5bbc46005f -> 02ffb4b415)

2023-11-25 Thread ephraimanierobi
This is an automated email from the ASF dual-hosted git repository. ephraimanierobi pushed a change to branch v2-8-test in repository https://gitbox.apache.org/repos/asf/airflow.git discard 5bbc46005f Update RELEASE_NOTES.rst discard 20cb92e52a Update version to 2.8.0 discard 55914e14cb Add

Re: [PR] Send explicit task logs when marking tasks stuck in queued as failed [airflow]

2023-11-25 Thread via GitHub
pankajkoti commented on PR #35857: URL: https://github.com/apache/airflow/pull/35857#issuecomment-1826680847 cc: @RNHTTR @vatsrahul1001 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Send explicit task logs when marking tasks stuck in queued as failed [airflow]

2023-11-25 Thread via GitHub
pankajkoti commented on PR #35857: URL: https://github.com/apache/airflow/pull/35857#issuecomment-1826679595 https://github.com/apache/airflow/assets/10206082/e8d87e6e-1b39-487f-a606-06d06905bffd;> -- This is an automated message from the Apache Git Service. To respond to the message,

[PR] Send explicit task logs when marking tasks stuck in queued as failed [airflow]

2023-11-25 Thread via GitHub
pankajkoti opened a new pull request, #35857: URL: https://github.com/apache/airflow/pull/35857 Using the feature built in #32646, when the scheduler marks tasks stuck in queued as failed, send such an explicit log indicating the action to the task logs so that it helps users

Re: [PR] Fix gcs listing - ensure blobs are loaded [airflow]

2023-11-25 Thread via GitHub
eladkal commented on PR #34919: URL: https://github.com/apache/airflow/pull/34919#issuecomment-1826659574 @atrbgithub can you rebase? (You disabled allowing mantainers to do changes on your branch so I can't do it for you) -- This is an automated message from the Apache Git Service. To

Re: [PR] KubernetesPodTrigger: add exception stack trace in TriggerEvent [airflow]

2023-11-25 Thread via GitHub
eladkal commented on code in PR #35716: URL: https://github.com/apache/airflow/pull/35716#discussion_r1405333949 ## airflow/providers/cncf/kubernetes/operators/pod.py: ## @@ -672,6 +672,7 @@ def invoke_defer_method(self): ) def execute_complete(self, context:

Re: [PR] add guideline for missing image when running breeze [airflow]

2023-11-25 Thread via GitHub
Lee-W commented on PR #35813: URL: https://github.com/apache/airflow/pull/35813#issuecomment-1826477533 Wow, it's a huge PR. thanks for helping out -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

(airflow) 01/01: Optimize CI/PROD image waiting and verification in CI workflow

2023-11-25 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch optimize-image-wait-verify in repository https://gitbox.apache.org/repos/asf/airflow.git commit 7b856296bae7236445e2e685de3c352a7dc75bab Author: Jarek Potiuk AuthorDate: Sat Nov 25 14:05:23 2023

(airflow) branch optimize-image-wait-verify updated (70d94de75d -> 7b856296ba)

2023-11-25 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch optimize-image-wait-verify in repository https://gitbox.apache.org/repos/asf/airflow.git omit 70d94de75d Optimize CI/PROD image waiting and verification in CI workflow new 7b856296ba

(airflow) branch optimize-image-wait-verify updated (4b63d7c787 -> 70d94de75d)

2023-11-25 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch optimize-image-wait-verify in repository https://gitbox.apache.org/repos/asf/airflow.git omit 4b63d7c787 Optimize CI/PROD image waiting and verification in CI workflow new 70d94de75d

(airflow) 01/01: Optimize CI/PROD image waiting and verification in CI workflow

2023-11-25 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch optimize-image-wait-verify in repository https://gitbox.apache.org/repos/asf/airflow.git commit 70d94de75dd2d7acf39311028d831c29ae0db692 Author: Jarek Potiuk AuthorDate: Sat Nov 25 14:05:23 2023

(airflow) branch optimize-image-wait-verify updated (af556b4b19 -> 4b63d7c787)

2023-11-25 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch optimize-image-wait-verify in repository https://gitbox.apache.org/repos/asf/airflow.git omit af556b4b19 Optimize CI/PROD image waiting and verification in CI workflow new 4b63d7c787

(airflow) 01/01: Optimize CI/PROD image waiting and verification in CI workflow

2023-11-25 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch optimize-image-wait-verify in repository https://gitbox.apache.org/repos/asf/airflow.git commit 4b63d7c787577c225a639edfb71bdcf0ef547bda Author: Jarek Potiuk AuthorDate: Sat Nov 25 14:05:23 2023

(airflow-ci-infra) branch main updated: Bump version of docker/docker-compose for stability

2023-11-25 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow-ci-infra.git The following commit(s) were added to refs/heads/main by this push: new 9e31fb0 Bump version of

(airflow) 01/01: Optimize CI/PROD image waiting and verification in CI workflow

2023-11-25 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch optimize-image-wait-verify in repository https://gitbox.apache.org/repos/asf/airflow.git commit af556b4b199b832374c2bffc76b32cece8fc1683 Author: Jarek Potiuk AuthorDate: Sat Nov 25 14:05:23 2023

(airflow) branch optimize-image-wait-verify updated (5db4097cab -> af556b4b19)

2023-11-25 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch optimize-image-wait-verify in repository https://gitbox.apache.org/repos/asf/airflow.git omit 5db4097cab Optimize CI/PROD image waiting and verification in CI workflow new af556b4b19

(airflow) 01/01: Optimize CI/PROD image waiting and verification in CI workflow

2023-11-25 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch optimize-image-wait-verify in repository https://gitbox.apache.org/repos/asf/airflow.git commit 5db4097cab689402ee47583b9cbc962023791398 Author: Jarek Potiuk AuthorDate: Sat Nov 25 14:05:23 2023

(airflow) branch optimize-image-wait-verify updated (9f206c240d -> 5db4097cab)

2023-11-25 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch optimize-image-wait-verify in repository https://gitbox.apache.org/repos/asf/airflow.git omit 9f206c240d Optimize CI/PROD image waiting and verification in CI workflow new 5db4097cab

(airflow) 01/01: Optimize CI/PROD image waiting and verification in CI workflow

2023-11-25 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch optimize-image-wait-verify in repository https://gitbox.apache.org/repos/asf/airflow.git commit 9f206c240d04c6697ca989a8bb94857a310e23ac Author: Jarek Potiuk AuthorDate: Sat Nov 25 14:05:23 2023

(airflow) branch optimize-image-wait-verify updated (102e1067b9 -> 9f206c240d)

2023-11-25 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch optimize-image-wait-verify in repository https://gitbox.apache.org/repos/asf/airflow.git omit 102e1067b9 Optimize CI/PROD image waiting and verification in CI workflow new 9f206c240d

(airflow) branch optimize-image-wait-verify updated (b4e48f937e -> 102e1067b9)

2023-11-25 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch optimize-image-wait-verify in repository https://gitbox.apache.org/repos/asf/airflow.git omit b4e48f937e Optimize CI/PROD image waiting and verification in CI workflow new 102e1067b9

(airflow) 01/01: Optimize CI/PROD image waiting and verification in CI workflow

2023-11-25 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch optimize-image-wait-verify in repository https://gitbox.apache.org/repos/asf/airflow.git commit 102e1067b93f7c1d8afd77a954dc854acf3eca49 Author: Jarek Potiuk AuthorDate: Sat Nov 25 14:05:23 2023

(airflow) 01/01: Optimize CI/PROD image waiting and verification in CI workflow

2023-11-25 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch optimize-image-wait-verify in repository https://gitbox.apache.org/repos/asf/airflow.git commit b4e48f937e61307dcfa11c94b10a285b4c29f606 Author: Jarek Potiuk AuthorDate: Sat Nov 25 14:05:23 2023

(airflow) branch optimize-image-wait-verify updated (d948d94198 -> b4e48f937e)

2023-11-25 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch optimize-image-wait-verify in repository https://gitbox.apache.org/repos/asf/airflow.git omit d948d94198 Optimize CI/PROD image waiting and verification in CI workflow new b4e48f937e

(airflow) branch optimize-image-wait-verify updated (5f2d019c1a -> d948d94198)

2023-11-25 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch optimize-image-wait-verify in repository https://gitbox.apache.org/repos/asf/airflow.git omit 5f2d019c1a Optimize CI/PROD image waiting and verification in CI workflow add d948d94198

(airflow) branch optimize-image-wait-verify updated (9945408ed4 -> 5f2d019c1a)

2023-11-25 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch optimize-image-wait-verify in repository https://gitbox.apache.org/repos/asf/airflow.git omit 9945408ed4 Optimize CI/PROD image waiting and verification in CI workflow add 5f2d019c1a

(airflow) 01/01: Optimize CI/PROD image waiting and verification in CI workflow

2023-11-25 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch optimize-image-wait-verify in repository https://gitbox.apache.org/repos/asf/airflow.git commit 9945408ed44d89c957fbb4aa7701e0e711c5ee46 Author: Jarek Potiuk AuthorDate: Sat Nov 25 14:05:23 2023

(airflow) branch optimize-image-wait-verify updated (788967ed2f -> 9945408ed4)

2023-11-25 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch optimize-image-wait-verify in repository https://gitbox.apache.org/repos/asf/airflow.git omit 788967ed2f Optimize CI/PROD image waiting and verification in CI workflow new 9945408ed4

(airflow) branch optimize-image-wait-verify updated (7d9fd35079 -> 788967ed2f)

2023-11-25 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch optimize-image-wait-verify in repository https://gitbox.apache.org/repos/asf/airflow.git omit 7d9fd35079 Optimize CI/PROD image waiting and verification in CI workflow add 788967ed2f

Re: [PR] feat: SQLExecuteQueryOperator sql_callable [airflow]

2023-11-25 Thread via GitHub
raphaelauv commented on PR #35844: URL: https://github.com/apache/airflow/pull/35844#issuecomment-1826453641 the "unload" I do ``` def unload_sql(sql_select_query: str, s3_export_path: str) -> str: sql_select_query = sql_select_query.replace("'", "''") return f"""

Re: [PR] feat: SQLExecuteQueryOperator sql_callable [airflow]

2023-11-25 Thread via GitHub
raphaelauv commented on PR #35844: URL: https://github.com/apache/airflow/pull/35844#issuecomment-1826453061 I would like to render values with airflow but since `params` can't be templated and I don't want to put airflow templating directly in the .sql file ( for separation of concern and

(airflow) 01/01: Optimize CI/PROD image waiting and verification in CI workflow

2023-11-25 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch optimize-image-wait-verify in repository https://gitbox.apache.org/repos/asf/airflow.git commit 7d9fd350790dc60a4ced0da7a4f258bc198e22af Author: Jarek Potiuk AuthorDate: Sat Nov 25 14:05:23 2023

(airflow) branch optimize-image-wait-verify updated (1b2f3a1ef4 -> 7d9fd35079)

2023-11-25 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch optimize-image-wait-verify in repository https://gitbox.apache.org/repos/asf/airflow.git omit 1b2f3a1ef4 Optimize CI/PROD image waiting and verification in CI workflow new 7d9fd35079

(airflow) branch optimize-image-wait-verify updated (2c4b326a96 -> 1b2f3a1ef4)

2023-11-25 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch optimize-image-wait-verify in repository https://gitbox.apache.org/repos/asf/airflow.git omit 2c4b326a96 Optimize CI/PROD image waiting and verification in CI workflow add 1b2f3a1ef4

(airflow) branch optimize-image-wait-verify updated (2788d3c150 -> 2c4b326a96)

2023-11-25 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch optimize-image-wait-verify in repository https://gitbox.apache.org/repos/asf/airflow.git omit 2788d3c150 Optimize CI/PROD image waiting and verification in CI workflow add 2c4b326a96

(airflow) branch optimize-image-wait-verify updated (fb0a155599 -> 2788d3c150)

2023-11-25 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch optimize-image-wait-verify in repository https://gitbox.apache.org/repos/asf/airflow.git omit fb0a155599 Optimize CI/PROD image waiting and verification in CI workflow new 2788d3c150

(airflow) 01/01: Optimize CI/PROD image waiting and verification in CI workflow

2023-11-25 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch optimize-image-wait-verify in repository https://gitbox.apache.org/repos/asf/airflow.git commit 2788d3c150bd5d68adbc67e8b12251a08e45619c Author: Jarek Potiuk AuthorDate: Sat Nov 25 14:05:23 2023

Re: [PR] Optimize CI/PROD image waiting and verification in CI workflow [airflow]

2023-11-25 Thread via GitHub
potiuk commented on PR #35856: URL: https://github.com/apache/airflow/pull/35856#issuecomment-1826450644 These diagrams: https://github.com/apache/airflow/blob/optimize-image-wait-verify/CI_DIAGRAMS.md and the https://github.com/apache/airflow/blob/optimize-image-wait-verify/CI.rst are

(airflow) 01/01: Optimize CI/PROD image waiting and verification in CI workflow

2023-11-25 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch optimize-image-wait-verify in repository https://gitbox.apache.org/repos/asf/airflow.git commit fb0a15559994e1ff2357022e8fdbe1974c7c28d0 Author: Jarek Potiuk AuthorDate: Sat Nov 25 14:05:23 2023

(airflow) branch optimize-image-wait-verify updated (5e93483e74 -> fb0a155599)

2023-11-25 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch optimize-image-wait-verify in repository https://gitbox.apache.org/repos/asf/airflow.git omit 5e93483e74 Optimize CI/PROD image waiting and verification in CI workflow new fb0a155599

(airflow) branch optimize-image-wait-verify updated (81e6bbce0a -> 5e93483e74)

2023-11-25 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch optimize-image-wait-verify in repository https://gitbox.apache.org/repos/asf/airflow.git omit 81e6bbce0a Optimize CI/PROD image waiting and verification in CI workflow add 5e93483e74

Re: [PR] Fix multiNamespaceMode not working with executor KubernetesExecutor [airflow]

2023-11-25 Thread via GitHub
github-actions[bot] closed pull request #34505: Fix multiNamespaceMode not working with executor KubernetesExecutor URL: https://github.com/apache/airflow/pull/34505 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

(airflow) branch optimize-image-wait-verify updated (305ee4dc5d -> 81e6bbce0a)

2023-11-25 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch optimize-image-wait-verify in repository https://gitbox.apache.org/repos/asf/airflow.git omit 305ee4dc5d Optimize CI/PROD image waiting and verification in CI workflow add 81e6bbce0a

(airflow) 01/01: Optimize CI/PROD image waiting and verification in CI workflow

2023-11-25 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch optimize-image-wait-verify in repository https://gitbox.apache.org/repos/asf/airflow.git commit 305ee4dc5d02776fd52a4c234f4932ba2acaace3 Author: Jarek Potiuk AuthorDate: Sat Nov 25 14:05:23 2023

(airflow) branch optimize-image-wait-verify updated (111cdbf98d -> 305ee4dc5d)

2023-11-25 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch optimize-image-wait-verify in repository https://gitbox.apache.org/repos/asf/airflow.git omit 111cdbf98d Optimize CI/PROD image waiting and verification in CI workflow new 305ee4dc5d

(airflow) branch optimize-image-wait-verify updated (0d0a5a7688 -> 111cdbf98d)

2023-11-25 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch optimize-image-wait-verify in repository https://gitbox.apache.org/repos/asf/airflow.git omit 0d0a5a7688 Optimize CI/PROD image waiting and verification in CI workflow add 111cdbf98d

[PR] Optimize CI/PROD image waiting and verification in CI workflow [airflow]

2023-11-25 Thread via GitHub
potiuk opened a new pull request, #35856: URL: https://github.com/apache/airflow/pull/35856 Currently both "wait-for-ci-images" and "preview-constraints" jobs are waiting for images to be built - which means that they both take a running worker slot (public runner) just to do the waiting

(airflow) 01/01: Optimize CI/PROD image waiting and verification in CI workflow

2023-11-25 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch optimize-image-wait-verify in repository https://gitbox.apache.org/repos/asf/airflow.git commit 0d0a5a768835b99fd7887dc53637183d9b256de0 Author: Jarek Potiuk AuthorDate: Sat Nov 25 14:05:23 2023

(airflow) branch optimize-image-wait-verify created (now 0d0a5a7688)

2023-11-25 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch optimize-image-wait-verify in repository https://gitbox.apache.org/repos/asf/airflow.git at 0d0a5a7688 Optimize CI/PROD image waiting and verification in CI workflow This branch includes

[PR] Fix vault gcp auth. [airflow]

2023-11-25 Thread via GitHub
fpopic opened a new pull request, #35855: URL: https://github.com/apache/airflow/pull/35855 --- **^ Add meaningful description above** Read the **[Pull Request

Re: [I] Deferrable ExternalTaskSensor cannot wait for an entire DAG [airflow]

2023-11-25 Thread via GitHub
Throne3d commented on issue #34207: URL: https://github.com/apache/airflow/issues/34207#issuecomment-1826442973 I've opened up https://github.com/apache/airflow/pull/35854 to add a note here on the sensor page -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] Note unsupported ExternalTaskSensor deferrable modes [airflow]

2023-11-25 Thread via GitHub
boring-cyborg[bot] commented on PR #35854: URL: https://github.com/apache/airflow/pull/35854#issuecomment-1826442273 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution

[PR] Note unsupported ExternalTaskSensor deferrable modes [airflow]

2023-11-25 Thread via GitHub
Throne3d opened a new pull request, #35854: URL: https://github.com/apache/airflow/pull/35854 related: #34207 Based on issue #34207 and my own testing, it seems ExternalTaskSensor doesn't work properly with `deferrable=True` when trying to sense a full DAG (i.e. when

Re: [PR] Fix passing dags.gitSync.sshKeySecret with dags.persistence.enabled=true [airflow]

2023-11-25 Thread via GitHub
potiuk commented on PR #34331: URL: https://github.com/apache/airflow/pull/34331#issuecomment-1826437589 I would strongly discourage using both git-sync and persistence. I am very close to propose (I did have a PR in the past about it) such combo should be forbidden because it leads to

Re: [I] Status of testing Providers that were prepared on November 24, 2023 [airflow]

2023-11-25 Thread via GitHub
dashton90 commented on issue #35845: URL: https://github.com/apache/airflow/issues/35845#issuecomment-1826434027 #35790 works -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] On the front-end page, the content is not displayed in the Executor Class field [airflow]

2023-11-25 Thread via GitHub
potiuk commented on issue #35847: URL: https://github.com/apache/airflow/issues/35847#issuecomment-1826432848 Feel free to work on it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] TypedDict for Context should mark Required/NotRequired attributes correctly [airflow]

2023-11-25 Thread via GitHub
ShivamPatel-24 commented on issue #35823: URL: https://github.com/apache/airflow/issues/35823#issuecomment-1826431982 Hello @potiuk, thank you for letting me know! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [I] Deferrable ExternalTaskSensor cannot wait for an entire DAG [airflow]

2023-11-25 Thread via GitHub
potiuk commented on issue #34207: URL: https://github.com/apache/airflow/issues/34207#issuecomment-1826431958 > Hi there! I just hit into this too - would it be possible to add a note that this isn't yet supported in the meantime? Sure - it's marked as "good-first-issue" so anyone

Re: [I] triggerer block_watchdog is crying "wolf!" [airflow]

2023-11-25 Thread via GitHub
potiuk commented on issue #35806: URL: https://github.com/apache/airflow/issues/35806#issuecomment-1826430516 Yes. Very good analysis (thanks), however I believe it has some flawed assumptions. I would love to get some more insights on that one and I have two questions @dlesco

Re: [I] Add path to airflow.cfg in docker-compose.yml [airflow]

2023-11-25 Thread via GitHub
potiuk commented on issue #35812: URL: https://github.com/apache/airflow/issues/35812#issuecomment-1826427680 Yes @svvema -> if you feel like improving the compose file you are absolutely welcome to. Would you like to contribute back in exchange for getting not only free software but also

(airflow) branch main updated: Disable mypy check for referencing package (#35850)

2023-11-25 Thread husseinawala
This is an automated email from the ASF dual-hosted git repository. husseinawala pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 2d58e70b3b Disable mypy check for

Re: [PR] Disable mypy check for referencing package [airflow]

2023-11-25 Thread via GitHub
hussein-awala merged PR #35850: URL: https://github.com/apache/airflow/pull/35850 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

(airflow) branch main updated: Add a new config to configure the host of the scheduler health check server (#35616)

2023-11-25 Thread husseinawala
This is an automated email from the ASF dual-hosted git repository. husseinawala pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 454e63fd66 Add a new config to configure the

Re: [PR] Add a new config to configure the host of the scheduler health check server [airflow]

2023-11-25 Thread via GitHub
hussein-awala merged PR #35616: URL: https://github.com/apache/airflow/pull/35616 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Add a new config to configure the host of the scheduler health check server [airflow]

2023-11-25 Thread via GitHub
hussein-awala commented on PR #35616: URL: https://github.com/apache/airflow/pull/35616#issuecomment-1826412613 Static checks failure is not related to this PR; I will try to fix it in a separate PR. -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] Add support for catchup="ignore_first" to disable catchup for the first DagRun [airflow]

2023-11-25 Thread via GitHub
hussein-awala commented on PR #35392: URL: https://github.com/apache/airflow/pull/35392#issuecomment-1826411539 I updated the dag details endpoint and made it b/c (https://github.com/apache/airflow/pull/35392/commits/510b10e819ffe5b897337c2aa1463376d694850b), and the catchup doc

Re: [PR] feat: SQLExecuteQueryOperator sql_callable [airflow]

2023-11-25 Thread via GitHub
eladkal commented on PR #35844: URL: https://github.com/apache/airflow/pull/35844#issuecomment-1826405918 I am not sure the suggestion simplify things. I am worried that it might cause more confusion. I replied on a similar problem in https://stackoverflow.com/a/72246305/14624409

Re: [PR] Add backward compatibility for elasticsearch<8 [airflow]

2023-11-25 Thread via GitHub
eladkal commented on PR #33281: URL: https://github.com/apache/airflow/pull/33281#issuecomment-1826401956 https://github.com/apache/airflow/pull/35707 raised to remove the functionality added by this PR -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] Add support for catchup="ignore_first" to disable catchup for the first DagRun [airflow]

2023-11-25 Thread via GitHub
eladkal commented on PR #35392: URL: https://github.com/apache/airflow/pull/35392#issuecomment-1826401001 > Bas is not saying catchup false is not intuitive in general, he's talking about the specific behavior that a run is immediately created even when catchup is false. It runs

Re: [PR] Add support for catchup="ignore_first" to disable catchup for the first DagRun [airflow]

2023-11-25 Thread via GitHub
hussein-awala commented on PR #35392: URL: https://github.com/apache/airflow/pull/35392#issuecomment-1826398524 > > > @hussein-awala can you explain what `ignore_first` means, i.e. what the behavior is as compared with `enable`, perhaps with some examples? > > > > > > Yes, I will

Re: [PR] Fix passing dags.gitSync.sshKeySecret with dags.persistence.enabled=true [airflow]

2023-11-25 Thread via GitHub
dolfinus commented on PR #34331: URL: https://github.com/apache/airflow/pull/34331#issuecomment-1826396118 > even though we don't need it Please describe this in more details. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] Add explanation about cleaning logs with helm [airflow]

2023-11-25 Thread via GitHub
eladkal commented on code in PR #35783: URL: https://github.com/apache/airflow/pull/35783#discussion_r1405198620 ## airflow/config_templates/config.yml: ## @@ -658,6 +658,8 @@ logging: description: | Whether the local log files for GCS, S3, WASB and OSS remote

(airflow) branch main updated: Enhance docs for zombie tasks (#35825)

2023-11-25 Thread pankajkoti
This is an automated email from the ASF dual-hosted git repository. pankajkoti pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 177da9016b Enhance docs for zombie tasks

Re: [I] Enhance the docs on zombie tasks to elaborate on how they are detected [airflow]

2023-11-25 Thread via GitHub
pankajkoti closed issue #35698: Enhance the docs on zombie tasks to elaborate on how they are detected URL: https://github.com/apache/airflow/issues/35698 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Enhance docs for zombie tasks [airflow]

2023-11-25 Thread via GitHub
pankajkoti merged PR #35825: URL: https://github.com/apache/airflow/pull/35825 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Add support for service account impersonation with computeEngineSSHHook (google provider) and IAP tunnel [airflow]

2023-11-25 Thread via GitHub
ginolegigot commented on PR #35136: URL: https://github.com/apache/airflow/pull/35136#issuecomment-1826372468 Thanks a lot for your reviews and information. Great day! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Stop logo clashing when pinwheel animates [airflow]

2023-11-25 Thread via GitHub
eladkal commented on PR #35564: URL: https://github.com/apache/airflow/pull/35564#issuecomment-1826372173 > Besides lagging the "Easter Egg" history I very much like the animation and would prefer to keep it. I am not a 100% designed but I treated the text shift as rather "cool" :-D But

Re: [PR] Add expanded-error to default status of dag_import_errors [airflow]

2023-11-25 Thread via GitHub
eladkal commented on PR #35745: URL: https://github.com/apache/airflow/pull/35745#issuecomment-1826371098 > @eladkal Yes, I had thought about that, but speaking honestly I don't have experience with really big deployments of Airflow. What is your experience of using the DAG Import Error UI

Re: [PR] Add expanded-error to default status of dag_import_errors [airflow]

2023-11-25 Thread via GitHub
caitpj commented on PR #35745: URL: https://github.com/apache/airflow/pull/35745#issuecomment-1826355327 > I am not sure this is sutiable for large deployments where there can be dozens of broken dags at a given time. WDYT? @eladkal Yes, I had thought about that, but speaking

Re: [PR] Fix passing dags.gitSync.sshKeySecret with dags.persistence.enabled=true [airflow]

2023-11-25 Thread via GitHub
romsharon98 commented on PR #34331: URL: https://github.com/apache/airflow/pull/34331#issuecomment-1826353980 This pr will solve the problem. But isnt this solution is better? https://github.com/apache/airflow/pull/30896 Because in the current solution we will add git-ssh even though

(airflow) branch main updated: Make livy connection failure test less picky (#35852)

2023-11-25 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 7c2885d21e Make livy connection failure test less

Re: [PR] Make livy connection failure test less picky [airflow]

2023-11-25 Thread via GitHub
potiuk merged PR #35852: URL: https://github.com/apache/airflow/pull/35852 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[PR] Make livy connection failure test less picky [airflow]

2023-11-25 Thread via GitHub
potiuk opened a new pull request, #35852: URL: https://github.com/apache/airflow/pull/35852 The new dependencies released changed slightly the response returned when connection failure occured ("Temporary Failure in Name Resolution" instead of "Host name cannot be found"). This PR

Re: [I] Airflow Web UI: Inconsistency on clearing task between Grid and Graph View [airflow]

2023-11-25 Thread via GitHub
jscheffl closed issue #35849: Airflow Web UI: Inconsistency on clearing task between Grid and Graph View URL: https://github.com/apache/airflow/issues/35849 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [I] Airflow Web UI: Inconsistency on clearing task between Grid and Graph View [airflow]

2023-11-25 Thread via GitHub
jscheffl commented on issue #35849: URL: https://github.com/apache/airflow/issues/35849#issuecomment-1826302108 It seems you are on a older version of Airflow, unfortunately you did not specify in your bug ticket. In the current version the Graph view has been reworked totally, the

[PR] Disable mypy check for referencing package [airflow]

2023-11-25 Thread via GitHub
potiuk opened a new pull request, #35850: URL: https://github.com/apache/airflow/pull/35850 The package uses some ignores that use old MyPy check names and generate syntax error, so we have to disable it to unblock mypy failures in main/new dependencies.

Re: [PR] Add expanded-error to default status of dag_import_errors [airflow]

2023-11-25 Thread via GitHub
eladkal commented on PR #35745: URL: https://github.com/apache/airflow/pull/35745#issuecomment-1826300237 I am not sure this is sutiable for large deployments where there can be dozens of broken dags at a given time. -- This is an automated message from the Apache Git Service. To respond

Re: [I] Add AthenaSqlHook based on PyAthena [airflow]

2023-11-25 Thread via GitHub
flolas commented on issue #34823: URL: https://github.com/apache/airflow/issues/34823#issuecomment-1826298499 Hi @treyyi Are you working on this? I'm really interested in this hook. Happy to help if you need so -- This is an automated message from the Apache Git Service. To respond to

Re: [I] Allow dynamically set `pool_slots` for potentially heavy tasks. [airflow]

2023-11-25 Thread via GitHub
potiuk commented on issue #35803: URL: https://github.com/apache/airflow/issues/35803#issuecomment-1826298106 Just to set expectations here. This is something that is going to be super complex and I am not sure if we are ever going to do that. Scheduler performs selection on which

Re: [PR] Fix `application_file` failure when parsing dictionary in `SparkKubernetesOperator` [airflow]

2023-11-25 Thread via GitHub
eladkal commented on PR #34996: URL: https://github.com/apache/airflow/pull/34996#issuecomment-1826294133 Fixed in https://github.com/apache/airflow/pull/35848 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Fix `application_file` failure when parsing dictionary in `SparkKubernetesOperator` [airflow]

2023-11-25 Thread via GitHub
eladkal closed pull request #34996: Fix `application_file` failure when parsing dictionary in `SparkKubernetesOperator` URL: https://github.com/apache/airflow/pull/34996 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

(airflow) branch main updated: Feature pass dictionary configuration in application_file in SparkKubernetesOperator (#35848)

2023-11-25 Thread husseinawala
This is an automated email from the ASF dual-hosted git repository. husseinawala pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new e57232ff8f Feature pass dictionary

Re: [PR] Feature pass dictionary configuration in application_file in SparkKubernetesOperator [airflow]

2023-11-25 Thread via GitHub
hussein-awala merged PR #35848: URL: https://github.com/apache/airflow/pull/35848 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Refactor and simplify breeze-based pre-commits [airflow]

2023-11-25 Thread via GitHub
potiuk commented on PR #35830: URL: https://github.com/apache/airflow/pull/35830#issuecomment-182629 That is also a good idea as a "learning" exercise for those who will be reviewing it - i will try to turn it into more of a `breeze tutorial` explaining those changes in more detail

Re: [PR] Refactor and simplify breeze-based pre-commits [airflow]

2023-11-25 Thread via GitHub
potiuk commented on PR #35830: URL: https://github.com/apache/airflow/pull/35830#issuecomment-1826289848 Sure - yeah. good idea. I will split it. It grew quite a bit more than initial refactor, but I think I can split it to logical chunks. It started from the idea that it will be a

Re: [PR] Feature pass dictionary configuration in application_file in SparkKubernetesOperator [airflow]

2023-11-25 Thread via GitHub
romsharon98 commented on code in PR #35848: URL: https://github.com/apache/airflow/pull/35848#discussion_r1404843058 ## airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py: ## @@ -59,7 +59,7 @@ class SparkKubernetesOperator(BaseOperator): def __init__(

Re: [PR] Feature pass dictionary configuration in application_file in SparkKubernetesOperator [airflow]

2023-11-25 Thread via GitHub
romsharon98 commented on code in PR #35848: URL: https://github.com/apache/airflow/pull/35848#discussion_r1404843058 ## airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py: ## @@ -59,7 +59,7 @@ class SparkKubernetesOperator(BaseOperator): def __init__(

Re: [I] Airflow StatsD Configuration Error: XML 'ResourceNotFound' Message for statsd_host Environment Variable [airflow]

2023-11-25 Thread via GitHub
jscheffl commented on issue #35817: URL: https://github.com/apache/airflow/issues/35817#issuecomment-1826282503 Okay, then I assume the root cause originally is the error ``` ERROR - Unexpected error: encoding with 'idna' codec failed (UnicodeError: label empty or too long) ```

Re: [PR] Feature pass dictionary configuration in application_file in SparkKubernetesOperator [airflow]

2023-11-25 Thread via GitHub
romsharon98 commented on code in PR #35848: URL: https://github.com/apache/airflow/pull/35848#discussion_r1404843058 ## airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py: ## @@ -59,7 +59,7 @@ class SparkKubernetesOperator(BaseOperator): def __init__(

  1   2   >