Re: [PR] AIP-59-performance-dags [airflow]

2024-09-03 Thread via GitHub
potiuk commented on code in PR #41961: URL: https://github.com/apache/airflow/pull/41961#discussion_r1741529552 ## performance/src/performance_dags/elastic_dag/elastic_dag.py: ## @@ -0,0 +1,322 @@ +""" +Elastic dag copied from apache/airflow master and adjusted to work on python

Re: [PR] AIP-59-performance-dags [airflow]

2024-09-03 Thread via GitHub
potiuk commented on code in PR #41961: URL: https://github.com/apache/airflow/pull/41961#discussion_r1741529923 ## performance/src/performance_dags/elastic_dag/elastic_dag.py: ## @@ -0,0 +1,322 @@ +""" +Elastic dag copied from apache/airflow master and adjusted to work on python

Re: [PR] AIP-59-performance-dags [airflow]

2024-09-03 Thread via GitHub
potiuk commented on code in PR #41961: URL: https://github.com/apache/airflow/pull/41961#discussion_r1741532812 ## performance/src/performance_dags/elastic_dag/elastic_dag.py: ## @@ -0,0 +1,322 @@ +""" +Elastic dag copied from apache/airflow master and adjusted to work on python

Re: [PR] AIP-59-performance-dags [airflow]

2024-09-03 Thread via GitHub
potiuk commented on PR #41961: URL: https://github.com/apache/airflow/pull/41961#issuecomment-2325754744 I thnk it also needs a bit cleanup for outdated stuff - generally modernising the code and see what can be reused from plain Airflow 2 perspective. Also possibly we should renama

Re: [PR] Add triggered_by field to DAG Run model to distinguish the source of a trigger [airflow]

2024-09-03 Thread via GitHub
potiuk commented on PR #39165: URL: https://github.com/apache/airflow/pull/39165#issuecomment-2325801888 I attempted to solve conflict - but It needs more static check fixes @molcay -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

[I] Infinite retries when calling dag.test() [airflow]

2024-09-03 Thread via GitHub
miroslav-trifonov opened a new issue, #41972: URL: https://github.com/apache/airflow/issues/41972 ### Apache Airflow version Other Airflow 2 version (please specify below) ### If "Other Airflow 2 version" selected, which one? 2.7.2 ### What happened? We are

Re: [I] Infinite retries when calling dag.test() [airflow]

2024-09-03 Thread via GitHub
boring-cyborg[bot] commented on issue #41972: URL: https://github.com/apache/airflow/issues/41972#issuecomment-2325805020 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for ap

Re: [PR] UI API Add views tests [airflow]

2024-09-03 Thread via GitHub
pierrejeambrun commented on PR #41957: URL: https://github.com/apache/airflow/pull/41957#issuecomment-2325806974 [bugraoz93](https://github.com/bugraoz93) I would that the initial idea is to just migrate existing endpoints to the new API (FastAPI), modifying only endpoints if necessary so w

Re: [PR] Generalize caching of connection in DbApiHook to improve performance [airflow]

2024-09-03 Thread via GitHub
dabla commented on PR #40751: URL: https://github.com/apache/airflow/pull/40751#issuecomment-2325846487 > Hey @dabla -> you need to also upgrade `common.sql` version in their provider.yaml - otherwise PROD image build is not able to find common.sql >= 1.17.0.dev0 > > The build is sma

Re: [PR] UI API Add views tests [airflow]

2024-09-03 Thread via GitHub
potiuk commented on PR #41957: URL: https://github.com/apache/airflow/pull/41957#issuecomment-2325847567 > I'll write the documentation down, If we are happy with it and agree with the general strategy I can present it to the rest of the community. (This thurday if I manage to do it before

Re: [PR] UI API Add views tests [airflow]

2024-09-03 Thread via GitHub
potiuk commented on PR #41957: URL: https://github.com/apache/airflow/pull/41957#issuecomment-2325848950 And even if we have no write-up - we can just intiate the discussion and outline the idea this Thursday -- This is an automated message from the Apache Git Service. To respond to the m

Re: [PR] Introduce notion of dialects in DbApiHook [airflow]

2024-09-03 Thread via GitHub
dabla commented on PR #41327: URL: https://github.com/apache/airflow/pull/41327#issuecomment-2325860567 @eladkal and @potiuk what do you think about this implementation? I would ideally want to use entry points to register the dialects, so that additional dialects can be loaded through dif

Re: [PR] Generalize caching of connection in DbApiHook to improve performance [airflow]

2024-09-03 Thread via GitHub
dabla commented on PR #40751: URL: https://github.com/apache/airflow/pull/40751#issuecomment-2325870292 > Hey @dabla -> you need to also upgrade `common.sql` version in their provider.yaml - otherwise PROD image build is not able to find common.sql >= 1.17.0.dev0 > > The build is sma

Re: [PR] [FEAT] databricks repair run with reason match and appropriate new settings [airflow]

2024-09-03 Thread via GitHub
gaurav7261 commented on PR #41412: URL: https://github.com/apache/airflow/pull/41412#issuecomment-2325875691 @Lee-W currently we are checking in root state_message only, should we also check `error` key in every fail task array as well ? -- This is an automated message from the Apache Gi

Re: [PR] Generalize caching of connection in DbApiHook to improve performance [airflow]

2024-09-03 Thread via GitHub
potiuk commented on PR #40751: URL: https://github.com/apache/airflow/pull/40751#issuecomment-2325882466 Just add version in the provider.yaml. all the rest should happen by pre-commits You might want to add a line at top of changelog explaining better what was added in common.sql (

Re: [PR] Move projects metadata for SBOM export to Google Spreadsheet [airflow]

2024-09-03 Thread via GitHub
pierrejeambrun commented on code in PR #41930: URL: https://github.com/apache/airflow/pull/41930#discussion_r1741637062 ## dev/breeze/src/airflow_breeze/utils/projects_google_spreadsheet.py: ## @@ -0,0 +1,253 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or

Re: [I] 'CeleryKubernetesExecutor' object has no attribute '_task_event_logs' [airflow]

2024-09-03 Thread via GitHub
hadarsharon commented on issue #41891: URL: https://github.com/apache/airflow/issues/41891#issuecomment-2325901918 I just checked and can confirm that this issue is resolved in [v2.10.1.rc1](https://pypi.python.org/pypi/apache-airflow/2.10.1.rc1) Cheers -- This is an automated mess

Re: [I] Status of testing of Apache Airflow 2.10.1rc1 [airflow]

2024-09-03 Thread via GitHub
hadarsharon commented on issue #41956: URL: https://github.com/apache/airflow/issues/41956#issuecomment-2325901266 Yes @potiuk I just checked and can confirm that #41891 is resolved in [v2.10.1.rc1](https://pypi.python.org/pypi/apache-airflow/2.10.1.rc1) Cheers -- This is an automa

Re: [PR] Move projects metadata for SBOM export to Google Spreadsheet [airflow]

2024-09-03 Thread via GitHub
pierrejeambrun commented on code in PR #41930: URL: https://github.com/apache/airflow/pull/41930#discussion_r1741646923 ## dev/breeze/src/airflow_breeze/utils/projects_google_spreadsheet.py: ## @@ -0,0 +1,253 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or

Re: [PR] Move projects metadata for SBOM export to Google Spreadsheet [airflow]

2024-09-03 Thread via GitHub
pierrejeambrun commented on code in PR #41930: URL: https://github.com/apache/airflow/pull/41930#discussion_r1741646923 ## dev/breeze/src/airflow_breeze/utils/projects_google_spreadsheet.py: ## @@ -0,0 +1,253 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or

Re: [PR] UI API Add views tests [airflow]

2024-09-03 Thread via GitHub
pierrejeambrun merged PR #41957: URL: https://github.com/apache/airflow/pull/41957 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@a

(airflow) branch main updated: Add views tests (#41957)

2024-09-03 Thread pierrejeambrun
This is an automated email from the ASF dual-hosted git repository. pierrejeambrun pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new f84d16e6fc Add views tests (#41957) f84d16e

Re: [PR] Move projects metadata for SBOM export to Google Spreadsheet [airflow]

2024-09-03 Thread via GitHub
potiuk commented on code in PR #41930: URL: https://github.com/apache/airflow/pull/41930#discussion_r1741657117 ## dev/breeze/src/airflow_breeze/utils/projects_google_spreadsheet.py: ## @@ -0,0 +1,253 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more co

Re: [PR] Add triggered_by field to DAG Run model to distinguish the source of a trigger [airflow]

2024-09-03 Thread via GitHub
molcay commented on PR #39165: URL: https://github.com/apache/airflow/pull/39165#issuecomment-2325918859 Hi @potiuk, Thank you for the attempt. I will have a look ASAP -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Move projects metadata for SBOM export to Google Spreadsheet [airflow]

2024-09-03 Thread via GitHub
potiuk commented on code in PR #41930: URL: https://github.com/apache/airflow/pull/41930#discussion_r1741661126 ## dev/breeze/src/airflow_breeze/utils/projects_google_spreadsheet.py: ## @@ -0,0 +1,253 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more co

Re: [PR] Move projects metadata for SBOM export to Google Spreadsheet [airflow]

2024-09-03 Thread via GitHub
potiuk commented on PR #41930: URL: https://github.com/apache/airflow/pull/41930#issuecomment-2325928503 > Glad to see SBOMs are getting some great improvement Indeed. With thos changes we are getting to the next level - from just "reporting as things are" to "makins some actionable s

Re: [PR] Generalize caching of connection in DbApiHook to improve performance [airflow]

2024-09-03 Thread via GitHub
dabla commented on PR #40751: URL: https://github.com/apache/airflow/pull/40751#issuecomment-2325957958 > Just add version in the provider.yaml. all the rest should happen by pre-commits > > You might want to add a line at top of changelog explaining better what was added in common.s

Re: [PR] Better map_index and downstream/upstream tasks handling [airflow]

2024-09-03 Thread via GitHub
uranusjr commented on code in PR #41384: URL: https://github.com/apache/airflow/pull/41384#discussion_r1741682178 ## tests/models/test_taskinstance.py: ## @@ -5312,3 +5312,65 @@ def test_swallow_mini_scheduler_exceptions(_schedule_downstream_mock, create_tas ti.schedule_do

Re: [PR] Generalize caching of connection in DbApiHook to improve performance [airflow]

2024-09-03 Thread via GitHub
potiuk commented on PR #40751: URL: https://github.com/apache/airflow/pull/40751#issuecomment-2325977332 > Nice, ok thank you @potiuk , wanted to make sure before bumping the version, will add a comment line in top of file explaining what was changed, always better to document anyway and go

Re: [PR] Generalize caching of connection in DbApiHook to improve performance [airflow]

2024-09-03 Thread via GitHub
dabla commented on PR #40751: URL: https://github.com/apache/airflow/pull/40751#issuecomment-2325986475 > > Nice, ok thank you @potiuk , wanted to make sure before bumping the version, will add a comment line in top of file explaining what was changed, always better to document anyway and g

Re: [PR] Move projects metadata for SBOM export to Google Spreadsheet [airflow]

2024-09-03 Thread via GitHub
potiuk merged PR #41930: URL: https://github.com/apache/airflow/pull/41930 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

(airflow) branch main updated (f84d16e6fc -> c3cabc1305)

2024-09-03 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from f84d16e6fc Add views tests (#41957) add c3cabc1305 Move projects metadata for SBOM export to Google Spreadsheet (

Re: [PR] Enable workload identity authentication for the Databricks provider [airflow]

2024-09-03 Thread via GitHub
BasPH commented on PR #41639: URL: https://github.com/apache/airflow/pull/41639#issuecomment-2326020217 Looks okay to me, although there's quite a lot of code duplication between the `_get_aad_token_for_default_az_credential` and `_a _get_aad_token_for_default_az_credential` methods. Any wa

Re: [I] glob syntax can be similar to .gitignore in .airflowignore files [airflow]

2024-09-03 Thread via GitHub
poorvirohidekar commented on issue #40672: URL: https://github.com/apache/airflow/issues/40672#issuecomment-2326027591 @YuvaPhani27 - Yes, I did some analysis and will test my changes in the next day or two. -- This is an automated message from the Apache Git Service. To respond to the m

Re: [PR] Generalize caching of connection in DbApiHook to improve performance [airflow]

2024-09-03 Thread via GitHub
potiuk commented on PR #40751: URL: https://github.com/apache/airflow/pull/40751#issuecomment-2326034038 > If it causes issues, I can revert it as I added this in a dedicated commit. We will see and RM can modify it on-line. How it works is that it simply updates the changelog locally

[PR] V3.1 [airflow]

2024-09-03 Thread via GitHub
legmac opened a new pull request, #41973: URL: https://github.com/apache/airflow/pull/41973 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pul

Re: [PR] V3.1 [airflow]

2024-09-03 Thread via GitHub
boring-cyborg[bot] commented on PR #41973: URL: https://github.com/apache/airflow/pull/41973#issuecomment-2326116267 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors'

Re: [PR] V3.1 [airflow]

2024-09-03 Thread via GitHub
hussein-awala closed pull request #41973: V3.1 URL: https://github.com/apache/airflow/pull/41973 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits

Re: [PR] V3.1 [airflow]

2024-09-03 Thread via GitHub
hussein-awala commented on PR #41973: URL: https://github.com/apache/airflow/pull/41973#issuecomment-2326127962 If you're trying to test something, you can create a PR in your fork. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

[PR] fix key permissions and ownership [airflow]

2024-09-03 Thread via GitHub
flypenguin opened a new pull request, #41974: URL: https://github.com/apache/airflow/pull/41974 Using the **helm chart** I was unable to get SSH-based git sync to run. Debugging lead to two issues: - The secret is mounted as `root`, with permissions `0660`, which makes it unreadable

Re: [PR] fix key permissions and ownership [airflow]

2024-09-03 Thread via GitHub
boring-cyborg[bot] commented on PR #41974: URL: https://github.com/apache/airflow/pull/41974#issuecomment-2326129484 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors'

Re: [PR] Introduce notion of dialects in DbApiHook [airflow]

2024-09-03 Thread via GitHub
potiuk commented on PR #41327: URL: https://github.com/apache/airflow/pull/41327#issuecomment-2326136008 We actually already use entrypoints - the provider.yaml "subset" is already exposed in providers and various provider's capabilities are available this way. They are even automatically e

Re: [PR] Allow custom api versions in MSGraphAsyncOperator [airflow]

2024-09-03 Thread via GitHub
dabla commented on PR #41331: URL: https://github.com/apache/airflow/pull/41331#issuecomment-2326159383 > You will need to make some tests db -tests and apparently `power-bi` connection is not created in older versions of airlfow (why would they be - so some test compat code should be added

Re: [PR] Allow custom api versions in MSGraphAsyncOperator [airflow]

2024-09-03 Thread via GitHub
dabla commented on PR #41331: URL: https://github.com/apache/airflow/pull/41331#issuecomment-2326161561 @potiuk I would also like to have this PR merged for the next providers release as this fixes also fixes proxy issue with PowerBIDatasetRefreshOperator and allows custom api versions to b

Re: [PR] Configure HttpHook's auth_type from Connection [airflow]

2024-09-03 Thread via GitHub
dabla commented on PR #35591: URL: https://github.com/apache/airflow/pull/35591#issuecomment-2326207394 @potiuk Maybe we can merge this one for Airflow 3.0.0 ? Ofc dunno what the implications will be regarding UI as it will not be FAB anymore, but I'm willing to help there also if needed.

Re: [PR] fix: rm `skip_if` and `run_if` in python source [airflow]

2024-09-03 Thread via GitHub
phi-friday commented on code in PR #41832: URL: https://github.com/apache/airflow/pull/41832#discussion_r1741866840 ## tests/utils/test_decorators.py: ## @@ -0,0 +1,41 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. Se

Re: [I] Airflow DAG run missing on few days [airflow]

2024-09-03 Thread via GitHub
nathadfield commented on issue #41833: URL: https://github.com/apache/airflow/issues/41833#issuecomment-2326253115 Without a reproducible example or, at the very least more information and context, I doubt you are going to get much help on this one because this is not a known problem or any

[PR] Metrics basic deprecated validators removed in favor of pattern matching [airflow]

2024-09-03 Thread via GitHub
dirrao opened a new pull request, #41975: URL: https://github.com/apache/airflow/pull/41975 Metrics basic deprecated validators (AllowListValidator and BlockListValidator) removed in favor of pattern matching -- This is an automated message from the Apache Git Service. To respond to t

[I] DAG import timeouts recorded in Airflow Database and displayed in Airflow UI [airflow]

2024-09-03 Thread via GitHub
VladaZakharova opened a new issue, #41976: URL: https://github.com/apache/airflow/issues/41976 ### Description The following DAG: ``` import datetime import time import airflow from airflow.operators.python import PythonOperator time.sleep(1000) with ai

Re: [I] DAG import timeouts recorded in Airflow Database and displayed in Airflow UI [airflow]

2024-09-03 Thread via GitHub
VladaZakharova commented on issue #41976: URL: https://github.com/apache/airflow/issues/41976#issuecomment-2326283548 @potiuk @uranusjr @Taragolis Hi all! This is another ticket for improving our Airflow UI :) Need your advice before implementing this, do you have any ideas? Thanks!

Re: [I] DAG import timeouts recorded in Airflow Database and displayed in Airflow UI [airflow]

2024-09-03 Thread via GitHub
potiuk commented on issue #41976: URL: https://github.com/apache/airflow/issues/41976#issuecomment-2326309995 Yeah - since we have `ParseImportError` -> it should likely be passed same way. But maybe it already is? I know there is a problem with visibility of ImportErrors when DAG has acces

[I] Triggered DAG button looks like disabled with TriggerDagRunOperator [airflow]

2024-09-03 Thread via GitHub
vatsrahul1001 opened a new issue, #41977: URL: https://github.com/apache/airflow/issues/41977 ### Apache Airflow version main (development) ### If "Other Airflow 2 version" selected, which one? 2.10.1.RC1 ### What happened? Noticed Triggered DAG button looks

Re: [PR] Introduce notion of dialects in DbApiHook [airflow]

2024-09-03 Thread via GitHub
dabla commented on PR #41327: URL: https://github.com/apache/airflow/pull/41327#issuecomment-2326411953 > We actually already use entrypoints - the `provider.yaml` "subset" is already exposed in providers and various provider's capabilities are available this way. They are even automaticall

Re: [I] Status of testing of Apache Airflow 2.10.1rc1 [airflow]

2024-09-03 Thread via GitHub
vatsrahul1001 commented on issue #41956: URL: https://github.com/apache/airflow/issues/41956#issuecomment-2326419319 Noticed [issue](https://github.com/apache/airflow/issues/41977) while using TriggerDagRunOperator -- This is an automated message from the Apache Git Service. To respond to

Re: [I] Triggered DAG button looks like disabled with TriggerDagRunOperator [airflow]

2024-09-03 Thread via GitHub
raphaelauv commented on issue #41977: URL: https://github.com/apache/airflow/issues/41977#issuecomment-2326428418 I have the problem with 2.10.0 and it was also the case in 2.9.X ![image](https://github.com/user-attachments/assets/370027ee-3526-4f97-9ac6-ca57da62522e) -- This i

Re: [I] DAG import timeouts recorded in Airflow Database and displayed in Airflow UI [airflow]

2024-09-03 Thread via GitHub
VladaZakharova commented on issue #41976: URL: https://github.com/apache/airflow/issues/41976#issuecomment-2326432524 Yes, looks very similar. But i think that problem cannot be resolved easily (for some reasons haha), so i can try to at least catch this kind of errors with AirflowTaskTimeo

Re: [I] Triggered DAG button looks like disabled with TriggerDagRunOperator [airflow]

2024-09-03 Thread via GitHub
vatsrahul1001 commented on issue #41977: URL: https://github.com/apache/airflow/issues/41977#issuecomment-2326437493 I did not notice this issue in our deployment with 2.10.0 ![image](https://github.com/user-attachments/assets/1a0cc252-4683-4a21-869e-35f6d805d7a7) -- This is an aut

Re: [I] DAG import timeouts recorded in Airflow Database and displayed in Airflow UI [airflow]

2024-09-03 Thread via GitHub
potiuk commented on issue #41976: URL: https://github.com/apache/airflow/issues/41976#issuecomment-2326531289 > Yes, looks very similar. But i think that problem cannot be resolved easily (for some reasons haha), so i can try to at least catch this kind of errors with AirflowTaskTimeout and

Re: [PR] Introduce notion of dialects in DbApiHook [airflow]

2024-09-03 Thread via GitHub
potiuk commented on PR #41327: URL: https://github.com/apache/airflow/pull/41327#issuecomment-2326559107 I think in mssql provider, we do have a few "loosely related" things in providers already and cross-provider dependencies (both explicit and implicit) and sometimes where things are "be

Re: [I] Status of testing of Apache Airflow 2.10.1rc1 [airflow]

2024-09-03 Thread via GitHub
joaopamaral commented on issue #41956: URL: https://github.com/apache/airflow/issues/41956#issuecomment-2326596343 Tested https://github.com/apache/airflow/pull/41809 with the following FAB version: - 1.2.2: permissions are synced correctly with the old access control format. - 1.3.0:

Re: [PR] Correct scheduled slots documentation and missing open telemetry span [airflow]

2024-09-03 Thread via GitHub
potiuk commented on PR #41899: URL: https://github.com/apache/airflow/pull/41899#issuecomment-2326603613 cc: @howardyoo @ferruzzi ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [I] Exceptions become a mess in logs using a Python 3.11 docker operator [airflow]

2024-09-03 Thread via GitHub
notatallshaw-gts commented on issue #33692: URL: https://github.com/apache/airflow/issues/33692#issuecomment-2326652700 Still an issue, I just have yet not managed to prioritize raising a PR 🙁 -- This is an automated message from the Apache Git Service. To respond to the message, please l

Re: [I] Exceptions become a mess in logs using a Python 3.11 docker operator [airflow]

2024-09-03 Thread via GitHub
potiuk commented on issue #33692: URL: https://github.com/apache/airflow/issues/33692#issuecomment-2326677110 Maybe someone can step up :) ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [I] Triggered DAG button looks like disabled with TriggerDagRunOperator [airflow]

2024-09-03 Thread via GitHub
ephraimbuddy commented on issue #41977: URL: https://github.com/apache/airflow/issues/41977#issuecomment-2326683152 > I have the problem with 2.10.0 and it was also the case in 2.9.X > > ![image](https://private-user-images.githubusercontent.com/10202690/363986114-c98c16ad-2a0c-4721-b

Re: [I] Airflow DAG run missing on few days [airflow]

2024-09-03 Thread via GitHub
potiuk closed issue #41833: Airflow DAG run missing on few days URL: https://github.com/apache/airflow/issues/41833 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscri

Re: [PR] Introduce notion of dialects in DbApiHook [airflow]

2024-09-03 Thread via GitHub
dabla commented on PR #41327: URL: https://github.com/apache/airflow/pull/41327#issuecomment-2326753141 > I think in mssql provider, we do have a few "loosely related" things in providers already and cross-provider dependencies (both explicit and implicit) and sometimes where things are "be

Re: [I] DAG import timeouts recorded in Airflow Database and displayed in Airflow UI [airflow]

2024-09-03 Thread via GitHub
VladaZakharova commented on issue #41976: URL: https://github.com/apache/airflow/issues/41976#issuecomment-2326781843 Actually, as i have checked my problem is a little bit different from what you have mentioned: in the #29897 i can see that for Admin user ImportError can be visible on UI (

Re: [I] DAG import timeouts recorded in Airflow Database and displayed in Airflow UI [airflow]

2024-09-03 Thread via GitHub
potiuk commented on issue #41976: URL: https://github.com/apache/airflow/issues/41976#issuecomment-2326821753 Yeah - so likely creating a ParseImportError speciffically when there is a timeout should help :) -- This is an automated message from the Apache Git Service. To respond to the me

Re: [PR] Allow custom api versions in MSGraphAsyncOperator [airflow]

2024-09-03 Thread via GitHub
potiuk commented on PR #41331: URL: https://github.com/apache/airflow/pull/41331#issuecomment-2326824353 Tests failing though -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[I] DagBag import timeout errors not propagating to Airflow UI [airflow]

2024-09-03 Thread via GitHub
RNHTTR opened a new issue, #41979: URL: https://github.com/apache/airflow/issues/41979 ### Apache Airflow version 2.10.0 ### If "Other Airflow 2 version" selected, which one? _No response_ ### What happened? When top-level DAG code causes a DagBag import tim

Re: [I] Templates: macros datetime.utcnow and datetime.utcfromtimestamp are deprecated in Python 3.12 [airflow]

2024-09-03 Thread via GitHub
iamtodor-fb commented on issue #41772: URL: https://github.com/apache/airflow/issues/41772#issuecomment-232686 @potiuk seems like my mentioned work-around didn't work for me cause bigquery adjusts the timezone, so the timezone has to be explicitly provided. We managed to provide it as f

Re: [PR] Implement `on_killed` support for tasks during DAG run timeout [airflow]

2024-09-03 Thread via GitHub
MRLab12 closed pull request #41627: Implement `on_killed` support for tasks during DAG run timeout URL: https://github.com/apache/airflow/pull/41627 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [PR] Add kerberos related connection fields(principal, keytab) on SparkSubmitHook [airflow]

2024-09-03 Thread via GitHub
seyoon-lim commented on code in PR #40757: URL: https://github.com/apache/airflow/pull/40757#discussion_r1742312400 ## airflow/providers/apache/spark/hooks/spark_submit.py: ## @@ -282,10 +313,57 @@ def _resolve_connection(self) -> dict[str, Any]: def get_conn(self) -> Any:

Re: [PR] Add kerberos related connection fields(principal, keytab) on SparkSubmitHook [airflow]

2024-09-03 Thread via GitHub
seyoon-lim commented on code in PR #40757: URL: https://github.com/apache/airflow/pull/40757#discussion_r1742312863 ## airflow/providers/apache/spark/hooks/spark_submit.py: ## @@ -282,10 +313,57 @@ def _resolve_connection(self) -> dict[str, Any]: def get_conn(self) -> Any:

(airflow) branch constraints-main updated: Updating constraints. Github run id:10683983372

2024-09-03 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new 721178c811 Updating con

Re: [PR] Warn on mini scheduler failures instead of debug [airflow]

2024-09-03 Thread via GitHub
dstandish commented on PR #39760: URL: https://github.com/apache/airflow/pull/39760#issuecomment-2326957871 related https://github.com/apache/airflow/pull/38914 https://github.com/apache/airflow/pull/39745 -- This is an automated message from the Apache Git Service. To respond t

Re: [I] Dynamically Mapped Tasks: DB performance issues [airflow]

2024-09-03 Thread via GitHub
dstandish commented on issue #39680: URL: https://github.com/apache/airflow/issues/39680#issuecomment-2326958977 related https://github.com/apache/airflow/pull/38914 https://github.com/apache/airflow/pull/39745 -- This is an automated message from the Apache Git Service. To respo

Re: [PR] Use SKIP LOCKED instead of NOWAIT in mini scheduler [airflow]

2024-09-03 Thread via GitHub
dstandish commented on PR #39745: URL: https://github.com/apache/airflow/pull/39745#issuecomment-2326961978 related https://github.com/apache/airflow/pull/38914 https://github.com/apache/airflow/pull/39760 https://github.com/apache/airflow/issues/39680 https://github.com/apache/ai

Re: [I] Rerunning a dag in Airflow 2.7.3 causes a missing table issue [airflow]

2024-09-03 Thread via GitHub
jeremiahishere commented on issue #41894: URL: https://github.com/apache/airflow/issues/41894#issuecomment-2326974576 It is failing in test for us. Production is not failing because the dag execution dates are based on the current time. It started failing when upgrading from airflow 2.6 t

Re: [I] Rerunning a dag in Airflow 2.7.3 causes a missing table issue [airflow]

2024-09-03 Thread via GitHub
jeremiahishere commented on issue #41894: URL: https://github.com/apache/airflow/issues/41894#issuecomment-2326995284 Is there an official way to clean the database between tests? I found this cleanup dag: https://github.com/apache/airflow/blob/v2-10-stable/airflow/utils/db_cleanup.py. I

Re: [I] DagBag import timeout errors not propagating to Airflow UI [airflow]

2024-09-03 Thread via GitHub
potiuk commented on issue #41979: URL: https://github.com/apache/airflow/issues/41979#issuecomment-2327026545 Duplicate of https://github.com/apache/airflow/issues/41976 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [I] DagBag import timeout errors not propagating to Airflow UI [airflow]

2024-09-03 Thread via GitHub
potiuk closed issue #41979: DagBag import timeout errors not propagating to Airflow UI URL: https://github.com/apache/airflow/issues/41979 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [I] DagBag import timeout errors not propagating to Airflow UI [airflow]

2024-09-03 Thread via GitHub
potiuk commented on issue #41979: URL: https://github.com/apache/airflow/issues/41979#issuecomment-2327026973 Closing as duplicate -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [I] Rerunning a dag in Airflow 2.7.3 causes a missing table issue [airflow]

2024-09-03 Thread via GitHub
potiuk commented on issue #41894: URL: https://github.com/apache/airflow/issues/41894#issuecomment-2327051114 I do not know too much about the root cause, but let me post some things that might help you to straighten things out: * 'airflow db reset` will reset the database. I believe

Re: [I] Rerunning a dag in Airflow 2.7.3 causes a missing table issue [airflow]

2024-09-03 Thread via GitHub
potiuk commented on issue #41894: URL: https://github.com/apache/airflow/issues/41894#issuecomment-2327052731 Also converting it into discussion. This is not "airflow issue" per se - I doubt we will be able to make a PR from it that will `fix` something - it's more of possibly troubleshooti

Re: [I] Rerunning a dag in Airflow 2.7.3 causes a missing table issue [airflow]

2024-09-03 Thread via GitHub
potiuk closed issue #41894: Rerunning a dag in Airflow 2.7.3 causes a missing table issue URL: https://github.com/apache/airflow/issues/41894 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [PR] Add kerberos related connection fields(principal, keytab) on SparkSubmitHook [airflow]

2024-09-03 Thread via GitHub
nevcohen commented on code in PR #40757: URL: https://github.com/apache/airflow/pull/40757#discussion_r1742450126 ## airflow/providers/apache/spark/hooks/spark_submit.py: ## @@ -282,10 +313,57 @@ def _resolve_connection(self) -> dict[str, Any]: def get_conn(self) -> Any:

Re: [PR] Add kerberos related connection fields(principal, keytab) on SparkSubmitHook [airflow]

2024-09-03 Thread via GitHub
nevcohen commented on code in PR #40757: URL: https://github.com/apache/airflow/pull/40757#discussion_r1742450532 ## airflow/providers/apache/spark/hooks/spark_submit.py: ## @@ -282,10 +313,57 @@ def _resolve_connection(self) -> dict[str, Any]: def get_conn(self) -> Any:

Re: [PR] Correct scheduled slots documentation and missing open telemetry span [airflow]

2024-09-03 Thread via GitHub
howardyoo commented on PR #41899: URL: https://github.com/apache/airflow/pull/41899#issuecomment-2327173389 > cc: @howardyoo @ferruzzi ? The change looks good to me. Thanks for adding these missing attributes to the span :-) -- This is an automated message from the Apache Git Serv

Re: [PR] Add support for semicolon stripping to DbApiHook, PrestoHook, and TrinoHook [airflow]

2024-09-03 Thread via GitHub
Illumaria commented on PR #41916: URL: https://github.com/apache/airflow/pull/41916#issuecomment-2327205062 Whoops... Sorry. 🙏🏻 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

[I] Xcom view broken for non-JSON values [airflow]

2024-09-03 Thread via GitHub
jkramer-ginkgo opened a new issue, #41981: URL: https://github.com/apache/airflow/issues/41981 ### Apache Airflow version 2.10.0 ### If "Other Airflow 2 version" selected, which one? _No response_ ### What happened? Non-JSON values are broken in the Xcom UI

Re: [PR] Kubernetes Pod Operator: check pod completion reason if completed during deffered logging interval [airflow]

2024-09-03 Thread via GitHub
johnhoran commented on PR #41868: URL: https://github.com/apache/airflow/pull/41868#issuecomment-2327219191 > Can you please fix the failing tests? @RNHTTR all resolved now, I've also added a test case which would have failed if the failure status wasn't handled. -- This is an

Re: [PR] Metrics basic deprecated validators removed in favor of pattern matching [airflow]

2024-09-03 Thread via GitHub
ferruzzi commented on code in PR #41975: URL: https://github.com/apache/airflow/pull/41975#discussion_r1742570262 ## airflow/metrics/validators.py: ## @@ -118,7 +106,7 @@ def get_validator() -> ListValidator: else: list_type = DEFAULT_VALIDATOR_TYPE -return v

Re: [PR] Skip database isolation case for task mapping taskinstance tests (#41954) [airflow]

2024-09-03 Thread via GitHub
potiuk merged PR #41954: URL: https://github.com/apache/airflow/pull/41954 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

(airflow) branch v2-10-test updated: Skip database isolation case for task mapping taskinstance tests (#41954) (#41954)

2024-09-03 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch v2-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/v2-10-test by this push: new 7f49cfc06b Skip database isolation case

Re: [PR] Metrics basic deprecated validators removed in favor of pattern matching [airflow]

2024-09-03 Thread via GitHub
ferruzzi commented on code in PR #41975: URL: https://github.com/apache/airflow/pull/41975#discussion_r1742580638 ## tests/core/test_stats.py: ## @@ -345,76 +326,48 @@ def test_regex_matches(self, match_pattern, expect_incr): statsd_client.assert_not_called() -c

Re: [PR] Correct scheduled slots documentation and missing open telemetry span [airflow]

2024-09-03 Thread via GitHub
ferruzzi merged PR #41899: URL: https://github.com/apache/airflow/pull/41899 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow

(airflow) branch main updated: Correct scheduled slots documentation and missing open telemetry span (#41899)

2024-09-03 Thread ferruzzi
This is an automated email from the ASF dual-hosted git repository. ferruzzi pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 0a816c6f0b Correct scheduled slots documentation

Re: [PR] Correct scheduled slots documentation and missing open telemetry span [airflow]

2024-09-03 Thread via GitHub
boring-cyborg[bot] commented on PR #41899: URL: https://github.com/apache/airflow/pull/41899#issuecomment-2327285987 Awesome work, congrats on your first merged pull request! You are invited to check our [Issue Tracker](https://github.com/apache/airflow/issues) for additional contributions.

(airflow) branch constraints-2-10 updated: Updating constraints. Github run id:10689463297

2024-09-03 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-2-10 in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-2-10 by this push: new ef84517159 Updating con

  1   2   >