Re: [PR] Send explicit task logs when marking tasks stuck in queued as failed [airflow]

2023-11-26 Thread via GitHub
pankajkoti commented on code in PR #35857: URL: https://github.com/apache/airflow/pull/35857#discussion_r1405349816 ## airflow/jobs/scheduler_job_runner.py: ## @@ -1577,15 +1577,18 @@ def _fail_tasks_stuck_in_queued(self, session: Session = NEW_SESSION) -> None: )

Re: [PR] Send explicit task logs when marking tasks stuck in queued as failed [airflow]

2023-11-26 Thread via GitHub
ephraimbuddy commented on code in PR #35857: URL: https://github.com/apache/airflow/pull/35857#discussion_r1405350011 ## airflow/jobs/scheduler_job_runner.py: ## @@ -1577,15 +1577,18 @@ def _fail_tasks_stuck_in_queued(self, session: Session = NEW_SESSION) -> None:

Re: [PR] Send explicit task logs when marking tasks stuck in queued as failed [airflow]

2023-11-26 Thread via GitHub
pankajkoti commented on code in PR #35857: URL: https://github.com/apache/airflow/pull/35857#discussion_r1405350927 ## airflow/jobs/scheduler_job_runner.py: ## @@ -1577,15 +1577,18 @@ def _fail_tasks_stuck_in_queued(self, session: Session = NEW_SESSION) -> None: )

Re: [PR] Send explicit task logs when marking tasks stuck in queued as failed [airflow]

2023-11-26 Thread via GitHub
ephraimbuddy commented on code in PR #35857: URL: https://github.com/apache/airflow/pull/35857#discussion_r1405351054 ## airflow/jobs/scheduler_job_runner.py: ## @@ -1577,15 +1577,18 @@ def _fail_tasks_stuck_in_queued(self, session: Session = NEW_SESSION) -> None:

Re: [PR] Fix passing dags.gitSync.sshKeySecret with dags.persistence.enabled=true [airflow]

2023-11-26 Thread via GitHub
dolfinus commented on PR #34331: URL: https://github.com/apache/airflow/pull/34331#issuecomment-1826720961 I'd rather prefer to show some kind of warning such a case than completely forbid it. Some users may successfully use this combination, there is no statistics related to this -- Th

Re: [PR] Send explicit task logs when marking tasks stuck in queued as failed [airflow]

2023-11-26 Thread via GitHub
pankajkoti closed pull request #35857: Send explicit task logs when marking tasks stuck in queued as failed URL: https://github.com/apache/airflow/pull/35857 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] Send explicit task logs when marking tasks stuck in queued as failed [airflow]

2023-11-26 Thread via GitHub
pankajkoti merged PR #35857: URL: https://github.com/apache/airflow/pull/35857 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airfl

(airflow) branch main updated: Send explicit task logs when marking tasks stuck in queued as failed (#35857)

2023-11-26 Thread pankajkoti
This is an automated email from the ASF dual-hosted git repository. pankajkoti pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new c7e13061bd Send explicit task logs when markin

Re: [PR] feat: SQLExecuteQueryOperator sql_callable [airflow]

2023-11-26 Thread via GitHub
Taragolis commented on PR #35844: URL: https://github.com/apache/airflow/pull/35844#issuecomment-1826743579 There is couple of solution already exists, note this not end of list, I believe there is also exist couple solution, so this list not final **Solution 1**: one of the, is J

Re: [PR] Optimize CI/PROD image waiting and verification in CI workflow [airflow]

2023-11-26 Thread via GitHub
potiuk merged PR #35856: URL: https://github.com/apache/airflow/pull/35856 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

(airflow) branch main updated: Optimize CI/PROD image waiting and verification in CI workflow (#35856)

2023-11-26 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new d36848828b Optimize CI/PROD image waiting and verif

[PR] Clarify constraints use in canary run in CI diagrams [airflow]

2023-11-26 Thread via GitHub
potiuk opened a new pull request, #35858: URL: https://github.com/apache/airflow/pull/35858 Small follow-up after #35856 - cache building uses constraints so rather than describing it, we add arrow-dependency --- **^ Add meaningful description above**

Re: [PR] Clarify constraints use in canary run in CI diagrams [airflow]

2023-11-26 Thread via GitHub
potiuk commented on PR #35858: URL: https://github.com/apache/airflow/pull/35858#issuecomment-1826754865 More accurate: https://github.com/apache/airflow/assets/595491/29e13a85-2244-4eab-88ad-982abfd86c1f";> -- This is an automated message from the Apache Git Service. To respond

[PR] Skip running "fix ownership" when Breeze image pre-commits are skipped [airflow]

2023-11-26 Thread via GitHub
potiuk opened a new pull request, #35859: URL: https://github.com/apache/airflow/pull/35859 When we skip breeze image pre-commits, we do not have breeze image available, because we want to run them as fast as possible. In this case we should not attempt to run "ownership fixing" - because:

Re: [PR] Skip running "fix ownership" when Breeze image pre-commits are skipped [airflow]

2023-11-26 Thread via GitHub
potiuk commented on PR #35859: URL: https://github.com/apache/airflow/pull/35859#issuecomment-1826758445 This is why "basic static checks" in https://github.com/apache/airflow/actions/runs/6995199303/job/19029787821?pr=35858#step:8:183 took 1m32 seconds instead of 30 seconds. -- This is

[PR] Make environment tests for breeze branch-agnostic [airflow]

2023-11-26 Thread via GitHub
potiuk opened a new pull request, #35860: URL: https://github.com/apache/airflow/pull/35860 The new tests in breeze depended implicitly on the fact they were running in `main` branch, but when run in v2*test, they should use the AIRFLOW_BRANCH as the expected branch rather than `main` expli

(airflow) branch main updated: Skip running "fix ownership" when Breeze image pre-commits are skipped (#35859)

2023-11-26 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new ca97dd67d4 Skip running "fix ownership" when Breeze

Re: [PR] Skip running "fix ownership" when Breeze image pre-commits are skipped [airflow]

2023-11-26 Thread via GitHub
potiuk merged PR #35859: URL: https://github.com/apache/airflow/pull/35859 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

Re: [PR] Make environment tests for breeze branch-agnostic [airflow]

2023-11-26 Thread via GitHub
potiuk merged PR #35860: URL: https://github.com/apache/airflow/pull/35860 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

(airflow) branch main updated: Make environment tests for breeze branch-agnostic (#35860)

2023-11-26 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new c3719d966e Make environment tests for breeze branch

Re: [PR] Clarify constraints use in canary run in CI diagrams [airflow]

2023-11-26 Thread via GitHub
potiuk merged PR #35858: URL: https://github.com/apache/airflow/pull/35858 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

(airflow) branch main updated (c3719d966e -> df6631ec08)

2023-11-26 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from c3719d966e Make environment tests for breeze branch-agnostic (#35860) add df6631ec08 Clarify constraints use in ca

[PR] Modify breeze commands and flags for standalone shell commands [airflow]

2023-11-26 Thread via GitHub
potiuk opened a new pull request, #35862: URL: https://github.com/apache/airflow/pull/35862 Breeze `shell` command has been foreseen eventually to also allow single commands in CI and pre-commit, and we are preparing it to be able to do so with all the bells and whistles. There are a few im

Re: [PR] Modify breeze commands and flags for standalone shell commands [airflow]

2023-11-26 Thread via GitHub
potiuk commented on PR #35862: URL: https://github.com/apache/airflow/pull/35862#issuecomment-1826790086 This is a logically-separated PR from #35830 as discussed in https://github.com/apache/airflow/pull/35830#pullrequestreview-1748817163 It's mostly revolving about making `breeze sh

[PR] Implement task duration page in react. [airflow]

2023-11-26 Thread via GitHub
tirkarthi opened a new pull request, #35863: URL: https://github.com/apache/airflow/pull/35863 closes: #30328 related: #30328 ![task_duration_1](https://github.com/apache/airflow/assets/3972343/0301c18d-40fe-4a80-ba59-a447dfb02490) -- This is an automated message from the A

[PR] Add more ways to connect to weaviate [airflow]

2023-11-26 Thread via GitHub
ephraimbuddy opened a new pull request, #35864: URL: https://github.com/apache/airflow/pull/35864 There are other options for connecting to weaviate. This commit adds these other options and also improved the imports/typing -- This is an automated message from the Apache Git Servi

Re: [PR] Modify breeze commands and flags for standalone shell commands [airflow]

2023-11-26 Thread via GitHub
potiuk commented on PR #35862: URL: https://github.com/apache/airflow/pull/35862#issuecomment-1826794060 BTW. Even with those small fixes adde - it should be FAR easier to review than https://github.com/apache/airflow/pull/35830 -- This is an automated message from the Apache Git Service

Re: [PR] Refactor and simplify breeze-based pre-commits [airflow]

2023-11-26 Thread via GitHub
potiuk commented on PR #35830: URL: https://github.com/apache/airflow/pull/35830#issuecomment-1826794451 As discussed with @jscheffl -> the "breeze command flag update" separated part ot of this PR has been moved to #35862 . I will rebase this one on top of it. -- This is an automated

Re: [PR] add guideline for missing image when running breeze [airflow]

2023-11-26 Thread via GitHub
potiuk commented on PR #35813: URL: https://github.com/apache/airflow/pull/35813#issuecomment-182681 Actually, that fix was separated out to a slightly smaller one https://github.com/apache/airflow/pull/35862 I **could** split it out even further if reviewers think it makes sense

Re: [PR] add guideline for missing image when running breeze [airflow]

2023-11-26 Thread via GitHub
potiuk commented on PR #35813: URL: https://github.com/apache/airflow/pull/35813#issuecomment-1826801193 The part that fixes the problem you had is here: https://github.com/apache/airflow/pull/35862/files#diff-6ef5b756e45d03d3723ba4a5c4b0f1417b115fe25560f293351c30290ec67b31R127

Re: [PR] Fix passing dags.gitSync.sshKeySecret with dags.persistence.enabled=true [airflow]

2023-11-26 Thread via GitHub
potiuk commented on PR #34331: URL: https://github.com/apache/airflow/pull/34331#issuecomment-1826801975 > I'd rather prefer to show some kind of warning in such a case than completely forbid it. Some users may successfully use this combination, there is no statistics related to this

Re: [PR] Fix passing dags.gitSync.sshKeySecret with dags.persistence.enabled=true [airflow]

2023-11-26 Thread via GitHub
potiuk commented on PR #34331: URL: https://github.com/apache/airflow/pull/34331#issuecomment-1826802063 > I'd rather prefer to show some kind of warning in such a case than completely forbid it. Some users may successfully use this combination, there is no statistics related to this

Re: [PR] Fix passing dags.gitSync.sshKeySecret with dags.persistence.enabled=true [airflow]

2023-11-26 Thread via GitHub
dolfinus commented on PR #34331: URL: https://github.com/apache/airflow/pull/34331#issuecomment-1826802369 No idea, and I'm not interested in this kind if solution. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

Re: [PR] Fix passing dags.gitSync.sshKeySecret with dags.persistence.enabled=true [airflow]

2023-11-26 Thread via GitHub
potiuk commented on PR #34331: URL: https://github.com/apache/airflow/pull/34331#issuecomment-1826802597 > No idea, and I'm not interested in this kind if solution. Pity. You loose opportunity to become one of 2700 contributors to Airflow. -- This is an automated message from the A

Re: [I] run_grant_dataset_view_access fails if view already exists [airflow]

2023-11-26 Thread via GitHub
kadai0308 commented on issue #35795: URL: https://github.com/apache/airflow/issues/35795#issuecomment-1826803699 Hi I am Kadai, new to here and looking forward to contribute. After go through the description and source code, i think i can fix this issue. Can you please assign this issue to

Re: [PR] Refactor and simplify breeze-based pre-commits [airflow]

2023-11-26 Thread via GitHub
potiuk commented on PR #35830: URL: https://github.com/apache/airflow/pull/35830#issuecomment-1826803741 Marking as Draft so that it's not merged accidentally before #35862 gets merged , but generally speaking - the last commit is quite ready and "complete" for review. -- This is an auto

Re: [PR] Fix passing dags.gitSync.sshKeySecret with dags.persistence.enabled=true [airflow]

2023-11-26 Thread via GitHub
dolfinus commented on PR #34331: URL: https://github.com/apache/airflow/pull/34331#issuecomment-1826804501 > You loose opportunity to become one of 2700 contributors to Airflow. I already am. If maintainers are not interested of merging PR which avoids producing wrong pod descr

[I] Someties test_cli_webserver_background sometimes detect processes left behind [airflow]

2023-11-26 Thread via GitHub
potiuk opened a new issue, #35865: URL: https://github.com/apache/airflow/issues/35865 ### Body Sometimes (rather rarely) the "check_processes" in `tests/cli/commands/test_webserver_command.py::TestCliWebServer::test_cli_webserver_background` detects some remaining processes.

Re: [PR] Fix passing dags.gitSync.sshKeySecret with dags.persistence.enabled=true [airflow]

2023-11-26 Thread via GitHub
potiuk commented on PR #34331: URL: https://github.com/apache/airflow/pull/34331#issuecomment-1826808549 > If maintainers are not interested of merging PR which avoids producing wrong pod description, and instead prefer to do nothing with that for months - ok, but very pitty to hear.

Re: [I] run_grant_dataset_view_access fails if view already exists [airflow]

2023-11-26 Thread via GitHub
kadai0308 commented on issue #35795: URL: https://github.com/apache/airflow/issues/35795#issuecomment-1826809213 After digging deeper, the issue already been fixed in google-cloud-bigquery version 3.13 [MR](https://github.com/googleapis/python-bigquery/pull/1682) -- This is an automated m

Re: [I] Status of testing Providers that were prepared on November 24, 2023 [airflow]

2023-11-26 Thread via GitHub
eladkal commented on issue #35845: URL: https://github.com/apache/airflow/issues/35845#issuecomment-1826812024 > #35790 works > > It has caused a problem with the example DAG (addressed in #35839) but the operator changes themselves work Thanks. Since functionality work as expe

Re: [PR] Add a migration script tooling support to migrate off mssql [airflow]

2023-11-26 Thread via GitHub
potiuk commented on PR #35861: URL: https://github.com/apache/airflow/pull/35861#issuecomment-1826815324 I am not going to block it in this form - but - as explained in a number of messages - I have a huge concern about a) making it part of official airflow tooling b) making it p

Re: [PR] [WIP] Fix vault gcp auth [airflow]

2023-11-26 Thread via GitHub
fpopic commented on code in PR #35855: URL: https://github.com/apache/airflow/pull/35855#discussion_r1405422826 ## airflow/providers/hashicorp/_internal_client/vault_client.py: ## @@ -291,13 +298,47 @@ def _auth_gcp(self, _client: hvac.Client) -> None: ) scop

Re: [PR] Add a migration script tooling support to migrate off mssql [airflow]

2023-11-26 Thread via GitHub
potiuk commented on PR #35861: URL: https://github.com/apache/airflow/pull/35861#issuecomment-1826819752 I wonder - what others think about it. Especially those who were in Airflow when we migrated to Airflow 2. from 1.10.15. And just to add a bit more context. We did release https:/

Re: [PR] [WIP] Fix vault gcp auth [airflow]

2023-11-26 Thread via GitHub
fpopic commented on code in PR #35855: URL: https://github.com/apache/airflow/pull/35855#discussion_r1405424437 ## airflow/providers/hashicorp/_internal_client/vault_client.py: ## @@ -291,13 +298,47 @@ def _auth_gcp(self, _client: hvac.Client) -> None: ) scop

Re: [PR] [WIP] Fix vault gcp auth [airflow]

2023-11-26 Thread via GitHub
fpopic commented on code in PR #35855: URL: https://github.com/apache/airflow/pull/35855#discussion_r1405424437 ## airflow/providers/hashicorp/_internal_client/vault_client.py: ## @@ -291,13 +298,47 @@ def _auth_gcp(self, _client: hvac.Client) -> None: ) scop

Re: [PR] Add a migration script tooling support to migrate off mssql [airflow]

2023-11-26 Thread via GitHub
jscheffl commented on PR #35861: URL: https://github.com/apache/airflow/pull/35861#issuecomment-1826823839 Hi @potiuk Yes, maybe this is a mis-understanding. I did NOT want to make this script a release piece. If we see we need it in other cases we _could_ leverage it. So it is now i

Re: [PR] Add a migration script tooling support to migrate off mssql [airflow]

2023-11-26 Thread via GitHub
potiuk commented on PR #35861: URL: https://github.com/apache/airflow/pull/35861#issuecomment-1826824513 And... Fantastic you are doing it ! Thanks for that! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

Re: [PR] Add a migration script tooling support to migrate off mssql [airflow]

2023-11-26 Thread via GitHub
eladkal commented on PR #35861: URL: https://github.com/apache/airflow/pull/35861#issuecomment-1826833780 > making it part of our repository in general I see no problem with that. Mssql is considered experimental. We can publish it under statmant of best effort. In the mailing l

Re: [I] Status of testing Providers that were prepared on November 24, 2023 [airflow]

2023-11-26 Thread via GitHub
hussein-awala commented on issue #35845: URL: https://github.com/apache/airflow/issues/35845#issuecomment-1826836426 I tested https://github.com/apache/airflow/pull/35185 and https://github.com/apache/airflow/pull/35635, they work as expected. -- This is an automated message from the Apac

[PR] Add encryption methods to serde for primitives [airflow]

2023-11-26 Thread via GitHub
bolkedebruin opened a new pull request, #35867: URL: https://github.com/apache/airflow/pull/35867 Several serialized object could expose sensitive information in the UI through XCom values. This generalizes how encryption works, including versioning of the protocol. This is a pr

[PR] Remove MSSQL support from Airflow core [airflow]

2023-11-26 Thread via GitHub
jscheffl opened a new pull request, #35868: URL: https://github.com/apache/airflow/pull/35868 DRAFT PR to remove MSSQL support from Airflow Open items: - Green build - Check that breeze works as expected - Discuss whether the matix action in Github needs a change for coverage?

Re: [PR] Simplify io.path [airflow]

2023-11-26 Thread via GitHub
bolkedebruin merged PR #35747: URL: https://github.com/apache/airflow/pull/35747 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@air

(airflow) branch main updated: Simplify io.path (#35747)

2023-11-26 Thread bolke
This is an automated email from the ASF dual-hosted git repository. bolke pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new fa94ee9ebb Simplify io.path (#35747) fa94ee9ebb is d

Re: [PR] feat: SQLExecuteQueryOperator sql_callable [airflow]

2023-11-26 Thread via GitHub
raphaelauv commented on PR #35844: URL: https://github.com/apache/airflow/pull/35844#issuecomment-1826844693 I agree that a general solution for all operator is better than adding the feature to that specific operator So the main question is Do we agree that a pythonOperator b

Re: [I] Status of testing Providers that were prepared on November 24, 2023 [airflow]

2023-11-26 Thread via GitHub
pankajkoti commented on issue #35845: URL: https://github.com/apache/airflow/issues/35845#issuecomment-1826851639 Tested my set of PRs, all work fine, thank you for the the release efforts! -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] Make pyodbc.Row and databricks.Row JSON-serializable via new `make_serializable` method [airflow]

2023-11-26 Thread via GitHub
bolkedebruin commented on PR #32319: URL: https://github.com/apache/airflow/pull/32319#issuecomment-1826863178 I missed this - why was the non standard `make_serializable` chosen over the standard `serialize` and `deserialize` methods? It also missed versioning and adds to the pre-commits h

(airflow) branch constraints-main updated: Updating constraints. Github run id:6997046175

2023-11-26 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new 90477fa3e5 Updating con

Re: [I] Status of testing Providers that were prepared on November 24, 2023 [airflow]

2023-11-26 Thread via GitHub
bolkedebruin commented on issue #35845: URL: https://github.com/apache/airflow/issues/35845#issuecomment-1826867395 I would like to see a re-discussion of https://github.com/apache/airflow/pull/32319 before including it. -- This is an automated message from the Apache Git Service. To resp

Re: [PR] Make pyodbc.Row and databricks.Row JSON-serializable via new `make_serializable` method [airflow]

2023-11-26 Thread via GitHub
Joffreybvn commented on PR #32319: URL: https://github.com/apache/airflow/pull/32319#issuecomment-1826875406 I'm whiling to create a new PR and pause/drop this one. But, during the implementation of this PR, I could not create/instantiate pyobc.Row objects ([which is a C++ object](ht

Re: [PR] Make pyodbc.Row and databricks.Row JSON-serializable via new `make_serializable` method [airflow]

2023-11-26 Thread via GitHub
potiuk commented on PR #32319: URL: https://github.com/apache/airflow/pull/32319#issuecomment-1826875811 I do not think it has anything to do with "standard" behaviour. This is an internal implementation of Hooks - each DB hooks might make their own decision on how to make the (already comm

Re: [I] Status of testing Providers that were prepared on November 24, 2023 [airflow]

2023-11-26 Thread via GitHub
potiuk commented on issue #35845: URL: https://github.com/apache/airflow/issues/35845#issuecomment-1826875916 > I would like to see a re-discussion of #32319 before including it. I think this is misunderstanding what #32319 wants to achieve. Commented there. -- This is an automated

Re: [PR] Modify breeze commands and flags for standalone shell commands [airflow]

2023-11-26 Thread via GitHub
potiuk commented on PR #35862: URL: https://github.com/apache/airflow/pull/35862#issuecomment-182681 > First I was fearing the size to review - until I realized that 50% was boilerplate code changes and 50% SVG image generated :-D LGTM! Precisely :). -- This is an automated me

Re: [PR] Modify breeze commands and flags for standalone shell commands [airflow]

2023-11-26 Thread via GitHub
potiuk merged PR #35862: URL: https://github.com/apache/airflow/pull/35862 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

Re: [PR] Refactor and simplify breeze-based pre-commits [airflow]

2023-11-26 Thread via GitHub
potiuk commented on PR #35830: URL: https://github.com/apache/airflow/pull/35830#issuecomment-1826878279 OK. Merged https://github.com/apache/airflow/pull/35862 so made it ready for review. -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [I] Databricks: add SQL endpoint operators [airflow]

2023-11-26 Thread via GitHub
wojkam73159 commented on issue #21377: URL: https://github.com/apache/airflow/issues/21377#issuecomment-1826878579 Hi, is this issue is still open? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Refactor and simplify breeze-based pre-commits [airflow]

2023-11-26 Thread via GitHub
potiuk commented on PR #35830: URL: https://github.com/apache/airflow/pull/35830#issuecomment-1826878756 BTW. If this is still too big I am happy to move pre-commit by pre-commit separately . -- This is an automated message from the Apache Git Service. To respond to the message, please lo

[I] Scheduler keeps getting killed [airflow]

2023-11-26 Thread via GitHub
Tony-1993 opened a new issue, #35869: URL: https://github.com/apache/airflow/issues/35869 ### Official Helm Chart version 1.11.0 (latest released) ### Apache Airflow version 2.3.4 ### Kubernetes Version 1.24.16-gke.500 ### Helm Chart configuration

Re: [I] Scheduler keeps getting killed [airflow]

2023-11-26 Thread via GitHub
boring-cyborg[bot] commented on issue #35869: URL: https://github.com/apache/airflow/issues/35869#issuecomment-1826881506 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for ap

Re: [PR] Make pyodbc.Row and databricks.Row JSON-serializable via new `make_serializable` method [airflow]

2023-11-26 Thread via GitHub
bolkedebruin commented on PR #32319: URL: https://github.com/apache/airflow/pull/32319#issuecomment-1826883406 Hey @Joffreybvn thanks for responding and apologies for being late to the game and being a bit rough. It isn't required to return the same during deserialization as is provided thr

Re: [PR] Send explicit task logs when marking tasks stuck in queued as failed [airflow]

2023-11-26 Thread via GitHub
dstandish commented on code in PR #35857: URL: https://github.com/apache/airflow/pull/35857#discussion_r1405470805 ## airflow/jobs/scheduler_job_runner.py: ## @@ -1577,15 +1577,18 @@ def _fail_tasks_stuck_in_queued(self, session: Session = NEW_SESSION) -> None: )

Re: [PR] Make pyodbc.Row and databricks.Row JSON-serializable via new `make_serializable` method [airflow]

2023-11-26 Thread via GitHub
potiuk commented on PR #32319: URL: https://github.com/apache/airflow/pull/32319#issuecomment-1826887657 > Why is the common _make_serializable method required? For the reason described above - comon "airflow standard" returned output of DBApiHook. All the DBApiHook implementations r

Re: [PR] Make pyodbc.Row and databricks.Row JSON-serializable via new `make_serializable` method [airflow]

2023-11-26 Thread via GitHub
potiuk commented on PR #32319: URL: https://github.com/apache/airflow/pull/32319#issuecomment-1826888353 Also -I think really you are mixing the problem here It's not serialize/deserialize we are talking about here. And IMHO the name very well reflects the intention. We are not trying t

Re: [I] Status of testing Providers that were prepared on November 24, 2023 [airflow]

2023-11-26 Thread via GitHub
potiuk commented on issue #35845: URL: https://github.com/apache/airflow/issues/35845#issuecomment-1826889795 Conflirmed my template + reproducible builds worl (works great! PR shortly to a better PMC verification step is coming soon :) ) -- This is an automated message from the Apache

[I] Simultaneous tasks completion causing missing dataset triggered dag runs [airflow]

2023-11-26 Thread via GitHub
boushphong opened a new issue, #35870: URL: https://github.com/apache/airflow/issues/35870 ### Apache Airflow version 2.7.3 ### What happened When multiple airflow tasks finish at about the same time, and those tasks are also responsible for triggering other Dag via Data

Re: [PR] Make pyodbc.Row and databricks.Row JSON-serializable via new `make_serializable` method [airflow]

2023-11-26 Thread via GitHub
bolkedebruin commented on PR #32319: URL: https://github.com/apache/airflow/pull/32319#issuecomment-1826892062 > Also -I think really you are mixing the problem here It's not serialize/deserialize we are talking about here. And IMHO the name very well reflects the intention. We are not

[PR] Add reproducible build check step in the proces of Provider releases [airflow]

2023-11-26 Thread via GitHub
potiuk opened a new pull request, #35871: URL: https://github.com/apache/airflow/pull/35871 The reproducible build we just added makes it possible to run full reproducible package builds (byte-to-byte reproducible) - which means that whoever uses the same sources of Airflow (and official 3r

[I] Airflow can't extract host and port in Connection uri that has multiple hosts [airflow]

2023-11-26 Thread via GitHub
boushphong opened a new issue, #35872: URL: https://github.com/apache/airflow/issues/35872 ### Apache Airflow version 2.7.3 ### What happened We have a Secret backend that retrieves connection uri. We encountered an error where if we have a connection uri that has multip

Re: [PR] confirmation dialog box for DAG run actions [airflow]

2023-11-26 Thread via GitHub
pierrejeambrun commented on code in PR #35393: URL: https://github.com/apache/airflow/pull/35393#discussion_r1405477319 ## airflow/www/static/js/dag/details/dagRun/MarkRunAs.tsx: ## @@ -58,42 +65,115 @@ const MarkRunAs = ({ runId, state, ...otherProps }: Props) => { markSu

Re: [PR] confirmation dialog box for DAG run actions [airflow]

2023-11-26 Thread via GitHub
pierrejeambrun commented on code in PR #35393: URL: https://github.com/apache/airflow/pull/35393#discussion_r1405477686 ## airflow/www/static/js/dag/details/dagRun/MarkRunAs.tsx: ## @@ -58,42 +65,115 @@ const MarkRunAs = ({ runId, state, ...otherProps }: Props) => { markSu

Re: [PR] confirmation dialog box for DAG run actions [airflow]

2023-11-26 Thread via GitHub
pierrejeambrun commented on code in PR #35393: URL: https://github.com/apache/airflow/pull/35393#discussion_r1405477686 ## airflow/www/static/js/dag/details/dagRun/MarkRunAs.tsx: ## @@ -58,42 +65,115 @@ const MarkRunAs = ({ runId, state, ...otherProps }: Props) => { markSu

Re: [PR] Add XCom tab to Grid [airflow]

2023-11-26 Thread via GitHub
hduong-mwam commented on code in PR #35719: URL: https://github.com/apache/airflow/pull/35719#discussion_r1405478899 ## airflow/www/static/js/dag/details/index.tsx: ## @@ -138,6 +146,7 @@ const Details = ({ const isGroupOrMappedTaskSummary = isGroup || isMappedTaskSummary;

Re: [PR] confirmation dialog box for DAG run actions [airflow]

2023-11-26 Thread via GitHub
pierrejeambrun commented on code in PR #35393: URL: https://github.com/apache/airflow/pull/35393#discussion_r1405477686 ## airflow/www/static/js/dag/details/dagRun/MarkRunAs.tsx: ## @@ -58,42 +65,115 @@ const MarkRunAs = ({ runId, state, ...otherProps }: Props) => { markSu

Re: [PR] confirmation dialog box for DAG run actions [airflow]

2023-11-26 Thread via GitHub
pierrejeambrun commented on code in PR #35393: URL: https://github.com/apache/airflow/pull/35393#discussion_r1405477686 ## airflow/www/static/js/dag/details/dagRun/MarkRunAs.tsx: ## @@ -58,42 +65,115 @@ const MarkRunAs = ({ runId, state, ...otherProps }: Props) => { markSu

Re: [PR] confirmation dialog box for DAG run actions [airflow]

2023-11-26 Thread via GitHub
pierrejeambrun commented on code in PR #35393: URL: https://github.com/apache/airflow/pull/35393#discussion_r1405477686 ## airflow/www/static/js/dag/details/dagRun/MarkRunAs.tsx: ## @@ -58,42 +65,115 @@ const MarkRunAs = ({ runId, state, ...otherProps }: Props) => { markSu

Re: [PR] Add XCom tab to Grid [airflow]

2023-11-26 Thread via GitHub
hduong-mwam commented on code in PR #35719: URL: https://github.com/apache/airflow/pull/35719#discussion_r1405479106 ## airflow/www/static/js/dag/details/taskInstance/Xcom/index.tsx: ## @@ -0,0 +1,110 @@ +/*! + * Licensed to the Apache Software Foundation (ASF) under one + * or

Re: [PR] confirmation dialog box for DAG run actions [airflow]

2023-11-26 Thread via GitHub
pierrejeambrun commented on code in PR #35393: URL: https://github.com/apache/airflow/pull/35393#discussion_r1405477319 ## airflow/www/static/js/dag/details/dagRun/MarkRunAs.tsx: ## @@ -58,42 +65,115 @@ const MarkRunAs = ({ runId, state, ...otherProps }: Props) => { markSu

Re: [PR] confirmation dialog box for DAG run actions [airflow]

2023-11-26 Thread via GitHub
pierrejeambrun commented on code in PR #35393: URL: https://github.com/apache/airflow/pull/35393#discussion_r1405477686 ## airflow/www/static/js/dag/details/dagRun/MarkRunAs.tsx: ## @@ -58,42 +65,115 @@ const MarkRunAs = ({ runId, state, ...otherProps }: Props) => { markSu

Re: [PR] Make pyodbc.Row and databricks.Row JSON-serializable via new `make_serializable` method [airflow]

2023-11-26 Thread via GitHub
potiuk commented on PR #32319: URL: https://github.com/apache/airflow/pull/32319#issuecomment-1826902225 > You are converting from one format to another. This is effectively serialization. This is why serde calls to serialize. By removing type information and not including versioning you ca

Re: [PR] Add multiselect to run state in grid view [airflow]

2023-11-26 Thread via GitHub
pierrejeambrun commented on code in PR #35403: URL: https://github.com/apache/airflow/pull/35403#discussion_r1405480918 ## airflow/www/views.py: ## @@ -3527,13 +3527,15 @@ def grid_data(self): with create_session() as session: query = select(DagRun).where(D

Re: [PR] Add reproducible build check step in the proces of Provider releases [airflow]

2023-11-26 Thread via GitHub
potiuk commented on PR #35871: URL: https://github.com/apache/airflow/pull/35871#issuecomment-1826904276 > Really nice Indeeed. Those "reproducible builds" - once we have good tooling - are really powerful in making "secure" supply chain possible -- This is an automated message fro

Re: [PR] Add reproducible build check step in the proces of Provider releases [airflow]

2023-11-26 Thread via GitHub
hussein-awala commented on code in PR #35871: URL: https://github.com/apache/airflow/pull/35871#discussion_r1405482513 ## dev/README_RELEASE_PROVIDER_PACKAGES.md: ## @@ -667,6 +667,102 @@ docker run --rm --entrypoint "airflow" local/airflow info docker image rm local/airflow `

Re: [PR] Add reproducible build check step in the proces of Provider releases [airflow]

2023-11-26 Thread via GitHub
potiuk merged PR #35871: URL: https://github.com/apache/airflow/pull/35871 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

(airflow) branch main updated: Add reproducible build check step in the proces of Provider releases (#35871)

2023-11-26 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 162d0f00d4 Add reproducible build check step in the

[PR] Ignore E402 (import not at the top of file) in pre-commits [airflow]

2023-11-26 Thread via GitHub
potiuk opened a new pull request, #35873: URL: https://github.com/apache/airflow/pull/35873 Pre-commits often import some modules from elsewhere after inserting a common folder at PYTHONPATH. Pre-commits are pretty special, because they are supposed to be run as standalone scripts by pre-co

Re: [PR] Refactor and simplify breeze-based pre-commits [airflow]

2023-11-26 Thread via GitHub
potiuk commented on PR #35830: URL: https://github.com/apache/airflow/pull/35830#issuecomment-1826909513 I started to separate things out https://github.com/apache/airflow/pull/35873 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] Refactor and simplify breeze-based pre-commits [airflow]

2023-11-26 Thread via GitHub
jscheffl commented on code in PR #35830: URL: https://github.com/apache/airflow/pull/35830#discussion_r1405486136 ## scripts/ci/pre_commit/common_precommit_utils.py: ## @@ -71,3 +80,74 @@ def get_directory_hash(directory: Path, skip_path_regexp: str | None = None) -> i

Re: [PR] Add XCom tab to Grid [airflow]

2023-11-26 Thread via GitHub
hduong-mwam commented on PR #35719: URL: https://github.com/apache/airflow/pull/35719#issuecomment-1826912752 > I really _love_ the UI changes gradually migrating from legacy to new. > > When I tested locally I saw still the old link pointing to the old XCom page in the second row of

Re: [PR] Add XCom tab to Grid [airflow]

2023-11-26 Thread via GitHub
hduong-mwam commented on code in PR #35719: URL: https://github.com/apache/airflow/pull/35719#discussion_r1405490868 ## airflow/www/static/js/dag/details/taskInstance/Xcom/XcomEntry.tsx: ## @@ -0,0 +1,82 @@ +/*! + * Licensed to the Apache Software Foundation (ASF) under one + *

Re: [PR] Make pyodbc.Row and databricks.Row JSON-serializable via new `make_serializable` method [airflow]

2023-11-26 Thread via GitHub
bolkedebruin commented on PR #32319: URL: https://github.com/apache/airflow/pull/32319#issuecomment-1826915604 > > You are converting from one format to another. This is effectively serialization. This is why serde calls to serialize. By removing type information and not including versionin

  1   2   >