Re: [PR] Create FAB provider and move FAB auth manager in it [airflow]

2023-12-09 Thread via GitHub
potiuk commented on PR #35926: URL: https://github.com/apache/airflow/pull/35926#issuecomment-1848320351 Woooh. Compat tests passed :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

[PR] Remove generated provider_packages dir [airflow]

2023-12-09 Thread via GitHub
potiuk opened a new pull request, #36138: URL: https://github.com/apache/airflow/pull/36138 When release manager generates provider packages, sources for them are temporarily copied to subfolders of `dist\provider_packages`. Generally we remove all such code (unless `--skip-deletin

Re: [I] Gitsync over https denied, unsupported by Github [airflow]

2023-12-09 Thread via GitHub
potiuk closed issue #36118: Gitsync over https denied, unsupported by Github URL: https://github.com/apache/airflow/issues/36118 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Remove generated provider_packages dir [airflow]

2023-12-09 Thread via GitHub
potiuk merged PR #36138: URL: https://github.com/apache/airflow/pull/36138 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

(airflow) branch main updated: Remove generated provider_packages dir (#36138)

2023-12-09 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new ada9f15be8 Remove generated provider_packages dir (

Re: [PR] Create latest log dir symlink as relative link [airflow]

2023-12-09 Thread via GitHub
potiuk merged PR #36019: URL: https://github.com/apache/airflow/pull/36019 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

(airflow) branch main updated: Create latest log dir symlink as relative link (#36019)

2023-12-09 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 17e91727b2 Create latest log dir symlink as relativ

Re: [PR] Unify how docker commands get executed in Breeze [airflow]

2023-12-09 Thread via GitHub
potiuk commented on code in PR #36131: URL: https://github.com/apache/airflow/pull/36131#discussion_r1421389322 ## dev/breeze/src/airflow_breeze/commands/release_management_commands.py: ## @@ -170,60 +161,6 @@ class VersionedFile(NamedTuple): comparable_version: Version

Re: [PR] Unify how docker commands get executed in Breeze [airflow]

2023-12-09 Thread via GitHub
potiuk commented on code in PR #36131: URL: https://github.com/apache/airflow/pull/36131#discussion_r1421389322 ## dev/breeze/src/airflow_breeze/commands/release_management_commands.py: ## @@ -170,60 +161,6 @@ class VersionedFile(NamedTuple): comparable_version: Version

Re: [PR] Fix broken log streaming from #35677 [airflow]

2023-12-09 Thread via GitHub
eladkal commented on PR #36127: URL: https://github.com/apache/airflow/pull/36127#issuecomment-1848360645 I will cancel RC1 for docker peovider and cut rc2 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] Fix broken log streaming from #35677 [airflow]

2023-12-09 Thread via GitHub
eladkal merged PR #36127: URL: https://github.com/apache/airflow/pull/36127 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.

(airflow) branch main updated: Fix broken log streaming from #35677 (#36127)

2023-12-09 Thread eladkal
This is an automated email from the ASF dual-hosted git repository. eladkal pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new f9c952bb7f Fix broken log streaming from #35677 (#

Re: [I] Status of testing Providers that were prepared on December 08, 2023 [airflow]

2023-12-09 Thread via GitHub
eladkal commented on issue #36117: URL: https://github.com/apache/airflow/issues/36117#issuecomment-1848360928 Docker provider will be excluded from rc1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[PR] Fix start-airflow command [airflow]

2023-12-09 Thread via GitHub
potiuk opened a new pull request, #36140: URL: https://github.com/apache/airflow/pull/36140 Recent refactors in installation parameters, accidentally broke start-airflow command in main. This PR fixes it by synchronizing missing args. --- **^ Add meaning

Re: [PR] Fix start-airflow command [airflow]

2023-12-09 Thread via GitHub
potiuk merged PR #36140: URL: https://github.com/apache/airflow/pull/36140 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

(airflow) branch main updated: Fix start-airflow command (#36140)

2023-12-09 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 9c0b0cd98f Fix start-airflow command (#36140) 9c0b0

[PR] Change retry type for Google Dataflow Client to async one [airflow]

2023-12-09 Thread via GitHub
potiuk opened a new pull request, #36141: URL: https://github.com/apache/airflow/pull/36141 Google Dataflow Client 0.8.6 implemented bugfix where retry type was changed to async. This caused our canary builds to fail. We change the client to Async now and bump min version of the clien

Re: [I] Status of testing Providers that were prepared on December 08, 2023 [airflow]

2023-12-09 Thread via GitHub
hussein-awala commented on issue #36117: URL: https://github.com/apache/airflow/issues/36117#issuecomment-1848434947 I tested my change on the Kubernetes executor, and most of the changes in the Amazon provider, all work as expected. -- This is an automated message from the Apache Git Ser

Re: [PR] GCSHook only perform list if delimiter specified [airflow]

2023-12-09 Thread via GitHub
hussein-awala commented on code in PR #36130: URL: https://github.com/apache/airflow/pull/36130#discussion_r1421434993 ## airflow/providers/google/cloud/hooks/gcs.py: ## @@ -821,7 +821,8 @@ def _list( delimiter=delimiter, versions=versio

Re: [PR] Make pyodbc.Row and databricks.Row JSON-serializable via new `make_serializable` method [airflow]

2023-12-09 Thread via GitHub
pankajastro commented on code in PR #32319: URL: https://github.com/apache/airflow/pull/32319#discussion_r1421437350 ## airflow/providers/databricks/hooks/databricks_sql.py: ## @@ -240,6 +240,13 @@ def run( else: return results +@staticmethod +def

Re: [PR] Change retry type for Google Dataflow Client to async one [airflow]

2023-12-09 Thread via GitHub
potiuk commented on PR #36141: URL: https://github.com/apache/airflow/pull/36141#issuecomment-1848443934 Needs few more libraries and more complete/selective use of AsyncRetry - seems the fixes were applied to a number of Google clients. -- This is an automated message from the Apache Git

Re: [PR] EventsTimetable schedules past events even if catchup=False [airflow]

2023-12-09 Thread via GitHub
Taragolis commented on code in PR #36134: URL: https://github.com/apache/airflow/pull/36134#discussion_r1421437692 ## airflow/timetables/events.py: ## @@ -79,22 +82,34 @@ def next_dagrun_info( last_automated_data_interval: DataInterval | None, restriction: Time

Re: [I] Status of testing Providers that were prepared on December 08, 2023 [airflow]

2023-12-09 Thread via GitHub
pankajastro commented on issue #36117: URL: https://github.com/apache/airflow/issues/36117#issuecomment-1848445737 As @atrbgithub has mentioned above https://github.com/apache/airflow/pull/34919 is breaking an existing DAG. so we might need to exclude Google provider -- This is an automa

[I] Can the BigQuery Partition Sensor be updated to sense a list of partition IDs instead of a single partition ID (string) [airflow]

2023-12-09 Thread via GitHub
noeun-kim opened a new issue, #36142: URL: https://github.com/apache/airflow/issues/36142 ### Description ### **Summary** I'm looking to work on sensing multiple partition IDs and I'm attempting to modify the relevant class for this purpose. I believe it's a simple adjustmen

Re: [I] Can the BigQuery Partition Sensor be updated to sense a list of partition IDs instead of a single partition ID (string) [airflow]

2023-12-09 Thread via GitHub
boring-cyborg[bot] commented on issue #36142: URL: https://github.com/apache/airflow/issues/36142#issuecomment-1848448691 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for ap

Re: [PR] Change retry type for Google Dataflow Client to async one [airflow]

2023-12-09 Thread via GitHub
potiuk commented on PR #36141: URL: https://github.com/apache/airflow/pull/36141#issuecomment-1848449776 Right .. That one should be green (and I believe good for RC2 of google provider). -- This is an automated message from the Apache Git Service. To respond to the message, please log o

Re: [I] Delete DagRun event is not registered in audit log [airflow]

2023-12-09 Thread via GitHub
eladkal commented on issue #36057: URL: https://github.com/apache/airflow/issues/36057#issuecomment-1848450007 > Actually @eladkal , wondering if the above is sufficient for what you need... the problem is that the action_logging decorator only logs a single row for both a single delete (if

Re: [I] Status of testing Providers that were prepared on December 08, 2023 [airflow]

2023-12-09 Thread via GitHub
eladkal commented on issue #36117: URL: https://github.com/apache/airflow/issues/36117#issuecomment-1848482088 > so we might need to exclude Google provider Yes. Google provider will also have RC2 -- This is an automated message from the Apache Git Service. To respond to the message

Re: [PR] Change retry type for Google Clients to async ones where needed [airflow]

2023-12-09 Thread via GitHub
potiuk commented on PR #36141: URL: https://github.com/apache/airflow/pull/36141#issuecomment-1848592427 OK. I found one more edge case in the new "provider verification" a fix to that will follow - then we will be able to merge it when It is merged. The problem is that if we bump a p

Re: [I] BigQuery with impersonation_chain does not accept custom scopes [airflow]

2023-12-09 Thread via GitHub
buu-nguyen commented on issue #33400: URL: https://github.com/apache/airflow/issues/33400#issuecomment-1848596780 > Possible workaround: > > * Step1: Extend a BigQueryHook class and overwrite [`GoogleBaseHook#scopes`](https://github.com/apache/airflow/blob/0f73647bdab79ac6c3096122

[PR] Use constraints from current CI image to install providers for tests [airflow]

2023-12-09 Thread via GitHub
potiuk opened a new pull request, #36143: URL: https://github.com/apache/airflow/pull/36143 We were using constraints from GitHub repository "constraints-main" branch when installing and testing providers in main. Hewever, when we are testing provider in their "main" form, we should use the

Re: [PR] Use constraints from current CI image to install providers for tests [airflow]

2023-12-09 Thread via GitHub
potiuk commented on PR #36143: URL: https://github.com/apache/airflow/pull/36143#issuecomment-1848598233 fixes problem found in #36141. The problem in https://github.com/apache/airflow/actions/runs/7152402263/job/19477871828?pr=36141 ``` ERROR: Cannot install apache-airflow-provi

Re: [PR] Change retry type for Google Clients to async ones where needed [airflow]

2023-12-09 Thread via GitHub
potiuk commented on PR #36141: URL: https://github.com/apache/airflow/pull/36141#issuecomment-1848598276 Fix in https://github.com/apache/airflow/pull/36143 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

Re: [PR] Use constraints from current CI image to install providers for tests [airflow]

2023-12-09 Thread via GitHub
eladkal commented on PR #36143: URL: https://github.com/apache/airflow/pull/36143#issuecomment-1848599219 > We were using constraints from GitHub repository "constraints-main" branch when installing and testing providers in main. Hewever, when we are testing provider in their "main" form, w

Re: [PR] Sync v2-8-stable with v2-8-test to release 2.8.0 [airflow]

2023-12-09 Thread via GitHub
eladkal commented on code in PR #36065: URL: https://github.com/apache/airflow/pull/36065#discussion_r1421500778 ## RELEASE_NOTES.rst: ## @@ -63,6 +64,14 @@ New Features Improvements +- Account for change in UTC offset when calculating next schedule (#35887) Re

Re: [PR] Use constraints from current CI image to install providers for tests [airflow]

2023-12-09 Thread via GitHub
pankajkoti commented on code in PR #36143: URL: https://github.com/apache/airflow/pull/36143#discussion_r1421501109 ## .github/workflows/ci.yml: ## @@ -762,14 +762,20 @@ jobs: run: > breeze release-management generate-issue-content-providers --only

Re: [PR] Use constraints from current CI image to install providers for tests [airflow]

2023-12-09 Thread via GitHub
potiuk commented on PR #36143: URL: https://github.com/apache/airflow/pull/36143#issuecomment-1848603934 > Then How did this ever worked? It's a very recent change when I split installation to two (last week), and it worked fine as long as you have not added a limit that violated the

Re: [PR] Create latest log dir symlink as relative link [airflow]

2023-12-09 Thread via GitHub
eladkal commented on PR #36019: URL: https://github.com/apache/airflow/pull/36019#issuecomment-1848604366 window for 2.8.0 is closed will have to wait for next release -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

Re: [PR] Use constraints from current CI image to install providers for tests [airflow]

2023-12-09 Thread via GitHub
pankajkoti commented on code in PR #36143: URL: https://github.com/apache/airflow/pull/36143#discussion_r1421501109 ## .github/workflows/ci.yml: ## @@ -762,14 +762,20 @@ jobs: run: > breeze release-management generate-issue-content-providers --only

Re: [PR] Use constraints from current CI image to install providers for tests [airflow]

2023-12-09 Thread via GitHub
potiuk commented on code in PR #36143: URL: https://github.com/apache/airflow/pull/36143#discussion_r1421501735 ## .github/workflows/ci.yml: ## @@ -762,14 +762,20 @@ jobs: run: > breeze release-management generate-issue-content-providers --only-ava

Re: [PR] Create latest log dir symlink as relative link [airflow]

2023-12-09 Thread via GitHub
potiuk commented on PR #36019: URL: https://github.com/apache/airflow/pull/36019#issuecomment-1848605415 Ech... yeah sneaking things in the last moment :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

(airflow) branch main updated: Use constraints from current CI image to install providers for tests (#36143)

2023-12-09 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 6e419001d3 Use constraints from current CI image to

Re: [PR] Use constraints from current CI image to install providers for tests [airflow]

2023-12-09 Thread via GitHub
potiuk merged PR #36143: URL: https://github.com/apache/airflow/pull/36143 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[PR] Fix the bug that affected the DAG end date. [airflow]

2023-12-09 Thread via GitHub
avkirilishin opened a new pull request, #36144: URL: https://github.com/apache/airflow/pull/36144 Closes: https://github.com/apache/airflow/issues/31555 The state management has been relocated from the `mark_task` module to the `DagRun`'s set state. Changes to attributes were imple

[PR] fix connection type webhdfs [airflow]

2023-12-09 Thread via GitHub
ankurbajaj9 opened a new pull request, #36145: URL: https://github.com/apache/airflow/pull/36145 These changes will enable "webhdfs" as a connection type in list of connection types when the webhdfs plugin is installed. Earlier there was a "hdfs" type of connection but that was

[PR] Fixu using latest constraints in sdist package tests [airflow]

2023-12-09 Thread via GitHub
potiuk opened a new pull request, #36146: URL: https://github.com/apache/airflow/pull/36146 Follow-up after #36143 - I realized sdist package tests (only run in canary builds) will have the same problem as wheel installation. I moved it closer in the workflow to the wheel installation

[PR] Fix broken regex for [airflow]

2023-12-09 Thread via GitHub
tobiaszorzetto opened a new pull request, #36147: URL: https://github.com/apache/airflow/pull/36147 This PR aims to fix the problem involving a broken regex in the function utilized for standardizing the allowed_deserialization_classes, as can be seen in issu

Re: [PR] Fix broken regex for [airflow]

2023-12-09 Thread via GitHub
boring-cyborg[bot] commented on PR #36147: URL: https://github.com/apache/airflow/pull/36147#issuecomment-1848678642 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution

Re: [PR] Fixu using latest constraints in sdist package tests [airflow]

2023-12-09 Thread via GitHub
potiuk commented on code in PR #36146: URL: https://github.com/apache/airflow/pull/36146#discussion_r1421569175 ## .github/workflows/ci.yml: ## @@ -796,6 +796,69 @@ jobs: breeze release-management prepare-provider-packages --skip-tag-check --package-format

[PR] Fix sftp warrning log [airflow]

2023-12-09 Thread via GitHub
romsharon98 opened a new pull request, #36148: URL: https://github.com/apache/airflow/pull/36148 Add spaces for logging warning. --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/mai

Re: [PR] Fix using latest constraints in sdist package tests [airflow]

2023-12-09 Thread via GitHub
potiuk merged PR #36146: URL: https://github.com/apache/airflow/pull/36146 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

(airflow) branch main updated: Fixu using latest constraints in sdist package tests (#36146)

2023-12-09 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 7b6cadd0e5 Fixu using latest constraints in sdist p

Re: [PR] Change retry type for Google Clients to async ones where needed [airflow]

2023-12-09 Thread via GitHub
potiuk merged PR #36141: URL: https://github.com/apache/airflow/pull/36141 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

Re: [PR] Change retry type for Google Clients to async ones where needed [airflow]

2023-12-09 Thread via GitHub
potiuk commented on PR #36141: URL: https://github.com/apache/airflow/pull/36141#issuecomment-1848685224 The fix worked :). and even had a follow-up. Merging it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

(airflow) branch main updated: Change retry type for Google Dataflow Client to async one (#36141)

2023-12-09 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 8d0c5d9008 Change retry type for Google Dataflow Cl

Re: [PR] Fix the bug that affected the DAG end date. [airflow]

2023-12-09 Thread via GitHub
avkirilishin closed pull request #36144: Fix the bug that affected the DAG end date. URL: https://github.com/apache/airflow/pull/36144 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

(airflow) branch constraints-main updated: Updating constraints. Github run id:7153571617

2023-12-09 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new d36843a9a0 Updating con

[PR] Fix mapped task group no longer works as expected [airflow]

2023-12-09 Thread via GitHub
avkirilishin opened a new pull request, #36149: URL: https://github.com/apache/airflow/pull/36149 Closes: https://github.com/apache/airflow/issues/35541 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [I] [QUARANTINED] test_scheduler_task_start_date is flaky [airflow]

2023-12-09 Thread via GitHub
avkirilishin commented on issue #15085: URL: https://github.com/apache/airflow/issues/15085#issuecomment-1848773842 Did it happen again? https://github.com/apache/airflow/actions/runs/7153230354/job/19479506575?pr=36144#step:6:14652 I can't catch it on my laptop. I've run it ~1000

Re: [I] Airflow StatsD Configuration Error: XML 'ResourceNotFound' Message for statsd_host Environment Variable [airflow]

2023-12-09 Thread via GitHub
github-actions[bot] commented on issue #35817: URL: https://github.com/apache/airflow/issues/35817#issuecomment-1848790180 This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further

Re: [PR] Prevent users from misusing _PIP_ADDITIONAL_REQUIREMENTS [airflow]

2023-12-09 Thread via GitHub
github-actions[bot] closed pull request #33879: Prevent users from misusing _PIP_ADDITIONAL_REQUIREMENTS URL: https://github.com/apache/airflow/pull/33879 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] Add documentation on serde [airflow]

2023-12-09 Thread via GitHub
eladkal commented on PR #35885: URL: https://github.com/apache/airflow/pull/35885#issuecomment-1848869720 > I believe generally for quite long time we've been following the patter to separte the "developer/contributor" documentation from user documentation. True. Contributor docs are

[PR] SparkSubmit Operator fixes [airflow]

2023-12-09 Thread via GitHub
pateash opened a new pull request, #36151: URL: https://github.com/apache/airflow/pull/36151 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)

Re: [I] Wrong try_number after rescheduling makes remote logging store all the logs with wrong log_id [airflow]

2023-12-09 Thread via GitHub
github-actions[bot] commented on issue #18080: URL: https://github.com/apache/airflow/issues/18080#issuecomment-1848877512 This issue has been closed because it has not received response from the issue author. -- This is an automated message from the Apache Git Service. To respond to the

Re: [I] Wrong try_number after rescheduling makes remote logging store all the logs with wrong log_id [airflow]

2023-12-09 Thread via GitHub
github-actions[bot] closed issue #18080: Wrong try_number after rescheduling makes remote logging store all the logs with wrong log_id URL: https://github.com/apache/airflow/issues/18080 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to G