Re: [PR] Fix compatibility of updated `airflow.io` with released providers [airflow]

2023-12-12 Thread via GitHub
potiuk commented on code in PR #36186: URL: https://github.com/apache/airflow/pull/36186#discussion_r1424966653 ## airflow/io/__init__.py: ## @@ -86,7 +93,13 @@ def get_fs( raise ValueError(f"No filesystem registered for scheme {scheme}") from None options =

(airflow) branch main updated: Improve CI.rst document clarity. (#36181)

2023-12-12 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 82b3c11277 Improve CI.rst document clarity.

Re: [PR] Improve CI.rst document clarity. [airflow]

2023-12-12 Thread via GitHub
potiuk merged PR #36181: URL: https://github.com/apache/airflow/pull/36181 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Improve CI.rst document clarity. [airflow]

2023-12-12 Thread via GitHub
potiuk commented on PR #36181: URL: https://github.com/apache/airflow/pull/36181#issuecomment-1853413106 Nice. @UncertainBadg3r - I wonder how you prepared the PR - did you use some LLMS or other tools to generate the clearer version of the description ? Or was that written by hand ? --

Re: [PR] adding triggerer option in breeze command [airflow]

2023-12-12 Thread via GitHub
potiuk commented on PR #36063: URL: https://github.com/apache/airflow/pull/36063#issuecomment-1853405709 Yeah. Apologies I missed that 臘 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] Create CLI command `airflow providers auth-managers` [airflow]

2023-12-12 Thread via GitHub
potiuk commented on issue #36193: URL: https://github.com/apache/airflow/issues/36193#issuecomment-1853406263 Sure :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

(airflow) annotated tag 2.8.0rc2 updated (34791d72d6 -> 0e8ee5bc2a)

2023-12-12 Thread ephraimanierobi
This is an automated email from the ASF dual-hosted git repository. ephraimanierobi pushed a change to annotated tag 2.8.0rc2 in repository https://gitbox.apache.org/repos/asf/airflow.git *** WARNING: tag 2.8.0rc2 was modified! *** from 34791d72d6 (commit) to 0e8ee5bc2a (tag)

Re: [PR] Add field for specifying other Airflow version in bug template [airflow]

2023-12-12 Thread via GitHub
eladkal commented on code in PR #36197: URL: https://github.com/apache/airflow/pull/36197#discussion_r1424897286 ## .github/ISSUE_TEMPLATE/airflow_bug_report.yml: ## @@ -31,17 +31,26 @@ body: - "Other Airflow 2 version (please specify below)" validations:

svn commit: r66027 - /dev/airflow/2.8.0rc1/

2023-12-12 Thread ephraimanierobi
Author: ephraimanierobi Date: Wed Dec 13 06:18:41 2023 New Revision: 66027 Log: Remove old release: 2.8.0rc1 Removed: dev/airflow/2.8.0rc1/

svn commit: r66026 - /dev/airflow/2.8.0rc2/

2023-12-12 Thread ephraimanierobi
Author: ephraimanierobi Date: Wed Dec 13 06:18:08 2023 New Revision: 66026 Log: Add artifacts for Airflow 2.8.0rc2 Added: dev/airflow/2.8.0rc2/ dev/airflow/2.8.0rc2/apache-airflow-2.8.0-source.tar.gz (with props) dev/airflow/2.8.0rc2/apache-airflow-2.8.0-source.tar.gz.asc

(airflow) annotated tag constraints-2.8.0rc2 updated (35ebec1d9a -> 619fcf9acd)

2023-12-12 Thread ephraimanierobi
This is an automated email from the ASF dual-hosted git repository. ephraimanierobi pushed a change to annotated tag constraints-2.8.0rc2 in repository https://gitbox.apache.org/repos/asf/airflow.git *** WARNING: tag constraints-2.8.0rc2 was modified! *** from 35ebec1d9a (commit) to

Re: [PR] Add create_or_replace_objects method to weaviate provider [airflow]

2023-12-12 Thread via GitHub
sunank200 commented on code in PR #36177: URL: https://github.com/apache/airflow/pull/36177#discussion_r1424884362 ## airflow/providers/weaviate/hooks/weaviate.py: ## @@ -396,28 +399,46 @@ def batch_data( .. seealso:: `batch_config_params options

(airflow) branch v2-8-stable updated (d0c1c452ce -> 34791d72d6)

2023-12-12 Thread ephraimanierobi
This is an automated email from the ASF dual-hosted git repository. ephraimanierobi pushed a change to branch v2-8-stable in repository https://gitbox.apache.org/repos/asf/airflow.git from d0c1c452ce Update RELEASE_NOTES.rst add 34791d72d6 Fix compatibility of updated `airflow.io` with

Re: [PR] Sync v2-8-stable with v2-7-test to release 2.8.0 [airflow]

2023-12-12 Thread via GitHub
ephraimbuddy merged PR #36189: URL: https://github.com/apache/airflow/pull/36189 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[PR] Update `retry` param typing in PubSubAsyncHook [airflow]

2023-12-12 Thread via GitHub
josh-fell opened a new pull request, #36198: URL: https://github.com/apache/airflow/pull/36198 The `google-cloud-pubsub` library was [recently released](https://pypi.org/project/google-cloud-pubsub/2.19.0/) with a bug fix to update the `retry` parameter when using the SubscriberAsyncClient

Re: [PR] adding triggerer option in breeze command [airflow]

2023-12-12 Thread via GitHub
o-nikolas commented on PR #36063: URL: https://github.com/apache/airflow/pull/36063#issuecomment-1853290953 > ah can i work on this part of starting the triggerer with system-tests @potiuk @o-nikolas ? > > Edit: I will open it as a different PR and submit the fix Yes please!

Re: [I] Create CLI command `airflow providers auth-managers` [airflow]

2023-12-12 Thread via GitHub
Bowrna commented on issue #36193: URL: https://github.com/apache/airflow/issues/36193#issuecomment-1853279483 Can i pick this issue too? @potiuk @vincbeck -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] adding triggerer option in breeze command [airflow]

2023-12-12 Thread via GitHub
Bowrna commented on PR #36063: URL: https://github.com/apache/airflow/pull/36063#issuecomment-1853278229 ah can i work on this part of starting the triggerer with system-tests @potiuk @o-nikolas ? -- This is an automated message from the Apache Git Service. To respond to the message,

[PR] Add field for specifying other Airflow version in bug template [airflow]

2023-12-12 Thread via GitHub
josh-fell opened a new pull request, #36197: URL: https://github.com/apache/airflow/pull/36197 The current Airflow bug template asks users to specify other Airflow 2.X versions "below" if not the latest pre-release, stable release, or `main`, but never setting aside a place for this input.

Re: [PR] Improve CI.rst document clarity. [airflow]

2023-12-12 Thread via GitHub
UncertainBadg3r commented on PR #36181: URL: https://github.com/apache/airflow/pull/36181#issuecomment-1853226292 > Static checks are failing, but this looks great! Ah, trailing whitespace! It should all be removed now. -- This is an automated message from the Apache Git Service.

(airflow) branch main updated: fix connection type webhdfs (#36145)

2023-12-12 Thread joshfell
This is an automated email from the ASF dual-hosted git repository. joshfell pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 6ee4d402e5 fix connection type webhdfs (#36145)

Re: [PR] fix connection type webhdfs [airflow]

2023-12-12 Thread via GitHub
josh-fell merged PR #36145: URL: https://github.com/apache/airflow/pull/36145 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Add create_or_replace_objects method to weaviate provider [airflow]

2023-12-12 Thread via GitHub
josh-fell commented on code in PR #36177: URL: https://github.com/apache/airflow/pull/36177#discussion_r1424807715 ## airflow/providers/weaviate/hooks/weaviate.py: ## @@ -606,3 +627,220 @@ def object_exists(self, uuid: str | UUID, **kwargs) -> bool: """

(airflow) branch main updated: Fix typo in dependency approach chapter of README (#36196)

2023-12-12 Thread joshfell
This is an automated email from the ASF dual-hosted git repository. joshfell pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 09b6ab9081 Fix typo in dependency approach

Re: [PR] fix typo in README.md [airflow]

2023-12-12 Thread via GitHub
josh-fell merged PR #36196: URL: https://github.com/apache/airflow/pull/36196 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Implement AthenaSQLHook [airflow]

2023-12-12 Thread via GitHub
josh-fell commented on code in PR #36171: URL: https://github.com/apache/airflow/pull/36171#discussion_r1424797740 ## airflow/providers/amazon/aws/hooks/athena_sql.py: ## @@ -0,0 +1,107 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor

[PR] fix typo in README.md [airflow]

2023-12-12 Thread via GitHub
cabbagepatchman opened a new pull request, #36196: URL: https://github.com/apache/airflow/pull/36196 Noticed a minor typo while reviewing the readme -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Syedahsn/aws executors ecs credentials refresh [airflow]

2023-12-12 Thread via GitHub
syedahsn commented on code in PR #36179: URL: https://github.com/apache/airflow/pull/36179#discussion_r1424742197 ## airflow/providers/amazon/aws/utils/exponential_backoff_retry.py: ## @@ -0,0 +1,60 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more

Re: [PR] Syedahsn/aws executors ecs credentials refresh [airflow]

2023-12-12 Thread via GitHub
syedahsn commented on code in PR #36179: URL: https://github.com/apache/airflow/pull/36179#discussion_r1424741783 ## airflow/providers/amazon/aws/utils/exponential_backoff_retry.py: ## @@ -0,0 +1,60 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more

Re: [PR] Syedahsn/aws executors ecs credentials refresh [airflow]

2023-12-12 Thread via GitHub
syedahsn commented on code in PR #36179: URL: https://github.com/apache/airflow/pull/36179#discussion_r1424732236 ## airflow/providers/amazon/aws/executors/ecs/ecs_executor.py: ## @@ -288,6 +333,9 @@ def attempt_task_runs(self): _failure_reasons = []

Re: [I] dag_processing.processes should be a gauge, not a counter [airflow]

2023-12-12 Thread via GitHub
potiuk commented on issue #36188: URL: https://github.com/apache/airflow/issues/36188#issuecomment-1853104463 cc: @ferruzzi ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] ExternalTaskSensor doesn't timeout if external DAG doesn't exist - dag.test() [airflow]

2023-12-12 Thread via GitHub
cbuffett commented on issue #34497: URL: https://github.com/apache/airflow/issues/34497#issuecomment-1853083625 Unfortunately I'm still seeing the infinite rescheduling issue after upgrading to Airflow 2.7.2. I believe the issue still revolves around the fact that the following code

Re: [PR] Add TabularVttsSensor [airflow]

2023-12-12 Thread via GitHub
github-actions[bot] commented on PR #32786: URL: https://github.com/apache/airflow/pull/32786#issuecomment-1853061371 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for

Re: [PR] Replace numpy example with practical exercise demonstrating top-level code [airflow]

2023-12-12 Thread via GitHub
github-actions[bot] closed pull request #35097: Replace numpy example with practical exercise demonstrating top-level code URL: https://github.com/apache/airflow/pull/35097 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

(airflow) branch main updated: Improve/simplify/remove superfluous flags in release provider process (#36195)

2023-12-12 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 4a4913c6f7 Improve/simplify/remove superfluous

Re: [PR] Improve/simplify/remove superfluous flags in release provider process [airflow]

2023-12-12 Thread via GitHub
potiuk merged PR #36195: URL: https://github.com/apache/airflow/pull/36195 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[PR] Improve/simplify/remove superfluous flags in release provider process [airflow]

2023-12-12 Thread via GitHub
potiuk opened a new pull request, #36195: URL: https://github.com/apache/airflow/pull/36195 Few small nits in commands used for relase process for providers --- **^ Add meaningful description above** Read the **[Pull Request

Re: [I] Add unit tests to retrieve fsspec from providers including backwards compatibility [airflow]

2023-12-12 Thread via GitHub
potiuk commented on issue #36187: URL: https://github.com/apache/airflow/issues/36187#issuecomment-1852905056 > We should indeed test, but should we not yank the older released providers? This seems over doing the backwards compat (and maintaining it!) While the feature , somewhat

Re: [I] Add unit tests to retrieve fsspec from providers including backwards compatibility [airflow]

2023-12-12 Thread via GitHub
potiuk commented on issue #36187: URL: https://github.com/apache/airflow/issues/36187#issuecomment-1852897234 > Well maybe not as they are existing providers, unlike common.io. Still it feels weird. If a change ends up in a provider before generally usable and still being worked on we

Re: [PR] Fix compatibility of updated `airflow.io` with released providers [airflow]

2023-12-12 Thread via GitHub
potiuk commented on code in PR #36186: URL: https://github.com/apache/airflow/pull/36186#discussion_r1424629695 ## airflow/io/__init__.py: ## @@ -86,7 +93,13 @@ def get_fs( raise ValueError(f"No filesystem registered for scheme {scheme}") from None options =

Re: [PR] Fix compatibility of updated `airflow.io` with released providers [airflow]

2023-12-12 Thread via GitHub
potiuk commented on code in PR #36186: URL: https://github.com/apache/airflow/pull/36186#discussion_r1424629695 ## airflow/io/__init__.py: ## @@ -86,7 +93,13 @@ def get_fs( raise ValueError(f"No filesystem registered for scheme {scheme}") from None options =

Re: [PR] Fix compatibility of updated `airflow.io` with released providers [airflow]

2023-12-12 Thread via GitHub
bolkedebruin commented on code in PR #36186: URL: https://github.com/apache/airflow/pull/36186#discussion_r1424623825 ## airflow/io/__init__.py: ## @@ -86,7 +93,13 @@ def get_fs( raise ValueError(f"No filesystem registered for scheme {scheme}") from None

[I] Status of testing Providers that were prepared on December 12, 2023 [airflow]

2023-12-12 Thread via GitHub
potiuk opened a new issue, #36194: URL: https://github.com/apache/airflow/issues/36194 ### Body I have a kind request for all the contributors to the latest provider packages release. Could you please help us to test the RC versions of the providers? The guidelines on how

Re: [I] Add unit tests to retrieve fsspec from providers including backwards compatibility [airflow]

2023-12-12 Thread via GitHub
bolkedebruin commented on issue #36187: URL: https://github.com/apache/airflow/issues/36187#issuecomment-1852883499 We should indeed test, but should we not yank the older released providers? This seems over doing the backwards compat (and maintaining it!) While the feature , somewhat

[PR] Add documentation for packages - 2023-12-12 [airflow-site]

2023-12-12 Thread via GitHub
potiuk opened a new pull request, #908: URL: https://github.com/apache/airflow-site/pull/908 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

(airflow-site) branch add-documentation-2023-12-12 created (now 1f579da82f)

2023-12-12 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch add-documentation-2023-12-12 in repository https://gitbox.apache.org/repos/asf/airflow-site.git at 1f579da82f Add documentation for packages - 2023-12-12 This branch includes the

Re: [PR] Fix compatibility of updated `airflow.io` with released providers [airflow]

2023-12-12 Thread via GitHub
bolkedebruin commented on PR #36186: URL: https://github.com/apache/airflow/pull/36186#issuecomment-1852873466 Mmm yes I understand. Sorry for this, I mentioned when merging the other it would require a release of the providers and wanted to prevent backwards compat check (which is silly

(airflow) branch main updated: Add capability of passing a bucket to `example_emr` system test (#36192)

2023-12-12 Thread vincbeck
This is an automated email from the ASF dual-hosted git repository. vincbeck pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 6f57ee1256 Add capability of passing a bucket to

Re: [PR] Add capability of passing a bucket to `example_emr` system test [airflow]

2023-12-12 Thread via GitHub
vincbeck merged PR #36192: URL: https://github.com/apache/airflow/pull/36192 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

svn commit: r66014 - /dev/airflow/providers/

2023-12-12 Thread potiuk
Author: potiuk Date: Tue Dec 12 21:10:16 2023 New Revision: 66014 Log: Add artifacts for Airflow Providers 2023-12-12 Added: dev/airflow/providers/apache_airflow_providers_daskexecutor-1.1.1-py3-none-any.whl (with props)

svn commit: r66013 - /release/airflow/KEYS

2023-12-12 Thread potiuk
Author: potiuk Date: Tue Dec 12 21:03:56 2023 New Revision: 66013 Log: Add new PGP key of Jarek Potiuk Modified: release/airflow/KEYS Modified: release/airflow/KEYS == --- release/airflow/KEYS (original) +++

(airflow) branch main updated: Prepare docs 1st wave of Providers December 2023 RC2 (#36190)

2023-12-12 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 64931b1a65 Prepare docs 1st wave of Providers

Re: [PR] Prepare docs 1st wave of Providers December 2023 RC2 [airflow]

2023-12-12 Thread via GitHub
potiuk merged PR #36190: URL: https://github.com/apache/airflow/pull/36190 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [I] AIP-56 - FAB AM - Move FAB auth manager to new provider [airflow]

2023-12-12 Thread via GitHub
vincbeck commented on issue #32210: URL: https://github.com/apache/airflow/issues/32210#issuecomment-1852735740 I was just starting this task but I did not think about the consequences. If we deprecate configs in the `webserver` section in core Airflow and copy these to the FAB provider

[I] Create CLI command `airflow providers auth-managers` [airflow]

2023-12-12 Thread via GitHub
vincbeck opened a new issue, #36193: URL: https://github.com/apache/airflow/issues/36193 ### Description Providers can provide custom auth managers. It would be interesting that, in the same fashion as `airflow providers executors`, to have a CLI command `airflow providers

Re: [PR] Prepare docs 1st wave of Providers December 2023 RC2 [airflow]

2023-12-12 Thread via GitHub
potiuk commented on code in PR #36190: URL: https://github.com/apache/airflow/pull/36190#discussion_r1424513423 ## airflow/providers/daskexecutor/__init__.py: ## @@ -27,16 +27,16 @@ __all__ = ["__version__"] -__version__ = "1.1.0" +__version__ = "1.2.0" try: from

Re: [PR] Prepare docs 1st wave of Providers December 2023 RC2 [airflow]

2023-12-12 Thread via GitHub
eladkal commented on code in PR #36190: URL: https://github.com/apache/airflow/pull/36190#discussion_r1424507863 ## airflow/providers/daskexecutor/__init__.py: ## @@ -27,16 +27,16 @@ __all__ = ["__version__"] -__version__ = "1.1.0" +__version__ = "1.2.0" try: from

(airflow) branch main updated: Avoid bumping min-airflow-version in prepare-provider-documentation (#36191)

2023-12-12 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new c2f440d827 Avoid bumping min-airflow-version in

Re: [PR] Avoid bumping removed min-airflow-version in prepare-provider-documentation [airflow]

2023-12-12 Thread via GitHub
potiuk merged PR #36191: URL: https://github.com/apache/airflow/pull/36191 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Prepare docs 1st wave of Providers December 2023 RC2 [airflow]

2023-12-12 Thread via GitHub
eladkal commented on code in PR #36190: URL: https://github.com/apache/airflow/pull/36190#discussion_r1424483662 ## airflow/providers/daskexecutor/CHANGELOG.rst: ## @@ -27,11 +27,29 @@ Changelog - -1.1.1 +1.2.0 Review Comment: if

Re: [PR] Fix daskexecutor provider docs [airflow-site]

2023-12-12 Thread via GitHub
eladkal closed pull request #907: Fix daskexecutor provider docs URL: https://github.com/apache/airflow-site/pull/907 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Convert DagFileProcessor.execute_callbacks to Internal API [airflow]

2023-12-12 Thread via GitHub
vincbeck commented on code in PR #28900: URL: https://github.com/apache/airflow/pull/28900#discussion_r1424467194 ## airflow/models/dag.py: ## @@ -1383,8 +1384,43 @@ def normalized_schedule_interval(self) -> ScheduleInterval: _schedule_interval =

Re: [PR] Fix daskexecutor provider docs [airflow-site]

2023-12-12 Thread via GitHub
eladkal commented on PR #907: URL: https://github.com/apache/airflow-site/pull/907#issuecomment-1852652281 Following our discussion closed due to https://github.com/apache/airflow/pull/36191 Thanks @potiuk -- This is an automated message from the Apache Git Service. To respond to

Re: [I] Add a command line option to start a triggerer process with breeze [airflow]

2023-12-12 Thread via GitHub
potiuk commented on issue #31826: URL: https://github.com/apache/airflow/issues/31826#issuecomment-1852647677 Reopened as @o-nikolas mentioned the idea here was for SYSTEM TESTS to support triggerer - see https://github.com/apache/airflow/pull/36063#issuecomment-1852622579 -- This is an

Re: [PR] adding triggerer option in breeze command [airflow]

2023-12-12 Thread via GitHub
potiuk commented on PR #36063: URL: https://github.com/apache/airflow/pull/36063#issuecomment-1852646164 Ah No. Indeed it's only start-airflow 臘 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[I] Add a command line option to start a triggerer process with breeze [airflow]

2023-12-12 Thread via GitHub
vandonr-amz opened a new issue, #31826: URL: https://github.com/apache/airflow/issues/31826 ### Description As discussed in https://lists.apache.org/thread/ss0605b561bcrrlogydgww1pzo97691j It would be helpful to be able to start a triggerer process from the same command line that

[PR] Add capability of passing a bucket to `example_emr` system test [airflow]

2023-12-12 Thread via GitHub
vincbeck opened a new pull request, #36192: URL: https://github.com/apache/airflow/pull/36192 Sometimes the system test `example_emr` is failing because the underlying EMR step is failing. In such case, for debugging purposes, we might want to keep the bucket. Adding an option to pass a

[PR] Avoid bumping min-airflow-version in prepare-provider-documentation [airflow]

2023-12-12 Thread via GitHub
potiuk opened a new pull request, #36191: URL: https://github.com/apache/airflow/pull/36191 The change #36160 included removed providers in documentation generation - which is unnecesary and we should not do it. This PR avoids doing it for removed providers.

Re: [PR] adding triggerer option in breeze command [airflow]

2023-12-12 Thread via GitHub
o-nikolas commented on PR #36063: URL: https://github.com/apache/airflow/pull/36063#issuecomment-1852622579 But will this start the triggerer for system tests as well? Or just when using start-airflow? -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] Prepare docs 1st wave of Providers December 2023 RC2 [airflow]

2023-12-12 Thread via GitHub
potiuk commented on code in PR #36190: URL: https://github.com/apache/airflow/pull/36190#discussion_r1424434055 ## airflow/providers/daskexecutor/CHANGELOG.rst: ## @@ -27,11 +27,29 @@ Changelog - -1.1.1 +1.2.0 Review Comment: See the discussion in

Re: [I] logging/remote_logging - option to disable creation of cloudwatch log groups [airflow]

2023-12-12 Thread via GitHub
o-nikolas commented on issue #35164: URL: https://github.com/apache/airflow/issues/35164#issuecomment-1852604590 Closing this one for now, since we lost engagement a bit. If we see this again we could think about adding more logging, but so far we just have one data point. -- This is an

Re: [PR] Remove duplicate dropdown entry in providers issue template [airflow]

2023-12-12 Thread via GitHub
potiuk commented on PR #36180: URL: https://github.com/apache/airflow/pull/36180#issuecomment-1852596743 > I had a feeling that the very mention of "we should add this to CI" would trigger a Jarek-special PR.  Very much so :D -- This is an automated message from the Apache Git

Re: [PR] Fix daskexecutor provider docs [airflow-site]

2023-12-12 Thread via GitHub
potiuk commented on PR #907: URL: https://github.com/apache/airflow-site/pull/907#issuecomment-1852591497 > That means modifying pre-commit and enabling 2.5.0 back Does it? I think we do not do any pre-commit that would prevent dask executor to keep 2.5.0 (but maybe I am wrong and

Re: [I] Add AthenaSqlHook based on PyAthena [airflow]

2023-12-12 Thread via GitHub
shubham22 commented on issue #34823: URL: https://github.com/apache/airflow/issues/34823#issuecomment-1852585468 @treyyi - please let us know if you're looking into this. If not Treyyi@, @flolas, would you have time to implement this? -- This is an automated message from the Apache Git

Re: [PR] Syedahsn/aws executors ecs credentials refresh [airflow]

2023-12-12 Thread via GitHub
o-nikolas commented on code in PR #36179: URL: https://github.com/apache/airflow/pull/36179#discussion_r1424417256 ## airflow/providers/amazon/aws/utils/exponential_backoff_retry.py: ## @@ -0,0 +1,60 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more

Re: [PR] Fix daskexecutor provider docs [airflow-site]

2023-12-12 Thread via GitHub
eladkal commented on PR #907: URL: https://github.com/apache/airflow-site/pull/907#issuecomment-1852574305 > We should not bump min-airflow version for this package - it should be identical as the last released version with the only difference being the description in PyPI. There is no

Re: [PR] Remove duplicate dropdown entry in providers issue template [airflow]

2023-12-12 Thread via GitHub
josh-fell commented on PR #36180: URL: https://github.com/apache/airflow/pull/36180#issuecomment-1852573398 Thanks for whipping that up @potiuk! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Remove duplicate dropdown entry in providers issue template [airflow]

2023-12-12 Thread via GitHub
josh-fell commented on PR #36180: URL: https://github.com/apache/airflow/pull/36180#issuecomment-1852572656 I had a feeling that the very mention of "we should add this to CI" would trigger a Jarek-special PR.  -- This is an automated message from the Apache Git Service. To respond to

Re: [PR] Remove duplicate dropdown entry in providers issue template [airflow]

2023-12-12 Thread via GitHub
josh-fell closed pull request #36180: Remove duplicate dropdown entry in providers issue template URL: https://github.com/apache/airflow/pull/36180 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Fix daskexecutor provider docs [airflow-site]

2023-12-12 Thread via GitHub
potiuk commented on PR #907: URL: https://github.com/apache/airflow-site/pull/907#issuecomment-1852571177 Ah then 1.2.0 is wrong. We should not bump Airflow min version for daskexecutor. Ths "upgrade min airflow" version will sklp bumping it at least. We should not bump min-airflow

Re: [PR] Fix daskexecutor provider docs [airflow-site]

2023-12-12 Thread via GitHub
potiuk commented on PR #907: URL: https://github.com/apache/airflow-site/pull/907#issuecomment-1852565967 So both - releasing 1.1.1 package and documentation for it are absolutely intended and needed. -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] Fix daskexecutor provider docs [airflow-site]

2023-12-12 Thread via GitHub
eladkal commented on PR #907: URL: https://github.com/apache/airflow-site/pull/907#issuecomment-1852566766 > Well. We SHOULD release daskexecutor 1.1.1 as part of this wave. That's the whole point of having the last release which will have "removed" notice in the `PyPI` description - there

Re: [PR] Fix daskexecutor provider docs [airflow-site]

2023-12-12 Thread via GitHub
potiuk commented on PR #907: URL: https://github.com/apache/airflow-site/pull/907#issuecomment-1852564367 Well. We SHOULD release daskexecutor 1.1.1 as part of this wave. That's the whole point of having the last release which will have "removed" notice in the `PyPI` description - there is

Re: [PR] Prepare docs 1st wave of Providers December 2023 RC2 [airflow]

2023-12-12 Thread via GitHub
eladkal closed pull request #36190: Prepare docs 1st wave of Providers December 2023 RC2 URL: https://github.com/apache/airflow/pull/36190 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[PR] Prepare docs 1st wave of Providers December 2023 RC2 [airflow]

2023-12-12 Thread via GitHub
eladkal opened a new pull request, #36190: URL: https://github.com/apache/airflow/pull/36190 Providers for RC2: daskexecutor, databricks, docker, google, odbc -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[PR] Fix daskexecutor provider docs [airflow-site]

2023-12-12 Thread via GitHub
eladkal opened a new pull request, #907: URL: https://github.com/apache/airflow-site/pull/907 There is no daskexecutor 1.1.1 It seems that doc build created this package though it wasn't part of the release wave - I will investigate and fix the problem with the doc build -- This is an

(airflow-site) branch dask created (now b70120e764)

2023-12-12 Thread eladkal
This is an automated email from the ASF dual-hosted git repository. eladkal pushed a change to branch dask in repository https://gitbox.apache.org/repos/asf/airflow-site.git at b70120e764 Fix daskexecutor provider docs This branch includes the following new commits: new b70120e764

(airflow) branch constraints-2-8 updated: Updating constraints. Github run id:7184642982

2023-12-12 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-2-8 in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-2-8 by this push: new c96b5c79b3 Updating

[PR] Sync v2-8-stable with v2-7-test to release 2.8.0 [airflow]

2023-12-12 Thread via GitHub
ephraimbuddy opened a new pull request, #36189: URL: https://github.com/apache/airflow/pull/36189 Time for `2.8.0rc2` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

(airflow) branch v2-8-test updated: Fix compatibility of updated `airflow.io` with released providers (#36186)

2023-12-12 Thread ephraimanierobi
This is an automated email from the ASF dual-hosted git repository. ephraimanierobi pushed a commit to branch v2-8-test in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/v2-8-test by this push: new 34791d72d6 Fix compatibility of

(airflow) branch main updated: Fix compatibility of updated `airflow.io` with released providers (#36186)

2023-12-12 Thread ephraimanierobi
This is an automated email from the ASF dual-hosted git repository. ephraimanierobi pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 7799c51e15 Fix compatibility of updated

Re: [PR] Fix compatibility of updated `airflow.io` with released providers [airflow]

2023-12-12 Thread via GitHub
ephraimbuddy merged PR #36186: URL: https://github.com/apache/airflow/pull/36186 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Add create_or_replace_objects method to weaviate provider [airflow]

2023-12-12 Thread via GitHub
mpgreg commented on code in PR #36177: URL: https://github.com/apache/airflow/pull/36177#discussion_r1424283026 ## airflow/providers/weaviate/hooks/weaviate.py: ## @@ -606,3 +627,220 @@ def object_exists(self, uuid: str | UUID, **kwargs) -> bool: """ client =

Re: [PR] Add create_or_replace_objects method to weaviate provider [airflow]

2023-12-12 Thread via GitHub
mpgreg commented on code in PR #36177: URL: https://github.com/apache/airflow/pull/36177#discussion_r1424109843 ## airflow/providers/weaviate/hooks/weaviate.py: ## @@ -396,28 +399,46 @@ def batch_data( .. seealso:: `batch_config_params options

[I] dag_processing.processes should be a gauge, not a counter [airflow]

2023-12-12 Thread via GitHub
Jeoffreybauvin opened a new issue, #36188: URL: https://github.com/apache/airflow/issues/36188 ### What do you see as an issue? I wasn't sure if it's a doc issue or a code issue. For me, dag_processing.processes should be considered as a gauge, not a counter, due to negative

Re: [I] Add unit tests to retrieve fsspec from providers including backwards compatibility [airflow]

2023-12-12 Thread via GitHub
potiuk commented on issue #36187: URL: https://github.com/apache/airflow/issues/36187#issuecomment-1852328549 cc: @bolkedebruin -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Fix compatibility of updated `airflow.io` with released providers [airflow]

2023-12-12 Thread via GitHub
potiuk commented on code in PR #36186: URL: https://github.com/apache/airflow/pull/36186#discussion_r1424225335 ## airflow/io/__init__.py: ## @@ -86,7 +93,13 @@ def get_fs( raise ValueError(f"No filesystem registered for scheme {scheme}") from None options =

Re: [PR] Fix compatibility of updated `airflow.io` with released providers [airflow]

2023-12-12 Thread via GitHub
potiuk commented on code in PR #36186: URL: https://github.com/apache/airflow/pull/36186#discussion_r1424216835 ## airflow/io/__init__.py: ## @@ -86,7 +93,13 @@ def get_fs( raise ValueError(f"No filesystem registered for scheme {scheme}") from None options =

Re: [PR] Fix compatibility of updated `airflow.io` with released providers [airflow]

2023-12-12 Thread via GitHub
ephraimbuddy commented on code in PR #36186: URL: https://github.com/apache/airflow/pull/36186#discussion_r1424205810 ## airflow/io/__init__.py: ## @@ -86,7 +93,13 @@ def get_fs( raise ValueError(f"No filesystem registered for scheme {scheme}") from None

[PR] Fix compatibility of updated `airflow.io` with released providers [airflow]

2023-12-12 Thread via GitHub
potiuk opened a new pull request, #36186: URL: https://github.com/apache/airflow/pull/36186 The released providers added support to previous version of the `airflow.io` - where options were not passed to `get_fs` method that provides Fsspec compatible FileSystem. However #35820 added

  1   2   >