Re: [I] max_active_tis_per_dag blocks tasks too aggressively when backfilling [airflow]

2023-10-03 Thread via GitHub
github-actions[bot] commented on issue #26078: URL: https://github.com/apache/airflow/issues/26078#issuecomment-1744324823 This issue has been closed because it has not received response from the issue author. -- This is an automated message from the Apache Git Service. To respond to the

Re: [I] max_active_tis_per_dag blocks tasks too aggressively when backfilling [airflow]

2023-10-03 Thread via GitHub
github-actions[bot] closed issue #26078: max_active_tis_per_dag blocks tasks too aggressively when backfilling URL: https://github.com/apache/airflow/issues/26078 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

Re: [PR] Fix Memorystore Memcached system test [airflow]

2023-10-03 Thread via GitHub
moiseenkov commented on PR #34447: URL: https://github.com/apache/airflow/pull/34447#issuecomment-1744344912 @eladkal , @kaxil , @Taragolis , hi, can we merge this please? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Fix system tests for PostgresToGCSOperator and MySQLToGCSOperator [airflow]

2023-10-03 Thread via GitHub
moiseenkov commented on PR #34502: URL: https://github.com/apache/airflow/pull/34502#issuecomment-1744345331 @eladkal , @kaxil , @Taragolis , hi, can we merge this please? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Refactor DataFusionInstanceLink usage [airflow]

2023-10-03 Thread via GitHub
moiseenkov commented on PR #34514: URL: https://github.com/apache/airflow/pull/34514#issuecomment-1744345481 @eladkal , @kaxil , @Taragolis , hi, can we merge this please? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Fix GCSToGoogleDriveOperator and gdrive system tests [airflow]

2023-10-03 Thread via GitHub
moiseenkov commented on PR #34545: URL: https://github.com/apache/airflow/pull/34545#issuecomment-1744345645 @eladkal , @kaxil , @Taragolis , hi, can we merge this please? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Fix Memorystore Memcached system test [airflow]

2023-10-03 Thread via GitHub
Taragolis merged PR #34447: URL: https://github.com/apache/airflow/pull/34447 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflo

[airflow] branch main updated: Fix Memorystore Memcached system test (#34447)

2023-10-03 Thread taragolis
This is an automated email from the ASF dual-hosted git repository. taragolis pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 78f7da46c3 Fix Memorystore Memcached system test

Re: [PR] feat: enable HTTPRoute for Gateway API [airflow]

2023-10-03 Thread via GitHub
Kyrremann commented on code in PR #34716: URL: https://github.com/apache/airflow/pull/34716#discussion_r1343624611 ## chart/values.yaml: ## @@ -220,6 +220,31 @@ ingress: # the name of a pre-created Secret containing a TLS private key and certificate secretName: ""

Re: [PR] feat: enable HTTPRoute for Gateway API [airflow]

2023-10-03 Thread via GitHub
Kyrremann commented on code in PR #34716: URL: https://github.com/apache/airflow/pull/34716#discussion_r1343629040 ## chart/templates/webserver/webserver-httproute.yaml: ## @@ -0,0 +1,82 @@ +{{/* + Licensed to the Apache Software Foundation (ASF) under one + or more contributor

Re: [PR] Fixed rows count in the migration script [airflow]

2023-10-03 Thread via GitHub
boring-cyborg[bot] commented on PR #34348: URL: https://github.com/apache/airflow/pull/34348#issuecomment-1744366725 Awesome work, congrats on your first merged pull request! You are invited to check our [Issue Tracker](https://github.com/apache/airflow/issues) for additional contributions.

Re: [PR] Fixed rows count in the migration script [airflow]

2023-10-03 Thread via GitHub
ephraimbuddy merged PR #34348: URL: https://github.com/apache/airflow/pull/34348 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@air

[airflow] branch main updated: Fixed rows count in the migration script (#34348)

2023-10-03 Thread ephraimanierobi
This is an automated email from the ASF dual-hosted git repository. ephraimanierobi pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new f349fda125 Fixed rows count in the migrati

Re: [PR] Add `deferred_as_active` param to consider deferred state as an active state [airflow]

2023-10-03 Thread via GitHub
uranusjr commented on PR #34700: URL: https://github.com/apache/airflow/pull/34700#issuecomment-1744375234 I wonder if we should have separate parameters for active + deferred instead of having a flag to switch between behaviours for the same flags. Not sure. -- This is an automated messa

Re: [I] Links to specific dagruns don't work when older than 25 runs [airflow]

2023-10-03 Thread via GitHub
hussein-awala commented on issue #34723: URL: https://github.com/apache/airflow/issues/34723#issuecomment-1744378524 I cannot reproduce with main: ![image](https://github.com/apache/airflow/assets/21311487/6c79f1a5-1805-4136-a599-edc3700ca1fe) ![image](https://github.com/apache/airflo

[I] Hiding Run Id and Logical date from trigger DAG UI [airflow]

2023-10-03 Thread via GitHub
hterik opened a new issue, #34726: URL: https://github.com/apache/airflow/issues/34726 ### Description With the new more user friendly Trigger UI (`/dags/mydag/trigger`), one can direct more users to using Airflow UI directly. However, the first two questions a user has to answ

Re: [I] Links to specific dagruns don't work when older than 25 runs [airflow]

2023-10-03 Thread via GitHub
hterik commented on issue #34723: URL: https://github.com/apache/airflow/issues/34723#issuecomment-1744403887 Sure, hope this is enough ![image](https://github.com/apache/airflow/assets/89977373/e34007db-2b20-41c3-bce5-90e8e8544417) The url was given by clicking a run in the **

Re: [PR] Check if the lower of provided values are sensitives in config endpoint [airflow]

2023-10-03 Thread via GitHub
ephraimbuddy commented on code in PR #34712: URL: https://github.com/apache/airflow/pull/34712#discussion_r1343679602 ## airflow/api_connexion/endpoints/config_endpoint.py: ## @@ -123,7 +123,7 @@ def get_value(section: str, option: str) -> Response: "Config not

Re: [PR] Open Search Provider [airflow]

2023-10-03 Thread via GitHub
eladkal commented on code in PR #34705: URL: https://github.com/apache/airflow/pull/34705#discussion_r1343689432 ## airflow/providers/opensearch/provider.yaml: ## @@ -0,0 +1,52 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreem

Re: [PR] Fix system tests for PostgresToGCSOperator and MySQLToGCSOperator [airflow]

2023-10-03 Thread via GitHub
Taragolis merged PR #34502: URL: https://github.com/apache/airflow/pull/34502 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflo

[airflow] branch main updated (f349fda125 -> fb21d093b8)

2023-10-03 Thread taragolis
This is an automated email from the ASF dual-hosted git repository. taragolis pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from f349fda125 Fixed rows count in the migration script (#34348) add fb21d093b8 Fix system tests for PostgresToGCSO

Re: [PR] feat: enable HTTPRoute for Gateway API [airflow]

2023-10-03 Thread via GitHub
Kyrremann commented on PR #34716: URL: https://github.com/apache/airflow/pull/34716#issuecomment-1744478997 @hussein-awala just to be sure, the helm tests are written in Python? So I'm guessing I can use `helm_tests/webserver/test_ingress_flower.py` as inspiration. Are we manually adding

Re: [I] Status of testing of Apache Airflow Helm Chart 1.11.0rc2 [airflow]

2023-10-03 Thread via GitHub
Francesko90 commented on issue #34660: URL: https://github.com/apache/airflow/issues/34660#issuecomment-1744510319 Tested #31868 and #34631. Works fine -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

[I] Impersonation logic missing in BigQuery Async operators [airflow]

2023-10-03 Thread via GitHub
hkc-8010 opened a new issue, #34727: URL: https://github.com/apache/airflow/issues/34727 ### Apache Airflow version main (development) ### What happened BigQueryAsyncOperators uses writing credentials to a file using `GoogleBaseHook.provide_gcp_credential_file_as_context

[PR] Bump min airflow version of providers [airflow]

2023-10-03 Thread via GitHub
eladkal opened a new pull request, #34728: URL: https://github.com/apache/airflow/pull/34728 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-

Re: [PR] Implement `is_authorized()` in auth manager [airflow]

2023-10-03 Thread via GitHub
utkarsharma2 commented on code in PR #33213: URL: https://github.com/apache/airflow/pull/33213#discussion_r1343788762 ## airflow/www/auth.py: ## @@ -35,16 +44,100 @@ def get_access_denied_message(): return conf.get("webserver", "access_denied_message") -def has_access(p

Re: [PR] Bump min airflow version of providers [airflow]

2023-10-03 Thread via GitHub
Taragolis commented on PR #34728: URL: https://github.com/apache/airflow/pull/34728#issuecomment-1744573040 I guess we also need to bump version in compatibility check CI job https://github.com/apache/airflow/blob/fb21d093b82a7f2f9a0c24016d795d39f945b872/.github/workflows/ci.yml#L729 --

Re: [PR] allows users to write dag_id and task_id in their national characters, added display name for dag / task [airflow]

2023-10-03 Thread via GitHub
uranusjr commented on PR #32520: URL: https://github.com/apache/airflow/pull/32520#issuecomment-1744604052 I tried to add a commit to address the error mentioned above, but there are multiple linting errors here and a merge conflict that must be fixed first. -- This is an automated messag

[airflow] branch main updated: Parse 'docker context ls --format=json' correctly (#34711)

2023-10-03 Thread uranusjr
This is an automated email from the ASF dual-hosted git repository. uranusjr pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 19284981f8 Parse 'docker context ls --format=json

Re: [PR] Parse 'docker context ls --format=json' correctly [airflow]

2023-10-03 Thread via GitHub
uranusjr merged PR #34711: URL: https://github.com/apache/airflow/pull/34711 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow

Re: [PR] Sqlalchemy 2.0 changes to `DagFileProcessorManager` [airflow]

2023-10-03 Thread via GitHub
uranusjr commented on code in PR #34665: URL: https://github.com/apache/airflow/pull/34665#discussion_r1343831841 ## airflow/dag_processing/manager.py: ## @@ -526,13 +527,11 @@ def deactivate_stale_dags( Deactivate them and remove them in the serialized_dag table.

Re: [PR] Sqlalchemy 2.0 changes to `DagFileProcessorManager` [airflow]

2023-10-03 Thread via GitHub
uranusjr commented on code in PR #34665: URL: https://github.com/apache/airflow/pull/34665#discussion_r1343831841 ## airflow/dag_processing/manager.py: ## @@ -526,13 +527,11 @@ def deactivate_stale_dags( Deactivate them and remove them in the serialized_dag table.

Re: [PR] Sqlalchemy 2.0 changes to `DagFileProcessorManager` [airflow]

2023-10-03 Thread via GitHub
uranusjr commented on code in PR #34665: URL: https://github.com/apache/airflow/pull/34665#discussion_r1343831841 ## airflow/dag_processing/manager.py: ## @@ -526,13 +527,11 @@ def deactivate_stale_dags( Deactivate them and remove them in the serialized_dag table.

Re: [PR] Sqlalchemy 2.0 changes to `DagFileProcessorManager` [airflow]

2023-10-03 Thread via GitHub
uranusjr commented on code in PR #34665: URL: https://github.com/apache/airflow/pull/34665#discussion_r1343835869 ## airflow/dag_processing/manager.py: ## @@ -547,11 +546,13 @@ def deactivate_stale_dags( to_deactivate.add(dag.dag_id) if to_deactivate:

Re: [PR] Sqlalchemy 2.0 changes to `DagFileProcessorManager` [airflow]

2023-10-03 Thread via GitHub
uranusjr commented on code in PR #34665: URL: https://github.com/apache/airflow/pull/34665#discussion_r1343839914 ## airflow/dag_processing/manager.py: ## @@ -695,15 +696,16 @@ def _fetch_callbacks_with_retries(self, max_callbacks: int, session: Session): """Fetch call

Re: [PR] Sqlalchemy 2.0 changes to `DagFileProcessorManager` [airflow]

2023-10-03 Thread via GitHub
uranusjr commented on code in PR #34665: URL: https://github.com/apache/airflow/pull/34665#discussion_r1343839914 ## airflow/dag_processing/manager.py: ## @@ -695,15 +696,16 @@ def _fetch_callbacks_with_retries(self, max_callbacks: int, session: Session): """Fetch call

Re: [PR] Use timeout when inntializing _timeout_sec of TaskStateTrigger [airflow]

2023-10-03 Thread via GitHub
Xiroo commented on code in PR #34715: URL: https://github.com/apache/airflow/pull/34715#discussion_r1343843951 ## airflow/sensors/external_task.py: ## @@ -341,6 +341,7 @@ def execute(self, context: Context) -> None: states=self.allowed_states,

Re: [PR] Use timeout when inntializing _timeout_sec of TaskStateTrigger [airflow]

2023-10-03 Thread via GitHub
Xiroo commented on code in PR #34715: URL: https://github.com/apache/airflow/pull/34715#discussion_r1343843951 ## airflow/sensors/external_task.py: ## @@ -341,6 +341,7 @@ def execute(self, context: Context) -> None: states=self.allowed_states,

Re: [PR] Use timeout when inntializing _timeout_sec of TaskStateTrigger [airflow]

2023-10-03 Thread via GitHub
Xiroo commented on code in PR #34715: URL: https://github.com/apache/airflow/pull/34715#discussion_r1343843951 ## airflow/sensors/external_task.py: ## @@ -341,6 +341,7 @@ def execute(self, context: Context) -> None: states=self.allowed_states,

[PR] WIP: Add Airflow FS [airflow]

2023-10-03 Thread via GitHub
bolkedebruin opened a new pull request, #34729: URL: https://github.com/apache/airflow/pull/34729 This adds Airflow FS, which is a generic abstraction on top of a filesystem or object storage. The idea is that you can do the following ```python import airflow.io.fs as afs af

Re: [PR] Use timeout when inntializing _timeout_sec of TaskStateTrigger [airflow]

2023-10-03 Thread via GitHub
Xiroo commented on code in PR #34715: URL: https://github.com/apache/airflow/pull/34715#discussion_r1343843951 ## airflow/sensors/external_task.py: ## @@ -341,6 +341,7 @@ def execute(self, context: Context) -> None: states=self.allowed_states,

Re: [PR] Use timeout when inntializing _timeout_sec of TaskStateTrigger [airflow]

2023-10-03 Thread via GitHub
Xiroo commented on code in PR #34715: URL: https://github.com/apache/airflow/pull/34715#discussion_r1343843951 ## airflow/sensors/external_task.py: ## @@ -341,6 +341,7 @@ def execute(self, context: Context) -> None: states=self.allowed_states,

Re: [PR] WIP: Add Airflow FS [airflow]

2023-10-03 Thread via GitHub
eladkal commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1343924059 ## airflow/io/__init__.py: ## Review Comment: this seems to be mostly used by providers (for transfer operators). so maybe it's better to add it as `common.fs

Re: [PR] WIP: Add Airflow FS [airflow]

2023-10-03 Thread via GitHub
bolkedebruin commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1343934484 ## airflow/io/__init__.py: ## Review Comment: That would makes sense, but the intention is to make xcom and Dag parsing work on top of it -- This is an

Re: [PR] WIP: Add Airflow FS [airflow]

2023-10-03 Thread via GitHub
bolkedebruin commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1343934484 ## airflow/io/__init__.py: ## Review Comment: That would makes sense, but the intention is to make xcom and Dag parsing work on top of it (at least have th

Re: [PR] Sqlalchemy 2.0 changes to `DagFileProcessorManager` [airflow]

2023-10-03 Thread via GitHub
phanikumv commented on code in PR #34665: URL: https://github.com/apache/airflow/pull/34665#discussion_r1343958429 ## airflow/dag_processing/manager.py: ## @@ -547,11 +546,13 @@ def deactivate_stale_dags( to_deactivate.add(dag.dag_id) if to_deactivate

Re: [PR] Sqlalchemy 2.0 changes to `DagFileProcessorManager` [airflow]

2023-10-03 Thread via GitHub
phanikumv commented on code in PR #34665: URL: https://github.com/apache/airflow/pull/34665#discussion_r1343957193 ## airflow/dag_processing/manager.py: ## @@ -695,15 +696,16 @@ def _fetch_callbacks_with_retries(self, max_callbacks: int, session: Session): """Fetch cal

[PR] Support git-sync v4 in Helm Chart [airflow]

2023-10-03 Thread via GitHub
Swalloow opened a new pull request, #34731: URL: https://github.com/apache/airflow/pull/34731 Support git-sync v4 in Helm Chart. Add `dags.gitSync.ref` option for backwards compatibility. closes: #32335 -- This is an automated message from the Apache Git Service. To

Re: [PR] Sqlalchemy 2.0 changes to `DagFileProcessorManager` [airflow]

2023-10-03 Thread via GitHub
phanikumv commented on code in PR #34665: URL: https://github.com/apache/airflow/pull/34665#discussion_r1343990513 ## airflow/dag_processing/manager.py: ## @@ -695,15 +696,16 @@ def _fetch_callbacks_with_retries(self, max_callbacks: int, session: Session): """Fetch cal

Re: [I] Links to specific dagruns don't work when older than 25 runs [airflow]

2023-10-03 Thread via GitHub
hussein-awala commented on issue #34723: URL: https://github.com/apache/airflow/issues/34723#issuecomment-1744840318 This date is the start date for the X dag runs, when you choose 2023-01-01 as start date, and 25 as a number of runs, the UI will show you the next 25 runs since 2023-01-01.

Re: [PR] Use timeout when inntializing _timeout_sec of TaskStateTrigger [airflow]

2023-10-03 Thread via GitHub
Xiroo commented on code in PR #34715: URL: https://github.com/apache/airflow/pull/34715#discussion_r1343843951 ## airflow/sensors/external_task.py: ## @@ -341,6 +341,7 @@ def execute(self, context: Context) -> None: states=self.allowed_states,

Re: [I] Links to specific dagruns don't work when older than 25 runs [airflow]

2023-10-03 Thread via GitHub
hterik commented on issue #34723: URL: https://github.com/apache/airflow/issues/34723#issuecomment-1744877344 This was not the case in Airflow 2.5, i could click on links referring to any dagrun, no matter how old, and it would correctly select it in the grid-view. The selection of date

[PR] Name params while invoking ClientSecretCredential [airflow]

2023-10-03 Thread via GitHub
irajankumar opened a new pull request, #34732: URL: https://github.com/apache/airflow/pull/34732 Name function parameters while invoking ``ClientSecretCredential`` function of Azure Identity. --- **^ Add meaningful description above** Read the **[Pull

Re: [PR] Name params while invoking ClientSecretCredential [airflow]

2023-10-03 Thread via GitHub
boring-cyborg[bot] commented on PR #34732: URL: https://github.com/apache/airflow/pull/34732#issuecomment-1744944783 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution

Re: [PR] Include AWS Lambda execution logs to task logs [airflow]

2023-10-03 Thread via GitHub
vincbeck merged PR #34692: URL: https://github.com/apache/airflow/pull/34692 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow

[airflow] branch main updated: Include AWS Lambda execution logs to task logs (#34692)

2023-10-03 Thread vincbeck
This is an automated email from the ASF dual-hosted git repository. vincbeck pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 3064812280 Include AWS Lambda execution logs to t

Re: [PR] Fix WSGI middleware example in the doc by import AUTH_REMOTE_USER from fab [airflow]

2023-10-03 Thread via GitHub
vincbeck commented on PR #34721: URL: https://github.com/apache/airflow/pull/34721#issuecomment-1744986361 > I wonder if this should be covered by AIP-56 cc @vincbeck It is something that we might have moved aspart of AIP-56, but this definitely helps AIP-56 :) Thank you! -- This i

Re: [PR] Implement `is_authorized()` in auth manager [airflow]

2023-10-03 Thread via GitHub
vincbeck commented on code in PR #33213: URL: https://github.com/apache/airflow/pull/33213#discussion_r1344123947 ## airflow/www/auth.py: ## @@ -35,16 +44,100 @@ def get_access_denied_message(): return conf.get("webserver", "access_denied_message") -def has_access(permi

Re: [PR] Open Search Provider [airflow]

2023-10-03 Thread via GitHub
vincbeck commented on PR #34705: URL: https://github.com/apache/airflow/pull/34705#issuecomment-1745014377 > For system test DAG to succeed would an OpenSearch domain need to be setup and if so what is the approach for standing up such infrastructure for integration tests? System tes

Re: [PR] Warning when viewing docs for previous versions of Airflow [airflow]

2023-10-03 Thread via GitHub
vincbeck commented on PR #34639: URL: https://github.com/apache/airflow/pull/34639#issuecomment-1745019573 There is an issue for it: #29283 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [PR] Allow passing context to DruidDbApiHook [airflow]

2023-10-03 Thread via GitHub
Taragolis merged PR #34603: URL: https://github.com/apache/airflow/pull/34603 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflo

Re: [PR] Allow passing context to DruidDbApiHook [airflow]

2023-10-03 Thread via GitHub
boring-cyborg[bot] commented on PR #34603: URL: https://github.com/apache/airflow/pull/34603#issuecomment-1745089246 Awesome work, congrats on your first merged pull request! You are invited to check our [Issue Tracker](https://github.com/apache/airflow/issues) for additional contributions.

[airflow] branch main updated: Allow passing context to DruidDbApiHook (#34603)

2023-10-03 Thread taragolis
This is an automated email from the ASF dual-hosted git repository. taragolis pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 89df63b7ce Allow passing context to DruidDbApiHo

Re: [PR] feat: enable HTTPRoute for Gateway API [airflow]

2023-10-03 Thread via GitHub
Kyrremann commented on PR #34716: URL: https://github.com/apache/airflow/pull/34716#issuecomment-1745128719 Added one test for now, making another tomorrow. Couldn't manage to run the test locally (neither through Breeze or virtualenv pytest). Hopefully they will be run automatically with a

Re: [PR] Open search integrations [airflow]

2023-10-03 Thread via GitHub
Taragolis commented on code in PR #34693: URL: https://github.com/apache/airflow/pull/34693#discussion_r1344266894 ## airflow/providers/amazon/aws/hooks/opensearch.py: ## @@ -0,0 +1,130 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor li

Re: [PR] Open Search Provider [airflow]

2023-10-03 Thread via GitHub
cjames23 commented on code in PR #34705: URL: https://github.com/apache/airflow/pull/34705#discussion_r1344279037 ## airflow/providers/opensearch/provider.yaml: ## @@ -0,0 +1,52 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agree

[PR] Add k8s recommended labels [airflow]

2023-10-03 Thread via GitHub
Owen-CH-Leung opened a new pull request, #34735: URL: https://github.com/apache/airflow/pull/34735 fixes #34048 As per the discussion, this PR modifes all existing yaml files inside airflow helm chart to include the recommended k8s labels. See [this](https://kubernetes.io/docs/conce

Re: [PR] Open Search Provider [airflow]

2023-10-03 Thread via GitHub
cjames23 commented on code in PR #34705: URL: https://github.com/apache/airflow/pull/34705#discussion_r1344298941 ## generated/provider_dependencies.json: ## @@ -28,6 +28,10 @@ "boto3>=1.28.0", "botocore>=1.31.0", "jsonpath_ng>=1.5.3", + "mypy-boto3-app

Re: [PR] Open Search Provider [airflow]

2023-10-03 Thread via GitHub
cjames23 commented on code in PR #34705: URL: https://github.com/apache/airflow/pull/34705#discussion_r1344300110 ## airflow/providers/opensearch/provider.yaml: ## @@ -0,0 +1,52 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agree

Re: [PR] Name params while invoking ClientSecretCredential [airflow]

2023-10-03 Thread via GitHub
uranusjr commented on PR #34732: URL: https://github.com/apache/airflow/pull/34732#issuecomment-1745278754 Please fix the errors. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [PR] Open Search Provider [airflow]

2023-10-03 Thread via GitHub
eladkal commented on code in PR #34705: URL: https://github.com/apache/airflow/pull/34705#discussion_r1344387696 ## airflow/providers/opensearch/provider.yaml: ## @@ -0,0 +1,52 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreem

Re: [PR] Open Search Provider [airflow]

2023-10-03 Thread via GitHub
eladkal commented on code in PR #34705: URL: https://github.com/apache/airflow/pull/34705#discussion_r1344387696 ## airflow/providers/opensearch/provider.yaml: ## @@ -0,0 +1,52 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreem

Re: [PR] WIP: Add Airflow FS [airflow]

2023-10-03 Thread via GitHub
dstandish commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1344389497 ## airflow/io/__init__.py: ## Review Comment: but at least the provider-specific code should be moved to providers. like the implementations of the interface

[I] AzureContainerInstancesOperator doesn't work any more after version 6.3.0 of apache-airflow-providers-microsoft-azure [airflow]

2023-10-03 Thread via GitHub
strictness opened a new issue, #34736: URL: https://github.com/apache/airflow/issues/34736 ### Apache Airflow version 2.7.1 ### What happened We have to hardcode our version of `apache-airflow-providers-microsoft-azure` to 6.2.4 in order for AzureContainerInstancesOpera

Re: [I] AzureContainerInstancesOperator doesn't work any more after version 6.3.0 of apache-airflow-providers-microsoft-azure [airflow]

2023-10-03 Thread via GitHub
boring-cyborg[bot] commented on issue #34736: URL: https://github.com/apache/airflow/issues/34736#issuecomment-1745315118 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for ap

Re: [PR] WIP: Add Airflow FS [airflow]

2023-10-03 Thread via GitHub
eladkal commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1344421695 ## airflow/io/__init__.py: ## Review Comment: Ideally all code in hooks/sensors/operatos should be in providers. That at least should be what we aim for --

Re: [PR] WIP: Add Airflow FS [airflow]

2023-10-03 Thread via GitHub
eladkal commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1344421695 ## airflow/io/__init__.py: ## Review Comment: Ideally all code in hooks/sensors/operatos should be in providers. That at least should be what we should aim for

Re: [PR] WIP: Add Airflow FS [airflow]

2023-10-03 Thread via GitHub
bolkedebruin commented on PR #34729: URL: https://github.com/apache/airflow/pull/34729#issuecomment-1745354692 > Design wise: I'm not sure about the `fs.mount` affecting global state -- could we instead do something like this: > > ```python > s3fs = afs.mount("s3://warehouse", "/mn

Re: [PR] Use auth manager `is_authorized_` APIs to check user permissions in Rest API [airflow]

2023-10-03 Thread via GitHub
vincbeck commented on PR #34317: URL: https://github.com/apache/airflow/pull/34317#issuecomment-1745380219 All comments are addressed. Reviews are welcome :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

Re: [PR] WIP: Add Airflow FS [airflow]

2023-10-03 Thread via GitHub
bolkedebruin commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r137639 ## airflow/io/__init__.py: ## Review Comment: I am happy to move the provider specific code to the providers. For now it was just convenient to deal with i

Re: [PR] Created INSTALL.md and working to improve it [airflow]

2023-10-03 Thread via GitHub
KesharwaniArpita commented on PR #34714: URL: https://github.com/apache/airflow/pull/34714#issuecomment-1745416495 SO while running the pre=commits in the repo, this is what I get: "Lint JSON Schema files with JSON SchemaFailed - hook id: lin

Re: [PR] Created INSTALL.md and working to improve it [airflow]

2023-10-03 Thread via GitHub
KesharwaniArpita commented on code in PR #34714: URL: https://github.com/apache/airflow/pull/34714#discussion_r1344476869 ## INSTALL.md: ## @@ -0,0 +1,117 @@ +# INSTALL / BUILD instructions for Apache Airflow + +This is a generic installation method that requires a number of dep

Re: [PR] Open search integrations [airflow]

2023-10-03 Thread via GitHub
vincbeck commented on code in PR #34693: URL: https://github.com/apache/airflow/pull/34693#discussion_r1344502835 ## airflow/providers/amazon/aws/operators/opensearch.py: ## @@ -0,0 +1,189 @@ +# Review Comment: Quite related to this [comment thread](https://github.com/apach

[PR] Bump azure-mgmt-containerinstance [airflow]

2023-10-03 Thread via GitHub
pankajastro opened a new pull request, #34738: URL: https://github.com/apache/airflow/pull/34738 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guideli

Re: [I] AzureContainerInstancesOperator doesn't work any more after version 6.3.0 of apache-airflow-providers-microsoft-azure [airflow]

2023-10-03 Thread via GitHub
Taragolis commented on issue #34736: URL: https://github.com/apache/airflow/issues/34736#issuecomment-1745468981 I not familiar with internals of Azure libraries but guess it somehow related to: - https://github.com/Azure/azure-sdk-for-python/issues/16908 - https://github.com/Azure/azu

Re: [I] Hiding Run Id and Logical date from trigger DAG UI [airflow]

2023-10-03 Thread via GitHub
jens-scheffler-bosch commented on issue #34726: URL: https://github.com/apache/airflow/issues/34726#issuecomment-1745511216 I also once thought about this but myself then had use cases for testing where I needed these fields. So I agree that in most cases hiding these fields would make life

Re: [PR] Open search integrations [airflow]

2023-10-03 Thread via GitHub
cjames23 commented on code in PR #34693: URL: https://github.com/apache/airflow/pull/34693#discussion_r1344556720 ## airflow/providers/amazon/aws/operators/opensearch.py: ## @@ -0,0 +1,189 @@ +# Review Comment: This PR is on hold right now until the other PR is ready. Once t

Re: [PR] Open search integrations [airflow]

2023-10-03 Thread via GitHub
vincbeck commented on code in PR #34693: URL: https://github.com/apache/airflow/pull/34693#discussion_r1344564045 ## airflow/providers/amazon/aws/operators/opensearch.py: ## @@ -0,0 +1,189 @@ +# Review Comment: Sounds like very good to me 👍 ## airflow/providers/a

Re: [I] Links to specific dagruns don't work when older than 25 runs [airflow]

2023-10-03 Thread via GitHub
jens-scheffler-bosch commented on issue #34723: URL: https://github.com/apache/airflow/issues/34723#issuecomment-1745571114 I did a test with an example DAG, triggered this manually >25 times. As the GRID view shows 25 items per default all is fine if you click on one of the recent 25 items

[PR] Order cherrypicking candidates by the closed date [airflow]

2023-10-03 Thread via GitHub
ephraimbuddy opened a new pull request, #34739: URL: https://github.com/apache/airflow/pull/34739 Currently, the candidates are ordered by the PR number but ordering by the closed date is better and will help avoid some conflicts -- This is an automated message from the Apache Git

Re: [I] Status of testing of Apache Airflow Helm Chart 1.11.0rc2 [airflow]

2023-10-03 Thread via GitHub
brokenjacobs commented on issue #34660: URL: https://github.com/apache/airflow/issues/34660#issuecomment-1745597041 https://github.com/apache/airflow/issues/32590 uses the wrong port to connect to postgres, it defaults to port 6543 (the port provisioned for pgbouncer) instead of port 5433.

Re: [I] Airflow2.0.2 python3.7.1 statsd not work [airflow]

2023-10-03 Thread via GitHub
iw-pavan commented on issue #16905: URL: https://github.com/apache/airflow/issues/16905#issuecomment-1745605208 Facing same issue on postgres. https://github.com/apache/airflow/discussions/33897 -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [I] Status of testing of Apache Airflow Helm Chart 1.11.0rc2 [airflow]

2023-10-03 Thread via GitHub
hussein-awala commented on issue #34660: URL: https://github.com/apache/airflow/issues/34660#issuecomment-1745613375 > #32590 uses the wrong port to connect to postgres, it defaults to port 6543 (the port provisioned for pgbouncer) instead of port 5433. Sorry I was out when this ask came in

Re: [I] Kubernetes provider - Error parsing timestamp in logs [airflow]

2023-10-03 Thread via GitHub
rudo-ro commented on issue #34388: URL: https://github.com/apache/airflow/issues/34388#issuecomment-1745618398 we are seeing this on airflow 2.7.0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [I] AzureContainerInstancesOperator doesn't work any more after version 6.3.0 of apache-airflow-providers-microsoft-azure [airflow]

2023-10-03 Thread via GitHub
eladkal commented on issue #34736: URL: https://github.com/apache/airflow/issues/34736#issuecomment-1745634062 Note that version 6.3.0 updated upstream library: `Bump azure-mgmt-containerinstance>=7.0.0,<9.0.0 (#33696)` Airflow did not break the interface of the operator (this is wh

Re: [I] AzureContainerInstancesOperator doesn't work any more after version 6.3.0 of apache-airflow-providers-microsoft-azure [airflow]

2023-10-03 Thread via GitHub
eladkal closed issue #34736: AzureContainerInstancesOperator doesn't work any more after version 6.3.0 of apache-airflow-providers-microsoft-azure URL: https://github.com/apache/airflow/issues/34736 -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [I] Kubernetes provider - Error parsing timestamp in logs [airflow]

2023-10-03 Thread via GitHub
eladkal commented on issue #34388: URL: https://github.com/apache/airflow/issues/34388#issuecomment-1745636944 > we are seeing this on airflow 2.7.0 This is not related to Airflow version. It's related to `apache-airflow-providers-cncf-kubernetes` version. -- This is an automated m

[I] not using pgbouncer connection values still use pgbouncer ports/names [airflow]

2023-10-03 Thread via GitHub
brokenjacobs opened a new issue, #34740: URL: https://github.com/apache/airflow/issues/34740 ### Apache Airflow version 2.7.1 ### What happened I missed the rc test window last week, sorry was out of town. When you use: `values.keda.usePgbouncer: false` The sett

Re: [I] not using pgbouncer connection values still use pgbouncer ports/names [airflow]

2023-10-03 Thread via GitHub
brokenjacobs commented on issue #34740: URL: https://github.com/apache/airflow/issues/34740#issuecomment-1745637574 Once again my apologies for missing the test window on this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub a

Re: [PR] Fix parsing KubernetesPodOperator multiline logs [airflow]

2023-10-03 Thread via GitHub
eladkal commented on PR #34412: URL: https://github.com/apache/airflow/pull/34412#issuecomment-1745638117 @hussein-awala can you fix conflicts and rebase? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

  1   2   >