Re: [I] Celery worker enters a catatonic state after redis restart [airflow]

2024-01-29 Thread via GitHub
eladkal commented on issue #26542: URL: https://github.com/apache/airflow/issues/26542#issuecomment-1916233877 > So I think if we ALSO have a way how to bring older celery configuration to "reasonable defaults", it's worth considering to explain that in the documentation and even our Helm

Re: [I] Button on UI to pause all DAGs [airflow]

2024-01-29 Thread via GitHub
eladkal commented on issue #36994: URL: https://github.com/apache/airflow/issues/36994#issuecomment-1916229857 I would rather this not be a UI feature. This is very dangerous operation and with a single click of a button that was done by mistake can cause serious problems to large

Re: [PR] Add GKEStartKueueInsideClusterOperator to support Kueue inside clusters [airflow]

2024-01-29 Thread via GitHub
hussein-awala commented on PR #37072: URL: https://github.com/apache/airflow/pull/37072#issuecomment-1916205910 > We usually use infrastructure as code to install the cluster and additional things together. what is the significance installing Kueue alone? Not true, we have an

Re: [PR] Add Qdrant Provider [airflow]

2024-01-29 Thread via GitHub
Anush008 commented on code in PR #36805: URL: https://github.com/apache/airflow/pull/36805#discussion_r1470662843 ## pyproject.toml: ## @@ -533,6 +533,7 @@ alibaba = [ "oss2>=2.14.0", ] amazon = [ + "PyAthena>=3.0.10", Review Comment: It came in with one of the merge

Re: [PR] Add Qdrant Provider [airflow]

2024-01-29 Thread via GitHub
Anush008 commented on code in PR #36805: URL: https://github.com/apache/airflow/pull/36805#discussion_r1470662469 ## tests/providers/qdrant/operators/test_qdrant.py: ## @@ -0,0 +1,70 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license

Re: [PR] Add GKEStartKueueInsideClusterOperator to support Kueue inside clusters [airflow]

2024-01-29 Thread via GitHub
VladaZakharova commented on PR #37072: URL: https://github.com/apache/airflow/pull/37072#issuecomment-1916161115 Hey @dirrao @raphaelauv ! Thank you for looking my changes. This operator is the first one in a set operators that are needed to cover functionality we have for GKE

[I] Dynamic Task Mapping TriggerDagRunOperator breaks grid/graph view [airflow]

2024-01-29 Thread via GitHub
jkarun opened a new issue, #37084: URL: https://github.com/apache/airflow/issues/37084 ### Apache Airflow version 2.8.1 ### If "Other Airflow 2 version" selected, which one? 2.6.1 ### What happened? I am using dynamic task mapping of TriggerDagRunOperator

Re: [I] Dynamic Task Mapping TriggerDagRunOperator breaks grid/graph view [airflow]

2024-01-29 Thread via GitHub
boring-cyborg[bot] commented on issue #37084: URL: https://github.com/apache/airflow/issues/37084#issuecomment-1916150562 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for

Re: [I] Dynamic mapping over task group is broken with multiple_outputs [airflow]

2024-01-29 Thread via GitHub
tanelk commented on issue #37073: URL: https://github.com/apache/airflow/issues/37073#issuecomment-1916128216 The same issue occurs when not using task groups, but it is handled a bit more gracefully:

Re: [I] Dynamic mapping over task group is broken with multiple_outputs [airflow]

2024-01-29 Thread via GitHub
tanelk commented on issue #37073: URL: https://github.com/apache/airflow/issues/37073#issuecomment-1916117531 I looked at it a bit more and the issue seems to be with the airflow.models.taskmap.TaskMap design. The airflow.models.taskinstance._record_task_map_for_downstreams method does not

[PR] Fixing a visual in breeze cheatsheet [airflow]

2024-01-29 Thread via GitHub
amoghrajesh opened a new pull request, #37083: URL: https://github.com/apache/airflow/pull/37083 Small visual fix for breeze cheatsheet --- **^ Add meaningful description above** Read the **[Pull Request

Re: [PR] confirmation dialog box for DAG run actions [airflow]

2024-01-29 Thread via GitHub
amoghrajesh commented on PR #35393: URL: https://github.com/apache/airflow/pull/35393#issuecomment-1916093168 @bbovenzi @potiuk are we good for a merge here? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] New document for security fixes in Airflow [airflow]

2024-01-29 Thread via GitHub
amoghrajesh commented on PR #37082: URL: https://github.com/apache/airflow/pull/37082#issuecomment-1916089538 @potiuk it is not complete, but would probably require few levels of reviews here. Appreciate your patience -- This is an automated message from the Apache Git Service. To

[PR] New document for security fixes in Airflow [airflow]

2024-01-29 Thread via GitHub
amoghrajesh opened a new pull request, #37082: URL: https://github.com/apache/airflow/pull/37082 Recently, it came to my attention that we have few unsaid rules and practices followed regarding the security issues and their fixes in Airflow. Introducing a new document to

Re: [PR] Added ability to enable/disable scheduler and webserver [airflow]

2024-01-29 Thread via GitHub
jedcunningham commented on code in PR #36991: URL: https://github.com/apache/airflow/pull/36991#discussion_r1470591576 ## chart/templates/scheduler/scheduler-deployment.yaml: ## @@ -20,7 +20,8 @@ ## Airflow Scheduler Deployment/StatefulSet

Re: [PR] Fixing status icon alignment for various views [airflow]

2024-01-29 Thread via GitHub
amoghrajesh commented on PR #36804: URL: https://github.com/apache/airflow/pull/36804#issuecomment-1916066467 Oops, that was a bad rebase, removed all reviewers who got added due to it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] Fixing status icon alignment for various views [airflow]

2024-01-29 Thread via GitHub
amoghrajesh commented on PR #36804: URL: https://github.com/apache/airflow/pull/36804#issuecomment-1916059253 @pierrejeambrun I made the changes using flex of Box, results look the same now: ![image](https://github.com/apache/airflow/assets/35884252/d3f819c4-36bc-4373-98f8-78b9488c632f)

Re: [PR] Added ability to enable/disable scheduler and webserver [airflow]

2024-01-29 Thread via GitHub
amoghrajesh commented on code in PR #36991: URL: https://github.com/apache/airflow/pull/36991#discussion_r1470577903 ## chart/templates/scheduler/scheduler-deployment.yaml: ## @@ -20,7 +20,8 @@ ## Airflow Scheduler Deployment/StatefulSet

Re: [PR] Fixing status icon alignment for various views [airflow]

2024-01-29 Thread via GitHub
amoghrajesh commented on PR #36804: URL: https://github.com/apache/airflow/pull/36804#issuecomment-1916051280 > I don't think that manually translating by 2 px is the way to go, this is hacky. (That might not work on every browser, screen size etc...). Just using the flex property of the

[I] Quickstart: not having `virtualenv` installed results in broken DAGs [airflow-site]

2024-01-29 Thread via GitHub
merobi-hub opened a new issue, #945: URL: https://github.com/apache/airflow-site/issues/945 Error-free installation of Airflow using the [Quickstart](https://airflow.apache.org/docs/apache-airflow/stable/start.html seems to require `virtualenv`, but there is no mention of this in the

[PR] feat(providers/google): add service_file support to GKEPodAsyncHook [airflow]

2024-01-29 Thread via GitHub
Lee-W opened a new pull request, #37081: URL: https://github.com/apache/airflow/pull/37081 --- **^ Add meaningful description above** Read the **[Pull Request

Re: [PR] The task is stuck in a queued state forever in case of pod launch errors [airflow]

2024-01-29 Thread via GitHub
jedcunningham commented on PR #36882: URL: https://github.com/apache/airflow/pull/36882#issuecomment-1916006948 > > I see you added retires counter. What do you think about custom delay between each retry of exceeded quota also? My issue is the high rate of requests to Kubernetes API and

Re: [PR] The task is stuck in a queued state forever in case of pod launch errors [airflow]

2024-01-29 Thread via GitHub
jedcunningham commented on code in PR #36882: URL: https://github.com/apache/airflow/pull/36882#discussion_r1470543337 ## airflow/providers/cncf/kubernetes/provider.yaml: ## @@ -350,6 +350,15 @@ config: type: string example: ~ default: "" +

Re: [PR] Added ability to enable/disable scheduler and webserver [airflow]

2024-01-29 Thread via GitHub
jedcunningham commented on code in PR #36991: URL: https://github.com/apache/airflow/pull/36991#discussion_r1470539324 ## helm_tests/airflow_core/test_scheduler.py: ## @@ -901,3 +902,19 @@ def test_overridden_automount_service_account_token(self):

Re: [PR] Add Qdrant Provider [airflow]

2024-01-29 Thread via GitHub
josh-fell commented on code in PR #36805: URL: https://github.com/apache/airflow/pull/36805#discussion_r1470517045 ## pyproject.toml: ## @@ -533,6 +533,7 @@ alibaba = [ "oss2>=2.14.0", ] amazon = [ + "PyAthena>=3.0.10", Review Comment: Is this a related change?

Re: [PR] Airflow Teradata Provider: Initial implementation of Airflow Teradata Provider [airflow]

2024-01-29 Thread via GitHub
josh-fell commented on code in PR #36953: URL: https://github.com/apache/airflow/pull/36953#discussion_r1470489935 ## airflow/providers/teradata/example_dags/example_teradata_operator.py: ## @@ -0,0 +1,132 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +#

Re: [PR] The task is stuck in a queued state forever in case of pod launch errors [airflow]

2024-01-29 Thread via GitHub
dirrao commented on PR #36882: URL: https://github.com/apache/airflow/pull/36882#issuecomment-1915929151 > I see you added retires counter. What do you think about custom delay between each retry of exceeded quota also? My issue is the high rate of requests to Kubernetes API and currently

Re: [I] Add two new fields (gunicorn_logconfig & gunicorn_logconfig_dict) to airflow.cfg to allow users to configure Airflow Webserver logs [airflow]

2024-01-29 Thread via GitHub
shayoo9814 commented on issue #37078: URL: https://github.com/apache/airflow/issues/37078#issuecomment-1915906602 I still believe there is value to be gained from the proposed change for the following reasons. 1. Consistent interface for Airflow customizations `airflow.cfg` is what

Re: [PR] draft for map_index_template to overwrite map_index in the UI [airflow]

2024-01-29 Thread via GitHub
RNHTTR commented on code in PR #36797: URL: https://github.com/apache/airflow/pull/36797#discussion_r1470456522 ## airflow/serialization/serialized_objects.py: ## @@ -1098,6 +1098,7 @@ def deserialize_operator(cls, encoded_op: dict[str, Any]) -> Operator:

Re: [I] KubernetesPodOperator does not return XCOM on pod failure [airflow]

2024-01-29 Thread via GitHub
hussein-awala commented on issue #8792: URL: https://github.com/apache/airflow/issues/8792#issuecomment-1915863325 As a workaround, we can use the new callbacks class: ```python class XComCallbacks(KubernetesPodOperatorCallback): @staticmethod def on_pod_completion(*,

Re: [PR] SqlToS3Operator: feat/ add max_rows_per_file parameter [airflow]

2024-01-29 Thread via GitHub
potiuk commented on PR #37055: URL: https://github.com/apache/airflow/pull/37055#issuecomment-1915824822 > Now that you explicitly fail if both parameters are specified, you could potentially simplify this statement: `if self.max_rows_per_file and not self.groupby_kwargs:` But I'll leave

(airflow) branch main updated: SqlToS3Operator: feat/ add max_rows_per_file parameter (#37055)

2024-01-29 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 8914e49551 SqlToS3Operator: feat/ add

Re: [PR] SqlToS3Operator: feat/ add max_rows_per_file parameter [airflow]

2024-01-29 Thread via GitHub
boring-cyborg[bot] commented on PR #37055: URL: https://github.com/apache/airflow/pull/37055#issuecomment-1915824330 Awesome work, congrats on your first merged pull request! You are invited to check our [Issue Tracker](https://github.com/apache/airflow/issues) for additional

Re: [PR] SqlToS3Operator: feat/ add max_rows_per_file parameter [airflow]

2024-01-29 Thread via GitHub
potiuk merged PR #37055: URL: https://github.com/apache/airflow/pull/37055 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [I] Add two new fields (gunicorn_logconfig & gunicorn_logconfig_dict) to airflow.cfg to allow users to configure Airflow Webserver logs [airflow]

2024-01-29 Thread via GitHub
potiuk closed issue #37078: Add two new fields (gunicorn_logconfig & gunicorn_logconfig_dict) to airflow.cfg to allow users to configure Airflow Webserver logs URL: https://github.com/apache/airflow/issues/37078 -- This is an automated message from the Apache Git Service. To respond to the

Re: [I] Add two new fields (gunicorn_logconfig & gunicorn_logconfig_dict) to airflow.cfg to allow users to configure Airflow Webserver logs [airflow]

2024-01-29 Thread via GitHub
potiuk commented on issue #37078: URL: https://github.com/apache/airflow/issues/37078#issuecomment-1915819055 You can configure gunicorn settings by GUNICORN_CMD_ARGS variable

Re: [PR] Update DatabricksSqlOperator to work with namedtuple [airflow]

2024-01-29 Thread via GitHub
SuperBo commented on PR #37025: URL: https://github.com/apache/airflow/pull/37025#issuecomment-1915809441 hi @Joffreybvn, I fixed the code style -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [I] Config variable execute_tasks_new_python_interpreter not sufficiently documented [airflow]

2024-01-29 Thread via GitHub
potiuk closed issue #37074: Config variable execute_tasks_new_python_interpreter not sufficiently documented URL: https://github.com/apache/airflow/issues/37074 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Add run_id to event logs API [airflow]

2024-01-29 Thread via GitHub
github-actions[bot] closed pull request #34224: Add run_id to event logs API URL: https://github.com/apache/airflow/pull/34224 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] BeamRunPythonPipelineOperator: push_xcom dataflow_job_id right before starting pipeline [airflow]

2024-01-29 Thread via GitHub
github-actions[bot] closed pull request #34707: BeamRunPythonPipelineOperator: push_xcom dataflow_job_id right before starting pipeline URL: https://github.com/apache/airflow/pull/34707 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [I] "Celery command failed", but "Marking task as SUCCESS" [airflow]

2024-01-29 Thread via GitHub
github-actions[bot] commented on issue #35610: URL: https://github.com/apache/airflow/issues/35610#issuecomment-1915805486 This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further

Re: [PR] Fix macros jinja template example [airflow]

2024-01-29 Thread via GitHub
github-actions[bot] commented on PR #36222: URL: https://github.com/apache/airflow/pull/36222#issuecomment-1915805441 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for

Re: [I] KubernetesPodOperator does not return XCOM on pod failure [airflow]

2024-01-29 Thread via GitHub
hussein-awala commented on issue #8792: URL: https://github.com/apache/airflow/issues/8792#issuecomment-1915799699 I'm interested in this proposed feature (btw it is not a bug). I tried to create something generic as @potiuk suggested, but it's too complicated since we don't return any

Re: [PR] Add an option to KubernetesPodOperator to push the result XCom on failure [airflow]

2024-01-29 Thread via GitHub
hussein-awala commented on PR #37079: URL: https://github.com/apache/airflow/pull/37079#issuecomment-1915788086 I will add some tests after discussing the solution. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[PR] Add an option to KubernetesPodOperator to push the result XCom on failure [airflow]

2024-01-29 Thread via GitHub
hussein-awala opened a new pull request, #37079: URL: https://github.com/apache/airflow/pull/37079 Closes: #8792 This PR adds a new argument `do_xcom_push_on_failure` to `KubernetesPodOperator` to load and push the XCom with a different key when the pod fails. -- This is an

Re: [I] Add two new fields (gunicorn_logconfig & gunicorn_logconfig_dict) to airflow.cfg to allow users to configure Airflow Webserver logs [airflow]

2024-01-29 Thread via GitHub
boring-cyborg[bot] commented on issue #37078: URL: https://github.com/apache/airflow/issues/37078#issuecomment-1915763537 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for

[I] Add two new fields (gunicorn_logconfig & gunicorn_logconfig_dict) to airflow.cfg to allow users to configure Airflow Webserver logs [airflow]

2024-01-29 Thread via GitHub
shayoo9814 opened a new issue, #37078: URL: https://github.com/apache/airflow/issues/37078 ### Description Today, Airflow users lack the ability to configure Airflow Webserver logs. (Users can only configure [access logs](https://docs.gunicorn.org/en/stable/settings.html#accesslog),

Re: [I] Audit logs entry as clear event even if we click on the tasks. [airflow]

2024-01-29 Thread via GitHub
jscheffl closed issue #36119: Audit logs entry as clear event even if we click on the tasks. URL: https://github.com/apache/airflow/issues/36119 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] Audit logs entry as clear event even if we click on the tasks. [airflow]

2024-01-29 Thread via GitHub
jscheffl commented on issue #36119: URL: https://github.com/apache/airflow/issues/36119#issuecomment-1915583326 Fixed in #36941 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Simplify hive client connection [airflow]

2024-01-29 Thread via GitHub
eladkal commented on code in PR #37043: URL: https://github.com/apache/airflow/pull/37043#discussion_r1470167194 ## airflow/providers/apache/hive/CHANGELOG.rst: ## @@ -27,6 +27,17 @@ Changelog - +7.0.0 +. + + +Breaking changes + + +*

Re: [PR] Standardize deprecations in providers [part1] [airflow]

2024-01-29 Thread via GitHub
dstandish commented on PR #36876: URL: https://github.com/apache/airflow/pull/36876#issuecomment-1915475826 > I am good for it - @dstandish ? What do you think? Seems we just replace warnings.warn with deprecated decorator, seems fine, to me -- This is an automated message from

Re: [I] Config variable execute_tasks_new_python_interpreter not sufficiently documented [airflow]

2024-01-29 Thread via GitHub
jscheffl commented on issue #37074: URL: https://github.com/apache/airflow/issues/37074#issuecomment-1915443970 Have you considered raising a PR and contributing an improvement? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] AIP-58: Add object storage backend for xcom [airflow]

2024-01-29 Thread via GitHub
hussein-awala commented on PR #37058: URL: https://github.com/apache/airflow/pull/37058#issuecomment-1915398665 > > > If we chose 2 then it makes sense to move it into common.io (with all the extras that it requires cause it is a 'new' capability for a provider). > > > > > > I

Re: [PR] Fix KPO docs argument precedence [airflow]

2024-01-29 Thread via GitHub
ketozhang commented on code in PR #37047: URL: https://github.com/apache/airflow/pull/37047#discussion_r1470039181 ## docs/apache-airflow-providers-cncf-kubernetes/operators.rst: ## @@ -77,8 +77,8 @@ You can print out the Kubernetes manifest for the pod that would be created

Re: [PR] AIP-58: Add object storage backend for xcom [airflow]

2024-01-29 Thread via GitHub
bolkedebruin commented on PR #37058: URL: https://github.com/apache/airflow/pull/37058#issuecomment-1915338997 Ok ill move it to the provider. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Add doc page with providers deprecations [airflow]

2024-01-29 Thread via GitHub
eladkal commented on code in PR #37075: URL: https://github.com/apache/airflow/pull/37075#discussion_r1469998144 ## docs/apache-airflow-providers/core-extensions/deprecations.rst: ## @@ -0,0 +1,57 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or more

Re: [PR] Add doc page with providers deprecations [airflow]

2024-01-29 Thread via GitHub
eladkal commented on code in PR #37075: URL: https://github.com/apache/airflow/pull/37075#discussion_r1469994293 ## docs/spelling_wordlist.txt: ## @@ -122,6 +122,7 @@ avp Avro avro aws +AwsBaseAsyncHook Review Comment: That is odd I don't think the right fix is to add

Re: [PR] Add doc page with providers deprecations [airflow]

2024-01-29 Thread via GitHub
kacpermuda commented on code in PR #37075: URL: https://github.com/apache/airflow/pull/37075#discussion_r1469992933 ## docs/apache-airflow-providers/core-extensions/deprecations.rst: ## @@ -0,0 +1,57 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or

Re: [PR] Add doc page with providers deprecations [airflow]

2024-01-29 Thread via GitHub
kacpermuda commented on code in PR #37075: URL: https://github.com/apache/airflow/pull/37075#discussion_r1469991675 ## docs/apache-airflow-providers/core-extensions/deprecations.rst: ## @@ -0,0 +1,57 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or

Re: [PR] Added ability to enable/disable scheduler and webserver [airflow]

2024-01-29 Thread via GitHub
eladkal commented on code in PR #36991: URL: https://github.com/apache/airflow/pull/36991#discussion_r1469990838 ## helm_tests/airflow_core/test_scheduler.py: ## @@ -901,3 +902,18 @@ def test_overridden_automount_service_account_token(self):

Re: [PR] Added ability to enable/disable scheduler and webserver [airflow]

2024-01-29 Thread via GitHub
eladkal commented on code in PR #36991: URL: https://github.com/apache/airflow/pull/36991#discussion_r1469989502 ## helm_tests/airflow_core/test_scheduler.py: ## @@ -901,3 +902,19 @@ def test_overridden_automount_service_account_token(self):

Re: [PR] Add doc page with providers deprecations [airflow]

2024-01-29 Thread via GitHub
kacpermuda commented on code in PR #37075: URL: https://github.com/apache/airflow/pull/37075#discussion_r1469989464 ## docs/spelling_wordlist.txt: ## @@ -122,6 +122,7 @@ avp Avro avro aws +AwsBaseAsyncHook Review Comment: Ah, this will be needed when #36876 is merged

Re: [PR] Add doc page with providers deprecations [airflow]

2024-01-29 Thread via GitHub
eladkal commented on code in PR #37075: URL: https://github.com/apache/airflow/pull/37075#discussion_r1469966259 ## docs/apache-airflow-providers/core-extensions/deprecations.rst: ## @@ -0,0 +1,57 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or more

Re: [I] Add a Reset Password button under Security > List Users > Edit user [airflow]

2024-01-29 Thread via GitHub
eladm26 commented on issue #37030: URL: https://github.com/apache/airflow/issues/37030#issuecomment-1915231534 Hi, Will be happy to take this one as a first issue -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[PR] Yandex Cloud Lockbox Secrets Backend: use endpoint from connection if not specified [airflow]

2024-01-29 Thread via GitHub
ImpressionableRaccoon opened a new pull request, #37076: URL: https://github.com/apache/airflow/pull/37076 try to use endpoint from connection if it not specified in `backend_kwargs` --- **^ Add meaningful description above** Read the **[Pull Request

Re: [I] apache-airflow-providers-cncf-kubernetes requires latest kubernetes version [airflow]

2024-01-29 Thread via GitHub
potiuk commented on issue #36678: URL: https://github.com/apache/airflow/issues/36678#issuecomment-1915219666 We release our providers roughly every two weeks. Byut if you REALLY want you can easily build your own provider packages we have instructions available to do so - this way you

Re: [PR] AIP-58: Add object storage backend for xcom [airflow]

2024-01-29 Thread via GitHub
potiuk commented on PR #37058: URL: https://github.com/apache/airflow/pull/37058#issuecomment-1915207986 Yeah. My suggestion was bad idea and you pointed out it's not needed indeed. And yes I thought that the change could actually be split In two : Base XCOM change for small/big values for

Re: [I] apache-airflow-providers-cncf-kubernetes requires latest kubernetes version [airflow]

2024-01-29 Thread via GitHub
raphaelauv commented on issue #36678: URL: https://github.com/apache/airflow/issues/36678#issuecomment-1915166094 https://github.com/apache/airflow/issues?q=is%3Aissue+Status+of+testing+Providers+is%3Aclosed+ -- This is an automated message from the Apache Git Service. To respond to the

Re: [I] apache-airflow-providers-cncf-kubernetes requires latest kubernetes version [airflow]

2024-01-29 Thread via GitHub
paramjeet01 commented on issue #36678: URL: https://github.com/apache/airflow/issues/36678#issuecomment-1915158946 @raphaelauv When are we planning to release the new version ? or is it automated by pipelines and we wait till then ? -- This is an automated message from the Apache Git

Re: [I] apache-airflow-providers-cncf-kubernetes requires latest kubernetes version [airflow]

2024-01-29 Thread via GitHub
raphaelauv commented on issue #36678: URL: https://github.com/apache/airflow/issues/36678#issuecomment-1915150429 close cause fix is merged , does not mean it was deploy in a new release and `7.14.0` is 2 days old , It can't include a fix merged 30 minutes ago -- This is an

Re: [I] apache-airflow-providers-cncf-kubernetes requires latest kubernetes version [airflow]

2024-01-29 Thread via GitHub
gbonazzoli commented on issue #36678: URL: https://github.com/apache/airflow/issues/36678#issuecomment-1915131712 The issue is said to be closed but it seems that `apache-airflow-providers-cncf-kubernetes 7.14.0` does not think so. ``` ERROR: pip's dependency resolver does not

Re: [PR] Release notes for helm chart 1.12.0 [airflow]

2024-01-29 Thread via GitHub
jedcunningham commented on PR #36917: URL: https://github.com/apache/airflow/pull/36917#issuecomment-1915114549 Okay, this should be ready now :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[PR] Add doc page with providers deprecations [airflow]

2024-01-29 Thread via GitHub
kacpermuda opened a new pull request, #37075: URL: https://github.com/apache/airflow/pull/37075 ## TLDR I want to add a documentation page that will have all the deprecated classes, methods, functions etc. It will get more information when #36876 is merged (where i switched

Re: [PR] Add GKEStartKueueInsideClusterOperator to support Kueue inside clusters [airflow]

2024-01-29 Thread via GitHub
raphaelauv commented on PR #37072: URL: https://github.com/apache/airflow/pull/37072#issuecomment-1915062317 Why add an operator in airflow to install `kueue` in a GKE ? the installation thanks to kubectl is not enough ?

Re: [PR] Make Amazon Provider tests compatible with `moto>=5` [airflow]

2024-01-29 Thread via GitHub
Taragolis commented on PR #37060: URL: https://github.com/apache/airflow/pull/37060#issuecomment-1915055196 Ok, seems like last changes finally fix all places where old `moto` decorators/context managers are uses. Just need to rebase it -- This is an automated message from the Apache

Re: [PR] bump: kubernetes client version [airflow]

2024-01-29 Thread via GitHub
raphaelauv closed pull request #36684: bump: kubernetes client version URL: https://github.com/apache/airflow/pull/36684 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Fix location requirement in DataflowTemplatedJobStartOperator [airflow]

2024-01-29 Thread via GitHub
dirrao commented on code in PR #37069: URL: https://github.com/apache/airflow/pull/37069#discussion_r1469838242 ## airflow/providers/google/cloud/operators/dataflow.py: ## @@ -677,6 +677,9 @@ def set_current_job(current_job): options = self.dataflow_default_options

(airflow) branch main updated: Kubernetes version bump (#37040)

2024-01-29 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new c36c4db3f7 Kubernetes version bump (#37040)

Re: [PR] Kubernetes version bump [airflow]

2024-01-29 Thread via GitHub
potiuk merged PR #37040: URL: https://github.com/apache/airflow/pull/37040 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [I] apache-airflow-providers-cncf-kubernetes requires latest kubernetes version [airflow]

2024-01-29 Thread via GitHub
potiuk closed issue #36678: apache-airflow-providers-cncf-kubernetes requires latest kubernetes version URL: https://github.com/apache/airflow/issues/36678 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Kubernetes version bump [airflow]

2024-01-29 Thread via GitHub
potiuk commented on PR #37040: URL: https://github.com/apache/airflow/pull/37040#issuecomment-1915025497 Oh. Fantastic ! Thank you ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Make Amazon Provider tests compatible with `moto>=5` [airflow]

2024-01-29 Thread via GitHub
potiuk commented on PR #37060: URL: https://github.com/apache/airflow/pull/37060#issuecomment-1915016353 :eyes: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [I] Button on UI to pause all DAGs [airflow]

2024-01-29 Thread via GitHub
potiuk commented on issue #36994: URL: https://github.com/apache/airflow/issues/36994#issuecomment-1915013335 Cool -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Added ability to enable/disable scheduler and webserver [airflow]

2024-01-29 Thread via GitHub
kajanajustin commented on PR #36991: URL: https://github.com/apache/airflow/pull/36991#issuecomment-1915007044 @potiuk static checks and the test_can_be_disabled for the scheduler were failing so I added a new class for the function. Let me know if you think everything looks good so we can

Re: [PR] The task is stuck in a queued state forever in case of pod launch errors [airflow]

2024-01-29 Thread via GitHub
chenyair commented on PR #36882: URL: https://github.com/apache/airflow/pull/36882#issuecomment-1914986603 I see you added retires counter. What do you think about custom delay between each retry of exceeded quota also? My issue is the high rate of requests to Kubernetes API and currently

Re: [I] Update airflow.io.path.ObjectStoragePath for upcoming universal_pathlib release [airflow]

2024-01-29 Thread via GitHub
bolkedebruin commented on issue #37067: URL: https://github.com/apache/airflow/issues/37067#issuecomment-1914980481 Hey @ap-- , That just sounds awesome! Obviously any simplification while retaining flexibility sounds great. The generic need for a custom class on top of

Re: [I] Config variable execute_tasks_new_python_interpreter not sufficiently documented [airflow]

2024-01-29 Thread via GitHub
boring-cyborg[bot] commented on issue #37074: URL: https://github.com/apache/airflow/issues/37074#issuecomment-1914962812 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for

Re: [PR] Update DatabricksSqlOperator to work with namedtuple [airflow]

2024-01-29 Thread via GitHub
Joffreybvn commented on PR #37025: URL: https://github.com/apache/airflow/pull/37025#issuecomment-1914962675 Just need to fix the code formatting. A run of `breeze static-checks` will do it :) -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [I] DatabricksSQLOperator struggles with parsing the data [airflow]

2024-01-29 Thread via GitHub
Joffreybvn commented on issue #36839: URL: https://github.com/apache/airflow/issues/36839#issuecomment-1914955285 I could not reproduce your issue. But I found another bug where a single namedtuple 'Row' fails to be serialized, while a list of those namedtuples works fine. Is your query

Re: [PR] AIP-58: Add object storage backend for xcom [airflow]

2024-01-29 Thread via GitHub
bolkedebruin commented on PR #37058: URL: https://github.com/apache/airflow/pull/37058#issuecomment-1914950829 > > If we chose 2 then it makes sense to move it into common.io (with all the extras that it requires cause it is a 'new' capability for a provider). > > I think I''d

Re: [PR] Draft: kubernetes version bump [airflow]

2024-01-29 Thread via GitHub
dirrao commented on PR #37040: URL: https://github.com/apache/airflow/pull/37040#issuecomment-1914902451 @potiuk I have fixed rest of the issues. Thank you. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [I] Airflow migrations depends on alembic>=1.13 [airflow]

2024-01-29 Thread via GitHub
Taragolis closed issue #37071: Airflow migrations depends on alembic>=1.13 URL: https://github.com/apache/airflow/issues/37071 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[I] Dynamic mapping over task group is broken with multiple_outputs [airflow]

2024-01-29 Thread via GitHub
tanelk opened a new issue, #37073: URL: https://github.com/apache/airflow/issues/37073 ### Apache Airflow version main (development) ### If "Other Airflow 2 version" selected, which one? _No response_ ### What happened? When returning multiple outputs from

[PR] Add GKEStartKueueInsideClusterOperator to support Kueue inside clusters [airflow]

2024-01-29 Thread via GitHub
VladaZakharova opened a new pull request, #37072: URL: https://github.com/apache/airflow/pull/37072 This PR adds the ability to install Kueue inside specific types of clusters in GKE. For more details please check:

[I] Airflow migrations depends on alembic>=1.13 [airflow]

2024-01-29 Thread via GitHub
cesar-vermeulen opened a new issue, #37071: URL: https://github.com/apache/airflow/issues/37071 ### Apache Airflow version 2.8.1 ### If "Other Airflow 2 version" selected, which one? _No response_ ### What happened? Airflow failed to upgrade from version

[PR] Fix location requirement in DataflowTemplatedJobStartOperator [airflow]

2024-01-29 Thread via GitHub
shardbread opened a new pull request, #37069: URL: https://github.com/apache/airflow/pull/37069 An error was encountered when calling the execute method on a DataflowTemplateJobStartOperator instance with an unspecified location. This fix resolves the issue with an empty location in the

Re: [PR] Deferrable sensors can timeout with no retries [airflow]

2024-01-29 Thread via GitHub
robg-eb commented on PR #33718: URL: https://github.com/apache/airflow/pull/33718#issuecomment-1914788890 @dstandish and team - Thank you for creating this PR, as we are currently unable to properly use retries with Deferrable (Async) sensors, because of the inconsistency in retry /

Re: [PR] Introduce DatasetTimetable [airflow]

2024-01-29 Thread via GitHub
sunank200 commented on code in PR #36710: URL: https://github.com/apache/airflow/pull/36710#discussion_r1469598323 ## airflow/timetables/datasets.py: ## @@ -0,0 +1,90 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

[I] Update airflow.io.path.ObjectStoragePath for upcoming universal_pathlib release [airflow]

2024-01-29 Thread via GitHub
ap-- opened a new issue, #37067: URL: https://github.com/apache/airflow/issues/37067 ### Apache Airflow version main (development) ### If "Other Airflow 2 version" selected, which one? _No response_ ### What happened? I'm opening this issue to track a

Re: [I] Update airflow.io.path.ObjectStoragePath for upcoming universal_pathlib release [airflow]

2024-01-29 Thread via GitHub
boring-cyborg[bot] commented on issue #37067: URL: https://github.com/apache/airflow/issues/37067#issuecomment-1914666408 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for

  1   2   >