Re: [PR] Celery provider: Add task_acks_late setting [airflow]

2024-01-30 Thread via GitHub
ninsbl commented on PR #37066: URL: https://github.com/apache/airflow/pull/37066#issuecomment-1917698973 > I just checked the conversation in the original issue, and personally I prefer to keep the default value to True and waits for community feedback before deciding if we need to change

Re: [I] Status of testing Providers that were prepared on January 30, 2024 [airflow]

2024-01-30 Thread via GitHub
Taragolis commented on issue #37102: URL: https://github.com/apache/airflow/issues/37102#issuecomment-1917678848 I've check my change and could confirm that it work as desired -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [I] DAG in tutorial doesn't exist [airflow]

2024-01-30 Thread via GitHub
danbachar commented on issue #37097: URL: https://github.com/apache/airflow/issues/37097#issuecomment-1917652945 Do you have virtualenv installed? From: Elad Kalif ***@***.***> Sent: Tuesday, January 30, 2024 7:18:54 PM To: apache/airflow

Re: [PR] Introduce `DatasetOrTimeSchedule` [airflow]

2024-01-30 Thread via GitHub
dstandish commented on code in PR #36710: URL: https://github.com/apache/airflow/pull/36710#discussion_r1471754215 ## airflow/timetables/datasets.py: ## @@ -0,0 +1,90 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

Re: [PR] Celery provider: Add task_acks_late setting [airflow]

2024-01-30 Thread via GitHub
hussein-awala commented on PR #37066: URL: https://github.com/apache/airflow/pull/37066#issuecomment-1917645760 I just checked the conversation in the original issue, and personally I prefer to keep the default value to True and waits for community feedback before deciding if we need to

Re: [PR] Celery provider: Add task_acks_late setting [airflow]

2024-01-30 Thread via GitHub
hussein-awala commented on code in PR #37066: URL: https://github.com/apache/airflow/pull/37066#discussion_r1471747200 ## airflow/providers/celery/provider.yaml: ## @@ -276,6 +276,20 @@ config: type: float example: ~ default: "1.0" +

Re: [I] DAG in tutorial doesn't exist [airflow]

2024-01-30 Thread via GitHub
aritra24 commented on issue #37097: URL: https://github.com/apache/airflow/issues/37097#issuecomment-1917640199 @eladkal I haven't tried it out yet, will try it out in a bit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [I] DAG in tutorial doesn't exist [airflow]

2024-01-30 Thread via GitHub
eladkal commented on issue #37097: URL: https://github.com/apache/airflow/issues/37097#issuecomment-1917631022 I was not able to reproduce it. All example dags are loaded fine for me. @aritra24 were you able to reproduce? -- This is an automated message from the Apache Git Service.

Re: [I] DAG in tutorial doesn't exist [airflow]

2024-01-30 Thread via GitHub
danbachar commented on issue #37097: URL: https://github.com/apache/airflow/issues/37097#issuecomment-1917586529 Sure. PR is already out and published -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [I] DAG in tutorial doesn't exist [airflow]

2024-01-30 Thread via GitHub
aritra24 commented on issue #37097: URL: https://github.com/apache/airflow/issues/37097#issuecomment-1917585253 @danbachar if you're working on this let me assign this to you? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] Fix KPO docs argument precedence [airflow]

2024-01-30 Thread via GitHub
ketozhang commented on PR #37047: URL: https://github.com/apache/airflow/pull/37047#issuecomment-1917580816 Perhaps out of scope. `pod_template_file` and `pod_template_dict` is mutually exclusive. If both are specified then `pod_template_dict` is ignored. This should be in the API docs.

Re: [PR] Fix KPO docs argument precedence [airflow]

2024-01-30 Thread via GitHub
ketozhang commented on code in PR #37047: URL: https://github.com/apache/airflow/pull/37047#discussion_r1471693809 ## docs/apache-airflow-providers-cncf-kubernetes/operators.rst: ## @@ -77,8 +77,8 @@ You can print out the Kubernetes manifest for the pod that would be created

(airflow) branch constraints-main updated: Updating constraints. Github run id:7714067073

2024-01-30 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new 8e6c1f81b9 Updating

Re: [PR] feature: callable for template_fields [airflow]

2024-01-30 Thread via GitHub
uranusjr commented on PR #37028: URL: https://github.com/apache/airflow/pull/37028#issuecomment-1917484651 I wonder if we should turn the return value of the callable into a template and render that. Since a callable can do whatever it wants, supporting returning a template seems redundant

[I] Status of testing Providers that were prepared on January 30, 2024 [airflow]

2024-01-30 Thread via GitHub
eladkal opened a new issue, #37102: URL: https://github.com/apache/airflow/issues/37102 ### Body I have a kind request for all the contributors to the latest provider packages release. Could you please help us to test the RC versions of the providers? The guidelines on

[PR] Add documentation for packages - 2024-01-30 [airflow-site]

2024-01-30 Thread via GitHub
eladkal opened a new pull request, #946: URL: https://github.com/apache/airflow-site/pull/946 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

(airflow-site) branch add-documentation-2024-01-22 deleted (was 0d3167f060)

2024-01-30 Thread eladkal
This is an automated email from the ASF dual-hosted git repository. eladkal pushed a change to branch add-documentation-2024-01-22 in repository https://gitbox.apache.org/repos/asf/airflow-site.git was 0d3167f060 Add documentation for packages - 2024-01-22 The revisions that were on this

Re: [PR] Add documentation for packages - 2024-01-22 [airflow-site]

2024-01-30 Thread via GitHub
eladkal closed pull request #940: Add documentation for packages - 2024-01-22 URL: https://github.com/apache/airflow-site/pull/940 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

(airflow-site) branch add-documentation-2024-01-30 created (now 2bff057388)

2024-01-30 Thread eladkal
This is an automated email from the ASF dual-hosted git repository. eladkal pushed a change to branch add-documentation-2024-01-30 in repository https://gitbox.apache.org/repos/asf/airflow-site.git at 2bff057388 Add documentation for packages - 2024-01-30 This branch includes the

[PR] Add dataset conditional syntax [airflow]

2024-01-30 Thread via GitHub
sunank200 opened a new pull request, #37101: URL: https://github.com/apache/airflow/pull/37101 Title: **Introducing Logical Operators for Dataset Dependencies in Airflow** PR Message: --- We've expanded the dataset dependency handling in Airflow, building on

Re: [PR] Update start.rst [airflow]

2024-01-30 Thread via GitHub
danbachar commented on PR #37098: URL: https://github.com/apache/airflow/pull/37098#issuecomment-1917435004 Without virtualenv the DAG mentioned in the tutorial does not exist. Get Outlook for Android From: Gopal Dirisala

(airflow-site) branch add-documentation-2024-01-30 deleted (was 3d0e32f605)

2024-01-30 Thread eladkal
This is an automated email from the ASF dual-hosted git repository. eladkal pushed a change to branch add-documentation-2024-01-30 in repository https://gitbox.apache.org/repos/asf/airflow-site.git was 3d0e32f605 Add documentation for packages - 2024-01-26 (#944) The revisions that were

(airflow-site) branch add-documentation-2024-01-30 created (now 3d0e32f605)

2024-01-30 Thread eladkal
This is an automated email from the ASF dual-hosted git repository. eladkal pushed a change to branch add-documentation-2024-01-30 in repository https://gitbox.apache.org/repos/asf/airflow-site.git at 3d0e32f605 Add documentation for packages - 2024-01-26 (#944) No new revisions were

(airflow) annotated tag providers-microsoft-azure/9.0.0rc1 updated (ba053f79ed -> 9aa58ad3e1)

2024-01-30 Thread eladkal
This is an automated email from the ASF dual-hosted git repository. eladkal pushed a change to annotated tag providers-microsoft-azure/9.0.0rc1 in repository https://gitbox.apache.org/repos/asf/airflow.git *** WARNING: tag providers-microsoft-azure/9.0.0rc1 was modified! *** from

svn commit: r67054 - /dev/airflow/providers/

2024-01-30 Thread eladkal
Author: eladkal Date: Tue Jan 30 16:13:22 2024 New Revision: 67054 Log: Add artifacts for Airflow Providers 2024-01-30 Added: dev/airflow/providers/apache_airflow_providers_microsoft_azure-9.0.0-py3-none-any.whl (with props)

(airflow) branch main updated: Prepare docs for ad-hoc release of microsoft.azure provider (#37099)

2024-01-30 Thread eladkal
This is an automated email from the ASF dual-hosted git repository. eladkal pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new ba053f79ed Prepare docs for ad-hoc release of

Re: [PR] Prepare docs for ad-hoc release of microsoft.azure provider [airflow]

2024-01-30 Thread via GitHub
eladkal merged PR #37099: URL: https://github.com/apache/airflow/pull/37099 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Update start.rst [airflow]

2024-01-30 Thread via GitHub
dirrao commented on code in PR #37098: URL: https://github.com/apache/airflow/pull/37098#discussion_r1471510252 ## docs/apache-airflow/start.rst: ## @@ -24,7 +24,7 @@ This quick start guide will help you bootstrap an Airflow standalone instance on .. note:: - Successful

[PR] Prepare docs for ad-hoc release of microsoft.azure provider [airflow]

2024-01-30 Thread via GitHub
eladkal opened a new pull request, #37099: URL: https://github.com/apache/airflow/pull/37099 --- **^ Add meaningful description above** Read the **[Pull Request

Re: [PR] Update start.rst [airflow]

2024-01-30 Thread via GitHub
boring-cyborg[bot] commented on PR #37098: URL: https://github.com/apache/airflow/pull/37098#issuecomment-1917218710 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our

[PR] Update start.rst [airflow]

2024-01-30 Thread via GitHub
danbachar opened a new pull request, #37098: URL: https://github.com/apache/airflow/pull/37098 add virtualenv as a requirement --- **^ Add meaningful description above** Read the **[Pull Request

[I] DAG in tutorial doesn't exist [airflow]

2024-01-30 Thread via GitHub
danbachar opened a new issue, #37097: URL: https://github.com/apache/airflow/issues/37097 ### What do you see as an issue? After installing Airflow and running it standalone with `$ airflow standalone`, the The Quick Start tutorial on the 4th step instructs the user to "Enable the

Re: [I] DAG in tutorial doesn't exist [airflow]

2024-01-30 Thread via GitHub
boring-cyborg[bot] commented on issue #37097: URL: https://github.com/apache/airflow/issues/37097#issuecomment-1917194583 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for

[PR] docs: update description in airflow provider.yaml [airflow]

2024-01-30 Thread via GitHub
ImpressionableRaccoon opened a new pull request, #37096: URL: https://github.com/apache/airflow/pull/37096 This is for yandex provider Home page https://airflow.apache.org/docs/apache-airflow-providers-yandex/stable/index.html --- **^ Add meaningful

[PR] docs: some grammatical improvements for providers index [airflow]

2024-01-30 Thread via GitHub
ImpressionableRaccoon opened a new pull request, #37095: URL: https://github.com/apache/airflow/pull/37095 grammatical improvements for providers index template --- **^ Add meaningful description above** Read the **[Pull Request

(airflow) branch main updated: Add missing `v2-8` branches to codecov.yml (#37094)

2024-01-30 Thread taragolis
This is an automated email from the ASF dual-hosted git repository. taragolis pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 8b5a1ff5e9 Add missing `v2-8` branches to

Re: [PR] Add missing `v2-8` branches to codecov.yml [airflow]

2024-01-30 Thread via GitHub
Taragolis merged PR #37094: URL: https://github.com/apache/airflow/pull/37094 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Fix location requirement in DataflowTemplatedJobStartOperator [airflow]

2024-01-30 Thread via GitHub
shardbread commented on code in PR #37069: URL: https://github.com/apache/airflow/pull/37069#discussion_r1471186463 ## airflow/providers/google/cloud/operators/dataflow.py: ## @@ -677,6 +677,9 @@ def set_current_job(current_job): options = self.dataflow_default_options

(airflow) branch main updated: Workaround backwards-incompatibility in google-core-api 2.16.0 (#37093)

2024-01-30 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 762f544741 Workaround backwards-incompatibility in

Re: [PR] Workaround backwards-incompatibility in google-core-api 2.16.0 [airflow]

2024-01-30 Thread via GitHub
potiuk merged PR #37093: URL: https://github.com/apache/airflow/pull/37093 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[PR] Workaround backwards-incompatibility in google-core-api 2.16.0 [airflow]

2024-01-30 Thread via GitHub
potiuk opened a new pull request, #37093: URL: https://github.com/apache/airflow/pull/37093 There is a backwards-incompatible change in google.api_core.retry.AsyncRetry imports. In 2.16.0 version of google-api-core, AsyncRetry was moved to google.api_core.retry_unary_async and

Re: [PR] AIP-58: Add object storage backend for xcom [airflow]

2024-01-30 Thread via GitHub
bolkedebruin commented on code in PR #37058: URL: https://github.com/apache/airflow/pull/37058#discussion_r1471129414 ## airflow/config_templates/config.yml: ## @@ -375,6 +375,32 @@ core: type: string example: "path.to.CustomXCom" default:

Re: [PR] Update DatabricksSqlOperator to work with namedtuple [airflow]

2024-01-30 Thread via GitHub
dirrao closed pull request #37025: Update DatabricksSqlOperator to work with namedtuple URL: https://github.com/apache/airflow/pull/37025 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

(airflow) branch main updated: fix static checks for openlineage provider (#37092)

2024-01-30 Thread mobuchowski
This is an automated email from the ASF dual-hosted git repository. mobuchowski pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 674ea5dd9e fix static checks for openlineage

Re: [PR] Update DatabricksSqlOperator to work with namedtuple [airflow]

2024-01-30 Thread via GitHub
dirrao commented on PR #37025: URL: https://github.com/apache/airflow/pull/37025#issuecomment-1916724800 Closing and reopening PR to run the test cases again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] fix static checks for openlineage provider [airflow]

2024-01-30 Thread via GitHub
mobuchowski merged PR #37092: URL: https://github.com/apache/airflow/pull/37092 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Fix KPO docs argument precedence [airflow]

2024-01-30 Thread via GitHub
dirrao commented on code in PR #37047: URL: https://github.com/apache/airflow/pull/37047#discussion_r1471103997 ## docs/apache-airflow-providers-cncf-kubernetes/operators.rst: ## @@ -77,8 +77,8 @@ You can print out the Kubernetes manifest for the pod that would be created at r

Re: [PR] New document for security fixes in Airflow [airflow]

2024-01-30 Thread via GitHub
amoghrajesh commented on code in PR #37082: URL: https://github.com/apache/airflow/pull/37082#discussion_r1471081020 ## HOW_TO_FIX_A_SECURITY_ISSUE.md: ## @@ -0,0 +1,60 @@ + + +# Fixing Security Issues in Airflow + +Generally speaking, Apache Airflow addresses security issues

Re: [PR] New document for security fixes in Airflow [airflow]

2024-01-30 Thread via GitHub
amoghrajesh commented on code in PR #37082: URL: https://github.com/apache/airflow/pull/37082#discussion_r1471080150 ## HOW_TO_FIX_A_SECURITY_ISSUE.md: ## @@ -0,0 +1,60 @@ + + +# Fixing Security Issues in Airflow + +Generally speaking, Apache Airflow addresses security issues

Re: [PR] check sagemaker training job status before deferring SageMakerTrainingOperator [airflow]

2024-01-30 Thread via GitHub
pankajkoti commented on code in PR #36685: URL: https://github.com/apache/airflow/pull/36685#discussion_r1471079764 ## airflow/providers/amazon/aws/operators/sagemaker.py: ## @@ -1085,8 +1089,50 @@ def execute(self, context: Context) -> dict: raise

Re: [PR] Add GKEStartKueueInsideClusterOperator to support Kueue inside clusters [airflow]

2024-01-30 Thread via GitHub
VladaZakharova commented on PR #37072: URL: https://github.com/apache/airflow/pull/37072#issuecomment-1916680545 > There is already KubernetesCreateResourceOperator > > For creating any resource in k8s Yes, thank you for the idea, it makes sense to inherit existing logic from

[PR] fix static checks for openlineage provider [airflow]

2024-01-30 Thread via GitHub
ImpressionableRaccoon opened a new pull request, #37092: URL: https://github.com/apache/airflow/pull/37092 related to this PR: https://github.com/apache/airflow/pull/36222 --- **^ Add meaningful description above** Read the **[Pull Request

[I] Dynamic task gets "upstream_failed" when with trigger_rule='none_failed' and upstream skipped. [airflow]

2024-01-30 Thread via GitHub
kmehkeri opened a new issue, #37091: URL: https://github.com/apache/airflow/issues/37091 ### Apache Airflow version 2.8.1 ### If "Other Airflow 2 version" selected, which one? _No response_ ### What happened? When a task that uses dynamic task mapping has

[I] 404 Pod not found can be seen often [airflow]

2024-01-30 Thread via GitHub
paramjeet01 opened a new issue, #37090: URL: https://github.com/apache/airflow/issues/37090 ### Apache Airflow Provider(s) cncf-kubernetes ### Versions of Apache Airflow Providers apache-airflow-providers-cncf-kubernetes==7.11.0 ### Apache Airflow version

Re: [PR] check sagemaker training job status before deferring SageMakerTrainingOperator [airflow]

2024-01-30 Thread via GitHub
Lee-W commented on code in PR #36685: URL: https://github.com/apache/airflow/pull/36685#discussion_r1471055969 ## airflow/providers/amazon/aws/operators/sagemaker.py: ## @@ -1085,8 +1089,50 @@ def execute(self, context: Context) -> dict: raise

Re: [PR] check sagemaker training job status before deferring SageMakerTrainingOperator [airflow]

2024-01-30 Thread via GitHub
Lee-W commented on code in PR #36685: URL: https://github.com/apache/airflow/pull/36685#discussion_r1471055450 ## airflow/providers/amazon/aws/hooks/sagemaker.py: ## @@ -1300,3 +1300,13 @@ def create_auto_ml_job( if "BestCandidate" in res: return

Re: [PR] confirmation dialog box for DAG run actions [airflow]

2024-01-30 Thread via GitHub
potiuk commented on PR #35393: URL: https://github.com/apache/airflow/pull/35393#issuecomment-1916637582 Something failed last time - looks like intermittent EOF on docker build. I rebased to see if it is OK -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] Add new run type "triggered" to distinguish manual DAG runs and triggered DAG runs [airflow]

2024-01-30 Thread via GitHub
molcay commented on PR #37087: URL: https://github.com/apache/airflow/pull/37087#issuecomment-1916616603 Hi @uranusjr, I think the use case is a bit different from the datasets. Let's have an following DAG as an example; ```python3 import random from airflow import

Re: [I] common-sql SQLThresholdCheckOperator always returns error if query yields value of 0 [airflow]

2024-01-30 Thread via GitHub
boring-cyborg[bot] commented on issue #37089: URL: https://github.com/apache/airflow/issues/37089#issuecomment-1916606770 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for

[I] common-sql SQLThresholdCheckOperator always returns error if query yields value of 0 [airflow]

2024-01-30 Thread via GitHub
Don-Burns opened a new issue, #37089: URL: https://github.com/apache/airflow/issues/37089 ### Apache Airflow Provider(s) common-sql ### Versions of Apache Airflow Providers apache-airflow-providers-common-sql==1.5.2 (from constraints file) ### Apache Airflow

(airflow) branch main updated: Make Amazon Provider tests compatible with `moto>=5` (#37060)

2024-01-30 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 7ea40fe972 Make Amazon Provider tests compatible

Re: [PR] Make Amazon Provider tests compatible with `moto>=5` [airflow]

2024-01-30 Thread via GitHub
potiuk merged PR #37060: URL: https://github.com/apache/airflow/pull/37060 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [I] Upgrade our tests to moto 5 [airflow]

2024-01-30 Thread via GitHub
potiuk closed issue #37053: Upgrade our tests to moto 5 URL: https://github.com/apache/airflow/issues/37053 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] Make Amazon Provider tests compatible with `moto>=5` [airflow]

2024-01-30 Thread via GitHub
potiuk commented on code in PR #37060: URL: https://github.com/apache/airflow/pull/37060#discussion_r1471012125 ## airflow/providers/amazon/provider.yaml: ## @@ -118,10 +118,7 @@ additional-extras: devel-dependencies: - aiobotocore>=2.7.0 - aws_xray_sdk>=2.12.0 - # Moto

Re: [PR] Add GKEStartKueueInsideClusterOperator to support Kueue inside clusters [airflow]

2024-01-30 Thread via GitHub
raphaelauv commented on PR #37072: URL: https://github.com/apache/airflow/pull/37072#issuecomment-1916527506 There is already KubernetesCreateResourceOperator For creating any resource in k8s -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] Make Amazon Provider tests compatible with `moto>=5` [airflow]

2024-01-30 Thread via GitHub
Taragolis commented on code in PR #37060: URL: https://github.com/apache/airflow/pull/37060#discussion_r1470955111 ## airflow/providers/amazon/provider.yaml: ## @@ -118,10 +118,7 @@ additional-extras: devel-dependencies: - aiobotocore>=2.7.0 - aws_xray_sdk>=2.12.0 - #

Re: [PR] New document for security fixes in Airflow [airflow]

2024-01-30 Thread via GitHub
uranusjr commented on code in PR #37082: URL: https://github.com/apache/airflow/pull/37082#discussion_r1470870995 ## HOW_TO_FIX_A_SECURITY_ISSUE.md: ## @@ -0,0 +1,60 @@ + + +# Fixing Security Issues in Airflow + +Generally speaking, Apache Airflow addresses security issues

[PR] Add new run type "triggered" to distinguish manual DAG runs and triggered DAG runs [airflow]

2024-01-30 Thread via GitHub
molcay opened a new pull request, #37087: URL: https://github.com/apache/airflow/pull/37087 Add new run type "triggered" to distinguish manual DAG runs and triggered DAG runs (`TriggerDagRunOperator`). ### Justification to introduce a new RunType Before this improvement, we were

Re: [PR] Add new run type "triggered" to distinguish manual DAG runs and triggered DAG runs [airflow]

2024-01-30 Thread via GitHub
uranusjr commented on PR #37087: URL: https://github.com/apache/airflow/pull/37087#issuecomment-1916457787 Is using datasets an alternative? This is one of the improvements the mechanism brought. -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] Make Amazon Provider tests compatible with `moto>=5` [airflow]

2024-01-30 Thread via GitHub
uranusjr commented on code in PR #37060: URL: https://github.com/apache/airflow/pull/37060#discussion_r1470835092 ## airflow/providers/amazon/provider.yaml: ## @@ -118,10 +118,7 @@ additional-extras: devel-dependencies: - aiobotocore>=2.7.0 - aws_xray_sdk>=2.12.0 - #

Re: [PR] Fix macros jinja template example [airflow]

2024-01-30 Thread via GitHub
mobuchowski merged PR #36222: URL: https://github.com/apache/airflow/pull/36222 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Fix macros jinja template example [airflow]

2024-01-30 Thread via GitHub
boring-cyborg[bot] commented on PR #36222: URL: https://github.com/apache/airflow/pull/36222#issuecomment-1916403082 Awesome work, congrats on your first merged pull request! You are invited to check our [Issue Tracker](https://github.com/apache/airflow/issues) for additional

Re: [PR] AIP-58: Add object storage backend for xcom [airflow]

2024-01-30 Thread via GitHub
uranusjr commented on code in PR #37058: URL: https://github.com/apache/airflow/pull/37058#discussion_r1470830063 ## airflow/config_templates/config.yml: ## @@ -375,6 +375,32 @@ core: type: string example: "path.to.CustomXCom" default:

Re: [PR] AIP-58: Add object storage backend for xcom [airflow]

2024-01-30 Thread via GitHub
uranusjr commented on code in PR #37058: URL: https://github.com/apache/airflow/pull/37058#discussion_r1470830063 ## airflow/config_templates/config.yml: ## @@ -375,6 +375,32 @@ core: type: string example: "path.to.CustomXCom" default:

Re: [PR] New document for security fixes in Airflow [airflow]

2024-01-30 Thread via GitHub
uranusjr commented on code in PR #37082: URL: https://github.com/apache/airflow/pull/37082#discussion_r1470871945 ## HOW_TO_FIX_A_SECURITY_ISSUE.md: ## @@ -0,0 +1,60 @@ + + +# Fixing Security Issues in Airflow + +Generally speaking, Apache Airflow addresses security issues

(airflow) branch main updated: Fix macros jinja template example (#36222)

2024-01-30 Thread mobuchowski
This is an automated email from the ASF dual-hosted git repository. mobuchowski pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new c81be6f78c Fix macros jinja template example

Re: [PR] AIP-58: Add object storage backend for xcom [airflow]

2024-01-30 Thread via GitHub
uranusjr commented on code in PR #37058: URL: https://github.com/apache/airflow/pull/37058#discussion_r1470830063 ## airflow/config_templates/config.yml: ## @@ -375,6 +375,32 @@ core: type: string example: "path.to.CustomXCom" default:

Re: [I] Add two new fields (gunicorn_logconfig & gunicorn_logconfig_dict) to airflow.cfg to allow users to configure Airflow Webserver logs [airflow]

2024-01-30 Thread via GitHub
potiuk commented on issue #37078: URL: https://github.com/apache/airflow/issues/37078#issuecomment-1916382058 No. This is wrong assessment. We already have few 100s of options in airflow config and we do not want to bloat it with even more stuff. Airflow config is actually one of the

Re: [PR] Standardize deprecations in providers [part1] [airflow]

2024-01-30 Thread via GitHub
mobuchowski merged PR #36876: URL: https://github.com/apache/airflow/pull/36876 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] AIP-58: Add object storage backend for xcom [airflow]

2024-01-30 Thread via GitHub
ephraimbuddy commented on code in PR #37058: URL: https://github.com/apache/airflow/pull/37058#discussion_r1470785477 ## airflow/config_templates/config.yml: ## @@ -375,6 +375,32 @@ core: type: string example: "path.to.CustomXCom" default:

Re: [PR] Add GKEStartKueueInsideClusterOperator to support Kueue inside clusters [airflow]

2024-01-30 Thread via GitHub
VladaZakharova commented on PR #37072: URL: https://github.com/apache/airflow/pull/37072#issuecomment-1916355819 Okay, then maybe it makes sense to convert this operator to just method call inside operators for CreateClusterQueue and CreateLocalQueue? WDYT? -- This is an automated

Re: [PR] Add GKEStartKueueInsideClusterOperator to support Kueue inside clusters [airflow]

2024-01-30 Thread via GitHub
dirrao commented on PR #37072: URL: https://github.com/apache/airflow/pull/37072#issuecomment-1916339750 > Why add an operator in airflow to install `kueue` in a GKE ? > > the installation thanks to kubectl is not enough ?

Re: [PR] Add doc page with providers deprecations [airflow]

2024-01-30 Thread via GitHub
kacpermuda commented on code in PR #37075: URL: https://github.com/apache/airflow/pull/37075#discussion_r1470781860 ## docs/spelling_wordlist.txt: ## @@ -122,6 +122,7 @@ avp Avro avro aws +AwsBaseAsyncHook Review Comment: In the doc page we are providing instructions on

Re: [PR] Order XCom entries by map index in API [airflow]

2024-01-30 Thread via GitHub
uranusjr commented on code in PR #37086: URL: https://github.com/apache/airflow/pull/37086#discussion_r1470778761 ## airflow/api_connexion/endpoints/xcom_endpoint.py: ## @@ -71,7 +71,7 @@ def get_xcom_entries( query = query.where(XCom.map_index == map_index) if

Re: [PR] Add doc page with providers deprecations [airflow]

2024-01-30 Thread via GitHub
kacpermuda commented on code in PR #37075: URL: https://github.com/apache/airflow/pull/37075#discussion_r1470777540 ## docs/apache-airflow-providers/core-extensions/deprecations.rst: ## @@ -0,0 +1,57 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or

Re: [PR] Fixing a visual in breeze cheatsheet [airflow]

2024-01-30 Thread via GitHub
amoghrajesh closed pull request #37083: Fixing a visual in breeze cheatsheet URL: https://github.com/apache/airflow/pull/37083 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] AIP-58: Add object storage backend for xcom [airflow]

2024-01-30 Thread via GitHub
hussein-awala commented on PR #37058: URL: https://github.com/apache/airflow/pull/37058#issuecomment-1916322884 > @hussein-awala sure I can do that. Is that also what we want? Shouldn't we bump the requirements of the provider? I understand that this forces people to upgrade but the

[PR] Order XCom entries by map index in API [airflow]

2024-01-30 Thread via GitHub
uranusjr opened a new pull request, #37086: URL: https://github.com/apache/airflow/pull/37086 This makes the resulting list more stable. I found this in a failed CI run: https://github.com/apache/airflow/actions/runs/7698956149/job/20980348665 While it is technically possible

Re: [PR] The task is stuck in a queued state forever in case of pod launch errors [airflow]

2024-01-30 Thread via GitHub
dirrao commented on code in PR #36882: URL: https://github.com/apache/airflow/pull/36882#discussion_r1470762084 ## airflow/providers/cncf/kubernetes/provider.yaml: ## @@ -350,6 +350,15 @@ config: type: string example: ~ default: "" +

Re: [PR] The task is stuck in a queued state forever in case of pod launch errors [airflow]

2024-01-30 Thread via GitHub
dirrao commented on code in PR #36882: URL: https://github.com/apache/airflow/pull/36882#discussion_r1470762084 ## airflow/providers/cncf/kubernetes/provider.yaml: ## @@ -350,6 +350,15 @@ config: type: string example: ~ default: "" +

Re: [PR] AIP-58: Add object storage backend for xcom [airflow]

2024-01-30 Thread via GitHub
bolkedebruin commented on PR #37058: URL: https://github.com/apache/airflow/pull/37058#issuecomment-1916308845 @hussein-awala sure I can do that. Is that also what we want? Shouldn't we bump the requirements of the provider? I understand that this forces people to upgrade but the

Re: [PR] The task is stuck in a queued state forever in case of pod launch errors [airflow]

2024-01-30 Thread via GitHub
dirrao commented on PR #36882: URL: https://github.com/apache/airflow/pull/36882#issuecomment-1916298950 > > > I see you added retires counter. What do you think about custom delay between each retry of exceeded quota also? My issue is the high rate of requests to Kubernetes API and

Re: [PR] Add GKEStartKueueInsideClusterOperator to support Kueue inside clusters [airflow]

2024-01-30 Thread via GitHub
dirrao commented on PR #37072: URL: https://github.com/apache/airflow/pull/37072#issuecomment-1916289946 > > We usually use infrastructure as code to install the cluster and additional things together. what is the significance installing Kueue alone? > > Not true, we have an

Re: [PR] add service_file support to GKEPodAsyncHook [airflow]

2024-01-30 Thread via GitHub
Lee-W commented on code in PR #37081: URL: https://github.com/apache/airflow/pull/37081#discussion_r1470746026 ## airflow/providers/google/cloud/hooks/kubernetes_engine.py: ## @@ -507,33 +507,37 @@ async def get_pod(self, name: str, namespace: str) -> V1Pod: :param

Re: [PR] add service_file support to GKEPodAsyncHook [airflow]

2024-01-30 Thread via GitHub
dirrao commented on code in PR #37081: URL: https://github.com/apache/airflow/pull/37081#discussion_r1470744567 ## airflow/providers/google/cloud/hooks/kubernetes_engine.py: ## @@ -507,33 +507,37 @@ async def get_pod(self, name: str, namespace: str) -> V1Pod: :param

[I] Helm Chart Release ? [airflow]

2024-01-30 Thread via GitHub
dntosas opened a new issue, #37085: URL: https://github.com/apache/airflow/issues/37085 ### Official Helm Chart version 1.11.0 (latest released) ### Apache Airflow version 2.8.1 ### Kubernetes Version 1.28 ### Helm Chart configuration _No