Re: [PR] Fix error handling in EksCreateClusterOperator in deferrable mode [airflow]

2023-10-16 Thread via GitHub
theaadya commented on code in PR #34966: URL: https://github.com/apache/airflow/pull/34966#discussion_r1361522739 ## airflow/providers/amazon/aws/operators/eks.py: ## @@ -397,10 +397,10 @@ def deferrable_create_cluster_next(self, context: Context, event: dict[str, Any]

Re: [PR] Migrate provider-specific executor docs to providers [airflow]

2023-10-16 Thread via GitHub
eladkal commented on PR #34809: URL: https://github.com/apache/airflow/pull/34809#issuecomment-176394 Seems there are conflicts -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Migrate provider-specific executor docs to providers [airflow]

2023-10-16 Thread via GitHub
RNHTTR commented on code in PR #34809: URL: https://github.com/apache/airflow/pull/34809#discussion_r1361422911 ## docs/apache-airflow/core-concepts/executor/index.rst: ## @@ -51,28 +51,27 @@ If you want to check which executor is currently set, you can use the ``airflow

[PR] clear_not_launched_queued_tasks optimization using kube list pods bat… [airflow]

2023-10-16 Thread via GitHub
dirrao opened a new pull request, #34985: URL: https://github.com/apache/airflow/pull/34985 Problem: Airflow running the clear_not_launched_queued_tasks function on a certain frequency (default 30 seconds). When we run the airflow on a large Kube cluster (pods more than > 5K). Internally

Re: [PR] Update GCS hook to get crc32c hash for CMEK-protected objects [airflow]

2023-10-16 Thread via GitHub
heaven00 commented on PR #34982: URL: https://github.com/apache/airflow/pull/34982#issuecomment-1765512475 sure, so the function that has been changed is here is a private function, so we can't test the logic that has been added but from what I can understand there should be some

Re: [PR] Update GCS hook to get crc32c hash for CMEK-protected objects [airflow]

2023-10-16 Thread via GitHub
dmedora commented on PR #34982: URL: https://github.com/apache/airflow/pull/34982#issuecomment-1765496038 > Shouldn't this need a test against whatever that blob should be? Not sure I understand, could you please explain further or with an example? Thanks. -- This is an automated

Re: [PR] Update GCS hook to get crc32c hash for CMEK-protected objects [airflow]

2023-10-16 Thread via GitHub
heaven00 commented on PR #34982: URL: https://github.com/apache/airflow/pull/34982#issuecomment-1765470742 Shouldn't this need a test against whatever that blob should be? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [I] Running tasks marked as skipped on DagRun timeout [airflow]

2023-10-16 Thread via GitHub
github-actions[bot] closed issue #30264: Running tasks marked as skipped on DagRun timeout URL: https://github.com/apache/airflow/issues/30264 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] Running tasks marked as skipped on DagRun timeout [airflow]

2023-10-16 Thread via GitHub
github-actions[bot] commented on issue #30264: URL: https://github.com/apache/airflow/issues/30264#issuecomment-1765457628 This issue has been closed because it has not received response from the issue author. -- This is an automated message from the Apache Git Service. To respond to the

Re: [I] psycopg2 deadlock detected after task run completed [airflow]

2023-10-16 Thread via GitHub
github-actions[bot] commented on issue #34357: URL: https://github.com/apache/airflow/issues/34357#issuecomment-1765457494 This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further

Re: [PR] Move Rest API auth related methods to FAB auth manager [airflow]

2023-10-16 Thread via GitHub
o-nikolas commented on code in PR #34924: URL: https://github.com/apache/airflow/pull/34924#discussion_r1361313195 ## airflow/auth/managers/fab/api/auth/backend/kerberos_auth.py: ## @@ -0,0 +1,153 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more

Re: [PR] AWS ECS Executor [airflow]

2023-10-16 Thread via GitHub
ferruzzi commented on code in PR #34381: URL: https://github.com/apache/airflow/pull/34381#discussion_r1361311213 ## airflow/providers/amazon/aws/executors/ecs/utils.py: ## @@ -0,0 +1,272 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor

Re: [PR] Move Rest API auth related methods to FAB auth manager [airflow]

2023-10-16 Thread via GitHub
o-nikolas commented on code in PR #34924: URL: https://github.com/apache/airflow/pull/34924#discussion_r1361311059 ## airflow/auth/managers/fab/api/auth/backend/kerberos_auth.py: ## @@ -0,0 +1,153 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more

Re: [PR] Move Rest API auth related methods to FAB auth manager [airflow]

2023-10-16 Thread via GitHub
vincbeck commented on code in PR #34924: URL: https://github.com/apache/airflow/pull/34924#discussion_r1361303612 ## airflow/auth/managers/fab/api/auth/backend/kerberos_auth.py: ## @@ -0,0 +1,153 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more

Re: [PR] Move Rest API auth related methods to FAB auth manager [airflow]

2023-10-16 Thread via GitHub
vincbeck commented on code in PR #34924: URL: https://github.com/apache/airflow/pull/34924#discussion_r1361304073 ## airflow/auth/managers/fab/api/auth/backend/kerberos_auth.py: ## @@ -0,0 +1,153 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more

Re: [PR] Move Rest API auth related methods to FAB auth manager [airflow]

2023-10-16 Thread via GitHub
o-nikolas commented on code in PR #34924: URL: https://github.com/apache/airflow/pull/34924#discussion_r1361299228 ## airflow/auth/managers/fab/api/auth/backend/kerberos_auth.py: ## @@ -0,0 +1,153 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more

Re: [PR] Use auth manager `is_authorized_` APIs to check user permissions in Rest API [airflow]

2023-10-16 Thread via GitHub
vincbeck commented on code in PR #34317: URL: https://github.com/apache/airflow/pull/34317#discussion_r1361283764 ## airflow/api_connexion/security.py: ## @@ -48,10 +61,194 @@ def requires_access_decorator(func: T): @wraps(func) def decorated(*args, **kwargs):

Re: [PR] Use auth manager `is_authorized_` APIs to check user permissions in Rest API [airflow]

2023-10-16 Thread via GitHub
vincbeck commented on code in PR #34317: URL: https://github.com/apache/airflow/pull/34317#discussion_r1361283427 ## airflow/auth/managers/fab/fab_auth_manager.py: ## @@ -65,22 +79,24 @@ CLICommand, ) -_MAP_METHOD_NAME_TO_FAB_ACTION_NAME: dict[ResourceMethod,

[airflow] branch main updated: Optimise and migrate to SA2-compatible syntax for TaskReschedule (#33720)

2023-10-16 Thread ephraimanierobi
This is an automated email from the ASF dual-hosted git repository. ephraimanierobi pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 85fd0e1102 Optimise and migrate to

Re: [PR] Optimise and migrate to SA2-compatible syntax for TaskReschedule [airflow]

2023-10-16 Thread via GitHub
ephraimbuddy merged PR #33720: URL: https://github.com/apache/airflow/pull/33720 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [I] Allow UPSERT when call `SnowflakeHook.insert_rows` [airflow]

2023-10-16 Thread via GitHub
Taragolis commented on issue #34860: URL: https://github.com/apache/airflow/issues/34860#issuecomment-1765265188 [REPLACE](https://dev.mysql.com/doc/refman/8.0/en/replace.html) it is UPSERT variant in MySQL, I guess years ago it was added for MySQL and after that comes to generic DBApiHook

Re: [I] Users Error [airflow]

2023-10-16 Thread via GitHub
Taragolis closed issue #34970: Users Error URL: https://github.com/apache/airflow/issues/34970 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [I] Users Error [airflow]

2023-10-16 Thread via GitHub
Taragolis commented on issue #34970: URL: https://github.com/apache/airflow/issues/34970#issuecomment-1765257873 Seems like you installed WTForms greater or equal than 3.1.0 see related issues: - https://github.com/dpgaspar/Flask-AppBuilder/issues/2137 -

Re: [I] Users Error [airflow]

2023-10-16 Thread via GitHub
atessier6 commented on issue #34970: URL: https://github.com/apache/airflow/issues/34970#issuecomment-1765241787 I've got the same issue, here's the traceback: ``` [2023-10-16T16:37:54.854-0400] {app.py:1744} ERROR - Exception on /users/list/ [GET] Traceback (most recent call

Re: [I] Skipping an Airflow task make the dynamic task skipped too even with TriggerRule.NONE_FAILED [airflow]

2023-10-16 Thread via GitHub
Taragolis commented on issue #34754: URL: https://github.com/apache/airflow/issues/34754#issuecomment-1765238654 I've unable to reproduce on current main, maybe it already fixed (I can't find related issue/PR) However this one worked as expected ```python from

Re: [PR] Glue DataBrew operator [airflow]

2023-10-16 Thread via GitHub
vincbeck merged PR #34807: URL: https://github.com/apache/airflow/pull/34807 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[airflow] branch main updated: Glue `DataBrew` operator (#34807)

2023-10-16 Thread vincbeck
This is an automated email from the ASF dual-hosted git repository. vincbeck pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 4a3567 Glue `DataBrew` operator (#34807)

Re: [I] Airflow Operator for Starting Amazon Glue DataBrew Job [airflow]

2023-10-16 Thread via GitHub
vincbeck closed issue #22037: Airflow Operator for Starting Amazon Glue DataBrew Job URL: https://github.com/apache/airflow/issues/22037 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Sets EcsRunTaskOperator default waiter duration to 70 days [airflow]

2023-10-16 Thread via GitHub
boring-cyborg[bot] commented on PR #34928: URL: https://github.com/apache/airflow/pull/34928#issuecomment-1765218061 Awesome work, congrats on your first merged pull request! You are invited to check our [Issue Tracker](https://github.com/apache/airflow/issues) for additional

Re: [PR] Sets EcsRunTaskOperator default waiter duration to 70 days [airflow]

2023-10-16 Thread via GitHub
vincbeck merged PR #34928: URL: https://github.com/apache/airflow/pull/34928 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[airflow] branch main updated: Set `EcsRunTaskOperator` default waiter duration to 70 days (#34928)

2023-10-16 Thread vincbeck
This is an automated email from the ASF dual-hosted git repository. vincbeck pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new b392f66c42 Set `EcsRunTaskOperator` default

Re: [I] EcsRunTaskOperator -`date value out of range` on deferrable execution - default waiter_max_attempts [airflow]

2023-10-16 Thread via GitHub
vincbeck closed issue #34889: EcsRunTaskOperator -`date value out of range` on deferrable execution - default waiter_max_attempts URL: https://github.com/apache/airflow/issues/34889 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] Update GCS hook to get crc32c hash for CMEK-protected objects [airflow]

2023-10-16 Thread via GitHub
boring-cyborg[bot] commented on PR #34982: URL: https://github.com/apache/airflow/pull/34982#issuecomment-1765202922 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution

[PR] Update GCS hook to get crc32c hash for CMEK-protected objects [airflow]

2023-10-16 Thread via GitHub
dmedora opened a new pull request, #34982: URL: https://github.com/apache/airflow/pull/34982 Closes: #34980 Updated _prepare_sync_plan function to get blobs using the Objects Get API (.get_blob(..)) so that their crc32c value is included in the returned Blob object. As discussed in

Re: [I] Allow UPSERT when call `SnowflakeHook.insert_rows` [airflow]

2023-10-16 Thread via GitHub
DEworkerDE commented on issue #34860: URL: https://github.com/apache/airflow/issues/34860#issuecomment-1765200214 @Taragolis thx, but maybe 'upsert' here is not really valid name for the issue? Because logic of 'replace into' and 'upsert' both have different idea, where upsert will

Re: [I] Mapped tasks with `operator_extra_links` property object causes SerializationError [airflow]

2023-10-16 Thread via GitHub
Taragolis closed issue #25243: Mapped tasks with `operator_extra_links` property object causes SerializationError URL: https://github.com/apache/airflow/issues/25243 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [I] Mapped tasks with `operator_extra_links` property object causes SerializationError [airflow]

2023-10-16 Thread via GitHub
Taragolis commented on issue #25243: URL: https://github.com/apache/airflow/issues/25243#issuecomment-1765193291 - Should be finally resolved in #34506 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Use base classes for AWS Lambda Operators/Sensors [airflow]

2023-10-16 Thread via GitHub
Taragolis commented on code in PR #34890: URL: https://github.com/apache/airflow/pull/34890#discussion_r1361213575 ## airflow/providers/amazon/aws/operators/lambda_function.py: ## @@ -62,13 +61,15 @@ class LambdaCreateFunctionOperator(BaseOperator): :param aws_conn_id: The

Re: [I] Buggy Cleanup job on Airflow 2.7.2 [airflow]

2023-10-16 Thread via GitHub
potiuk closed issue #34975: Buggy Cleanup job on Airflow 2.7.2 URL: https://github.com/apache/airflow/issues/34975 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [I] Allow UPSERT when call `SnowflakeHook.insert_rows` [airflow]

2023-10-16 Thread via GitHub
Taragolis commented on issue #34860: URL: https://github.com/apache/airflow/issues/34860#issuecomment-176517 My assumption this method broken for multiple Hooks which depend on [`common.sql`

Re: [I] Buggy Cleanup job on Airflow 2.7.2 [airflow]

2023-10-16 Thread via GitHub
hakuno commented on issue #34975: URL: https://github.com/apache/airflow/issues/34975#issuecomment-1765168600 So the workaround on Helm chart is `` cleanup: env: - name: "AIRFLOW__CORE__EXECUTOR" value: "CeleryKubernetesExecutor" ```

Re: [PR] Open search integrations [airflow]

2023-10-16 Thread via GitHub
cjames23 commented on PR #34693: URL: https://github.com/apache/airflow/pull/34693#issuecomment-1765158146 Closing this PR in favor of actually just adding the auth required in the OpenSearch provider to simplify dependency management and code management. I will open a new PR in the coming

Re: [PR] Open search integrations [airflow]

2023-10-16 Thread via GitHub
cjames23 closed pull request #34693: Open search integrations URL: https://github.com/apache/airflow/pull/34693 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [I] GCSSynchronizeBucketsOperator does not retrieve crc32c hash for CMEK objects [airflow]

2023-10-16 Thread via GitHub
boring-cyborg[bot] commented on issue #34980: URL: https://github.com/apache/airflow/issues/34980#issuecomment-1765150130 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for

[I] GCSSynchronizeBucketsOperator does not retrieve crc32c hash for CMEK objects [airflow]

2023-10-16 Thread via GitHub
dmedora opened a new issue, #34980: URL: https://github.com/apache/airflow/issues/34980 ### Apache Airflow version 2.7.2 ### What happened The GCSSynchronizeBucketsOperator eventually calls the `_prepare_sync_plan` function in GCS' `hooks.py`. This function retrieves

Re: [I] Dag File Processor Does Not Detect Dynamically Generated Dags [airflow]

2023-10-16 Thread via GitHub
Taragolis closed issue #34978: Dag File Processor Does Not Detect Dynamically Generated Dags URL: https://github.com/apache/airflow/issues/34978 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] Buggy Cleanup job on Airflow 2.7.2 [airflow]

2023-10-16 Thread via GitHub
Taragolis commented on issue #34975: URL: https://github.com/apache/airflow/issues/34975#issuecomment-1765146479 Then try to set `CeleryKubernetesExecutor`, executor related commands moved into the providers and injected into the CLI depend on configured executor. Locally it also

Re: [I] Airflow web doesn't work when set sql_alchemy_pool_enabled to false [airflow]

2023-10-16 Thread via GitHub
Taragolis commented on issue #34950: URL: https://github.com/apache/airflow/issues/34950#issuecomment-1765135617 > Already fixed at v3.0.0, can airflow upgrade the version of Flask-SQLAlchemy to 3.0.0 or higher? Until FAB (Flask-AppBuilder) won't support it we can't upgrade it.

[airflow] branch main updated: Add `check_interval` and `max_attempts` as parameter of `DynamoDBToS3Operator` (#34972)

2023-10-16 Thread ferruzzi
This is an automated email from the ASF dual-hosted git repository. ferruzzi pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new b1196460db Add `check_interval` and

Re: [PR] Add `check_interval` and `max_attempts` as parameter of `DynamoDBToS3Operator` [airflow]

2023-10-16 Thread via GitHub
ferruzzi merged PR #34972: URL: https://github.com/apache/airflow/pull/34972 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [I] Dag File Processor Does Not Detect Dynamically Generated Dags [airflow]

2023-10-16 Thread via GitHub
boring-cyborg[bot] commented on issue #34978: URL: https://github.com/apache/airflow/issues/34978#issuecomment-1765111534 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for

[I] Dag File Processor Does Not Detect Dynamically Generated Dags [airflow]

2023-10-16 Thread via GitHub
respecht opened a new issue, #34978: URL: https://github.com/apache/airflow/issues/34978 ### Apache Airflow version 2.7.2 ### What happened If a DAG is dynamically generated, with the code for that generation in a separate file, the Dag File Processor will not pick up

Re: [PR] Use base classes for AWS Lambda Operators/Sensors [airflow]

2023-10-16 Thread via GitHub
ferruzzi commented on code in PR #34890: URL: https://github.com/apache/airflow/pull/34890#discussion_r1361153782 ## airflow/providers/amazon/aws/operators/lambda_function.py: ## @@ -62,13 +61,15 @@ class LambdaCreateFunctionOperator(BaseOperator): :param aws_conn_id: The

Re: [PR] Use base classes for AWS Lambda Operators/Sensors [airflow]

2023-10-16 Thread via GitHub
ferruzzi commented on code in PR #34890: URL: https://github.com/apache/airflow/pull/34890#discussion_r1361153782 ## airflow/providers/amazon/aws/operators/lambda_function.py: ## @@ -62,13 +61,15 @@ class LambdaCreateFunctionOperator(BaseOperator): :param aws_conn_id: The

Re: [I] Buggy Cleanup job on Airflow 2.7.2 [airflow]

2023-10-16 Thread via GitHub
hakuno commented on issue #34975: URL: https://github.com/apache/airflow/issues/34975#issuecomment-1765103855 @Taragolis I tested it on my running scheduler pod (healthy) and it worked like a charm. Also it worked with `CeleryKubernetesExecutor`. But I need the Celery

Re: [I] Depth-first dynamic task execution fails entire DAG when using `dag.test()` [airflow]

2023-10-16 Thread via GitHub
Taragolis closed issue #31944: Depth-first dynamic task execution fails entire DAG when using `dag.test()` URL: https://github.com/apache/airflow/issues/31944 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [I] SIGTERM not catched on callbacks [airflow]

2023-10-16 Thread via GitHub
Taragolis closed issue #34471: SIGTERM not catched on callbacks URL: https://github.com/apache/airflow/issues/34471 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [I] Buggy Cleanup job on Airflow 2.7.2 [airflow]

2023-10-16 Thread via GitHub
Taragolis commented on issue #34975: URL: https://github.com/apache/airflow/issues/34975#issuecomment-1765093467 What happen if you explicitly set to KubernetesExecutor before run comman? E.g. ```shell ❯ export AIRFLOW__CORE__EXECUTOR="KubernetesExecutor" ❯ airflow kubernetes

Re: [PR] Add Open Search Provider [airflow]

2023-10-16 Thread via GitHub
ferruzzi commented on PR #34705: URL: https://github.com/apache/airflow/pull/34705#issuecomment-1765090521 Nice, congrats! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Added "overrides" parameter for CloudRunExecuteJobOperator [airflow]

2023-10-16 Thread via GitHub
ChloeSheasby commented on PR #34874: URL: https://github.com/apache/airflow/pull/34874#issuecomment-1765064695 Functionality is there and will be released in v0.10.0: https://github.com/googleapis/python-run/pull/127 -- This is an automated message from the Apache Git Service. To respond

[I] Buggy Cleanup job on Airflow 2.7.2 [airflow]

2023-10-16 Thread via GitHub
hakuno opened a new issue, #34975: URL: https://github.com/apache/airflow/issues/34975 ### Apache Airflow version 2.7.2 ### What happened The cleanup job doesn't work after the upgrade. The container calls ``` airflow kubernetes cleanup-pods

Re: [I] Buggy Cleanup job on Airflow 2.7.2 [airflow]

2023-10-16 Thread via GitHub
boring-cyborg[bot] commented on issue #34975: URL: https://github.com/apache/airflow/issues/34975#issuecomment-1765046315 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for

Re: [PR] Fix _SECRET and _CMD broker configuration [airflow]

2023-10-16 Thread via GitHub
JonnyWaffles commented on PR #34782: URL: https://github.com/apache/airflow/pull/34782#issuecomment-1765011741 It looks like the test kit [uses](https://github.com/apache/airflow/blob/ad15af5cab6fc7aebb10d194fd5ca28a5e185492/tests/providers/celery/executors/test_celery_executor.py#L71) the

Re: [I] DAGs Stuck in Queued State with Google Composer and Airflow 2.5.3 [airflow]

2023-10-16 Thread via GitHub
Taragolis closed issue #34967: DAGs Stuck in Queued State with Google Composer and Airflow 2.5.3 URL: https://github.com/apache/airflow/issues/34967 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [I] Token expiring with deferred SnowflakeOperator [airflow]

2023-10-16 Thread via GitHub
Taragolis commented on issue #34968: URL: https://github.com/apache/airflow/issues/34968#issuecomment-1764990241 > /usr/local/lib/python3.11/site-packages/astronomer/providers/snowflake/operators/snowflake.py" This provider is not a part of [Community

Re: [I] Token expiring with deferred SnowflakeOperator [airflow]

2023-10-16 Thread via GitHub
Taragolis closed issue #34968: Token expiring with deferred SnowflakeOperator URL: https://github.com/apache/airflow/issues/34968 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] Users Error [airflow]

2023-10-16 Thread via GitHub
Taragolis commented on issue #34970: URL: https://github.com/apache/airflow/issues/34970#issuecomment-1764984295 @renato-rt could you post full traceback as a text -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [I] PythonVirtualEnv Operator expecting output dir 'script.out' causing DAG to fail [airflow]

2023-10-16 Thread via GitHub
potiuk commented on issue #34958: URL: https://github.com/apache/airflow/issues/34958#issuecomment-1764981102 Feel free -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [I] Backfill job detects deadlock over dynamic task mapping [airflow]

2023-10-16 Thread via GitHub
Taragolis commented on issue #33994: URL: https://github.com/apache/airflow/issues/33994#issuecomment-1764972648 Current backfill logic is a complicated, and should be refactored during - https://github.com/apache/airflow/issues/33249 -- This is an automated message from the Apache Git

Re: [I] PythonVirtualEnv Operator expecting output dir 'script.out' causing DAG to fail [airflow]

2023-10-16 Thread via GitHub
2Rahul02 commented on issue #34958: URL: https://github.com/apache/airflow/issues/34958#issuecomment-1764967197 Hey, i wanna work on it. can you please assign it to me. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Let auth managers provide their own API endpoints [airflow]

2023-10-16 Thread via GitHub
vincbeck commented on code in PR #34349: URL: https://github.com/apache/airflow/pull/34349#discussion_r1361023435 ## airflow/www/extensions/init_views.py: ## @@ -308,5 +318,16 @@ def init_api_experimental(app): "The authenticated user has full access.",

Re: [PR] Let auth managers provide their own API endpoints [airflow]

2023-10-16 Thread via GitHub
vincbeck commented on code in PR #34349: URL: https://github.com/apache/airflow/pull/34349#discussion_r1361022706 ## airflow/www/extensions/init_views.py: ## @@ -308,5 +318,16 @@ def init_api_experimental(app): "The authenticated user has full access.",

[airflow] branch constraints-main updated: Updating constraints. Github run id:6536358542

2023-10-16 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new 7e4d36aec0 Updating

Re: [I] Status of testing Providers that were prepared on October 13, 2023 [airflow]

2023-10-16 Thread via GitHub
asherkhb commented on issue #34930: URL: https://github.com/apache/airflow/issues/34930#issuecomment-1764912184 Confirming Amazon 8.8.0rc1 resolves the issues related to my changes  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] Introduce a generic export for containerized executor logging [airflow]

2023-10-16 Thread via GitHub
o-nikolas commented on code in PR #34903: URL: https://github.com/apache/airflow/pull/34903#discussion_r1360963519 ## airflow/cli/commands/task_command.py: ## @@ -326,7 +326,7 @@ def _move_task_handlers_to_root(ti: TaskInstance) -> Generator[None, None, None]

Re: [PR] Add example PSRP DAG [airflow]

2023-10-16 Thread via GitHub
Taragolis commented on PR #34907: URL: https://github.com/apache/airflow/pull/34907#issuecomment-1764830481 Since you create System test it would be also nice to add examples to the https://airflow.apache.org/docs/apache-airflow-providers-microsoft-psrp/stable/operators/index.html

Re: [PR] Let auth managers provide their own API endpoints [airflow]

2023-10-16 Thread via GitHub
potiuk commented on code in PR #34349: URL: https://github.com/apache/airflow/pull/34349#discussion_r1360920084 ## airflow/www/extensions/init_views.py: ## @@ -308,5 +318,16 @@ def init_api_experimental(app): "The authenticated user has full access.",

Re: [PR] Update documentation to enable test connection [airflow]

2023-10-16 Thread via GitHub
potiuk merged PR #34905: URL: https://github.com/apache/airflow/pull/34905 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Update documentation to enable test connection [airflow]

2023-10-16 Thread via GitHub
boring-cyborg[bot] commented on PR #34905: URL: https://github.com/apache/airflow/pull/34905#issuecomment-1764827310 Awesome work, congrats on your first merged pull request! You are invited to check our [Issue Tracker](https://github.com/apache/airflow/issues) for additional

[airflow] branch main updated: Update documentation to enable test connection (#34905)

2023-10-16 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new ad15af5cab Update documentation to enable test

Re: [I] Status of testing Providers that were prepared on October 13, 2023 [airflow]

2023-10-16 Thread via GitHub
hussein-awala commented on issue #34930: URL: https://github.com/apache/airflow/issues/34930#issuecomment-1764797360 I checked if all my changes are present, and I tested some of them, all looks good :+1: -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] Use auth manager `is_authorized_` APIs to check user permissions in Rest API [airflow]

2023-10-16 Thread via GitHub
vincbeck commented on code in PR #34317: URL: https://github.com/apache/airflow/pull/34317#discussion_r1360872279 ## airflow/auth/managers/fab/fab_auth_manager.py: ## @@ -65,22 +79,24 @@ CLICommand, ) -_MAP_METHOD_NAME_TO_FAB_ACTION_NAME: dict[ResourceMethod,

Re: [PR] Use auth manager `is_authorized_` APIs to check user permissions in Rest API [airflow]

2023-10-16 Thread via GitHub
vincbeck commented on code in PR #34317: URL: https://github.com/apache/airflow/pull/34317#discussion_r1360871612 ## airflow/www/security_manager.py: ## @@ -269,125 +272,47 @@ def get_user_roles(user=None): user = g.user return user.roles -def

[airflow] branch main updated (9b06798a28 -> 6bb6ae8f9f)

2023-10-16 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 9b06798a28 Use requires_access to check read permission on dag instead of checking it explicitly (#34940) add

Re: [PR] `PostgresOperator` should not overwrite `SQLExecuteQueryOperator.template_fields` [airflow]

2023-10-16 Thread via GitHub
potiuk merged PR #34969: URL: https://github.com/apache/airflow/pull/34969 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[airflow] branch main updated: Use requires_access to check read permission on dag instead of checking it explicitly (#34940)

2023-10-16 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 9b06798a28 Use requires_access to check read

Re: [PR] Use requires_access to check read permission on dag instead of checking it explicitly [airflow]

2023-10-16 Thread via GitHub
potiuk merged PR #34940: URL: https://github.com/apache/airflow/pull/34940 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[PR] Add `check_interval` and `max_attempts` as parameter of `DynamoDBToS3Operator` [airflow]

2023-10-16 Thread via GitHub
vincbeck opened a new pull request, #34972: URL: https://github.com/apache/airflow/pull/34972 Add `check_interval` and `max_attempts` as parameter of `DynamoDBToS3Operator` to override default values if needed. I modified the system test `example_dynamodb_to_s3.py` as well to

[airflow] branch dependabot/npm_and_yarn/airflow/www/undici-5.26.3 created (now c288bda4bf)

2023-10-16 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/npm_and_yarn/airflow/www/undici-5.26.3 in repository https://gitbox.apache.org/repos/asf/airflow.git at c288bda4bf Bump undici from 5.19.1 to 5.26.3 in /airflow/www No new

[PR] Bump undici from 5.19.1 to 5.26.3 in /airflow/www [airflow]

2023-10-16 Thread via GitHub
dependabot[bot] opened a new pull request, #34971: URL: https://github.com/apache/airflow/pull/34971 Bumps [undici](https://github.com/nodejs/undici) from 5.19.1 to 5.26.3. Release notes Sourced from https://github.com/nodejs/undici/releases;>undici's releases. v5.26.3 No

Re: [PR] Fix error handling in EksCreateClusterOperator in deferrable mode [airflow]

2023-10-16 Thread via GitHub
Taragolis commented on code in PR #34966: URL: https://github.com/apache/airflow/pull/34966#discussion_r1360820902 ## airflow/providers/amazon/aws/operators/eks.py: ## @@ -397,10 +397,10 @@ def deferrable_create_cluster_next(self, context: Context, event: dict[str, Any]

Re: [I] DAGs Stuck in Queued State with Google Composer and Airflow 2.5.3 [airflow]

2023-10-16 Thread via GitHub
eladkal commented on issue #34967: URL: https://github.com/apache/airflow/issues/34967#issuecomment-1764700052 Queue state issues were mostly resolved with Airflow 2.6.0 I suggest to update Airflow version first. -- This is an automated message from the Apache Git Service. To

Re: [PR] Fix error handling in EksCreateClusterOperator in deferrable mode [airflow]

2023-10-16 Thread via GitHub
Taragolis commented on code in PR #34966: URL: https://github.com/apache/airflow/pull/34966#discussion_r1360827505 ## airflow/providers/amazon/aws/operators/eks.py: ## @@ -397,10 +397,10 @@ def deferrable_create_cluster_next(self, context: Context, event: dict[str, Any]

Re: [PR] Airflow-34909 - Fix gcs listing - ensure blobs are loaded [airflow]

2023-10-16 Thread via GitHub
atrbgithub commented on PR #34919: URL: https://github.com/apache/airflow/pull/34919#issuecomment-1764678627 Please see [here](https://github.com/googleapis/python-storage/issues/863#issuecomment-1245740122) for more info. In our case `blobs.prefixes` was not being populated.

Re: [PR] Fix error handling in EksCreateClusterOperator in deferrable mode [airflow]

2023-10-16 Thread via GitHub
vincbeck commented on PR #34966: URL: https://github.com/apache/airflow/pull/34966#issuecomment-1764642390 @Taragolis -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [I] Users Error [airflow]

2023-10-16 Thread via GitHub
boring-cyborg[bot] commented on issue #34970: URL: https://github.com/apache/airflow/issues/34970#issuecomment-1764631476 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for

[I] Users Error [airflow]

2023-10-16 Thread via GitHub
renato-rt opened a new issue, #34970: URL: https://github.com/apache/airflow/issues/34970 ### Apache Airflow version 2.7.2 ### What happened When clicking on 'List Users' I am getting an error called: 'Airflow is used by many users, and it is very likely that others

[airflow] branch main updated: Some improvements/fixes for dag_run and task_instance endpoints (#34942)

2023-10-16 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 84d99404bb Some improvements/fixes for dag_run and

Re: [PR] Some improvements/fixes for dag_run and task_instance endpoints [airflow]

2023-10-16 Thread via GitHub
potiuk merged PR #34942: URL: https://github.com/apache/airflow/pull/34942 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Some improvements/fixes for dag_run and task_instance endpoints [airflow]

2023-10-16 Thread via GitHub
vincbeck commented on PR #34942: URL: https://github.com/apache/airflow/pull/34942#issuecomment-1764625106 > Oh yeah. Two days ago I noticed the same problems while reviewing #34317 > > See my comments here: [#34317

  1   2   >