Re: [I] UNEXPECTED COMMIT - THIS WILL BREAK HA LOCKS! [airflow]
amoghrajesh commented on issue #50421: URL: https://github.com/apache/airflow/issues/50421#issuecomment-2876591107 I will self-assign this to myself and look at it when I have some bandwidth -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] UNEXPECTED COMMIT - THIS WILL BREAK HA LOCKS! [airflow]
IgV52 commented on issue #50421: URL: https://github.com/apache/airflow/issues/50421#issuecomment-2875078324 @amoghrajesh ` Traceback (most recent call last): airflow_scheduler | File "/app/.venv/lib/python3.12/site-packages/airflow/providers/amazon/aws/log/s3_task_handler.py", line 120, in write airflow_scheduler | if append and self.s3_log_exists(remote_log_location): airflow_scheduler | ^^^ airflow_scheduler | File "/app/.venv/lib/python3.12/site-packages/airflow/providers/amazon/aws/log/s3_task_handler.py", line 81, in s3_log_exists airflow_scheduler | return self.hook.check_for_key(remote_log_location) airflow_scheduler | airflow_scheduler | File "/app/.venv/lib/python3.12/site-packages/airflow/providers/amazon/aws/hooks/s3.py", line 153, in wrapper airflow_scheduler | return func(*bound_args.args, **bound_args.kwargs) airflow_scheduler | ^^^ airflow_scheduler | File "/app/.venv/lib/python3.12/site-packages/airflow/providers/amazon/aws/hooks/s3.py", line 126, in wrapper airflow_scheduler | return func(*bound_args.args, **bound_args.kwargs) airflow_scheduler | ^^^ airflow_scheduler | File "/app/.venv/lib/python3.12/site-packages/airflow/providers/amazon/aws/hooks/s3.py", line 953, in check_for_key airflow_scheduler | obj = self.head_object(key, bucket_name) airflow_scheduler | ^^ airflow_scheduler | File "/app/.venv/lib/python3.12/site-packages/airflow/providers/amazon/aws/hooks/s3.py", line 153, in wrapper airflow_scheduler | return func(*bound_args.args, **bound_args.kwargs) airflow_scheduler | ^^^ airflow_scheduler | File "/app/.venv/lib/python3.12/site-packages/airflow/providers/amazon/aws/hooks/s3.py", line 126, in wrapper airflow_scheduler | return func(*bound_args.args, **bound_args.kwargs) airflow_scheduler | ^^^ airflow_scheduler | File "/app/.venv/lib/python3.12/site-packages/airflow/providers/amazon/aws/hooks/s3.py", line 934, in head_object airflow_scheduler | return self.get_conn().head_object(Bucket=bucket_name, Key=key) airflow_scheduler | airflow_scheduler | File "/app/.venv/lib/python3.12/site-packages/botocore/client.py", line 570, in _api_call airflow_scheduler | return self._make_api_call(operation_name, kwargs) airflow_scheduler | ^^^ airflow_scheduler | File "/app/.venv/lib/python3.12/site-packages/botocore/context.py", line 123, in wrapper airflow_scheduler | return func(*args, **kwargs) airflow_scheduler |^ airflow_scheduler | File "/app/.venv/lib/python3.12/site-packages/botocore/client.py", line 1013, in _make_api_call airflow_scheduler | http, parsed_response = se
Re: [I] UNEXPECTED COMMIT - THIS WILL BREAK HA LOCKS! [airflow]
IgV52 commented on issue #50421: URL: https://github.com/apache/airflow/issues/50421#issuecomment-2873484834 @amoghrajesh  start error, MetastoreBackend use get_connection, but @provide_session adds an already used session and wrapped in prohibit_commit maybe needs refactoring airflow.jobs.scheduler_job_runner -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] UNEXPECTED COMMIT - THIS WILL BREAK HA LOCKS! [airflow]
amoghrajesh commented on issue #50421:
URL: https://github.com/apache/airflow/issues/50421#issuecomment-2872759764
@IgV52 let me explain.
I am trying to narrow down my search radius to fix the issue being faced
here. Your issue description is not very clear for me to able to reproduce the
bug you are facing and help in fixing it.
I configured my airflow with this:
```
export AIRFLOW__SECRETS__BACKEND="airflow.secrets.metastore.MetastoreBackend"
```
And then tried a simple dag that gets a connection, somewhat like:
```
from __future__ import annotations
from airflow.models.baseoperator import BaseOperator
from airflow import DAG
from airflow.sdk import Variable
from airflow.sdk.definitions.connection import Connection
class CustomOperator(BaseOperator):
def execute(self, context):
gc = Connection.get("athena_default")
print("The conn is", gc)
print("Conn uri is", gc.get_uri())
with DAG("example_get_connection_from_defn", schedule=None, catchup=False)
as dag:
CustomOperator(task_id="print_conn")
```
This till doesn't report the error you are reporting. I get this error:
```
[2025-05-12, 19:45:14] INFO - DAG bundles loaded: dags-folder:
source="airflow.dag_processing.bundles.manager.DagBundlesManager"
[2025-05-12, 19:45:14] INFO - Filling up the DagBag from
/files/dags/dags/get_connection_sdk.py: source="airflow.models.dagbag.DagBag"
[2025-05-12, 19:45:14] INFO - Secrets backends loaded for worker: count=2:
backend_classes=["MetastoreBackend","EnvironmentVariablesBackend"]:
source="supervisor"
[2025-05-12, 19:45:14] ERROR - Unable to retrieve connection from secrets
backend (MetastoreBackend). Checking subsequent secrets backend.: source="task"
RuntimeError: Direct database access via the ORM is not allowed in Airflow
[3](http://localhost:28080/dags/example_get_connection_from_defn/runs/manual__2025-05-12T14:15:13.756176+00:00/tasks/print_conn?try_number=1#3).0
File "/opt/airflow/task-sdk/src/airflow/sdk/execution_time/context.py", line
129 in _get_connection
File "/opt/airflow/airflow-core/src/airflow/utils/session.py", line 100 in
wrapper
File "/usr/local/lib/python3.9/contextlib.py", line 119 in __enter__
File "/opt/airflow/airflow-core/src/airflow/utils/session.py", line
[4](http://localhost:28080/dags/example_get_connection_from_defn/runs/manual__2025-05-12T14:15:13.756176+00:00/tasks/print_conn?try_number=1#4)1
in create_session
File "/opt/airflow/task-sdk/src/airflow/sdk/execution_time/supervisor.py",
line 253 in __init__
[2025-05-12, 19:45:14] INFO - The conn is
Connection(conn_id='athena_default', conn_type='athena', description=None,
host=None, schema=None, login=None, password=None, port=None, extra=None):
chan="stdout": source="task"
[202[5](http://localhost:28080/dags/example_get_connection_from_defn/runs/manual__2025-05-12T14:15:13.756176+00:00/tasks/print_conn?try_number=1#5)-05-12,
19:45:14] INFO - Conn uri is athena://: chan="stdout": source="task"
```
Errors, but works. Hence I am asking for input here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [I] UNEXPECTED COMMIT - THIS WILL BREAK HA LOCKS! [airflow]
IgV52 commented on issue #50421: URL: https://github.com/apache/airflow/issues/50421#issuecomment-2872639459 @amoghrajesh sorry but I don't understand why the purpose of use is important if the module doesn't work in principle. I use s3 to store logs, after the task is completed, s3_task_handler.py can't access secrets, although secrets are loaded from the database, but an error appears that does not allow using the connection. The exception should occur immediately, or you need to come up with another solution to get a session without calling commit -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] UNEXPECTED COMMIT - THIS WILL BREAK HA LOCKS! [airflow]
IgV52 commented on issue #50421:
URL: https://github.com/apache/airflow/issues/50421#issuecomment-2872517004
> [@IgV52](https://github.com/IgV52) could you please help with the DAG
here? It is not too clear from the issue.
`
from airflow.sdk import DAG
from airflow.providers.standard.operators.python import PythonOperator
from datetime import datetime
def test():
print("TEST")
with DAG(
"test",
start_date=datetime(2025, 1, 1),
) as dag:
task = PythonOperator(
task_id="test",
python_callable=test,
)
`
When using MetastoreBackend as a backend for secrets, an error occurs, but
the database query itself is executed and the result can be obtained.
The point of the error is to abort the commit, but then the entire
MetastoreBackend module breaks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [I] UNEXPECTED COMMIT - THIS WILL BREAK HA LOCKS! [airflow]
amoghrajesh commented on issue #50421: URL: https://github.com/apache/airflow/issues/50421#issuecomment-2872280679 @IgV52 could you please help with the DAG here? It is not too clear from the issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] UNEXPECTED COMMIT - THIS WILL BREAK HA LOCKS! [airflow]
boring-cyborg[bot] commented on issue #50421: URL: https://github.com/apache/airflow/issues/50421#issuecomment-2868396149 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
