BretaGlac opened a new issue, #43151:
URL: https://github.com/apache/airflow/issues/43151
### Apache Airflow version
2.10.2
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
We are using custom plugin with listener on_dag_run_running. That is being
used within scheduler. It should fetch the variable defined by either Airflow
UI or Airflow CLI, but fails to do it.
`
KeyError: 'Variable monitoring_api_key does not exist'
`
I've been debugging this a bit and checked how
airflow.models.variable.Variable picks it up.
So I've used ensure_secrets_loaded and iterated through secret backends,
only without try/except block.
`
for secrets_backend in ensure_secrets_loaded():
var_val = secrets_backend.get_variable(key=key)
print(var_val)
`
Everything was fine until it reached Metastore and throw error
`
[2024-10-14T14:30:15.578+0200] {variable.py:357} ERROR - Unable to retrieve
variable from secrets backend (MetastoreBackend). Checking subsequent secrets
backend.
Traceback (most recent call last):
File
"/data01/airflow/.pyenv/versions/3.12.6/lib/python3.12/site-packages/airflow/models/variable.py",
line 353, in get_variable_from_secrets
var_val = secrets_backend.get_variable(key=key)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/data01/airflow/.pyenv/versions/3.12.6/lib/python3.12/site-packages/airflow/utils/session.py",
line 96, in wrapper
with create_session() as session:
File
"/data01/airflow/.pyenv/versions/3.12.6/lib/python3.12/contextlib.py", line
144, in __exit__
next(self.gen)
File
"/data01/airflow/.pyenv/versions/3.12.6/lib/python3.12/site-packages/airflow/utils/session.py",
line 57, in create_session
session.commit()
File
"/data01/airflow/.pyenv/versions/3.12.6/lib/python3.12/site-packages/sqlalchemy/orm/session.py",
line 1454, in commit
self._transaction.commit(_to_root=self.future)
File
"/data01/airflow/.pyenv/versions/3.12.6/lib/python3.12/site-packages/sqlalchemy/orm/session.py",
line 832, in commit
self._prepare_impl()
File
"/data01/airflow/.pyenv/versions/3.12.6/lib/python3.12/site-packages/sqlalchemy/orm/session.py",
line 800, in _prepare_impl
self.session.dispatch.before_commit(self.session)
File
"/data01/airflow/.pyenv/versions/3.12.6/lib/python3.12/site-packages/sqlalchemy/event/attr.py",
line 346, in __call__
fn(*args, **kw)
File
"/data01/airflow/.pyenv/versions/3.12.6/lib/python3.12/site-packages/airflow/utils/sqlalchemy.py",
line 424, in _validate_commit
raise RuntimeError("UNEXPECTED COMMIT - THIS WILL BREAK HA LOCKS!")
RuntimeError: UNEXPECTED COMMIT - THIS WILL BREAK HA LOCKS!
`
If variable is used from envrionment or Hashicorp vault, then everything is
fine. Only issue is if the variable exists only in Metastore.
### What you think should happen instead?
Scheduler should be able to pickup variables from Metastore as well.
### How to reproduce
Set variable with Airflow UI or Airflow CLI. Do not set variable into
environment or secrets backend.
Try to use variable within scheduler - eg. use event listener
on_dag_run_running and use
`
import logging
from airflow.models import Variable
from datetime import datetime
from airflow.listeners import hookimpl
from airflow.plugins_manager import AirflowPlugin
class Plg(AirflowPlugin):
class Listener:
@hookimpl
def on_dag_run_running(self, dag_run, msg: str):
"""
This method is called when dag run state changes to RUNNING.
"""
start_date = datetime.utcnow()
state = dag_run.get_state()
var_val = Variable.get("your_variable")
logging.info(f"LSNR Dag running, status:{start_date}
state:{state}, variable {var_val}")
`
### Operating System
Red Hat Enterprise Linux 8.10 (Ootpa)
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==8.28.0
apache-airflow-providers-celery==3.8.1
apache-airflow-providers-common-compat==1.2.0
apache-airflow-providers-common-io==1.4.0
apache-airflow-providers-common-sql==1.16.0
apache-airflow-providers-dbt-cloud==3.10.0
apache-airflow-providers-fab==1.3.0
apache-airflow-providers-ftp==3.11.0
apache-airflow-providers-google==10.22.0
apache-airflow-providers-hashicorp==3.8.0
apache-airflow-providers-http==4.13.0
apache-airflow-providers-imap==3.7.0
apache-airflow-providers-jdbc==4.5.0
apache-airflow-providers-microsoft-mssql==3.9.0
apache-airflow-providers-mysql==5.7.0
apache-airflow-providers-odbc==4.7.0
apache-airflow-providers-oracle==3.11.0
apache-airflow-providers-postgres==5.12.0
apache-airflow-providers-salesforce==5.8.0
apache-airflow-providers-sftp==4.11.0
apache-airflow-providers-slack==8.9.0
apache-airflow-providers-smtp==1.8.0
apache-airflow-providers-snowflake==5.7.0
apache-airflow-providers-sqlite==3.9.0
apache-airflow-providers-ssh==3.13.1
### Deployment
Virtualenv installation
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]