tatiana opened a new issue, #32287:
URL: https://github.com/apache/airflow/issues/32287
### Apache Airflow version
2.6.2
### What happened
An Airflow user could not list any DAG Runs in the UI. The reason was that
the DAG Run `config` attribute had a value that could not be parsed in the
webserver. This error was reproduced using different versions of Airflow and -
until this date (30 June 2023) - still happens in the `main` branch.
The original error message was:
```
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 2447, in
wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1952, in
full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1821, in
handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.9/site-packages/flask/_compat.py", line 39, in
reraise
raise value
File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1950, in
full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1936, in
dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/local/lib/python3.9/site-packages/airflow/www/auth.py", line 51,
in decorated
return func(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/airflow/www/decorators.py",
line 109, in view_func
return f(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/airflow/www/decorators.py",
line 72, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/airflow/utils/session.py", line
70, in wrapper
return func(*args, session=session, **kwargs)
File "/usr/local/lib/python3.9/site-packages/airflow/www/views.py", line
2291, in tree
session.query(DagRun)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line
3373, in all
return list(self)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/loading.py",
line 100, in instances
cursor.close()
File
"/usr/local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line
68, in exit
compat.raise_(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/compat.py",
line 182, in raise_
raise exception
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/loading.py",
line 80, in instances
rows = [proc(row) for row in fetch]
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/loading.py",
line 80, in <listcomp>
rows = [proc(row) for row in fetch]
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/loading.py",
line 579, in _instance
_populate_full(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/loading.py",
line 725, in _populate_full
dict_[key] = getter(row)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/sqltypes.py",
line 1723, in process
return loads(value)
ModuleNotFoundError: No module named
'unusual_prefix_000a97efabd6e981b062d199b3a7ce82d192aeca_scenario-flow'
```
### What you think should happen instead
There are a few ways to deal with this problem. Some possibilities:
1. Users should not be allowed to create DAG Runs with a `config` that can't
be parsed on the webserver.
2. Airflow UI should list DAG Runs, which could be parsed without issues,
perhaps giving an error message for the ones that couldn't.
3. Airflow UI should not expose `config` values for DAG Runs since, at the
moment, we accept anything that could be pickled, which may not be renderable
in the UI.
### How to reproduce
The following lines can be added to `airflow/tests/www/views.py` to
reproduce a similar issue:
```
class CustomClass:
attribute = None
@pytest.fixture()
def dag_run_with_problematic_config(session):
""" The Airflow Scheduler allows users to create DAG Runs with `config`
values that
cannot be parsed in the AIrflow Webserver."""
invalid_config = {"custom_class_instance": CustomClass()}
dag = DagBag().get_dag("example_bash_operator")
execution_date = timezone.datetime(2016, 1, 9)
dr = dag.create_dagrun(
state="running",
execution_date=execution_date,
data_interval=(execution_date, execution_date),
run_id="test_dag_runs_action",
conf=invalid_config,
session=session,
)
session.add(dr)
tis = [
TaskInstance(dag.get_task("runme_0"), run_id=dr.run_id,
state="success"),
TaskInstance(dag.get_task("runme_1"), run_id=dr.run_id,
state="failed"),
]
session.bulk_save_objects(tis)
session.commit()
yield dr
session.query(DagRun).delete()
session.query(TaskInstance).delete()
def test_get_dagrun_error_problematic_dagrun_config(
session, dag_run_with_problematic_config, client_dr_without_dag_edit
):
"""Test that a config which cannot be parsed in the Airflow Webserver
does not interfere with listing DAG Runs"""
assert session.query(DagRun).filter(DagRun.dag_id ==
dag_run_with_problematic_config.dag_id).count() == 1
resp = client_dr_without_dag_edit.get("/dagrun/list/",
follow_redirects=True)
with client_dr_without_dag_edit.application.test_request_context():
url = flask.url_for(
"Airflow.graph",
dag_id=dag_run_with_problematic_config.dag_id,
execution_date=dag_run_with_problematic_config.execution_date,
)
dag_url_link = markupsafe.Markup('<a
href="{url}">{dag_id}</a>').format(
url=url, dag_id=dag_run_with_problematic_config.dag_id
)
check_content_in_response(dag_url_link, resp)
```
### Operating System
Reproduced on Debian (GNU LInux) and MacOS
### Versions of Apache Airflow Providers
Not relevant.
### Deployment
Astronomer
### Deployment details
Not relevant.
### Anything else
Original issues raised by customers: `17199` and `23588`.
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
