nickwood opened a new issue, #33884:
URL: https://github.com/apache/airflow/issues/33884
### Apache Airflow version
2.7.0
### What happened
I have a DAG with the following operator defined:
`
delete_old_from_metadata_db = BashOperator(
task_id="delete_old_from_metadata_db",
bash_command=f"airflow db clean --clean-before-timestamp '2023-08-29
12:00:00' --skip-archive --yes"
)
`
On 2.6.3 this correctly deleted everything from the airflow metadata
database prior to the specified date. I'm using CeleryExecutor
On 2.7.0 this fails however, with the following logs:
[2023-08-29, 12:18:11 UTC] {subprocess.py:63} INFO - Tmp dir root location:
/tmp
[2023-08-29, 12:18:11 UTC] {subprocess.py:75} INFO - Running command:
['/bin/bash', '-c', "*** db clean --clean-before-timestamp
'2023-08-29T12:17:09.799435+00:00' --skip-archive --yes"]
[2023-08-29, 12:18:11 UTC] {subprocess.py:86} INFO - Output:
[2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO -
/home/***/.local/lib/python3.9/site-packages/***/utils/db.py:1037 SAWarning:
Skipped unsupported reflection of expression-based index idx_ab_user_username
[2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO -
/home/***/.local/lib/python3.9/site-packages/***/utils/db.py:1037 SAWarning:
Skipped unsupported reflection of expression-based index
idx_ab_register_user_username
[2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO -
/home/***/.local/lib/python3.9/site-packages/***/utils/db_cleanup.py:174
SAWarning: Class CreateTableAs will not make use of SQL compilation caching as
it does not set the 'inherit_cache' attribute to ``True``. This can have
significant performance implications including some performance degradations in
comparison to prior SQLAlchemy versions. Set this attribute to True if this
object can make use of the cache key generated by the superclass.
Alternatively, this attribute may be set to False which will disable this
warning. (Background on this error at: https://sqlalche.me/e/14/cprf)
[2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO -
[2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO - Checking table
callback_request
[2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO - Found 0 rows meeting
deletion criteria.
[2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO -
[2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO - Checking table
celery_taskmeta
[2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO - Found 22 rows meeting
deletion criteria.
[2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO - Performing Delete...
[2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO - Moving data to table
_***_deleted__celery_taskmeta__20230829121812
[2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO - Traceback (most recent
call last):
[2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO - File
"/home/***/.local/bin/***", line 8, in <module>
[2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO - sys.exit(main())
[2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO - File
"/home/***/.local/lib/python3.9/site-packages/***/__main__.py", line 60, in main
[2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO - args.func(args)
[2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO - File
"/home/***/.local/lib/python3.9/site-packages/***/cli/cli_config.py", line 49,
in command
[2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO - return func(*args,
**kwargs)
[2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO - File
"/home/***/.local/lib/python3.9/site-packages/***/utils/cli.py", line 113, in
wrapper
[2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO - return f(*args,
**kwargs)
[2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO - File
"/home/***/.local/lib/python3.9/site-packages/***/utils/providers_configuration_loader.py",
line 56, in wrapped_function
[2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO - return func(*args,
**kwargs)
[2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO - File
"/home/***/.local/lib/python3.9/site-packages/***/cli/commands/db_command.py",
line 241, in cleanup_tables
[2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO - run_cleanup(
[2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO - File
"/home/***/.local/lib/python3.9/site-packages/***/utils/session.py", line 77,
in wrapper
[2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO - return func(*args,
session=session, **kwargs)
[2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO - File
"/home/***/.local/lib/python3.9/site-packages/***/utils/db_cleanup.py", line
437, in run_cleanup
[2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO - _cleanup_table(
[2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO - File
"/home/***/.local/lib/python3.9/site-packages/***/utils/db_cleanup.py", line
302, in _cleanup_table
[2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO -
_do_delete(query=query, orm_model=orm_model, skip_archive=skip_archive,
session=session)
[2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO - File
"/home/***/.local/lib/python3.9/site-packages/***/utils/db_cleanup.py", line
197, in _do_delete
[2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO - target_table.drop()
[2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO - File
"/home/***/.local/lib/python3.9/site-packages/sqlalchemy/sql/schema.py", line
978, in drop
[2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO - bind =
_bind_or_error(self)
[2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO - File
"/home/***/.local/lib/python3.9/site-packages/sqlalchemy/sql/base.py", line
1659, in _bind_or_error
[2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO - raise
exc.UnboundExecutionError(msg)
[2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO -
sqlalchemy.exc.UnboundExecutionError: Table object
'_***_deleted__celery_taskmeta__20230829121812' is not bound to an Engine or
Connection. Execution can not proceed without a database to execute against.
[2023-08-29, 12:18:13 UTC] {subprocess.py:97} INFO - Command exited with
return code 1
[2023-08-29, 12:18:13 UTC] {taskinstance.py:1943} ERROR - Task failed with
exception
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.9/site-packages/airflow/operators/bash.py",
line 210, in execute
raise AirflowException(
airflow.exceptions.AirflowException: Bash command failed. The command
returned a non-zero exit code 1.
### What you think should happen instead
_No response_
### How to reproduce
Using Airflow Instance with Celery Executor, create the DAG:
`from airflow import DAG
from airflow.operators.bash import BashOperator
with DAG(
) as dag:
execution_date_minus_1 = "{{ execution_date -
macros.timedelta(minutes=1) }}"
delete_old_from_metadata_db = BashOperator(
task_id="delete_old_from_metadata_db",
bash_command=f"airflow db clean --clean-before-timestamp
'{execution_date_minus_1}' --skip-archive --yes",
)
delete_old_from_metadata_db
`
The run the DAG - you may need to run it twice as the failure is predicated
on having old data to prune
### Operating System
Docker image
apache/airflow:slim-2.7.0-python3.9@sha256:535f3ef8b0a2bbf90a5401e649076c46d305daf24e6f11a77e67ea41602c77ed
### Versions of Apache Airflow Providers
apache-airflow-providers-celery==3.2.1
apache-airflow-providers-common-sql==1.6.0
apache-airflow-providers-ftp==3.4.2
apache-airflow-providers-http==4.5.0
apache-airflow-providers-imap==3.2.2
apache-airflow-providers-postgres==5.5.2
apache-airflow-providers-redis==3.2.1
apache-airflow-providers-sqlite==3.4.2
### Deployment
Docker-Compose
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]