nickwood opened a new issue, #33884:
URL: https://github.com/apache/airflow/issues/33884

   ### Apache Airflow version
   
   2.7.0
   
   ### What happened
   
   I have a DAG with the following operator defined:
   `
       delete_old_from_metadata_db = BashOperator(
           task_id="delete_old_from_metadata_db",
           bash_command=f"airflow db clean --clean-before-timestamp '2023-08-29 
12:00:00' --skip-archive --yes"
       )
   `
   On 2.6.3 this correctly deleted everything from the airflow metadata 
database prior to the specified date. I'm using CeleryExecutor
   
   On 2.7.0 this fails however, with the following logs:
   [2023-08-29, 12:18:11 UTC] {subprocess.py:63} INFO - Tmp dir root location: 
/tmp
   [2023-08-29, 12:18:11 UTC] {subprocess.py:75} INFO - Running command: 
['/bin/bash', '-c', "*** db clean --clean-before-timestamp 
'2023-08-29T12:17:09.799435+00:00' --skip-archive --yes"]
   [2023-08-29, 12:18:11 UTC] {subprocess.py:86} INFO - Output:
   [2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO - 
/home/***/.local/lib/python3.9/site-packages/***/utils/db.py:1037 SAWarning: 
Skipped unsupported reflection of expression-based index idx_ab_user_username
   [2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO - 
/home/***/.local/lib/python3.9/site-packages/***/utils/db.py:1037 SAWarning: 
Skipped unsupported reflection of expression-based index 
idx_ab_register_user_username
   [2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO - 
/home/***/.local/lib/python3.9/site-packages/***/utils/db_cleanup.py:174 
SAWarning: Class CreateTableAs will not make use of SQL compilation caching as 
it does not set the 'inherit_cache' attribute to ``True``.  This can have 
significant performance implications including some performance degradations in 
comparison to prior SQLAlchemy versions.  Set this attribute to True if this 
object can make use of the cache key generated by the superclass.  
Alternatively, this attribute may be set to False which will disable this 
warning. (Background on this error at: https://sqlalche.me/e/14/cprf)
   [2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO - 
   [2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO - Checking table 
callback_request
   [2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO - Found 0 rows meeting 
deletion criteria.
   [2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO - 
   [2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO - Checking table 
celery_taskmeta
   [2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO - Found 22 rows meeting 
deletion criteria.
   [2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO - Performing Delete...
   [2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO - Moving data to table 
_***_deleted__celery_taskmeta__20230829121812
   [2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO - Traceback (most recent 
call last):
   [2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO -   File 
"/home/***/.local/bin/***", line 8, in <module>
   [2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO -     sys.exit(main())
   [2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO -   File 
"/home/***/.local/lib/python3.9/site-packages/***/__main__.py", line 60, in main
   [2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO -     args.func(args)
   [2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO -   File 
"/home/***/.local/lib/python3.9/site-packages/***/cli/cli_config.py", line 49, 
in command
   [2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO -     return func(*args, 
**kwargs)
   [2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO -   File 
"/home/***/.local/lib/python3.9/site-packages/***/utils/cli.py", line 113, in 
wrapper
   [2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO -     return f(*args, 
**kwargs)
   [2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO -   File 
"/home/***/.local/lib/python3.9/site-packages/***/utils/providers_configuration_loader.py",
 line 56, in wrapped_function
   [2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO -     return func(*args, 
**kwargs)
   [2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO -   File 
"/home/***/.local/lib/python3.9/site-packages/***/cli/commands/db_command.py", 
line 241, in cleanup_tables
   [2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO -     run_cleanup(
   [2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO -   File 
"/home/***/.local/lib/python3.9/site-packages/***/utils/session.py", line 77, 
in wrapper
   [2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO -     return func(*args, 
session=session, **kwargs)
   [2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO -   File 
"/home/***/.local/lib/python3.9/site-packages/***/utils/db_cleanup.py", line 
437, in run_cleanup
   [2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO -     _cleanup_table(
   [2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO -   File 
"/home/***/.local/lib/python3.9/site-packages/***/utils/db_cleanup.py", line 
302, in _cleanup_table
   [2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO -     
_do_delete(query=query, orm_model=orm_model, skip_archive=skip_archive, 
session=session)
   [2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO -   File 
"/home/***/.local/lib/python3.9/site-packages/***/utils/db_cleanup.py", line 
197, in _do_delete
   [2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO -     target_table.drop()
   [2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO -   File 
"/home/***/.local/lib/python3.9/site-packages/sqlalchemy/sql/schema.py", line 
978, in drop
   [2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO -     bind = 
_bind_or_error(self)
   [2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO -   File 
"/home/***/.local/lib/python3.9/site-packages/sqlalchemy/sql/base.py", line 
1659, in _bind_or_error
   [2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO -     raise 
exc.UnboundExecutionError(msg)
   [2023-08-29, 12:18:12 UTC] {subprocess.py:93} INFO - 
sqlalchemy.exc.UnboundExecutionError: Table object 
'_***_deleted__celery_taskmeta__20230829121812' is not bound to an Engine or 
Connection.  Execution can not proceed without a database to execute against.
   [2023-08-29, 12:18:13 UTC] {subprocess.py:97} INFO - Command exited with 
return code 1
   [2023-08-29, 12:18:13 UTC] {taskinstance.py:1943} ERROR - Task failed with 
exception
   Traceback (most recent call last):
     File 
"/home/airflow/.local/lib/python3.9/site-packages/airflow/operators/bash.py", 
line 210, in execute
       raise AirflowException(
   airflow.exceptions.AirflowException: Bash command failed. The command 
returned a non-zero exit code 1.
   
   ### What you think should happen instead
   
   _No response_
   
   ### How to reproduce
   
   Using Airflow Instance with Celery Executor, create the DAG:
   
   `from airflow import DAG
   from airflow.operators.bash import BashOperator
   
   with DAG(
   
   ) as dag:
       execution_date_minus_1 = "{{ execution_date - 
macros.timedelta(minutes=1) }}"
       delete_old_from_metadata_db = BashOperator(
           task_id="delete_old_from_metadata_db",
           bash_command=f"airflow db clean --clean-before-timestamp 
'{execution_date_minus_1}' --skip-archive --yes",
       )
       delete_old_from_metadata_db
   `
   The run the DAG - you may need to run it twice as the failure is predicated 
on having old data to prune
   
   
   ### Operating System
   
   Docker image 
apache/airflow:slim-2.7.0-python3.9@sha256:535f3ef8b0a2bbf90a5401e649076c46d305daf24e6f11a77e67ea41602c77ed
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow-providers-celery==3.2.1
   apache-airflow-providers-common-sql==1.6.0
   apache-airflow-providers-ftp==3.4.2
   apache-airflow-providers-http==4.5.0
   apache-airflow-providers-imap==3.2.2
   apache-airflow-providers-postgres==5.5.2
   apache-airflow-providers-redis==3.2.1
   apache-airflow-providers-sqlite==3.4.2
   
   
   ### Deployment
   
   Docker-Compose
   
   ### Deployment details
   
   _No response_
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to