GitHub user ilgiznurgaliev created a discussion: S3 Dag Bundle won't remove
files removed on remote S3
Hello.
I'm using S3 Dag Bundle and it works perfectly except one thing. Files removed
remotely (in S3 bucket) won't be removed in Airflow. My source S3 bucket
contains some config files additionally to dag files. I use these config files
to generate dags dynamically like that:
```
configs_dir = Path(__file__).resolve().parent / 'configs'
for file in glob(f'{configs_dir}/*.yaml'):
with open(file, 'r') as f:
config = yaml.load(f)
# Generate pipeline
@dag(
dag_id=config['dag_id'],
schedule=config['cron'],
tags=config['tags'],
)
def yac_instance_autoshutdown_pipeline():
...
```
So number of dags have to be the same as number of yaml files in `configs`
folder.
When one of the yaml files removed from `configs` folder it's expected that it
will be removed locally too and appropriate dag will be deactivated.
Actual behavior: dags will appear when new config files created, but won't be
deactivated when config files removed.
Is there any way to clean up files that were removed on remote S3 bucket? Any
workaround would help. Would be best to allow something like `--delete` flag
for S3 Bundle configuration, but currently it is not supported.
GitHub link: https://github.com/apache/airflow/discussions/64775
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
