tirkarthi commented on issue #41641:
URL: https://github.com/apache/airflow/issues/41641#issuecomment-2439879558
I have started prototyping a small package based on LibCST to build a Python
2to3 like tool for Airflow 2to3 that does simple and straight forward
replacements. My main motivation was around lot of our users in our Airflow
instance using `schedule_interval` in Airflow 2 that was deprecated and renamed
to `schedule` in Airflow 3. It would require updating thousands of dags
manually and some automation could help. This could also help in places with
import statements changes .E.g. Task SDK need to be updated from `from airflow
import DAG` to `from airflow.sdk import DAG`. Something like this could
eventually become part of Airflow cli so that users can run `airflow migrate
/airflow/dags` for migration or serve as a starter point for migration. It can
update the file in place or show diff. Currently it does the following changes
:
Dags
* schedule_interval -> schedule
* timetable -> schedule
* concurrency -> max_active_tasks
* Removal of unused full_filepath parameter
* default_view (tree -> grid)
Operators
* task_concurrency -> max_active_tis_per_dag
* trigger_rule (none_failed_or_skipped -> none_failed_min_one_success)
Sample file
```python
import datetime
from airflow import DAG
from airflow.decorators import dag, task
from airflow.operators.empty import EmptyOperator
from airflow.timetables.events import EventsTimetable
with DAG(
dag_id="my_dag_name",
default_view="tree",
start_date=datetime.datetime(2021, 1, 1),
schedule_interval="@daily",
concurrency=2,
):
op = EmptyOperator(
task_id="task", task_concurrency=1,
trigger_rule="none_failed_or_skipped"
)
@dag(
default_view="graph",
start_date=datetime.datetime(2021, 1, 1),
schedule_interval=EventsTimetable(event_dates=[datetime.datetime(2022,
4, 5)]),
max_active_tasks=2,
full_filepath="/tmp/test_dag.py"
)
def my_decorated_dag():
op = EmptyOperator(task_id="task")
my_decorated_dag()
```
Sample usage
```shell
python -m libcst.tool codemod dag_fixer.DagFixerCommand -u 1
tests/test_dag.py
Calculating full-repo metadata...
Executing codemod...
reformatted -
All done! ✨ 🍰 ✨
1 file reformatted.
--- /home/karthikeyan/stuff/python/libcst-tut/tests/test_dag.py
+++ /home/karthikeyan/stuff/python/libcst-tut/tests/test_dag.py
@@ -10,6 +10,6 @@
dag_id="my_dag_name",
- default_view="tree",
+ default_view="grid",
start_date=datetime.datetime(2021, 1, 1),
- schedule_interval="@daily",
- concurrency=2,
+ schedule="@daily",
+ max_active_tasks=2,
):
@@ -23,5 +23,4 @@
start_date=datetime.datetime(2021, 1, 1),
- schedule_interval=EventsTimetable(event_dates=[datetime.datetime(2022,
4, 5)]),
+ schedule=EventsTimetable(event_dates=[datetime.datetime(2022, 4, 5)]),
max_active_tasks=2,
- full_filepath="/tmp/test_dag.py"
)
Finished codemodding 1 files!
- Transformed 1 files successfully.
- Skipped 0 files.
- Failed to codemod 0 files.
- 0 warnings were generated.
```
Repo : https://github.com/tirkarthi/Airflow-2to3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
