tirkarthi commented on issue #41641:
URL: https://github.com/apache/airflow/issues/41641#issuecomment-2440023575
Thanks @potiuk for the details. I will start a discussion on this at the
devlist and continue there. Bowler looks interesting. Using `libcst.tool` from
cli parallelizes the process. Right now this needs `python -m libcst.tool` to
execute it as a codemod. Initially I had designed them as standalone
Transformer for each category like (dag, operator) where the updated AST from
one transformer can be passed to another. The codemod looked like a recommended
abstraction for running it and changed it that way to later find cli accepts
only one codemod at a time. I need to check how composable they are.
```
python -m libcst.tool codemod --help | grep -i jobs
usage: libcst.tool codemod [-h] [-x] [-j JOBS] [-p VERSION] [-u [CONTEXT]]
-j JOBS, --jobs JOBS Number of jobs to use when processing files. Defaults
time python -m libcst.tool codemod dag_fixer.DagFixerCommand -u 1
~/airflow/dags > /dev/null 2>&1
python -m libcst.tool codemod dag_fixer.DagFixerCommand -u 1 ~/airflow/dags
>
6.95s user 0.61s system 410% cpu 1.843 total
# Single core
time python -m libcst.tool codemod dag_fixer.DagFixerCommand -u 1 -j 1
~/airflow/dags > /dev/null 2>&1
python -m libcst.tool codemod dag_fixer.DagFixerCommand -u 1 -j 1 >
/dev/nul 4.66s user 0.38s system 99% cpu 5.035 total
# 4 core
python -m libcst.tool codemod dag_fixer.DagFixerCommand -u 1 -j 4
~/airflow/dags > /dev/null 2>&1
python -m libcst.tool codemod dag_fixer.DagFixerCommand -u 1 -j 4 >
/dev/nul 5.45s user 0.54s system 253% cpu 2.358 total
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
