pingzh commented on PR #30104: URL: https://github.com/apache/airflow/pull/30104#issuecomment-1468846541
> Can you share a bit more details about your use case ? What is the expected benefit, shorter dag parsing times ? In what situation do you have many python files being wrongly picked up by the heuristic ? hi @vandonr-amz right now, the heuristic checks the work `dag` and `airflow` in a python file and decides if there is dag or not. For example, a python file can easily have work `dag` in a comment or a helper methods. This will add workload for the dag file parsing and even can create lots of import_errors (it happens in our case, we had to add lots of file paths to the `.airflowignore`, however, it isn't very scalable given there are several thousands of files in our repo). Having this pluggable callable, it will be it easier to define rules. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
