a-meledin opened a new issue, #35623: URL: https://github.com/apache/airflow/issues/35623
### Description Problem: - When a task logic is written in another language (C/C#/etc) and Airflow is used just for: (1) traversing a Dag; (2) executing remote logic over SSH/REST API; (3) taking logs or their summary back through SFTP/other methods ... In such a setting Local/CeleryExcutor multiprocessing logic gives too much overhead. - E.g. we have a production DAG with 1500 task instances to run in parallel. It requires setting approximately 50 celeryd's with 32 threads each. Resource/CPU consumption as compared to real work done by Airflow in such a case cannot be justified. Possible solution: - Async multithreaded executor seems to be an option. - As each task is executed the same way: -- @task code is empty or contains only configuration -- configuration is handled by user defined class that is passed to executor. NB: Triggerer is not considered as an option, because the process for ti is spawn several times. This multiplies overhead. ### Use case/motivation _No response_ ### Related issues _No response_ ### Are you willing to submit a PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
