a-meledin opened a new issue, #35623:
URL: https://github.com/apache/airflow/issues/35623

   ### Description
   
   Problem: 
   - When a task logic is written in another language (C/C#/etc) and Airflow is 
used just for: (1) traversing a Dag; (2) executing remote logic over SSH/REST 
API; (3) taking logs or their summary back through SFTP/other methods ... In 
such a setting Local/CeleryExcutor multiprocessing logic gives too much 
overhead. 
   - E.g. we have a production DAG with 1500 task instances to run in parallel. 
It requires setting approximately 50 celeryd's with 32 threads each. 
Resource/CPU consumption as compared to real work done by Airflow in such a 
case cannot be justified. 
   
   Possible solution: 
   - Async multithreaded executor seems to be an option. 
   - As each task is executed the same way:
   -- @task code is empty or contains only configuration
   -- configuration is handled by user defined class that is passed to 
executor.  
   
   NB: Triggerer is not considered as an option, because the process for ti is 
spawn several times. This multiplies overhead.
   
   ### Use case/motivation
   
   _No response_
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to