luoyuliuyin commented on PR #39484: URL: https://github.com/apache/airflow/pull/39484#issuecomment-2101380558
> I am a little skeptical that the overhead for process pool creation is meaningful. Because i think the dominant performance consideration is the blocking IO sending the task to redis and waiting for response. > > In order for the process pool creation to be dominant, you'd have to be sending very few tasks i'd think, in which case it would be fast anyway. No? > > E.g. in your example above, you claim that it sometimes took 4 seconds to send 32 tasks. But, you did not establish that the bottleneck was process pool creation. How do you know it wasn't mostly just waiting for the broker to respond? Are you claiming that it takes 4 seconds to create the process pool? Maybe you would be able to get at this by adding a log message immediately after entering the context? I'm testing it through the logs. It takes a while. It usually takes about 2 days to see that the scheduling process slows down significantly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
