ArvidMartensRenson commented on issue #50185: URL: https://github.com/apache/airflow/issues/50185#issuecomment-2889837087
For now I solved througt the creation of an async Task SDK client which the triggers use directly. In my setup the triggers always have the api server running on the same instances, hence I could use localhost traffic. In this way it is rather similar like the in-process traffic but I do not overload the data pipe between trigger runner and supervisor. From my investigation it seems that using too much external workload triggers (count calls) together with the supervisor-runner syncs congests that pipe in such a way that all calls were blocked indefinitely resulting in a trigger process that seems alive (heartbeats still worked) but in reality it is stuck. Probably this is not the designed pattern, but for now this solution makes it possible for me to role out airflow 3.0 at our office. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
