dstandish commented on PR #36227: URL: https://github.com/apache/airflow/pull/36227#issuecomment-1856504013
Hi @troyharvey , my secret master plan is to do a larger refactor of KPO logging. In short here's the plan: * run a log fetcher in a thread per container * each log fetcher passes the log lines to a queue * main thread reads from queue and logs the messages * separate thread checks pod status periodically to see if it's done. if done, stop everything it may sound complicated but i think it's ultimately simpler and more reliable than what we have now. and it should resolve your problem. however, in order to do this refactor, i want to try to not regress the various fixes people have contributed (such as #34412), so i need to understand what the heck it is doing or at least rely on the tests. but the tests pass even with the code removed. which is why i published this PR, so that the author of #34412 can help clarify the situation, and we can make this better. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
