VladaZakharova commented on issue #39613: URL: https://github.com/apache/airflow/issues/39613#issuecomment-3253239824
After my investigation on this issue I see two big problems with StackdriverTaskHandler. In addition I need to say that StackdriverTaskHandler has never worked as expected. From its creating this StackdriverTaskHandler has these problems and the community several times started discussing these problems. In my opinion the best solution here is refactoring for this StackdriverTaskHandler. Problems: - StackdriverTaskHandler sends only partially logs to Google Stackdriver service. This problem happens because BackgroundThreadTransport, which is used as a default transport to send logs to Stackdriver, doesn't work as expected and sends only part of the logs. This issue is inside BackgroundThreadTransport and isn't related to Airflow, more information about it here: https://github.com/googleapis/python-logging/issues/550 Like a temporary solution we can change default transport to SyncTransport and this problem will disappear. But, I didn't find what was the reason to use BackgroundThreadTransport as a default. - StackdriverTaskHandler doesn't read logs in real time from Google Stackdriver service. For this problem I don't see any solution now. Documentation for StackdriverTaskHandler mentions that this handler prints logs to Airflow UI in real time, but when I run DAG this behavior doesn't happen. At the current moment StackdriverTaskHandler waits for the task ending and then pulls logs from Stackdriver. We can update the docs and describe to users real behavior for this handler or we can start a refactoring for implement behavior which was described in the docs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
