potiuk commented on issue #45061: URL: https://github.com/apache/airflow/issues/45061#issuecomment-2556774687
I think it would be a great idea to review and make it "common" and reused accross the components. Also we have another, related discussion about logging in https://github.com/apache/airflow/issues/45079 where a solutions is proposed to handle both - streaming logs and serving logs after task is completed - with regards to OOM issues. Currently we merge and store logs in memory which - in some situations might in some situations lead to OOMs. And some refactoring and restructuring of the logging mechanism might be necessary to handle things better. I think both of you @mrk-andreev and @jason810496 are very well skilled and prepared to handle that - under @dstandish @eladkal (and a little of mine) guidande and mentoring, so if you both work on this together to get both sides of logging improved. I know it's not directly the same issue, but having a small team of skilled people to focus on both and collaborate, might be a great idea. @dstandish - WDYT? Having the three of maintainers here to overlook it while Mark and LIU ZHE YOU work on those, might be a good idea to distract from other Airlfow 3 streams? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
