Hi, In my Spark application, I need to log data row by row directly from the executors to avoid overwhelming the driver's memory, which is already going places.
I am exploring the possibility of implementing a distributed logging strategy where each executor logs its output directly, rather than collecting all data to the driver first. Can you recommend best practices or tools for implementing distributed logging from executors in Spark applications? Thank you very much for your time and consideration. I look forward to your response. Best, ZS
