Re: [DISCUSS] SPIP: Structured Spark Logging

Thanks everyone for the valuable feedback! Given the generally positive feedback received, I plan to move forward by initiating the voting thread. I encourage you to participate in the upcoming thread. Warm regards, Gengliang On Sat, Mar 9, 2024 at 12:55 PM Mich Talebzadeh wrote: > Splendid.

Re: [DISCUSS] SPIP: Structured Spark Logging

Splendid. Thanks Gengliang Mich Talebzadeh, Dad | Technologist | Solutions Architect | Engineer London United Kingdom view my Linkedin profile https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* The information provided

Re: [DISCUSS] SPIP: Structured Spark Logging

Hi Mich, Thanks for your suggestions. I agree that we should avoid confusion with Spark Structured Streaming. So, I'll go with "Structured Logging Framework for Apache Spark". This keeps the standard term "Structured Logging" and distinguishes it from "Structured Streaming" clearly. Thanks for

Re: [DISCUSS] SPIP: Structured Spark Logging

Hi Gengliang, Thanks for taking the initiative to improve the Spark logging system. Transitioning to structured logs seems like a worthy way to enhance the ability to analyze and troubleshoot Spark jobs and hopefully the future integration with cloud logging systems. While "Structured Spark

Re: [DISCUSS] SPIP: Structured Spark Logging

Hi Gengling, Thanks for sharing this ! I added a few queries to the proposal doc, and we can continue discussing there, but overall I am in favor of this. Regards, Mridul On Fri, Mar 1, 2024 at 1:35 AM Gengliang Wang wrote: > Hi All, > > I propose to enhance our logging system by

[DISCUSS] SPIP: Structured Spark Logging

Hi All, I propose to enhance our logging system by transitioning to structured logs. This initiative is designed to tackle the challenges of analyzing distributed logs from drivers, workers, and executors by allowing them to be queried using a fixed schema. The goal is to improve the