Hi Gengliang,
Thanks for taking the initiative to improve the Spark logging system.
Transitioning to structured logs seems like a worthy way to enhance the
ability to analyze and troubleshoot Spark jobs and hopefully the future
integration with cloud logging systems. While "Structured Spark
Hi Gengling,
Thanks for sharing this !
I added a few queries to the proposal doc, and we can continue discussing
there, but overall I am in favor of this.
Regards,
Mridul
On Fri, Mar 1, 2024 at 1:35 AM Gengliang Wang wrote:
> Hi All,
>
> I propose to enhance our logging system by
Hi,
It seems to me that there are issues related to below
* I think when a task failed in between and retry task started and
completed it may create duplicate as failed task has some data + retry task
has full data. but my question is why spark keeps delta data or
according to you if
