Re: Spark Event Log Forwarding and Offset Tracking

2021-02-04 Thread Raymond Tan
Thanks Jacek. Can you point me to some sample implementations of this that I can use as a reference? On Sun, Jan 17, 2021 at 10:09 PM Jacek Laskowski wrote: > Hi, > > > Forwarding Spark Event Logs to identify critical events like job start, > executor failures, job failures etc to ElasticSearch

Re: Spark Event Log Forwarding and Offset Tracking

2021-01-17 Thread Jacek Laskowski
Hi, > Forwarding Spark Event Logs to identify critical events like job start, executor failures, job failures etc to ElasticSearch via log4j. However I could not find any way to foward event log via log4j configurations. Is there any other recommended approach to track these application events? I

Spark Event Log Forwarding and Offset Tracking

2021-01-13 Thread raymond.tan
Hello here, I am new to spark and am trying to add some monitoring for spark applications specifically to handle the below situations - 1 - Forwarding Spark Event Logs to identify critical events like job start, executor failures, job failures etc to ElasticSearch via log4j. However I could not fin