Re: 'ExecutorTaskSummary' alternative in Spark 2.3 onwards

2020-01-02 Thread Ninja Coder
Any advise/help here is much appreciated. On Mon, Dec 30, 2019 at 4:16 PM Ninja Coder wrote: > I have a spark streaming application (currently spark 2.2) which is using > `org.apache.spark.ui.exec.ExecutorTaskSummary` to grab executor details > like > duration, tasks failed, tasks c

'ExecutorTaskSummary' alternative in Spark 2.3 onwards

2019-12-30 Thread Ninja Coder
I have a spark streaming application (currently spark 2.2) which is using `org.apache.spark.ui.exec.ExecutorTaskSummary` to grab executor details like duration, tasks failed, tasks completed, GC time, etc after each batch is completed. These metrics are then loaded to prometheus and pagerduty alert