Re: 'ExecutorTaskSummary' alternative in Spark 2.3 onwards

Any advise/help here is much appreciated. On Mon, Dec 30, 2019 at 4:16 PM Ninja Coder wrote: > I have a spark streaming application (currently spark 2.2) which is using > `org.apache.spark.ui.exec.ExecutorTaskSummary` to grab executor details > like > duration, tasks failed, tasks c

'ExecutorTaskSummary' alternative in Spark 2.3 onwards

I have a spark streaming application (currently spark 2.2) which is using `org.apache.spark.ui.exec.ExecutorTaskSummary` to grab executor details like duration, tasks failed, tasks completed, GC time, etc after each batch is completed. These metrics are then loaded to prometheus and pagerduty

Using Aggregate and group by on spark Dataset api

JavaRDD prdd = sc.textFile("c:\\fls\\people.txt").map( new Function() { public Person call(String line) throws Exception { String[] parts = line.split(","); Person person = new Person();