Re: 'ExecutorTaskSummary' alternative in Spark 2.3 onwards

[Ninja Coder]

Any advise/help here is much appreciated.

On Mon, Dec 30, 2019 at 4:16 PM Ninja Coder  wrote:

> I have a spark streaming application (currently spark 2.2) which is using
> `org.apache.spark.ui.exec.ExecutorTaskSummary` to grab executor details
> like
> duration, tasks failed, tasks completed, GC time, etc after each batch is
> completed. These metrics are then loaded to prometheus and pagerduty alerts
> are set on it.
>
> We are planning to upgrade spark and looks like`ExecutorTaskSummary` is no
> longer available from spark 2.3
>
> I would like to know what are the other alternatives which I can use.
>
>
>
> --
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>
> -
> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>
>

'ExecutorTaskSummary' alternative in Spark 2.3 onwards

[Ninja Coder]

I have a spark streaming application (currently spark 2.2) which is using
`org.apache.spark.ui.exec.ExecutorTaskSummary` to grab executor details like
duration, tasks failed, tasks completed, GC time, etc after each batch is
completed. These metrics are then loaded to prometheus and pagerduty alerts
are set on it. 

We are planning to upgrade spark and looks like`ExecutorTaskSummary` is no
longer available from spark 2.3

I would like to know what are the other alternatives which I can use. 



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Using Aggregate and group by on spark Dataset api

[coder]

JavaRDD prdd = sc.textFile("c:\\fls\\people.txt").map(
  new Function() {
public Person call(String line) throws Exception {
  String[] parts = line.split(",");
  Person person = new Person();
  person.setName(parts[0]);
  person.setAge(Integer.parseInt(parts[1].trim()));
 person.setSal(Integer.parseInt(parts[2].trim()));
  return person;
}
  });

   RDDpersonRDD =  prdd.toRDD(prdd);
   Dataset dss= sqlContext.createDataset(personRDD ,  
Encoders.bean(Person.class));
   GroupedDataset dq=dss.groupBy(new Column("name"));

I have to calculate sum of age and salary group by name on the dataset.
Please help how to query dataset ? I tried using GroupedDataset but don't
know how to proceed with it. 

I acn not find much help for using dataset api.
Please help




--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Using-Aggregate-and-group-by-on-spark-Dataset-api-tp26824.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org