Re: Secondary Sort using Apache Spark 1.6

2017-03-29 Thread Pariksheet Barapatre
D[(DeviceKey, Int)] = ShuffledRDD[1] at > repartitionAndSortWithinPartitions at :30 > > > Yong > > > -- > *From:* Pariksheet Barapatre <pbarapa...@gmail.com> > *Sent:* Wednesday, March 29, 2017 9:02 AM > *To:* user > *Subject:* Second

Re: Secondary Sort using Apache Spark 1.6

2017-03-29 Thread Yong Zhang
la> t.repartitionAndSortWithinPartitions(new DeviceKeyPartitioner(2)) res0: org.apache.spark.rdd.RDD[(DeviceKey, Int)] = ShuffledRDD[1] at repartitionAndSortWithinPartitions at :30 Yong From: Pariksheet Barapatre <pbarapa...@gmail.com> Sent: Wednesday, March

Secondary Sort using Apache Spark 1.6

2017-03-29 Thread Pariksheet Barapatre
Hi, I am referring web link http://codingjunkie.net/spark-secondary-sort/ to implement secondary sort in my spark job. I have defined my key case class as case class DeviceKey(serialNum: String, eventDate: