Re: detecting last record of partition

2016-10-13 Thread Holden Karau
It sounds like mapPartitionsWithIndex will give you the information you want over flatMap. On Thursday, October 13, 2016, Shushant Arora wrote: > Hi > > I have a transformation on a pair rdd using flatmap function. > > 1.Can I detect in flatmap whether the current

detecting last record of partition

2016-10-13 Thread Shushant Arora
Hi I have a transformation on a pair rdd using flatmap function. 1.Can I detect in flatmap whether the current record is last record of partition being processed and 2. what is the partition index of this partition. public Iterable> call(Tuple2 t) throws