It sounds like mapPartitionsWithIndex will give you the information you want over flatMap.
On Thursday, October 13, 2016, Shushant Arora <shushantaror...@gmail.com> wrote: > Hi > > I have a transformation on a pair rdd using flatmap function. > > 1.Can I detect in flatmap whether the current record is last record of > partition being processed and > 2. what is the partition index of this partition. > > public Iterable<Tuple2<String, String>> call(Tuple2<String, String> t) > throws Exception { > //whether element is last elemet of current partiton or not ?? > //partition index to which element belongs > to > } > }); > Thanks > > -- Cell : 425-233-8271 Twitter: https://twitter.com/holdenkarau