or you could
1) convert dataframe to RDD
2) use mapPartitions and zipWithIndex within each partition
3) convert RDD back to dataframe you will need to make sure you preserve
partitioning
Deenar
On 1 June 2015 at 02:23, ayan guha guha.a...@gmail.com wrote:
If you are on spark 1.3, use
If you are on spark 1.3, use repartitionandSort followed by mappartition.
In 1.4, window functions will be supported, it seems
On 1 Jun 2015 04:10, Ricardo Almeida ricardo.alme...@actnowib.com wrote:
That's great and how would you create an ordered index by partition (by
product in this
One way I can see is to -
1. get rdd from your df
2. call rdd.zipWithIndex to get a new rdd
3. turn your new rdd to a new df
On Fri, May 29, 2015 at 5:43 AM, Cesar Flores ces...@gmail.com wrote:
Assuming that I have the next data frame:
flag | price
--
1
Assuming that I have the next data frame:
flag | price
--
1|47.808764653746
1|47.808764653746
1|31.9869279512204
1|47.7907893713564
1|16.7599200038239
1|16.7599200038239
1|20.3916014172137
How can I create a data frame with an extra indexed column
