Re: dataframe sorting and find the index of the maximum element

2016-04-06 Thread Angel Angel
hi, Thanks for help. Actually i didn't declare the function. Can you give me the best way so solve this problem. Here i am explaining my problem: I have 3 tables and i have to find the maximum element in each table. Table1 Table2 Table3 21 31 41 55 1 12 32 19 96 2 14 28 Then

Re: dataframe sorting and find the index of the maximum element

2016-04-05 Thread Michael Armbrust
You should generally think of a DataFrame as unordered, unless you are explicitly asking for an order. One way to order and assign an index is with window functions . On Tue, Apr 5, 2016 at 4:17 AM, Angel

Re: dataframe sorting and find the index of the maximum element

2016-04-05 Thread Ted Yu
The error was due to REPL expecting an integer (index to the Array) whereas "MAX(count)" was a String. What do you want to achieve ? On Tue, Apr 5, 2016 at 4:17 AM, Angel Angel wrote: > Hello, > > i am writing one spark application i which i need the index of the

Re: dataframe sorting and find the index of the maximum element

2016-04-05 Thread Ted Yu
Did you define idxmax() method yourself ? Thanks On Tue, Apr 5, 2016 at 4:17 AM, Angel Angel wrote: > Hello, > > i am writing one spark application i which i need the index of the maximum > element. > > My table has one column only and i want the index of the maximum

dataframe sorting and find the index of the maximum element

2016-04-05 Thread Angel Angel
Hello, i am writing one spark application i which i need the index of the maximum element. My table has one column only and i want the index of the maximum element. MAX(count) 23 32 3 Here is my code the data type of the array is org.apache.spark.sql.Dataframe. Thanks in advance. Also please