Re: Problem with WINDOW functions?

2015-12-30 Thread Vadim Tkachenko
Davies, Thank you, I will wait on 1.6 release. http://apache-spark-user-list.1001560.n3.nabble.com/Problem-with-WINDOW-functions-tt25833.html ? On Wed, Dec 30, 2015 at 12:06 AM, Davies Liu wrote: > Window functions are improved in 1.6 release, could you try 1.6-RC4 >

Re: Problem with WINDOW functions?

2015-12-30 Thread Vadim Tkachenko
Gokula, Thanks, I will try this. I am just SQL kind of guy :), but I will try your suggestion Thanks, Vadim On Wed, Dec 30, 2015 at 1:07 PM, Gokula Krishnan D wrote: > Hello Vadim - > > Alternatively, you can achieve by using the *window functions* which is > available

Re: Problem with WINDOW functions?

2015-12-30 Thread Gokula Krishnan D
Hello Vadim - Alternatively, you can achieve by using the *window functions* which is available from 1.4.0 *code_value.txt (Input)* = 1000,200,Descr-200,01 1000,200,Descr-200-new,02 1000,201,Descr-201,01 1000,202,Descr-202-new,03 1000,202,Descr-202,01

Re: Problem with WINDOW functions?

2015-12-30 Thread Davies Liu
Window functions are improved in 1.6 release, could you try 1.6-RC4 (or wait until next week for the final release)? Even In 1.6, the buffer of rows for window function does not support spilling (also does not use memory efficiently), there is a JIRA for it:

Re: Problem with WINDOW functions?

2015-12-29 Thread Chris Fregly
on quick glance, it appears that you're calling collect() in there which is bringing down a huge amount of data down to the single Driver. this is why, when you allocated more memory to the Driver, a different error emerges most -definitely related to stop-the-world GC to cause the node to

RE: Problem with WINDOW functions?

2015-12-29 Thread Cheng, Hao
Which version are you using? Have you tried the 1.6? From: Vadim Tkachenko [mailto:apache...@gmail.com] Sent: Wednesday, December 30, 2015 10:17 AM To: Cheng, Hao Cc: user@spark.apache.org Subject: Re: Problem with WINDOW functions? When I allocate 200g to executor, it is able to make better

RE: Problem with WINDOW functions?

2015-12-29 Thread Cheng, Hao
Can you try to write the result into another file instead? Let's see if there any issue in the executors side . sqlContext.sql("SELECT day,page,dense_rank() OVER (PARTITION BY day ORDER BY pageviews DESC) as rank FROM d1").filter("rank <= 20").sort($"day",$"rank").write.parquet("/path/to/file")

Re: Problem with WINDOW functions?

2015-12-29 Thread Vadim Tkachenko
a bigger memory for executors? > > -Original Message- > From: va...@percona.com [mailto:va...@percona.com] On Behalf Of Vadim > Tkachenko > Sent: Wednesday, December 30, 2015 9:51 AM > To: Cheng, Hao > Cc: user@spark.apache.org > Subject: Re: Problem with WINDOW fu

RE: Problem with WINDOW functions?

2015-12-29 Thread Cheng, Hao
Is there any improvement if you set a bigger memory for executors? -Original Message- From: va...@percona.com [mailto:va...@percona.com] On Behalf Of Vadim Tkachenko Sent: Wednesday, December 30, 2015 9:51 AM To: Cheng, Hao Cc: user@spark.apache.org Subject: Re: Problem with WINDOW

RE: Problem with WINDOW functions?

2015-12-29 Thread Cheng, Hao
s etc. will be more helpful in understanding your problem. From: Vadim Tkachenko [mailto:apache...@gmail.com] Sent: Wednesday, December 30, 2015 10:49 AM To: Cheng, Hao Subject: Re: Problem with WINDOW functions? I use 1.5.2. Where can I get 1.6? I do not see it on http://spark.apache.org/downloads.html T