Gokula,
Thanks,
I will try this.
I am just SQL kind of guy :), but I will try your suggestion
Thanks,
Vadim
On Wed, Dec 30, 2015 at 1:07 PM, Gokula Krishnan D
wrote:
> Hello Vadim -
>
> Alternatively, you can achieve by using the *window functions* which is
> available from 1.4.0
>
> *code_va
Hello Vadim -
Alternatively, you can achieve by using the *window functions* which is
available from 1.4.0
*code_value.txt (Input)*
=
1000,200,Descr-200,01
1000,200,Descr-200-new,02
1000,201,Descr-201,01
1000,202,Descr-202-new,03
1000,202,Descr-202,01
1000,202,Descr-202-old,02
Davies,
Thank you, I will wait on 1.6 release.
http://apache-spark-user-list.1001560.n3.nabble.com/Problem-with-WINDOW-functions-tt25833.html
?
On Wed, Dec 30, 2015 at 12:06 AM, Davies Liu wrote:
> Window functions are improved in 1.6 release, could you try 1.6-RC4
> (or wait until next week f
Window functions are improved in 1.6 release, could you try 1.6-RC4
(or wait until next week for the final release)?
Even In 1.6, the buffer of rows for window function does not support
spilling (also does not use memory efficiently), there is a JIRA for
it: https://issues.apache.org/jira/browse/S
ll be more helpful in understanding your problem.
From: Vadim Tkachenko [mailto:apache...@gmail.com]
Sent: Wednesday, December 30, 2015 10:49 AM
To: Cheng, Hao
Subject: Re: Problem with WINDOW functions?
I use 1.5.2.
Where can I get 1.6? I do not see it on http://spark.apache.org/downloads.html
T
Which version are you using? Have you tried the 1.6?
From: Vadim Tkachenko [mailto:apache...@gmail.com]
Sent: Wednesday, December 30, 2015 10:17 AM
To: Cheng, Hao
Cc: user@spark.apache.org
Subject: Re: Problem with WINDOW functions?
When I allocate 200g to executor, it is able to make better
ors?
>
> -Original Message-
> From: va...@percona.com [mailto:va...@percona.com] On Behalf Of Vadim
> Tkachenko
> Sent: Wednesday, December 30, 2015 9:51 AM
> To: Cheng, Hao
> Cc: user@spark.apache.org
> Subject: Re: Problem with WINDOW functions?
>
> Hi,
>
Is there any improvement if you set a bigger memory for executors?
-Original Message-
From: va...@percona.com [mailto:va...@percona.com] On Behalf Of Vadim Tkachenko
Sent: Wednesday, December 30, 2015 9:51 AM
To: Cheng, Hao
Cc: user@spark.apache.org
Subject: Re: Problem with WINDOW
on quick glance, it appears that you're calling collect() in there which is
bringing down a huge amount of data down to the single Driver. this is why,
when you allocated more memory to the Driver, a different error emerges most
-definitely related to stop-the-world GC to cause the node to beco
Can you try to write the result into another file instead? Let's see if there
any issue in the executors side .
sqlContext.sql("SELECT day,page,dense_rank() OVER (PARTITION BY day ORDER BY
pageviews DESC) as rank FROM d1").filter("rank <=
20").sort($"day",$"rank").write.parquet("/path/to/file")
10 matches
Mail list logo