Re: [IMPORTANT] Committers please update merge script

2015-05-23 Thread Patrick Wendell
Thanks Ted - there is no need for people to upgrade at this point, since the changes in the release scripts just modify it not to rely on default behavior. On Sat, May 23, 2015 at 7:06 AM, Ted Yu yuzhih...@gmail.com wrote: INFRA-9646 has been resolved. FYI On Wed, May 13, 2015 at 6:00 PM,

Kryo option changed

2015-05-23 Thread Debasish Das
Hi, I am on last week's master but all the examples that set up the following .set(spark.kryoserializer.buffer, 8m) are failing with the following error: Exception in thread main java.lang.IllegalArgumentException: spark.kryoserializer.buffer must be less than 2048 mb, got: + 8192 mb. looks

Re: Kryo option changed

2015-05-23 Thread Ted Yu
bq. it shuld be 8mb Please use the above syntax. Cheers On Sat, May 23, 2015 at 6:04 PM, Debasish Das debasish.da...@gmail.com wrote: Hi, I am on last week's master but all the examples that set up the following .set(spark.kryoserializer.buffer, 8m) are failing with the following error:

spark packages

2015-05-23 Thread Debasish Das
Hi, Is it possible to add GPL/LGPL code on spark packages or it must be licensed under Apache as well ? I want to expose Professor Tim Davis's LGPL library for sparse algebra and ECOS GPL library through the package. Thanks. Deb

Re: Kryo option changed

2015-05-23 Thread Debasish Das
Tried 8mb...still I am failing on the same error... On Sat, May 23, 2015 at 6:10 PM, Ted Yu yuzhih...@gmail.com wrote: bq. it shuld be 8mb Please use the above syntax. Cheers On Sat, May 23, 2015 at 6:04 PM, Debasish Das debasish.da...@gmail.com wrote: Hi, I am on last week's master

Re: Kryo option changed

2015-05-23 Thread Ted Yu
Pardon me. Please use '8192k' Cheers On Sat, May 23, 2015 at 6:24 PM, Debasish Das debasish.da...@gmail.com wrote: Tried 8mb...still I am failing on the same error... On Sat, May 23, 2015 at 6:10 PM, Ted Yu yuzhih...@gmail.com wrote: bq. it shuld be 8mb Please use the above syntax.

Re: Kryo option changed

2015-05-23 Thread Josh Rosen
Which commit of master are you building off? It looks like there was a bugfix for an issue related to KryoSerializer buffer configuration: https://github.com/apache/spark/pull/5934 That patch was committed two weeks ago, but you mentioned that you're building off a newer version of master.

Re: Tungsten's Vectorized Execution

2015-05-23 Thread Yijie Shen
Davies and Reynold, Glad to hear about the status. I’ve seen [SPARK-7813](https://issues.apache.org/jira/browse/SPARK-7813) and watching it now. If I understand correctly, it’s aimed at moving CodeGenerator’s expressionEvaluator’s code-gen logic into each expressions’ eval() and eliminating

Re: [IMPORTANT] Committers please update merge script

2015-05-23 Thread Ted Yu
INFRA-9646 has been resolved. FYI On Wed, May 13, 2015 at 6:00 PM, Patrick Wendell pwend...@gmail.com wrote: Hi All - unfortunately the fix introduced another bug, which is that fixVersion was not updated properly. I've updated the script and had one other person test it. So committers

Re: spark packages

2015-05-23 Thread Patrick Wendell
Yes - spark packages can include non ASF licenses. On Sat, May 23, 2015 at 6:16 PM, Debasish Das debasish.da...@gmail.com wrote: Hi, Is it possible to add GPL/LGPL code on spark packages or it must be licensed under Apache as well ? I want to expose Professor Tim Davis's LGPL library for

Power iteration clustering

2015-05-23 Thread Debasish Das
Hi, What was the motivation to write power iteration clustering using graphx and not a vector matrix multiplication over similarity matrix represented as say coordinate matrix ? We can use gemv in that flow to block the computation. Over graphx can we do all k eigen vector computation together

Re: spark packages

2015-05-23 Thread DB Tsai
I thought LGPL is okay but GPL is not okay for Apache project. On Saturday, May 23, 2015, Patrick Wendell pwend...@gmail.com wrote: Yes - spark packages can include non ASF licenses. On Sat, May 23, 2015 at 6:16 PM, Debasish Das debasish.da...@gmail.com javascript:; wrote: Hi, Is it

Re: spark packages

2015-05-23 Thread Reynold Xin
That's the nice thing about Spark packages. It is just a package index for libraries and applications built on top of Spark and not part of the Spark codebase, so it is not restricted to follow only ASF-compatible licenses. On Sat, May 23, 2015 at 10:12 PM, DB Tsai dbt...@dbtsai.com wrote: I