Re: Pyspark Error when broadcast numpy array

2014-11-12 Thread bliuab
or upcoming 1.2 release, could you upgrade to 1.1? Davies On Tue, Nov 11, 2014 at 8:37 PM, bliuab [hidden email] http://user/SendEmail.jtp?type=nodenode=18684i=0 wrote: Dear Liu: Thank you very much for your help. I will update that patch. By the way, as I have succeed to broadcast

Pyspark Error when broadcast numpy array

2014-11-11 Thread bliuab
In spark-1.0.2, I have come across an error when I try to broadcast a quite large numpy array(with 35M dimension). The error information except the java.lang.NegativeArraySizeException error and details is listed below. Moreover, when broadcast a relatively smaller numpy array(30M dimension),

Re: Pyspark Error when broadcast numpy array

2014-11-11 Thread bliuab
at 12:29 PM, Davies Liu-2 [via Apache Spark User List] ml-node+s1001560n18673...@n3.nabble.com wrote: This PR fix the problem: https://github.com/apache/spark/pull/2659 cc @josh Davies On Tue, Nov 11, 2014 at 7:47 PM, bliuab [hidden email] http://user/SendEmail.jtp?type=nodenode=18673i=0

Re: Pyspark Error when broadcast numpy array

2014-11-11 Thread bliuab
didn't read your post carefully. The broadcast in Python had been improved much since 1.1, I think it will work in 1.1 or upcoming 1.2 release, could you upgrade to 1.1? Davies On Tue, Nov 11, 2014 at 8:37 PM, bliuab [hidden email] http://user/SendEmail.jtp?type=nodenode=18684i=0 wrote