Apache Spark broadcast error: Error sending message as driverActor is null [message = UpdateBlockInfo(BlockManagerId(4)

2015-01-22 Thread Zijing Guo
HiI'm using Apache Spark 1.1.0 and I'm currently having issue with broadcast method. So when I call broadcast function on a small dataset to a 5 nodes cluster, I experiencing the Error sending message as driverActor is null after broadcast the variables several times (apps running under jboss). 

Apache Spark broadcast error: Error sending message as driverActor is null [message = UpdateBlockInfo(BlockManagerId

2015-01-22 Thread Edwin
help would be appreciate. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Apache-Spark-broadcast-error-Error-sending-message-as-driverActor-is-null-message-UpdateBlockInfo-Bld-tp21320.html Sent from the Apache Spark User List mailing list archive

spark broadcast error

2014-11-24 Thread Ke Wang
to spark, can anyone help me? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-broadcast-error-tp19643.html Sent from the Apache Spark User List mailing list archive at Nabble.com

Multiple Applications(Spark Contexts) Concurrently Fail With Broadcast Error

2014-11-07 Thread ryaminal
-Concurrently-Fail-With-Broadcast-Error-tp18374.html Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: Broadcast error

2014-09-16 Thread Chengi Liu
Cool.. While let me try that.. any other suggestion(s) on things I can try? On Mon, Sep 15, 2014 at 9:59 AM, Davies Liu dav...@databricks.com wrote: I think the 1.1 will be really helpful for you, it's all compatitble with 1.0, so it's not hard to upgrade to 1.1. On Mon, Sep 15, 2014 at

Re: Broadcast error

2014-09-15 Thread Chengi Liu
Hi Akhil, So with your config (specifically with set(spark.akka.frameSize , 1000)) , I see the error: org.apache.spark.SparkException: Job aborted due to stage failure: Serialized task 0:0 was 401970046 bytes which exceeds spark.akka.frameSize (10485760 bytes). Consider using broadcast

Re: Broadcast error

2014-09-15 Thread Akhil Das
Try: rdd = sc.broadcast(matrix) Or rdd = sc.parallelize(matrix,100) // Just increase the number of slices, give it a try. Thanks Best Regards On Mon, Sep 15, 2014 at 2:18 PM, Chengi Liu chengi.liu...@gmail.com wrote: Hi Akhil, So with your config (specifically with

Re: Broadcast error

2014-09-15 Thread Chengi Liu
So.. same result with parallelize (matrix,1000) with broadcast.. seems like I got jvm core dump :-/ 4/09/15 02:31:22 INFO BlockManagerInfo: Registering block manager host:47978 with 19.2 GB RAM 14/09/15 02:31:22 INFO BlockManagerInfo: Registering block manager host:43360 with 19.2 GB RAM Unhandled

Re: Broadcast error

2014-09-15 Thread Davies Liu
I think the 1.1 will be really helpful for you, it's all compatitble with 1.0, so it's not hard to upgrade to 1.1. On Mon, Sep 15, 2014 at 2:35 AM, Chengi Liu chengi.liu...@gmail.com wrote: So.. same result with parallelize (matrix,1000) with broadcast.. seems like I got jvm core dump :-/

Broadcast error

2014-09-14 Thread Chengi Liu
Hi, I am trying to create an rdd out of large matrix sc.parallelize suggest to use broadcast But when I do sc.broadcast(data) I get this error: Traceback (most recent call last): File stdin, line 1, in module File /usr/common/usg/spark/1.0.2/python/pyspark/context.py, line 370, in

Re: Broadcast error

2014-09-14 Thread Chengi Liu
Specifically the error I see when I try to operate on rdd created by sc.parallelize method : org.apache.spark.SparkException: Job aborted due to stage failure: Serialized task 12:12 was 12062263 bytes which exceeds spark.akka.frameSize (10485760 bytes). Consider using broadcast variables for large

Re: Broadcast error

2014-09-14 Thread Chengi Liu
How? Example please.. Also, if I am running this in pyspark shell.. how do i configure spark.akka.frameSize ?? On Sun, Sep 14, 2014 at 7:43 AM, Akhil Das ak...@sigmoidanalytics.com wrote: When the data size is huge, you better of use the torrentBroadcastFactory. Thanks Best Regards On

Re: Broadcast error

2014-09-14 Thread Chengi Liu
And when I use sparksubmit script, I get the following error: py4j.protocol.Py4JJavaError: An error occurred while calling o26.trainKMeansModel. : org.apache.spark.SparkException: Job aborted due to stage failure: All masters are unresponsive! Giving up. at

Re: Broadcast error

2014-09-14 Thread Davies Liu
Hey Chengi, What's the version of Spark you are using? It have big improvements about broadcast in 1.1, could you try it? On Sun, Sep 14, 2014 at 8:29 PM, Chengi Liu chengi.liu...@gmail.com wrote: Any suggestions.. I am really blocked on this one On Sun, Sep 14, 2014 at 2:43 PM, Chengi Liu

Re: Broadcast error

2014-09-14 Thread Chengi Liu
I am using spark1.0.2. This is my work cluster.. so I can't setup a new version readily... But right now, I am not using broadcast .. conf = SparkConf().set(spark.executor.memory, 32G).set(spark.akka.frameSize, 1000) sc = SparkContext(conf = conf) rdd = sc.parallelize(matrix,5) from

Re: Broadcast error

2014-09-14 Thread Chengi Liu
And the thing is code runs just fine if I reduce the number of rows in my data? On Sun, Sep 14, 2014 at 8:45 PM, Chengi Liu chengi.liu...@gmail.com wrote: I am using spark1.0.2. This is my work cluster.. so I can't setup a new version readily... But right now, I am not using broadcast ..

pyspark broadcast error

2014-03-11 Thread Brad Miller
Hi All, When I run the program shown below, I receive the error shown below. I am running the current version of branch-0.9 from github. Note that I do not receive the error when I replace 2 ** 29 with 2 ** X, where X 29. More interestingly, I do not receive the error when X = 30, and when X