HiI'm using Apache Spark 1.1.0 and I'm currently having issue with broadcast
method. So when I call broadcast function on a small dataset to a 5 nodes
cluster, I experiencing the Error sending message as driverActor is null
after broadcast the variables several times (apps running under jboss).
help would be appreciate.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Apache-Spark-broadcast-error-Error-sending-message-as-driverActor-is-null-message-UpdateBlockInfo-Bld-tp21320.html
Sent from the Apache Spark User List mailing list archive
to spark, can anyone help me?
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/spark-broadcast-error-tp19643.html
Sent from the Apache Spark User List mailing list archive at Nabble.com
-Concurrently-Fail-With-Broadcast-Error-tp18374.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Cool.. While let me try that.. any other suggestion(s) on things I can try?
On Mon, Sep 15, 2014 at 9:59 AM, Davies Liu dav...@databricks.com wrote:
I think the 1.1 will be really helpful for you, it's all compatitble
with 1.0, so it's
not hard to upgrade to 1.1.
On Mon, Sep 15, 2014 at
Hi Akhil,
So with your config (specifically with set(spark.akka.frameSize ,
1000)) , I see the error:
org.apache.spark.SparkException: Job aborted due to stage failure:
Serialized task 0:0 was 401970046 bytes which exceeds spark.akka.frameSize
(10485760 bytes). Consider using broadcast
Try:
rdd = sc.broadcast(matrix)
Or
rdd = sc.parallelize(matrix,100) // Just increase the number of slices,
give it a try.
Thanks
Best Regards
On Mon, Sep 15, 2014 at 2:18 PM, Chengi Liu chengi.liu...@gmail.com wrote:
Hi Akhil,
So with your config (specifically with
So.. same result with parallelize (matrix,1000)
with broadcast.. seems like I got jvm core dump :-/
4/09/15 02:31:22 INFO BlockManagerInfo: Registering block manager
host:47978 with 19.2 GB RAM
14/09/15 02:31:22 INFO BlockManagerInfo: Registering block manager
host:43360 with 19.2 GB RAM
Unhandled
I think the 1.1 will be really helpful for you, it's all compatitble
with 1.0, so it's
not hard to upgrade to 1.1.
On Mon, Sep 15, 2014 at 2:35 AM, Chengi Liu chengi.liu...@gmail.com wrote:
So.. same result with parallelize (matrix,1000)
with broadcast.. seems like I got jvm core dump :-/
Hi,
I am trying to create an rdd out of large matrix sc.parallelize
suggest to use broadcast
But when I do
sc.broadcast(data)
I get this error:
Traceback (most recent call last):
File stdin, line 1, in module
File /usr/common/usg/spark/1.0.2/python/pyspark/context.py, line 370,
in
Specifically the error I see when I try to operate on rdd created by
sc.parallelize method
: org.apache.spark.SparkException: Job aborted due to stage failure:
Serialized task 12:12 was 12062263 bytes which exceeds spark.akka.frameSize
(10485760 bytes). Consider using broadcast variables for large
How? Example please..
Also, if I am running this in pyspark shell.. how do i configure
spark.akka.frameSize ??
On Sun, Sep 14, 2014 at 7:43 AM, Akhil Das ak...@sigmoidanalytics.com
wrote:
When the data size is huge, you better of use the torrentBroadcastFactory.
Thanks
Best Regards
On
And when I use sparksubmit script, I get the following error:
py4j.protocol.Py4JJavaError: An error occurred while calling
o26.trainKMeansModel.
: org.apache.spark.SparkException: Job aborted due to stage failure: All
masters are unresponsive! Giving up.
at
Hey Chengi,
What's the version of Spark you are using? It have big improvements
about broadcast in 1.1, could you try it?
On Sun, Sep 14, 2014 at 8:29 PM, Chengi Liu chengi.liu...@gmail.com wrote:
Any suggestions.. I am really blocked on this one
On Sun, Sep 14, 2014 at 2:43 PM, Chengi Liu
I am using spark1.0.2.
This is my work cluster.. so I can't setup a new version readily...
But right now, I am not using broadcast ..
conf = SparkConf().set(spark.executor.memory,
32G).set(spark.akka.frameSize, 1000)
sc = SparkContext(conf = conf)
rdd = sc.parallelize(matrix,5)
from
And the thing is code runs just fine if I reduce the number of rows in my
data?
On Sun, Sep 14, 2014 at 8:45 PM, Chengi Liu chengi.liu...@gmail.com wrote:
I am using spark1.0.2.
This is my work cluster.. so I can't setup a new version readily...
But right now, I am not using broadcast ..
Hi All,
When I run the program shown below, I receive the error shown below.
I am running the current version of branch-0.9 from github. Note that
I do not receive the error when I replace 2 ** 29 with 2 ** X,
where X 29. More interestingly, I do not receive the error when X =
30, and when X
17 matches
Mail list logo