Unsubscribe

Unsubscribe

Spark Streaming - Increasing number of executors slows down processing rate

ures) spark.streaming.kafka.consumer.poll.ms=1 Have tried increasing driver memory to 4GB and also increased driver.cores to 4. If anybody has faced similar issues please provide some pointers to how to address this issue. Thanks a lot for your time. Regards, Edwin

Re: Spark Streaming from Kafka, deal with initial heavy load.

Hi, You can enable backpressure to handle this. spark.streaming.backpressure.enabled spark.streaming.receiver.maxRate Thanks, Edwin On Mar 18, 2017, 12:53 AM -0400, sagarcasual . <sagarcas...@gmail.com>, wrote: > Hi, we have spark 1.6.1 streaming from Kafka (0.10.1) topic usi

RE: RE: Fast write datastore...

to man Yarn and impala co-exist. Thanks, Edwin On Mar 16, 2017, 5:44 AM -0400, yohann jardin <yohannjar...@hotmail.com>, wrote: > Hello everyone, > > I'm also really interested in the answers as I will be facing the same issue > soon. > Muthu, if you evaluate again Apache

Apache Spark broadcast error: Error sending message as driverActor is null [message = UpdateBlockInfo(BlockManagerId

I'm using Apache Spark 1.1.0 and I'm currently having issue with broadcast method. So when I call broadcast function on a small dataset to a 5 nodes cluster, I experiencing the Error sending message as driverActor is null after broadcast the variables several times (apps running under jboss). Any

Proper way to check SparkContext's status within code

Hi, Is there a way to check the status of the SparkContext regarding whether it's alive or not through the code, not through UI or anything else? Thanks Edwin -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Proper-way-to-check-SparkContext-s-status

cache function is not working on RDD from parallelize

). So does anyone have any idea on this? Thanks Edwin -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/cache-function-is-not-working-on-RDD-from-parallelize-tp18219.html Sent from the Apache Spark User List mailing list archive at Nabble.com

apache spark union function cause executors disassociate (Lost executor 1 on 172.32.1.12: remote Akka client disassociated)

be a problem (can tell from the UI). what worth mentioning is that one rdd is significant bigger than the other one (much bigger), does anyone have any idea why? Thanks Edwin -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/apache-spark-union-function-cause

Re: apache spark union function cause executors disassociate (Lost executor 1 on 172.32.1.12: remote Akka client disassociated)

does union function cause any data shuffling? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/apache-spark-union-function-cause-executors-disassociate-Lost-executor-1-on-172-32-1-12-remote-Akka--tp15442p15444.html Sent from the Apache Spark User List

Re: apache spark union function cause executors disassociate (Lost executor 1 on 172.32.1.12: remote Akka client disassociated)

19:02:45,963 INFO [org.apache.spark.MapOutputTrackerMaster] (spark-akka.actor.default-dispatcher-14) Size of output statuses for shuffle 1 is 216 bytes 19:02:45,964 INFO [org.apache.spark.scheduler.DAGScheduler] (spark-akka.actor.default-dispatcher-14) Got job 5 (getCallSite at null:-1) with

Transient association error on a 3 nodes cluster

bytes. Thanks Edwin -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Transient-association-error-on-a-3-nodes-cluster-tp14914.html Sent from the Apache Spark User List mailing list archive at Nabble.com