It turns out there is a bug in the code which makes an infinite loop some
time after start. :)
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-Stuck-After-10mins-Issue-tp23189p23210.html
Sent from the Apache Spark User List mailing list
Hi,
I have a Spark Streaming application that reads messages from Kafka
(multiple topics) and does aggregation on the data via updateStateByKey with
50 Spark workers where each has 1 core and 6G RAM. It is working fine for
the first 10mins or so, but then it will stuck in the foreachRDD
And here is the Thread Dump, where seems every worker is waiting for Executor
#6 Thread 95: sparkExecutor-akka.actor.default-dispatcher-22 (RUNNABLE) to
be complete:
Thread 41: BLOCK_MANAGER cleanup timer (WAITING)
Thread 42: BROADCAST_VARS cleanup timer (WAITING)
Thread 44: shuffle-client-0
Hi all,
I'm building a Spark Streaming application that will continuously read
multiple kafka topics at the same time. However, I found a weird issue that
it reads only hundreds of messages then it stopped reading any more. If I
changed the three topic to only one topic, then it is fine and it
Hi all,
I'm trying to run the sample Spark application in version v1.2.0 and above.
However, I've encountered a weird issue like below. This issue only be seen
in v1.2.0 and above, but v1.1.0 and v1.1.1 are fine.
The sample code:
val sc : SparkContext = new SparkContext(conf)
val
Hi all,
I've been using Spark 1.1.0 for a while, and now would like to upgrade to
Spark 1.1.1 or above. However, it throws the following errors:
18:05:31.522 [sparkDriver-akka.actor.default-dispatcher-3hread] ERROR
TaskSchedulerImpl - Lost executor 37 on hcompute001: remote Akka client
Hi,
Are there any way that I can setup a remote HDFS for Spark (more specific,
for Spark Streaming checkpoints)? The reason I'm asking is that our Spark
and HDFS do not run on the same machines. I've been looked around but still
no clue so far.
Thanks,
EH
--
View this message in context
checkpoints? Thank you for your help.
Best,
EH
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Setup-Remote-HDFS-for-Spark-tp19481p19485.html
Sent from the Apache Spark User List mailing list archive at Nabble.com
