hi, here is problem description, I write a custom networkreceiver to receive image data from camera. I had confirmed all the data received correctly.
1)when data received, only the networkreceiver node run at full speed, while other nodes keep idle, my spark cluster has 6 nodes. 2)And every image data is calculated many times, which I expected to calculate once. 3)How to distribute tasks to the whole cluster? I tried dstream.repartitioin.map(), problems in 2) is still existed ,and sometimes job faidled. here is spark configration, conf.set("spark.executor.memory", "3g"); conf.set("spark.shuffle.netty.connect.timeout", "300000") conf.set("spark.storage.blockManagerSlaveTimeoutMs", "300000") conf.set("spark.akka.timeout","600") conf.set("spark.akka.threads","8") node : 8 cores, mem 6G. NetWork : 1Gb/bps Any suggestions will be appreciated. thanks, QingFeng -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-NeteorkReceiver-problems-tp7109.html Sent from the Apache Spark User List mailing list archive at Nabble.com.