Apologies in delay in getting back on this. It seems the Kinesis example does not run on Spark 1.1.0 even when it is built using kinesis-acl profile because of a dependency conflict in http client (same issue as http://mail-archives.apache.org/mod_mbox/spark-dev/201409.mbox/%3ccajob8btdxks-7-spjj5jmnw0xsnrjwdpcqqtjht1hun6j4z...@mail.gmail.com%3E). I had to add a later version of http client in kinesis-acl profile to make it run. Then, the Kinesis example sets master as local so it does not honour the MASTER environment variable as other examples do. Once I was able to resolve these issues, I was finally able to reproduce the issue. The example works fine in local mode but does not do anything when receiver runs in remote workers.
Spark streaming does not report any blocks received from the receivers even though I can see the following lines in the app logs (I modified the debug line to print size as well): 14/09/19 12:30:18 DEBUG ReceiverSupervisorImpl: Pushed block input-0-1411129664668 in 15 ms 14/09/19 12:30:18 DEBUG ReceiverSupervisorImpl: Reported block input-0-1411129664668 of size 1 Here are the screenshots of Spark admin: http://imgur.com/a/gWKYm I also ran other examples (custom receiver, etc) in both local and distributed mode and they seem to be working fine. Any ideas? Thanks, Aniket On 12 September 2014 02:49, Tathagata Das <tathagata.das1...@gmail.com> wrote: > This is very puzzling, given that this works in the local mode. > > Does running the kinesis example work with your spark-submit? > > > https://github.com/apache/spark/blob/master/extras/kinesis-asl/src/main/scala/org/apache/spark/examples/streaming/KinesisWordCountASL.scala > > The instructions are present in the streaming guide. > https://github.com/apache/spark/blob/master/docs/streaming-kinesis-integration.md > > If that does not work on cluster, then I would see the streaming UI for > the number records that are being received, and the stages page for whether > jobs are being executed for every batch or not. Can tell use whether that > is working well. > > Also ccing, chris fregly who wrote Kinesis integration. > > TD > > > > > On Thu, Sep 11, 2014 at 4:51 AM, Aniket Bhatnagar < > aniket.bhatna...@gmail.com> wrote: > >> Hi all >> >> I am trying to run kinesis spark streaming application on a standalone >> spark cluster. The job works find in local mode but when I submit it (using >> spark-submit), it doesn't do anything. I enabled logs >> for org.apache.spark.streaming.kinesis package and I regularly get the >> following in worker logs: >> >> 14/09/11 11:41:25 DEBUG KinesisRecordProcessor: Stored: Worker >> x.x.x.x:b88a9210-cbb9-4c31-8da7-35fd92faba09 stored 34 records for shardId >> shardId-000000000000 >> 14/09/11 11:41:25 DEBUG KinesisRecordProcessor: Stored: Worker >> x.x.x.x:b2e9c20f-470a-44fe-a036-630c776919fb stored 31 records for shardId >> shardId-000000000001 >> >> But the job does not perform any operations defined on DStream. To >> investigate this further, I changed the kinesis-asl's KinesisUtils to >> perform the following computation on the DStream created >> using ssc.receiverStream(new KinesisReceiver...): >> >> stream.count().foreachRDD(rdd => rdd.foreach(tuple => logInfo("Emitted " >> + tuple))) >> >> Even the above line does not results in any corresponding log entries >> both in driver and worker logs. The only relevant logs that I could find in >> driver logs are: >> 14/09/11 11:40:58 INFO DAGScheduler: Stage 2 (foreach at >> KinesisUtils.scala:68) finished in 0.398 s >> 14/09/11 11:40:58 INFO SparkContext: Job finished: foreach at >> KinesisUtils.scala:68, took 4.926449985 s >> 14/09/11 11:40:58 INFO JobScheduler: Finished job streaming job >> 1410435653000 ms.0 from job set of time 1410435653000 ms >> 14/09/11 11:40:58 INFO JobScheduler: Starting job streaming job >> 1410435653000 ms.1 from job set of time 1410435653000 ms >> 14/09/11 11:40:58 INFO SparkContext: Starting job: foreach at >> KinesisUtils.scala:68 >> 14/09/11 11:40:58 INFO DAGScheduler: Registering RDD 13 (union at >> DStream.scala:489) >> 14/09/11 11:40:58 INFO DAGScheduler: Got job 3 (foreach at >> KinesisUtils.scala:68) with 2 output partitions (allowLocal=false) >> 14/09/11 11:40:58 INFO DAGScheduler: Final stage: Stage 5(foreach at >> KinesisUtils.scala:68) >> >> After the above logs, nothing shows up corresponding to KinesisUtils. I >> am out of ideas on this one and any help on this would greatly appreciated. >> >> Thanks, >> Aniket >> > >
