Hello, I want to add that, I don't even see the streaming tab in the application UI on port 4040 when I run it on the cluster. The cluster on EC2 has 1 master node and 1 worker node. The cores used on the worker node is 2 of 2 and memory used is 6GB of 6.3GB.
Can I run a spark streaming job with just 2 cores? Appreciate your time and help. Thanks On Fri, Jan 13, 2017 at 10:46 PM, shyla deshpande <deshpandesh...@gmail.com> wrote: > Hello, > > My spark streaming app that reads kafka topics and prints the DStream > works fine on my laptop, but on AWS cluster it produces no output and no > errors. > > Please help me debug. > > I am using Spark 2.0.2 and kafka-0-10 > > Thanks > > The following is the output of the spark streaming app... > > > 17/01/14 06:22:41 WARN NativeCodeLoader: Unable to load native-hadoop library > for your platform... using builtin-java classes where applicable > 17/01/14 06:22:43 WARN Checkpoint: Checkpoint directory check1 does not exist > Creating new context > 17/01/14 06:22:45 WARN SparkContext: Use an existing SparkContext, some > configuration may not take effect. > 17/01/14 06:22:45 WARN KafkaUtils: overriding enable.auto.commit to false for > executor > 17/01/14 06:22:45 WARN KafkaUtils: overriding auto.offset.reset to none for > executor > 17/01/14 06:22:45 WARN KafkaUtils: overriding executor group.id to > spark-executor-whilDataStream > 17/01/14 06:22:45 WARN KafkaUtils: overriding receive.buffer.bytes to 65536 > see KAFKA-3135 > > >
