Hello, I checked the log file on the worker node and don't see any error there. This is the first time I am asked to run on such a small cluster. I feel its the resources issue, but it will be great help is somebody can confirm this or share your experience. Thanks
On Sat, Jan 14, 2017 at 4:01 PM, shyla deshpande <deshpandesh...@gmail.com> wrote: > Hello, > > I want to add that, > I don't even see the streaming tab in the application UI on port 4040 when > I run it on the cluster. > The cluster on EC2 has 1 master node and 1 worker node. > The cores used on the worker node is 2 of 2 and memory used is 6GB of > 6.3GB. > > Can I run a spark streaming job with just 2 cores? > > Appreciate your time and help. > > Thanks > > > > > > On Fri, Jan 13, 2017 at 10:46 PM, shyla deshpande < > deshpandesh...@gmail.com> wrote: > >> Hello, >> >> My spark streaming app that reads kafka topics and prints the DStream >> works fine on my laptop, but on AWS cluster it produces no output and no >> errors. >> >> Please help me debug. >> >> I am using Spark 2.0.2 and kafka-0-10 >> >> Thanks >> >> The following is the output of the spark streaming app... >> >> >> 17/01/14 06:22:41 WARN NativeCodeLoader: Unable to load native-hadoop >> library for your platform... using builtin-java classes where applicable >> 17/01/14 06:22:43 WARN Checkpoint: Checkpoint directory check1 does not exist >> Creating new context >> 17/01/14 06:22:45 WARN SparkContext: Use an existing SparkContext, some >> configuration may not take effect. >> 17/01/14 06:22:45 WARN KafkaUtils: overriding enable.auto.commit to false >> for executor >> 17/01/14 06:22:45 WARN KafkaUtils: overriding auto.offset.reset to none for >> executor >> 17/01/14 06:22:45 WARN KafkaUtils: overriding executor group.id to >> spark-executor-whilDataStream >> 17/01/14 06:22:45 WARN KafkaUtils: overriding receive.buffer.bytes to 65536 >> see KAFKA-3135 >> >> >> >
