Hello Everyone, A bit confused on this one...I have set up the KafkaWordCount found here: https://github.com/apache/spark/blob/master/examples/scala-2.10/src/main/java/org/apache/spark/examples/streaming/JavaKafkaWordCount.java
Everything runs fine when I run it using this on instance A: repository: https://github.com/aijibd/KafkaSpark The word count works fine, etc. I literally clone that repository and try to run it on instance B and get the following error: I've gotten NoClassDefFoundErrors while building the pom file, but I don't think missing dependencies is the issue here as it says "/usr/lib/hadoop/bin/hadoop: No such file or directory" which is true, hadoop is under usr/bin/ not usr/lib. I'm confused why it is looking for a file there...I don't recall changing any settings regarding this. Thank you! line 83: /usr/lib/hadoop/bin/hadoop: No such file or directory Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream at org.apache.spark.deploy.SparkSubmitArguments.parse$1(SparkSubmitArguments.scala:307) at org.apache.spark.deploy.SparkSubmitArguments.parseOpts(SparkSubmitArguments.scala:220) at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:75) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:70) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 5 more
