Thanks Tathagata! You are right, I have packaged the contents of the spark shipped example jar into my jar....which contains serveral HDFS configuration files like hdfs-default.xml etc. Thanks!
bit1...@163.com From: Tathagata Das Date: 2015-02-24 12:04 To: bit1...@163.com CC: yuzhihong; silvio.fiorito; user Subject: Re: Re_ Re_ Does Spark Streaming depend on Hadoop_(4) You could have a hdfs configuration files in the classpath of the program. HDFS libraries that Spark uses automatically picks those up and starts using them. TD On Mon, Feb 23, 2015 at 7:47 PM, bit1...@163.com <bit1...@163.com> wrote: I am crazy for frequent mail rejection so I create a new thread SMTP error, DOT: 552 spam score (5.7) exceeded threshold (FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_REPLY,HTML_FONT_FACE_BAD,HTML_MESSAGE,RCVD_IN_BL_SPAMCOP_NET,SPF_PASS Hi Silvio and Ted I know there is a configuration parameter to control to write log to HDFS, but I didn't enable it. From the stack trace, looks like accessing HDFS is triggered in my code, but I didn't use HDFS, following is my code: object MyKafkaWordCount { def main(args: Array[String]) { println("Start to run MyKafkaWordCount") val conf = new SparkConf().setAppName("MyKafkaWordCount").setMaster("local[20]") val ssc = new StreamingContext(conf, Seconds(3)) val topicMap = Map("topic-p6-r2"->1) val zkQuorum = "localhost:2181"; val group = "topic-p6-r2-consumer-group" //Kakfa has 6 partitions, here create 6 Receiver val streams = (1 to 6).map ( _ => KafkaUtils.createStream(ssc, zkQuorum, group, topicMap).map(_._2) ) //repartition to 18, 3 times of the receiver val partitions = ssc.union(streams).repartition(18).map("DataReceived: " + _) partitions.print() ssc.start() ssc.awaitTermination() } } bit1...@163.com