Hi Tathagata, Thanks very much for this tip and for the quick reply. It absolutely did the trick!
One small note for others: make sure your command-line invocation of the spark-submit script has the argument “—master” also setting “local[n]” with n > 1. Thanks again, Rishi On Aug 28, 2014, at 1:09 PM, Tathagata Das <tathagata.das1...@gmail.com<mailto:tathagata.das1...@gmail.com>> wrote: Try using "local[n]" with n > 1, instead of local. Since receivers take up 1 slot, and "local" is basically 1 slot, there is no slot left to process the data. That's why nothing gets printed. TD On Thu, Aug 28, 2014 at 10:28 AM, Verma, Rishi (398J) <rishi.ve...@jpl.nasa.gov<mailto:rishi.ve...@jpl.nasa.gov>> wrote: Hi Folks, I’d like to find out tips on how to convert the RDDs inside a Spark Streaming DStream to a set of SchemaRDDs. My DStream contains JSON data pushed over from Kafka, and I’d like to use SparkSQL’s JSON import function (i.e. jsonRDD) to register the JSON dataset as a table, and perform queries on it. Here’s a code snippet of my latest attempt (in Scala): … val sc = new SparkContext(conf) val ssc = new StreamingContext("local", this.getClass.getName, Seconds(1)) ssc.checkpoint("checkpoint") val stream = KafkaUtils.createStream(ssc, "localhost:2181", “group", Map(“topic" -> 10)).map(_._2) val sql = new SQLContext(sc) stream.foreachRDD(rdd => { if (rdd.count > 0) { // message received val sqlRDD = sql.jsonRDD(rdd) sqlRDD.printSchema() } else { println(“No message received") } }) … This compiles and runs when I submit it to Spark (local-mode); however, I never seem to be able to successfully see a schema printed on my console, via the “sqlRDD.printSchema()” method when Kafka is streaming my JSON messages to the “topic” topic name. I know my JSON is valid and my Kafka connection works fine, I’ve been able to print the stream messages in their raw format, just not as SchemaRDDs. Any tips? Suggestions? Thanks much, --- Rishi Verma NASA Jet Propulsion Laboratory California Institute of Technology --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org<mailto:user-unsubscr...@spark.apache.org> For additional commands, e-mail: user-h...@spark.apache.org<mailto:user-h...@spark.apache.org> --- Rishi Verma NASA Jet Propulsion Laboratory California Institute of Technology
