Hi Buntu,
You could something similar to the following:
val receiver_stream = new ReceiverInputDStream(ssc) {
override def getReceiver(): Receiver[Nothing] = ??? //Whatever
}.map((x : String) = (null, x))
val config = new Configuration()
config.set(mongo.output.uri,
You can convert this ReceiverInputDStream
http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.streaming.dstream.ReceiverInputDStream
into PairRDDFuctions
http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.rdd.PairRDDFunctions
and call the
of
org.apache.spark.streaming.dstream.DStream[String]
Any help on how to go about saving a DStream to Hadoop would be appreciated.
Thanks!
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/How-to-save-ReceiverInputDStream-to-Hadoop-using-saveAsNewAPIHadoopFile-tp16062.html
Sent
be appreciated.
Thanks!
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/How-to-save-ReceiverInputDStream-to-Hadoop-using-saveAsNewAPIHadoopFile-tp16062.html
Sent from the Apache Spark User List mailing list archive at Nabble.com
to Hadoop would be
appreciated.
Thanks!
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/How-to-save-ReceiverInputDStream-to-Hadoop-using-saveAsNewAPIHadoopFile-tp16062.html
Sent from the Apache Spark User List mailing list archive at Nabble.com
Your RDD does not contain pairs, since you .map(_._2) (BTW that can
just be .values). Hadoop files means SequenceFiles and those
store key-value pairs. That's why the method only appears for
RDD[(K,V)].
On Fri, Oct 10, 2014 at 3:50 AM, Buntu Dev buntu...@gmail.com wrote:
Thanks Sean, but I'm
