Re: text file stream to HDFS

2015-07-04 Thread Ted Yu 
Please take a look
at 
streaming/src/main/scala/org/apache/spark/streaming/dstream/PairDStreamFunctions.scala

  def saveAsHadoopFiles[F : OutputFormat[K, V]](
  prefix: String,
  suffix: String
)(implicit fm: ClassTag[F]): Unit = ssc.withScope {

Cheers

On Sat, Jul 4, 2015 at 5:23 PM, ravi tella ddpis...@gmail.com wrote:

 Hello,
 How should I write a text file stream DStream to HDFS.
 I tried the the following
  val lines = ssc.textFileStream(hdfs:/user/hadoop/spark/streaming/input/)
 lines.saveAsTextFile(hdfs:/user/hadoop/output1)

  val lines = ssc.textFileStream(hdfs:/user/hadoop/spark/streaming/input/)
 lines.saveAsHadoopFiles(hdfs:/user/hadoop/output1,out)

 in both cases i received compile errors saying:
  is not a member of org.apache.spark.streaming.dstream.DStream[String]

 Thanks in advance for help.

text file stream to HDFS

2015-07-04 Thread ravi tella 
Hello,
How should I write a text file stream DStream to HDFS.
I tried the the following
 val lines = ssc.textFileStream(hdfs:/user/hadoop/spark/streaming/input/)
lines.saveAsTextFile(hdfs:/user/hadoop/output1)

 val lines = ssc.textFileStream(hdfs:/user/hadoop/spark/streaming/input/)
lines.saveAsHadoopFiles(hdfs:/user/hadoop/output1,out)

in both cases i received compile errors saying:
 is not a member of org.apache.spark.streaming.dstream.DStream[String]

Thanks in advance for help.