[ https://issues.apache.org/jira/browse/SPARK-3276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14113704#comment-14113704 ]
Sean Owen commented on SPARK-3276: ---------------------------------- Given the nature of a stream processing framework, when would you want to keep reprocessing all old data? that is something you can do, but, doesn't require Spark Streaming > Provide a API to specify whether the old files need to be ignored in file > input text DStream > -------------------------------------------------------------------------------------------- > > Key: SPARK-3276 > URL: https://issues.apache.org/jira/browse/SPARK-3276 > Project: Spark > Issue Type: Bug > Components: Streaming > Affects Versions: 1.0.2 > Reporter: Jack Hu > > Currently, only one API called textFileStream in StreamingContext to specify > the text file dstream, which ignores the old files always. On some times, the > old files is still useful. > Need a API to let user choose whether the old files need to be ingored or not > . > The API currently in StreamingContext: > def textFileStream(directory: String): DStream[String] = { > fileStream[LongWritable, Text, > TextInputFormat](directory).map(_._2.toString) > } -- This message was sent by Atlassian JIRA (v6.2#6252) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org