[ https://issues.apache.org/jira/browse/SPARK-3276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14502702#comment-14502702 ]
Emre Sevinç commented on SPARK-3276: ------------------------------------ Can someone with enough access rights assign this issue to me, (currently it is not assigned to anyone)? (Now that I've already discussed it with Spark developers and prepared a Pull Request on Github) > Provide a API to specify MIN_REMEMBER_DURATION for files to consider as input > in streaming > ------------------------------------------------------------------------------------------ > > Key: SPARK-3276 > URL: https://issues.apache.org/jira/browse/SPARK-3276 > Project: Spark > Issue Type: Improvement > Components: Streaming > Affects Versions: 1.2.0 > Reporter: Jack Hu > Priority: Minor > > Currently, only one API called textFileStream in StreamingContext to specify > the text file dstream, which ignores the old files always. On some times, the > old files is still useful. > Need a API to let user choose whether the old files need to be ingored or not > . > The API currently in StreamingContext: > def textFileStream(directory: String): DStream[String] = { > fileStream[LongWritable, Text, > TextInputFormat](directory).map(_._2.toString) > } -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org