[ https://issues.apache.org/jira/browse/SPARK-3276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen updated SPARK-3276: ----------------------------- Summary: Provide a API to specify MIN_REMEMBER_DURATION for files to consider as input in streaming (was: Provide a API to specify whether the old files need to be ignored in file input text DStream) See SPARK-6061. I'm going to hijack this slightly to suggest that the real issue is not being able to choose to see only new files, which you already can, but controlling {{MIN_REMEMBER_DURATION}}, which is currently hard-coded. > Provide a API to specify MIN_REMEMBER_DURATION for files to consider as input > in streaming > ------------------------------------------------------------------------------------------ > > Key: SPARK-3276 > URL: https://issues.apache.org/jira/browse/SPARK-3276 > Project: Spark > Issue Type: Improvement > Components: Streaming > Affects Versions: 1.2.0 > Reporter: Jack Hu > Priority: Minor > > Currently, only one API called textFileStream in StreamingContext to specify > the text file dstream, which ignores the old files always. On some times, the > old files is still useful. > Need a API to let user choose whether the old files need to be ingored or not > . > The API currently in StreamingContext: > def textFileStream(directory: String): DStream[String] = { > fileStream[LongWritable, Text, > TextInputFormat](directory).map(_._2.toString) > } -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org