Jack Hu created SPARK-3276:
------------------------------
Summary: Provide a API to specify whether the old files need to be
ignored in file input text DStream
Key: SPARK-3276
URL: https://issues.apache.org/jira/browse/SPARK-3276
Project: Spark
Issue Type: Bug
Components: Streaming
Affects Versions: 1.0.2
Reporter: Jack Hu
Currently, only one API called textFileStream in StreamingContext to specify
the text file dstream, which ignores the old files always. On some times, the
old files is still useful.
Need a API to let user choose whether the old files need to be ingored or not .
The API currently in StreamingContext:
def textFileStream(directory: String): DStream[String] = {
fileStream[LongWritable, Text,
TextInputFormat](directory).map(_._2.toString)
}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
