Re: fileStream with old files

2015-07-15 Thread Terry Hole
); } Files.deleteIfExists(testDir); } *From:* Tathagata Das [mailto:t...@databricks.com] *Sent:* Wednesday, July 15, 2015 00:01 *To:* Terry Hole *Cc:* Hunter Morgan; user@spark.apache.org *Subject:* Re: fileStream with old files It was added, but its not documented publicly. I

RE: fileStream with old files

2015-07-15 Thread Hunter Morgan
(eachTempFile); } Files.deleteIfExists(testDir); } From: Tathagata Das [mailto:t...@databricks.com] Sent: Wednesday, July 15, 2015 00:01 To: Terry Hole Cc: Hunter Morgan; user@spark.apache.org Subject: Re: fileStream with old files It was added, but its not documented publicly. I am planning

Re: fileStream with old files

2015-07-14 Thread Tathagata Das
, 1654564); to big/small. Are there known limitations of the onlyNewFiles=false? Am I doing something wrong? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/fileStream-with-old-files-tp23802.html Sent from the Apache Spark User List mailing list archive

Re: fileStream with old files

2015-07-13 Thread Terry Hole
(spark.streaming.minRememberDuration, 1654564); to big/small. Are there known limitations of the onlyNewFiles=false? Am I doing something wrong? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/fileStream-with-old-files-tp23802.html Sent from the Apache