GitHub user lw-lin opened a pull request: https://github.com/apache/spark/pull/17120
[SPARK-19715][Structured Streaming] Option to Strip Paths in FileSource ## What changes were proposed in this pull request? Today, we compare the whole path when deciding if a file is new in the FileSource for structured streaming. However, this cause cause false negatives in the case where the path has changed in a cosmetic way (i.e. changing `s3n` to `s3a`). This patch adds an option `fileNameOnly` that causes the new file check to be based only on the filename (but still store the whole path in the log). ## Usage ```scala spark .readStream .option("fileNameOnly", true) .text("s3n://bucket/dir1/dir2") .writeStream ... ``` ## How was this patch tested? Added a test case You can merge this pull request into a Git repository by running: $ git pull https://github.com/lw-lin/spark filename-only Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/17120.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #17120 ---- commit aeb10d100a24ca644745fb8b26985b584fd5118e Author: Liwei Lin <lwl...@gmail.com> Date: 2017-02-28T15:29:17Z Add support for `fileNameOnly` ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org