[GitHub] spark issue #20683: [SPARK-8605] Exclude files in StreamingContext. textFile...
Github user gaborgsomogyi commented on the issue: https://github.com/apache/spark/pull/20683 Don't really understand the issue itself. Which filesystem used this case? Why is it not possible to use Hadoop-compatible filesystem like HDFS for instance? This supports atomic rename. [See here](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/filesystem/introduction.html#Atomicity) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20683: [SPARK-8605] Exclude files in StreamingContext. textFile...
Github user ConcurrencyPractitioner commented on the issue: https://github.com/apache/spark/pull/20683 @jerryshao In Spark Streaming, I think ```.tmp``` is used as a suffix to indicate that the object was a file, although I do not know if this is universal. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20683: [SPARK-8605] Exclude files in StreamingContext. textFile...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/20683 > a extra boolean expression was added to test if a regex was present. Can you please explain what's the meaning of "if a regex was present"? Seems the fix is not so necessary. If you want to filter out some temp files, you can write your own `filter` instead of using Spark Streaming's default one. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20683: [SPARK-8605] Exclude files in StreamingContext. textFile...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20683 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20683: [SPARK-8605] Exclude files in StreamingContext. textFile...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20683 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20683: [SPARK-8605] Exclude files in StreamingContext. textFile...
Github user ConcurrencyPractitioner commented on the issue: https://github.com/apache/spark/pull/20683 Jenkins test this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
