[jira] [Commented] (SPARK-18512) FileNotFoundException on _temporary directory with Spark Streaming 2.0.1 and S3A

[ https://issues.apache.org/jira/browse/SPARK-18512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751577#comment-15751577 ] Mathieu D commented on SPARK-18512: --- [SPARK-18883] > FileNotFoundException on _temporary directory

[jira] [Commented] (SPARK-18512) FileNotFoundException on _temporary directory with Spark Streaming 2.0.1 and S3A

[ https://issues.apache.org/jira/browse/SPARK-18512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751492#comment-15751492 ] Steve Loughran commented on SPARK-18512: yes, file a SPARK one and that can be a base for blame

[jira] [Commented] (SPARK-18512) FileNotFoundException on _temporary directory with Spark Streaming 2.0.1 and S3A

[ https://issues.apache.org/jira/browse/SPARK-18512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751329#comment-15751329 ] Mathieu D commented on SPARK-18512: --- I'm experiencing the same problem, but we're using HDFS, not S3.

[jira] [Commented] (SPARK-18512) FileNotFoundException on _temporary directory with Spark Streaming 2.0.1 and S3A

[ https://issues.apache.org/jira/browse/SPARK-18512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15726310#comment-15726310 ] Steve Loughran commented on SPARK-18512: It'd be good to get some more details from people who

[jira] [Commented] (SPARK-18512) FileNotFoundException on _temporary directory with Spark Streaming 2.0.1 and S3A

[ https://issues.apache.org/jira/browse/SPARK-18512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15726148#comment-15726148 ] Adrian Bridgett commented on SPARK-18512: - Thanks Steve - just seen a "next few weeks" mentioned

[jira] [Commented] (SPARK-18512) FileNotFoundException on _temporary directory with Spark Streaming 2.0.1 and S3A

[ https://issues.apache.org/jira/browse/SPARK-18512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15726135#comment-15726135 ] Steve Loughran commented on SPARK-18512: ah, the "when will 2.8 ship" question. Really close, Jun

[jira] [Commented] (SPARK-18512) FileNotFoundException on _temporary directory with Spark Streaming 2.0.1 and S3A

[ https://issues.apache.org/jira/browse/SPARK-18512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15725242#comment-15725242 ] Adrian Bridgett commented on SPARK-18512: - So currently, with Spark2 there's no sensible way to

[jira] [Commented] (SPARK-18512) FileNotFoundException on _temporary directory with Spark Streaming 2.0.1 and S3A

[ https://issues.apache.org/jira/browse/SPARK-18512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722354#comment-15722354 ] Steve Loughran commented on SPARK-18512: of course, if you do switch to EMRFS, you should get

[jira] [Commented] (SPARK-18512) FileNotFoundException on _temporary directory with Spark Streaming 2.0.1 and S3A

[ https://issues.apache.org/jira/browse/SPARK-18512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722320#comment-15722320 ] Steve Loughran commented on SPARK-18512: Actually, this is the problem whcih MAPREDUCE-6478 deals

[jira] [Commented] (SPARK-18512) FileNotFoundException on _temporary directory with Spark Streaming 2.0.1 and S3A

[ https://issues.apache.org/jira/browse/SPARK-18512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722304#comment-15722304 ] Steve Loughran commented on SPARK-18512: no. What you are seeing is an eventual consistency

[jira] [Commented] (SPARK-18512) FileNotFoundException on _temporary directory with Spark Streaming 2.0.1 and S3A

[ https://issues.apache.org/jira/browse/SPARK-18512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702656#comment-15702656 ] Giuseppe Bonaccorso commented on SPARK-18512: - Thanks! I've already set mergeSchema=false,

[jira] [Commented] (SPARK-18512) FileNotFoundException on _temporary directory with Spark Streaming 2.0.1 and S3A

[ https://issues.apache.org/jira/browse/SPARK-18512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702394#comment-15702394 ] Giuseppe Bonaccorso commented on SPARK-18512: - It happens after about 1.000.000 writings of

[jira] [Commented] (SPARK-18512) FileNotFoundException on _temporary directory with Spark Streaming 2.0.1 and S3A

[ https://issues.apache.org/jira/browse/SPARK-18512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702373#comment-15702373 ] Steve Loughran commented on SPARK-18512: one question: what's the size of data being committed

[jira] [Commented] (SPARK-18512) FileNotFoundException on _temporary directory with Spark Streaming 2.0.1 and S3A

[ https://issues.apache.org/jira/browse/SPARK-18512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702368#comment-15702368 ] Steve Loughran commented on SPARK-18512: This looks like a consistency problem; s3 listing always

[jira] [Commented] (SPARK-18512) FileNotFoundException on _temporary directory with Spark Streaming 2.0.1 and S3A

[ https://issues.apache.org/jira/browse/SPARK-18512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15695800#comment-15695800 ] Genmao Yu commented on SPARK-18512: --- [~giuseppe.bonaccorso] How can i reproduce this failure? Can u

[jira] [Commented] (SPARK-18512) FileNotFoundException on _temporary directory with Spark Streaming 2.0.1 and S3A

[ https://issues.apache.org/jira/browse/SPARK-18512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15686061#comment-15686061 ] Giuseppe Bonaccorso commented on SPARK-18512: - No, I didn't. I'm afraid that relaunching a

[jira] [Commented] (SPARK-18512) FileNotFoundException on _temporary directory with Spark Streaming 2.0.1 and S3A

[ https://issues.apache.org/jira/browse/SPARK-18512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15684863#comment-15684863 ] Shixiong Zhu commented on SPARK-18512: -- Did you enable speculation? > FileNotFoundException on