[jira] [Commented] (SPARK-17593) list files on s3 very slow

[ https://issues.apache.org/jira/browse/SPARK-17593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16247886#comment-16247886 ] Steve Loughran commented on SPARK-17593: Hey nick, yes, need to move to FileSys

[jira] [Commented] (SPARK-17593) list files on s3 very slow

[ https://issues.apache.org/jira/browse/SPARK-17593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16244887#comment-16244887 ] Nick Dimiduk commented on SPARK-17593: -- So the fix in Hadoop 2.8 is for any variant

[jira] [Commented] (SPARK-17593) list files on s3 very slow

[ https://issues.apache.org/jira/browse/SPARK-17593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15742480#comment-15742480 ] Steve Loughran commented on SPARK-17593: Marking as a dependency of HADOOP-13208,

[jira] [Commented] (SPARK-17593) list files on s3 very slow

[ https://issues.apache.org/jira/browse/SPARK-17593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15562171#comment-15562171 ] Gaurav Shah commented on SPARK-17593: - added detail explanation and solution here ht

[jira] [Commented] (SPARK-17593) list files on s3 very slow

[ https://issues.apache.org/jira/browse/SPARK-17593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15503675#comment-15503675 ] Gaurav Shah commented on SPARK-17593: - I definitely agree that flattening out will he

[jira] [Commented] (SPARK-17593) list files on s3 very slow

[ https://issues.apache.org/jira/browse/SPARK-17593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15503668#comment-15503668 ] Gaurav Shah commented on SPARK-17593: - Thanks [~ste...@apache.org] S3 is definitely s

[jira] [Commented] (SPARK-17593) list files on s3 very slow

[ https://issues.apache.org/jira/browse/SPARK-17593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15503663#comment-15503663 ] Steve Loughran commented on SPARK-17593: Looking at the dir tree, anything you co

[jira] [Commented] (SPARK-17593) list files on s3 very slow

[ https://issues.apache.org/jira/browse/SPARK-17593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15503583#comment-15503583 ] Steve Loughran commented on SPARK-17593: Sean is right: this is primarily S3, or

[jira] [Commented] (SPARK-17593) list files on s3 very slow

[ https://issues.apache.org/jira/browse/SPARK-17593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15503207#comment-15503207 ] Gaurav Shah commented on SPARK-17593: - Thanks [~srowen] tried after your comment, but

[jira] [Commented] (SPARK-17593) list files on s3 very slow

[ https://issues.apache.org/jira/browse/SPARK-17593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15503164#comment-15503164 ] Gaurav Shah commented on SPARK-17593: - Thanks [~srowen] my spark code does use `s3n`

[jira] [Commented] (SPARK-17593) list files on s3 very slow

[ https://issues.apache.org/jira/browse/SPARK-17593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15503128#comment-15503128 ] Sean Owen commented on SPARK-17593: --- I'm not sure this is a Spark problem. It seems S3