[jira] [Commented] (SPARK-21137) Spark reads many small files slowly

2017-06-27 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065423#comment-16065423 ] Steve Loughran commented on SPARK-21137: Looking at this. something is trying to

[jira] [Commented] (SPARK-21137) Spark reads many small files slowly

2017-06-27 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065428#comment-16065428 ] Steve Loughran commented on SPARK-21137: ps, for now, do it in parallel: {{mapre

[jira] [Commented] (SPARK-21137) Spark reads many small files slowly

2017-06-27 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065445#comment-16065445 ] Steve Loughran commented on SPARK-21137: Filed HADOOP-14600. Looks like a v. old

[jira] [Commented] (SPARK-21137) Spark reads many small files slowly

2017-06-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065476#comment-16065476 ] Apache Spark commented on SPARK-21137: -- User 'srowen' has created a pull request for

[jira] [Commented] (SPARK-21137) Spark reads many small files slowly

2017-06-27 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065475#comment-16065475 ] Sean Owen commented on SPARK-21137: --- OK, so it is something that could be optimized in

[jira] [Commented] (SPARK-21137) Spark reads many small files slowly

2017-06-27 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065515#comment-16065515 ] Steve Loughran commented on SPARK-21137: bq. so it is something that could be opt