[jira] [Commented] (SPARK-16188) Spark sql create a lot of small files

2017-08-17 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131796#comment-16131796 ] cen yuhai commented on SPARK-16188: --- [~xianlongZhang] yes, you are right, I has impleme

[jira] [Commented] (SPARK-16188) Spark sql create a lot of small files

2017-08-17 Thread xianlongZhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131772#comment-16131772 ] xianlongZhang commented on SPARK-16188: --- cen yuhai,thanks for your advice, but my

[jira] [Commented] (SPARK-16188) Spark sql create a lot of small files

2017-08-16 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129815#comment-16129815 ] cen yuhai commented on SPARK-16188: --- [~xianlongZhang] You can use distribute by rand()

[jira] [Commented] (SPARK-16188) Spark sql create a lot of small files

2017-08-16 Thread xianlongZhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129742#comment-16129742 ] xianlongZhang commented on SPARK-16188: --- But when we use Spark sql, we can not call

[jira] [Commented] (SPARK-16188) Spark sql create a lot of small files

2016-06-26 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15350038#comment-15350038 ] cen yuhai commented on SPARK-16188: --- Found 4 items -rw-r--r-- 3 xiaoju supergroup

[jira] [Commented] (SPARK-16188) Spark sql create a lot of small files

2016-06-26 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15350035#comment-15350035 ] cen yuhai commented on SPARK-16188: --- Maybe SPARK-9858 can help me, I will try that.

[jira] [Commented] (SPARK-16188) Spark sql create a lot of small files

2016-06-26 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15350033#comment-15350033 ] Sean Owen commented on SPARK-16188: --- Just repartition/coalesce to fewer partitions firs

[jira] [Commented] (SPARK-16188) Spark sql create a lot of small files

2016-06-26 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15350032#comment-15350032 ] cen yuhai commented on SPARK-16188: --- [~srowen]en] already works? Can you provide me PR

[jira] [Commented] (SPARK-16188) Spark sql create a lot of small files

2016-06-25 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15350015#comment-15350015 ] Sean Owen commented on SPARK-16188: --- Generally the idea is to merge to fewer partitions

[jira] [Commented] (SPARK-16188) Spark sql create a lot of small files

2016-06-25 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15350013#comment-15350013 ] cen yuhai commented on SPARK-16188: --- [~hyukjin.kwon] It is not just empty files， but a

[jira] [Commented] (SPARK-16188) Spark sql create a lot of small files

2016-06-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15349581#comment-15349581 ] Hyukjin Kwon commented on SPARK-16188: -- Is this a duplicated of SPARK-10216 maybe?