[jira] [Commented] (SPARK-17998) Reading Parquet files coalesces parts into too few in-memory partitions

2019-01-31 Thread Nicholas Resnick (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16757326#comment-16757326 ] Nicholas Resnick commented on SPARK-17998: -- Going to answer my question: it is in fact a

[jira] [Commented] (SPARK-17998) Reading Parquet files coalesces parts into too few in-memory partitions

2019-01-30 Thread Nicholas Resnick (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756801#comment-16756801 ] Nicholas Resnick commented on SPARK-17998: -- I reproduced the OP's steps above on my local

[jira] [Commented] (SPARK-17998) Reading Parquet files coalesces parts into too few in-memory partitions

2018-01-15 Thread Fernando Pereira (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326195#comment-16326195 ] Fernando Pereira commented on SPARK-17998: -- [~sams] Did you have the change to check

[jira] [Commented] (SPARK-17998) Reading Parquet files coalesces parts into too few in-memory partitions

2018-01-10 Thread Fernando Pereira (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16321234#comment-16321234 ] Fernando Pereira commented on SPARK-17998: -- It says spark.sql.files.maxPartitionBytes in this

[jira] [Commented] (SPARK-17998) Reading Parquet files coalesces parts into too few in-memory partitions

2018-01-10 Thread sam (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16320041#comment-16320041 ] sam commented on SPARK-17998: - [~srowen] Thanks, no idea where I got that from, cursed weakly typed silently

[jira] [Commented] (SPARK-17998) Reading Parquet files coalesces parts into too few in-memory partitions

2018-01-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318684#comment-16318684 ] Sean Owen commented on SPARK-17998: --- [~ferdonline] I checked the code and spark.files.maxPartitionBytes

[jira] [Commented] (SPARK-17998) Reading Parquet files coalesces parts into too few in-memory partitions

2018-01-09 Thread Fernando Pereira (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318243#comment-16318243 ] Fernando Pereira commented on SPARK-17998: -- The documentation

[jira] [Commented] (SPARK-17998) Reading Parquet files coalesces parts into too few in-memory partitions

2017-10-13 Thread sam (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203877#comment-16203877 ] sam commented on SPARK-17998: - [~lwlin] I think this is a regression. We used to be able to easily control

[jira] [Commented] (SPARK-17998) Reading Parquet files coalesces parts into too few in-memory partitions

2016-10-19 Thread Shea Parkes (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590680#comment-15590680 ] Shea Parkes commented on SPARK-17998: - That definitely answers it. I would say the default of 128MB

[jira] [Commented] (SPARK-17998) Reading Parquet files coalesces parts into too few in-memory partitions

2016-10-19 Thread Liwei Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590675#comment-15590675 ] Liwei Lin commented on SPARK-17998: --- Hi [~shea.parkes], for your case, the number is determined at