Johannes Mayer created SPARK-23771:
--------------------------------------
Summary: Uneven Rowgroup size after repartition
Key: SPARK-23771
URL: https://issues.apache.org/jira/browse/SPARK-23771
Project: Spark
Issue Type: Bug
Components: Input/Output, Shuffle, SQL
Affects Versions: 1.6.0
Environment: Cloudera CDH 5.13.1
Reporter: Johannes Mayer
I have a Hive table on AVRO files, that i want to read and store as a
partitioned parquet files (one file per partition).
What i do is:
{code:java}
// read the AVRO table and distribute by the partition column
val data = sql("select * from avro_table distribute by part_col")
// write data as partitioned parquet files
data.write.partitionBy(part_col).parquet("output/path/")
{code}
I get one file per partition as expected. But often I run into OutOfMemory
Errors. Investigating the issue I found out, that some row groups are very big
and since all data of a row group is held in memory before it is flushed to
disk, i think this causes the OutOfMemory. Other row groups are very small,
containing almost no data. See the output from parquet-tools meta:
{code:java}
row group 1: RC:5740100 TS:566954562 OFFSET:4
row group 2: RC:33769 TS:2904145 OFFSET:117971092
row group 3: RC:31822 TS:2772650 OFFSET:118905225
row group 4: RC:29854 TS:2704127 OFFSET:119793188
row group 5: RC:28050 TS:2356729 OFFSET:120660675
row group 6: RC:26507 TS:2111983 OFFSET:121406541
row group 7: RC:25143 TS:1967731 OFFSET:122069351
row group 8: RC:23876 TS:1991238 OFFSET:122682160
row group 9: RC:22584 TS:2069463 OFFSET:123303246
row group 10: RC:21225 TS:1955748 OFFSET:123960700
row group 11: RC:19960 TS:1931889 OFFSET:124575333
row group 12: RC:18806 TS:1725871 OFFSET:125132862
row group 13: RC:17719 TS:1653309 OFFSET:125668057
row group 14: RC:1617743 TS:157973949 OFFSET:134217728{code}
One thing to notice is, that this file was written in a Spark application
running on 13 executors. Is it possible, that local data is in the big row
group and the remote reads go into seperate (small) row groups? The shuffle is
involved, because data is read with distribute by clause.
Is this a known bug? Is there a workaround to get even row group sizes? I want
to decrease the row group size using
sc.hadoopConfiguration.setInt("parquet.block.size", 64 * 1024 * 1024)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
