[
https://issues.apache.org/jira/browse/SPARK-30735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tomohiro Tanaka updated SPARK-30735:
------------------------------------
Description:
h1. New functionality for {{partitionBy}}
To enhance performance using partitionBy , calling {{repartition}} method based
on columns is much good before calling {{partitionBy}}. I added new function:
{color:#0747a6}{{partitionBy(<True | False>, columns>}}{color} to
{{partitionBy}}.
h2. Problems when not using {{repartition}} before {{partitionBy}}.
When using {{paritionBy}}, following problems happen because of specified
columns in {{partitionBy}} are located separately.
* The spark application which includes {{partitionBy}} takes much longer (for
example, [[python - partitionBy taking too long while saving a dataset on S3
using Pyspark - Stack
Overflow|https://stackoverflow.com/questions/56496387/partitionby-taking-too-long-while-saving-a-dataset-on-s3-using-pyspark]])]
* When using {{partitionBy}}, memory usage increases much high compared with
not using {{partitionBy}} (as follows I tested with Spark ver.2.4.3). Please
check the attachment (the left figure shows "not using repartition based on
columns before partitionBy", the other shows "using repartition".
h2. How to use?
It's very simple. If you want to use repartition method before {{partitionBy}},
just you specify {color:#0747a6}{{true}}{color} in {{partitionBy}}.
Example:
{code:java}
val df = spark.read.format("csv").option("header", true).load(<INPUT_PATH>)
df.write.format("json").partitionBy(true, columns).save(<OUTPUT_PATH>){code}
was:
h1. New functionality for {{partitionBy}}
To enhance performance using partitionBy , calling {{repartition}} method based
on columns is much good before calling {{partitionBy}}. I added new function:
{color:#0747a6}{{partitionBy(<True | False>, columns>}}{color} to
{{partitionBy}}.
h2. Problems when not using {{repartition}} before {{partitionBy}}.
When using {{paritionBy}}, following problems happen because of specified
columns in {{partitionBy}} are located separately.
* The spark application which includes {{partitionBy}} takes much longer (for
example, [[python - partitionBy taking too long while saving a dataset on S3
using Pyspark - Stack
Overflow|https://stackoverflow.com/questions/56496387/partitionby-taking-too-long-while-saving-a-dataset-on-s3-using-pyspark]|https://stackoverflow.com/questions/56496387/partitionby-taking-too-long-while-saving-a-dataset-on-s3-using-pyspark])]
* When using {{partitionBy}}, memory usage increases much high compared with
not using {{partitionBy}} (as follows I tested with Spark ver.2.4.3).
** Not using repartition before partitionBy:
** Using repartition before partitionBy
h2. How to use?
It's very simple. If you want to use repartition method before {{partitionBy}},
just you specify {color:#0747a6}{{true}}{color} in {{partitionBy}}.
Example:
{code:java}
val df = spark.read.format("csv").option("header", true).load(<INPUT_PATH>)
df.write.format("json").partitionBy(true, columns).save(<OUTPUT_PATH>){code}
> Improving writing performance by adding repartition based on columns to
> partitionBy for DataFrameWriter
> -------------------------------------------------------------------------------------------------------
>
> Key: SPARK-30735
> URL: https://issues.apache.org/jira/browse/SPARK-30735
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.4.3, 2.4.4
> Environment: * Spark-3.0.0
> * Scala: version 2.12.10
> * sbt 0.13.18, script ver: 1.3.7 (Built using sbt)
> * Java: 1.8.0_231
> ** Java(TM) SE Runtime Environment (build 1.8.0_231-b11)
> ** Java HotSpot(TM) 64-Bit Server VM (build 25.231-b11, mixed mode)
> Reporter: Tomohiro Tanaka
> Priority: Trivial
> Labels: performance, pull-request-available
> Fix For: 3.0.0, 3.1.0
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> h1. New functionality for {{partitionBy}}
> To enhance performance using partitionBy , calling {{repartition}} method
> based on columns is much good before calling {{partitionBy}}. I added new
> function: {color:#0747a6}{{partitionBy(<True | False>, columns>}}{color} to
> {{partitionBy}}.
>
> h2. Problems when not using {{repartition}} before {{partitionBy}}.
> When using {{paritionBy}}, following problems happen because of specified
> columns in {{partitionBy}} are located separately.
> * The spark application which includes {{partitionBy}} takes much longer
> (for example, [[python - partitionBy taking too long while saving a dataset
> on S3 using Pyspark - Stack
> Overflow|https://stackoverflow.com/questions/56496387/partitionby-taking-too-long-while-saving-a-dataset-on-s3-using-pyspark]])]
> * When using {{partitionBy}}, memory usage increases much high compared with
> not using {{partitionBy}} (as follows I tested with Spark ver.2.4.3). Please
> check the attachment (the left figure shows "not using repartition based on
> columns before partitionBy", the other shows "using repartition".
> h2. How to use?
> It's very simple. If you want to use repartition method before
> {{partitionBy}}, just you specify {color:#0747a6}{{true}}{color} in
> {{partitionBy}}.
> Example:
> {code:java}
> val df = spark.read.format("csv").option("header", true).load(<INPUT_PATH>)
> df.write.format("json").partitionBy(true, columns).save(<OUTPUT_PATH>){code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
