[ https://issues.apache.org/jira/browse/SPARK-30735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17031099#comment-17031099 ]
Tomohiro Tanaka commented on SPARK-30735: ----------------------------------------- Hello, [~dongjoon]! Thanks for your checking this JIRA and PR. Sorry for my wrong inputs, and I understand guidelines about "versions" based on your comments. Also, I update the title of my PR from [SPARK-30735][CORE] to [SPARK-30735][SQL] (If it's no need to be changed, please let me know). Thank you very much for your kind help. > Improving writing performance by adding repartition based on columns to > partitionBy for DataFrameWriter > ------------------------------------------------------------------------------------------------------- > > Key: SPARK-30735 > URL: https://issues.apache.org/jira/browse/SPARK-30735 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 3.1.0 > Environment: * Spark-3.0.0 > * Scala: version 2.12.10 > * sbt 0.13.18, script ver: 1.3.7 (Built using sbt) > * Java: 1.8.0_231 > ** Java(TM) SE Runtime Environment (build 1.8.0_231-b11) > ** Java HotSpot(TM) 64-Bit Server VM (build 25.231-b11, mixed mode) > Reporter: Tomohiro Tanaka > Priority: Trivial > Labels: performance, pull-request-available > Attachments: repartition-before-partitionby.png > > Original Estimate: 336h > Remaining Estimate: 336h > > h1. New functionality for {{partitionBy}} > To enhance performance using partitionBy , calling {{repartition}} method > based on columns is much good before calling {{partitionBy}}. I added new > function: {color:#0747a6}{{partitionBy(<True | False>, columns>}}{color} to > {{partitionBy}}. > > h2. Problems when not using {{repartition}} before {{partitionBy}}. > When using {{paritionBy}}, following problems happen because of specified > columns in {{partitionBy}} are located separately. > * The spark application which includes {{partitionBy}} takes much longer > (for example, [[python - partitionBy taking too long while saving a dataset > on S3 using Pyspark - Stack > Overflow|https://stackoverflow.com/questions/56496387/partitionby-taking-too-long-while-saving-a-dataset-on-s3-using-pyspark]])] > * When using {{partitionBy}}, memory usage increases much high compared with > not using {{partitionBy}} (as follows I tested with Spark ver.2.4.3). > * Additional information about memory usage affection by partitionBy: Please > check the attachment (the left figure shows "using partitionBy", the other > shows "not using partitionBy)". > h2. How to use? > It's very simple. If you want to use repartition method before > {{partitionBy}}, just you specify {color:#0747a6}{{true}}{color} in > {{partitionBy}}. > Example: > {code:java} > val df = spark.read.format("csv").option("header", true).load(<INPUT_PATH>) > df.write.format("json").partitionBy(true, columns).save(<OUTPUT_PATH>){code} > > -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org