GitHub user cloud-fan opened a pull request:
https://github.com/apache/spark/pull/16898
[SPARK-19563][SQL] advoid unnecessary sort in FileFormatWriter
## What changes were proposed in this pull request?
In `FileFormatWriter`, we will sort the input rows by partition columns and
bucket id and sort columns, if we want to write data out partitioned or
bucketed.
However, if the data is already sorted, we will sort it again, which is
unnecssary.
This PR removes the sorting logic in `FileFormatWriter` and use `SortExec`
instead. We will not add `SortExec` if the data is already sorted.
## How was this patch tested?
I did a micro benchmark manually
```
val df = spark.range(10000000).select($"id", $"id" % 10 as
"part").sort("part")
spark.time(df.write.partitionBy("part").parquet("/tmp/test"))
```
The result was about 6.4 seconds before this PR, and is 5.7 seconds
afterwards.
close https://github.com/apache/spark/pull/16724
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cloud-fan/spark writer
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16898.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16898
----
commit 602a1e48f2ca0c8286601a35c40b0e0f23331f23
Author: Wenchen Fan <wenc...@databricks.com>
Date: 2017-02-12T09:01:53Z
advoid unnecessary sort in FileFormatWriter
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
