Takeshi Yamamuro created SPARK-14193:
----------------------------------------
Summary: Skip unnecessary sorts if input data have been already
ordered in InMemoryRelation
Key: SPARK-14193
URL: https://issues.apache.org/jira/browse/SPARK-14193
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 1.6.1
Reporter: Takeshi Yamamuro
This pr is to skip unnecessary sorts if input data have been already ordered in
InMemoryTable.
Let's say we have a cached table with column 'a' sorted;
```
val df1 = Seq((1, 0), (3, 0), (2, 0), (1, 0)).toDF("a", "b")
val df2 = df1.sort("a").cache
df2.show // just cache data
```
If you say `df2.sort("a")`, the current spark generates a plan like;
```
== Physical Plan ==
Sort [a#13 ASC], true, 0
+- InMemoryColumnarTableScan [a#13,b#14], InMemoryRelation [a#13,b#14], true,
10000, StorageLevel(true, true, false, true, 1), Sort [a#13 ASC], true, 0, None
```
This pr removes this unncessary sort.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
