[ https://issues.apache.org/jira/browse/SPARK-14193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Takeshi Yamamuro updated SPARK-14193: ------------------------------------- Description: This ticket is to skip unnecessary sorts if input data have been already ordered in InMemoryTable. Let's say we have a cached table with column 'a' sorted; {code} val df1 = Seq((1, 0), (3, 0), (2, 0), (1, 0)).toDF("a", "b") val df2 = df1.sort("a").cache df2.show // just cache data {code} If you say `df2.sort("a")`, the current spark generates a plan like; {code} == Physical Plan == Sort [a#13 ASC], true, 0 +- InMemoryColumnarTableScan [a#13,b#14], InMemoryRelation [a#13,b#14], true, 10000, StorageLevel(true, true, false, true, 1), Sort [a#13 ASC], true, 0, None {code} This ticket targets at removing this unncessary sort. was: This ticket is to skip unnecessary sorts if input data have been already ordered in InMemoryTable. Let's say we have a cached table with column 'a' sorted; {code} val df1 = Seq((1, 0), (3, 0), (2, 0), (1, 0)).toDF("a", "b") val df2 = df1.sort("a").cache df2.show // just cache data {code} If you say `df2.sort("a")`, the current spark generates a plan like; {code} == Physical Plan == Sort [a#13 ASC], true, 0 +- InMemoryColumnarTableScan [a#13,b#14], InMemoryRelation [a#13,b#14], true, 10000, StorageLevel(true, true, false, true, 1), Sort [a#13 ASC], true, 0, None {code} This pr removes this unncessary sort. > Skip unnecessary sorts if input data have been already ordered in > InMemoryRelation > ---------------------------------------------------------------------------------- > > Key: SPARK-14193 > URL: https://issues.apache.org/jira/browse/SPARK-14193 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 1.6.1 > Reporter: Takeshi Yamamuro > > This ticket is to skip unnecessary sorts if input data have been already > ordered in InMemoryTable. > Let's say we have a cached table with column 'a' sorted; > {code} > val df1 = Seq((1, 0), (3, 0), (2, 0), (1, 0)).toDF("a", "b") > val df2 = df1.sort("a").cache > df2.show // just cache data > {code} > If you say `df2.sort("a")`, the current spark generates a plan like; > {code} > == Physical Plan == > Sort [a#13 ASC], true, 0 > +- InMemoryColumnarTableScan [a#13,b#14], InMemoryRelation [a#13,b#14], true, > 10000, StorageLevel(true, true, false, true, 1), Sort [a#13 ASC], true, 0, > None > {code} > This ticket targets at removing this unncessary sort. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org