[jira] [Updated] (SPARK-15076) Add ReorderAssociativeOperator optimizer
[
https://issues.apache.org/jira/browse/SPARK-15076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-15076:
--
Component/s: SQL
> Add ReorderAssociativeOperator optimizer
>
>
> Key: SPARK-15076
> URL: https://issues.apache.org/jira/browse/SPARK-15076
> Project: Spark
> Issue Type: Improvement
> Components: Optimizer, SQL
>Reporter: Dongjoon Hyun
>Assignee: Dongjoon Hyun
> Fix For: 2.1.0
>
>
> This issue add a new optimizer `ReorderAssociativeOperator` by taking
> advantage of integral associative property. Currently, Spark works like the
> following.
> 1) Can optimize `1 + 2 + 3 + 4 + 5 + 6 + 7 + 8 + 9 + a` into `45 + a`.
> 2) Cannot optimize `a + 1 + 2 + 3 + 4 + 5 + 6 + 7 + 8 + 9`.
> This issue can handle Case 2 for *Add/Multiply* expression whose data types
> are `ByteType`, `ShortType`, `IntegerType`, and `LongType`. The followings
> are the plan comparison between `before` and `after` this issue.
> **Before**
> {code}
> scala> sql("select a+1+2+3+4+5+6+7+8+9 from (select explode(array(1))
> a)").explain
> == Physical Plan ==
> WholeStageCodegen
> : +- Project [(a#7 + 1) + 2) + 3) + 4) + 5) + 6) + 7) + 8) + 9) AS
> (a + 1) + 2) + 3) + 4) + 5) + 6) + 7) + 8) + 9)#8]
> : +- INPUT
> +- Generate explode([1]), false, false, [a#7]
>+- Scan OneRowRelation[]
> {code}
> **After**
> {code}
> scala> sql("select a+1+2+3+4+5+6+7+8+9 from (select explode(array(1))
> a)").explain
> == Physical Plan ==
> WholeStageCodegen
> : +- Project [(a#7 + 45) AS (a + 1) + 2) + 3) + 4) + 5) + 6) + 7) +
> 8) + 9)#8]
> : +- INPUT
> +- Generate explode([1]), false, false, [a#7]
>+- Scan OneRowRelation[]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-15076) Add ReorderAssociativeOperator optimizer
[
https://issues.apache.org/jira/browse/SPARK-15076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenchen Fan updated SPARK-15076:
Assignee: Dongjoon Hyun
> Add ReorderAssociativeOperator optimizer
>
>
> Key: SPARK-15076
> URL: https://issues.apache.org/jira/browse/SPARK-15076
> Project: Spark
> Issue Type: Improvement
> Components: Optimizer
>Reporter: Dongjoon Hyun
>Assignee: Dongjoon Hyun
> Fix For: 2.1.0
>
>
> This issue add a new optimizer `ReorderAssociativeOperator` by taking
> advantage of integral associative property. Currently, Spark works like the
> following.
> 1) Can optimize `1 + 2 + 3 + 4 + 5 + 6 + 7 + 8 + 9 + a` into `45 + a`.
> 2) Cannot optimize `a + 1 + 2 + 3 + 4 + 5 + 6 + 7 + 8 + 9`.
> This issue can handle Case 2 for *Add/Multiply* expression whose data types
> are `ByteType`, `ShortType`, `IntegerType`, and `LongType`. The followings
> are the plan comparison between `before` and `after` this issue.
> **Before**
> {code}
> scala> sql("select a+1+2+3+4+5+6+7+8+9 from (select explode(array(1))
> a)").explain
> == Physical Plan ==
> WholeStageCodegen
> : +- Project [(a#7 + 1) + 2) + 3) + 4) + 5) + 6) + 7) + 8) + 9) AS
> (a + 1) + 2) + 3) + 4) + 5) + 6) + 7) + 8) + 9)#8]
> : +- INPUT
> +- Generate explode([1]), false, false, [a#7]
>+- Scan OneRowRelation[]
> {code}
> **After**
> {code}
> scala> sql("select a+1+2+3+4+5+6+7+8+9 from (select explode(array(1))
> a)").explain
> == Physical Plan ==
> WholeStageCodegen
> : +- Project [(a#7 + 45) AS (a + 1) + 2) + 3) + 4) + 5) + 6) + 7) +
> 8) + 9)#8]
> : +- INPUT
> +- Generate explode([1]), false, false, [a#7]
>+- Scan OneRowRelation[]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-15076) Add ReorderAssociativeOperator optimizer
[
https://issues.apache.org/jira/browse/SPARK-15076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-15076:
--
Description:
This issue add a new optimizer `ReorderAssociativeOperator` by taking
advantage of integral associative property. Currently, Spark works like the
following.
1) Can optimize `1 + 2 + 3 + 4 + 5 + 6 + 7 + 8 + 9 + a` into `45 + a`.
2) Cannot optimize `a + 1 + 2 + 3 + 4 + 5 + 6 + 7 + 8 + 9`.
This issue can handle Case 2 for *Add/Multiply* expression whose data types are
`ByteType`, `ShortType`, `IntegerType`, and `LongType`. The followings are the
plan comparison between `before` and `after` this issue.
**Before**
{code}
scala> sql("select a+1+2+3+4+5+6+7+8+9 from (select explode(array(1))
a)").explain
== Physical Plan ==
WholeStageCodegen
: +- Project [(a#7 + 1) + 2) + 3) + 4) + 5) + 6) + 7) + 8) + 9) AS
(a + 1) + 2) + 3) + 4) + 5) + 6) + 7) + 8) + 9)#8]
: +- INPUT
+- Generate explode([1]), false, false, [a#7]
+- Scan OneRowRelation[]
{code}
**After**
{code}
scala> sql("select a+1+2+3+4+5+6+7+8+9 from (select explode(array(1))
a)").explain
== Physical Plan ==
WholeStageCodegen
: +- Project [(a#7 + 45) AS (a + 1) + 2) + 3) + 4) + 5) + 6) + 7) + 8)
+ 9)#8]
: +- INPUT
+- Generate explode([1]), false, false, [a#7]
+- Scan OneRowRelation[]
{code}
was:
This issue add a new optimizer `ConstantFolding` by taking advantage of
integral associative property. Currently, `ConstantFolding` works like the
following.
1) Can optimize `1 + 2 + 3 + 4 + 5 + 6 + 7 + 8 + 9 + a` into `45 + a`.
2) Cannot optimize `a + 1 + 2 + 3 + 4 + 5 + 6 + 7 + 8 + 9`.
This issue enables `ConstantFolding` to handle Case 2 for *Add/Multiply*
expression whose data types are `ByteType`, `ShortType`, `IntegerType`, and
`LongType`. The followings are the plan comparision between `before` and
`after` this issue.
**Before**
{code}
scala> sql("select a+1+2+3+4+5+6+7+8+9 from (select explode(array(1))
a)").explain
== Physical Plan ==
WholeStageCodegen
: +- Project [(a#7 + 1) + 2) + 3) + 4) + 5) + 6) + 7) + 8) + 9) AS
(a + 1) + 2) + 3) + 4) + 5) + 6) + 7) + 8) + 9)#8]
: +- INPUT
+- Generate explode([1]), false, false, [a#7]
+- Scan OneRowRelation[]
{code}
**After**
{code}
scala> sql("select a+1+2+3+4+5+6+7+8+9 from (select explode(array(1))
a)").explain
== Physical Plan ==
WholeStageCodegen
: +- Project [(a#7 + 45) AS (a + 1) + 2) + 3) + 4) + 5) + 6) + 7) + 8)
+ 9)#8]
: +- INPUT
+- Generate explode([1]), false, false, [a#7]
+- Scan OneRowRelation[]
{code}
> Add ReorderAssociativeOperator optimizer
>
>
> Key: SPARK-15076
> URL: https://issues.apache.org/jira/browse/SPARK-15076
> Project: Spark
> Issue Type: Improvement
> Components: Optimizer
>Reporter: Dongjoon Hyun
>
> This issue add a new optimizer `ReorderAssociativeOperator` by taking
> advantage of integral associative property. Currently, Spark works like the
> following.
> 1) Can optimize `1 + 2 + 3 + 4 + 5 + 6 + 7 + 8 + 9 + a` into `45 + a`.
> 2) Cannot optimize `a + 1 + 2 + 3 + 4 + 5 + 6 + 7 + 8 + 9`.
> This issue can handle Case 2 for *Add/Multiply* expression whose data types
> are `ByteType`, `ShortType`, `IntegerType`, and `LongType`. The followings
> are the plan comparison between `before` and `after` this issue.
> **Before**
> {code}
> scala> sql("select a+1+2+3+4+5+6+7+8+9 from (select explode(array(1))
> a)").explain
> == Physical Plan ==
> WholeStageCodegen
> : +- Project [(a#7 + 1) + 2) + 3) + 4) + 5) + 6) + 7) + 8) + 9) AS
> (a + 1) + 2) + 3) + 4) + 5) + 6) + 7) + 8) + 9)#8]
> : +- INPUT
> +- Generate explode([1]), false, false, [a#7]
>+- Scan OneRowRelation[]
> {code}
> **After**
> {code}
> scala> sql("select a+1+2+3+4+5+6+7+8+9 from (select explode(array(1))
> a)").explain
> == Physical Plan ==
> WholeStageCodegen
> : +- Project [(a#7 + 45) AS (a + 1) + 2) + 3) + 4) + 5) + 6) + 7) +
> 8) + 9)#8]
> : +- INPUT
> +- Generate explode([1]), false, false, [a#7]
>+- Scan OneRowRelation[]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-15076) Add ReorderAssociativeOperator optimizer
[
https://issues.apache.org/jira/browse/SPARK-15076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-15076:
--
Summary: Add ReorderAssociativeOperator optimizer (was: Improve
ConstantFolding optimizer to use integral associative property)
> Add ReorderAssociativeOperator optimizer
>
>
> Key: SPARK-15076
> URL: https://issues.apache.org/jira/browse/SPARK-15076
> Project: Spark
> Issue Type: Improvement
> Components: Optimizer
>Reporter: Dongjoon Hyun
>
> This issue improves `ConstantFolding` optimizer by taking advantage of
> integral associative property. Currently, `ConstantFolding` works like the
> following.
> 1) Can optimize `1 + 2 + 3 + 4 + 5 + 6 + 7 + 8 + 9 + a` into `45 + a`.
> 2) Cannot optimize `a + 1 + 2 + 3 + 4 + 5 + 6 + 7 + 8 + 9`.
> This issue enables `ConstantFolding` to handle Case 2 for *Add/Multiply*
> expression whose data types are `ByteType`, `ShortType`, `IntegerType`, and
> `LongType`. The followings are the plan comparision between `before` and
> `after` this issue.
> **Before**
> {code}
> scala> sql("select a+1+2+3+4+5+6+7+8+9 from (select explode(array(1))
> a)").explain
> == Physical Plan ==
> WholeStageCodegen
> : +- Project [(a#7 + 1) + 2) + 3) + 4) + 5) + 6) + 7) + 8) + 9) AS
> (a + 1) + 2) + 3) + 4) + 5) + 6) + 7) + 8) + 9)#8]
> : +- INPUT
> +- Generate explode([1]), false, false, [a#7]
>+- Scan OneRowRelation[]
> {code}
> **After**
> {code}
> scala> sql("select a+1+2+3+4+5+6+7+8+9 from (select explode(array(1))
> a)").explain
> == Physical Plan ==
> WholeStageCodegen
> : +- Project [(a#7 + 45) AS (a + 1) + 2) + 3) + 4) + 5) + 6) + 7) +
> 8) + 9)#8]
> : +- INPUT
> +- Generate explode([1]), false, false, [a#7]
>+- Scan OneRowRelation[]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-15076) Add ReorderAssociativeOperator optimizer
[
https://issues.apache.org/jira/browse/SPARK-15076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-15076:
--
Description:
This issue add a new optimizer `ConstantFolding` by taking advantage of
integral associative property. Currently, `ConstantFolding` works like the
following.
1) Can optimize `1 + 2 + 3 + 4 + 5 + 6 + 7 + 8 + 9 + a` into `45 + a`.
2) Cannot optimize `a + 1 + 2 + 3 + 4 + 5 + 6 + 7 + 8 + 9`.
This issue enables `ConstantFolding` to handle Case 2 for *Add/Multiply*
expression whose data types are `ByteType`, `ShortType`, `IntegerType`, and
`LongType`. The followings are the plan comparision between `before` and
`after` this issue.
**Before**
{code}
scala> sql("select a+1+2+3+4+5+6+7+8+9 from (select explode(array(1))
a)").explain
== Physical Plan ==
WholeStageCodegen
: +- Project [(a#7 + 1) + 2) + 3) + 4) + 5) + 6) + 7) + 8) + 9) AS
(a + 1) + 2) + 3) + 4) + 5) + 6) + 7) + 8) + 9)#8]
: +- INPUT
+- Generate explode([1]), false, false, [a#7]
+- Scan OneRowRelation[]
{code}
**After**
{code}
scala> sql("select a+1+2+3+4+5+6+7+8+9 from (select explode(array(1))
a)").explain
== Physical Plan ==
WholeStageCodegen
: +- Project [(a#7 + 45) AS (a + 1) + 2) + 3) + 4) + 5) + 6) + 7) + 8)
+ 9)#8]
: +- INPUT
+- Generate explode([1]), false, false, [a#7]
+- Scan OneRowRelation[]
{code}
was:
This issue improves `ConstantFolding` optimizer by taking advantage of integral
associative property. Currently, `ConstantFolding` works like the following.
1) Can optimize `1 + 2 + 3 + 4 + 5 + 6 + 7 + 8 + 9 + a` into `45 + a`.
2) Cannot optimize `a + 1 + 2 + 3 + 4 + 5 + 6 + 7 + 8 + 9`.
This issue enables `ConstantFolding` to handle Case 2 for *Add/Multiply*
expression whose data types are `ByteType`, `ShortType`, `IntegerType`, and
`LongType`. The followings are the plan comparision between `before` and
`after` this issue.
**Before**
{code}
scala> sql("select a+1+2+3+4+5+6+7+8+9 from (select explode(array(1))
a)").explain
== Physical Plan ==
WholeStageCodegen
: +- Project [(a#7 + 1) + 2) + 3) + 4) + 5) + 6) + 7) + 8) + 9) AS
(a + 1) + 2) + 3) + 4) + 5) + 6) + 7) + 8) + 9)#8]
: +- INPUT
+- Generate explode([1]), false, false, [a#7]
+- Scan OneRowRelation[]
{code}
**After**
{code}
scala> sql("select a+1+2+3+4+5+6+7+8+9 from (select explode(array(1))
a)").explain
== Physical Plan ==
WholeStageCodegen
: +- Project [(a#7 + 45) AS (a + 1) + 2) + 3) + 4) + 5) + 6) + 7) + 8)
+ 9)#8]
: +- INPUT
+- Generate explode([1]), false, false, [a#7]
+- Scan OneRowRelation[]
{code}
> Add ReorderAssociativeOperator optimizer
>
>
> Key: SPARK-15076
> URL: https://issues.apache.org/jira/browse/SPARK-15076
> Project: Spark
> Issue Type: Improvement
> Components: Optimizer
>Reporter: Dongjoon Hyun
>
> This issue add a new optimizer `ConstantFolding` by taking advantage of
> integral associative property. Currently, `ConstantFolding` works like the
> following.
> 1) Can optimize `1 + 2 + 3 + 4 + 5 + 6 + 7 + 8 + 9 + a` into `45 + a`.
> 2) Cannot optimize `a + 1 + 2 + 3 + 4 + 5 + 6 + 7 + 8 + 9`.
> This issue enables `ConstantFolding` to handle Case 2 for *Add/Multiply*
> expression whose data types are `ByteType`, `ShortType`, `IntegerType`, and
> `LongType`. The followings are the plan comparision between `before` and
> `after` this issue.
> **Before**
> {code}
> scala> sql("select a+1+2+3+4+5+6+7+8+9 from (select explode(array(1))
> a)").explain
> == Physical Plan ==
> WholeStageCodegen
> : +- Project [(a#7 + 1) + 2) + 3) + 4) + 5) + 6) + 7) + 8) + 9) AS
> (a + 1) + 2) + 3) + 4) + 5) + 6) + 7) + 8) + 9)#8]
> : +- INPUT
> +- Generate explode([1]), false, false, [a#7]
>+- Scan OneRowRelation[]
> {code}
> **After**
> {code}
> scala> sql("select a+1+2+3+4+5+6+7+8+9 from (select explode(array(1))
> a)").explain
> == Physical Plan ==
> WholeStageCodegen
> : +- Project [(a#7 + 45) AS (a + 1) + 2) + 3) + 4) + 5) + 6) + 7) +
> 8) + 9)#8]
> : +- INPUT
> +- Generate explode([1]), false, false, [a#7]
>+- Scan OneRowRelation[]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
