[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...

2014-06-11 Thread ueshin
Github user ueshin commented on a diff in the pull request:

https://github.com/apache/spark/pull/990#discussion_r13635448
  
--- Diff: 
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantFoldingSuite.scala
 ---
@@ -173,4 +173,63 @@ class ConstantFoldingSuite extends OptimizerTest {
 
 comparePlans(optimized, correctAnswer)
   }
+
+  test(Constant folding test: expressions have null literals) {
+val originalQuery =
+  testRelation
+.select(
+  IsNull(Literal(null)) as 'c1,
+  IsNotNull(Literal(null)) as 'c2,
+
+  GetItem(Literal(null, ArrayType(IntegerType)), 1) as 'c3,
--- End diff --

Ah, I see, that's right!
I'll move them back.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...

2014-06-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/990#issuecomment-45706927
  
 Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...

2014-06-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/990#issuecomment-45706938
  
Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...

2014-06-11 Thread marmbrus
Github user marmbrus commented on the pull request:

https://github.com/apache/spark/pull/990#issuecomment-45785433
  
Hey, this isn't mergable anymore.  Mind rebasing?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...

2014-06-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/990#issuecomment-45813703
  
 Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...

2014-06-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/990#issuecomment-45813714
  
Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...

2014-06-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/990#issuecomment-45818717
  
All automated tests passed.
Refer to this link for build results: 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15693/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...

2014-06-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/990#issuecomment-45818716
  
Merged build finished. All automated tests passed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...

2014-06-11 Thread asfgit
Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/990


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...

2014-06-11 Thread marmbrus
Github user marmbrus commented on the pull request:

https://github.com/apache/spark/pull/990#issuecomment-45819003
  
Thanks!

Merged into master and 1.0.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...

2014-06-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/990#issuecomment-45582226
  
 Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...

2014-06-10 Thread ueshin
Github user ueshin commented on the pull request:

https://github.com/apache/spark/pull/990#issuecomment-45582389
  
@marmbrus I checked and remove the unneeded rules from `NullPropagation`.
Could you please check the changes?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...

2014-06-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/990#issuecomment-45583299
  

Refer to this link for build results: 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15606/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...

2014-06-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/990#issuecomment-45583276
  
Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...

2014-06-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/990#issuecomment-45583298
  
Merged build finished. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...

2014-06-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/990#issuecomment-45569478
  
 Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...

2014-06-09 Thread ueshin
Github user ueshin commented on a diff in the pull request:

https://github.com/apache/spark/pull/990#discussion_r13576056
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
 ---
@@ -154,6 +155,10 @@ object NullPropagation extends Rule[LogicalPlan] {
 case left :: Literal(null, _) :: Nil = Literal(null, e.dataType)
 case _ = e
   }
+  case e: CaseConversionExpression = e.children match {
--- End diff --

Fixed the two points.
Should I remove the mentioned lines?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...

2014-06-09 Thread marmbrus
Github user marmbrus commented on a diff in the pull request:

https://github.com/apache/spark/pull/990#discussion_r13576069
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
 ---
@@ -154,6 +155,10 @@ object NullPropagation extends Rule[LogicalPlan] {
 case left :: Literal(null, _) :: Nil = Literal(null, e.dataType)
 case _ = e
   }
+  case e: CaseConversionExpression = e.children match {
--- End diff --

Yes please.
On Jun 9, 2014 8:16 PM, Takuya UESHIN notificati...@github.com wrote:

 In
 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala:

  @@ -154,6 +155,10 @@ object NullPropagation extends Rule[LogicalPlan] {
   case left :: Literal(null, _) :: Nil = Literal(null, 
e.dataType)
   case _ = e
 }
  +  case e: CaseConversionExpression = e.children match {

 Fixed the two points.
 Should I remove the mentioned lines?

 —
 Reply to this email directly or view it on GitHub
 https://github.com/apache/spark/pull/990/files#r13576056.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...

2014-06-09 Thread ueshin
Github user ueshin commented on a diff in the pull request:

https://github.com/apache/spark/pull/990#discussion_r13576722
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
 ---
@@ -154,6 +155,10 @@ object NullPropagation extends Rule[LogicalPlan] {
 case left :: Literal(null, _) :: Nil = Literal(null, e.dataType)
 case _ = e
   }
+  case e: CaseConversionExpression = e.children match {
--- End diff --

Ok, I do.
BTW, I noticed that there are some Expressions in the same situation like 
`UnaryMinus`, `Cast`, or `Not`, ( or `BinaryArithmetic`, `BinaryComparison`, 
`StringRegexExpression`, too ? )
I think they also can be handled by constant folding.
What do you think?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...

2014-06-09 Thread marmbrus
Github user marmbrus commented on a diff in the pull request:

https://github.com/apache/spark/pull/990#discussion_r13577271
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
 ---
@@ -154,6 +155,10 @@ object NullPropagation extends Rule[LogicalPlan] {
 case left :: Literal(null, _) :: Nil = Literal(null, e.dataType)
 case _ = e
   }
+  case e: CaseConversionExpression = e.children match {
--- End diff --

Good catch! Can you double check that constant folding works as expected 
and then remove the unneeded rules?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...

2014-06-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/990#issuecomment-45574334
  
Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...

2014-06-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/990#issuecomment-45574335
  
Merged build finished. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...

2014-06-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/990#issuecomment-45574336
  

Refer to this link for build results: 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15597/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...

2014-06-09 Thread ueshin
Github user ueshin commented on a diff in the pull request:

https://github.com/apache/spark/pull/990#discussion_r13530167
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
 ---
@@ -288,3 +293,18 @@ object SimplifyCasts extends Rule[LogicalPlan] {
 case Cast(e, dataType) if e.dataType == dataType = e
   }
 }
+
+/**
+ * Removes the inner [[catalyst.expressions.CaseConversionExpression]] 
that are unnecessary because
+ * the inner conversion is overwritten by the outer one.
+ */
+object SimplifyCaseConversionExpressions extends Rule[LogicalPlan] {
+  def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+case q: LogicalPlan = q transformExpressionsUp {
--- End diff --

Hi,
I just cared that the conversions were used redundantly like 
`UPPER(LOWER(str))`, which would be a mistake in most case, though.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...

2014-06-09 Thread rxin
Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/990#discussion_r13530208
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
 ---
@@ -288,3 +293,18 @@ object SimplifyCasts extends Rule[LogicalPlan] {
 case Cast(e, dataType) if e.dataType == dataType = e
   }
 }
+
+/**
+ * Removes the inner [[catalyst.expressions.CaseConversionExpression]] 
that are unnecessary because
+ * the inner conversion is overwritten by the outer one.
+ */
+object SimplifyCaseConversionExpressions extends Rule[LogicalPlan] {
+  def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+case q: LogicalPlan = q transformExpressionsUp {
--- End diff --

Do you actually have queries that trigger this?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...

2014-06-09 Thread ueshin
Github user ueshin commented on a diff in the pull request:

https://github.com/apache/spark/pull/990#discussion_r13530241
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
 ---
@@ -288,3 +293,18 @@ object SimplifyCasts extends Rule[LogicalPlan] {
 case Cast(e, dataType) if e.dataType == dataType = e
   }
 }
+
+/**
+ * Removes the inner [[catalyst.expressions.CaseConversionExpression]] 
that are unnecessary because
+ * the inner conversion is overwritten by the outer one.
+ */
+object SimplifyCaseConversionExpressions extends Rule[LogicalPlan] {
+  def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+case q: LogicalPlan = q transformExpressionsUp {
--- End diff --

Just use `UPPER(LOWER(str))` or `UPPER(UPPER(str))`, or something like 
those.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...

2014-06-09 Thread rxin
Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/990#issuecomment-45463867
  
This looks good to me. Can you add some unit tests for collapsing case 
statements?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...

2014-06-09 Thread ueshin
Github user ueshin commented on the pull request:

https://github.com/apache/spark/pull/990#issuecomment-45465573
  
Thank you for your comments.
I added some tests.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...

2014-06-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/990#issuecomment-45465776
  
 Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...

2014-06-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/990#issuecomment-45469127
  
Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...

2014-06-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/990#issuecomment-45473517
  
All automated tests passed.
Refer to this link for build results: 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15561/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...

2014-06-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/990#issuecomment-45473514
  
Merged build finished. All automated tests passed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...

2014-06-09 Thread marmbrus
Github user marmbrus commented on a diff in the pull request:

https://github.com/apache/spark/pull/990#discussion_r13558814
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
 ---
@@ -154,6 +155,10 @@ object NullPropagation extends Rule[LogicalPlan] {
 case left :: Literal(null, _) :: Nil = Literal(null, e.dataType)
 case _ = e
   }
+  case e: CaseConversionExpression = e.children match {
--- End diff --

This case would be more generally handled by constant folding.  Can you 
override foldable in these expressions to be true if the child is foldable?  
Also please fix the toString methods.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...

2014-06-08 Thread rxin
Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/990#discussion_r13529691
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
 ---
@@ -288,3 +293,18 @@ object SimplifyCasts extends Rule[LogicalPlan] {
 case Cast(e, dataType) if e.dataType == dataType = e
   }
 }
+
+/**
+ * Removes the inner [[catalyst.expressions.CaseConversionExpression]] 
that are unnecessary because
+ * the inner conversion is overwritten by the outer one.
+ */
+object SimplifyCaseConversionExpressions extends Rule[LogicalPlan] {
+  def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+case q: LogicalPlan = q transformExpressionsUp {
--- End diff --

Just curious - do you have a use case for this rule? I wonder how often it 
happens in practice. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...

2014-06-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/990#issuecomment-45308169
  
Merged build finished. All automated tests passed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...

2014-06-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/990#issuecomment-45308172
  
All automated tests passed.
Refer to this link for build results: 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15495/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...

2014-06-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/990#issuecomment-45303026
  
 Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...

2014-06-05 Thread ueshin
GitHub user ueshin opened a pull request:

https://github.com/apache/spark/pull/990

[SPARK-2052] [SQL] Add optimization for CaseConversionExpression's.

Add optimization for `CaseConversionExpression`'s.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ueshin/apache-spark issues/SPARK-2052

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/990.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #990


commit 9e39abd79e3119a9721655a3652275b9b2a36b07
Author: Takuya UESHIN ues...@happy-camper.st
Date:   2014-06-06T04:46:05Z

Add optimization for CaseConversionExpression's.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...

2014-06-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/990#issuecomment-45304700
  
Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---