[jira] [Commented] (PIG-4128) New logical optimizer rule: ConstantCalculator
[ https://issues.apache.org/jira/browse/PIG-4128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14138394#comment-14138394 ] Daniel Dai commented on PIG-4128: - Thanks Cheolsoo, really glad to know this! > New logical optimizer rule: ConstantCalculator > -- > > Key: PIG-4128 > URL: https://issues.apache.org/jira/browse/PIG-4128 > Project: Pig > Issue Type: New Feature > Components: impl >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 0.14.0 > > Attachments: PIG-4128-1.patch, PIG-4128-2.patch, PIG-4128-3.patch > > > Pig used to have a LogicExpressionSimplifier to simplify expression which > also calculates constant expression. The optimizer rule is buggy and we > disable it by default in PIG-2316. > However, we do need this feature especially in partition/predicate push down, > since both does not deal with complex constant expression, we'd like to > replace the expression with constant before the actual push down. Yes, user > may manually do the calculation and rewrite the query, but even rewrite is > sometimes not possible. Consider the case user want to push a datetime > predicate, user have to write a ToDate udf since Pig does not have datetime > constant. > In this Jira, I provide a new rule: ConstantCalculator, which is much simpler > and much less error prone, to replace LogicExpressionSimplifier. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PIG-4128) New logical optimizer rule: ConstantCalculator
[ https://issues.apache.org/jira/browse/PIG-4128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136539#comment-14136539 ] Cheolsoo Park commented on PIG-4128: [~daijy], I want to say thank you for this patch. One of common mistakes that I see is that partition column is long type, and user specifies a filter expression like {{ == }}. The partition used to be not pushed down in this case because Pig inserts a cast expression into the filer expression, and this was always confusing to users. But with your patch, this filter expression just works. > New logical optimizer rule: ConstantCalculator > -- > > Key: PIG-4128 > URL: https://issues.apache.org/jira/browse/PIG-4128 > Project: Pig > Issue Type: New Feature > Components: impl >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 0.14.0 > > Attachments: PIG-4128-1.patch, PIG-4128-2.patch, PIG-4128-3.patch > > > Pig used to have a LogicExpressionSimplifier to simplify expression which > also calculates constant expression. The optimizer rule is buggy and we > disable it by default in PIG-2316. > However, we do need this feature especially in partition/predicate push down, > since both does not deal with complex constant expression, we'd like to > replace the expression with constant before the actual push down. Yes, user > may manually do the calculation and rewrite the query, but even rewrite is > sometimes not possible. Consider the case user want to push a datetime > predicate, user have to write a ToDate udf since Pig does not have datetime > constant. > In this Jira, I provide a new rule: ConstantCalculator, which is much simpler > and much less error prone, to replace LogicExpressionSimplifier. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PIG-4128) New logical optimizer rule: ConstantCalculator
[ https://issues.apache.org/jira/browse/PIG-4128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14112498#comment-14112498 ] Daniel Dai commented on PIG-4128: - Oh, yes, I delete it because it is empty. I bring it back. > New logical optimizer rule: ConstantCalculator > -- > > Key: PIG-4128 > URL: https://issues.apache.org/jira/browse/PIG-4128 > Project: Pig > Issue Type: New Feature > Components: impl >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 0.14.0 > > Attachments: PIG-4128-1.patch, PIG-4128-2.patch, PIG-4128-3.patch > > > Pig used to have a LogicExpressionSimplifier to simplify expression which > also calculates constant expression. The optimizer rule is buggy and we > disable it by default in PIG-2316. > However, we do need this feature especially in partition/predicate push down, > since both does not deal with complex constant expression, we'd like to > replace the expression with constant before the actual push down. Yes, user > may manually do the calculation and rewrite the query, but even rewrite is > sometimes not possible. Consider the case user want to push a datetime > predicate, user have to write a ToDate udf since Pig does not have datetime > constant. > In this Jira, I provide a new rule: ConstantCalculator, which is much simpler > and much less error prone, to replace LogicExpressionSimplifier. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (PIG-4128) New logical optimizer rule: ConstantCalculator
[ https://issues.apache.org/jira/browse/PIG-4128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14112366#comment-14112366 ] Cheolsoo Park commented on PIG-4128: [~daijy], did you delete {{excluded-tests-23}} intentionally? It broke the build- {code} /home/jenkins/jenkins-slave/workspace/Pig-trunk/build.xml:905: Excludesfile /home/jenkins/jenkins-slave/workspace/Pig-trunk/test/excluded-tests-23 not found. {code} https://builds.apache.org/job/Pig-trunk/1639/console > New logical optimizer rule: ConstantCalculator > -- > > Key: PIG-4128 > URL: https://issues.apache.org/jira/browse/PIG-4128 > Project: Pig > Issue Type: New Feature > Components: impl >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 0.14.0 > > Attachments: PIG-4128-1.patch, PIG-4128-2.patch, PIG-4128-3.patch > > > Pig used to have a LogicExpressionSimplifier to simplify expression which > also calculates constant expression. The optimizer rule is buggy and we > disable it by default in PIG-2316. > However, we do need this feature especially in partition/predicate push down, > since both does not deal with complex constant expression, we'd like to > replace the expression with constant before the actual push down. Yes, user > may manually do the calculation and rewrite the query, but even rewrite is > sometimes not possible. Consider the case user want to push a datetime > predicate, user have to write a ToDate udf since Pig does not have datetime > constant. > In this Jira, I provide a new rule: ConstantCalculator, which is much simpler > and much less error prone, to replace LogicExpressionSimplifier. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (PIG-4128) New logical optimizer rule: ConstantCalculator
[ https://issues.apache.org/jira/browse/PIG-4128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14111505#comment-14111505 ] Thejas M Nair commented on PIG-4128: +1 > New logical optimizer rule: ConstantCalculator > -- > > Key: PIG-4128 > URL: https://issues.apache.org/jira/browse/PIG-4128 > Project: Pig > Issue Type: New Feature > Components: impl >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 0.14.0 > > Attachments: PIG-4128-1.patch, PIG-4128-2.patch, PIG-4128-3.patch > > > Pig used to have a LogicExpressionSimplifier to simplify expression which > also calculates constant expression. The optimizer rule is buggy and we > disable it by default in PIG-2316. > However, we do need this feature especially in partition/predicate push down, > since both does not deal with complex constant expression, we'd like to > replace the expression with constant before the actual push down. Yes, user > may manually do the calculation and rewrite the query, but even rewrite is > sometimes not possible. Consider the case user want to push a datetime > predicate, user have to write a ToDate udf since Pig does not have datetime > constant. > In this Jira, I provide a new rule: ConstantCalculator, which is much simpler > and much less error prone, to replace LogicExpressionSimplifier. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (PIG-4128) New logical optimizer rule: ConstantCalculator
[ https://issues.apache.org/jira/browse/PIG-4128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14100183#comment-14100183 ] Daniel Dai commented on PIG-4128: - RB link: https://reviews.apache.org/r/24789/ > New logical optimizer rule: ConstantCalculator > -- > > Key: PIG-4128 > URL: https://issues.apache.org/jira/browse/PIG-4128 > Project: Pig > Issue Type: New Feature > Components: impl >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 0.14.0 > > Attachments: PIG-4128-1.patch > > > Pig used to have a LogicExpressionSimplifier to simplify expression which > also calculates constant expression. The optimizer rule is buggy and we > disable it by default in PIG-2316. > However, we do need this feature especially in partition/predicate push down, > since both does not deal with complex constant expression, we'd like to > replace the expression with constant before the actual push down. Yes, user > may manually do the calculation and rewrite the query, but even rewrite is > sometimes not possible. Consider the case user want to push a datetime > predicate, user have to write a ToDate udf since Pig does not have datetime > constant. > In this Jira, I provide a new rule: ConstantCalculator, which is much simpler > and much less error prone, to replace LogicExpressionSimplifier. -- This message was sent by Atlassian JIRA (v6.2#6252)