[jira] [Updated] (CALCITE-3376) VolcanoPlanner CannotPlanException: best rel is null even though there is an option with non-infinite cost
[
https://issues.apache.org/jira/browse/CALCITE-3376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Quesada Lopez updated CALCITE-3376:
-
Attachment: logsTRACE.txt
> VolcanoPlanner CannotPlanException: best rel is null even though there is an
> option with non-infinite cost
> --
>
> Key: CALCITE-3376
> URL: https://issues.apache.org/jira/browse/CALCITE-3376
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.21.0
>Reporter: Ruben Quesada Lopez
>Priority: Major
> Attachments: Graphviz.png, logsTRACE.txt, stackTrace.txt
>
>
> The problem can be reproduced by adding this test to PlannerTest.java:
> {code:java}
> @Test public void testCannotPlanException() throws Exception {
> RelBuilder builder = RelBuilder.create(RelBuilderTest.config().build());
> RuleSet ruleSet =
> RuleSets.ofList(
> //EnumerableRules.ENUMERABLE_JOIN_RULE, // with this rule it
> works!
> JoinToCorrelateRule.INSTANCE,
> EnumerableRules.ENUMERABLE_CORRELATE_RULE,
> EnumerableRules.ENUMERABLE_PROJECT_RULE,
> EnumerableRules.ENUMERABLE_FILTER_RULE,
> EnumerableRules.ENUMERABLE_SORT_RULE,
> EnumerableRules.ENUMERABLE_UNION_RULE,
> EnumerableRules.ENUMERABLE_TABLE_SCAN_RULE);
> builder
> .scan("EMP")
> .scan("EMP")
> .union(true)
> .scan("EMP")
> .scan("EMP")
> .union(true)
> .join(
> JoinRelType.INNER,
> builder.equals(
> builder.field(2, 0, "DEPTNO"),
> builder.field(2, 1, "EMPNO")));
> RelNode relNode = builder.build();
> RelOptPlanner planner = relNode.getCluster().getPlanner();
> Program program = Programs.of(ruleSet);
> RelTraitSet toTraits = relNode.getTraitSet()
> .replace(EnumerableConvention.INSTANCE);
> RelNode output = program.run(planner, relNode, toTraits,
> ImmutableList.of(), ImmutableList.of());
> String outputStr = toString(output);
> }
> {code}
> Running this test causes the following exception (full stack trace attached):
> {code:java}
> org.apache.calcite.plan.RelOptPlanner$CannotPlanException: There are not
> enough rules to produce a node with desired properties:
> convention=ENUMERABLE, sort=[]. All the inputs have relevant nodes, however
> the cost is still infinite.
> Root: rel#13:Subset#2.ENUMERABLE.[]
> {code}
> The last part of the message (_All the inputs have relevant nodes, however
> the cost is still infinite_) seems relevant, because we can see that
> {{rel#13}}'s best is {{null}}, and it should be {{rel#21}} (which has a
> non-infinite cost):
> {code:java}
> rel#13:Subset#2.ENUMERABLE.[], best=null, importance=1.0
> rel#14:AbstractConverter.ENUMERABLE.[](input=RelSubset#12,
> convention=ENUMERABLE, sort=[]), rowcount=117.6, cumulative cost={inf}
> rel#21:EnumerableCorrelate.ENUMERABLE.[](left=RelSubset#19,
> right=RelSubset#20, correlation=$cor0, joinType=inner, requiredColumns={7}),
> rowcount=1.0, cumulative cost={1770.60001 rows, 2466.0 cpu, 0.0 io}
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-3376) VolcanoPlanner CannotPlanException: best rel is null even though there is an option with non-infinite cost
[
https://issues.apache.org/jira/browse/CALCITE-3376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Quesada Lopez updated CALCITE-3376:
-
Attachment: (was: logsTRACE.txt)
> VolcanoPlanner CannotPlanException: best rel is null even though there is an
> option with non-infinite cost
> --
>
> Key: CALCITE-3376
> URL: https://issues.apache.org/jira/browse/CALCITE-3376
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.21.0
>Reporter: Ruben Quesada Lopez
>Priority: Major
> Attachments: Graphviz.png, logsTRACE.txt, stackTrace.txt
>
>
> The problem can be reproduced by adding this test to PlannerTest.java:
> {code:java}
> @Test public void testCannotPlanException() throws Exception {
> RelBuilder builder = RelBuilder.create(RelBuilderTest.config().build());
> RuleSet ruleSet =
> RuleSets.ofList(
> //EnumerableRules.ENUMERABLE_JOIN_RULE, // with this rule it
> works!
> JoinToCorrelateRule.INSTANCE,
> EnumerableRules.ENUMERABLE_CORRELATE_RULE,
> EnumerableRules.ENUMERABLE_PROJECT_RULE,
> EnumerableRules.ENUMERABLE_FILTER_RULE,
> EnumerableRules.ENUMERABLE_SORT_RULE,
> EnumerableRules.ENUMERABLE_UNION_RULE,
> EnumerableRules.ENUMERABLE_TABLE_SCAN_RULE);
> builder
> .scan("EMP")
> .scan("EMP")
> .union(true)
> .scan("EMP")
> .scan("EMP")
> .union(true)
> .join(
> JoinRelType.INNER,
> builder.equals(
> builder.field(2, 0, "DEPTNO"),
> builder.field(2, 1, "EMPNO")));
> RelNode relNode = builder.build();
> RelOptPlanner planner = relNode.getCluster().getPlanner();
> Program program = Programs.of(ruleSet);
> RelTraitSet toTraits = relNode.getTraitSet()
> .replace(EnumerableConvention.INSTANCE);
> RelNode output = program.run(planner, relNode, toTraits,
> ImmutableList.of(), ImmutableList.of());
> String outputStr = toString(output);
> }
> {code}
> Running this test causes the following exception (full stack trace attached):
> {code:java}
> org.apache.calcite.plan.RelOptPlanner$CannotPlanException: There are not
> enough rules to produce a node with desired properties:
> convention=ENUMERABLE, sort=[]. All the inputs have relevant nodes, however
> the cost is still infinite.
> Root: rel#13:Subset#2.ENUMERABLE.[]
> {code}
> The last part of the message (_All the inputs have relevant nodes, however
> the cost is still infinite_) seems relevant, because we can see that
> {{rel#13}}'s best is {{null}}, and it should be {{rel#21}} (which has a
> non-infinite cost):
> {code:java}
> rel#13:Subset#2.ENUMERABLE.[], best=null, importance=1.0
> rel#14:AbstractConverter.ENUMERABLE.[](input=RelSubset#12,
> convention=ENUMERABLE, sort=[]), rowcount=117.6, cumulative cost={inf}
> rel#21:EnumerableCorrelate.ENUMERABLE.[](left=RelSubset#19,
> right=RelSubset#20, correlation=$cor0, joinType=inner, requiredColumns={7}),
> rowcount=1.0, cumulative cost={1770.60001 rows, 2466.0 cpu, 0.0 io}
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-3376) VolcanoPlanner CannotPlanException: best rel is null even though there is an option with non-infinite cost
[
https://issues.apache.org/jira/browse/CALCITE-3376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Quesada Lopez updated CALCITE-3376:
-
Attachment: logsTRACE.txt
> VolcanoPlanner CannotPlanException: best rel is null even though there is an
> option with non-infinite cost
> --
>
> Key: CALCITE-3376
> URL: https://issues.apache.org/jira/browse/CALCITE-3376
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.21.0
>Reporter: Ruben Quesada Lopez
>Priority: Major
> Attachments: Graphviz.png, logsTRACE.txt, stackTrace.txt
>
>
> The problem can be reproduced by adding this test to PlannerTest.java:
> {code:java}
> @Test public void testCannotPlanException() throws Exception {
> RelBuilder builder = RelBuilder.create(RelBuilderTest.config().build());
> RuleSet ruleSet =
> RuleSets.ofList(
> //EnumerableRules.ENUMERABLE_JOIN_RULE, // with this rule it
> works!
> JoinToCorrelateRule.INSTANCE,
> EnumerableRules.ENUMERABLE_CORRELATE_RULE,
> EnumerableRules.ENUMERABLE_PROJECT_RULE,
> EnumerableRules.ENUMERABLE_FILTER_RULE,
> EnumerableRules.ENUMERABLE_SORT_RULE,
> EnumerableRules.ENUMERABLE_UNION_RULE,
> EnumerableRules.ENUMERABLE_TABLE_SCAN_RULE);
> builder
> .scan("EMP")
> .scan("EMP")
> .union(true)
> .scan("EMP")
> .scan("EMP")
> .union(true)
> .join(
> JoinRelType.INNER,
> builder.equals(
> builder.field(2, 0, "DEPTNO"),
> builder.field(2, 1, "EMPNO")));
> RelNode relNode = builder.build();
> RelOptPlanner planner = relNode.getCluster().getPlanner();
> Program program = Programs.of(ruleSet);
> RelTraitSet toTraits = relNode.getTraitSet()
> .replace(EnumerableConvention.INSTANCE);
> RelNode output = program.run(planner, relNode, toTraits,
> ImmutableList.of(), ImmutableList.of());
> String outputStr = toString(output);
> }
> {code}
> Running this test causes the following exception (full stack trace attached):
> {code:java}
> org.apache.calcite.plan.RelOptPlanner$CannotPlanException: There are not
> enough rules to produce a node with desired properties:
> convention=ENUMERABLE, sort=[]. All the inputs have relevant nodes, however
> the cost is still infinite.
> Root: rel#13:Subset#2.ENUMERABLE.[]
> {code}
> The last part of the message (_All the inputs have relevant nodes, however
> the cost is still infinite_) seems relevant, because we can see that
> {{rel#13}}'s best is {{null}}, and it should be {{rel#21}} (which has a
> non-infinite cost):
> {code:java}
> rel#13:Subset#2.ENUMERABLE.[], best=null, importance=1.0
> rel#14:AbstractConverter.ENUMERABLE.[](input=RelSubset#12,
> convention=ENUMERABLE, sort=[]), rowcount=117.6, cumulative cost={inf}
> rel#21:EnumerableCorrelate.ENUMERABLE.[](left=RelSubset#19,
> right=RelSubset#20, correlation=$cor0, joinType=inner, requiredColumns={7}),
> rowcount=1.0, cumulative cost={1770.60001 rows, 2466.0 cpu, 0.0 io}
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-2959) Support collation on struct fields
[
https://issues.apache.org/jira/browse/CALCITE-2959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Quesada Lopez updated CALCITE-2959:
-
Description:
Currently, the class {{RelFieldCollation}} is used to define _"the ordering of
one field of a RelNode whose output is to be sorted"_. This representation can
hold only "simple" fields. In case of struct fields, a projection needs to be
applied in order to reference the struct field as a simple one. For example,
given this table:
{code}
CREATE TYPE Address AS (
street VARCHAR(20) NOT NULL,
zipcode VARCHAR(20) NOT NULL,
city VARCHAR(20) NOT NULL);
CREATE TABLE Person (
id VARCHAR(20) NOT NULL,
name VARCHAR(20) NOT NULL,
address Address NOT NULL);
{code}
With a SQL query such as: "{{SELECT p.name, p.address.city FROM Person p ORDER
BY p.address.city}}" the pseudo-plan generated would look like:
{code}
Sort(1) // --> Collation: [1]
Project(0=$1, 1=$2.city)
Scan(table=Person)
{code}
However, what would happen if we had a specific Scan operator that would
guarantee us that the records would be scanned already ordered by address.city?
Something like:
{code}
EnhancedScan(table=Person, sort=$2.city) // --> Collation???
{code}
The collation of such an operator cannot be represented with the current
Calcite capabilities (RelFieldCollation), because it would not be a "simple"
field, but a struct field, i.e. we would need a new collation abstraction to
represent it, e.g. [2.city] or [2.2]
I would like to open the discussion to see if / how we could find a solution to
represent this case.
was:
Currently, the class {{RelFieldCollation}} is used to define _"the ordering of
one field of a RelNode whose output is to be sorted"_. This representation can
hold only "simple" fields. In case of struct fields, a projection needs to be
applied in order to reference the struct field as a simple one. For example,
given this table:
{code}
CREATE TYPE Address AS (
street VARCHAR(20) NOT NULL,
zipcode VARCHAR(20) NOT NULL,
city VARCHAR(20) NOT NULL);
CREATE TABLE Person (
id VARCHAR(20) NOT NULL,
name VARCHAR(20) NOT NULL,
address Address NOT NULL);
{code}
With a SQL query such as: "{{SELECT p.name, p.address.city FROM Person p ORDER
BY p.address.city}}" the pseudo-plan generated would look like:
{code}
Sort(1) // --> Collation: [1]
Project(0=$1, 1=$2.city)
Scan(table=Person)
{code}
However, what would happen if we had a specific Scan operator that would
guarantee us that the records would be scanned already ordered by address.city?
Something like:
{code}
EnhancedScan(table=Person, sort=$2.city) // --> Collation???
{code}
The collation of such an operator cannot be represented with the current
Calcite capabilities (RelFieldCollation), because it would not be a "simple"
field, but a struct field, i.e. we would need a new collation abstraction to
represent it, e.g. [2.city] or [2.2]
I would like to open the discuss to see if / how we could find a solution to
represent this case.
> Support collation on struct fields
> --
>
> Key: CALCITE-2959
> URL: https://issues.apache.org/jira/browse/CALCITE-2959
> Project: Calcite
> Issue Type: New Feature
>Reporter: Ruben Quesada Lopez
>Priority: Major
>
> Currently, the class {{RelFieldCollation}} is used to define _"the ordering
> of one field of a RelNode whose output is to be sorted"_. This representation
> can hold only "simple" fields. In case of struct fields, a projection needs
> to be applied in order to reference the struct field as a simple one. For
> example, given this table:
> {code}
> CREATE TYPE Address AS (
> street VARCHAR(20) NOT NULL,
> zipcode VARCHAR(20) NOT NULL,
> city VARCHAR(20) NOT NULL);
> CREATE TABLE Person (
> id VARCHAR(20) NOT NULL,
> name VARCHAR(20) NOT NULL,
> address Address NOT NULL);
> {code}
> With a SQL query such as: "{{SELECT p.name, p.address.city FROM Person p
> ORDER BY p.address.city}}" the pseudo-plan generated would look like:
> {code}
> Sort(1) // --> Collation: [1]
> Project(0=$1, 1=$2.city)
> Scan(table=Person)
> {code}
> However, what would happen if we had a specific Scan operator that would
> guarantee us that the records would be scanned already ordered by
> address.city? Something like:
> {code}
> EnhancedScan(table=Person, sort=$2.city) // --> Collation???
> {code}
> The collation of such an operator cannot be represented with the current
> Calcite capabilities (RelFieldCollation), because it would not be a "simple"
> field, but a struct field, i.e. we would need a new collation abstraction to
> represent it, e.g. [2.city] or [2.2]
> I would like to open the discussion to see if / how we could find a solution
> to represent this case.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Resolved] (CALCITE-3309) Refactor generatePredicate method from EnumerableNestedLoopJoin/EnumerableHashJoin/EnumerableBatchNestedLoopJoin into a single location
[
https://issues.apache.org/jira/browse/CALCITE-3309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Quesada Lopez resolved CALCITE-3309.
--
Resolution: Fixed
Fixed via
https://github.com/apache/calcite/commit/55cc3abfbcb906935a703a5cf77b1b0359fca734
> Refactor generatePredicate method from
> EnumerableNestedLoopJoin/EnumerableHashJoin/EnumerableBatchNestedLoopJoin
> into a single location
> ---
>
> Key: CALCITE-3309
> URL: https://issues.apache.org/jira/browse/CALCITE-3309
> Project: Calcite
> Issue Type: Improvement
>Reporter: Ruben Quesada Lopez
>Assignee: Ruben Quesada Lopez
>Priority: Trivial
> Labels: pull-request-available
> Fix For: 1.21.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The method {{EnumerableNestedLoopJoin#predicate}} (that generates a Predicate
> Expression based on a RexNode condition) has been copied pasted as
> {{EnumerableBatchNestedLoopJoin#generatePredicate}} due to CALCITE-2979, and
> {{EnumerableHashJoin#generatePredicate}} due to CALCITE-2973.
> The goal of this ticket is to refactor that method into a single location
> (e.g. EnumUtils) where it can be accessible by all 3 Enumerable Joins.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (CALCITE-3307) PigRelExTest, PigRelOpTest and PigScriptTest fail on Windows
[
https://issues.apache.org/jira/browse/CALCITE-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919575#comment-16919575
]
Ruben Quesada Lopez commented on CALCITE-3307:
--
BTW, I just noticed that there is another "skip for Windows" in
PigRelBuilderStyleTest:
{code}
public class PigRelBuilderStyleTest extends AbstractPigTest {
public PigRelBuilderStyleTest() {
Assume.assumeThat("Pigs don't like Windows", File.separatorChar, is('/'));
}
...
{code}
If I remove that line, I get the same exception {{java.io.IOException: Could
not locate executable null\bin\winutils.exe in the Hadoop binaries.}} in those
tests.
> PigRelExTest, PigRelOpTest and PigScriptTest fail on Windows
>
>
> Key: CALCITE-3307
> URL: https://issues.apache.org/jira/browse/CALCITE-3307
> Project: Calcite
> Issue Type: Bug
>Reporter: Julian Hyde
>Assignee: Ruben Quesada Lopez
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.21.0
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Running {{PigRelExTest}} on Windows, after CALCITE-3122 was merged, gives the
> following error:
> {noformat}
> 2019-08-29 15:33:23,229 [main] ERROR - Failed to locate the winutils binary
> in the hadoop binary path
> java.io.IOException: Could not locate executable null\bin\winutils.exe in the
> Hadoop binaries.
> at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:382)
> at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:397)
> at org.apache.hadoop.util.Shell.(Shell.java:390)
> at org.apache.hadoop.util.StringUtils.(StringUtils.java:80)
> at
> org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611)
> at
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:273)
> at
> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:261)
> at
> org.apache.hadoop.security.UserGroupInformation.isAuthenticationMethodEnabled(UserGroupInformation.java:338)
> at
> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:332)
> at
> org.apache.pig.backend.hadoop.HKerberos.tryKerberosKeytabLogin(HKerberos.java:70)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:220)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:112)
> at org.apache.pig.impl.PigContext.connect(PigContext.java:305)
> at org.apache.pig.PigServer.(PigServer.java:231)
> at org.apache.pig.PigServer.(PigServer.java:219)
> at org.apache.pig.PigServer.(PigServer.java:211)
> at org.apache.pig.PigServer.(PigServer.java:207)
> at org.apache.calcite.piglet.PigConverter.(PigConverter.java:107)
> at org.apache.calcite.piglet.PigConverter.create(PigConverter.java:112)
> at
> org.apache.calcite.test.PigRelTestBase.testSetup(PigRelTestBase.java:34)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> {noformat}
> This issue has arisen many times; see HADOOP-10775 and SPARK-2356.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Comment Edited] (CALCITE-3307) PigRelExTest, PigRelOpTest and PigScriptTest fail on Windows
[
https://issues.apache.org/jira/browse/CALCITE-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919575#comment-16919575
]
Ruben Quesada Lopez edited comment on CALCITE-3307 at 8/30/19 2:05 PM:
---
BTW, I just noticed that there was already another "skip for Windows" in
PigRelBuilderStyleTest:
{code}
public class PigRelBuilderStyleTest extends AbstractPigTest {
public PigRelBuilderStyleTest() {
Assume.assumeThat("Pigs don't like Windows", File.separatorChar, is('/'));
}
...
{code}
If I remove that line, I get the same exception {{java.io.IOException: Could
not locate executable null\bin\winutils.exe in the Hadoop binaries.}} in those
tests.
was (Author: rubenql):
BTW, I just noticed that there is another "skip for Windows" in
PigRelBuilderStyleTest:
{code}
public class PigRelBuilderStyleTest extends AbstractPigTest {
public PigRelBuilderStyleTest() {
Assume.assumeThat("Pigs don't like Windows", File.separatorChar, is('/'));
}
...
{code}
If I remove that line, I get the same exception {{java.io.IOException: Could
not locate executable null\bin\winutils.exe in the Hadoop binaries.}} in those
tests.
> PigRelExTest, PigRelOpTest and PigScriptTest fail on Windows
>
>
> Key: CALCITE-3307
> URL: https://issues.apache.org/jira/browse/CALCITE-3307
> Project: Calcite
> Issue Type: Bug
>Reporter: Julian Hyde
>Assignee: Ruben Quesada Lopez
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.21.0
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Running {{PigRelExTest}} on Windows, after CALCITE-3122 was merged, gives the
> following error:
> {noformat}
> 2019-08-29 15:33:23,229 [main] ERROR - Failed to locate the winutils binary
> in the hadoop binary path
> java.io.IOException: Could not locate executable null\bin\winutils.exe in the
> Hadoop binaries.
> at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:382)
> at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:397)
> at org.apache.hadoop.util.Shell.(Shell.java:390)
> at org.apache.hadoop.util.StringUtils.(StringUtils.java:80)
> at
> org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611)
> at
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:273)
> at
> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:261)
> at
> org.apache.hadoop.security.UserGroupInformation.isAuthenticationMethodEnabled(UserGroupInformation.java:338)
> at
> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:332)
> at
> org.apache.pig.backend.hadoop.HKerberos.tryKerberosKeytabLogin(HKerberos.java:70)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:220)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:112)
> at org.apache.pig.impl.PigContext.connect(PigContext.java:305)
> at org.apache.pig.PigServer.(PigServer.java:231)
> at org.apache.pig.PigServer.(PigServer.java:219)
> at org.apache.pig.PigServer.(PigServer.java:211)
> at org.apache.pig.PigServer.(PigServer.java:207)
> at org.apache.calcite.piglet.PigConverter.(PigConverter.java:107)
> at org.apache.calcite.piglet.PigConverter.create(PigConverter.java:112)
> at
> org.apache.calcite.test.PigRelTestBase.testSetup(PigRelTestBase.java:34)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at
[jira] [Commented] (CALCITE-3307) PigRelExTest, PigRelOpTest and PigScriptTest fail on Windows
[
https://issues.apache.org/jira/browse/CALCITE-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919520#comment-16919520
]
Ruben Quesada Lopez commented on CALCITE-3307:
--
You're welcome! :)
> PigRelExTest, PigRelOpTest and PigScriptTest fail on Windows
>
>
> Key: CALCITE-3307
> URL: https://issues.apache.org/jira/browse/CALCITE-3307
> Project: Calcite
> Issue Type: Bug
>Reporter: Julian Hyde
>Assignee: Ruben Quesada Lopez
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.21.0
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Running {{PigRelExTest}} on Windows, after CALCITE-3122 was merged, gives the
> following error:
> {noformat}
> 2019-08-29 15:33:23,229 [main] ERROR - Failed to locate the winutils binary
> in the hadoop binary path
> java.io.IOException: Could not locate executable null\bin\winutils.exe in the
> Hadoop binaries.
> at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:382)
> at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:397)
> at org.apache.hadoop.util.Shell.(Shell.java:390)
> at org.apache.hadoop.util.StringUtils.(StringUtils.java:80)
> at
> org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611)
> at
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:273)
> at
> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:261)
> at
> org.apache.hadoop.security.UserGroupInformation.isAuthenticationMethodEnabled(UserGroupInformation.java:338)
> at
> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:332)
> at
> org.apache.pig.backend.hadoop.HKerberos.tryKerberosKeytabLogin(HKerberos.java:70)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:220)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:112)
> at org.apache.pig.impl.PigContext.connect(PigContext.java:305)
> at org.apache.pig.PigServer.(PigServer.java:231)
> at org.apache.pig.PigServer.(PigServer.java:219)
> at org.apache.pig.PigServer.(PigServer.java:211)
> at org.apache.pig.PigServer.(PigServer.java:207)
> at org.apache.calcite.piglet.PigConverter.(PigConverter.java:107)
> at org.apache.calcite.piglet.PigConverter.create(PigConverter.java:112)
> at
> org.apache.calcite.test.PigRelTestBase.testSetup(PigRelTestBase.java:34)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> {noformat}
> This issue has arisen many times; see HADOOP-10775 and SPARK-2356.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (CALCITE-3307) PigRelExTest, PigRelOpTest and PigScriptTest fail on Windows
[
https://issues.apache.org/jira/browse/CALCITE-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Quesada Lopez updated CALCITE-3307:
-
Fix Version/s: 1.21.0
> PigRelExTest, PigRelOpTest and PigScriptTest fail on Windows
>
>
> Key: CALCITE-3307
> URL: https://issues.apache.org/jira/browse/CALCITE-3307
> Project: Calcite
> Issue Type: Bug
>Reporter: Julian Hyde
>Assignee: Ruben Quesada Lopez
>Priority: Major
> Fix For: 1.21.0
>
>
> Running {{PigRelExTest}} on Windows, after CALCITE-3122 was merged, gives the
> following error:
> {noformat}
> 2019-08-29 15:33:23,229 [main] ERROR - Failed to locate the winutils binary
> in the hadoop binary path
> java.io.IOException: Could not locate executable null\bin\winutils.exe in the
> Hadoop binaries.
> at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:382)
> at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:397)
> at org.apache.hadoop.util.Shell.(Shell.java:390)
> at org.apache.hadoop.util.StringUtils.(StringUtils.java:80)
> at
> org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611)
> at
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:273)
> at
> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:261)
> at
> org.apache.hadoop.security.UserGroupInformation.isAuthenticationMethodEnabled(UserGroupInformation.java:338)
> at
> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:332)
> at
> org.apache.pig.backend.hadoop.HKerberos.tryKerberosKeytabLogin(HKerberos.java:70)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:220)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:112)
> at org.apache.pig.impl.PigContext.connect(PigContext.java:305)
> at org.apache.pig.PigServer.(PigServer.java:231)
> at org.apache.pig.PigServer.(PigServer.java:219)
> at org.apache.pig.PigServer.(PigServer.java:211)
> at org.apache.pig.PigServer.(PigServer.java:207)
> at org.apache.calcite.piglet.PigConverter.(PigConverter.java:107)
> at org.apache.calcite.piglet.PigConverter.create(PigConverter.java:112)
> at
> org.apache.calcite.test.PigRelTestBase.testSetup(PigRelTestBase.java:34)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> {noformat}
> This issue has arisen many times; see HADOOP-10775 and SPARK-2356.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Comment Edited] (CALCITE-3307) PigRelExTest, PigRelOpTest and PigScriptTest fail on Windows
[
https://issues.apache.org/jira/browse/CALCITE-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919308#comment-16919308
]
Ruben Quesada Lopez edited comment on CALCITE-3307 at 8/30/19 8:11 AM:
---
Following the suggestion by [~julianhyde] in CALCITE-3122, I can take care of
this by just ignoring the:
- 29 tests in PigRelExTest
- 32 tests in PigRelOpTest
- 1 test in PigScriptTest
By adding:
{code}
Assume.assumeFalse("Skip: Pig/Hadoop tests do not work on Windows",
isWindows());
{code}
Plus the corresponding {{isWindows}} method in a common location (probably
{{PigRelTestBase}}):
{code}
static boolean isWindows() {
return System.getProperty("os.name").startsWith("Windows");
}
{code}
was (Author: rubenql):
Following the suggestion by [~julianhyde] in CALCITE-3122, I can take care of
this by just ignoring the:
- 32 tests in PigRelOpTest
- 29 tests in PigRelExTest
- 1 test in PigScriptTest
By adding:
{code}
Assume.assumeFalse("Skip: Pig/Hadoop tests do not work on Windows",
isWindows());
{code}
Plus the corresponding {{isWindows}} method in a common location (probably
{{PigRelTestBase}}):
{code}
static boolean isWindows() {
return System.getProperty("os.name").startsWith("Windows");
}
{code}
> PigRelExTest, PigRelOpTest and PigScriptTest fail on Windows
>
>
> Key: CALCITE-3307
> URL: https://issues.apache.org/jira/browse/CALCITE-3307
> Project: Calcite
> Issue Type: Bug
>Reporter: Julian Hyde
>Assignee: Ruben Quesada Lopez
>Priority: Major
>
> Running {{PigRelExTest}} on Windows, after CALCITE-3122 was merged, gives the
> following error:
> {noformat}
> 2019-08-29 15:33:23,229 [main] ERROR - Failed to locate the winutils binary
> in the hadoop binary path
> java.io.IOException: Could not locate executable null\bin\winutils.exe in the
> Hadoop binaries.
> at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:382)
> at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:397)
> at org.apache.hadoop.util.Shell.(Shell.java:390)
> at org.apache.hadoop.util.StringUtils.(StringUtils.java:80)
> at
> org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611)
> at
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:273)
> at
> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:261)
> at
> org.apache.hadoop.security.UserGroupInformation.isAuthenticationMethodEnabled(UserGroupInformation.java:338)
> at
> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:332)
> at
> org.apache.pig.backend.hadoop.HKerberos.tryKerberosKeytabLogin(HKerberos.java:70)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:220)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:112)
> at org.apache.pig.impl.PigContext.connect(PigContext.java:305)
> at org.apache.pig.PigServer.(PigServer.java:231)
> at org.apache.pig.PigServer.(PigServer.java:219)
> at org.apache.pig.PigServer.(PigServer.java:211)
> at org.apache.pig.PigServer.(PigServer.java:207)
> at org.apache.calcite.piglet.PigConverter.(PigConverter.java:107)
> at org.apache.calcite.piglet.PigConverter.create(PigConverter.java:112)
> at
> org.apache.calcite.test.PigRelTestBase.testSetup(PigRelTestBase.java:34)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at
[jira] [Assigned] (CALCITE-3307) PigRelExTest, PigRelOpTest and PigScriptTest fail on Windows
[
https://issues.apache.org/jira/browse/CALCITE-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Quesada Lopez reassigned CALCITE-3307:
Assignee: Ruben Quesada Lopez
> PigRelExTest, PigRelOpTest and PigScriptTest fail on Windows
>
>
> Key: CALCITE-3307
> URL: https://issues.apache.org/jira/browse/CALCITE-3307
> Project: Calcite
> Issue Type: Bug
>Reporter: Julian Hyde
>Assignee: Ruben Quesada Lopez
>Priority: Major
>
> Running {{PigRelExTest}} on Windows, after CALCITE-3122 was merged, gives the
> following error:
> {noformat}
> 2019-08-29 15:33:23,229 [main] ERROR - Failed to locate the winutils binary
> in the hadoop binary path
> java.io.IOException: Could not locate executable null\bin\winutils.exe in the
> Hadoop binaries.
> at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:382)
> at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:397)
> at org.apache.hadoop.util.Shell.(Shell.java:390)
> at org.apache.hadoop.util.StringUtils.(StringUtils.java:80)
> at
> org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611)
> at
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:273)
> at
> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:261)
> at
> org.apache.hadoop.security.UserGroupInformation.isAuthenticationMethodEnabled(UserGroupInformation.java:338)
> at
> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:332)
> at
> org.apache.pig.backend.hadoop.HKerberos.tryKerberosKeytabLogin(HKerberos.java:70)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:220)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:112)
> at org.apache.pig.impl.PigContext.connect(PigContext.java:305)
> at org.apache.pig.PigServer.(PigServer.java:231)
> at org.apache.pig.PigServer.(PigServer.java:219)
> at org.apache.pig.PigServer.(PigServer.java:211)
> at org.apache.pig.PigServer.(PigServer.java:207)
> at org.apache.calcite.piglet.PigConverter.(PigConverter.java:107)
> at org.apache.calcite.piglet.PigConverter.create(PigConverter.java:112)
> at
> org.apache.calcite.test.PigRelTestBase.testSetup(PigRelTestBase.java:34)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> {noformat}
> This issue has arisen many times; see HADOOP-10775 and SPARK-2356.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Comment Edited] (CALCITE-3307) PigRelExTest, PigRelOpTest and PigScriptTest fail on Windows
[
https://issues.apache.org/jira/browse/CALCITE-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919308#comment-16919308
]
Ruben Quesada Lopez edited comment on CALCITE-3307 at 8/30/19 8:09 AM:
---
Following the suggestion by [~julianhyde] in CALCITE-3122, I can take care of
this by just ignoring the:
- 32 tests in PigRelOpTest
- 29 tests in PigRelExTest
- 1 test in PigScriptTest
By adding:
{code}
Assume.assumeFalse("Skip: Pig/Hadoop tests do not work on Windows",
isWindows());
{code}
Plus the corresponding {{isWindows}} method in a common location (probably
{{PigRelTestBase}}):
{code}
static boolean isWindows() {
return System.getProperty("os.name").startsWith("Windows");
}
{code}
was (Author: rubenql):
Following the suggestion by [~julianhyde] in CALCITE-3122, I can take care of
this by just ignoring the:
- 32 tests in PigRelOpTest
- 29 tests in PigRelExTest
- 1 test in PigScriptTest
By adding:
{code}
Assume.assumeFalse("Skip: Pig/Hadoop tests do not work on Windows",
isWindows());
{code}
Plus the corresponding {{isWindows}} method in a common location (probably
PigRelTestBas):
{code}
static boolean isWindows() {
return System.getProperty("os.name").startsWith("Windows");
}
{code}
> PigRelExTest, PigRelOpTest and PigScriptTest fail on Windows
>
>
> Key: CALCITE-3307
> URL: https://issues.apache.org/jira/browse/CALCITE-3307
> Project: Calcite
> Issue Type: Bug
>Reporter: Julian Hyde
>Priority: Major
>
> Running {{PigRelExTest}} on Windows, after CALCITE-3122 was merged, gives the
> following error:
> {noformat}
> 2019-08-29 15:33:23,229 [main] ERROR - Failed to locate the winutils binary
> in the hadoop binary path
> java.io.IOException: Could not locate executable null\bin\winutils.exe in the
> Hadoop binaries.
> at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:382)
> at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:397)
> at org.apache.hadoop.util.Shell.(Shell.java:390)
> at org.apache.hadoop.util.StringUtils.(StringUtils.java:80)
> at
> org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611)
> at
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:273)
> at
> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:261)
> at
> org.apache.hadoop.security.UserGroupInformation.isAuthenticationMethodEnabled(UserGroupInformation.java:338)
> at
> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:332)
> at
> org.apache.pig.backend.hadoop.HKerberos.tryKerberosKeytabLogin(HKerberos.java:70)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:220)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:112)
> at org.apache.pig.impl.PigContext.connect(PigContext.java:305)
> at org.apache.pig.PigServer.(PigServer.java:231)
> at org.apache.pig.PigServer.(PigServer.java:219)
> at org.apache.pig.PigServer.(PigServer.java:211)
> at org.apache.pig.PigServer.(PigServer.java:207)
> at org.apache.calcite.piglet.PigConverter.(PigConverter.java:107)
> at org.apache.calcite.piglet.PigConverter.create(PigConverter.java:112)
> at
> org.apache.calcite.test.PigRelTestBase.testSetup(PigRelTestBase.java:34)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at
[jira] [Comment Edited] (CALCITE-3307) PigRelExTest, PigRelOpTest and PigScriptTest fail on Windows
[
https://issues.apache.org/jira/browse/CALCITE-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919308#comment-16919308
]
Ruben Quesada Lopez edited comment on CALCITE-3307 at 8/30/19 8:09 AM:
---
Following the suggestion by [~julianhyde] in CALCITE-3122, I can take care of
this by just ignoring the:
- 32 tests in PigRelOpTest
- 29 tests in PigRelExTest
- 1 test in PigScriptTest
By adding:
{code}
Assume.assumeFalse("Skip: Pig/Hadoop tests do not work on Windows",
isWindows());
{code}
Plus the corresponding {{isWindows}} method in a common location (probably
{{PigRelTestBase}}):
{code}
static boolean isWindows() {
return System.getProperty("os.name").startsWith("Windows");
}
{code}
was (Author: rubenql):
Following the suggestion by [~julianhyde] in CALCITE-3122, I can take care of
this by just ignoring the:
- 32 tests in PigRelOpTest
- 29 tests in PigRelExTest
- 1 test in PigScriptTest
By adding:
{code}
Assume.assumeFalse("Skip: Pig/Hadoop tests do not work on Windows",
isWindows());
{code}
Plus the corresponding {{isWindows}} method in a common location (probably
{{PigRelTestBase}}):
{code}
static boolean isWindows() {
return System.getProperty("os.name").startsWith("Windows");
}
{code}
> PigRelExTest, PigRelOpTest and PigScriptTest fail on Windows
>
>
> Key: CALCITE-3307
> URL: https://issues.apache.org/jira/browse/CALCITE-3307
> Project: Calcite
> Issue Type: Bug
>Reporter: Julian Hyde
>Priority: Major
>
> Running {{PigRelExTest}} on Windows, after CALCITE-3122 was merged, gives the
> following error:
> {noformat}
> 2019-08-29 15:33:23,229 [main] ERROR - Failed to locate the winutils binary
> in the hadoop binary path
> java.io.IOException: Could not locate executable null\bin\winutils.exe in the
> Hadoop binaries.
> at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:382)
> at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:397)
> at org.apache.hadoop.util.Shell.(Shell.java:390)
> at org.apache.hadoop.util.StringUtils.(StringUtils.java:80)
> at
> org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611)
> at
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:273)
> at
> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:261)
> at
> org.apache.hadoop.security.UserGroupInformation.isAuthenticationMethodEnabled(UserGroupInformation.java:338)
> at
> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:332)
> at
> org.apache.pig.backend.hadoop.HKerberos.tryKerberosKeytabLogin(HKerberos.java:70)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:220)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:112)
> at org.apache.pig.impl.PigContext.connect(PigContext.java:305)
> at org.apache.pig.PigServer.(PigServer.java:231)
> at org.apache.pig.PigServer.(PigServer.java:219)
> at org.apache.pig.PigServer.(PigServer.java:211)
> at org.apache.pig.PigServer.(PigServer.java:207)
> at org.apache.calcite.piglet.PigConverter.(PigConverter.java:107)
> at org.apache.calcite.piglet.PigConverter.create(PigConverter.java:112)
> at
> org.apache.calcite.test.PigRelTestBase.testSetup(PigRelTestBase.java:34)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at
[jira] [Comment Edited] (CALCITE-3307) PigRelExTest, PigRelOpTest and PigScriptTest fail on Windows
[
https://issues.apache.org/jira/browse/CALCITE-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919308#comment-16919308
]
Ruben Quesada Lopez edited comment on CALCITE-3307 at 8/30/19 8:09 AM:
---
Following the suggestion by [~julianhyde] in CALCITE-3122, I can take care of
this by just ignoring the:
- 32 tests in PigRelOpTest
- 29 tests in PigRelExTest
- 1 test in PigScriptTest
By adding:
{code}
Assume.assumeFalse("Skip: Pig/Hadoop tests do not work on Windows",
isWindows());
{code}
Plus the corresponding {{isWindows}} method in a common location (probably
PigRelTestBas):
{code}
static boolean isWindows() {
return System.getProperty("os.name").startsWith("Windows");
}
{code}
was (Author: rubenql):
Following the suggestion by [~julianhyde] in CALCITE-3122, I can take care of
this by just ignoring the:
- 32 tests in PigRelOpTest
- 29 tests in PigRelExTest
- 1 test in PigScriptTest
By adding:
{code}
Assume.assumeFalse("Skip: Pig/Hadoop tests do not work on Windows",
isWindows());
{code}
Plus the corresponding {{isWindows}} method in a common location (probably
PigRelTestBas):
{code}
static boolean isWindows() {
return System.getProperty("os.name").startsWith("Windows");
}
{code}
> PigRelExTest, PigRelOpTest and PigScriptTest fail on Windows
>
>
> Key: CALCITE-3307
> URL: https://issues.apache.org/jira/browse/CALCITE-3307
> Project: Calcite
> Issue Type: Bug
>Reporter: Julian Hyde
>Priority: Major
>
> Running {{PigRelExTest}} on Windows, after CALCITE-3122 was merged, gives the
> following error:
> {noformat}
> 2019-08-29 15:33:23,229 [main] ERROR - Failed to locate the winutils binary
> in the hadoop binary path
> java.io.IOException: Could not locate executable null\bin\winutils.exe in the
> Hadoop binaries.
> at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:382)
> at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:397)
> at org.apache.hadoop.util.Shell.(Shell.java:390)
> at org.apache.hadoop.util.StringUtils.(StringUtils.java:80)
> at
> org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611)
> at
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:273)
> at
> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:261)
> at
> org.apache.hadoop.security.UserGroupInformation.isAuthenticationMethodEnabled(UserGroupInformation.java:338)
> at
> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:332)
> at
> org.apache.pig.backend.hadoop.HKerberos.tryKerberosKeytabLogin(HKerberos.java:70)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:220)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:112)
> at org.apache.pig.impl.PigContext.connect(PigContext.java:305)
> at org.apache.pig.PigServer.(PigServer.java:231)
> at org.apache.pig.PigServer.(PigServer.java:219)
> at org.apache.pig.PigServer.(PigServer.java:211)
> at org.apache.pig.PigServer.(PigServer.java:207)
> at org.apache.calcite.piglet.PigConverter.(PigConverter.java:107)
> at org.apache.calcite.piglet.PigConverter.create(PigConverter.java:112)
> at
> org.apache.calcite.test.PigRelTestBase.testSetup(PigRelTestBase.java:34)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at
[jira] [Updated] (CALCITE-3307) PigRelExTest, PigRelOpTest and PigScriptTest fail on Windows
[
https://issues.apache.org/jira/browse/CALCITE-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Quesada Lopez updated CALCITE-3307:
-
Summary: PigRelExTest, PigRelOpTest and PigScriptTest fail on Windows
(was: PigRelOpTest, PigRelExTest fails on Windows)
> PigRelExTest, PigRelOpTest and PigScriptTest fail on Windows
>
>
> Key: CALCITE-3307
> URL: https://issues.apache.org/jira/browse/CALCITE-3307
> Project: Calcite
> Issue Type: Bug
>Reporter: Julian Hyde
>Priority: Major
>
> Running {{PigRelExTest}} on Windows, after CALCITE-3122 was merged, gives the
> following error:
> {noformat}
> 2019-08-29 15:33:23,229 [main] ERROR - Failed to locate the winutils binary
> in the hadoop binary path
> java.io.IOException: Could not locate executable null\bin\winutils.exe in the
> Hadoop binaries.
> at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:382)
> at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:397)
> at org.apache.hadoop.util.Shell.(Shell.java:390)
> at org.apache.hadoop.util.StringUtils.(StringUtils.java:80)
> at
> org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611)
> at
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:273)
> at
> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:261)
> at
> org.apache.hadoop.security.UserGroupInformation.isAuthenticationMethodEnabled(UserGroupInformation.java:338)
> at
> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:332)
> at
> org.apache.pig.backend.hadoop.HKerberos.tryKerberosKeytabLogin(HKerberos.java:70)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:220)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:112)
> at org.apache.pig.impl.PigContext.connect(PigContext.java:305)
> at org.apache.pig.PigServer.(PigServer.java:231)
> at org.apache.pig.PigServer.(PigServer.java:219)
> at org.apache.pig.PigServer.(PigServer.java:211)
> at org.apache.pig.PigServer.(PigServer.java:207)
> at org.apache.calcite.piglet.PigConverter.(PigConverter.java:107)
> at org.apache.calcite.piglet.PigConverter.create(PigConverter.java:112)
> at
> org.apache.calcite.test.PigRelTestBase.testSetup(PigRelTestBase.java:34)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> {noformat}
> This issue has arisen many times; see HADOOP-10775 and SPARK-2356.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (CALCITE-3307) PigRelOpTest, PigRelExTest fails on Windows
[
https://issues.apache.org/jira/browse/CALCITE-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Quesada Lopez updated CALCITE-3307:
-
Summary: PigRelOpTest, PigRelExTest fails on Windows (was: PigRelExTest
fails on Windows)
> PigRelOpTest, PigRelExTest fails on Windows
> ---
>
> Key: CALCITE-3307
> URL: https://issues.apache.org/jira/browse/CALCITE-3307
> Project: Calcite
> Issue Type: Bug
>Reporter: Julian Hyde
>Priority: Major
>
> Running {{PigRelExTest}} on Windows, after CALCITE-3122 was merged, gives the
> following error:
> {noformat}
> 2019-08-29 15:33:23,229 [main] ERROR - Failed to locate the winutils binary
> in the hadoop binary path
> java.io.IOException: Could not locate executable null\bin\winutils.exe in the
> Hadoop binaries.
> at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:382)
> at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:397)
> at org.apache.hadoop.util.Shell.(Shell.java:390)
> at org.apache.hadoop.util.StringUtils.(StringUtils.java:80)
> at
> org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611)
> at
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:273)
> at
> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:261)
> at
> org.apache.hadoop.security.UserGroupInformation.isAuthenticationMethodEnabled(UserGroupInformation.java:338)
> at
> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:332)
> at
> org.apache.pig.backend.hadoop.HKerberos.tryKerberosKeytabLogin(HKerberos.java:70)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:220)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:112)
> at org.apache.pig.impl.PigContext.connect(PigContext.java:305)
> at org.apache.pig.PigServer.(PigServer.java:231)
> at org.apache.pig.PigServer.(PigServer.java:219)
> at org.apache.pig.PigServer.(PigServer.java:211)
> at org.apache.pig.PigServer.(PigServer.java:207)
> at org.apache.calcite.piglet.PigConverter.(PigConverter.java:107)
> at org.apache.calcite.piglet.PigConverter.create(PigConverter.java:112)
> at
> org.apache.calcite.test.PigRelTestBase.testSetup(PigRelTestBase.java:34)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> {noformat}
> This issue has arisen many times; see HADOOP-10775 and SPARK-2356.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (CALCITE-3309) Refactor generatePredicate method from EnumerableNestedLoopJoin/EnumerableHashJoin/EnumerableBatchNestedLoopJoin into a single location
Ruben Quesada Lopez created CALCITE-3309:
Summary: Refactor generatePredicate method from
EnumerableNestedLoopJoin/EnumerableHashJoin/EnumerableBatchNestedLoopJoin into
a single location
Key: CALCITE-3309
URL: https://issues.apache.org/jira/browse/CALCITE-3309
Project: Calcite
Issue Type: Improvement
Reporter: Ruben Quesada Lopez
Assignee: Ruben Quesada Lopez
Fix For: 1.21.0
The method {{EnumerableNestedLoopJoin#predicate}} (that generates a Predicate
Expression based on a RexNode condition) has been copied pasted as
{{EnumerableBatchNestedLoopJoin#generatePredicate}} due to CALCITE-2979, and
{{EnumerableHashJoin#generatePredicate}} due to CALCITE-2973.
The goal of this ticket is to refactor that method into a single location (e.g.
EnumUtils) where it can be accessible by all 3 Enumerable Joins.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (CALCITE-3307) PigRelExTest fails on Windows
[
https://issues.apache.org/jira/browse/CALCITE-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919278#comment-16919278
]
Ruben Quesada Lopez commented on CALCITE-3307:
--
I confirm that (on my local Windows environment) by ignoring those 3 tests
mentioned above we get a successful build.
> PigRelExTest fails on Windows
> -
>
> Key: CALCITE-3307
> URL: https://issues.apache.org/jira/browse/CALCITE-3307
> Project: Calcite
> Issue Type: Bug
>Reporter: Julian Hyde
>Priority: Major
>
> Running {{PigRelExTest}} on Windows, after CALCITE-3122 was merged, gives the
> following error:
> {noformat}
> 2019-08-29 15:33:23,229 [main] ERROR - Failed to locate the winutils binary
> in the hadoop binary path
> java.io.IOException: Could not locate executable null\bin\winutils.exe in the
> Hadoop binaries.
> at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:382)
> at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:397)
> at org.apache.hadoop.util.Shell.(Shell.java:390)
> at org.apache.hadoop.util.StringUtils.(StringUtils.java:80)
> at
> org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611)
> at
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:273)
> at
> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:261)
> at
> org.apache.hadoop.security.UserGroupInformation.isAuthenticationMethodEnabled(UserGroupInformation.java:338)
> at
> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:332)
> at
> org.apache.pig.backend.hadoop.HKerberos.tryKerberosKeytabLogin(HKerberos.java:70)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:220)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:112)
> at org.apache.pig.impl.PigContext.connect(PigContext.java:305)
> at org.apache.pig.PigServer.(PigServer.java:231)
> at org.apache.pig.PigServer.(PigServer.java:219)
> at org.apache.pig.PigServer.(PigServer.java:211)
> at org.apache.pig.PigServer.(PigServer.java:207)
> at org.apache.calcite.piglet.PigConverter.(PigConverter.java:107)
> at org.apache.calcite.piglet.PigConverter.create(PigConverter.java:112)
> at
> org.apache.calcite.test.PigRelTestBase.testSetup(PigRelTestBase.java:34)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> {noformat}
> This issue has arisen many times; see HADOOP-10775 and SPARK-2356.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Comment Edited] (CALCITE-3307) PigRelExTest fails on Windows
[
https://issues.apache.org/jira/browse/CALCITE-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919278#comment-16919278
]
Ruben Quesada Lopez edited comment on CALCITE-3307 at 8/30/19 7:01 AM:
---
I confirm that (on my local Windows environment) by ignoring those 3 tests
mentioned above Iget a successful build.
was (Author: rubenql):
I confirm that (on my local Windows environment) by ignoring those 3 tests
mentioned above we get a successful build.
> PigRelExTest fails on Windows
> -
>
> Key: CALCITE-3307
> URL: https://issues.apache.org/jira/browse/CALCITE-3307
> Project: Calcite
> Issue Type: Bug
>Reporter: Julian Hyde
>Priority: Major
>
> Running {{PigRelExTest}} on Windows, after CALCITE-3122 was merged, gives the
> following error:
> {noformat}
> 2019-08-29 15:33:23,229 [main] ERROR - Failed to locate the winutils binary
> in the hadoop binary path
> java.io.IOException: Could not locate executable null\bin\winutils.exe in the
> Hadoop binaries.
> at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:382)
> at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:397)
> at org.apache.hadoop.util.Shell.(Shell.java:390)
> at org.apache.hadoop.util.StringUtils.(StringUtils.java:80)
> at
> org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611)
> at
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:273)
> at
> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:261)
> at
> org.apache.hadoop.security.UserGroupInformation.isAuthenticationMethodEnabled(UserGroupInformation.java:338)
> at
> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:332)
> at
> org.apache.pig.backend.hadoop.HKerberos.tryKerberosKeytabLogin(HKerberos.java:70)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:220)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:112)
> at org.apache.pig.impl.PigContext.connect(PigContext.java:305)
> at org.apache.pig.PigServer.(PigServer.java:231)
> at org.apache.pig.PigServer.(PigServer.java:219)
> at org.apache.pig.PigServer.(PigServer.java:211)
> at org.apache.pig.PigServer.(PigServer.java:207)
> at org.apache.calcite.piglet.PigConverter.(PigConverter.java:107)
> at org.apache.calcite.piglet.PigConverter.create(PigConverter.java:112)
> at
> org.apache.calcite.test.PigRelTestBase.testSetup(PigRelTestBase.java:34)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> {noformat}
> This issue has arisen many times; see HADOOP-10775 and SPARK-2356.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Comment Edited] (CALCITE-3307) PigRelExTest fails on Windows
[
https://issues.apache.org/jira/browse/CALCITE-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919278#comment-16919278
]
Ruben Quesada Lopez edited comment on CALCITE-3307 at 8/30/19 7:01 AM:
---
I confirm that (on my local Windows environment) by ignoring those 3 tests
mentioned above I get a successful build.
was (Author: rubenql):
I confirm that (on my local Windows environment) by ignoring those 3 tests
mentioned above Iget a successful build.
> PigRelExTest fails on Windows
> -
>
> Key: CALCITE-3307
> URL: https://issues.apache.org/jira/browse/CALCITE-3307
> Project: Calcite
> Issue Type: Bug
>Reporter: Julian Hyde
>Priority: Major
>
> Running {{PigRelExTest}} on Windows, after CALCITE-3122 was merged, gives the
> following error:
> {noformat}
> 2019-08-29 15:33:23,229 [main] ERROR - Failed to locate the winutils binary
> in the hadoop binary path
> java.io.IOException: Could not locate executable null\bin\winutils.exe in the
> Hadoop binaries.
> at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:382)
> at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:397)
> at org.apache.hadoop.util.Shell.(Shell.java:390)
> at org.apache.hadoop.util.StringUtils.(StringUtils.java:80)
> at
> org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611)
> at
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:273)
> at
> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:261)
> at
> org.apache.hadoop.security.UserGroupInformation.isAuthenticationMethodEnabled(UserGroupInformation.java:338)
> at
> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:332)
> at
> org.apache.pig.backend.hadoop.HKerberos.tryKerberosKeytabLogin(HKerberos.java:70)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:220)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:112)
> at org.apache.pig.impl.PigContext.connect(PigContext.java:305)
> at org.apache.pig.PigServer.(PigServer.java:231)
> at org.apache.pig.PigServer.(PigServer.java:219)
> at org.apache.pig.PigServer.(PigServer.java:211)
> at org.apache.pig.PigServer.(PigServer.java:207)
> at org.apache.calcite.piglet.PigConverter.(PigConverter.java:107)
> at org.apache.calcite.piglet.PigConverter.create(PigConverter.java:112)
> at
> org.apache.calcite.test.PigRelTestBase.testSetup(PigRelTestBase.java:34)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> {noformat}
> This issue has arisen many times; see HADOOP-10775 and SPARK-2356.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Comment Edited] (CALCITE-3284) Enumerable hash semijoin / antijoin support non-equi join conditions
[ https://issues.apache.org/jira/browse/CALCITE-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919274#comment-16919274 ] Ruben Quesada Lopez edited comment on CALCITE-3284 at 8/30/19 6:56 AM: --- Fixed (together with CALCITE-2973) via https://github.com/apache/calcite/commit/6cec81178e6739d9b1f3474dc5dbd7488dda2464 Thanks for the PR [~hhlai1990]! was (Author: rubenql): Fixed (together with CALCITE-2973) via https://github.com/apache/calcite/commit/6cec81178e6739d9b1f3474dc5dbd7488dda2464 Thanks for the Pr Lai Zhou! > Enumerable hash semijoin / antijoin support non-equi join conditions > > > Key: CALCITE-3284 > URL: https://issues.apache.org/jira/browse/CALCITE-3284 > Project: Calcite > Issue Type: Improvement >Reporter: Haisheng Yuan >Priority: Major > Fix For: 1.21.0 > > > Calcite should be able to generate enumerable hash semijoin / antijoin with > non-equi join conditions, as long as there are equi-join condtions, so that > we can do hash look up. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Comment Edited] (CALCITE-2973) Allow theta joins that have equi conditions to be executed using a hash join algorithm
[ https://issues.apache.org/jira/browse/CALCITE-2973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919273#comment-16919273 ] Ruben Quesada Lopez edited comment on CALCITE-2973 at 8/30/19 6:55 AM: --- Fixed via https://github.com/apache/calcite/commit/6cec81178e6739d9b1f3474dc5dbd7488dda2464 Thanks for the PR [~hhlai1990]! was (Author: rubenql): Fixed via https://github.com/apache/calcite/commit/6cec81178e6739d9b1f3474dc5dbd7488dda2464 Thanks for the Pr [~hhlai1990]! > Allow theta joins that have equi conditions to be executed using a hash join > algorithm > -- > > Key: CALCITE-2973 > URL: https://issues.apache.org/jira/browse/CALCITE-2973 > Project: Calcite > Issue Type: New Feature > Components: core >Affects Versions: 1.19.0 >Reporter: Lai Zhou >Priority: Minor > Labels: pull-request-available > Fix For: 1.21.0 > > Time Spent: 9.5h > Remaining Estimate: 0h > > Now the EnumerableMergeJoinRule only supports an inner and equi join. > If users make a theta-join query for a large dataset (such as 1*1), > the nested-loop join process will take dozens of time than the sort-merge > join process . > So if we can apply merge-join or hash-join rule for a theta join, it will > improve the performance greatly. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Resolved] (CALCITE-3284) Enumerable hash semijoin / antijoin support non-equi join conditions
[ https://issues.apache.org/jira/browse/CALCITE-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruben Quesada Lopez resolved CALCITE-3284. -- Resolution: Fixed Fixed (together with CALCITE-2973) via https://github.com/apache/calcite/commit/6cec81178e6739d9b1f3474dc5dbd7488dda2464 Thanks for the Pr Lai Zhou! > Enumerable hash semijoin / antijoin support non-equi join conditions > > > Key: CALCITE-3284 > URL: https://issues.apache.org/jira/browse/CALCITE-3284 > Project: Calcite > Issue Type: Improvement >Reporter: Haisheng Yuan >Priority: Major > Fix For: 1.21.0 > > > Calcite should be able to generate enumerable hash semijoin / antijoin with > non-equi join conditions, as long as there are equi-join condtions, so that > we can do hash look up. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Resolved] (CALCITE-2973) Allow theta joins that have equi conditions to be executed using a hash join algorithm
[ https://issues.apache.org/jira/browse/CALCITE-2973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruben Quesada Lopez resolved CALCITE-2973. -- Resolution: Fixed Fixed via https://github.com/apache/calcite/commit/6cec81178e6739d9b1f3474dc5dbd7488dda2464 Thanks for the Pr [~hhlai1990]! > Allow theta joins that have equi conditions to be executed using a hash join > algorithm > -- > > Key: CALCITE-2973 > URL: https://issues.apache.org/jira/browse/CALCITE-2973 > Project: Calcite > Issue Type: New Feature > Components: core >Affects Versions: 1.19.0 >Reporter: Lai Zhou >Priority: Minor > Labels: pull-request-available > Fix For: 1.21.0 > > Time Spent: 9.5h > Remaining Estimate: 0h > > Now the EnumerableMergeJoinRule only supports an inner and equi join. > If users make a theta-join query for a large dataset (such as 1*1), > the nested-loop join process will take dozens of time than the sort-merge > join process . > So if we can apply merge-join or hash-join rule for a theta join, it will > improve the performance greatly. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Comment Edited] (CALCITE-3122) Convert Pig Latin scripts into Calcite logical plans
[
https://issues.apache.org/jira/browse/CALCITE-3122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918630#comment-16918630
]
Ruben Quesada Lopez edited comment on CALCITE-3122 at 8/29/19 1:50 PM:
---
If I run PigRelExTest or PigRelOpTest isolated, this is the first error I get
in the logs:
{code}
2019-08-29 15:33:23,229 [main] ERROR - Failed to locate the winutils binary in
the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the
Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:382)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:397)
at org.apache.hadoop.util.Shell.(Shell.java:390)
at org.apache.hadoop.util.StringUtils.(StringUtils.java:80)
at
org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611)
at
org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:273)
at
org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:261)
at
org.apache.hadoop.security.UserGroupInformation.isAuthenticationMethodEnabled(UserGroupInformation.java:338)
at
org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:332)
at
org.apache.pig.backend.hadoop.HKerberos.tryKerberosKeytabLogin(HKerberos.java:70)
at
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:220)
at
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:112)
at org.apache.pig.impl.PigContext.connect(PigContext.java:305)
at org.apache.pig.PigServer.(PigServer.java:231)
at org.apache.pig.PigServer.(PigServer.java:219)
at org.apache.pig.PigServer.(PigServer.java:211)
at org.apache.pig.PigServer.(PigServer.java:207)
at org.apache.calcite.piglet.PigConverter.(PigConverter.java:107)
at org.apache.calcite.piglet.PigConverter.create(PigConverter.java:112)
at
org.apache.calcite.test.PigRelTestBase.testSetup(PigRelTestBase.java:34)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at
com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
at
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
{code}
And then I start getting all the InvalidPathException like:
{code}
java.nio.file.InvalidPathException: Illegal char <:> at index 4:
file:///C:/work/calcite/piglet/test
{code}
Googling the error message "{{Could not locate executable null\bin\winutils.exe
in the Hadoop binaries}}", it seems a "well-known issue".
was (Author: rubenql):
If I run PigRelExTest or PigRelOpTest isolated, this is the first error I get
in the logs:
{code}
2019-08-29 15:33:23,229 [main] ERROR - Failed to locate the winutils binary in
the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the
Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:382)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:397)
at org.apache.hadoop.util.Shell.(Shell.java:390)
at
[jira] [Commented] (CALCITE-3122) Convert Pig Latin scripts into Calcite logical plans
[
https://issues.apache.org/jira/browse/CALCITE-3122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918630#comment-16918630
]
Ruben Quesada Lopez commented on CALCITE-3122:
--
If I run PigRelExTest or PigRelOpTest isolated, this is the first error I get
in the logs:
{code}
2019-08-29 15:33:23,229 [main] ERROR - Failed to locate the winutils binary in
the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the
Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:382)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:397)
at org.apache.hadoop.util.Shell.(Shell.java:390)
at org.apache.hadoop.util.StringUtils.(StringUtils.java:80)
at
org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611)
at
org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:273)
at
org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:261)
at
org.apache.hadoop.security.UserGroupInformation.isAuthenticationMethodEnabled(UserGroupInformation.java:338)
at
org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:332)
at
org.apache.pig.backend.hadoop.HKerberos.tryKerberosKeytabLogin(HKerberos.java:70)
at
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:220)
at
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:112)
at org.apache.pig.impl.PigContext.connect(PigContext.java:305)
at org.apache.pig.PigServer.(PigServer.java:231)
at org.apache.pig.PigServer.(PigServer.java:219)
at org.apache.pig.PigServer.(PigServer.java:211)
at org.apache.pig.PigServer.(PigServer.java:207)
at org.apache.calcite.piglet.PigConverter.(PigConverter.java:107)
at org.apache.calcite.piglet.PigConverter.create(PigConverter.java:112)
at
org.apache.calcite.test.PigRelTestBase.testSetup(PigRelTestBase.java:34)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at
com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
at
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
{code}
And then I start getting all the InvalidPathException like:
{code}
java.nio.file.InvalidPathException: Illegal char <:> at index 4:
file:///C:/work/calcite/piglet/test
{code}
> Convert Pig Latin scripts into Calcite logical plans
> -
>
> Key: CALCITE-3122
> URL: https://issues.apache.org/jira/browse/CALCITE-3122
> Project: Calcite
> Issue Type: New Feature
> Components: core, piglet
>Reporter: Khai Tran
>Assignee: Julian Hyde
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.21.0
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> We create an internal Calcite repo at LinkedIn and develop APIs to parse any
> Pig Latin scripts into Calcite logical plan. The code was tested in nearly
> ~1000 Pig
[jira] [Commented] (CALCITE-3122) Convert Pig Latin scripts into Calcite logical plans
[ https://issues.apache.org/jira/browse/CALCITE-3122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918594#comment-16918594 ] Ruben Quesada Lopez commented on CALCITE-3122: -- [~zabetak] [~khaitran] I am with a local Windows environment, and I can reproduce the issue systematically. > Convert Pig Latin scripts into Calcite logical plans > - > > Key: CALCITE-3122 > URL: https://issues.apache.org/jira/browse/CALCITE-3122 > Project: Calcite > Issue Type: New Feature > Components: core, piglet >Reporter: Khai Tran >Assignee: Julian Hyde >Priority: Major > Labels: pull-request-available > Fix For: 1.21.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > We create an internal Calcite repo at LinkedIn and develop APIs to parse any > Pig Latin scripts into Calcite logical plan. The code was tested in nearly > ~1000 Pig scripts written at LinkedIn. > Changes: > 1. piglet: main conversion code live there, include: > * APIs to convert any Pig scripts into RelNode plans or SQL statements > * Use Pig Grunt parser to parse Pig Latin scripts into Pig logical plan > (DAGs) > * Convert Pig schemas into RelDatatype > * Traverse through Pig expression plan and convert Pig expressions into > RexNodes > * Map some basic Pig UDFs to Calcite SQL operators > * Build Calcite UDFs for any other Pig UDFs, including UDFs written in both > Java and Python > * Traverse (DFS) through Pig logical plans to convert each Pig logical nodes > to RelNodes > * Have an optimizer rule to optimize Pig group/cogroup into Aggregate > operators > 2. core: > * Implement other RelNode in Rel2Sql so that Pig can be translated into SQL > * Other minor changes in a few other classes to make Pig to Calcite works -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (CALCITE-3284) Enumerable hash semijoin / antijoin support non-equi join conditions
[ https://issues.apache.org/jira/browse/CALCITE-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruben Quesada Lopez updated CALCITE-3284: - Fix Version/s: 1.21.0 > Enumerable hash semijoin / antijoin support non-equi join conditions > > > Key: CALCITE-3284 > URL: https://issues.apache.org/jira/browse/CALCITE-3284 > Project: Calcite > Issue Type: Improvement >Reporter: Haisheng Yuan >Priority: Major > Fix For: 1.21.0 > > > Calcite should be able to generate enumerable hash semijoin / antijoin with > non-equi join conditions, as long as there are equi-join condtions, so that > we can do hash look up. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (CALCITE-2973) Allow theta joins that have equi conditions to be executed using a hash join algorithm
[ https://issues.apache.org/jira/browse/CALCITE-2973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruben Quesada Lopez updated CALCITE-2973: - Fix Version/s: 1.21.0 > Allow theta joins that have equi conditions to be executed using a hash join > algorithm > -- > > Key: CALCITE-2973 > URL: https://issues.apache.org/jira/browse/CALCITE-2973 > Project: Calcite > Issue Type: New Feature > Components: core >Affects Versions: 1.19.0 >Reporter: Lai Zhou >Priority: Minor > Labels: pull-request-available > Fix For: 1.21.0 > > Time Spent: 8h 20m > Remaining Estimate: 0h > > Now the EnumerableMergeJoinRule only supports an inner and equi join. > If users make a theta-join query for a large dataset (such as 1*1), > the nested-loop join process will take dozens of time than the sort-merge > join process . > So if we can apply merge-join or hash-join rule for a theta join, it will > improve the performance greatly. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (CALCITE-2973) Allow theta joins that have equi conditions to be executed using a hash join algorithm
[ https://issues.apache.org/jira/browse/CALCITE-2973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916457#comment-16916457 ] Ruben Quesada Lopez commented on CALCITE-2973: -- [~hyuan], [~danny0405], [~julianhyde], I think we could try to push this into 1.21, what do you think? > Allow theta joins that have equi conditions to be executed using a hash join > algorithm > -- > > Key: CALCITE-2973 > URL: https://issues.apache.org/jira/browse/CALCITE-2973 > Project: Calcite > Issue Type: New Feature > Components: core >Affects Versions: 1.19.0 >Reporter: Lai Zhou >Priority: Minor > Labels: pull-request-available > Time Spent: 8h 20m > Remaining Estimate: 0h > > Now the EnumerableMergeJoinRule only supports an inner and equi join. > If users make a theta-join query for a large dataset (such as 1*1), > the nested-loop join process will take dozens of time than the sort-merge > join process . > So if we can apply merge-join or hash-join rule for a theta join, it will > improve the performance greatly. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Comment Edited] (CALCITE-3284) Enumerable hash semijoin / antijoin support non-equi join conditions
[
https://issues.apache.org/jira/browse/CALCITE-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915539#comment-16915539
]
Ruben Quesada Lopez edited comment on CALCITE-3284 at 8/26/19 7:39 AM:
---
[~hhlai1990], you can create the test using RelBuilder (see
{{EnumerableJoinTest#equiAntiJoin}}, {{EnumerableJoinTest#nonEquiAntiJoin}}).
Unfortunately, I think it is not possible to create tests in SQL, since the SQL
parser does not generate joins with type SEMI/ANTI for the moment.
was (Author: rubenql):
[~hhlai1990], you can create the test using RelBuilder (see
{{EnumerableJoinTest#equiAntiJoin}}, {{EnumerableJoinTest#nonEquiAntiJoin}}).
Unfortunately, I think it is not possible to create tests in SQL, since the SQL
parse does not generate joins with type SEMI/ANTI for the moment.
> Enumerable hash semijoin / antijoin support non-equi join conditions
>
>
> Key: CALCITE-3284
> URL: https://issues.apache.org/jira/browse/CALCITE-3284
> Project: Calcite
> Issue Type: Improvement
>Reporter: Haisheng Yuan
>Priority: Major
>
> Calcite should be able to generate enumerable hash semijoin / antijoin with
> non-equi join conditions, as long as there are equi-join condtions, so that
> we can do hash look up.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (CALCITE-3284) Enumerable hash semijoin / antijoin support non-equi join conditions
[
https://issues.apache.org/jira/browse/CALCITE-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915539#comment-16915539
]
Ruben Quesada Lopez commented on CALCITE-3284:
--
[~hhlai1990], you can create the test using RelBuilder (see
{{EnumerableJoinTest#equiAntiJoin}}, {{EnumerableJoinTest#nonEquiAntiJoin}}).
Unfortunately, I think it is not possible to create tests in SQL, since the SQL
parse does not generate joins with type SEMI/ANTI for the moment.
> Enumerable hash semijoin / antijoin support non-equi join conditions
>
>
> Key: CALCITE-3284
> URL: https://issues.apache.org/jira/browse/CALCITE-3284
> Project: Calcite
> Issue Type: Improvement
>Reporter: Haisheng Yuan
>Priority: Major
>
> Calcite should be able to generate enumerable hash semijoin / antijoin with
> non-equi join conditions, as long as there are equi-join condtions, so that
> we can do hash look up.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Resolved] (CALCITE-2979) Add a batch-based nested loop join algorithm
[ https://issues.apache.org/jira/browse/CALCITE-2979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruben Quesada Lopez resolved CALCITE-2979. -- Resolution: Fixed Fixed via https://github.com/apache/calcite/commit/a8f9e4ee02c4253e1cbc150cbe3af8791a7beda8 Thanks for the PR [~khawlamhb]! > Add a batch-based nested loop join algorithm > > > Key: CALCITE-2979 > URL: https://issues.apache.org/jira/browse/CALCITE-2979 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.19.0 >Reporter: Stamatis Zampetakis >Assignee: Khawla Mouhoubi >Priority: Major > Labels: performance, pull-request-available > Fix For: 1.21.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > > Currently, Calcite provides a tuple-based nested loop join algorithm > implemented through EnumerableCorrelate and EnumerableDefaults.correlateJoin. > This means that for each tuple of the outer relation we probe (set variables) > in the inner relation. > The goal of this issue is to add new algorithm (or extend the correlateJoin > method) which first gathers blocks (batches) of tuples from the outer > relation and then probes the inner relation once per block. > There are cases (eg., indexes) where the inner relation can be accessed by > more than one value which can greatly improve the performance in particular > when the outer relation is big. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (CALCITE-3284) Enumerable hash semijoin / antijoin support non-equi join conditions
[ https://issues.apache.org/jira/browse/CALCITE-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16914001#comment-16914001 ] Ruben Quesada Lopez commented on CALCITE-3284: -- [~hyuan] thanks for creating the ticket. I have slightly modified its title and description to not forget about the antijoin (a.k.a. antisemijoin) case. > Enumerable hash semijoin / antijoin support non-equi join conditions > > > Key: CALCITE-3284 > URL: https://issues.apache.org/jira/browse/CALCITE-3284 > Project: Calcite > Issue Type: Improvement >Reporter: Haisheng Yuan >Priority: Major > > Calcite should be able to generate enumerable hash semijoin / antijoin with > non-equi join conditions, as long as there are equi-join condtions, so that > we can do hash look up. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (CALCITE-3284) Enumerable hash semijoin / antijoin support non-equi join conditions
[ https://issues.apache.org/jira/browse/CALCITE-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruben Quesada Lopez updated CALCITE-3284: - Description: Calcite should be able to generate enumerable hash semijoin / antijoin with non-equi join conditions, as long as there are equi-join condtions, so that we can do hash look up. was: Calcite should be able to generate enumerable hash semijoin with non-equi join conditions, as long as there are equi-join condtions, so that we can do hash look up. > Enumerable hash semijoin / antijoin support non-equi join conditions > > > Key: CALCITE-3284 > URL: https://issues.apache.org/jira/browse/CALCITE-3284 > Project: Calcite > Issue Type: Improvement >Reporter: Haisheng Yuan >Priority: Major > > Calcite should be able to generate enumerable hash semijoin / antijoin with > non-equi join conditions, as long as there are equi-join condtions, so that > we can do hash look up. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (CALCITE-2973) Allow theta joins that have equi conditions to be executed using a hash join algorithm
[
https://issues.apache.org/jira/browse/CALCITE-2973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913391#comment-16913391
]
Ruben Quesada Lopez commented on CALCITE-2973:
--
[~hhlai1990], thanks for this PR, I think it generally looks in a good shape. I
have one small concern though regarding the new implementation of
{{EnumerableDefaults#hashJoin_}}.
TLDR; I think this change may impact performance of RIGHT / FULL hash equi
joins.
The addition of the new predicate to support all types of join conditions (and
not just equi-joins) requires to change the {{Set unmatchedKeys}} into a
{{List innersUnmatched}}. If I understand correctly, it is required to
do so because with this change we may have two inner results with the same TKey
(which is based on the equi-condition), one being a match and the other not
being an actual match due to the new extra (non-equi) predicate. This
{{innersUnmatched}} list will be used in RIGHT / FULL joins to keep the results
from the right input which had no match, in order to finally generate the
results with null on the left. In case of INNER / LEFT join, this unmatch List
(or Set in the previous version) will not be used, so what I am about to say is
not really relevant.
The problem with the new {{List innersUnmatched}} for the RIGHT / FULL
joins is that it will be pre-filled with ALL the right input (inner) results,
and the process will remove the ones which actually had a match. Previously,
this removal was implemented via a {{HashSet#remove(TKey)}}, which has a better
performance than the new {{ArrayList#removeAll(List)}}, specially in
cases of RIGHT / FULL joins on big, big tables. This shall happen even if the
new predicate is null (i.e. we are dealing with a pure equi-join). What I am
trying to say is that, with this change, it can be expected a drop in the
performance of RIGHT / FULL hash equi joins, compared to previous calcite
versions, specially on big data volumes.
I have not make any measurements, so I'm not sure if the performance impact
will be relevant or not, so maybe I am making a big deal out of nothing.
A possible solution (although I'm not 100% convinced) could be keeping the
"old" EnumerableDefaults method for hash joins (maybe rename it as
{{hashEquiJoin_}}) and use it for pure equi joins. The new method {{hashJoin_}}
in the PR with the extra predicate will implement any type of join condition
(and will redirect to the previous one if predicate is null, i.e. if this is
actually an equi join). Pros: same performance for hash equi joins is
guaranteed. Cons: code duplicated, complex maintainability, etc.
> Allow theta joins that have equi conditions to be executed using a hash join
> algorithm
> --
>
> Key: CALCITE-2973
> URL: https://issues.apache.org/jira/browse/CALCITE-2973
> Project: Calcite
> Issue Type: New Feature
> Components: core
>Affects Versions: 1.19.0
>Reporter: Lai Zhou
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 6.5h
> Remaining Estimate: 0h
>
> Now the EnumerableMergeJoinRule only supports an inner and equi join.
> If users make a theta-join query for a large dataset (such as 1*1),
> the nested-loop join process will take dozens of time than the sort-merge
> join process .
> So if we can apply merge-join or hash-join rule for a theta join, it will
> improve the performance greatly.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Assigned] (CALCITE-2979) Add a batch-based nested loop join algorithm
[ https://issues.apache.org/jira/browse/CALCITE-2979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruben Quesada Lopez reassigned CALCITE-2979: Assignee: Ruben Quesada Lopez (was: Khawla Mouhoubi) > Add a batch-based nested loop join algorithm > > > Key: CALCITE-2979 > URL: https://issues.apache.org/jira/browse/CALCITE-2979 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.19.0 >Reporter: Stamatis Zampetakis >Assignee: Ruben Quesada Lopez >Priority: Major > Labels: performance, pull-request-available > Fix For: 1.21.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > Currently, Calcite provides a tuple-based nested loop join algorithm > implemented through EnumerableCorrelate and EnumerableDefaults.correlateJoin. > This means that for each tuple of the outer relation we probe (set variables) > in the inner relation. > The goal of this issue is to add new algorithm (or extend the correlateJoin > method) which first gathers blocks (batches) of tuples from the outer > relation and then probes the inner relation once per block. > There are cases (eg., indexes) where the inner relation can be accessed by > more than one value which can greatly improve the performance in particular > when the outer relation is big. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (CALCITE-2979) Add a batch-based nested loop join algorithm
[ https://issues.apache.org/jira/browse/CALCITE-2979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruben Quesada Lopez updated CALCITE-2979: - Fix Version/s: 1.21.0 > Add a batch-based nested loop join algorithm > > > Key: CALCITE-2979 > URL: https://issues.apache.org/jira/browse/CALCITE-2979 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.19.0 >Reporter: Stamatis Zampetakis >Assignee: Khawla Mouhoubi >Priority: Major > Labels: performance, pull-request-available > Fix For: 1.21.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > Currently, Calcite provides a tuple-based nested loop join algorithm > implemented through EnumerableCorrelate and EnumerableDefaults.correlateJoin. > This means that for each tuple of the outer relation we probe (set variables) > in the inner relation. > The goal of this issue is to add new algorithm (or extend the correlateJoin > method) which first gathers blocks (batches) of tuples from the outer > relation and then probes the inner relation once per block. > There are cases (eg., indexes) where the inner relation can be accessed by > more than one value which can greatly improve the performance in particular > when the outer relation is big. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (CALCITE-2979) Add a batch-based nested loop join algorithm
[ https://issues.apache.org/jira/browse/CALCITE-2979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruben Quesada Lopez reassigned CALCITE-2979: Assignee: Khawla Mouhoubi (was: Ruben Quesada Lopez) > Add a batch-based nested loop join algorithm > > > Key: CALCITE-2979 > URL: https://issues.apache.org/jira/browse/CALCITE-2979 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.19.0 >Reporter: Stamatis Zampetakis >Assignee: Khawla Mouhoubi >Priority: Major > Labels: performance, pull-request-available > Fix For: 1.21.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > Currently, Calcite provides a tuple-based nested loop join algorithm > implemented through EnumerableCorrelate and EnumerableDefaults.correlateJoin. > This means that for each tuple of the outer relation we probe (set variables) > in the inner relation. > The goal of this issue is to add new algorithm (or extend the correlateJoin > method) which first gathers blocks (batches) of tuples from the outer > relation and then probes the inner relation once per block. > There are cases (eg., indexes) where the inner relation can be accessed by > more than one value which can greatly improve the performance in particular > when the outer relation is big. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (CALCITE-3278) Simplify the use to translate RexNode to Expression for evaluating
[
https://issues.apache.org/jira/browse/CALCITE-3278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16912347#comment-16912347
]
Ruben Quesada Lopez commented on CALCITE-3278:
--
Please note that there is another (broader) ticket regarding
RexNode-to-Expression translation: CALCITE-3224. I'm not sure if it might
interfere or supersede this one.
> Simplify the use to translate RexNode to Expression for evaluating
> --
>
> Key: CALCITE-3278
> URL: https://issues.apache.org/jira/browse/CALCITE-3278
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Wang Yanlin
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The method *forAggregation* of *RexToLixTranslator* is designed to work for
> translating aggregate functions, with some parameters that we actually do not
> need, if we just want to translate a single RexNode.
> We lack a more common sense function to get an instance of
> RexToLixTranslator.
> And, the translated expression is a *ParameterExpression*, not fit for
> evaluating. When evaluating, we get an exception like this
> {code:java}
> java.lang.RuntimeException: parameter v not on stack
> at org.apache.calcite.linq4j.tree.Evaluator.peek(Evaluator.java:51)
> at
> org.apache.calcite.linq4j.tree.ParameterExpression.evaluate(ParameterExpression.java:55)
> at
> org.apache.calcite.linq4j.tree.GotoStatement.evaluate(GotoStatement.java:97)
> at
> org.apache.calcite.linq4j.tree.BlockStatement.evaluate(BlockStatement.java:83)
> at org.apache.calcite.linq4j.tree.Evaluator.evaluate(Evaluator.java:55)
> at
> org.apache.calcite.linq4j.tree.FunctionExpression.lambda$compile$0(FunctionExpression.java:87)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslatorTest.testRawTranslateRexNode(RexToLixTranslatorTest.java:57)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
> at
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (CALCITE-2979) Add a block-based nested loop join algorithm
[ https://issues.apache.org/jira/browse/CALCITE-2979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16912315#comment-16912315 ] Ruben Quesada Lopez commented on CALCITE-2979: -- Hi everyone, I think the PR is in a pretty good shape, if anyone has the opportunity to take a look at it, it will be very helpful. In any case, I believe we can safely push this into 1.21; even though this is a "brand new" join implementation, the risk is very limited since the new rule that generates the batch nested loop operator is not part of the "default" Calcite rule set, so this change should not break anything. > Add a block-based nested loop join algorithm > > > Key: CALCITE-2979 > URL: https://issues.apache.org/jira/browse/CALCITE-2979 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.19.0 >Reporter: Stamatis Zampetakis >Assignee: Khawla Mouhoubi >Priority: Major > Labels: performance, pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > > Currently, Calcite provides a tuple-based nested loop join algorithm > implemented through EnumerableCorrelate and EnumerableDefaults.correlateJoin. > This means that for each tuple of the outer relation we probe (set variables) > in the inner relation. > The goal of this issue is to add new algorithm (or extend the correlateJoin > method) which first gathers blocks (batches) of tuples from the outer > relation and then probes the inner relation once per block. > There are cases (eg., indexes) where the inner relation can be accessed by > more than one value which can greatly improve the performance in particular > when the outer relation is big. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (CALCITE-3226) RelBuilder doesn't keep the alias when scan from an expanded view
[
https://issues.apache.org/jira/browse/CALCITE-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903632#comment-16903632
]
Ruben Quesada Lopez commented on CALCITE-3226:
--
[~julianhyde], correct me if you are wrong, the test that you are requesting is:
{quote}the case where the view is “select * from t”, I.e. a single TableScan.
You code assumes that if the result of the expansion is a table scan expansion
did not occur.
{quote}
I think that the merged solution does not assume that. My understanding is
that, if the result of a expansion is a table scan, the expansion did occur,
and the alias would have already been correctly set, no need to explicitly set
the alias using {{RelBuilder#as}}. Apart from that, it would seem that this
scenario is not actually possible, because expansion always adds a Project as
last step, so I am not so sure how we could force that test case. Do you
confirm [~jinxing6...@126.com]?
> RelBuilder doesn't keep the alias when scan from an expanded view
> -
>
> Key: CALCITE-3226
> URL: https://issues.apache.org/jira/browse/CALCITE-3226
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: jin xing
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.21.0
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> In current implementation of {{RelBuilder::scan}}
> ([https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L1048]),
> the alias can be derived and recorded into {{Frame}} only when the
> {{RelNode}} is a {{TableScan}}
> ([https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L2754]).
> But when {{RelBuilder::scan}} from an expanded view, the node is not a
> {{TableScan}} and the alias is not kept. Below test failed - we cannot
> reference a field by alias "MYVIEW".
> {code:java}
> @Test public void testExpandViewShouldKeepAlias() throws SQLException {
> try (Connection connection =
> DriverManager.getConnection("jdbc:calcite:")) {
> final Frameworks.ConfigBuilder configBuilder =
> expandingConfig(connection);
> final RelOptTable.ViewExpander viewExpander =
> (RelOptTable.ViewExpander)
> Frameworks.getPlanner(configBuilder.build());
> final RelFactories.TableScanFactory tableScanFactory =
> RelFactories.expandingScanFactory(viewExpander,
> RelFactories.DEFAULT_TABLE_SCAN_FACTORY);
> configBuilder.context(Contexts.of(tableScanFactory));
> final RelBuilder builder = RelBuilder.create(configBuilder.build());
> RelNode node =
> builder.scan("MYVIEW")
> .project(
> builder.field(1, "MYVIEW", "EMPNO"), // Exception
> thrown from here.
> builder.field(1, "MYVIEW", "ENAME"))
> .build();
> String expected =
> "LogicalProject(EMPNO=[$0], ENAME=[$1])\n"
> + " LogicalFilter(condition=[=(1, 1)])\n"
> + "LogicalTableScan(table=[[scott, EMP]])\n";
> assertThat(node, hasTree(expected));
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Comment Edited] (CALCITE-3198) Enhance RexSimplify to handle (x<>a or x<>b)
[
https://issues.apache.org/jira/browse/CALCITE-3198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903332#comment-16903332
]
Ruben Quesada Lopez edited comment on CALCITE-3198 at 8/8/19 9:49 PM:
--
Ooops, my bad. Fixed via
https://github.com/apache/calcite/commit/fb797bbf428a0a949a792b3bc01b612049cb22fc
was (Author: rubenql):
Ooops, my bad. Fixing it via https://github.com/apache/calcite/pull/1365
> Enhance RexSimplify to handle (x<>a or x<>b)
>
>
> Key: CALCITE-3198
> URL: https://issues.apache.org/jira/browse/CALCITE-3198
> Project: Calcite
> Issue Type: Improvement
>Affects Versions: 1.20.0
>Reporter: Ruben Quesada Lopez
>Assignee: Ruben Quesada Lopez
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.21.0
>
> Time Spent: 7h 50m
> Remaining Estimate: 0h
>
> Currently, ReduceExpressionsRule.FILTER_INSTANCE can successfully reduce a
> query like this one (see RelOptRulesTest#testReduceConstantsDup):
> {code}
> // query:
> select d.deptno from dept d where d.deptno=7 and d.deptno=8
> // plan before:
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[AND(=($0, 7), =($0, 8))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after:
> LogicalProject(DEPTNO=[$0])
> LogicalValues(tuples=[[]])
> {code}
> As we can see, since the filter is 'always false', the
> LogicalTableScan+LogicalFilter are correctly replaced by an empty
> LogicalValues.
> However, the same filter with a NOT expression, is not correctly simplified:
> {code}
> // query:
> select d.deptno from dept d where not(d.deptno=7 and d.deptno=8)
> // plan before:
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[NOT(AND(=($0, 7), =($0, 8)))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after (actual, NOT distributivity for AND):
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[OR(<>($0, 7), <>($0, 8))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after (expected, when $0's type is NOT nullable, filter removed):
> LogicalProject(DEPTNO=[$0])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after (expected, when $0's type IS nullable, filter simplified as IS
> NOT NULL):
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[IS NOT NULL($0)])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Comment Edited] (CALCITE-3198) Enhance RexSimplify to handle (x<>a or x<>b)
[
https://issues.apache.org/jira/browse/CALCITE-3198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903332#comment-16903332
]
Ruben Quesada Lopez edited comment on CALCITE-3198 at 8/8/19 9:16 PM:
--
Ooops, my bad. Fixing it via https://github.com/apache/calcite/pull/1365
was (Author: rubenql):
Ooops, my bad. Fixing it...
> Enhance RexSimplify to handle (x<>a or x<>b)
>
>
> Key: CALCITE-3198
> URL: https://issues.apache.org/jira/browse/CALCITE-3198
> Project: Calcite
> Issue Type: Improvement
>Affects Versions: 1.20.0
>Reporter: Ruben Quesada Lopez
>Assignee: Ruben Quesada Lopez
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.21.0
>
> Time Spent: 7h 40m
> Remaining Estimate: 0h
>
> Currently, ReduceExpressionsRule.FILTER_INSTANCE can successfully reduce a
> query like this one (see RelOptRulesTest#testReduceConstantsDup):
> {code}
> // query:
> select d.deptno from dept d where d.deptno=7 and d.deptno=8
> // plan before:
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[AND(=($0, 7), =($0, 8))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after:
> LogicalProject(DEPTNO=[$0])
> LogicalValues(tuples=[[]])
> {code}
> As we can see, since the filter is 'always false', the
> LogicalTableScan+LogicalFilter are correctly replaced by an empty
> LogicalValues.
> However, the same filter with a NOT expression, is not correctly simplified:
> {code}
> // query:
> select d.deptno from dept d where not(d.deptno=7 and d.deptno=8)
> // plan before:
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[NOT(AND(=($0, 7), =($0, 8)))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after (actual, NOT distributivity for AND):
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[OR(<>($0, 7), <>($0, 8))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after (expected, when $0's type is NOT nullable, filter removed):
> LogicalProject(DEPTNO=[$0])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after (expected, when $0's type IS nullable, filter simplified as IS
> NOT NULL):
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[IS NOT NULL($0)])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (CALCITE-3198) Enhance RexSimplify to handle (x<>a or x<>b)
[
https://issues.apache.org/jira/browse/CALCITE-3198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903332#comment-16903332
]
Ruben Quesada Lopez commented on CALCITE-3198:
--
Ooops, my bad. Fixing it...
> Enhance RexSimplify to handle (x<>a or x<>b)
>
>
> Key: CALCITE-3198
> URL: https://issues.apache.org/jira/browse/CALCITE-3198
> Project: Calcite
> Issue Type: Improvement
>Affects Versions: 1.20.0
>Reporter: Ruben Quesada Lopez
>Assignee: Ruben Quesada Lopez
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.21.0
>
> Time Spent: 7.5h
> Remaining Estimate: 0h
>
> Currently, ReduceExpressionsRule.FILTER_INSTANCE can successfully reduce a
> query like this one (see RelOptRulesTest#testReduceConstantsDup):
> {code}
> // query:
> select d.deptno from dept d where d.deptno=7 and d.deptno=8
> // plan before:
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[AND(=($0, 7), =($0, 8))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after:
> LogicalProject(DEPTNO=[$0])
> LogicalValues(tuples=[[]])
> {code}
> As we can see, since the filter is 'always false', the
> LogicalTableScan+LogicalFilter are correctly replaced by an empty
> LogicalValues.
> However, the same filter with a NOT expression, is not correctly simplified:
> {code}
> // query:
> select d.deptno from dept d where not(d.deptno=7 and d.deptno=8)
> // plan before:
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[NOT(AND(=($0, 7), =($0, 8)))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after (actual, NOT distributivity for AND):
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[OR(<>($0, 7), <>($0, 8))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after (expected, when $0's type is NOT nullable, filter removed):
> LogicalProject(DEPTNO=[$0])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after (expected, when $0's type IS nullable, filter simplified as IS
> NOT NULL):
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[IS NOT NULL($0)])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (CALCITE-3226) RelBuilder doesn't keep the alias when scan from an expanded view
[
https://issues.apache.org/jira/browse/CALCITE-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903242#comment-16903242
]
Ruben Quesada Lopez commented on CALCITE-3226:
--
Sorry, [~julianhyde]. As described by [~jinxing6...@126.com], it seems it is
not possible to get to that scenario, because the view always produce a
{{Project+Scan}}, even in case of a simple {{“select * from t”}}.
> RelBuilder doesn't keep the alias when scan from an expanded view
> -
>
> Key: CALCITE-3226
> URL: https://issues.apache.org/jira/browse/CALCITE-3226
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: jin xing
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.21.0
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> In current implementation of {{RelBuilder::scan}}
> ([https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L1048]),
> the alias can be derived and recorded into {{Frame}} only when the
> {{RelNode}} is a {{TableScan}}
> ([https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L2754]).
> But when {{RelBuilder::scan}} from an expanded view, the node is not a
> {{TableScan}} and the alias is not kept. Below test failed - we cannot
> reference a field by alias "MYVIEW".
> {code:java}
> @Test public void testExpandViewShouldKeepAlias() throws SQLException {
> try (Connection connection =
> DriverManager.getConnection("jdbc:calcite:")) {
> final Frameworks.ConfigBuilder configBuilder =
> expandingConfig(connection);
> final RelOptTable.ViewExpander viewExpander =
> (RelOptTable.ViewExpander)
> Frameworks.getPlanner(configBuilder.build());
> final RelFactories.TableScanFactory tableScanFactory =
> RelFactories.expandingScanFactory(viewExpander,
> RelFactories.DEFAULT_TABLE_SCAN_FACTORY);
> configBuilder.context(Contexts.of(tableScanFactory));
> final RelBuilder builder = RelBuilder.create(configBuilder.build());
> RelNode node =
> builder.scan("MYVIEW")
> .project(
> builder.field(1, "MYVIEW", "EMPNO"), // Exception
> thrown from here.
> builder.field(1, "MYVIEW", "ENAME"))
> .build();
> String expected =
> "LogicalProject(EMPNO=[$0], ENAME=[$1])\n"
> + " LogicalFilter(condition=[=(1, 1)])\n"
> + "LogicalTableScan(table=[[scott, EMP]])\n";
> assertThat(node, hasTree(expected));
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Resolved] (CALCITE-3226) RelBuilder doesn't keep the alias when scan from an expanded view
[
https://issues.apache.org/jira/browse/CALCITE-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Quesada Lopez resolved CALCITE-3226.
--
Resolution: Fixed
Fix Version/s: 1.21.0
Fixed via
https://github.com/apache/calcite/commit/2678eb9a867c6589208b60c2a07446c658d2d8c6
Thanks for the PR [~jinxing6...@126.com]!
> RelBuilder doesn't keep the alias when scan from an expanded view
> -
>
> Key: CALCITE-3226
> URL: https://issues.apache.org/jira/browse/CALCITE-3226
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: jin xing
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.21.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> In current implementation of {{RelBuilder::scan}}
> ([https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L1048]),
> the alias can be derived and recorded into {{Frame}} only when the
> {{RelNode}} is a {{TableScan}}
> ([https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L2754]).
> But when {{RelBuilder::scan}} from an expanded view, the node is not a
> {{TableScan}} and the alias is not kept. Below test failed - we cannot
> reference a field by alias "MYVIEW".
> {code:java}
> @Test public void testExpandViewShouldKeepAlias() throws SQLException {
> try (Connection connection =
> DriverManager.getConnection("jdbc:calcite:")) {
> final Frameworks.ConfigBuilder configBuilder =
> expandingConfig(connection);
> final RelOptTable.ViewExpander viewExpander =
> (RelOptTable.ViewExpander)
> Frameworks.getPlanner(configBuilder.build());
> final RelFactories.TableScanFactory tableScanFactory =
> RelFactories.expandingScanFactory(viewExpander,
> RelFactories.DEFAULT_TABLE_SCAN_FACTORY);
> configBuilder.context(Contexts.of(tableScanFactory));
> final RelBuilder builder = RelBuilder.create(configBuilder.build());
> RelNode node =
> builder.scan("MYVIEW")
> .project(
> builder.field(1, "MYVIEW", "EMPNO"), // Exception
> thrown from here.
> builder.field(1, "MYVIEW", "ENAME"))
> .build();
> String expected =
> "LogicalProject(EMPNO=[$0], ENAME=[$1])\n"
> + " LogicalFilter(condition=[=(1, 1)])\n"
> + "LogicalTableScan(table=[[scott, EMP]])\n";
> assertThat(node, hasTree(expected));
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Resolved] (CALCITE-3198) Enhance RexSimplify to handle (x<>a or x<>b)
[
https://issues.apache.org/jira/browse/CALCITE-3198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Quesada Lopez resolved CALCITE-3198.
--
Resolution: Fixed
Fixed via
https://github.com/apache/calcite/commit/7f5b313bbc0f98379c1b03a3fb08f296143d64c2
> Enhance RexSimplify to handle (x<>a or x<>b)
>
>
> Key: CALCITE-3198
> URL: https://issues.apache.org/jira/browse/CALCITE-3198
> Project: Calcite
> Issue Type: Improvement
>Affects Versions: 1.20.0
>Reporter: Ruben Quesada Lopez
>Assignee: Ruben Quesada Lopez
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.21.0
>
> Time Spent: 7.5h
> Remaining Estimate: 0h
>
> Currently, ReduceExpressionsRule.FILTER_INSTANCE can successfully reduce a
> query like this one (see RelOptRulesTest#testReduceConstantsDup):
> {code}
> // query:
> select d.deptno from dept d where d.deptno=7 and d.deptno=8
> // plan before:
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[AND(=($0, 7), =($0, 8))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after:
> LogicalProject(DEPTNO=[$0])
> LogicalValues(tuples=[[]])
> {code}
> As we can see, since the filter is 'always false', the
> LogicalTableScan+LogicalFilter are correctly replaced by an empty
> LogicalValues.
> However, the same filter with a NOT expression, is not correctly simplified:
> {code}
> // query:
> select d.deptno from dept d where not(d.deptno=7 and d.deptno=8)
> // plan before:
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[NOT(AND(=($0, 7), =($0, 8)))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after (actual, NOT distributivity for AND):
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[OR(<>($0, 7), <>($0, 8))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after (expected, when $0's type is NOT nullable, filter removed):
> LogicalProject(DEPTNO=[$0])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after (expected, when $0's type IS nullable, filter simplified as IS
> NOT NULL):
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[IS NOT NULL($0)])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Resolved] (CALCITE-2624) Add a rule to copy a sort below a join operator
[ https://issues.apache.org/jira/browse/CALCITE-2624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruben Quesada Lopez resolved CALCITE-2624. -- Resolution: Fixed Fixed via https://github.com/apache/calcite/commit/80cc4253a3a5b0677382725d7471ed1163cc204e Thanks for the PR [~khawlamhb]! > Add a rule to copy a sort below a join operator > --- > > Key: CALCITE-2624 > URL: https://issues.apache.org/jira/browse/CALCITE-2624 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.17.0 >Reporter: Stamatis Zampetakis >Assignee: Khawla Mouhoubi >Priority: Major > Labels: pull-request-available > Fix For: 1.21.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Currently, the only rule that allows a sort to traverse a binary operator is > the SortJoinTransposeRule. The rule was introduced mainly to push limits in > the case of left and right outer joins (see CALCITE-831). > I assume that the main reason that we don't have more rules is that sorts > with limits and offsets cannot be pushed safely below many types of join > operators. However, in many cases, it is possible and beneficial for > optimization purposes to just push the sort without the limit and offset. > Since we do not know in advance if the join operator preserves the order we > cannot remove (that is why I am saying copy and not transpose) the sort > operator on top of the join. The latter is not really a problem since the > SortRemoveRule can detect such cases and remove the sort if it is redundant. > A few concrete examples where this optimization makes sense are outlined > below: > * allow the sort to be later absorbed by an index scan and disappear from > the plan (Sort + Tablescan => IndexScan with RelCollation); > * allow operators that require sorted inputs to be exploited more easily > (e.g., merge join); > * allow the sort to be performed on a possibly smaller result (assuming that > the physical binary operator that is going to be used preserves the order of > left/right input and the top sort operator can be removed entirely). > I propose to add a new rule (e.g., SortCopyBelowJoinRule, > SortJoinCopyBelowRule) which allows a sort to be copied to the left or right > (or to both if it is rather easy to decompose the sort) of a join operator > (excluding the limit and offset attributes) if the respective inputs are not > already sorted. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (CALCITE-2624) Add a rule to copy a sort below a join operator
[ https://issues.apache.org/jira/browse/CALCITE-2624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899478#comment-16899478 ] Ruben Quesada Lopez commented on CALCITE-2624: -- [~zabetak], I think we can push it to 1.21, as you suggested, just adding the rule and its tests, without including to the default ruleset of Calcite. I'll take another look and the PR and merge it in the following days. > Add a rule to copy a sort below a join operator > --- > > Key: CALCITE-2624 > URL: https://issues.apache.org/jira/browse/CALCITE-2624 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.17.0 >Reporter: Stamatis Zampetakis >Assignee: Khawla Mouhoubi >Priority: Major > Labels: pull-request-available > Fix For: 1.21.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Currently, the only rule that allows a sort to traverse a binary operator is > the SortJoinTransposeRule. The rule was introduced mainly to push limits in > the case of left and right outer joins (see CALCITE-831). > I assume that the main reason that we don't have more rules is that sorts > with limits and offsets cannot be pushed safely below many types of join > operators. However, in many cases, it is possible and beneficial for > optimization purposes to just push the sort without the limit and offset. > Since we do not know in advance if the join operator preserves the order we > cannot remove (that is why I am saying copy and not transpose) the sort > operator on top of the join. The latter is not really a problem since the > SortRemoveRule can detect such cases and remove the sort if it is redundant. > A few concrete examples where this optimization makes sense are outlined > below: > * allow the sort to be later absorbed by an index scan and disappear from > the plan (Sort + Tablescan => IndexScan with RelCollation); > * allow operators that require sorted inputs to be exploited more easily > (e.g., merge join); > * allow the sort to be performed on a possibly smaller result (assuming that > the physical binary operator that is going to be used preserves the order of > left/right input and the top sort operator can be removed entirely). > I propose to add a new rule (e.g., SortCopyBelowJoinRule, > SortJoinCopyBelowRule) which allows a sort to be copied to the left or right > (or to both if it is rather easy to decompose the sort) of a join operator > (excluding the limit and offset attributes) if the respective inputs are not > already sorted. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Resolved] (CALCITE-3225) JoinToMultiJoinRule should not match SEMI/ANTI LogicalJoin
[ https://issues.apache.org/jira/browse/CALCITE-3225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruben Quesada Lopez resolved CALCITE-3225. -- Resolution: Fixed Fixed via https://github.com/apache/calcite/commit/f82353c80223143673fc78561a05c79d6fd4f364 Thanks for the PR [~godfreyhe]! > JoinToMultiJoinRule should not match SEMI/ANTI LogicalJoin > -- > > Key: CALCITE-3225 > URL: https://issues.apache.org/jira/browse/CALCITE-3225 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.20.0 >Reporter: godfrey he >Priority: Critical > Labels: pull-request-available > Fix For: 1.21.0 > > Time Spent: 20m > Remaining Estimate: 0h > > this bug is found in flink: > [FLINK-13545|https://issues.apache.org/jira/browse/FLINK-13545] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (CALCITE-3198) Enhance RexSimplify to handle (x<>a or x<>b)
[
https://issues.apache.org/jira/browse/CALCITE-3198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Quesada Lopez updated CALCITE-3198:
-
Summary: Enhance RexSimplify to handle (x<>a or x<>b) (was:
ReduceExpressionsRule.FILTER_INSTANCE does not reduce 'NOT(x=a AND x=b)')
> Enhance RexSimplify to handle (x<>a or x<>b)
>
>
> Key: CALCITE-3198
> URL: https://issues.apache.org/jira/browse/CALCITE-3198
> Project: Calcite
> Issue Type: Improvement
>Affects Versions: 1.20.0
>Reporter: Ruben Quesada Lopez
>Assignee: Ruben Quesada Lopez
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.21.0
>
> Time Spent: 6h 40m
> Remaining Estimate: 0h
>
> Currently, ReduceExpressionsRule.FILTER_INSTANCE can successfully reduce a
> query like this one (see RelOptRulesTest#testReduceConstantsDup):
> {code}
> // query:
> select d.deptno from dept d where d.deptno=7 and d.deptno=8
> // plan before:
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[AND(=($0, 7), =($0, 8))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after:
> LogicalProject(DEPTNO=[$0])
> LogicalValues(tuples=[[]])
> {code}
> As we can see, since the filter is 'always false', the
> LogicalTableScan+LogicalFilter are correctly replaced by an empty
> LogicalValues.
> However, the same filter with a NOT expression, is not correctly simplified:
> {code}
> // query:
> select d.deptno from dept d where not(d.deptno=7 and d.deptno=8)
> // plan before:
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[NOT(AND(=($0, 7), =($0, 8)))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after (actual, NOT distributivity for AND):
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[OR(<>($0, 7), <>($0, 8))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after (expected, when $0's type is NOT nullable, filter removed):
> LogicalProject(DEPTNO=[$0])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after (expected, when $0's type IS nullable, filter simplified as IS
> NOT NULL):
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[IS NOT NULL($0)])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (CALCITE-3198) ReduceExpressionsRule.FILTER_INSTANCE does not reduce 'NOT(x=a AND x=b)'
[
https://issues.apache.org/jira/browse/CALCITE-3198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Quesada Lopez updated CALCITE-3198:
-
Issue Type: Improvement (was: Bug)
> ReduceExpressionsRule.FILTER_INSTANCE does not reduce 'NOT(x=a AND x=b)'
>
>
> Key: CALCITE-3198
> URL: https://issues.apache.org/jira/browse/CALCITE-3198
> Project: Calcite
> Issue Type: Improvement
>Affects Versions: 1.20.0
>Reporter: Ruben Quesada Lopez
>Assignee: Ruben Quesada Lopez
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.21.0
>
> Time Spent: 6h 40m
> Remaining Estimate: 0h
>
> Currently, ReduceExpressionsRule.FILTER_INSTANCE can successfully reduce a
> query like this one (see RelOptRulesTest#testReduceConstantsDup):
> {code}
> // query:
> select d.deptno from dept d where d.deptno=7 and d.deptno=8
> // plan before:
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[AND(=($0, 7), =($0, 8))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after:
> LogicalProject(DEPTNO=[$0])
> LogicalValues(tuples=[[]])
> {code}
> As we can see, since the filter is 'always false', the
> LogicalTableScan+LogicalFilter are correctly replaced by an empty
> LogicalValues.
> However, the same filter with a NOT expression, is not correctly simplified:
> {code}
> // query:
> select d.deptno from dept d where not(d.deptno=7 and d.deptno=8)
> // plan before:
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[NOT(AND(=($0, 7), =($0, 8)))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after (actual, NOT distributivity for AND):
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[OR(<>($0, 7), <>($0, 8))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after (expected, when $0's type is NOT nullable, filter removed):
> LogicalProject(DEPTNO=[$0])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after (expected, when $0's type IS nullable, filter simplified as IS
> NOT NULL):
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[IS NOT NULL($0)])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (CALCITE-481) Add "Spool" operator, to allow re-use of relational expressions
[ https://issues.apache.org/jira/browse/CALCITE-481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16898622#comment-16898622 ] Ruben Quesada Lopez commented on CALCITE-481: - Ok, [~julianhyde]. > Add "Spool" operator, to allow re-use of relational expressions > --- > > Key: CALCITE-481 > URL: https://issues.apache.org/jira/browse/CALCITE-481 > Project: Calcite > Issue Type: Bug >Reporter: Julian Hyde >Assignee: Jesus Camacho Rodriguez >Priority: Major > Fix For: 1.21.0 > > > If a sub-tree occurs more than once in a query an efficient plan would > probably evaluate once and have two readers read the same data. We propose a > "Spool" relational expression for this purpose. > Spool would have one input, the expression that populates it. > In the VolcanoPlanner, any RelNode can already have multiple consumers (each > of which sees the same row type and the same data) but an optimal plan does > not typically include multiple uses of the same node, so most implementors > (e.g. EnumerableRelImplementor) would just not notice, and generate the same > code twice. Having an explicit Spool would alert the implementor to re-use > the result. > We do not prescribe a mechanism for implementing Spool as a physical > operator. A job that populates a temporary table is one possible mechanism. > As part of this case, we should implement Spool in Enumerable convention, and > use it to evaluate some test queries. > The other reason to implement Spool is costing. The cost of a Spool with N > consumers is typically something like A + B . N. A, the fixed cost, is > significantly larger than B, the re-play cost. > Volcano's dynamic programming model does not make it easy to account for > re-use. There are approaches in academia based on integer linear programming; > see e.g. http://www.slideshare.net/INRIA-OAK/plreuse and > https://hal.inria.fr/hal-01353891/document. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Resolved] (CALCITE-3211) List of MutableRel may fail to be identified by SubstitutionVisitor during matching
[
https://issues.apache.org/jira/browse/CALCITE-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Quesada Lopez resolved CALCITE-3211.
--
Resolution: Fixed
Fixed in
https://github.com/apache/calcite/commit/d98856bf1a5f5c151d004b769e14bdd368a67234
Thanks for the PR [~jinxing6...@126.com]!
> List of MutableRel may fail to be identified by SubstitutionVisitor during
> matching
> ---
>
> Key: CALCITE-3211
> URL: https://issues.apache.org/jira/browse/CALCITE-3211
> Project: Calcite
> Issue Type: Bug
>Reporter: jin xing
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.21.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Current implementation of {{MutableRels::toMutables}} is as below:
> {code:java}
> private static List toMutables(List nodes) {
> return Lists.transform(nodes, MutableRels::toMutable);
> }
> {code}
> Thus every time we {{get}} from the result list, a new {{MutableRel}} will be
> created:
> {code:java}
> private static class TransformingRandomAccessList extends
> AbstractList
> implements RandomAccess, Serializable {
> final List fromList;
> final Function function;
> TransformingRandomAccessList(List fromList, Function extends T> function) {
> this.fromList = checkNotNull(fromList);
> this.function = checkNotNull(function);
> }
> @Override
> public T get(int index) {
> return function.apply(fromList.get(index));
> }
> ..
> {code}
> As a result,
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/plan/SubstitutionVisitor.java#L514
>{{SubstitutionVisitor}} will fail to check whether a node has been
> met/matched before
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (CALCITE-3211) List of MutableRel may fail to be identified by SubstitutionVisitor during matching
[
https://issues.apache.org/jira/browse/CALCITE-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Quesada Lopez updated CALCITE-3211:
-
Fix Version/s: 1.21.0
> List of MutableRel may fail to be identified by SubstitutionVisitor during
> matching
> ---
>
> Key: CALCITE-3211
> URL: https://issues.apache.org/jira/browse/CALCITE-3211
> Project: Calcite
> Issue Type: Bug
>Reporter: jin xing
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.21.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Current implementation of {{MutableRels::toMutables}} is as below:
> {code:java}
> private static List toMutables(List nodes) {
> return Lists.transform(nodes, MutableRels::toMutable);
> }
> {code}
> Thus every time we {{get}} from the result list, a new {{MutableRel}} will be
> created:
> {code:java}
> private static class TransformingRandomAccessList extends
> AbstractList
> implements RandomAccess, Serializable {
> final List fromList;
> final Function function;
> TransformingRandomAccessList(List fromList, Function extends T> function) {
> this.fromList = checkNotNull(fromList);
> this.function = checkNotNull(function);
> }
> @Override
> public T get(int index) {
> return function.apply(fromList.get(index));
> }
> ..
> {code}
> As a result,
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/plan/SubstitutionVisitor.java#L514
>{{SubstitutionVisitor}} will fail to check whether a node has been
> met/matched before
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Resolved] (CALCITE-3109) Improvements on algebraic operators to express recursive queries (RepeatUnion & TableSpool)
[
https://issues.apache.org/jira/browse/CALCITE-3109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Quesada Lopez resolved CALCITE-3109.
--
Resolution: Fixed
Fixed in
https://github.com/apache/calcite/commit/c1c4e31c849c4c9cb73056be95f5694ccd5fc2db
> Improvements on algebraic operators to express recursive queries (RepeatUnion
> & TableSpool)
> ---
>
> Key: CALCITE-3109
> URL: https://issues.apache.org/jira/browse/CALCITE-3109
> Project: Calcite
> Issue Type: Task
>Reporter: Ruben Quesada Lopez
>Assignee: Ruben Quesada Lopez
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.21.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> After the implementation of CALCITE-2812, there are some pending details to
> be addressed in order to make the RepeatUnion and TableSpool more consistent.
> TableSpool:
> - Replace {{String tableName}} with a {{Table}} (perhaps wrapped in a
> {{RelOptTable}}).
> RepeatUnion:
> - Rename {{maxRep}} as {{iterationLimit}}. Change this field to allow all
> integer values, and treat all negative values as "no limit".
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Resolved] (CALCITE-3209) When calling MutableMultiRel::setInput, exception thrown
[
https://issues.apache.org/jira/browse/CALCITE-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Quesada Lopez resolved CALCITE-3209.
--
Resolution: Fixed
Fixed in
https://github.com/apache/calcite/commit/9ba140ef66ad1e048789532211ddf8b3ca1cf8ec
Thanks for the PR [~jinxing6...@126.com]!
> When calling MutableMultiRel::setInput, exception thrown
>
>
> Key: CALCITE-3209
> URL: https://issues.apache.org/jira/browse/CALCITE-3209
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: jin xing
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.21.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> When digging code of materialization matching. I found the current
> implementation of {{MutableMultiRel::}}{{inputs}} is by an {{ImmutableList}},
> thus impossible to be changed by
>
> [https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/rel/mutable/MutableMultiRel.java#L42]
> {code:java}
> @Override public void setInput(int ordinalInParent, MutableRel input)
> {code}
> Seems a mistake.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (CALCITE-3209) When calling MutableMultiRel::setInput, exception thrown
[
https://issues.apache.org/jira/browse/CALCITE-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Quesada Lopez updated CALCITE-3209:
-
Fix Version/s: 1.21.0
> When calling MutableMultiRel::setInput, exception thrown
>
>
> Key: CALCITE-3209
> URL: https://issues.apache.org/jira/browse/CALCITE-3209
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: jin xing
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.21.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> When digging code of materialization matching. I found the current
> implementation of {{MutableMultiRel::}}{{inputs}} is by an {{ImmutableList}},
> thus impossible to be changed by
>
> [https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/rel/mutable/MutableMultiRel.java#L42]
> {code:java}
> @Override public void setInput(int ordinalInParent, MutableRel input)
> {code}
> Seems a mistake.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Comment Edited] (CALCITE-3198) ReduceExpressionsRule.FILTER_INSTANCE does not reduce 'NOT(x=a AND x=b)'
[
https://issues.apache.org/jira/browse/CALCITE-3198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16885045#comment-16885045
]
Ruben Quesada Lopez edited comment on CALCITE-3198 at 7/23/19 7:26 AM:
---
Same result is obtained with the following test, which is maybe more
straightforward (and by solving this, the one in the description might be also
solved as a side effect):
{code}
// query:
select d.deptno from dept d where d.deptno<>7 or d.deptno<>8
// plan before:
LogicalProject(DEPTNO=[$0])
LogicalFilter(condition=[OR(<>($0, 7), <>($0, 8))])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
// plan after (actual, unchanged):
LogicalProject(DEPTNO=[$0])
LogicalFilter(condition=[OR(<>($0, 7), <>($0, 8))])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
// plan after (expected, when $0's type is NOT nullable, filter removed):
LogicalProject(DEPTNO=[$0])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
// plan after (expected, when $0's type IS nullable, filter simplified as IS
NOT NULL):
LogicalProject(DEPTNO=[$0])
LogicalFilter(condition=[IS NOT NULL($0)])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
{code}
was (Author: rubenql):
Same result is obtained with the following test, which is maybe more
straightforward (and by solving this, the one in the description might be also
solved as a side effect):
{code}
// query:
select d.deptno from dept d where d.deptno<>7 or d.deptno<>8
// plan before:
LogicalProject(DEPTNO=[$0])
LogicalFilter(condition=[OR(<>($0, 7), <>($0, 8))])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
// plan after (actual, unchanged):
LogicalProject(DEPTNO=[$0])
LogicalFilter(condition=[OR(<>($0, 7), <>($0, 8))])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
// plan after (expected, when $0's type is NOT nullable, filter removed):
LogicalProject(DEPTNO=[$0])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
// plan after (expected, when $0's type IS nullable, filter simplified as IS
NOT NULL):
LogicalProject(DEPTNO=[$0])
LogicalFilter(condition=[$0 IS NOT NULL])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
{code}
> ReduceExpressionsRule.FILTER_INSTANCE does not reduce 'NOT(x=a AND x=b)'
>
>
> Key: CALCITE-3198
> URL: https://issues.apache.org/jira/browse/CALCITE-3198
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.20.0
>Reporter: Ruben Quesada Lopez
>Assignee: Ruben Quesada Lopez
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.21.0
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> Currently, ReduceExpressionsRule.FILTER_INSTANCE can successfully reduce a
> query like this one (see RelOptRulesTest#testReduceConstantsDup):
> {code}
> // query:
> select d.deptno from dept d where d.deptno=7 and d.deptno=8
> // plan before:
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[AND(=($0, 7), =($0, 8))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after:
> LogicalProject(DEPTNO=[$0])
> LogicalValues(tuples=[[]])
> {code}
> As we can see, since the filter is 'always false', the
> LogicalTableScan+LogicalFilter are correctly replaced by an empty
> LogicalValues.
> However, the same filter with a NOT expression, is not correctly simplified:
> {code}
> // query:
> select d.deptno from dept d where not(d.deptno=7 and d.deptno=8)
> // plan before:
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[NOT(AND(=($0, 7), =($0, 8)))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after (actual, NOT distributivity for AND):
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[OR(<>($0, 7), <>($0, 8))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after (expected, when $0's type is NOT nullable, filter removed):
> LogicalProject(DEPTNO=[$0])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after (expected, when $0's type IS nullable, filter simplified as IS
> NOT NULL):
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[IS NOT NULL($0)])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (CALCITE-3198) ReduceExpressionsRule.FILTER_INSTANCE does not reduce 'NOT(x=a AND x=b)'
[
https://issues.apache.org/jira/browse/CALCITE-3198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Quesada Lopez updated CALCITE-3198:
-
Description:
Currently, ReduceExpressionsRule.FILTER_INSTANCE can successfully reduce a
query like this one (see RelOptRulesTest#testReduceConstantsDup):
{code}
// query:
select d.deptno from dept d where d.deptno=7 and d.deptno=8
// plan before:
LogicalProject(DEPTNO=[$0])
LogicalFilter(condition=[AND(=($0, 7), =($0, 8))])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
// plan after:
LogicalProject(DEPTNO=[$0])
LogicalValues(tuples=[[]])
{code}
As we can see, since the filter is 'always false', the
LogicalTableScan+LogicalFilter are correctly replaced by an empty LogicalValues.
However, the same filter with a NOT expression, is not correctly simplified:
{code}
// query:
select d.deptno from dept d where not(d.deptno=7 and d.deptno=8)
// plan before:
LogicalProject(DEPTNO=[$0])
LogicalFilter(condition=[NOT(AND(=($0, 7), =($0, 8)))])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
// plan after (actual, NOT distributivity for AND):
LogicalProject(DEPTNO=[$0])
LogicalFilter(condition=[OR(<>($0, 7), <>($0, 8))])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
// plan after (expected, when $0's type is NOT nullable, filter removed):
LogicalProject(DEPTNO=[$0])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
// plan after (expected, when $0's type IS nullable, filter simplified as IS
NOT NULL):
LogicalProject(DEPTNO=[$0])
LogicalFilter(condition=[IS NOT NULL($0)])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
{code}
was:
Currently, ReduceExpressionsRule.FILTER_INSTANCE can successfully reduce a
query like this one (see RelOptRulesTest#testReduceConstantsDup):
{code}
// query:
select d.deptno from dept d where d.deptno=7 and d.deptno=8
// plan before:
LogicalProject(DEPTNO=[$0])
LogicalFilter(condition=[AND(=($0, 7), =($0, 8))])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
// plan after:
LogicalProject(DEPTNO=[$0])
LogicalValues(tuples=[[]])
{code}
As we can see, since the filter is 'always false', the
LogicalTableScan+LogicalFilter are correctly replaced by an empty LogicalValues.
However, the same filter with a NOT expression, is not correctly simplified:
{code}
// query:
select d.deptno from dept d where not(d.deptno=7 and d.deptno=8)
// plan before:
LogicalProject(DEPTNO=[$0])
LogicalFilter(condition=[NOT(AND(=($0, 7), =($0, 8)))])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
// plan after (actual, NOT distributivity for AND):
LogicalProject(DEPTNO=[$0])
LogicalFilter(condition=[OR(<>($0, 7), <>($0, 8))])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
// plan after (expected, when $0's type is NOT nullable, filter removed):
LogicalProject(DEPTNO=[$0])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
// plan after (expected, when $0's type IS nullable, filter simplified as IS
NOT NULL):
LogicalProject(DEPTNO=[$0])
LogicalFilter(condition=[$0 IS NOT NULL])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
{code}
> ReduceExpressionsRule.FILTER_INSTANCE does not reduce 'NOT(x=a AND x=b)'
>
>
> Key: CALCITE-3198
> URL: https://issues.apache.org/jira/browse/CALCITE-3198
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.20.0
>Reporter: Ruben Quesada Lopez
>Assignee: Ruben Quesada Lopez
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.21.0
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> Currently, ReduceExpressionsRule.FILTER_INSTANCE can successfully reduce a
> query like this one (see RelOptRulesTest#testReduceConstantsDup):
> {code}
> // query:
> select d.deptno from dept d where d.deptno=7 and d.deptno=8
> // plan before:
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[AND(=($0, 7), =($0, 8))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after:
> LogicalProject(DEPTNO=[$0])
> LogicalValues(tuples=[[]])
> {code}
> As we can see, since the filter is 'always false', the
> LogicalTableScan+LogicalFilter are correctly replaced by an empty
> LogicalValues.
> However, the same filter with a NOT expression, is not correctly simplified:
> {code}
> // query:
> select d.deptno from dept d where not(d.deptno=7 and d.deptno=8)
> // plan before:
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[NOT(AND(=($0, 7), =($0, 8)))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after (actual, NOT distributivity for AND):
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[OR(<>($0, 7), <>($0, 8))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after (expected, when $0's type is NOT nullable, filter removed):
[jira] [Comment Edited] (CALCITE-3198) ReduceExpressionsRule.FILTER_INSTANCE does not reduce 'NOT(x=a AND x=b)'
[
https://issues.apache.org/jira/browse/CALCITE-3198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16885045#comment-16885045
]
Ruben Quesada Lopez edited comment on CALCITE-3198 at 7/22/19 3:38 PM:
---
Same result is obtained with the following test, which is maybe more
straightforward (and by solving this, the one in the description might be also
solved as a side effect):
{code}
// query:
select d.deptno from dept d where d.deptno<>7 or d.deptno<>8
// plan before:
LogicalProject(DEPTNO=[$0])
LogicalFilter(condition=[OR(<>($0, 7), <>($0, 8))])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
// plan after (actual, unchanged):
LogicalProject(DEPTNO=[$0])
LogicalFilter(condition=[OR(<>($0, 7), <>($0, 8))])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
// plan after (expected, filter removed):
LogicalProject(DEPTNO=[$0])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
{code}
was (Author: rubenql):
Same result is obtained with the following test, which is maybe more
straightforward (and by solving this, the one in the description might be also
solved as a side effect):
{code}
// query:
select d.deptno from dept d where d.deptno<>7 or d.deptno=8
// plan before:
LogicalProject(DEPTNO=[$0])
LogicalFilter(condition=[OR(<>($0, 7), <>($0, 8))])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
// plan after (actual, unchanged):
LogicalProject(DEPTNO=[$0])
LogicalFilter(condition=[OR(<>($0, 7), <>($0, 8))])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
// plan after (expected, filter removed):
LogicalProject(DEPTNO=[$0])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
{code}
> ReduceExpressionsRule.FILTER_INSTANCE does not reduce 'NOT(x=a AND x=b)'
>
>
> Key: CALCITE-3198
> URL: https://issues.apache.org/jira/browse/CALCITE-3198
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.20.0
>Reporter: Ruben Quesada Lopez
>Assignee: Ruben Quesada Lopez
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.21.0
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> Currently, ReduceExpressionsRule.FILTER_INSTANCE can successfully reduce a
> query like this one (see RelOptRulesTest#testReduceConstantsDup):
> {code}
> // query:
> select d.deptno from dept d where d.deptno=7 and d.deptno=8
> // plan before:
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[AND(=($0, 7), =($0, 8))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after:
> LogicalProject(DEPTNO=[$0])
> LogicalValues(tuples=[[]])
> {code}
> As we can see, since the filter is 'always false', the
> LogicalTableScan+LogicalFilter are correctly replaced by an empty
> LogicalValues.
> However, the same filter with a NOT expression, is not correctly simplified:
> {code}
> // query:
> select d.deptno from dept d where not(d.deptno=7 and d.deptno=8)
> // plan before:
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[NOT(AND(=($0, 7), =($0, 8)))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after (actual, NOT distributivity for AND):
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[OR(<>($0, 7), <>($0, 8))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after (expected, when $0's type is NOT nullable, filter removed):
> LogicalProject(DEPTNO=[$0])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after (expected, when $0's type IS nullable, filter simplified as IS
> NOT NULL):
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[$0 IS NOT NULL])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Comment Edited] (CALCITE-3198) ReduceExpressionsRule.FILTER_INSTANCE does not reduce 'NOT(x=a AND x=b)'
[
https://issues.apache.org/jira/browse/CALCITE-3198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16885045#comment-16885045
]
Ruben Quesada Lopez edited comment on CALCITE-3198 at 7/22/19 3:38 PM:
---
Same result is obtained with the following test, which is maybe more
straightforward (and by solving this, the one in the description might be also
solved as a side effect):
{code}
// query:
select d.deptno from dept d where d.deptno<>7 or d.deptno<>8
// plan before:
LogicalProject(DEPTNO=[$0])
LogicalFilter(condition=[OR(<>($0, 7), <>($0, 8))])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
// plan after (actual, unchanged):
LogicalProject(DEPTNO=[$0])
LogicalFilter(condition=[OR(<>($0, 7), <>($0, 8))])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
// plan after (expected, when $0's type is NOT nullable, filter removed):
LogicalProject(DEPTNO=[$0])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
// plan after (expected, when $0's type IS nullable, filter simplified as IS
NOT NULL):
LogicalProject(DEPTNO=[$0])
LogicalFilter(condition=[$0 IS NOT NULL])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
{code}
was (Author: rubenql):
Same result is obtained with the following test, which is maybe more
straightforward (and by solving this, the one in the description might be also
solved as a side effect):
{code}
// query:
select d.deptno from dept d where d.deptno<>7 or d.deptno<>8
// plan before:
LogicalProject(DEPTNO=[$0])
LogicalFilter(condition=[OR(<>($0, 7), <>($0, 8))])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
// plan after (actual, unchanged):
LogicalProject(DEPTNO=[$0])
LogicalFilter(condition=[OR(<>($0, 7), <>($0, 8))])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
// plan after (expected, filter removed):
LogicalProject(DEPTNO=[$0])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
{code}
> ReduceExpressionsRule.FILTER_INSTANCE does not reduce 'NOT(x=a AND x=b)'
>
>
> Key: CALCITE-3198
> URL: https://issues.apache.org/jira/browse/CALCITE-3198
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.20.0
>Reporter: Ruben Quesada Lopez
>Assignee: Ruben Quesada Lopez
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.21.0
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> Currently, ReduceExpressionsRule.FILTER_INSTANCE can successfully reduce a
> query like this one (see RelOptRulesTest#testReduceConstantsDup):
> {code}
> // query:
> select d.deptno from dept d where d.deptno=7 and d.deptno=8
> // plan before:
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[AND(=($0, 7), =($0, 8))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after:
> LogicalProject(DEPTNO=[$0])
> LogicalValues(tuples=[[]])
> {code}
> As we can see, since the filter is 'always false', the
> LogicalTableScan+LogicalFilter are correctly replaced by an empty
> LogicalValues.
> However, the same filter with a NOT expression, is not correctly simplified:
> {code}
> // query:
> select d.deptno from dept d where not(d.deptno=7 and d.deptno=8)
> // plan before:
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[NOT(AND(=($0, 7), =($0, 8)))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after (actual, NOT distributivity for AND):
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[OR(<>($0, 7), <>($0, 8))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after (expected, when $0's type is NOT nullable, filter removed):
> LogicalProject(DEPTNO=[$0])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after (expected, when $0's type IS nullable, filter simplified as IS
> NOT NULL):
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[$0 IS NOT NULL])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (CALCITE-3198) ReduceExpressionsRule.FILTER_INSTANCE does not reduce 'NOT(x=a AND x=b)'
[
https://issues.apache.org/jira/browse/CALCITE-3198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Quesada Lopez updated CALCITE-3198:
-
Description:
Currently, ReduceExpressionsRule.FILTER_INSTANCE can successfully reduce a
query like this one (see RelOptRulesTest#testReduceConstantsDup):
{code}
// query:
select d.deptno from dept d where d.deptno=7 and d.deptno=8
// plan before:
LogicalProject(DEPTNO=[$0])
LogicalFilter(condition=[AND(=($0, 7), =($0, 8))])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
// plan after:
LogicalProject(DEPTNO=[$0])
LogicalValues(tuples=[[]])
{code}
As we can see, since the filter is 'always false', the
LogicalTableScan+LogicalFilter are correctly replaced by an empty LogicalValues.
However, the same filter with a NOT expression, is not correctly simplified:
{code}
// query:
select d.deptno from dept d where not(d.deptno=7 and d.deptno=8)
// plan before:
LogicalProject(DEPTNO=[$0])
LogicalFilter(condition=[NOT(AND(=($0, 7), =($0, 8)))])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
// plan after (actual, NOT distributivity for AND):
LogicalProject(DEPTNO=[$0])
LogicalFilter(condition=[OR(<>($0, 7), <>($0, 8))])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
// plan after (expected, when $0's type is NOT nullable, filter removed):
LogicalProject(DEPTNO=[$0])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
// plan after (expected, when $0's type IS nullable, filter simplified as IS
NOT NULL):
LogicalProject(DEPTNO=[$0])
LogicalFilter(condition=[$0 IS NOT NULL])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
{code}
was:
Currently, ReduceExpressionsRule.FILTER_INSTANCE can successfully reduce a
query like this one (see RelOptRulesTest#testReduceConstantsDup):
{code}
// query:
select d.deptno from dept d where d.deptno=7 and d.deptno=8
// plan before:
LogicalProject(DEPTNO=[$0])
LogicalFilter(condition=[AND(=($0, 7), =($0, 8))])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
// plan after:
LogicalProject(DEPTNO=[$0])
LogicalValues(tuples=[[]])
{code}
As we can see, since the filter is 'always false', the
LogicalTableScan+LogicalFilter are correctly replaced by an empty LogicalValues.
However, the same filter with a NOT expression, is not correctly simplified:
{code}
// query:
select d.deptno from dept d where not(d.deptno=7 and d.deptno=8)
// plan before:
LogicalProject(DEPTNO=[$0])
LogicalFilter(condition=[NOT(AND(=($0, 7), =($0, 8)))])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
// plan after (actual, NOT distributivity for AND):
LogicalProject(DEPTNO=[$0])
LogicalFilter(condition=[OR(<>($0, 7), <>($0, 8))])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
// plan after (expected, filter removed):
LogicalProject(DEPTNO=[$0])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
{code}
Since the filter is the negated of an 'always false filter' (the one used in
the previous query), it is therefore an 'always true filter', so the expected
behavior is that the LogicalFilter should be removed, and it is not.
> ReduceExpressionsRule.FILTER_INSTANCE does not reduce 'NOT(x=a AND x=b)'
>
>
> Key: CALCITE-3198
> URL: https://issues.apache.org/jira/browse/CALCITE-3198
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.20.0
>Reporter: Ruben Quesada Lopez
>Assignee: Ruben Quesada Lopez
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.21.0
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> Currently, ReduceExpressionsRule.FILTER_INSTANCE can successfully reduce a
> query like this one (see RelOptRulesTest#testReduceConstantsDup):
> {code}
> // query:
> select d.deptno from dept d where d.deptno=7 and d.deptno=8
> // plan before:
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[AND(=($0, 7), =($0, 8))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after:
> LogicalProject(DEPTNO=[$0])
> LogicalValues(tuples=[[]])
> {code}
> As we can see, since the filter is 'always false', the
> LogicalTableScan+LogicalFilter are correctly replaced by an empty
> LogicalValues.
> However, the same filter with a NOT expression, is not correctly simplified:
> {code}
> // query:
> select d.deptno from dept d where not(d.deptno=7 and d.deptno=8)
> // plan before:
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[NOT(AND(=($0, 7), =($0, 8)))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after (actual, NOT distributivity for AND):
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[OR(<>($0, 7), <>($0, 8))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after (expected, when $0's type is NOT nullable, filter removed):
>
[jira] [Resolved] (CALCITE-3182) Trim unused fields for plan of materialized-view before matching.
[
https://issues.apache.org/jira/browse/CALCITE-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Quesada Lopez resolved CALCITE-3182.
--
Resolution: Fixed
Fixed in
https://github.com/apache/calcite/commit/58b6ad6135d909c053f15fb902d8b73559d1fa53
Thanks for the PR [~jinxing6...@126.com]!
> Trim unused fields for plan of materialized-view before matching.
> -
>
> Key: CALCITE-3182
> URL: https://issues.apache.org/jira/browse/CALCITE-3182
> Project: Calcite
> Issue Type: Bug
>Reporter: jin xing
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.21.0
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> In current code, before matching query with materialized-view, unused fields
> of query is trimmed but materialized-view is not. Thus below simple SQL fails
> to be matched though query and materialized-
> view are exactly the same:
> {code:java}
> @Test public void testMaterializationAfterTrimingOfUnusedFields() {
> String sql =
> "select \"y\".\"deptno\", \"y\".\"name\", \"x\".\"sum_salary\"\n" +
> "from\n" +
> " (select \"deptno\", sum(\"salary\") \"sum_salary\" from \"emps\" group by
> \"deptno\") \"x\"\n" +
> " join\n" +
> " \"depts\" \"y\"\n" +
> " on \"x\".\"deptno\"=\"y\".\"deptno\"\n";
> checkMaterialize(sql, sql);
> }{code}
> Checking {{CalciteMaterializer}}
> [https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/prepare/CalciteMaterializer.java#L83]
> , I think the code intends to do the trimming, but didn't call the method.
> Since unused fields of query is trimmed anyway
> [https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/prepare/CalcitePrepareImpl.java#L995]
> , thus I think there's no necessity to keep the materialized-view un-trimmed
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (CALCITE-3182) Trim unused fields for plan of materialized-view before matching.
[
https://issues.apache.org/jira/browse/CALCITE-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Quesada Lopez updated CALCITE-3182:
-
Fix Version/s: 1.21.0
> Trim unused fields for plan of materialized-view before matching.
> -
>
> Key: CALCITE-3182
> URL: https://issues.apache.org/jira/browse/CALCITE-3182
> Project: Calcite
> Issue Type: Bug
>Reporter: jin xing
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.21.0
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> In current code, before matching query with materialized-view, unused fields
> of query is trimmed but materialized-view is not. Thus below simple SQL fails
> to be matched though query and materialized-
> view are exactly the same:
> {code:java}
> @Test public void testMaterializationAfterTrimingOfUnusedFields() {
> String sql =
> "select \"y\".\"deptno\", \"y\".\"name\", \"x\".\"sum_salary\"\n" +
> "from\n" +
> " (select \"deptno\", sum(\"salary\") \"sum_salary\" from \"emps\" group by
> \"deptno\") \"x\"\n" +
> " join\n" +
> " \"depts\" \"y\"\n" +
> " on \"x\".\"deptno\"=\"y\".\"deptno\"\n";
> checkMaterialize(sql, sql);
> }{code}
> Checking {{CalciteMaterializer}}
> [https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/prepare/CalciteMaterializer.java#L83]
> , I think the code intends to do the trimming, but didn't call the method.
> Since unused fields of query is trimmed anyway
> [https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/prepare/CalcitePrepareImpl.java#L995]
> , thus I think there's no necessity to keep the materialized-view un-trimmed
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (CALCITE-3198) ReduceExpressionsRule.FILTER_INSTANCE does not reduce 'NOT(x=a AND x=b)'
[
https://issues.apache.org/jira/browse/CALCITE-3198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16887746#comment-16887746
]
Ruben Quesada Lopez commented on CALCITE-3198:
--
PR available. As mentioned in the previous comments, the proposed fix is
actually located in {{RexSimplify}}.
> ReduceExpressionsRule.FILTER_INSTANCE does not reduce 'NOT(x=a AND x=b)'
>
>
> Key: CALCITE-3198
> URL: https://issues.apache.org/jira/browse/CALCITE-3198
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.20.0
>Reporter: Ruben Quesada Lopez
>Assignee: Ruben Quesada Lopez
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.21.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently, ReduceExpressionsRule.FILTER_INSTANCE can successfully reduce a
> query like this one (see RelOptRulesTest#testReduceConstantsDup):
> {code}
> // query:
> select d.deptno from dept d where d.deptno=7 and d.deptno=8
> // plan before:
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[AND(=($0, 7), =($0, 8))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after:
> LogicalProject(DEPTNO=[$0])
> LogicalValues(tuples=[[]])
> {code}
> As we can see, since the filter is 'always false', the
> LogicalTableScan+LogicalFilter are correctly replaced by an empty
> LogicalValues.
> However, the same filter with a NOT expression, is not correctly simplified:
> {code}
> // query:
> select d.deptno from dept d where not(d.deptno=7 and d.deptno=8)
> // plan before:
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[NOT(AND(=($0, 7), =($0, 8)))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after (actual, NOT distributivity for AND):
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[OR(<>($0, 7), <>($0, 8))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after (expected, filter removed):
> LogicalProject(DEPTNO=[$0])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> {code}
> Since the filter is the negated of an 'always false filter' (the one used in
> the previous query), it is therefore an 'always true filter', so the expected
> behavior is that the LogicalFilter should be removed, and it is not.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (CALCITE-3198) ReduceExpressionsRule.FILTER_INSTANCE does not reduce 'NOT(x=a AND x=b)'
[
https://issues.apache.org/jira/browse/CALCITE-3198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Quesada Lopez updated CALCITE-3198:
-
Fix Version/s: 1.21.0
> ReduceExpressionsRule.FILTER_INSTANCE does not reduce 'NOT(x=a AND x=b)'
>
>
> Key: CALCITE-3198
> URL: https://issues.apache.org/jira/browse/CALCITE-3198
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.20.0
>Reporter: Ruben Quesada Lopez
>Assignee: Ruben Quesada Lopez
>Priority: Minor
> Fix For: 1.21.0
>
>
> Currently, ReduceExpressionsRule.FILTER_INSTANCE can successfully reduce a
> query like this one (see RelOptRulesTest#testReduceConstantsDup):
> {code}
> // query:
> select d.deptno from dept d where d.deptno=7 and d.deptno=8
> // plan before:
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[AND(=($0, 7), =($0, 8))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after:
> LogicalProject(DEPTNO=[$0])
> LogicalValues(tuples=[[]])
> {code}
> As we can see, since the filter is 'always false', the
> LogicalTableScan+LogicalFilter are correctly replaced by an empty
> LogicalValues.
> However, the same filter with a NOT expression, is not correctly simplified:
> {code}
> // query:
> select d.deptno from dept d where not(d.deptno=7 and d.deptno=8)
> // plan before:
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[NOT(AND(=($0, 7), =($0, 8)))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after (actual, NOT distributivity for AND):
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[OR(<>($0, 7), <>($0, 8))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after (expected, filter removed):
> LogicalProject(DEPTNO=[$0])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> {code}
> Since the filter is the negated of an 'always false filter' (the one used in
> the previous query), it is therefore an 'always true filter', so the expected
> behavior is that the LogicalFilter should be removed, and it is not.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Assigned] (CALCITE-3198) ReduceExpressionsRule.FILTER_INSTANCE does not reduce 'NOT(x=a AND x=b)'
[
https://issues.apache.org/jira/browse/CALCITE-3198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Quesada Lopez reassigned CALCITE-3198:
Assignee: Ruben Quesada Lopez
> ReduceExpressionsRule.FILTER_INSTANCE does not reduce 'NOT(x=a AND x=b)'
>
>
> Key: CALCITE-3198
> URL: https://issues.apache.org/jira/browse/CALCITE-3198
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.20.0
>Reporter: Ruben Quesada Lopez
>Assignee: Ruben Quesada Lopez
>Priority: Minor
>
> Currently, ReduceExpressionsRule.FILTER_INSTANCE can successfully reduce a
> query like this one (see RelOptRulesTest#testReduceConstantsDup):
> {code}
> // query:
> select d.deptno from dept d where d.deptno=7 and d.deptno=8
> // plan before:
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[AND(=($0, 7), =($0, 8))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after:
> LogicalProject(DEPTNO=[$0])
> LogicalValues(tuples=[[]])
> {code}
> As we can see, since the filter is 'always false', the
> LogicalTableScan+LogicalFilter are correctly replaced by an empty
> LogicalValues.
> However, the same filter with a NOT expression, is not correctly simplified:
> {code}
> // query:
> select d.deptno from dept d where not(d.deptno=7 and d.deptno=8)
> // plan before:
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[NOT(AND(=($0, 7), =($0, 8)))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after (actual, NOT distributivity for AND):
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[OR(<>($0, 7), <>($0, 8))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after (expected, filter removed):
> LogicalProject(DEPTNO=[$0])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> {code}
> Since the filter is the negated of an 'always false filter' (the one used in
> the previous query), it is therefore an 'always true filter', so the expected
> behavior is that the LogicalFilter should be removed, and it is not.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (CALCITE-3198) ReduceExpressionsRule.FILTER_INSTANCE does not reduce 'NOT(x=a AND x=b)'
[
https://issues.apache.org/jira/browse/CALCITE-3198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16885912#comment-16885912
]
Ruben Quesada Lopez commented on CALCITE-3198:
--
[~julianhyde], you are right. I talked about {{ReduceExpressionsRule}} because
it seemed to me the "entry point" of this issue. However, in the end this rule
delegates much of the reduction process to {{RexSimplify}}, so I guess that is
where we should act to fix this.
> ReduceExpressionsRule.FILTER_INSTANCE does not reduce 'NOT(x=a AND x=b)'
>
>
> Key: CALCITE-3198
> URL: https://issues.apache.org/jira/browse/CALCITE-3198
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.20.0
>Reporter: Ruben Quesada Lopez
>Priority: Minor
>
> Currently, ReduceExpressionsRule.FILTER_INSTANCE can successfully reduce a
> query like this one (see RelOptRulesTest#testReduceConstantsDup):
> {code}
> // query:
> select d.deptno from dept d where d.deptno=7 and d.deptno=8
> // plan before:
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[AND(=($0, 7), =($0, 8))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after:
> LogicalProject(DEPTNO=[$0])
> LogicalValues(tuples=[[]])
> {code}
> As we can see, since the filter is 'always false', the
> LogicalTableScan+LogicalFilter are correctly replaced by an empty
> LogicalValues.
> However, the same filter with a NOT expression, is not correctly simplified:
> {code}
> // query:
> select d.deptno from dept d where not(d.deptno=7 and d.deptno=8)
> // plan before:
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[NOT(AND(=($0, 7), =($0, 8)))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after (actual, NOT distributivity for AND):
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[OR(<>($0, 7), <>($0, 8))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after (expected, filter removed):
> LogicalProject(DEPTNO=[$0])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> {code}
> Since the filter is the negated of an 'always false filter' (the one used in
> the previous query), it is therefore an 'always true filter', so the expected
> behavior is that the LogicalFilter should be removed, and it is not.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Comment Edited] (CALCITE-3198) ReduceExpressionsRule.FILTER_INSTANCE does not reduce 'NOT(x=a AND x=b)'
[
https://issues.apache.org/jira/browse/CALCITE-3198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16885045#comment-16885045
]
Ruben Quesada Lopez edited comment on CALCITE-3198 at 7/15/19 10:24 AM:
Same result is obtained with the following test, which is maybe more
straightforward (and by solving this, the one in the description might be also
solved as a side effect):
{code}
// query:
select d.deptno from dept d where d.deptno<>7 or d.deptno=8
// plan before:
LogicalProject(DEPTNO=[$0])
LogicalFilter(condition=[OR(<>($0, 7), <>($0, 8))])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
// plan after (actual, unchanged):
LogicalProject(DEPTNO=[$0])
LogicalFilter(condition=[OR(<>($0, 7), <>($0, 8))])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
// plan after (expected, filter removed):
LogicalProject(DEPTNO=[$0])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
{code}
was (Author: rubenql):
Same result is obtained with the following test:
{code}
// query:
select d.deptno from dept d where d.deptno<>7 or d.deptno=8
// plan before:
LogicalProject(DEPTNO=[$0])
LogicalFilter(condition=[OR(<>($0, 7), <>($0, 8))])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
// plan after (actual, unchanged):
LogicalProject(DEPTNO=[$0])
LogicalFilter(condition=[OR(<>($0, 7), <>($0, 8))])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
// plan after (expected, filter removed):
LogicalProject(DEPTNO=[$0])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
{code}
> ReduceExpressionsRule.FILTER_INSTANCE does not reduce 'NOT(x=a AND x=b)'
>
>
> Key: CALCITE-3198
> URL: https://issues.apache.org/jira/browse/CALCITE-3198
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.20.0
>Reporter: Ruben Quesada Lopez
>Priority: Minor
>
> Currently, ReduceExpressionsRule.FILTER_INSTANCE can successfully reduce a
> query like this one (see RelOptRulesTest#testReduceConstantsDup):
> {code}
> // query:
> select d.deptno from dept d where d.deptno=7 and d.deptno=8
> // plan before:
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[AND(=($0, 7), =($0, 8))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after:
> LogicalProject(DEPTNO=[$0])
> LogicalValues(tuples=[[]])
> {code}
> As we can see, since the filter is 'always false', the
> LogicalTableScan+LogicalFilter are correctly replaced by an empty
> LogicalValues.
> However, the same filter with a NOT expression, is not correctly simplified:
> {code}
> // query:
> select d.deptno from dept d where not(d.deptno=7 and d.deptno=8)
> // plan before:
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[NOT(AND(=($0, 7), =($0, 8)))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after (actual, NOT distributivity for AND):
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[OR(<>($0, 7), <>($0, 8))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after (expected, filter removed):
> LogicalProject(DEPTNO=[$0])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> {code}
> Since the filter is the negated of an 'always false filter' (the one used in
> the previous query), it is therefore an 'always true filter', so the expected
> behavior is that the LogicalFilter should be removed, and it is not.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (CALCITE-3198) ReduceExpressionsRule.FILTER_INSTANCE does not reduce 'NOT(x=a AND x=b)'
[
https://issues.apache.org/jira/browse/CALCITE-3198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Quesada Lopez updated CALCITE-3198:
-
Description:
Currently, ReduceExpressionsRule.FILTER_INSTANCE can successfully reduce a
query like this one (see RelOptRulesTest#testReduceConstantsDup):
{code}
// query:
select d.deptno from dept d where d.deptno=7 and d.deptno=8
// plan before:
LogicalProject(DEPTNO=[$0])
LogicalFilter(condition=[AND(=($0, 7), =($0, 8))])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
// plan after:
LogicalProject(DEPTNO=[$0])
LogicalValues(tuples=[[]])
{code}
As we can see, since the filter is 'always false', the
LogicalTableScan+LogicalFilter are correctly replaced by an empty LogicalValues.
However, the same filter with a NOT expression, is not correctly simplified:
{code}
// query:
select d.deptno from dept d where not(d.deptno=7 and d.deptno=8)
// plan before:
LogicalProject(DEPTNO=[$0])
LogicalFilter(condition=[NOT(AND(=($0, 7), =($0, 8)))])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
// plan after (actual, NOT distributivity for AND):
LogicalProject(DEPTNO=[$0])
LogicalFilter(condition=[OR(<>($0, 7), <>($0, 8))])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
// plan after (expected, filter removed):
LogicalProject(DEPTNO=[$0])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
{code}
Since the filter is the negated of an 'always false filter' (the one used in
the previous query), it is therefore an 'always true filter', so the expected
behavior is that the LogicalFilter should be removed, and it is not.
was:
Currently, ReduceExpressionsRule.FILTER_INSTANCE can successfully reduce a
query like this one (see RelOptRulesTest#testReduceConstantsDup):
{code}
// query:
select d.deptno from dept d where d.deptno=7 and d.deptno=8
// plan before:
LogicalProject(DEPTNO=[$0])
LogicalFilter(condition=[AND(=($0, 7), =($0, 8))])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
// plan after:
LogicalProject(DEPTNO=[$0])
LogicalValues(tuples=[[]])
{code}
As we can see, since the filter is 'always false', the
LogicalTableScan+LogicalFilter are correctly replaced by an empty LogicalValues.
However, the same filter with a NOT expression, is not correctly simplified:
{code}
// query:
select d.deptno from dept d where not(d.deptno=7 and d.deptno=8)
// plan before:
LogicalProject(DEPTNO=[$0])
LogicalFilter(condition=[NOT(AND(=($0, 7), =($0, 8)))])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
// plan after (actual):
LogicalProject(DEPTNO=[$0])
LogicalFilter(condition=[OR(<>($0, 7), <>($0, 8))])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
// plan after (expected):
LogicalProject(DEPTNO=[$0])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
{code}
Since the filter is the negated of an 'always false filter' (the one used in
the previous query), it is therefore an 'always true filter', so the expected
behavior is that the LogicalFilter should be removed, and it is not.
> ReduceExpressionsRule.FILTER_INSTANCE does not reduce 'NOT(x=a AND x=b)'
>
>
> Key: CALCITE-3198
> URL: https://issues.apache.org/jira/browse/CALCITE-3198
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.20.0
>Reporter: Ruben Quesada Lopez
>Priority: Minor
>
> Currently, ReduceExpressionsRule.FILTER_INSTANCE can successfully reduce a
> query like this one (see RelOptRulesTest#testReduceConstantsDup):
> {code}
> // query:
> select d.deptno from dept d where d.deptno=7 and d.deptno=8
> // plan before:
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[AND(=($0, 7), =($0, 8))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after:
> LogicalProject(DEPTNO=[$0])
> LogicalValues(tuples=[[]])
> {code}
> As we can see, since the filter is 'always false', the
> LogicalTableScan+LogicalFilter are correctly replaced by an empty
> LogicalValues.
> However, the same filter with a NOT expression, is not correctly simplified:
> {code}
> // query:
> select d.deptno from dept d where not(d.deptno=7 and d.deptno=8)
> // plan before:
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[NOT(AND(=($0, 7), =($0, 8)))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after (actual, NOT distributivity for AND):
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[OR(<>($0, 7), <>($0, 8))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after (expected, filter removed):
> LogicalProject(DEPTNO=[$0])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> {code}
> Since the filter is the negated of an 'always false filter' (the one used in
> the previous query), it is therefore an 'always true filter', so the expected
> behavior is that the LogicalFilter should
[jira] [Commented] (CALCITE-3198) ReduceExpressionsRule.FILTER_INSTANCE does not reduce 'NOT(x=a AND x=b)'
[
https://issues.apache.org/jira/browse/CALCITE-3198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16885045#comment-16885045
]
Ruben Quesada Lopez commented on CALCITE-3198:
--
Same result is obtained with the following test:
{code}
// query:
select d.deptno from dept d where d.deptno<>7 or d.deptno=8
// plan before:
LogicalProject(DEPTNO=[$0])
LogicalFilter(condition=[OR(<>($0, 7), <>($0, 8))])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
// plan after (actual, unchanged):
LogicalProject(DEPTNO=[$0])
LogicalFilter(condition=[OR(<>($0, 7), <>($0, 8))])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
// plan after (expected, filter removed):
LogicalProject(DEPTNO=[$0])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
{code}
> ReduceExpressionsRule.FILTER_INSTANCE does not reduce 'NOT(x=a AND x=b)'
>
>
> Key: CALCITE-3198
> URL: https://issues.apache.org/jira/browse/CALCITE-3198
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.20.0
>Reporter: Ruben Quesada Lopez
>Priority: Minor
>
> Currently, ReduceExpressionsRule.FILTER_INSTANCE can successfully reduce a
> query like this one (see RelOptRulesTest#testReduceConstantsDup):
> {code}
> // query:
> select d.deptno from dept d where d.deptno=7 and d.deptno=8
> // plan before:
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[AND(=($0, 7), =($0, 8))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after:
> LogicalProject(DEPTNO=[$0])
> LogicalValues(tuples=[[]])
> {code}
> As we can see, since the filter is 'always false', the
> LogicalTableScan+LogicalFilter are correctly replaced by an empty
> LogicalValues.
> However, the same filter with a NOT expression, is not correctly simplified:
> {code}
> // query:
> select d.deptno from dept d where not(d.deptno=7 and d.deptno=8)
> // plan before:
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[NOT(AND(=($0, 7), =($0, 8)))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after (actual):
> LogicalProject(DEPTNO=[$0])
> LogicalFilter(condition=[OR(<>($0, 7), <>($0, 8))])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> // plan after (expected):
> LogicalProject(DEPTNO=[$0])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> {code}
> Since the filter is the negated of an 'always false filter' (the one used in
> the previous query), it is therefore an 'always true filter', so the expected
> behavior is that the LogicalFilter should be removed, and it is not.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (CALCITE-3198) ReduceExpressionsRule.FILTER_INSTANCE does not reduce 'NOT(x=a AND x=b)'
Ruben Quesada Lopez created CALCITE-3198:
Summary: ReduceExpressionsRule.FILTER_INSTANCE does not reduce
'NOT(x=a AND x=b)'
Key: CALCITE-3198
URL: https://issues.apache.org/jira/browse/CALCITE-3198
Project: Calcite
Issue Type: Bug
Affects Versions: 1.20.0
Reporter: Ruben Quesada Lopez
Currently, ReduceExpressionsRule.FILTER_INSTANCE can successfully reduce a
query like this one (see RelOptRulesTest#testReduceConstantsDup):
{code}
// query:
select d.deptno from dept d where d.deptno=7 and d.deptno=8
// plan before:
LogicalProject(DEPTNO=[$0])
LogicalFilter(condition=[AND(=($0, 7), =($0, 8))])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
// plan after:
LogicalProject(DEPTNO=[$0])
LogicalValues(tuples=[[]])
{code}
As we can see, since the filter is 'always false', the
LogicalTableScan+LogicalFilter are correctly replaced by an empty LogicalValues.
However, the same filter with a NOT expression, is not correctly simplified:
{code}
// query:
select d.deptno from dept d where not(d.deptno=7 and d.deptno=8)
// plan before:
LogicalProject(DEPTNO=[$0])
LogicalFilter(condition=[NOT(AND(=($0, 7), =($0, 8)))])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
// plan after (actual):
LogicalProject(DEPTNO=[$0])
LogicalFilter(condition=[OR(<>($0, 7), <>($0, 8))])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
// plan after (expected):
LogicalProject(DEPTNO=[$0])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
{code}
Since the filter is the negated of an 'always false filter' (the one used in
the previous query), it is therefore an 'always true filter', so the expected
behavior is that the LogicalFilter should be removed, and it is not.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (CALCITE-3165) Project#accept(RexShuttle shuttle) does not update rowType
[
https://issues.apache.org/jira/browse/CALCITE-3165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Quesada Lopez updated CALCITE-3165:
-
Affects Version/s: 1.20.0
> Project#accept(RexShuttle shuttle) does not update rowType
> --
>
> Key: CALCITE-3165
> URL: https://issues.apache.org/jira/browse/CALCITE-3165
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.20.0
>Reporter: Ruben Quesada Lopez
>Assignee: Ruben Quesada Lopez
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.21.0
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> {{Project#accept(RexShuttle shuttle)}} can return a copy of the original
> Project, if the list of expressions is modified by the shuttle. However, this
> Project copy will maintain the original rowType, which might be wrong because
> this type is based on the expressions list (which was modified by the shuttle)
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (CALCITE-3165) Project#accept(RexShuttle shuttle) does not update rowType
[
https://issues.apache.org/jira/browse/CALCITE-3165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Quesada Lopez updated CALCITE-3165:
-
Fix Version/s: 1.21.0
> Project#accept(RexShuttle shuttle) does not update rowType
> --
>
> Key: CALCITE-3165
> URL: https://issues.apache.org/jira/browse/CALCITE-3165
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Ruben Quesada Lopez
>Assignee: Ruben Quesada Lopez
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.21.0
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> {{Project#accept(RexShuttle shuttle)}} can return a copy of the original
> Project, if the list of expressions is modified by the shuttle. However, this
> Project copy will maintain the original rowType, which might be wrong because
> this type is based on the expressions list (which was modified by the shuttle)
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Resolved] (CALCITE-3165) Project#accept(RexShuttle shuttle) does not update rowType
[
https://issues.apache.org/jira/browse/CALCITE-3165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Quesada Lopez resolved CALCITE-3165.
--
Resolution: Fixed
Fixed in
https://github.com/apache/calcite/commit/e5e5809d28f422896147bd9f069f4930f22629ad
> Project#accept(RexShuttle shuttle) does not update rowType
> --
>
> Key: CALCITE-3165
> URL: https://issues.apache.org/jira/browse/CALCITE-3165
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.20.0
>Reporter: Ruben Quesada Lopez
>Assignee: Ruben Quesada Lopez
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.21.0
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> {{Project#accept(RexShuttle shuttle)}} can return a copy of the original
> Project, if the list of expressions is modified by the shuttle. However, this
> Project copy will maintain the original rowType, which might be wrong because
> this type is based on the expressions list (which was modified by the shuttle)
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (CALCITE-3113) Equivalent MutableAggregates with different row types fail with AssertionError
[
https://issues.apache.org/jira/browse/CALCITE-3113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16881346#comment-16881346
]
Ruben Quesada Lopez commented on CALCITE-3113:
--
[~jinxing6...@126.com], I'm not familiar with this part of the code, but I
think your PR looks good. Moreover, I believe your change allows us to keep the
assertion, we just need to adapt it (please check my comment in the PR).
> Equivalent MutableAggregates with different row types fail with AssertionError
> --
>
> Key: CALCITE-3113
> URL: https://issues.apache.org/jira/browse/CALCITE-3113
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.19.0
>Reporter: Feng Zhu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Add test case in MaterializationTest:
> {code:java}
> @Test public void testAggregateAlias() {
> checkMaterialize(
> "select count(*) as c from \"emps\" group by \"empid\"",
> "select count(*) + 1 as c from \"emps\" group by \"empid\"");
> }
> {code}
> It fails due to different rowtype.
> {code:java}
> java.lang.AssertionError
> at
> org.apache.calcite.plan.SubstitutionVisitor.go(SubstitutionVisitor.java:504)
> at
> org.apache.calcite.plan.SubstitutionVisitor.go(SubstitutionVisitor.java:465)
> at
> org.apache.calcite.plan.MaterializedViewSubstitutionVisitor.go(MaterializedViewSubstitutionVisitor.java:56)
> at
> org.apache.calcite.plan.RelOptMaterializations.substitute(RelOptMaterializations.java:200)
> at
> org.apache.calcite.plan.RelOptMaterializations.useMaterializedViews(RelOptMaterializations.java:72)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.registerMaterializations(VolcanoPlanner.java:347)
> {code}
> However, according to MutableAggregate's hashCode implementation, this
> materialization can be reused, i.e., queryDescedant=targetDescendant.
> {code:java}
> queryDescendant: RecordType(JavaType(int) empid, BIGINT $f1)
> =
> Aggregate(groupSet: {0}, groupSets: [{0}], calls: [COUNT()])
> Project(projects: [$0])
> Scan(table: [hr, emps])
> targetDescendant: RecordType(JavaType(int) empid, BIGINT C)
> =
> Aggregate(groupSet: {0}, groupSets: [{0}], calls: [COUNT()])
> Project(projects: [$0])
> Scan(table: [hr, emps])
> {code}
> So, how can we align them?
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CALCITE-3185) Calcite Web: API doc search does not work (404)
Ruben Quesada Lopez created CALCITE-3185:
Summary: Calcite Web: API doc search does not work (404)
Key: CALCITE-3185
URL: https://issues.apache.org/jira/browse/CALCITE-3185
Project: Calcite
Issue Type: Bug
Affects Versions: 1.20.0
Reporter: Ruben Quesada Lopez
In https://calcite.apache.org/apidocs/ when we type something in the SEARCH
input, e.g. "RelBuilder", and we click on the corresponding item, we get
redirected to a wrong url (notice the 'undefined' in the middle):
https://calcite.apache.org/apidocs/undefined/org/apache/calcite/tools/RelBuilder.html
which shows the message:
{code}
Not Found
The requested URL /apidocs/undefined/org/apache/calcite/tools/RelBuilder.html
was not found on this server.
{code}
The proper url should be:
https://calcite.apache.org/apidocs/org/apache/calcite/tools/RelBuilder.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-3113) Equivalent MutableAggregates with different row types fail with AssertionError
[
https://issues.apache.org/jira/browse/CALCITE-3113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16881004#comment-16881004
]
Ruben Quesada Lopez commented on CALCITE-3113:
--
[~julianhyde], it seems that this assertion was inserted by you some time ago,
maybe you could help us here: is it too strict? Shall we remove it to fix cases
like the one described by [~jinxing6...@126.com] ?
> Equivalent MutableAggregates with different row types fail with AssertionError
> --
>
> Key: CALCITE-3113
> URL: https://issues.apache.org/jira/browse/CALCITE-3113
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.19.0
>Reporter: Feng Zhu
>Priority: Major
>
> Add test case in MaterializationTest:
> {code:java}
> @Test public void testAggregateAlias() {
> checkMaterialize(
> "select count(*) as c from \"emps\" group by \"empid\"",
> "select count(*) + 1 as c from \"emps\" group by \"empid\"");
> }
> {code}
> It fails due to different rowtype.
> {code:java}
> java.lang.AssertionError
> at
> org.apache.calcite.plan.SubstitutionVisitor.go(SubstitutionVisitor.java:504)
> at
> org.apache.calcite.plan.SubstitutionVisitor.go(SubstitutionVisitor.java:465)
> at
> org.apache.calcite.plan.MaterializedViewSubstitutionVisitor.go(MaterializedViewSubstitutionVisitor.java:56)
> at
> org.apache.calcite.plan.RelOptMaterializations.substitute(RelOptMaterializations.java:200)
> at
> org.apache.calcite.plan.RelOptMaterializations.useMaterializedViews(RelOptMaterializations.java:72)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.registerMaterializations(VolcanoPlanner.java:347)
> {code}
> However, according to MutableAggregate's hashCode implementation, this
> materialization can be reused, i.e., queryDescedant=targetDescendant.
> {code:java}
> queryDescendant: RecordType(JavaType(int) empid, BIGINT $f1)
> =
> Aggregate(groupSet: {0}, groupSets: [{0}], calls: [COUNT()])
> Project(projects: [$0])
> Scan(table: [hr, emps])
> targetDescendant: RecordType(JavaType(int) empid, BIGINT C)
> =
> Aggregate(groupSet: {0}, groupSets: [{0}], calls: [COUNT()])
> Project(projects: [$0])
> Scan(table: [hr, emps])
> {code}
> So, how can we align them?
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (CALCITE-3172) RelBuilder#empty does not keep aliases
[
https://issues.apache.org/jira/browse/CALCITE-3172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Quesada Lopez resolved CALCITE-3172.
--
Resolution: Fixed
Fixed in
https://github.com/apache/calcite/commit/046f9f4e2e3c779e4e2e61946d67a0dc124f7f72
> RelBuilder#empty does not keep aliases
> --
>
> Key: CALCITE-3172
> URL: https://issues.apache.org/jira/browse/CALCITE-3172
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Ruben Quesada Lopez
>Assignee: Ruben Quesada Lopez
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.21.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> {{RelBuilder#empty}} method _"Creates a relational expression that reads from
> an input and throws all of the rows away... The default implementation
> creates a Values with the same specified row type as the input"_. However,
> even though the "original" row-type is kept, its aliases are lost, which can
> lead to issues when building a plan (especially because this method is used
> internally by other methods, e.g. by RelBuilder#filter with an "always false"
> filter).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (CALCITE-3172) RelBuilder#empty does not keep aliases
[
https://issues.apache.org/jira/browse/CALCITE-3172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Quesada Lopez updated CALCITE-3172:
-
Description: {{RelBuilder#empty}} method _"Creates a relational expression
that reads from an input and throws all of the rows away... The default
implementation creates a Values with the same specified row type as the
input"_. However, even though the "original" row-type is kept, its aliases are
lost, which can lead to issues when building a plan (especially because this
method is used internally by other methods, e.g. by RelBuilder#filter with an
"always false" filter). (was: {{RelBuilder#empty}} method _"Creates a
relational expression that reads from an input and throws all of the rows
away... The default implementation creates a Values with the same specified row
type as the input"_. However, even though the "original" rowType is kept, its
aliases are lost, which can lead to issues when building a plan (especially
because this method is used internally by other ones, e.g. by RelBuilder#filter
with an "always false" filter).)
> RelBuilder#empty does not keep aliases
> --
>
> Key: CALCITE-3172
> URL: https://issues.apache.org/jira/browse/CALCITE-3172
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Ruben Quesada Lopez
>Assignee: Ruben Quesada Lopez
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.21.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{RelBuilder#empty}} method _"Creates a relational expression that reads from
> an input and throws all of the rows away... The default implementation
> creates a Values with the same specified row type as the input"_. However,
> even though the "original" row-type is kept, its aliases are lost, which can
> lead to issues when building a plan (especially because this method is used
> internally by other methods, e.g. by RelBuilder#filter with an "always false"
> filter).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (CALCITE-3172) RelBuilder#empty does not keep aliases
[
https://issues.apache.org/jira/browse/CALCITE-3172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878670#comment-16878670
]
Ruben Quesada Lopez edited comment on CALCITE-3172 at 7/4/19 2:02 PM:
--
Example of plan that will throw an Exception:
{code}
RelBuilder builder = ...
RexNode myFilter = ... // could be a parameter whose value we do not know
beforehand
builder
.scan("DEPT").as("d")
.filter(myFilter)
.project(
builder.field(1, "d", "DEPTNO"), // <-- exception if myFilter is
literal(false)
builder.field(1, "d", "DNAME"))
.build();
{code}
was (Author: rubenql):
Example of plan that will throw an Exception:
{code}
RelBuilder builder = ...
RexNode myFilter = ... // could be a parameter whose value we do not know
beforehand
builder
.scan("DEPT").as("d")
.filter(myFilter)
.project(
builder.field(1, "d", "DEPTNO"), // <-- exception if myFilter
is literal(false)
builder.field(1, "d", "DNAME"))
.build();
{code}
> RelBuilder#empty does not keep aliases
> --
>
> Key: CALCITE-3172
> URL: https://issues.apache.org/jira/browse/CALCITE-3172
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Ruben Quesada Lopez
>Assignee: Ruben Quesada Lopez
>Priority: Minor
> Fix For: 1.21.0
>
>
> {{RelBuilder#empty}} method _"Creates a relational expression that reads from
> an input and throws all of the rows away... The default implementation
> creates a Values with the same specified row type as the input"_. However,
> even though the "original" rowType is kept, its aliases are lost, which can
> lead to issues when building a plan (especially because this method is used
> internally by other ones, e.g. by RelBuilder#filter with an "always false"
> filter).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-3172) RelBuilder#empty does not keep aliases
[
https://issues.apache.org/jira/browse/CALCITE-3172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878670#comment-16878670
]
Ruben Quesada Lopez commented on CALCITE-3172:
--
Example of plan that will throw an Exception:
{code}
RelBuilder builder = ...
RexNode myFilter = ... // could be a parameter whose value we do not know
beforehand
builder
.scan("DEPT").as("d")
.filter(myFilter)
.project(
builder.field(1, "d", "DEPTNO"), // <-- exception if myFilter
is literal(false)
builder.field(1, "d", "DNAME"))
.build();
{code}
> RelBuilder#empty does not keep aliases
> --
>
> Key: CALCITE-3172
> URL: https://issues.apache.org/jira/browse/CALCITE-3172
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Ruben Quesada Lopez
>Assignee: Ruben Quesada Lopez
>Priority: Minor
> Fix For: 1.21.0
>
>
> {{RelBuilder#empty}} method _"Creates a relational expression that reads from
> an input and throws all of the rows away... The default implementation
> creates a Values with the same specified row type as the input"_. However,
> even though the "original" rowType is kept, its aliases are lost, which can
> lead to issues when building a plan (especially because this method is used
> internally by other ones, e.g. by RelBuilder#filter with an "always false"
> filter).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CALCITE-3172) RelBuilder#empty does not keep aliases
Ruben Quesada Lopez created CALCITE-3172:
Summary: RelBuilder#empty does not keep aliases
Key: CALCITE-3172
URL: https://issues.apache.org/jira/browse/CALCITE-3172
Project: Calcite
Issue Type: Bug
Components: core
Reporter: Ruben Quesada Lopez
Assignee: Ruben Quesada Lopez
Fix For: 1.21.0
{{RelBuilder#empty}} method _"Creates a relational expression that reads from
an input and throws all of the rows away... The default implementation creates
a Values with the same specified row type as the input"_. However, even though
the "original" rowType is kept, its aliases are lost, which can lead to issues
when building a plan (especially because this method is used internally by
other ones, e.g. by RelBuilder#filter with an "always false" filter).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-3165) Project#accept(RexShuttle shuttle) does not update rowType
[
https://issues.apache.org/jira/browse/CALCITE-3165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876807#comment-16876807
]
Ruben Quesada Lopez commented on CALCITE-3165:
--
Provided PR with unit test (which is a dummy scenario, but shows the issue).
> Project#accept(RexShuttle shuttle) does not update rowType
> --
>
> Key: CALCITE-3165
> URL: https://issues.apache.org/jira/browse/CALCITE-3165
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Ruben Quesada Lopez
>Assignee: Ruben Quesada Lopez
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{Project#accept(RexShuttle shuttle)}} can return a copy of the original
> Project, if the list of expressions is modified by the shuttle. However, this
> Project copy will maintain the original rowType, which might be wrong because
> this type is based on the expressions list (which was modified by the shuttle)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (CALCITE-3165) Project#accept(RexShuttle shuttle) does not update rowType
[
https://issues.apache.org/jira/browse/CALCITE-3165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876315#comment-16876315
]
Ruben Quesada Lopez edited comment on CALCITE-3165 at 7/1/19 4:15 PM:
--
Currently, {{Project#accept(RexShuttle shuttle)}} is implemented as:
{code}
public RelNode accept(RexShuttle shuttle) {
List exps = shuttle.apply(this.exps);
if (this.exps == exps) {
return this;
}
return copy(traitSet, getInput(), exps, rowType);
}
{code}
Thus, the original rowType is always kept, even though we know that the list of
expressions has been changed, so we might have potentially a different rowType.
The idea would be doing something in the line of:
{code}
public RelNode accept(RexShuttle shuttle) {
List exps = shuttle.apply(this.exps);
if (this.exps == exps) {
return this;
}
final RelDataType newRowType =
RexUtil.createStructType(
getInput().getCluster().getTypeFactory(),
exps,
rowType.getFieldNames(),
SqlValidatorUtil.F_SUGGESTER);
return copy(traitSet, getInput(), exps, newRowType);
}
{code}
I'll try to provide a unit test to show the problem.
was (Author: rubenql):
Currently, Project#accept(RexShuttle shuttle) does:
{code}
public RelNode accept(RexShuttle shuttle) {
List exps = shuttle.apply(this.exps);
if (this.exps == exps) {
return this;
}
return copy(traitSet, getInput(), exps, rowType);
}
{code}
Thus, the original rowType is always kept, even though we know that the list of
expressions has been changed, so we might have potentially a different rowType.
The idea would be doing something in the line of:
{code}
public RelNode accept(RexShuttle shuttle) {
List exps = shuttle.apply(this.exps);
if (this.exps == exps) {
return this;
}
final RelDataType newRowType =
RexUtil.createStructType(
getInput().getCluster().getTypeFactory(),
exps,
rowType.getFieldNames(),
SqlValidatorUtil.F_SUGGESTER);
return copy(traitSet, getInput(), exps, newRowType);
}
{code}
I'll try to provide a unit test to show the problem.
> Project#accept(RexShuttle shuttle) does not update rowType
> --
>
> Key: CALCITE-3165
> URL: https://issues.apache.org/jira/browse/CALCITE-3165
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Ruben Quesada Lopez
>Assignee: Ruben Quesada Lopez
>Priority: Minor
>
> {{Project#accept(RexShuttle shuttle)}} can return a copy of the original
> Project, if the list of expressions is modified by the shuttle. However, this
> Project copy will maintain the original rowType, which might be wrong because
> this type is based on the expressions list (which was modified by the shuttle)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-3165) Project#accept(RexShuttle shuttle) does not update rowType
[
https://issues.apache.org/jira/browse/CALCITE-3165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876315#comment-16876315
]
Ruben Quesada Lopez commented on CALCITE-3165:
--
Currently, Project#accept(RexShuttle shuttle) does:
{code}
public RelNode accept(RexShuttle shuttle) {
List exps = shuttle.apply(this.exps);
if (this.exps == exps) {
return this;
}
return copy(traitSet, getInput(), exps, rowType);
}
{code}
Thus, the original rowType is always kept, even though we know that the list of
expressions has been changed, so we might have potentially a different rowType.
The idea would be doing something in the line of:
{code}
public RelNode accept(RexShuttle shuttle) {
List exps = shuttle.apply(this.exps);
if (this.exps == exps) {
return this;
}
final RelDataType newRowType =
RexUtil.createStructType(
getInput().getCluster().getTypeFactory(),
exps,
rowType.getFieldNames(),
SqlValidatorUtil.F_SUGGESTER);
return copy(traitSet, getInput(), exps, newRowType);
}
{code}
I'll try to provide a unit test to show the problem.
> Project#accept(RexShuttle shuttle) does not update rowType
> --
>
> Key: CALCITE-3165
> URL: https://issues.apache.org/jira/browse/CALCITE-3165
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Ruben Quesada Lopez
>Assignee: Ruben Quesada Lopez
>Priority: Minor
>
> {{Project#accept(RexShuttle shuttle)}} can return a copy of the original
> Project, if the list of expressions is modified by the shuttle. However, this
> Project copy will maintain the original rowType, which might be wrong because
> this type is based on the expressions list (which was modified by the shuttle)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CALCITE-3165) Project#accept(RexShuttle shuttle) does not update rowType
Ruben Quesada Lopez created CALCITE-3165:
Summary: Project#accept(RexShuttle shuttle) does not update rowType
Key: CALCITE-3165
URL: https://issues.apache.org/jira/browse/CALCITE-3165
Project: Calcite
Issue Type: Bug
Components: core
Reporter: Ruben Quesada Lopez
Assignee: Ruben Quesada Lopez
{{Project#accept(RexShuttle shuttle)}} can return a copy of the original
Project, if the list of expressions is modified by the shuttle. However, this
Project copy will maintain the original rowType, which might be wrong because
this type is based on the expressions list (which was modified by the shuttle)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (CALCITE-3125) Remove completely CorrelateJoinType
[ https://issues.apache.org/jira/browse/CALCITE-3125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruben Quesada Lopez resolved CALCITE-3125. -- Resolution: Fixed Fixed in https://github.com/apache/calcite/commit/64a61cda18ada294e74ed6841d714d067bf0cb4e > Remove completely CorrelateJoinType > --- > > Key: CALCITE-3125 > URL: https://issues.apache.org/jira/browse/CALCITE-3125 > Project: Calcite > Issue Type: Task >Reporter: Ruben Quesada Lopez >Assignee: Ruben Quesada Lopez >Priority: Minor > Labels: pull-request-available > Fix For: 1.21.0 > > Time Spent: 20m > Remaining Estimate: 0h > > CorrelateJoinType was deprecated in 1.20 via CALCITE-3119. > It shall be kept as deprecated for one minor release (1.20); and then remove > completely for 1.21. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (CALCITE-3133) Remove completely SemiJoinType
[ https://issues.apache.org/jira/browse/CALCITE-3133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruben Quesada Lopez resolved CALCITE-3133. -- Resolution: Fixed Fixed in https://github.com/apache/calcite/commit/ceb13a134502b950ce1ce1c25d9267aa51ca14b5 > Remove completely SemiJoinType > -- > > Key: CALCITE-3133 > URL: https://issues.apache.org/jira/browse/CALCITE-3133 > Project: Calcite > Issue Type: Task >Reporter: Ruben Quesada Lopez >Assignee: Ruben Quesada Lopez >Priority: Minor > Labels: pull-request-available > Fix For: 1.21.0 > > Time Spent: 20m > Remaining Estimate: 0h > > SemiJoinType was deprecated in 1.20 via CALCITE-2969. > It shall be kept as deprecated for one minor release (1.20); and then remove > completely for 1.21. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (CALCITE-3161) Remove deprecated SemiJoinType usage completely
[ https://issues.apache.org/jira/browse/CALCITE-3161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruben Quesada Lopez closed CALCITE-3161. Resolution: Duplicate Closing as duplicated of CALCITE-3133 > Remove deprecated SemiJoinType usage completely > --- > > Key: CALCITE-3161 > URL: https://issues.apache.org/jira/browse/CALCITE-3161 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.20.0 >Reporter: Danny Chan >Assignee: Danny Chan >Priority: Major > Fix For: 1.21.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CALCITE-3161) Remove deprecated SemiJoinType usage completely
[ https://issues.apache.org/jira/browse/CALCITE-3161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876067#comment-16876067 ] Ruben Quesada Lopez commented on CALCITE-3161: -- Ok, [~danny0405], I'll start working on CALCITE-3133 and flag this one as duplicated. > Remove deprecated SemiJoinType usage completely > --- > > Key: CALCITE-3161 > URL: https://issues.apache.org/jira/browse/CALCITE-3161 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.20.0 >Reporter: Danny Chan >Assignee: Danny Chan >Priority: Major > Fix For: 1.21.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (CALCITE-3161) Remove deprecated SemiJoinType usage completely
[ https://issues.apache.org/jira/browse/CALCITE-3161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876042#comment-16876042 ] Ruben Quesada Lopez edited comment on CALCITE-3161 at 7/1/19 9:28 AM: -- [~danny0405], sorry, I had already created CALCITE-3133 "Remove completely SemiJoinType" for this purpose, but I have not started yet. If you are going to work on this right now, I can close my ticket as duplicated and we keep the current one; or you can close the current one and work on CALCITE-3133, whatever you prefer. was (Author: rubenql): [~danny0405], sorry, I had already created CALCITE-3133 "Remove completely SemiJoinType" for this purpose, but I have not started yet. If you are going to work on this right now I can close my ticket as duplicated and we keep the current one, or you can close the current one and work on CALCITE-3133, whatever you prefer. > Remove deprecated SemiJoinType usage completely > --- > > Key: CALCITE-3161 > URL: https://issues.apache.org/jira/browse/CALCITE-3161 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.20.0 >Reporter: Danny Chan >Assignee: Danny Chan >Priority: Major > Fix For: 1.21.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (CALCITE-3161) Remove deprecated SemiJoinType usage completely
[ https://issues.apache.org/jira/browse/CALCITE-3161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876042#comment-16876042 ] Ruben Quesada Lopez edited comment on CALCITE-3161 at 7/1/19 9:28 AM: -- [~danny0405], sorry, I had already created CALCITE-3133 "Remove completely SemiJoinType" for this purpose, but I have not started yet. If you are going to work on this right now I can close my ticket as duplicated and we keep the current one, or you can close the current one and work on CALCITE-3133, whatever you prefer. was (Author: rubenql): [~danny0405], sorry, I had already created CALCITE-3133 "Remove completely SemiJoinType" for this purpose, but if you are going to work on this right now I can close my ticket as duplicated and we keep the current one. > Remove deprecated SemiJoinType usage completely > --- > > Key: CALCITE-3161 > URL: https://issues.apache.org/jira/browse/CALCITE-3161 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.20.0 >Reporter: Danny Chan >Assignee: Danny Chan >Priority: Major > Fix For: 1.21.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CALCITE-3161) Remove deprecated SemiJoinType usage completely
[ https://issues.apache.org/jira/browse/CALCITE-3161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876042#comment-16876042 ] Ruben Quesada Lopez commented on CALCITE-3161: -- [~danny0405], sorry, I had already created CALCITE-3133 "Remove completely SemiJoinType" for this purpose, but if you are going to work on this right now I can close my ticket as duplicated and we keep the current one. > Remove deprecated SemiJoinType usage completely > --- > > Key: CALCITE-3161 > URL: https://issues.apache.org/jira/browse/CALCITE-3161 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.20.0 >Reporter: Danny Chan >Assignee: Danny Chan >Priority: Major > Fix For: 1.21.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (CALCITE-3133) Remove completely SemiJoinType
[ https://issues.apache.org/jira/browse/CALCITE-3133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruben Quesada Lopez reassigned CALCITE-3133: Assignee: Ruben Quesada Lopez > Remove completely SemiJoinType > -- > > Key: CALCITE-3133 > URL: https://issues.apache.org/jira/browse/CALCITE-3133 > Project: Calcite > Issue Type: Task >Reporter: Ruben Quesada Lopez >Assignee: Ruben Quesada Lopez >Priority: Minor > Fix For: 1.21.0 > > > SemiJoinType was deprecated in 1.20 via CALCITE-2969. > It shall be kept as deprecated for one minor release (1.20); and then remove > completely for 1.21. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-3133) Remove completely SemiJoinType
Ruben Quesada Lopez created CALCITE-3133: Summary: Remove completely SemiJoinType Key: CALCITE-3133 URL: https://issues.apache.org/jira/browse/CALCITE-3133 Project: Calcite Issue Type: Task Reporter: Ruben Quesada Lopez Fix For: 1.21.0 SemiJoinType was deprecated in 1.20 via CALCITE-2969. It shall be kept as deprecated for one minor release (1.20); and then remove completely for 1.21. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CALCITE-3119) Deprecate Linq4j CorrelateJoinType (in favor of JoinType)
[ https://issues.apache.org/jira/browse/CALCITE-3119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16863021#comment-16863021 ] Ruben Quesada Lopez commented on CALCITE-3119: -- [~michaelmior], in my opinion, yes. > Deprecate Linq4j CorrelateJoinType (in favor of JoinType) > - > > Key: CALCITE-3119 > URL: https://issues.apache.org/jira/browse/CALCITE-3119 > Project: Calcite > Issue Type: Task >Reporter: Ruben Quesada Lopez >Assignee: Ruben Quesada Lopez >Priority: Major > Labels: pull-request-available > Fix For: 1.20.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > CALCITE-2968 introduced in Linq4j a new (more general) enum JoinType. As a > result, we have right now two enums for (almost) the same purpose: > CorrelateJoinType (used for Correlation, thus defining only INNER, LEFT, > SEMI, ANTI) and the new JoinType (can be used for any purpose, defining all > types of join, mirroring core's JoinRelType). > The idea is to flag CorrelateJoinType as deprecated, and replace all its > usages by JoinType. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CALCITE-3109) Improvements on algebraic operators to express recursive queries (RepeatUnion & TableSpool)
[
https://issues.apache.org/jira/browse/CALCITE-3109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Quesada Lopez updated CALCITE-3109:
-
Fix Version/s: 1.21.0
> Improvements on algebraic operators to express recursive queries (RepeatUnion
> & TableSpool)
> ---
>
> Key: CALCITE-3109
> URL: https://issues.apache.org/jira/browse/CALCITE-3109
> Project: Calcite
> Issue Type: Task
>Reporter: Ruben Quesada Lopez
>Assignee: Ruben Quesada Lopez
>Priority: Minor
> Fix For: 1.21.0
>
>
> After the implementation of CALCITE-2812, there are some pending details to
> be addressed in order to make the RepeatUnion and TableSpool more consistent.
> TableSpool:
> - Replace {{String tableName}} with a {{Table}} (perhaps wrapped in a
> {{RelOptTable}}).
> RepeatUnion:
> - Rename {{maxRep}} as {{iterationLimit}}. Change this field to allow all
> integer values, and treat all negative values as "no limit".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
