[jira] [Commented] (IMPALA-7957) UNION ALL query returns incorrect results
[
https://issues.apache.org/jira/browse/IMPALA-7957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16856817#comment-16856817
]
ASF subversion and git services commented on IMPALA-7957:
-
Commit ae8295118191486f31da4d8d3c9d0f7e7e5d4b3a in impala's branch
refs/heads/master from stiga-huang
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=ae82951 ]
IMPALA-7957: Fix slot equivalences may be enforced multiple times
Predicates can be divided into three types according to the way they are
generated:
1) origin predicates that come from the query
2) auxiliary equal predicates generated for equivalence between a
label(alias) and its real expression
3) inferred predicates that inferred from the slot equivalences graph
The slot equivalences graph (valueTransferGraph in Analyzer) is
generated by the first two kinds of predicates. Analyzer will create
equivalence predicates for a PlanNode based on the unassigned predicates
and the valueTransferGraph. However, the current implementation can't
avoid creating inferred predicates that are duplicated with previously
created inferred predicates if they have been assigned before.
Duplicated inferred predicates are either redundant or wrong. Say, if we
create predicate p1: s1 = s2 for the current PlanNode and p1 duplicates
with a previously inferred predicate p0: s1 = s2 (same as s2 = s1), we
can prove that p1 is redundant or wrong:
1) p0 must have been assigned. Otherwise, p0 will be in the unassigned
conjuncts list and p1 won't be created.
2) p0 must have been assigned to an offspring node of the current
PlanNode since we create the PlanNodes in a depth first manner.
3) The origin predicates that infer to p0 have been assigned to an
offspring node too.
Then, rows that should be rejected have been filtered out either by p0
or the origin predicates that infer to p0. What's worse, assigning p1 on
top of the origin predicates may wrongly reject rows. Hence, p1 is
either redundant or wrong.
In inferring predicates based on slot equivalence (createEquivConjuncts)
we should update partialEquivSlots based on the previously assigned
equivalence predicates. So slot equivalence won't be enforced multiple
times.
This patch also adds some useful TRACE level logs.
Tests:
* Add tests for UNIONs in inline-view.test
* Run all tests locally in CORE exploration strategy
Change-Id: Ida2d5d8149b217e18ebae61e136848162503653e
Reviewed-on: http://gerrit.cloudera.org:8080/13051
Reviewed-by: Tim Armstrong
Tested-by: Impala Public Jenkins
> UNION ALL query returns incorrect results
> -
>
> Key: IMPALA-7957
> URL: https://issues.apache.org/jira/browse/IMPALA-7957
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Affects Versions: Impala 2.12.0
>Reporter: Luis E Martinez-Poblete
>Assignee: Quanlong Huang
>Priority: Blocker
> Labels: correctness
>
> Synopsis:
> =
> UNION ALL query returns incorrect results
> Problem:
>
> Customer reported a UNION ALL query returning incorrect results. The UNION
> ALL query has 2 legs, but Impala is only returning information from one leg.
> Issue can be reproduced in the latest version of Impala. Below is the
> reproduction case:
> {noformat}
> create table mytest_t (c1 timestamp, c2 timestamp, c3 int, c4 int);
> insert into mytest_t values (now(), ADDDATE (now(),1), 1,1);
> insert into mytest_t values (now(), ADDDATE (now(),1), 2,2);
> insert into mytest_t values (now(), ADDDATE (now(),1), 3,3);
> SELECT t.c1
> FROM
> (SELECT c1, c2
> FROM mytest_t) t
> LEFT JOIN
> (SELECT c1, c2
> FROM mytest_t
> WHERE c2 = c1) t2 ON (t.c2 = t2.c2)
> UNION ALL
> VALUES (NULL)
> {noformat}
> The above query produces the following execution plan:
> {noformat}
> ++
> | Explain String
>|
> ++
> | Max Per-Host Resource Reservation: Memory=34.02MB Threads=5
>|
> | Per-Host Resource Estimates: Memory=2.06GB
>|
> | WARNING: The following tables are missing relevant table and/or column
> statistics. |
> | default.mytest_t
>|
> |
>|
> | PLAN-ROOT SINK
>|
> | |
>|
> | 06:EXCHANGE [UNPARTITIONED]
>|
> | |
[jira] [Commented] (IMPALA-7957) UNION ALL query returns incorrect results
[
https://issues.apache.org/jira/browse/IMPALA-7957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820628#comment-16820628
]
Quanlong Huang commented on IMPALA-7957:
Hi [~Paul.Rogers], I have a patch for this for review:
[https://gerrit.cloudera.org/c/13051]
Could you have a look when you have time?
> UNION ALL query returns incorrect results
> -
>
> Key: IMPALA-7957
> URL: https://issues.apache.org/jira/browse/IMPALA-7957
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Affects Versions: Impala 2.12.0
>Reporter: Luis E Martinez-Poblete
>Assignee: Paul Rogers
>Priority: Blocker
> Labels: correctness
>
> Synopsis:
> =
> UNION ALL query returns incorrect results
> Problem:
>
> Customer reported a UNION ALL query returning incorrect results. The UNION
> ALL query has 2 legs, but Impala is only returning information from one leg.
> Issue can be reproduced in the latest version of Impala. Below is the
> reproduction case:
> {noformat}
> create table mytest_t (c1 timestamp, c2 timestamp, c3 int, c4 int);
> insert into mytest_t values (now(), ADDDATE (now(),1), 1,1);
> insert into mytest_t values (now(), ADDDATE (now(),1), 2,2);
> insert into mytest_t values (now(), ADDDATE (now(),1), 3,3);
> SELECT t.c1
> FROM
> (SELECT c1, c2
> FROM mytest_t) t
> LEFT JOIN
> (SELECT c1, c2
> FROM mytest_t
> WHERE c2 = c1) t2 ON (t.c2 = t2.c2)
> UNION ALL
> VALUES (NULL)
> {noformat}
> The above query produces the following execution plan:
> {noformat}
> ++
> | Explain String
>|
> ++
> | Max Per-Host Resource Reservation: Memory=34.02MB Threads=5
>|
> | Per-Host Resource Estimates: Memory=2.06GB
>|
> | WARNING: The following tables are missing relevant table and/or column
> statistics. |
> | default.mytest_t
>|
> |
>|
> | PLAN-ROOT SINK
>|
> | |
>|
> | 06:EXCHANGE [UNPARTITIONED]
>|
> | |
>|
> | 00:UNION
>|
> | | constant-operands=1
>|
> | |
>|
> | 04:SELECT
>|
> | | predicates: default.mytest_t.c1 = default.mytest_t.c2
>|
> | |
>|
> | 03:HASH JOIN [LEFT OUTER JOIN, BROADCAST]
>|
> | | hash predicates: c2 = c2
>|
> | |
>|
> | |--05:EXCHANGE [BROADCAST]
>|
> | | |
>|
> | | 02:SCAN HDFS [default.mytest_t]
>|
> | | partitions=1/1 files=3 size=192B
>|
> | | predicates: c2 = c1
>|
> | |
>|
> | 01:SCAN HDFS [default.mytest_t]
>|
> |partitions=1/1 files=3 size=192B
>|
> ++
> {noformat}
> The issue is in operator 4:
> {noformat}
> | 04:SELECT |
> | | predicates: default.mytest_t.c1 = default.mytest_t.c2 |
> {noformat}
> It's definitely a bug with predicate placement - that c1 = c2 predicate
> shouldn't be evaluated outside the right branch of the LEFT JOIN.
> Thanks,
> Luis Martinez.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For
[jira] [Commented] (IMPALA-7957) UNION ALL query returns incorrect results
[
https://issues.apache.org/jira/browse/IMPALA-7957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16810221#comment-16810221
]
Tim Armstrong commented on IMPALA-7957:
---
May be connected
> UNION ALL query returns incorrect results
> -
>
> Key: IMPALA-7957
> URL: https://issues.apache.org/jira/browse/IMPALA-7957
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Affects Versions: Impala 2.12.0
>Reporter: Luis E Martinez-Poblete
>Assignee: Paul Rogers
>Priority: Blocker
> Labels: correctness
>
> Synopsis:
> =
> UNION ALL query returns incorrect results
> Problem:
>
> Customer reported a UNION ALL query returning incorrect results. The UNION
> ALL query has 2 legs, but Impala is only returning information from one leg.
> Issue can be reproduced in the latest version of Impala. Below is the
> reproduction case:
> {noformat}
> create table mytest_t (c1 timestamp, c2 timestamp, c3 int, c4 int);
> insert into mytest_t values (now(), ADDDATE (now(),1), 1,1);
> insert into mytest_t values (now(), ADDDATE (now(),1), 2,2);
> insert into mytest_t values (now(), ADDDATE (now(),1), 3,3);
> SELECT t.c1
> FROM
> (SELECT c1, c2
> FROM mytest_t) t
> LEFT JOIN
> (SELECT c1, c2
> FROM mytest_t
> WHERE c2 = c1) t2 ON (t.c2 = t2.c2)
> UNION ALL
> VALUES (NULL)
> {noformat}
> The above query produces the following execution plan:
> {noformat}
> ++
> | Explain String
>|
> ++
> | Max Per-Host Resource Reservation: Memory=34.02MB Threads=5
>|
> | Per-Host Resource Estimates: Memory=2.06GB
>|
> | WARNING: The following tables are missing relevant table and/or column
> statistics. |
> | default.mytest_t
>|
> |
>|
> | PLAN-ROOT SINK
>|
> | |
>|
> | 06:EXCHANGE [UNPARTITIONED]
>|
> | |
>|
> | 00:UNION
>|
> | | constant-operands=1
>|
> | |
>|
> | 04:SELECT
>|
> | | predicates: default.mytest_t.c1 = default.mytest_t.c2
>|
> | |
>|
> | 03:HASH JOIN [LEFT OUTER JOIN, BROADCAST]
>|
> | | hash predicates: c2 = c2
>|
> | |
>|
> | |--05:EXCHANGE [BROADCAST]
>|
> | | |
>|
> | | 02:SCAN HDFS [default.mytest_t]
>|
> | | partitions=1/1 files=3 size=192B
>|
> | | predicates: c2 = c1
>|
> | |
>|
> | 01:SCAN HDFS [default.mytest_t]
>|
> |partitions=1/1 files=3 size=192B
>|
> ++
> {noformat}
> The issue is in operator 4:
> {noformat}
> | 04:SELECT |
> | | predicates: default.mytest_t.c1 = default.mytest_t.c2 |
> {noformat}
> It's definitely a bug with predicate placement - that c1 = c2 predicate
> shouldn't be evaluated outside the right branch of the LEFT JOIN.
> Thanks,
> Luis Martinez.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
