[Impala-ASF-CR] IMPALA-5494: Fixes the selectivity of NOT IN predicates

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change.

Change subject: IMPALA-5494: Fixes the selectivity of NOT IN predicates
..


Patch Set 8: Verified+1

-- 
To view, visit http://gerrit.cloudera.org:8080/7168
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I69e6217257b5618cb63e13b32ba3347fa0483b63
Gerrit-PatchSet: 8
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Vincent Tran 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Vincent Tran 
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5494: Fixes the selectivity of NOT IN predicates

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change.

Change subject: IMPALA-5494: Fixes the selectivity of NOT IN predicates
..


Patch Set 8:

Build started: http://jenkins.impala.io:8080/job/gerrit-verify-dryrun/744/

-- 
To view, visit http://gerrit.cloudera.org:8080/7168
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I69e6217257b5618cb63e13b32ba3347fa0483b63
Gerrit-PatchSet: 8
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Vincent Tran 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Vincent Tran 
Gerrit-HasComments: No

Re: [Impala-ASF-CR] IMPALA-5494: Fixes the selectivity of NOT IN predicates

[Alexander Behm]

Random failure to start Kudu:
F0616 16:05:10.200038 23162 tablet_server_main.cc:72] Check failed: _s.ok()
Bad status: Service unavailable: Cannot initialize clock: Error reading
clock. Clock considered unsynchronized

Will retry.

On Fri, Jun 16, 2017 at 11:24 AM, Impala Public Jenkins (Code Review) <
ger...@cloudera.org> wrote:

> Impala Public Jenkins has posted comments on this change.
>
> Change subject: IMPALA-5494: Fixes the selectivity of NOT IN predicates
> ..
>
>
> Patch Set 8: Verified-1
>
> Build failed: http://jenkins.impala.io:8080/job/gerrit-verify-dryrun/742/
>
> --
> To view, visit http://gerrit.cloudera.org:8080/7168
> To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
>
> Gerrit-MessageType: comment
> Gerrit-Change-Id: I69e6217257b5618cb63e13b32ba3347fa0483b63
> Gerrit-PatchSet: 8
> Gerrit-Project: Impala-ASF
> Gerrit-Branch: master
> Gerrit-Owner: Vincent Tran 
> Gerrit-Reviewer: Alex Behm 
> Gerrit-Reviewer: Bharath Vissapragada 
> Gerrit-Reviewer: Impala Public Jenkins
> Gerrit-Reviewer: Vincent Tran 
> Gerrit-HasComments: No
>
> --
> You received this message because you are subscribed to the Google Groups
> "impala-cr" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to impala-cr+unsubscr...@cloudera.com.
> For more options, visit https://groups.google.com/a/cloudera.com/d/optout.
>

[Impala-ASF-CR] IMPALA-5494: Fixes the selectivity of NOT IN predicates

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change.

Change subject: IMPALA-5494: Fixes the selectivity of NOT IN predicates
..


Patch Set 8: Verified-1

Build failed: http://jenkins.impala.io:8080/job/gerrit-verify-dryrun/742/

-- 
To view, visit http://gerrit.cloudera.org:8080/7168
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I69e6217257b5618cb63e13b32ba3347fa0483b63
Gerrit-PatchSet: 8
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Vincent Tran 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Vincent Tran 
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5494: Fixes the selectivity of NOT IN predicates

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change.

Change subject: IMPALA-5494: Fixes the selectivity of NOT IN predicates
..


Patch Set 8:

Build started: http://jenkins.impala.io:8080/job/gerrit-verify-dryrun/742/

-- 
To view, visit http://gerrit.cloudera.org:8080/7168
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I69e6217257b5618cb63e13b32ba3347fa0483b63
Gerrit-PatchSet: 8
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Vincent Tran 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Vincent Tran 
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5494: Fixes the selectivity of NOT IN predicates

[Alex Behm (Code Review)]

Alex Behm has posted comments on this change.

Change subject: IMPALA-5494: Fixes the selectivity of NOT IN predicates
..


Patch Set 8: Code-Review+2

-- 
To view, visit http://gerrit.cloudera.org:8080/7168
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I69e6217257b5618cb63e13b32ba3347fa0483b63
Gerrit-PatchSet: 8
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Vincent Tran 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Vincent Tran 
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5494: Fixes the selectivity of NOT IN predicates

[Vincent Tran (Code Review)]

Hello Impala Public Jenkins, Alex Behm,

I'd like you to reexamine a change.  Please visit

http://gerrit.cloudera.org:8080/7168

to look at the new patch set (#8).

Change subject: IMPALA-5494: Fixes the selectivity of NOT IN predicates
..

IMPALA-5494: Fixes the selectivity of NOT IN predicates

This change modifies the logic of NOT IN predicate so that
the planner can calculate the correct node cardinality. Prior
to this change, both IN and NOT IN predicates shared the
same selectivity, which resulted in the same cardinality
during planning.

The selectivity is calculated by the following heuristic:

selectivity = 1 - (num of predicate children /
num of distinct values)

Change-Id: I69e6217257b5618cb63e13b32ba3347fa0483b63
---
M fe/src/main/java/org/apache/impala/analysis/InPredicate.java
A fe/src/test/java/org/apache/impala/analysis/ExprSelectivityTest.java
M 
testdata/workloads/functional-planner/queries/PlannerTest/kudu-selectivity.test
3 files changed, 73 insertions(+), 6 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/7168/8
-- 
To view, visit http://gerrit.cloudera.org:8080/7168
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I69e6217257b5618cb63e13b32ba3347fa0483b63
Gerrit-PatchSet: 8
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Vincent Tran 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Vincent Tran 

[Impala-ASF-CR] IMPALA-5494: Fixes the selectivity of NOT IN predicates

[Vincent Tran (Code Review)]

Vincent Tran has posted comments on this change.

Change subject: IMPALA-5494: Fixes the selectivity of NOT IN predicates
..


Patch Set 7:

One query plan in kudu-selectivity.test had a NOT IN predicate. The actual scan 
node cardinality was changed from 4 to 5. The order of predicates was also 
changed (NOT IN has the highest cardinality so was moved to the most right side 
of the predicate list). Private build looks good now.

-- 
To view, visit http://gerrit.cloudera.org:8080/7168
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I69e6217257b5618cb63e13b32ba3347fa0483b63
Gerrit-PatchSet: 7
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Vincent Tran 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Vincent Tran 
Gerrit-HasComments: No

Re: [Impala-ASF-CR] IMPALA-5494: Fixes the selectivity of NOT IN predicates

[Alexander Behm]

Planner test failed in PlannerTest.testKuduSelectivity. Please fix,

On Wed, Jun 14, 2017 at 9:16 PM, Impala Public Jenkins (Code Review) <
ger...@cloudera.org> wrote:

> Impala Public Jenkins has posted comments on this change.
>
> Change subject: IMPALA-5494: Fixes the selectivity of NOT IN predicates
> ..
>
>
> Patch Set 7: Verified-1
>
> Build failed: http://jenkins.impala.io:8080/job/gerrit-verify-dryrun/731/
>
> --
> To view, visit http://gerrit.cloudera.org:8080/7168
> To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
>
> Gerrit-MessageType: comment
> Gerrit-Change-Id: I69e6217257b5618cb63e13b32ba3347fa0483b63
> Gerrit-PatchSet: 7
> Gerrit-Project: Impala-ASF
> Gerrit-Branch: master
> Gerrit-Owner: Vincent Tran 
> Gerrit-Reviewer: Alex Behm 
> Gerrit-Reviewer: Bharath Vissapragada 
> Gerrit-Reviewer: Impala Public Jenkins
> Gerrit-Reviewer: Vincent Tran 
> Gerrit-HasComments: No
>
> --
> You received this message because you are subscribed to the Google Groups
> "impala-cr" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to impala-cr+unsubscr...@cloudera.com.
> For more options, visit https://groups.google.com/a/cloudera.com/d/optout.
>

[Impala-ASF-CR] IMPALA-5494: Fixes the selectivity of NOT IN predicates

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change.

Change subject: IMPALA-5494: Fixes the selectivity of NOT IN predicates
..


Patch Set 7: Verified-1

Build failed: http://jenkins.impala.io:8080/job/gerrit-verify-dryrun/731/

-- 
To view, visit http://gerrit.cloudera.org:8080/7168
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I69e6217257b5618cb63e13b32ba3347fa0483b63
Gerrit-PatchSet: 7
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Vincent Tran 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Vincent Tran 
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5494: Fixes the selectivity of NOT IN predicates

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change.

Change subject: IMPALA-5494: Fixes the selectivity of NOT IN predicates
..


Patch Set 7:

Build started: http://jenkins.impala.io:8080/job/gerrit-verify-dryrun/731/

-- 
To view, visit http://gerrit.cloudera.org:8080/7168
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I69e6217257b5618cb63e13b32ba3347fa0483b63
Gerrit-PatchSet: 7
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Vincent Tran 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Vincent Tran 
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5494: Fixes the selectivity of NOT IN predicates

[Alex Behm (Code Review)]

Alex Behm has posted comments on this change.

Change subject: IMPALA-5494: Fixes the selectivity of NOT IN predicates
..


Patch Set 7: Code-Review+2

-- 
To view, visit http://gerrit.cloudera.org:8080/7168
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I69e6217257b5618cb63e13b32ba3347fa0483b63
Gerrit-PatchSet: 7
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Vincent Tran 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Vincent Tran 
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5494: Fixes the selectivity of NOT IN predicates

[Vincent Tran (Code Review)]

Vincent Tran has posted comments on this change.

Change subject: IMPALA-5494: Fixes the selectivity of NOT IN predicates
..


Patch Set 6:

(6 comments)

http://gerrit.cloudera.org:8080/#/c/7168/6/fe/src/test/java/org/apache/impala/analysis/ExprSelectivityTest.java
File fe/src/test/java/org/apache/impala/analysis/ExprSelectivityTest.java:

Line 31: public void verifySelectivityStmt(String stmtStr, double 
expectedSel)
> we indent 2
Done


Line 33:   SelectStmt stmt = (SelectStmt) ParsesOk(stmtStr);
> simplify with AnalyzesOk()
Done


Line 38:   assertEquals(calculatedSel,expectedSel);
> space after comma
Done


Line 43:   String stmtStr = "select * from functional.alltypes where id " +
> simplify this to "select  from functional.alltypes"
Done


Line 52: private double getPredSel(int numPredChild) {
> Very specific to IN predicate. How about we hardcode the expected values fo
Done


Line 69:   verifySel("in (1,3,5,7)", getPredSel(4));
> Seems cleaner to provide the full expr as input and not assume "id". That w
Done


-- 
To view, visit http://gerrit.cloudera.org:8080/7168
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I69e6217257b5618cb63e13b32ba3347fa0483b63
Gerrit-PatchSet: 6
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Vincent Tran 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Vincent Tran 
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5494: Fixes the selectivity of NOT IN predicates

[Alex Behm (Code Review)]

Alex Behm has posted comments on this change.

Change subject: IMPALA-5494: Fixes the selectivity of NOT IN predicates
..


Patch Set 6:

(6 comments)

Thanks for adding the new test!

http://gerrit.cloudera.org:8080/#/c/7168/6/fe/src/test/java/org/apache/impala/analysis/ExprSelectivityTest.java
File fe/src/test/java/org/apache/impala/analysis/ExprSelectivityTest.java:

Line 31: public void verifySelectivityStmt(String stmtStr, double 
expectedSel)
we indent 2


Line 33:   SelectStmt stmt = (SelectStmt) ParsesOk(stmtStr);
simplify with AnalyzesOk()


Line 38:   assertEquals(calculatedSel,expectedSel);
space after comma


Line 43:   String stmtStr = "select * from functional.alltypes where id " +
simplify this to "select  from functional.alltypes"


Line 52: private double getPredSel(int numPredChild) {
Very specific to IN predicate. How about we hardcode the expected values for 
now? It's not clear to me that it makes sense to re-implement the same logic 
again for testing purposes.


Line 69:   verifySel("in (1,3,5,7)", getPredSel(4));
Seems cleaner to provide the full expr as input and not assume "id". That way 
this framework should work for arbitrary exprs.


-- 
To view, visit http://gerrit.cloudera.org:8080/7168
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I69e6217257b5618cb63e13b32ba3347fa0483b63
Gerrit-PatchSet: 6
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Vincent Tran 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Vincent Tran 
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5494: Fixes the selectivity of NOT IN predicates

[Vincent Tran (Code Review)]

Vincent Tran has uploaded a new patch set (#6).

Change subject: IMPALA-5494: Fixes the selectivity of NOT IN predicates
..

IMPALA-5494: Fixes the selectivity of NOT IN predicates

This change modifies the logic of NOT IN predicate so that
the planner can calculate the correct node cardinality. Prior
to this change, both IN and NOT IN predicates shared the
same selectivity, which resulted in the same cardinality
during planning.

The selectivity is calculated by the following heuristic:

selectivity = 1 - (num of predicate children /
num of distinct values)

Change-Id: I69e6217257b5618cb63e13b32ba3347fa0483b63
---
M fe/src/main/java/org/apache/impala/analysis/InPredicate.java
A fe/src/test/java/org/apache/impala/analysis/ExprSelectivityTest.java
2 files changed, 79 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/7168/6
-- 
To view, visit http://gerrit.cloudera.org:8080/7168
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I69e6217257b5618cb63e13b32ba3347fa0483b63
Gerrit-PatchSet: 6
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Vincent Tran 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Vincent Tran 

[Impala-ASF-CR] IMPALA-5494: Fixes the selectivity of NOT IN predicates

[Vincent Tran (Code Review)]

Vincent Tran has uploaded a new patch set (#5).

Change subject: IMPALA-5494: Fixes the selectivity of NOT IN predicates
..

IMPALA-5494: Fixes the selectivity of NOT IN predicates

This change modifies the logic of NOT IN predicate so that
the planner can calculate the correct node cardinality. Prior
to this change, both IN and NOT IN predicates shared the
same selectivity, which resulted in the same cardinality
during planning.

The selectivity is calculated by the following heuristic:

selectivity = 1 - (num of predicate children /
num of distinct values)

Change-Id: I69e6217257b5618cb63e13b32ba3347fa0483b63
---
M fe/src/main/java/org/apache/impala/analysis/InPredicate.java
A fe/src/test/java/org/apache/impala/analysis/ExprSelectivityTest.java
2 files changed, 79 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/7168/5
-- 
To view, visit http://gerrit.cloudera.org:8080/7168
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I69e6217257b5618cb63e13b32ba3347fa0483b63
Gerrit-PatchSet: 5
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Vincent Tran 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Vincent Tran 

[Impala-ASF-CR] IMPALA-5494: Fixes the selectivity of NOT IN predicates

[Vincent Tran (Code Review)]

Vincent Tran has uploaded a new patch set (#4).

Change subject: IMPALA-5494: Fixes the selectivity of NOT IN predicates
..

IMPALA-5494: Fixes the selectivity of NOT IN predicates

This change modifies the logic of NOT IN predicate so that
the planner can calculate the correct node cardinality. Prior
to this change, both IN and NOT IN predicates shared the
same selectivity, which resulted in the same cardinality
during planning.

The selectivity is calculated by the following heuristic:

selectivity = 1 - (num of predicate children /
num of distinct values)

Change-Id: I69e6217257b5618cb63e13b32ba3347fa0483b63
---
M fe/src/main/java/org/apache/impala/analysis/InPredicate.java
1 file changed, 7 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/7168/4
-- 
To view, visit http://gerrit.cloudera.org:8080/7168
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I69e6217257b5618cb63e13b32ba3347fa0483b63
Gerrit-PatchSet: 4
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Vincent Tran 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Vincent Tran 

[Impala-ASF-CR] IMPALA-5494: Fixes the selectivity of NOT IN predicates

[Vincent Tran (Code Review)]

Vincent Tran has posted comments on this change.

Change subject: IMPALA-5494: Fixes the selectivity of NOT IN predicates
..


Patch Set 3:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/7168/3/tests/metadata/test_explain.py
File tests/metadata/test_explain.py:

Line 105: result = self.execute_query("explain select * from %s.%s where id 
"
> Instead of this end-to-end test, please add a new FE unit test ExprSelectiv
That makes sense. Done.


Line 105: result = self.execute_query("explain select * from %s.%s where id 
"
> Agree, e-e test seems like an overkill.
Done


-- 
To view, visit http://gerrit.cloudera.org:8080/7168
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I69e6217257b5618cb63e13b32ba3347fa0483b63
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Vincent Tran 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Vincent Tran 
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5494: Fixes the selectivity of NOT IN predicates

[Bharath Vissapragada (Code Review)]

Bharath Vissapragada has posted comments on this change.

Change subject: IMPALA-5494: Fixes the selectivity of NOT IN predicates
..


Patch Set 3:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/7168/3/tests/metadata/test_explain.py
File tests/metadata/test_explain.py:

Line 105: result = self.execute_query("explain select * from %s.%s where id 
"
> Instead of this end-to-end test, please add a new FE unit test ExprSelectiv
Agree, e-e test seems like an overkill.


-- 
To view, visit http://gerrit.cloudera.org:8080/7168
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I69e6217257b5618cb63e13b32ba3347fa0483b63
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Vincent Tran 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Vincent Tran 
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5494: Fixes the selectivity of NOT IN predicates

[Alex Behm (Code Review)]

Alex Behm has posted comments on this change.

Change subject: IMPALA-5494: Fixes the selectivity of NOT IN predicates
..


Patch Set 3:

(1 comment)

Thanks for finding and fixing this.

http://gerrit.cloudera.org:8080/#/c/7168/3/tests/metadata/test_explain.py
File tests/metadata/test_explain.py:

Line 105: result = self.execute_query("explain select * from %s.%s where id 
"
Instead of this end-to-end test, please add a new FE unit test 
ExprSelectivityTest similar to ExprNdvTest. Relying on a full explain seems 
like overkill, and checking the scan cardinality is really an indirect test.


-- 
To view, visit http://gerrit.cloudera.org:8080/7168
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I69e6217257b5618cb63e13b32ba3347fa0483b63
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Vincent Tran 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Vincent Tran 
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5494: Fixes the selectivity of NOT IN predicates

[Vincent Tran (Code Review)]

Vincent Tran has posted comments on this change.

Change subject: IMPALA-5494: Fixes the selectivity of NOT IN predicates
..


Patch Set 2:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/7168/2//COMMIT_MSG
Commit Message:

PS2, Line 7: NOT IN predicate shares the same selectivity as
   : IN predicate.
> Change to what this commit fixes may be? Something like Fix the selectivity
Done


PS2, Line 18: predicae
> typo
Done


http://gerrit.cloudera.org:8080/#/c/7168/2/tests/metadata/test_explain.py
File tests/metadata/test_explain.py:

Line 104: # IN predicate, cardinality should be 7300*(num of children/num 
of distinct values)
> Not totally sure if there is a better place to add these tests, Alex any id
Maybe I should just specify the expected value as a constant?


Line 109: # NOT IN predicate, cardinality should be 7300*(1-(num of 
children/num of distinct values))
> nit: long line.
Done


-- 
To view, visit http://gerrit.cloudera.org:8080/7168
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I69e6217257b5618cb63e13b32ba3347fa0483b63
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Vincent Tran 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Vincent Tran 
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5494: Fixes the selectivity of NOT IN predicates

[Vincent Tran (Code Review)]

Vincent Tran has uploaded a new patch set (#3).

Change subject: IMPALA-5494: Fixes the selectivity of NOT IN predicates
..

IMPALA-5494: Fixes the selectivity of NOT IN predicates

This change modifies the logic of NOT IN predicate so that
the planner can calculate the correct node cardinality. Prior
to this change, both IN and NOT IN predicates shared the
same selectivity, which resulted in the same cardinality
during planning.

The selectivity is calculated by the following heuristic:

selectivity = 1 - (num of predicate children /
num of distinct values)

Change-Id: I69e6217257b5618cb63e13b32ba3347fa0483b63
---
M fe/src/main/java/org/apache/impala/analysis/InPredicate.java
M tests/metadata/test_explain.py
2 files changed, 18 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/7168/3
-- 
To view, visit http://gerrit.cloudera.org:8080/7168
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I69e6217257b5618cb63e13b32ba3347fa0483b63
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Vincent Tran 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Vincent Tran