[jira] [Created] (KYLIN-2427) Auto adjust join order to make query executable
Kaige Liu created KYLIN-2427: - Summary: Auto adjust join order to make query executable Key: KYLIN-2427 URL: https://issues.apache.org/jira/browse/KYLIN-2427 Project: Kylin Issue Type: Bug Reporter: Kaige Liu KYLIN-2406 reports an issue: The order of joins will affect the result of query. For example, below query leads to "No model found" Below query triggers NPE {code} with tmp3 as ( select l_partkey, 0.5 * sum(l_quantity) as sum_quantity, l_suppkey from v_lineitem inner join supplier on l_suppkey = s_suppkey inner join nation on s_nationkey = n_nationkey inner join part on l_partkey = p_partkey where l_shipdate >= '1992-01-01' and l_shipdate <= '1995-01-01' and n_name = 'CANADA' and p_name like 'forest%' group by l_partkey, l_suppkey ) select s_name, s_address from v_partsupp inner join tmp3 on ps_partkey = l_partkey and ps_suppkey = l_suppkey inner join supplier on ps_suppkey = s_suppkey where ps_availqty > sum_quantity group by s_name, s_address order by s_name {code} While below query is OK. Only difference being the order of "inner join tmp3" and "inner join supplier" {code} with tmp3 as ( select l_partkey, 0.5 * sum(l_quantity) as sum_quantity, l_suppkey from v_lineitem inner join supplier on l_suppkey = s_suppkey inner join nation on s_nationkey = n_nationkey inner join part on l_partkey = p_partkey where l_shipdate >= '1992-01-01' and l_shipdate <= '1995-01-01' and n_name = 'CANADA' and p_name like 'forest%' group by l_partkey, l_suppkey ) select s_name, s_address from v_partsupp inner join supplier on ps_suppkey = s_suppkey inner join tmp3 on ps_partkey = l_partkey and ps_suppkey = l_suppkey where ps_availqty > sum_quantity group by s_name, s_address order by s_name {code} But below query is OK. {code} with tmp3 as ( select l_partkey, 0.5 * sum(l_quantity) as sum_quantity, l_suppkey from v_lineitem inner join supplier on l_suppkey = s_suppkey inner join nation on s_nationkey = n_nationkey inner join part on l_partkey = p_partkey where l_shipdate >= '1992-01-01' and l_shipdate <= '1995-01-01' and n_name = 'CANADA' and p_name like 'forest%' group by l_partkey, l_suppkey ) select s_name, s_address from v_partsupp inner join supplier on ps_suppkey = s_suppkey inner join tmp3 on ps_partkey = l_partkey and ps_suppkey = l_suppkey where ps_availqty > sum_quantity group by s_name, s_address order by s_name {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (KYLIN-2427) Auto adjust join order to make query executable
[ https://issues.apache.org/jira/browse/KYLIN-2427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaige Liu updated KYLIN-2427: - Description: KYLIN-2406 reports an issue: The order of joins will affect the result of query. For example, below query leads to "No model found" Below query triggers NPE {code} with tmp3 as ( select l_partkey, 0.5 * sum(l_quantity) as sum_quantity, l_suppkey from v_lineitem inner join supplier on l_suppkey = s_suppkey inner join nation on s_nationkey = n_nationkey inner join part on l_partkey = p_partkey where l_shipdate >= '1992-01-01' and l_shipdate <= '1995-01-01' and n_name = 'CANADA' and p_name like 'forest%' group by l_partkey, l_suppkey ) select s_name, s_address from v_partsupp inner join tmp3 on ps_partkey = l_partkey and ps_suppkey = l_suppkey inner join supplier on ps_suppkey = s_suppkey where ps_availqty > sum_quantity group by s_name, s_address order by s_name {code} While below query is OK. Only difference being the order of "inner join tmp3" and "inner join supplier" {code} with tmp3 as ( select l_partkey, 0.5 * sum(l_quantity) as sum_quantity, l_suppkey from v_lineitem inner join supplier on l_suppkey = s_suppkey inner join nation on s_nationkey = n_nationkey inner join part on l_partkey = p_partkey where l_shipdate >= '1992-01-01' and l_shipdate <= '1995-01-01' and n_name = 'CANADA' and p_name like 'forest%' group by l_partkey, l_suppkey ) select s_name, s_address from v_partsupp inner join supplier on ps_suppkey = s_suppkey inner join tmp3 on ps_partkey = l_partkey and ps_suppkey = l_suppkey where ps_availqty > sum_quantity group by s_name, s_address order by s_name {code} was: KYLIN-2406 reports an issue: The order of joins will affect the result of query. For example, below query leads to "No model found" Below query triggers NPE {code} with tmp3 as ( select l_partkey, 0.5 * sum(l_quantity) as sum_quantity, l_suppkey from v_lineitem inner join supplier on l_suppkey = s_suppkey inner join nation on s_nationkey = n_nationkey inner join part on l_partkey = p_partkey where l_shipdate >= '1992-01-01' and l_shipdate <= '1995-01-01' and n_name = 'CANADA' and p_name like 'forest%' group by l_partkey, l_suppkey ) select s_name, s_address from v_partsupp inner join tmp3 on ps_partkey = l_partkey and ps_suppkey = l_suppkey inner join supplier on ps_suppkey = s_suppkey where ps_availqty > sum_quantity group by s_name, s_address order by s_name {code} While below query is OK. Only difference being the order of "inner join tmp3" and "inner join supplier" {code} with tmp3 as ( select l_partkey, 0.5 * sum(l_quantity) as sum_quantity, l_suppkey from v_lineitem inner join supplier on l_suppkey = s_suppkey inner join nation on s_nationkey = n_nationkey inner join part on l_partkey = p_partkey where l_shipdate >= '1992-01-01' and l_shipdate <= '1995-01-01' and n_name = 'CANADA' and p_name like 'forest%' group by l_partkey, l_suppkey ) select s_name, s_address from v_partsupp inner join supplier on ps_suppkey = s_suppkey inner join tmp3 on ps_partkey = l_partkey and ps_suppkey = l_suppkey where ps_availqty > sum_quantity group by s_name, s_address order by s_name {code} But below query is OK. {code} with tmp3 as ( select l_partkey, 0.5 * sum(l_quantity) as sum_quantity, l_suppkey from v_lineitem inner join supplier on l_suppkey = s_suppkey inner join nation on s_nationkey = n_nationkey inner join part on l_partkey = p_partkey where l_shipdate >= '1992-01-01' and l_shipdate <= '1995-01-01' and n_name = 'CANADA' and p_name like 'forest%' group by l_partkey, l_suppkey ) select s_name, s_address from v_partsupp inner join supplier on ps_suppkey = s_suppkey inner join tmp3 on ps_partkey = l_partkey and ps_suppkey = l_suppkey where ps_availqty > sum_quantity group by s_name, s_address order by s_name {code} > Auto adjust join order to make query executable > --- > > Key: KYLIN-2427 > URL: https://issues.apache.org/jira/browse/KYLIN-2427 > Project: Kylin > Issue Type: Bug >Reporter: Kaige Liu > > KYLIN-2406 reports an issue: The order of joins will affect the result of > query. For example, below query leads to "No model found" > Below query triggers NPE > {code} > with tmp3 as ( > select l_partkey, 0.5 * sum(l_quantity) as sum_quantity, l_suppkey > from v_lineitem > inner join supplier on l_suppkey = s_suppkey > inner join nation on s_nationkey = n_nationkey > inner join part on l_partkey = p_partkey > where l_shipdate >= '1992-01-01' an
[jira] [Assigned] (KYLIN-2427) Auto adjust join order to make query executable
[ https://issues.apache.org/jira/browse/KYLIN-2427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaige Liu reassigned KYLIN-2427: - Assignee: Kaige Liu > Auto adjust join order to make query executable > --- > > Key: KYLIN-2427 > URL: https://issues.apache.org/jira/browse/KYLIN-2427 > Project: Kylin > Issue Type: Bug >Reporter: Kaige Liu >Assignee: Kaige Liu > > KYLIN-2406 reports an issue: The order of joins will affect the result of > query. For example, below query leads to "No model found" > Below query triggers NPE > {code} > with tmp3 as ( > select l_partkey, 0.5 * sum(l_quantity) as sum_quantity, l_suppkey > from v_lineitem > inner join supplier on l_suppkey = s_suppkey > inner join nation on s_nationkey = n_nationkey > inner join part on l_partkey = p_partkey > where l_shipdate >= '1992-01-01' and l_shipdate <= '1995-01-01' > and n_name = 'CANADA' > and p_name like 'forest%' > group by l_partkey, l_suppkey > ) > select > s_name, > s_address > from > v_partsupp > inner join tmp3 on ps_partkey = l_partkey and ps_suppkey = l_suppkey > inner join supplier on ps_suppkey = s_suppkey > where > ps_availqty > sum_quantity > group by > s_name, s_address > order by > s_name > {code} > While below query is OK. Only difference being the order of "inner join tmp3" > and "inner join supplier" > {code} > with tmp3 as ( > select l_partkey, 0.5 * sum(l_quantity) as sum_quantity, l_suppkey > from v_lineitem > inner join supplier on l_suppkey = s_suppkey > inner join nation on s_nationkey = n_nationkey > inner join part on l_partkey = p_partkey > where l_shipdate >= '1992-01-01' and l_shipdate <= '1995-01-01' > and n_name = 'CANADA' > and p_name like 'forest%' > group by l_partkey, l_suppkey > ) > select > s_name, > s_address > from > v_partsupp > inner join supplier on ps_suppkey = s_suppkey > inner join tmp3 on ps_partkey = l_partkey and ps_suppkey = l_suppkey > where > ps_availqty > sum_quantity > group by > s_name, s_address > order by > s_name > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (KYLIN-2406) TPC-H query 20, can triggers NPE
[ https://issues.apache.org/jira/browse/KYLIN-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaige Liu updated KYLIN-2406: - Attachment: KYLIN-2406-fix-NPE.patch For now this patch only prevent the NPE and give an error hint. A better solution will be given in KYLIN-2427 later. > TPC-H query 20, can triggers NPE > > > Key: KYLIN-2406 > URL: https://issues.apache.org/jira/browse/KYLIN-2406 > Project: Kylin > Issue Type: Bug >Reporter: liyang >Assignee: Kaige Liu > Attachments: KYLIN-2406-fix-NPE.patch > > > Below query triggers NPE > {code} > with tmp3 as ( > select l_partkey, 0.5 * sum(l_quantity) as sum_quantity, l_suppkey > from v_lineitem > inner join supplier on l_suppkey = s_suppkey > inner join nation on s_nationkey = n_nationkey > inner join part on l_partkey = p_partkey > where l_shipdate >= '1992-01-01' and l_shipdate <= '1995-01-01' > and n_name = 'CANADA' > and p_name like 'forest%' > group by l_partkey, l_suppkey > ) > select > s_name, > s_address > from > v_partsupp > inner join tmp3 on ps_partkey = l_partkey and ps_suppkey = l_suppkey > inner join supplier on ps_suppkey = s_suppkey > where > ps_availqty > sum_quantity > group by > s_name, s_address > order by > s_name > {code} > While below query is OK. Only difference being the order of "inner join tmp3" > and "inner join supplier" > {code} > with tmp3 as ( > select l_partkey, 0.5 * sum(l_quantity) as sum_quantity, l_suppkey > from v_lineitem > inner join supplier on l_suppkey = s_suppkey > inner join nation on s_nationkey = n_nationkey > inner join part on l_partkey = p_partkey > where l_shipdate >= '1992-01-01' and l_shipdate <= '1995-01-01' > and n_name = 'CANADA' > and p_name like 'forest%' > group by l_partkey, l_suppkey > ) > select > s_name, > s_address > from > v_partsupp > inner join supplier on ps_suppkey = s_suppkey > inner join tmp3 on ps_partkey = l_partkey and ps_suppkey = l_suppkey > where > ps_availqty > sum_quantity > group by > s_name, s_address > order by > s_name > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (KYLIN-2406) TPC-H query 20, can triggers NPE
[ https://issues.apache.org/jira/browse/KYLIN-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaige Liu updated KYLIN-2406: - Attachment: KYLIN-2406-fix-NPE.patch > TPC-H query 20, can triggers NPE > > > Key: KYLIN-2406 > URL: https://issues.apache.org/jira/browse/KYLIN-2406 > Project: Kylin > Issue Type: Bug >Reporter: liyang >Assignee: Kaige Liu > Attachments: KYLIN-2406-fix-NPE.patch > > > Below query triggers NPE > {code} > with tmp3 as ( > select l_partkey, 0.5 * sum(l_quantity) as sum_quantity, l_suppkey > from v_lineitem > inner join supplier on l_suppkey = s_suppkey > inner join nation on s_nationkey = n_nationkey > inner join part on l_partkey = p_partkey > where l_shipdate >= '1992-01-01' and l_shipdate <= '1995-01-01' > and n_name = 'CANADA' > and p_name like 'forest%' > group by l_partkey, l_suppkey > ) > select > s_name, > s_address > from > v_partsupp > inner join tmp3 on ps_partkey = l_partkey and ps_suppkey = l_suppkey > inner join supplier on ps_suppkey = s_suppkey > where > ps_availqty > sum_quantity > group by > s_name, s_address > order by > s_name > {code} > While below query is OK. Only difference being the order of "inner join tmp3" > and "inner join supplier" > {code} > with tmp3 as ( > select l_partkey, 0.5 * sum(l_quantity) as sum_quantity, l_suppkey > from v_lineitem > inner join supplier on l_suppkey = s_suppkey > inner join nation on s_nationkey = n_nationkey > inner join part on l_partkey = p_partkey > where l_shipdate >= '1992-01-01' and l_shipdate <= '1995-01-01' > and n_name = 'CANADA' > and p_name like 'forest%' > group by l_partkey, l_suppkey > ) > select > s_name, > s_address > from > v_partsupp > inner join supplier on ps_suppkey = s_suppkey > inner join tmp3 on ps_partkey = l_partkey and ps_suppkey = l_suppkey > where > ps_availqty > sum_quantity > group by > s_name, s_address > order by > s_name > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (KYLIN-2406) TPC-H query 20, can triggers NPE
[ https://issues.apache.org/jira/browse/KYLIN-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaige Liu updated KYLIN-2406: - Attachment: (was: KYLIN-2406-fix-NPE.patch) > TPC-H query 20, can triggers NPE > > > Key: KYLIN-2406 > URL: https://issues.apache.org/jira/browse/KYLIN-2406 > Project: Kylin > Issue Type: Bug >Reporter: liyang >Assignee: Kaige Liu > Attachments: KYLIN-2406-fix-NPE.patch > > > Below query triggers NPE > {code} > with tmp3 as ( > select l_partkey, 0.5 * sum(l_quantity) as sum_quantity, l_suppkey > from v_lineitem > inner join supplier on l_suppkey = s_suppkey > inner join nation on s_nationkey = n_nationkey > inner join part on l_partkey = p_partkey > where l_shipdate >= '1992-01-01' and l_shipdate <= '1995-01-01' > and n_name = 'CANADA' > and p_name like 'forest%' > group by l_partkey, l_suppkey > ) > select > s_name, > s_address > from > v_partsupp > inner join tmp3 on ps_partkey = l_partkey and ps_suppkey = l_suppkey > inner join supplier on ps_suppkey = s_suppkey > where > ps_availqty > sum_quantity > group by > s_name, s_address > order by > s_name > {code} > While below query is OK. Only difference being the order of "inner join tmp3" > and "inner join supplier" > {code} > with tmp3 as ( > select l_partkey, 0.5 * sum(l_quantity) as sum_quantity, l_suppkey > from v_lineitem > inner join supplier on l_suppkey = s_suppkey > inner join nation on s_nationkey = n_nationkey > inner join part on l_partkey = p_partkey > where l_shipdate >= '1992-01-01' and l_shipdate <= '1995-01-01' > and n_name = 'CANADA' > and p_name like 'forest%' > group by l_partkey, l_suppkey > ) > select > s_name, > s_address > from > v_partsupp > inner join supplier on ps_suppkey = s_suppkey > inner join tmp3 on ps_partkey = l_partkey and ps_suppkey = l_suppkey > where > ps_availqty > sum_quantity > group by > s_name, s_address > order by > s_name > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (KYLIN-2406) TPC-H query 20, prevent NPE and give error hint
[ https://issues.apache.org/jira/browse/KYLIN-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li resolved KYLIN-2406. Resolution: Fixed Fix Version/s: v2.0.0 Merged to master branch. Thanks Kaige! > TPC-H query 20, prevent NPE and give error hint > --- > > Key: KYLIN-2406 > URL: https://issues.apache.org/jira/browse/KYLIN-2406 > Project: Kylin > Issue Type: Bug >Reporter: liyang >Assignee: Kaige Liu > Fix For: v2.0.0 > > Attachments: KYLIN-2406-fix-NPE.patch > > > Below query triggers NPE > {code} > with tmp3 as ( > select l_partkey, 0.5 * sum(l_quantity) as sum_quantity, l_suppkey > from v_lineitem > inner join supplier on l_suppkey = s_suppkey > inner join nation on s_nationkey = n_nationkey > inner join part on l_partkey = p_partkey > where l_shipdate >= '1992-01-01' and l_shipdate <= '1995-01-01' > and n_name = 'CANADA' > and p_name like 'forest%' > group by l_partkey, l_suppkey > ) > select > s_name, > s_address > from > v_partsupp > inner join tmp3 on ps_partkey = l_partkey and ps_suppkey = l_suppkey > inner join supplier on ps_suppkey = s_suppkey > where > ps_availqty > sum_quantity > group by > s_name, s_address > order by > s_name > {code} > While below query is OK. Only difference being the order of "inner join tmp3" > and "inner join supplier" > {code} > with tmp3 as ( > select l_partkey, 0.5 * sum(l_quantity) as sum_quantity, l_suppkey > from v_lineitem > inner join supplier on l_suppkey = s_suppkey > inner join nation on s_nationkey = n_nationkey > inner join part on l_partkey = p_partkey > where l_shipdate >= '1992-01-01' and l_shipdate <= '1995-01-01' > and n_name = 'CANADA' > and p_name like 'forest%' > group by l_partkey, l_suppkey > ) > select > s_name, > s_address > from > v_partsupp > inner join supplier on ps_suppkey = s_suppkey > inner join tmp3 on ps_partkey = l_partkey and ps_suppkey = l_suppkey > where > ps_availqty > sum_quantity > group by > s_name, s_address > order by > s_name > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (KYLIN-2406) TPC-H query 20, prevent NPE and give error hint
[ https://issues.apache.org/jira/browse/KYLIN-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li updated KYLIN-2406: --- Request participants: (was: ) Summary: TPC-H query 20, prevent NPE and give error hint (was: TPC-H query 20, can triggers NPE) > TPC-H query 20, prevent NPE and give error hint > --- > > Key: KYLIN-2406 > URL: https://issues.apache.org/jira/browse/KYLIN-2406 > Project: Kylin > Issue Type: Bug >Reporter: liyang >Assignee: Kaige Liu > Fix For: v2.0.0 > > Attachments: KYLIN-2406-fix-NPE.patch > > > Below query triggers NPE > {code} > with tmp3 as ( > select l_partkey, 0.5 * sum(l_quantity) as sum_quantity, l_suppkey > from v_lineitem > inner join supplier on l_suppkey = s_suppkey > inner join nation on s_nationkey = n_nationkey > inner join part on l_partkey = p_partkey > where l_shipdate >= '1992-01-01' and l_shipdate <= '1995-01-01' > and n_name = 'CANADA' > and p_name like 'forest%' > group by l_partkey, l_suppkey > ) > select > s_name, > s_address > from > v_partsupp > inner join tmp3 on ps_partkey = l_partkey and ps_suppkey = l_suppkey > inner join supplier on ps_suppkey = s_suppkey > where > ps_availqty > sum_quantity > group by > s_name, s_address > order by > s_name > {code} > While below query is OK. Only difference being the order of "inner join tmp3" > and "inner join supplier" > {code} > with tmp3 as ( > select l_partkey, 0.5 * sum(l_quantity) as sum_quantity, l_suppkey > from v_lineitem > inner join supplier on l_suppkey = s_suppkey > inner join nation on s_nationkey = n_nationkey > inner join part on l_partkey = p_partkey > where l_shipdate >= '1992-01-01' and l_shipdate <= '1995-01-01' > and n_name = 'CANADA' > and p_name like 'forest%' > group by l_partkey, l_suppkey > ) > select > s_name, > s_address > from > v_partsupp > inner join supplier on ps_suppkey = s_suppkey > inner join tmp3 on ps_partkey = l_partkey and ps_suppkey = l_suppkey > where > ps_availqty > sum_quantity > group by > s_name, s_address > order by > s_name > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KYLIN-2424) Optimize the integration test's performance
[ https://issues.apache.org/jira/browse/KYLIN-2424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15853970#comment-15853970 ] hongbin ma commented on KYLIN-2424: --- [~Shaofengshi] great work! I can close KYLIN-2015 safely now [~yimingliu] looks like it's abbreviation for "true". If so it could be confusing, why not just use "true"? > Optimize the integration test's performance > --- > > Key: KYLIN-2424 > URL: https://issues.apache.org/jira/browse/KYLIN-2424 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI > Fix For: v2.0.0 > > > Kylin's integration test is slow, especially the ITCombinationTest. Most of > time are spent on H2 to execute the test queries. In a latest integration > test, this test case take 90 minutes to finish. > By checking H2's document, I think the main problem is the absence of index > on the tables, while index is very important for a relational database's > query performance. So when Kylin create the tables in H2, shoud create index > on the columns that will be used in the queries, like the pk/fk, the > filtering columns etc. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KYLIN-2015) replace h2 with alternatives like sqllite or mysql
[ https://issues.apache.org/jira/browse/KYLIN-2015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15853972#comment-15853972 ] hongbin ma commented on KYLIN-2015: --- The performance issue of H2 is solved in KYLIN-2424 without replacing H2 > replace h2 with alternatives like sqllite or mysql > -- > > Key: KYLIN-2015 > URL: https://issues.apache.org/jira/browse/KYLIN-2015 > Project: Kylin > Issue Type: Improvement >Reporter: hongbin ma >Assignee: hongbin ma > > in IT we compare kylin's result with H2's results to ensure query correctness. > however h2 only supports part of the SQL syntax. For example, it cannot > support functions like timestampadd, or (DATE'2013-01-02' + interval '3' > day). What's more, subqueries are observed to be very slow on H2. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (KYLIN-2424) Optimize the integration test's performance
[ https://issues.apache.org/jira/browse/KYLIN-2424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15853970#comment-15853970 ] hongbin ma edited comment on KYLIN-2424 at 2/6/17 1:20 PM: --- [~Shaofengshi] great work! [~yimingliu] looks like it's abbreviation for "true". If so it could be confusing, why not just use "true"? was (Author: mahongbin): [~Shaofengshi] great work! I can close KYLIN-2015 safely now [~yimingliu] looks like it's abbreviation for "true". If so it could be confusing, why not just use "true"? > Optimize the integration test's performance > --- > > Key: KYLIN-2424 > URL: https://issues.apache.org/jira/browse/KYLIN-2424 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI > Fix For: v2.0.0 > > > Kylin's integration test is slow, especially the ITCombinationTest. Most of > time are spent on H2 to execute the test queries. In a latest integration > test, this test case take 90 minutes to finish. > By checking H2's document, I think the main problem is the absence of index > on the tables, while index is very important for a relational database's > query performance. So when Kylin create the tables in H2, shoud create index > on the columns that will be used in the queries, like the pk/fk, the > filtering columns etc. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (KYLIN-2407) TPC-H query 20, why this query returns no result?
[ https://issues.apache.org/jira/browse/KYLIN-2407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaige Liu reassigned KYLIN-2407: - Assignee: Kaige Liu > TPC-H query 20, why this query returns no result? > - > > Key: KYLIN-2407 > URL: https://issues.apache.org/jira/browse/KYLIN-2407 > Project: Kylin > Issue Type: Bug >Reporter: liyang >Assignee: Kaige Liu > > Below query returns no result. > {code} > with tmp3 as ( > select l_partkey, 0.5 * sum(l_quantity) as sum_quantity, l_suppkey > from v_lineitem > inner join supplier on l_suppkey = s_suppkey > inner join nation on s_nationkey = n_nationkey > inner join part on l_partkey = p_partkey > where l_shipdate >= '1992-01-01' and l_shipdate <= '1995-01-01' > and n_name = 'CANADA' > and p_name like 'forest%' > group by l_partkey, l_suppkey > ), > tmp5 as ( > select > ps_suppkey > from > v_partsupp inner join tmp3 on ps_partkey = l_partkey and ps_suppkey = > l_suppkey > where > ps_availqty > sum_quantity > ) > select > s_name, > s_address > from > supplier > where > s_suppkey IN (select ps_suppkey from tmp5) > order by s_name > {code} > While another similar query returns correct result. > {code} > with tmp3 as ( > select l_partkey, 0.5 * sum(l_quantity) as sum_quantity, l_suppkey > from v_lineitem > inner join supplier on l_suppkey = s_suppkey > inner join nation on s_nationkey = n_nationkey > inner join part on l_partkey = p_partkey > where l_shipdate >= '1992-01-01' and l_shipdate <= '1995-01-01' > and n_name = 'CANADA' > and p_name like 'forest%' > group by l_partkey, l_suppkey > ) > select > s_name, > s_address > from > v_partsupp > inner join supplier on ps_suppkey = s_suppkey > inner join tmp3 on ps_partkey = l_partkey and ps_suppkey = l_suppkey > where > ps_availqty > sum_quantity > group by > s_name, s_address > order by > s_name > {code} > Maybe something wrong with the "where ... IN ..." clause? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (KYLIN-2428) Cleanup unnecessary shaded libraries for job/coprocessor/jdbc/server
Billy Liu created KYLIN-2428: Summary: Cleanup unnecessary shaded libraries for job/coprocessor/jdbc/server Key: KYLIN-2428 URL: https://issues.apache.org/jira/browse/KYLIN-2428 Project: Kylin Issue Type: Improvement Components: General Affects Versions: v1.6.0 Reporter: Billy Liu Assignee: Billy Liu Kylin releases three libraries: kylin-coprocessor, kylin-jdbc, kylin-job and one web application: server. Currently, all libraries have shaded some used third party libraries into the package. For example, guava, curator, commons, kyro in kylin-job. The duplicate libraries in runtime classpath may have potential class loading conflicts and waste computing resource. We should leverage the hadoop provided libraries at runtime instead of the shaded one. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (KYLIN-2428) Cleanup unnecessary shaded libraries for job/coprocessor/jdbc/server
[ https://issues.apache.org/jira/browse/KYLIN-2428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu updated KYLIN-2428: - Attachment: KYLIN-2428.patch > Cleanup unnecessary shaded libraries for job/coprocessor/jdbc/server > > > Key: KYLIN-2428 > URL: https://issues.apache.org/jira/browse/KYLIN-2428 > Project: Kylin > Issue Type: Improvement > Components: General >Affects Versions: v1.6.0 >Reporter: Billy Liu >Assignee: Billy Liu > Attachments: KYLIN-2428.patch > > > Kylin releases three libraries: kylin-coprocessor, kylin-jdbc, kylin-job and > one web application: server. > Currently, all libraries have shaded some used third party libraries into the > package. For example, guava, curator, commons, kyro in kylin-job. The > duplicate libraries in runtime classpath may have potential class loading > conflicts and waste computing resource. We should leverage the hadoop > provided libraries at runtime instead of the shaded one. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KYLIN-2428) Cleanup unnecessary shaded libraries for job/coprocessor/jdbc/server
[ https://issues.apache.org/jira/browse/KYLIN-2428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15854147#comment-15854147 ] Billy Liu commented on KYLIN-2428: -- Compared with kylin 1.6: kylin-coprocessor is reduced from 2.4M to 1.9M, by removing extendedset, commons-lang3, but also introduced tduning.math kylin-jdbc increased from 7.6M to 8.5M since Avatica 1.9 upgrade. kylin-job is reduced from 9.5M to 5.8M, by removing esotericsoftware, guava, jsch, extendedset, jsr305, commons-cli, commons-io, commons-lang, curator and objenesis. Web application has reduced from 94 libraries to 80. > Cleanup unnecessary shaded libraries for job/coprocessor/jdbc/server > > > Key: KYLIN-2428 > URL: https://issues.apache.org/jira/browse/KYLIN-2428 > Project: Kylin > Issue Type: Improvement > Components: General >Affects Versions: v1.6.0 >Reporter: Billy Liu >Assignee: Billy Liu > Attachments: KYLIN-2428.patch > > > Kylin releases three libraries: kylin-coprocessor, kylin-jdbc, kylin-job and > one web application: server. > Currently, all libraries have shaded some used third party libraries into the > package. For example, guava, curator, commons, kyro in kylin-job. The > duplicate libraries in runtime classpath may have potential class loading > conflicts and waste computing resource. We should leverage the hadoop > provided libraries at runtime instead of the shaded one. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KYLIN-2428) Cleanup unnecessary shaded libraries for job/coprocessor/jdbc/server
[ https://issues.apache.org/jira/browse/KYLIN-2428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15854211#comment-15854211 ] Billy Liu commented on KYLIN-2428: -- By refine the JDBC shade setting, the package has reduced from 7.6M to 5.4M, no duplicated shaded any more. > Cleanup unnecessary shaded libraries for job/coprocessor/jdbc/server > > > Key: KYLIN-2428 > URL: https://issues.apache.org/jira/browse/KYLIN-2428 > Project: Kylin > Issue Type: Improvement > Components: General >Affects Versions: v1.6.0 >Reporter: Billy Liu >Assignee: Billy Liu > Attachments: KYLIN-2428.patch > > > Kylin releases three libraries: kylin-coprocessor, kylin-jdbc, kylin-job and > one web application: server. > Currently, all libraries have shaded some used third party libraries into the > package. For example, guava, curator, commons, kyro in kylin-job. The > duplicate libraries in runtime classpath may have potential class loading > conflicts and waste computing resource. We should leverage the hadoop > provided libraries at runtime instead of the shaded one. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (KYLIN-2429) Variable initialized should be declared volatile in SparkCubingByLayer#execute()
Ted Yu created KYLIN-2429: - Summary: Variable initialized should be declared volatile in SparkCubingByLayer#execute() Key: KYLIN-2429 URL: https://issues.apache.org/jira/browse/KYLIN-2429 Project: Kylin Issue Type: Bug Reporter: Ted Yu {code} final JavaPairRDD encodedBaseRDD = intermediateTable.javaRDD().mapToPair(new PairFunction() { transient boolean initialized = false; ... public Tuple2 call(Row row) throws Exception { if (initialized == false) { synchronized (SparkCubingByLayer.class) { if (initialized == false) { {code} For double checked locking to work, initialized needs to be volatile. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (KYLIN-2430) Unnecessary exception catching in BulkLoadJob
kangkaisen created KYLIN-2430: - Summary: Unnecessary exception catching in BulkLoadJob Key: KYLIN-2430 URL: https://issues.apache.org/jira/browse/KYLIN-2430 Project: Kylin Issue Type: Bug Components: Storage - HBase Affects Versions: v1.6.0 Reporter: kangkaisen Assignee: kangkaisen FsShell.run has caught all exceptions, So we should get exitCode instead of catching exception. Currently code potentially result in infinite loop in {{LoadIncrementalHFiles}} if user use HBase 0.98.13 and don't set {{hbase.bulkload.retries.number}}. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (KYLIN-2329) Between 0.06 - 0.01 and 0.06 + 0.01, returns incorrect result
[ https://issues.apache.org/jira/browse/KYLIN-2329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyang resolved KYLIN-2329. --- Resolution: Fixed Fix Version/s: v2.0.0 > Between 0.06 - 0.01 and 0.06 + 0.01, returns incorrect result > - > > Key: KYLIN-2329 > URL: https://issues.apache.org/jira/browse/KYLIN-2329 > Project: Kylin > Issue Type: Bug >Reporter: liyang >Assignee: Kaige Liu > Fix For: v2.0.0 > > > A TPC-H query returns incorrect result: > {code} > select > sum(l_saleprice) as revenue > from > v_lineitem > where > l_shipdate >= '1993-01-01' > and l_shipdate < '1994-01-01' > and l_discount between 0.06 - 0.01 and 0.06 + 0.01 > and l_quantity < 25; > {code} > The result becomes correct if change condition to below > {code} > and l_discount between 0.05 and 0.07 > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KYLIN-2329) Between 0.06 - 0.01 and 0.06 + 0.01, returns incorrect result
[ https://issues.apache.org/jira/browse/KYLIN-2329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15855458#comment-15855458 ] liyang commented on KYLIN-2329: --- Cannot reproduce on latest master. Perhaps, the recent upgrade of calcite solved this. > Between 0.06 - 0.01 and 0.06 + 0.01, returns incorrect result > - > > Key: KYLIN-2329 > URL: https://issues.apache.org/jira/browse/KYLIN-2329 > Project: Kylin > Issue Type: Bug >Reporter: liyang >Assignee: Kaige Liu > Fix For: v2.0.0 > > > A TPC-H query returns incorrect result: > {code} > select > sum(l_saleprice) as revenue > from > v_lineitem > where > l_shipdate >= '1993-01-01' > and l_shipdate < '1994-01-01' > and l_discount between 0.06 - 0.01 and 0.06 + 0.01 > and l_quantity < 25; > {code} > The result becomes correct if change condition to below > {code} > and l_discount between 0.05 and 0.07 > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (KYLIN-2431) StorageCleanupJob will remove immediate tables created by other kylin instances
Dong Li created KYLIN-2431: -- Summary: StorageCleanupJob will remove immediate tables created by other kylin instances Key: KYLIN-2431 URL: https://issues.apache.org/jira/browse/KYLIN-2431 Project: Kylin Issue Type: Improvement Reporter: Dong Li Assignee: Dong Li Priority: Minor If QA and PROD instances are in same hive database, running StorageCleanupJob on QA will remove immediate tables created by PROD, which might fail Kylin jobs of PROD. A solution is to add metastore name to hive table prefix, then filter table names with metastore name during cleanup job. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KYLIN-2428) Cleanup unnecessary shaded libraries for job/coprocessor/jdbc/server
[ https://issues.apache.org/jira/browse/KYLIN-2428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15855491#comment-15855491 ] Billy Liu commented on KYLIN-2428: -- [INFO] Apache Kylin ... SUCCESS [ 3.045 s] [INFO] Apache Kylin - Calcite Overrides ... SUCCESS [ 2.161 s] [INFO] Apache Kylin - Core Common . SUCCESS [ 5.515 s] [INFO] Apache Kylin - Core Metadata ... SUCCESS [ 25.081 s] [INFO] Apache Kylin - Core Dictionary . SUCCESS [01:14 min] [INFO] Apache Kylin - Core Cube ... SUCCESS [01:44 min] [INFO] Apache Kylin - Core Job SUCCESS [01:58 min] [INFO] Apache Kylin - Core Storage SUCCESS [ 34.999 s] [INFO] Apache Kylin - MapReduce Engine SUCCESS [ 20.623 s] [INFO] Apache Kylin - HBase Storage ... SUCCESS [ 12.564 s] [INFO] Apache Kylin - Spark Engine SUCCESS [ 13.664 s] [INFO] Apache Kylin - Hive Source . SUCCESS [ 4.314 s] [INFO] Apache Kylin - Kafka Source SUCCESS [ 3.250 s] [INFO] Apache Kylin - Query ... SUCCESS [ 2.485 s] [INFO] Apache Kylin - Tool SUCCESS [ 4.199 s] [INFO] Apache Kylin - REST Server Base SUCCESS [ 6.802 s] [INFO] Apache Kylin - REST Server . SUCCESS [ 37.196 s] [INFO] Apache Kylin - JDBC Driver . SUCCESS [ 5.646 s] [INFO] Apache Kylin - Assembly SUCCESS [ 5.158 s] [INFO] Apache Kylin - Integration Test SUCCESS [ 01:08 h] [INFO] Apache Kylin - Tomcat Extension SUCCESS [ 2.810 s] [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 01:16 h [INFO] Finished at: 2017-02-07T15:27:23+00:00 [INFO] Final Memory: 75M/935M > Cleanup unnecessary shaded libraries for job/coprocessor/jdbc/server > > > Key: KYLIN-2428 > URL: https://issues.apache.org/jira/browse/KYLIN-2428 > Project: Kylin > Issue Type: Improvement > Components: General >Affects Versions: v1.6.0 >Reporter: Billy Liu >Assignee: Billy Liu > Attachments: KYLIN-2428.patch > > > Kylin releases three libraries: kylin-coprocessor, kylin-jdbc, kylin-job and > one web application: server. > Currently, all libraries have shaded some used third party libraries into the > package. For example, guava, curator, commons, kyro in kylin-job. The > duplicate libraries in runtime classpath may have potential class loading > conflicts and waste computing resource. We should leverage the hadoop > provided libraries at runtime instead of the shaded one. -- This message was sent by Atlassian JIRA (v6.3.15#6346)