[jira] [Updated] (HIVE-22162) MVs are not using ACID tables.
[ https://issues.apache.org/jira/browse/HIVE-22162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-22162: -- Status: Patch Available (was: Open) > MVs are not using ACID tables. > -- > > Key: HIVE-22162 > URL: https://issues.apache.org/jira/browse/HIVE-22162 > Project: Hive > Issue Type: Bug > Components: Materialized views >Affects Versions: 3.1.2 >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22162.1.patch, HIVE-22162.2.patch, > HIVE-22162.3.patch > > > {code} > SET hive.support.concurrency=true; > SET hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; > SET metastore.strict.managed.tables=true; > SET hive.default.fileformat=textfile; > SET hive.default.fileformat.managed=orc; > SET metastore.create.as.acid=true; > CREATE TABLE cmv_basetable_n4 (a int, b varchar(256), c decimal(10,2)); > INSERT INTO cmv_basetable_n4 VALUES (1, 'alfred', 10.30),(2, 'bob', 3.14),(2, > 'bonnie', 172342.2),(3, 'calvin', 978.76),(3, 'charlie', 9.8); > CREATE MATERIALIZED VIEW cmv_mat_view_n4 disable rewrite > AS SELECT a, b, c FROM cmv_basetable_n4; > DESCRIBE FORMATTED cmv_mat_view_n4; > {code} > {code} > POSTHOOK: query: DESCRIBE FORMATTED cmv_mat_view_n4 > ... > Table Type: MATERIALIZED_VIEW > Table Parameters: > COLUMN_STATS_ACCURATE > {\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"a\":\"true\",\"b\":\"true\",\"c\":\"true\"}} > bucketing_version 2 > numFiles1 > numRows 5 > rawDataSize 1025 > totalSize 509 > {code} > Missing table parameter > {code} > transaction = true > {code} > cc.: [~ashutoshc], [~gopalv], [~jcamachorodriguez] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22162) MVs are not using ACID tables.
[ https://issues.apache.org/jira/browse/HIVE-22162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-22162: -- Status: Open (was: Patch Available) > MVs are not using ACID tables. > -- > > Key: HIVE-22162 > URL: https://issues.apache.org/jira/browse/HIVE-22162 > Project: Hive > Issue Type: Bug > Components: Materialized views >Affects Versions: 3.1.2 >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22162.1.patch, HIVE-22162.2.patch, > HIVE-22162.3.patch > > > {code} > SET hive.support.concurrency=true; > SET hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; > SET metastore.strict.managed.tables=true; > SET hive.default.fileformat=textfile; > SET hive.default.fileformat.managed=orc; > SET metastore.create.as.acid=true; > CREATE TABLE cmv_basetable_n4 (a int, b varchar(256), c decimal(10,2)); > INSERT INTO cmv_basetable_n4 VALUES (1, 'alfred', 10.30),(2, 'bob', 3.14),(2, > 'bonnie', 172342.2),(3, 'calvin', 978.76),(3, 'charlie', 9.8); > CREATE MATERIALIZED VIEW cmv_mat_view_n4 disable rewrite > AS SELECT a, b, c FROM cmv_basetable_n4; > DESCRIBE FORMATTED cmv_mat_view_n4; > {code} > {code} > POSTHOOK: query: DESCRIBE FORMATTED cmv_mat_view_n4 > ... > Table Type: MATERIALIZED_VIEW > Table Parameters: > COLUMN_STATS_ACCURATE > {\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"a\":\"true\",\"b\":\"true\",\"c\":\"true\"}} > bucketing_version 2 > numFiles1 > numRows 5 > rawDataSize 1025 > totalSize 509 > {code} > Missing table parameter > {code} > transaction = true > {code} > cc.: [~ashutoshc], [~gopalv], [~jcamachorodriguez] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22162) MVs are not using ACID tables.
[ https://issues.apache.org/jira/browse/HIVE-22162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-22162: -- Attachment: HIVE-22162.3.patch > MVs are not using ACID tables. > -- > > Key: HIVE-22162 > URL: https://issues.apache.org/jira/browse/HIVE-22162 > Project: Hive > Issue Type: Bug > Components: Materialized views >Affects Versions: 3.1.2 >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22162.1.patch, HIVE-22162.2.patch, > HIVE-22162.3.patch > > > {code} > SET hive.support.concurrency=true; > SET hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; > SET metastore.strict.managed.tables=true; > SET hive.default.fileformat=textfile; > SET hive.default.fileformat.managed=orc; > SET metastore.create.as.acid=true; > CREATE TABLE cmv_basetable_n4 (a int, b varchar(256), c decimal(10,2)); > INSERT INTO cmv_basetable_n4 VALUES (1, 'alfred', 10.30),(2, 'bob', 3.14),(2, > 'bonnie', 172342.2),(3, 'calvin', 978.76),(3, 'charlie', 9.8); > CREATE MATERIALIZED VIEW cmv_mat_view_n4 disable rewrite > AS SELECT a, b, c FROM cmv_basetable_n4; > DESCRIBE FORMATTED cmv_mat_view_n4; > {code} > {code} > POSTHOOK: query: DESCRIBE FORMATTED cmv_mat_view_n4 > ... > Table Type: MATERIALIZED_VIEW > Table Parameters: > COLUMN_STATS_ACCURATE > {\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"a\":\"true\",\"b\":\"true\",\"c\":\"true\"}} > bucketing_version 2 > numFiles1 > numRows 5 > rawDataSize 1025 > totalSize 509 > {code} > Missing table parameter > {code} > transaction = true > {code} > cc.: [~ashutoshc], [~gopalv], [~jcamachorodriguez] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-21930) WINDOW COUNT DISTINCT return wrong value with PARTITION BY
[ https://issues.apache.org/jira/browse/HIVE-21930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16920717#comment-16920717 ] Igor commented on HIVE-21930: - Any updates on this? > WINDOW COUNT DISTINCT return wrong value with PARTITION BY > -- > > Key: HIVE-21930 > URL: https://issues.apache.org/jira/browse/HIVE-21930 > Project: Hive > Issue Type: Bug > Components: PTF-Windowing >Affects Versions: 3.1.0 > Environment: Beeline version 3.1.0.3.0.1.0-187 by Apache Hive >Reporter: Igor >Priority: Major > Labels: distinct, window_funcion > > count(distinct a) over (partiton by b) return wring result. For example (T is > CTE here): > {code:java} > select p, day, ts > , row_number() OVER (PARTITION BY phone ORDER BY ts ASC) as line_number > , count(1) OVER (PARTITION BY phone ROWS BETWEEN UNBOUNDED PRECEDING AND > UNBOUNDED FOLLOWING) as lines > , count(distinct day) OVER (PARTITION BY phone ROWS BETWEEN UNBOUNDED > PRECEDING AND UNBOUNDED FOLLOWING) as days > FROM T{code} > WINDOW specification doesn't affect on results: same wrong with and without > window. > count(1) and count(distinct day) return the same result. Count distinct is > wrong. > > I've add size(collect_set(day) OVER (PARTITION BY phone)) as days2 and > count(distinct return correct result. > Following query return non-empty result: > {code:java} > select A.*, B.days, B. from ( > select p, day, ts > , row_number() OVER (PARTITION BY phone ORDER BY ts ASC) as line_number > , count(1) OVER (PARTITION BY p ROWS BETWEEN UNBOUNDED PRECEDING AND > UNBOUNDED FOLLOWING) as lines > , count(distinct day) OVER (PARTITION BY phone ROWS BETWEEN UNBOUNDED > PRECEDING AND UNBOUNDED FOLLOWING) as days > , size(collect_set(day) OVER (PARTITION BY phone)) as days2 > , dense_rank() over (partition by phone order by day) + dense_rank() over > (partition by phone order by day desc) - 1 as days3 > FROM T ) as A > join ( > select p, day, ts > , row_number() OVER (PARTITION BY phone ORDER BY ts ASC) as line_number > , count(1) OVER (PARTITION BY phone ROWS BETWEEN UNBOUNDED PRECEDING AND > UNBOUNDED FOLLOWING) as lines > , count(distinct day) OVER (PARTITION BY phone ROWS BETWEEN UNBOUNDED > PRECEDING AND UNBOUNDED FOLLOWING) as days > FROM T > ) as B on A.p=B.p and A.line_number=B.line_number > where A.days!=B.days > order by A.p, A.line_number > {code} > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-21737) Upgrade Avro to version 1.9.1
[ https://issues.apache.org/jira/browse/HIVE-21737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong updated HIVE-21737: Summary: Upgrade Avro to version 1.9.1 (was: Upgrade Avro to version 1.9.0) > Upgrade Avro to version 1.9.1 > - > > Key: HIVE-21737 > URL: https://issues.apache.org/jira/browse/HIVE-21737 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Ismaël Mejía >Assignee: Fokko Driesprong >Priority: Minor > Labels: pull-request-available > Attachments: 0001-HIVE-21737-Bump-Apache-Avro-to-1.9.0.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Avro 1.9.0 was released recently. It brings a lot of fixes including a leaner > version of Avro without Jackson in the public API. Worth the update. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-21737) Upgrade Avro to version 1.9.1
[ https://issues.apache.org/jira/browse/HIVE-21737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong updated HIVE-21737: Attachment: (was: 0001-HIVE-21737-Bump-Apache-Avro-to-1.9.0.patch) > Upgrade Avro to version 1.9.1 > - > > Key: HIVE-21737 > URL: https://issues.apache.org/jira/browse/HIVE-21737 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Ismaël Mejía >Assignee: Fokko Driesprong >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Avro 1.9.0 was released recently. It brings a lot of fixes including a leaner > version of Avro without Jackson in the public API. Worth the update. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22149) Metastore: Unify codahale metrics.log json structure between hiveserver2 and metastore services
[ https://issues.apache.org/jira/browse/HIVE-22149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-22149: Attachment: metrics_metastore.log metrics_hiveserver2.log > Metastore: Unify codahale metrics.log json structure between hiveserver2 and > metastore services > --- > > Key: HIVE-22149 > URL: https://issues.apache.org/jira/browse/HIVE-22149 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Attachments: metrics_hiveserver2.log, metrics_metastore.log > > > While fixing HIVE-22140 I found some really annoying differences between the > codahale metric file structures between hiveserver2 and metastore, e.g. > open_connections: can be found in "counters" for hs2, but in "gauges" for ms > threads count: it's a proper "threads.count" for hs2, but a really ambiguous > "count" for ms > so I realized that "memory." and "threads." prefix is completely absent in ms > metrics file, which is misleading -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-21737) Upgrade Avro to version 1.9.1
[ https://issues.apache.org/jira/browse/HIVE-21737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong updated HIVE-21737: Attachment: 0001-HIVE-21737-Bump-Apache-Avro-to-1.9.1.patch > Upgrade Avro to version 1.9.1 > - > > Key: HIVE-21737 > URL: https://issues.apache.org/jira/browse/HIVE-21737 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Ismaël Mejía >Assignee: Fokko Driesprong >Priority: Minor > Labels: pull-request-available > Attachments: 0001-HIVE-21737-Bump-Apache-Avro-to-1.9.1.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Avro 1.9.0 was released recently. It brings a lot of fixes including a leaner > version of Avro without Jackson in the public API. Worth the update. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22149) Metastore: Unify codahale metrics.log json structure between hiveserver2 and metastore services
[ https://issues.apache.org/jira/browse/HIVE-22149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-22149: Attachment: HIVE-22149.01.patch > Metastore: Unify codahale metrics.log json structure between hiveserver2 and > metastore services > --- > > Key: HIVE-22149 > URL: https://issues.apache.org/jira/browse/HIVE-22149 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-22149.01.patch, metrics_hiveserver2.log, > metrics_metastore.log > > > While fixing HIVE-22140 I found some really annoying differences between the > codahale metric file structures between hiveserver2 and metastore, e.g. > open_connections: can be found in "counters" for hs2, but in "gauges" for ms > threads count: it's a proper "threads.count" for hs2, but a really ambiguous > "count" for ms > so I realized that "memory." and "threads." prefix is completely absent in ms > metrics file, which is misleading -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22149) Metastore: Unify codahale metrics.log json structure between hiveserver2 and metastore services
[ https://issues.apache.org/jira/browse/HIVE-22149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-22149: Status: Patch Available (was: Open) > Metastore: Unify codahale metrics.log json structure between hiveserver2 and > metastore services > --- > > Key: HIVE-22149 > URL: https://issues.apache.org/jira/browse/HIVE-22149 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-22149.01.patch, metrics_hiveserver2.log, > metrics_metastore.log > > > While fixing HIVE-22140 I found some really annoying differences between the > codahale metric file structures between hiveserver2 and metastore, e.g. > open_connections: can be found in "counters" for hs2, but in "gauges" for ms > threads count: it's a proper "threads.count" for hs2, but a really ambiguous > "count" for ms > so I realized that "memory." and "threads." prefix is completely absent in ms > metrics file, which is misleading -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-22162) MVs are not using ACID tables.
[ https://issues.apache.org/jira/browse/HIVE-22162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16920840#comment-16920840 ] Hive QA commented on HIVE-22162: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 7s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 0s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 41s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 38s{color} | {color:blue} ql in master has 2248 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 17s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 53s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 22m 46s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-18435/dev-support/hive-personality.sh | | git revision | master / 04397e5 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-18435/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > MVs are not using ACID tables. > -- > > Key: HIVE-22162 > URL: https://issues.apache.org/jira/browse/HIVE-22162 > Project: Hive > Issue Type: Bug > Components: Materialized views >Affects Versions: 3.1.2 >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22162.1.patch, HIVE-22162.2.patch, > HIVE-22162.3.patch > > > {code} > SET hive.support.concurrency=true; > SET hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; > SET metastore.strict.managed.tables=true; > SET hive.default.fileformat=textfile; > SET hive.default.fileformat.managed=orc; > SET metastore.create.as.acid=true; > CREATE TABLE cmv_basetable_n4 (a int, b varchar(256), c decimal(10,2)); > INSERT INTO cmv_basetable_n4 VALUES (1, 'alfred', 10.30),(2, 'bob', 3.14),(2, > 'bonnie', 172342.2),(3, 'calvin', 978.76),(3, 'charlie', 9.8); > CREATE MATERIALIZED VIEW cmv_mat_view_n4 disable rewrite > AS SELECT a, b, c FROM cmv_basetable_n4; > DESCRIBE FORMATTED cmv_mat_view_n4; > {code} > {code} > POSTHOOK: query: DESCRIBE FORMATTED cmv_mat_view_n4 > ... > Table Type: MATERIALIZED_VIEW > Table Parameters: > COLUMN_STATS_ACCURATE > {\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"a\":\"true\",\"b\":\"true\",\"c\":\"true\"}} > bucketing_version 2 > numFiles1 > numRows 5 > rawDataSize 1025 > totalSize 509
[jira] [Updated] (HIVE-22149) Metastore: Unify codahale metrics.log json structure between hiveserver2 and metastore services
[ https://issues.apache.org/jira/browse/HIVE-22149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-22149: Attachment: (was: HIVE-22149.01.patch) > Metastore: Unify codahale metrics.log json structure between hiveserver2 and > metastore services > --- > > Key: HIVE-22149 > URL: https://issues.apache.org/jira/browse/HIVE-22149 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-22149.01.patch, metrics_hiveserver2.log, > metrics_metastore.log > > > While fixing HIVE-22140 I found some really annoying differences between the > codahale metric file structures between hiveserver2 and metastore, e.g. > open_connections: can be found in "counters" for hs2, but in "gauges" for ms > threads count: it's a proper "threads.count" for hs2, but a really ambiguous > "count" for ms > so I realized that "memory." and "threads." prefix is completely absent in ms > metrics file, which is misleading -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22149) Metastore: Unify codahale metrics.log json structure between hiveserver2 and metastore services
[ https://issues.apache.org/jira/browse/HIVE-22149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-22149: Attachment: HIVE-22149.01.patch > Metastore: Unify codahale metrics.log json structure between hiveserver2 and > metastore services > --- > > Key: HIVE-22149 > URL: https://issues.apache.org/jira/browse/HIVE-22149 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-22149.01.patch, metrics_hiveserver2.log, > metrics_metastore.log > > > While fixing HIVE-22140 I found some really annoying differences between the > codahale metric file structures between hiveserver2 and metastore, e.g. > open_connections: can be found in "counters" for hs2, but in "gauges" for ms > threads count: it's a proper "threads.count" for hs2, but a really ambiguous > "count" for ms > so I realized that "memory." and "threads." prefix is completely absent in ms > metrics file, which is misleading -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-22162) MVs are not using ACID tables.
[ https://issues.apache.org/jira/browse/HIVE-22162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16920880#comment-16920880 ] Hive QA commented on HIVE-22162: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12979110/HIVE-22162.3.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 16746 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.llap.security.TestLlapSignerImpl.testSigning (batchId=361) org.apache.hadoop.hive.ql.TestTxnCommands.testMergeOnTezEdges (batchId=351) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/18435/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18435/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18435/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12979110 - PreCommit-HIVE-Build > MVs are not using ACID tables. > -- > > Key: HIVE-22162 > URL: https://issues.apache.org/jira/browse/HIVE-22162 > Project: Hive > Issue Type: Bug > Components: Materialized views >Affects Versions: 3.1.2 >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22162.1.patch, HIVE-22162.2.patch, > HIVE-22162.3.patch > > > {code} > SET hive.support.concurrency=true; > SET hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; > SET metastore.strict.managed.tables=true; > SET hive.default.fileformat=textfile; > SET hive.default.fileformat.managed=orc; > SET metastore.create.as.acid=true; > CREATE TABLE cmv_basetable_n4 (a int, b varchar(256), c decimal(10,2)); > INSERT INTO cmv_basetable_n4 VALUES (1, 'alfred', 10.30),(2, 'bob', 3.14),(2, > 'bonnie', 172342.2),(3, 'calvin', 978.76),(3, 'charlie', 9.8); > CREATE MATERIALIZED VIEW cmv_mat_view_n4 disable rewrite > AS SELECT a, b, c FROM cmv_basetable_n4; > DESCRIBE FORMATTED cmv_mat_view_n4; > {code} > {code} > POSTHOOK: query: DESCRIBE FORMATTED cmv_mat_view_n4 > ... > Table Type: MATERIALIZED_VIEW > Table Parameters: > COLUMN_STATS_ACCURATE > {\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"a\":\"true\",\"b\":\"true\",\"c\":\"true\"}} > bucketing_version 2 > numFiles1 > numRows 5 > rawDataSize 1025 > totalSize 509 > {code} > Missing table parameter > {code} > transaction = true > {code} > cc.: [~ashutoshc], [~gopalv], [~jcamachorodriguez] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (HIVE-20683) Add the Ability to push Dynamic Between and Bloom filters to Druid
[ https://issues.apache.org/jira/browse/HIVE-20683?focusedWorklogId=305285&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-305285 ] ASF GitHub Bot logged work on HIVE-20683: - Author: ASF GitHub Bot Created on: 02/Sep/19 17:02 Start Date: 02/Sep/19 17:02 Worklog Time Spent: 10m Work Description: nishantmonu51 commented on pull request #723: [HIVE-20683] Add the Ability to push Dynamic Between and Bloom filters to Druid URL: https://github.com/apache/hive/pull/723#discussion_r320020777 ## File path: druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandlerUtils.java ## @@ -894,4 +944,257 @@ public static IndexSpec getIndexSpec(Configuration jc) { ImmutableList aggregatorFactories = aggregatorFactoryBuilder.build(); return Pair.of(dimensions, aggregatorFactories.toArray(new AggregatorFactory[0])); } + + // Druid only supports String,Long,Float,Double selectors + private static Set druidSupportedTypeInfos = ImmutableSet.of( + TypeInfoFactory.stringTypeInfo, TypeInfoFactory.charTypeInfo, + TypeInfoFactory.varcharTypeInfo, TypeInfoFactory.byteTypeInfo, + TypeInfoFactory.intTypeInfo, TypeInfoFactory.longTypeInfo, + TypeInfoFactory.shortTypeInfo, TypeInfoFactory.doubleTypeInfo + ); + + private static Set stringTypeInfos = ImmutableSet.of( + TypeInfoFactory.stringTypeInfo, + TypeInfoFactory.charTypeInfo, TypeInfoFactory.varcharTypeInfo + ); + + + public static org.apache.druid.query.Query addDynamicFilters(org.apache.druid.query.Query query, + ExprNodeGenericFuncDesc filterExpr, Configuration conf, boolean resolveDynamicValues + ) { +List virtualColumns = Arrays +.asList(getVirtualColumns(query).getVirtualColumns()); +org.apache.druid.query.Query rv = query; +DimFilter joinReductionFilter = toDruidFilter(filterExpr, conf, virtualColumns, +resolveDynamicValues +); +if(joinReductionFilter != null) { + String type = query.getType(); + DimFilter filter = new AndDimFilter(joinReductionFilter, query.getFilter()); + switch (type) { + case org.apache.druid.query.Query.TIMESERIES: +rv = Druids.TimeseriesQueryBuilder.copy((TimeseriesQuery) query) +.filters(filter) +.virtualColumns(VirtualColumns.create(virtualColumns)) +.build(); +break; + case org.apache.druid.query.Query.TOPN: +rv = new TopNQueryBuilder((TopNQuery) query) +.filters(filter) +.virtualColumns(VirtualColumns.create(virtualColumns)) +.build(); +break; + case org.apache.druid.query.Query.GROUP_BY: +rv = new GroupByQuery.Builder((GroupByQuery) query) +.setDimFilter(filter) +.setVirtualColumns(VirtualColumns.create(virtualColumns)) +.build(); +break; + case org.apache.druid.query.Query.SCAN: +rv = ScanQuery.ScanQueryBuilder.copy((ScanQuery) query) +.filters(filter) +.virtualColumns(VirtualColumns.create(virtualColumns)) +.build(); +break; + case org.apache.druid.query.Query.SELECT: +rv = Druids.SelectQueryBuilder.copy((SelectQuery) query) +.filters(filter) +.virtualColumns(VirtualColumns.create(virtualColumns)) +.build(); +break; + default: +throw new UnsupportedOperationException("Unsupported Query type " + type); + } +} +return rv; + } + + private static DimFilter toDruidFilter(ExprNodeDesc filterExpr, Configuration configuration, + List virtualColumns, boolean resolveDynamicValues + ) { +if(filterExpr == null) { + return null; +} +Class genericUDFClass = getGenericUDFClassFromExprDesc(filterExpr); +if(FunctionRegistry.isOpAnd(filterExpr)) { + Iterator iterator = filterExpr.getChildren().iterator(); + List delegates = Lists.newArrayList(); + while (iterator.hasNext()) { +DimFilter filter = toDruidFilter(iterator.next(), configuration, virtualColumns, +resolveDynamicValues +); +if(filter != null) { + delegates.add(filter); +} + } + if(delegates != null && !delegates.isEmpty()) { +return new AndDimFilter(delegates); + } +} +if(FunctionRegistry.isOpOr(filterExpr)) { + Iterator iterator = filterExpr.getChildren().iterator(); + List delegates = Lists.newArrayList(); + while (iterator.hasNext()) { +DimFilter filter = toDruidFilter(iterator.next(), configuration, virtualColumns, +resolveDynamicValues +); +if(filter != null) { + delegates.add(filter); +} + } + if(delegates != null) { +return new OrDimFilter(delegates); + } +} else if(GenericUDFBetween.class ==
[jira] [Work logged] (HIVE-20683) Add the Ability to push Dynamic Between and Bloom filters to Druid
[ https://issues.apache.org/jira/browse/HIVE-20683?focusedWorklogId=305284&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-305284 ] ASF GitHub Bot logged work on HIVE-20683: - Author: ASF GitHub Bot Created on: 02/Sep/19 17:02 Start Date: 02/Sep/19 17:02 Worklog Time Spent: 10m Work Description: nishantmonu51 commented on pull request #723: [HIVE-20683] Add the Ability to push Dynamic Between and Bloom filters to Druid URL: https://github.com/apache/hive/pull/723#discussion_r320020745 ## File path: druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandlerUtils.java ## @@ -894,4 +944,257 @@ public static IndexSpec getIndexSpec(Configuration jc) { ImmutableList aggregatorFactories = aggregatorFactoryBuilder.build(); return Pair.of(dimensions, aggregatorFactories.toArray(new AggregatorFactory[0])); } + + // Druid only supports String,Long,Float,Double selectors + private static Set druidSupportedTypeInfos = ImmutableSet.of( + TypeInfoFactory.stringTypeInfo, TypeInfoFactory.charTypeInfo, + TypeInfoFactory.varcharTypeInfo, TypeInfoFactory.byteTypeInfo, + TypeInfoFactory.intTypeInfo, TypeInfoFactory.longTypeInfo, + TypeInfoFactory.shortTypeInfo, TypeInfoFactory.doubleTypeInfo + ); + + private static Set stringTypeInfos = ImmutableSet.of( + TypeInfoFactory.stringTypeInfo, + TypeInfoFactory.charTypeInfo, TypeInfoFactory.varcharTypeInfo + ); + + + public static org.apache.druid.query.Query addDynamicFilters(org.apache.druid.query.Query query, + ExprNodeGenericFuncDesc filterExpr, Configuration conf, boolean resolveDynamicValues + ) { +List virtualColumns = Arrays +.asList(getVirtualColumns(query).getVirtualColumns()); +org.apache.druid.query.Query rv = query; +DimFilter joinReductionFilter = toDruidFilter(filterExpr, conf, virtualColumns, +resolveDynamicValues +); +if(joinReductionFilter != null) { + String type = query.getType(); + DimFilter filter = new AndDimFilter(joinReductionFilter, query.getFilter()); + switch (type) { + case org.apache.druid.query.Query.TIMESERIES: +rv = Druids.TimeseriesQueryBuilder.copy((TimeseriesQuery) query) +.filters(filter) +.virtualColumns(VirtualColumns.create(virtualColumns)) +.build(); +break; + case org.apache.druid.query.Query.TOPN: +rv = new TopNQueryBuilder((TopNQuery) query) +.filters(filter) +.virtualColumns(VirtualColumns.create(virtualColumns)) +.build(); +break; + case org.apache.druid.query.Query.GROUP_BY: +rv = new GroupByQuery.Builder((GroupByQuery) query) +.setDimFilter(filter) +.setVirtualColumns(VirtualColumns.create(virtualColumns)) +.build(); +break; + case org.apache.druid.query.Query.SCAN: +rv = ScanQuery.ScanQueryBuilder.copy((ScanQuery) query) +.filters(filter) +.virtualColumns(VirtualColumns.create(virtualColumns)) +.build(); +break; + case org.apache.druid.query.Query.SELECT: +rv = Druids.SelectQueryBuilder.copy((SelectQuery) query) +.filters(filter) +.virtualColumns(VirtualColumns.create(virtualColumns)) +.build(); +break; + default: +throw new UnsupportedOperationException("Unsupported Query type " + type); + } +} +return rv; + } + + private static DimFilter toDruidFilter(ExprNodeDesc filterExpr, Configuration configuration, Review comment: added This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 305284) Time Spent: 1h 20m (was: 1h 10m) > Add the Ability to push Dynamic Between and Bloom filters to Druid > -- > > Key: HIVE-20683 > URL: https://issues.apache.org/jira/browse/HIVE-20683 > Project: Hive > Issue Type: New Feature > Components: Druid integration >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20683.1.patch, HIVE-20683.2.patch, > HIVE-20683.3.patch, HIVE-20683.4.patch, HIVE-20683.5.patch, > HIVE-20683.6.patch, HIVE-20683.8.patch, HIVE-20683.patch > > Time Spent: 1h 20m > Remaining Estimate: 0h > > For optimizing joins, Hive generates BETWEEN fil
[jira] [Work logged] (HIVE-20683) Add the Ability to push Dynamic Between and Bloom filters to Druid
[ https://issues.apache.org/jira/browse/HIVE-20683?focusedWorklogId=305286&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-305286 ] ASF GitHub Bot logged work on HIVE-20683: - Author: ASF GitHub Bot Created on: 02/Sep/19 17:03 Start Date: 02/Sep/19 17:03 Worklog Time Spent: 10m Work Description: nishantmonu51 commented on pull request #723: [HIVE-20683] Add the Ability to push Dynamic Between and Bloom filters to Druid URL: https://github.com/apache/hive/pull/723#discussion_r320020948 ## File path: druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandlerUtils.java ## @@ -894,4 +944,257 @@ public static IndexSpec getIndexSpec(Configuration jc) { ImmutableList aggregatorFactories = aggregatorFactoryBuilder.build(); return Pair.of(dimensions, aggregatorFactories.toArray(new AggregatorFactory[0])); } + + // Druid only supports String,Long,Float,Double selectors + private static Set druidSupportedTypeInfos = ImmutableSet.of( + TypeInfoFactory.stringTypeInfo, TypeInfoFactory.charTypeInfo, + TypeInfoFactory.varcharTypeInfo, TypeInfoFactory.byteTypeInfo, + TypeInfoFactory.intTypeInfo, TypeInfoFactory.longTypeInfo, + TypeInfoFactory.shortTypeInfo, TypeInfoFactory.doubleTypeInfo + ); + + private static Set stringTypeInfos = ImmutableSet.of( + TypeInfoFactory.stringTypeInfo, + TypeInfoFactory.charTypeInfo, TypeInfoFactory.varcharTypeInfo + ); + + + public static org.apache.druid.query.Query addDynamicFilters(org.apache.druid.query.Query query, Review comment: Not sure about it yet, as most of the logic is similar is for query filters probably we can separate things into DruidQueryUtils at some point in a separate PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 305286) Time Spent: 1h 40m (was: 1.5h) > Add the Ability to push Dynamic Between and Bloom filters to Druid > -- > > Key: HIVE-20683 > URL: https://issues.apache.org/jira/browse/HIVE-20683 > Project: Hive > Issue Type: New Feature > Components: Druid integration >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20683.1.patch, HIVE-20683.2.patch, > HIVE-20683.3.patch, HIVE-20683.4.patch, HIVE-20683.5.patch, > HIVE-20683.6.patch, HIVE-20683.8.patch, HIVE-20683.patch > > Time Spent: 1h 40m > Remaining Estimate: 0h > > For optimizing joins, Hive generates BETWEEN filter with min-max and BLOOM > filter for filtering one side of semi-join. > Druid 0.13.0 will have support for Bloom filters (Added via > https://github.com/apache/incubator-druid/pull/6222) > Implementation details - > # Hive generates and passes the filters as part of 'filterExpr' in TableScan. > # DruidQueryBasedRecordReader gets this filter passed as part of the conf. > # During execution phase, before sending the query to druid in > DruidQueryBasedRecordReader we will deserialize this filter, translate it > into a DruidDimFilter and add it to existing DruidQuery. Tez executor > already ensures that when we start reading results from the record reader, > all the dynamic values are initialized. > # Explaining a druid query also prints the query sent to druid as > {{druid.json.query}}. We also need to make sure to update the druid query > with the filters. During explain we do not have the actual values for the > dynamic values, so instead of values we will print the dynamic expression > itself as part of druid query. > Note:- This work needs druid to be updated to version 0.13.0 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (HIVE-20683) Add the Ability to push Dynamic Between and Bloom filters to Druid
[ https://issues.apache.org/jira/browse/HIVE-20683?focusedWorklogId=305287&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-305287 ] ASF GitHub Bot logged work on HIVE-20683: - Author: ASF GitHub Bot Created on: 02/Sep/19 17:03 Start Date: 02/Sep/19 17:03 Worklog Time Spent: 10m Work Description: nishantmonu51 commented on pull request #723: [HIVE-20683] Add the Ability to push Dynamic Between and Bloom filters to Druid URL: https://github.com/apache/hive/pull/723#discussion_r320020963 ## File path: ql/src/test/results/clientpositive/druid/druidmini_joins.q.out ## @@ -223,8 +228,8 @@ GROUP BY `tbl1`.`username` POSTHOOK: type: QUERY Review comment: added This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 305287) Time Spent: 1h 50m (was: 1h 40m) > Add the Ability to push Dynamic Between and Bloom filters to Druid > -- > > Key: HIVE-20683 > URL: https://issues.apache.org/jira/browse/HIVE-20683 > Project: Hive > Issue Type: New Feature > Components: Druid integration >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20683.1.patch, HIVE-20683.2.patch, > HIVE-20683.3.patch, HIVE-20683.4.patch, HIVE-20683.5.patch, > HIVE-20683.6.patch, HIVE-20683.8.patch, HIVE-20683.patch > > Time Spent: 1h 50m > Remaining Estimate: 0h > > For optimizing joins, Hive generates BETWEEN filter with min-max and BLOOM > filter for filtering one side of semi-join. > Druid 0.13.0 will have support for Bloom filters (Added via > https://github.com/apache/incubator-druid/pull/6222) > Implementation details - > # Hive generates and passes the filters as part of 'filterExpr' in TableScan. > # DruidQueryBasedRecordReader gets this filter passed as part of the conf. > # During execution phase, before sending the query to druid in > DruidQueryBasedRecordReader we will deserialize this filter, translate it > into a DruidDimFilter and add it to existing DruidQuery. Tez executor > already ensures that when we start reading results from the record reader, > all the dynamic values are initialized. > # Explaining a druid query also prints the query sent to druid as > {{druid.json.query}}. We also need to make sure to update the druid query > with the filters. During explain we do not have the actual values for the > dynamic values, so instead of values we will print the dynamic expression > itself as part of druid query. > Note:- This work needs druid to be updated to version 0.13.0 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (HIVE-20683) Add the Ability to push Dynamic Between and Bloom filters to Druid
[ https://issues.apache.org/jira/browse/HIVE-20683?focusedWorklogId=305288&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-305288 ] ASF GitHub Bot logged work on HIVE-20683: - Author: ASF GitHub Bot Created on: 02/Sep/19 17:04 Start Date: 02/Sep/19 17:04 Worklog Time Spent: 10m Work Description: nishantmonu51 commented on pull request #723: [HIVE-20683] Add the Ability to push Dynamic Between and Bloom filters to Druid URL: https://github.com/apache/hive/pull/723#discussion_r320021144 ## File path: ql/src/test/results/clientpositive/druid/druidmini_expressions.q.out ## @@ -1868,9 +1868,9 @@ POSTHOOK: query: SELECT DATE_ADD(cast(`__time` as date), CAST((cdouble / 1000) A POSTHOOK: type: QUERY POSTHOOK: Input: default@druid_table_alltypesorc POSTHOOK: Output: hdfs://### HDFS PATH ### -1969-02-26 1970-11-04 Review comment: checked it out, DATE_ADD(cast(`__time` as date), CAST((cdouble / 1000) changed as the cdouble value changed due to change in rollup. Added more columns to this query to make things more clear in this PR This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 305288) Time Spent: 2h (was: 1h 50m) > Add the Ability to push Dynamic Between and Bloom filters to Druid > -- > > Key: HIVE-20683 > URL: https://issues.apache.org/jira/browse/HIVE-20683 > Project: Hive > Issue Type: New Feature > Components: Druid integration >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20683.1.patch, HIVE-20683.2.patch, > HIVE-20683.3.patch, HIVE-20683.4.patch, HIVE-20683.5.patch, > HIVE-20683.6.patch, HIVE-20683.8.patch, HIVE-20683.patch > > Time Spent: 2h > Remaining Estimate: 0h > > For optimizing joins, Hive generates BETWEEN filter with min-max and BLOOM > filter for filtering one side of semi-join. > Druid 0.13.0 will have support for Bloom filters (Added via > https://github.com/apache/incubator-druid/pull/6222) > Implementation details - > # Hive generates and passes the filters as part of 'filterExpr' in TableScan. > # DruidQueryBasedRecordReader gets this filter passed as part of the conf. > # During execution phase, before sending the query to druid in > DruidQueryBasedRecordReader we will deserialize this filter, translate it > into a DruidDimFilter and add it to existing DruidQuery. Tez executor > already ensures that when we start reading results from the record reader, > all the dynamic values are initialized. > # Explaining a druid query also prints the query sent to druid as > {{druid.json.query}}. We also need to make sure to update the druid query > with the filters. During explain we do not have the actual values for the > dynamic values, so instead of values we will print the dynamic expression > itself as part of druid query. > Note:- This work needs druid to be updated to version 0.13.0 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (HIVE-20683) Add the Ability to push Dynamic Between and Bloom filters to Druid
[ https://issues.apache.org/jira/browse/HIVE-20683?focusedWorklogId=305289&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-305289 ] ASF GitHub Bot logged work on HIVE-20683: - Author: ASF GitHub Bot Created on: 02/Sep/19 17:04 Start Date: 02/Sep/19 17:04 Worklog Time Spent: 10m Work Description: nishantmonu51 commented on pull request #723: [HIVE-20683] Add the Ability to push Dynamic Between and Bloom filters to Druid URL: https://github.com/apache/hive/pull/723#discussion_r320021168 ## File path: ql/src/test/results/clientpositive/druid/druidmini_expressions.q.out ## @@ -1868,9 +1868,9 @@ POSTHOOK: query: SELECT DATE_ADD(cast(`__time` as date), CAST((cdouble / 1000) A POSTHOOK: type: QUERY POSTHOOK: Input: default@druid_table_alltypesorc POSTHOOK: Output: hdfs://### HDFS PATH ### -1969-02-26 1970-11-04 -1969-03-19 1970-10-14 -1969-11-13 1970-02-17 +1969-12-15 1970-01-16 +1969-12-15 1970-01-16 +1969-12-15 1970-01-16 Review comment: checked it out, DATE_ADD(cast(__time as date), CAST((cdouble / 1000) changed as the cdouble value changed due to change in rollup. Added more columns to this query to make things more clear in this PR This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 305289) Time Spent: 2h 10m (was: 2h) > Add the Ability to push Dynamic Between and Bloom filters to Druid > -- > > Key: HIVE-20683 > URL: https://issues.apache.org/jira/browse/HIVE-20683 > Project: Hive > Issue Type: New Feature > Components: Druid integration >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20683.1.patch, HIVE-20683.2.patch, > HIVE-20683.3.patch, HIVE-20683.4.patch, HIVE-20683.5.patch, > HIVE-20683.6.patch, HIVE-20683.8.patch, HIVE-20683.patch > > Time Spent: 2h 10m > Remaining Estimate: 0h > > For optimizing joins, Hive generates BETWEEN filter with min-max and BLOOM > filter for filtering one side of semi-join. > Druid 0.13.0 will have support for Bloom filters (Added via > https://github.com/apache/incubator-druid/pull/6222) > Implementation details - > # Hive generates and passes the filters as part of 'filterExpr' in TableScan. > # DruidQueryBasedRecordReader gets this filter passed as part of the conf. > # During execution phase, before sending the query to druid in > DruidQueryBasedRecordReader we will deserialize this filter, translate it > into a DruidDimFilter and add it to existing DruidQuery. Tez executor > already ensures that when we start reading results from the record reader, > all the dynamic values are initialized. > # Explaining a druid query also prints the query sent to druid as > {{druid.json.query}}. We also need to make sure to update the druid query > with the filters. During explain we do not have the actual values for the > dynamic values, so instead of values we will print the dynamic expression > itself as part of druid query. > Note:- This work needs druid to be updated to version 0.13.0 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (HIVE-20683) Add the Ability to push Dynamic Between and Bloom filters to Druid
[ https://issues.apache.org/jira/browse/HIVE-20683?focusedWorklogId=305290&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-305290 ] ASF GitHub Bot logged work on HIVE-20683: - Author: ASF GitHub Bot Created on: 02/Sep/19 17:05 Start Date: 02/Sep/19 17:05 Worklog Time Spent: 10m Work Description: nishantmonu51 commented on issue #723: [HIVE-20683] Add the Ability to push Dynamic Between and Bloom filters to Druid URL: https://github.com/apache/hive/pull/723#issuecomment-527206460 @b-slim : Updated based on your comments, please check. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 305290) Time Spent: 2h 20m (was: 2h 10m) > Add the Ability to push Dynamic Between and Bloom filters to Druid > -- > > Key: HIVE-20683 > URL: https://issues.apache.org/jira/browse/HIVE-20683 > Project: Hive > Issue Type: New Feature > Components: Druid integration >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20683.1.patch, HIVE-20683.2.patch, > HIVE-20683.3.patch, HIVE-20683.4.patch, HIVE-20683.5.patch, > HIVE-20683.6.patch, HIVE-20683.8.patch, HIVE-20683.patch > > Time Spent: 2h 20m > Remaining Estimate: 0h > > For optimizing joins, Hive generates BETWEEN filter with min-max and BLOOM > filter for filtering one side of semi-join. > Druid 0.13.0 will have support for Bloom filters (Added via > https://github.com/apache/incubator-druid/pull/6222) > Implementation details - > # Hive generates and passes the filters as part of 'filterExpr' in TableScan. > # DruidQueryBasedRecordReader gets this filter passed as part of the conf. > # During execution phase, before sending the query to druid in > DruidQueryBasedRecordReader we will deserialize this filter, translate it > into a DruidDimFilter and add it to existing DruidQuery. Tez executor > already ensures that when we start reading results from the record reader, > all the dynamic values are initialized. > # Explaining a druid query also prints the query sent to druid as > {{druid.json.query}}. We also need to make sure to update the druid query > with the filters. During explain we do not have the actual values for the > dynamic values, so instead of values we will print the dynamic expression > itself as part of druid query. > Note:- This work needs druid to be updated to version 0.13.0 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22162) MVs are not using ACID tables.
[ https://issues.apache.org/jira/browse/HIVE-22162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-22162: -- Attachment: HIVE-22162.4.patch > MVs are not using ACID tables. > -- > > Key: HIVE-22162 > URL: https://issues.apache.org/jira/browse/HIVE-22162 > Project: Hive > Issue Type: Bug > Components: Materialized views >Affects Versions: 3.1.2 >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22162.1.patch, HIVE-22162.2.patch, > HIVE-22162.3.patch, HIVE-22162.4.patch > > > {code} > SET hive.support.concurrency=true; > SET hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; > SET metastore.strict.managed.tables=true; > SET hive.default.fileformat=textfile; > SET hive.default.fileformat.managed=orc; > SET metastore.create.as.acid=true; > CREATE TABLE cmv_basetable_n4 (a int, b varchar(256), c decimal(10,2)); > INSERT INTO cmv_basetable_n4 VALUES (1, 'alfred', 10.30),(2, 'bob', 3.14),(2, > 'bonnie', 172342.2),(3, 'calvin', 978.76),(3, 'charlie', 9.8); > CREATE MATERIALIZED VIEW cmv_mat_view_n4 disable rewrite > AS SELECT a, b, c FROM cmv_basetable_n4; > DESCRIBE FORMATTED cmv_mat_view_n4; > {code} > {code} > POSTHOOK: query: DESCRIBE FORMATTED cmv_mat_view_n4 > ... > Table Type: MATERIALIZED_VIEW > Table Parameters: > COLUMN_STATS_ACCURATE > {\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"a\":\"true\",\"b\":\"true\",\"c\":\"true\"}} > bucketing_version 2 > numFiles1 > numRows 5 > rawDataSize 1025 > totalSize 509 > {code} > Missing table parameter > {code} > transaction = true > {code} > cc.: [~ashutoshc], [~gopalv], [~jcamachorodriguez] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22162) MVs are not using ACID tables.
[ https://issues.apache.org/jira/browse/HIVE-22162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-22162: -- Status: Open (was: Patch Available) > MVs are not using ACID tables. > -- > > Key: HIVE-22162 > URL: https://issues.apache.org/jira/browse/HIVE-22162 > Project: Hive > Issue Type: Bug > Components: Materialized views >Affects Versions: 3.1.2 >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22162.1.patch, HIVE-22162.2.patch, > HIVE-22162.3.patch, HIVE-22162.4.patch > > > {code} > SET hive.support.concurrency=true; > SET hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; > SET metastore.strict.managed.tables=true; > SET hive.default.fileformat=textfile; > SET hive.default.fileformat.managed=orc; > SET metastore.create.as.acid=true; > CREATE TABLE cmv_basetable_n4 (a int, b varchar(256), c decimal(10,2)); > INSERT INTO cmv_basetable_n4 VALUES (1, 'alfred', 10.30),(2, 'bob', 3.14),(2, > 'bonnie', 172342.2),(3, 'calvin', 978.76),(3, 'charlie', 9.8); > CREATE MATERIALIZED VIEW cmv_mat_view_n4 disable rewrite > AS SELECT a, b, c FROM cmv_basetable_n4; > DESCRIBE FORMATTED cmv_mat_view_n4; > {code} > {code} > POSTHOOK: query: DESCRIBE FORMATTED cmv_mat_view_n4 > ... > Table Type: MATERIALIZED_VIEW > Table Parameters: > COLUMN_STATS_ACCURATE > {\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"a\":\"true\",\"b\":\"true\",\"c\":\"true\"}} > bucketing_version 2 > numFiles1 > numRows 5 > rawDataSize 1025 > totalSize 509 > {code} > Missing table parameter > {code} > transaction = true > {code} > cc.: [~ashutoshc], [~gopalv], [~jcamachorodriguez] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22162) MVs are not using ACID tables.
[ https://issues.apache.org/jira/browse/HIVE-22162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-22162: -- Status: Patch Available (was: Open) > MVs are not using ACID tables. > -- > > Key: HIVE-22162 > URL: https://issues.apache.org/jira/browse/HIVE-22162 > Project: Hive > Issue Type: Bug > Components: Materialized views >Affects Versions: 3.1.2 >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22162.1.patch, HIVE-22162.2.patch, > HIVE-22162.3.patch, HIVE-22162.4.patch > > > {code} > SET hive.support.concurrency=true; > SET hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; > SET metastore.strict.managed.tables=true; > SET hive.default.fileformat=textfile; > SET hive.default.fileformat.managed=orc; > SET metastore.create.as.acid=true; > CREATE TABLE cmv_basetable_n4 (a int, b varchar(256), c decimal(10,2)); > INSERT INTO cmv_basetable_n4 VALUES (1, 'alfred', 10.30),(2, 'bob', 3.14),(2, > 'bonnie', 172342.2),(3, 'calvin', 978.76),(3, 'charlie', 9.8); > CREATE MATERIALIZED VIEW cmv_mat_view_n4 disable rewrite > AS SELECT a, b, c FROM cmv_basetable_n4; > DESCRIBE FORMATTED cmv_mat_view_n4; > {code} > {code} > POSTHOOK: query: DESCRIBE FORMATTED cmv_mat_view_n4 > ... > Table Type: MATERIALIZED_VIEW > Table Parameters: > COLUMN_STATS_ACCURATE > {\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"a\":\"true\",\"b\":\"true\",\"c\":\"true\"}} > bucketing_version 2 > numFiles1 > numRows 5 > rawDataSize 1025 > totalSize 509 > {code} > Missing table parameter > {code} > transaction = true > {code} > cc.: [~ashutoshc], [~gopalv], [~jcamachorodriguez] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-22162) MVs are not using ACID tables.
[ https://issues.apache.org/jira/browse/HIVE-22162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16921199#comment-16921199 ] Hive QA commented on HIVE-22162: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 10s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 0s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 40s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 39s{color} | {color:blue} ql in master has 2248 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 17s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 48s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 22m 36s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-18436/dev-support/hive-personality.sh | | git revision | master / 04397e5 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-18436/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > MVs are not using ACID tables. > -- > > Key: HIVE-22162 > URL: https://issues.apache.org/jira/browse/HIVE-22162 > Project: Hive > Issue Type: Bug > Components: Materialized views >Affects Versions: 3.1.2 >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22162.1.patch, HIVE-22162.2.patch, > HIVE-22162.3.patch, HIVE-22162.4.patch > > > {code} > SET hive.support.concurrency=true; > SET hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; > SET metastore.strict.managed.tables=true; > SET hive.default.fileformat=textfile; > SET hive.default.fileformat.managed=orc; > SET metastore.create.as.acid=true; > CREATE TABLE cmv_basetable_n4 (a int, b varchar(256), c decimal(10,2)); > INSERT INTO cmv_basetable_n4 VALUES (1, 'alfred', 10.30),(2, 'bob', 3.14),(2, > 'bonnie', 172342.2),(3, 'calvin', 978.76),(3, 'charlie', 9.8); > CREATE MATERIALIZED VIEW cmv_mat_view_n4 disable rewrite > AS SELECT a, b, c FROM cmv_basetable_n4; > DESCRIBE FORMATTED cmv_mat_view_n4; > {code} > {code} > POSTHOOK: query: DESCRIBE FORMATTED cmv_mat_view_n4 > ... > Table Type: MATERIALIZED_VIEW > Table Parameters: > COLUMN_STATS_ACCURATE > {\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"a\":\"true\",\"b\":\"true\",\"c\":\"true\"}} > bucketing_version 2 > numFiles1 > numRows 5 > rawDataSize 1025 > totalSize