[jira] [Commented] (HIVE-20006) Make materializations invalidation cache work with multiple active remote metastores
[ https://issues.apache.org/jira/browse/HIVE-20006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17601241#comment-17601241 ] comet commented on HIVE-20006: -- apology, is MaterializationsCacheCleanerTask is replaced with another class on Hive 3.1 onward? I would like to set below property, but class not found ``` metastore.task.threads.always org.apache.hadoop.hive.metastore.events.EventCleanerTask,org.apache.hadoop.hive.metastore.MaterializationsCacheCleanerTask ``` > Make materializations invalidation cache work with multiple active remote > metastores > > > Key: HIVE-20006 > URL: https://issues.apache.org/jira/browse/HIVE-20006 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Critical > Fix For: 4.0.0, 4.0.0-alpha-1 > > Attachments: HIVE-19027.01.patch, HIVE-19027.02.patch, > HIVE-19027.03.patch, HIVE-19027.04.patch, HIVE-20006.01.patch, > HIVE-20006.02.patch, HIVE-20006.03.patch, HIVE-20006.04.patch, > HIVE-20006.05.patch, HIVE-20006.06.patch, HIVE-20006.07.patch, > HIVE-20006.patch > > > The main points: > - Only MVs that use transactional tables can have a time window value of 0. > Those are the only MVs that can be guaranteed to not be outdated when a query > is executed. > - For MVs that +cannot be outdated+, comparison is based on valid write id > lists. > - For MVs that +can be outdated+: > ** The window for valid outdated MVs can be specified in intervals of 1 > minute. > ** A materialized view is outdated if it was built before that time window > and any source table has been modified since. > A time window of -1 means to always use the materialized view for rewriting > without any checks concerning its validity. If a materialized view uses an > external table, the only way to trigger the rewriting would be to set the > property to -1, since currently we do not capture for validation purposes > whether the external source tables have been modified since the MV was > created or not. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-20006) Make materializations invalidation cache work with multiple active remote metastores
[ https://issues.apache.org/jira/browse/HIVE-20006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544043#comment-16544043 ] Hive QA commented on HIVE-20006: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12931576/HIVE-20006.07.patch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 14648 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/12600/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12600/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12600/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12931576 - PreCommit-HIVE-Build > Make materializations invalidation cache work with multiple active remote > metastores > > > Key: HIVE-20006 > URL: https://issues.apache.org/jira/browse/HIVE-20006 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Critical > Attachments: HIVE-19027.01.patch, HIVE-19027.02.patch, > HIVE-19027.03.patch, HIVE-19027.04.patch, HIVE-20006.01.patch, > HIVE-20006.02.patch, HIVE-20006.03.patch, HIVE-20006.04.patch, > HIVE-20006.05.patch, HIVE-20006.06.patch, HIVE-20006.07.patch, > HIVE-20006.patch > > > The main points: > - Only MVs that use transactional tables and are stored in transactional > tables can have a time window value of 0. Those are the only MVs that can be > guaranteed to not be outdated when a query is executed. > - For MVs that +cannot be outdated+, comparison is based on valid write id > lists. > - For MVs that +can be outdated+: > ** The window for valid outdated MVs can be specified in intervals of 1 > minute. > ** A materialized view is outdated if it was built before that time window > and any source table has been modified since. > A time window of -1 means to always use the materialized view for rewriting > without any checks concerning its validity. If a materialized view uses an > external table, the only way to trigger the rewriting would be to set the > property to -1, since currently we do not capture for validation purposes > whether the external source tables have been modified since the MV was > created or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20006) Make materializations invalidation cache work with multiple active remote metastores
[ https://issues.apache.org/jira/browse/HIVE-20006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544022#comment-16544022 ] Hive QA commented on HIVE-20006: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 1s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 21s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 30s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 25s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 26s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 30s{color} | {color:blue} common in master has 64 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m 53s{color} | {color:blue} standalone-metastore/metastore-common in master has 217 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 45s{color} | {color:blue} ql in master has 2289 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m 9s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 21s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 21s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 23s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 23s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 15s{color} | {color:red} common: The patch generated 1 new + 422 unchanged - 5 fixed = 423 total (was 427) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 40s{color} | {color:red} ql: The patch generated 4 new + 380 unchanged - 4 fixed = 384 total (was 384) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 26s{color} | {color:red} root: The patch generated 1 new + 422 unchanged - 5 fixed = 423 total (was 427) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 6 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m 3s{color} | {color:red} standalone-metastore/metastore-common generated 3 new + 215 unchanged - 2 fixed = 218 total (was 217) {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m 54s{color} | {color:red} ql generated 1 new + 2288 unchanged - 1 fixed = 2289 total (was 2289) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s{color} | {color:green} common in the patch passed. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s{color} | {color:green} standalone-metastore_metastore-common generated 0 new + 53 unchanged - 1 fixed = 53 total (was 54) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s{color} | {color:green} ql in the patch passed. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m 6s{color} | {color:green} root generated 0 new + 370 unchanged - 1 fixed = 370 total (was 371) {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 12s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 66m 59s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:standalone-metastore/metastore-common | | | org.apache.hadoop.hive.metastore.txn.TxnHandler.lockMaterializationRebuild(String, String, long) passes a nonconstant String to an execute or addBatch method on an SQL statement At TxnHandler.java:nonconstant String to an execute or addBatch method on an SQL statement At TxnHandler.java:[line 1785] | | |
[jira] [Commented] (HIVE-20006) Make materializations invalidation cache work with multiple active remote metastores
[ https://issues.apache.org/jira/browse/HIVE-20006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16542443#comment-16542443 ] Hive QA commented on HIVE-20006: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12931331/HIVE-20006.06.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/12571/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12571/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12571/ Messages: {noformat} This message was trimmed, see log for full details Removing standalone-metastore/src/ + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at 20eb7b5 HIVE-20097 : Convert standalone-metastore to a submodule (Alexander Kolbasov reviewed by Vihang Karajgaonkar) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2018-07-13 02:54:18.459 + rm -rf ../yetus_PreCommit-HIVE-Build-12571 + mkdir ../yetus_PreCommit-HIVE-Build-12571 + git gc + cp -R . ../yetus_PreCommit-HIVE-Build-12571 + mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-12571/yetus + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch error: a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/MaterializedViewTask.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java: does not exist in index error: a/ql/src/test/queries/clientpositive/materialized_view_create_rewrite_time_window.q: does not exist in index error: a/ql/src/test/results/clientpositive/druid/druidmini_mv.q.out: does not exist in index error: a/ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite_5.q.out: does not exist in index error: a/ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite_time_window.q.out: does not exist in index error: a/ql/src/test/results/clientpositive/llap/materialized_view_rewrite_empty.q.out: does not exist in index error: a/standalone-metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp: does not exist in index error: a/standalone-metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h: does not exist in index error: a/standalone-metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp: does not exist in index error: a/standalone-metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp: does not exist in index error: a/standalone-metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h: does not exist in index error: a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CreationMetadata.java: does not exist in index error: a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FindSchemasByColsResp.java: does not exist in index error: a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Materialization.java: does not exist in index error: a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SchemaVersion.java: does not exist in index error: a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java: does not exist in index error: a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/WMFullResourcePlan.java: does not exist in index error: a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/WMGetAllResourcePlanResponse.java: does not exist in index error: a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/WMGetTriggersForResourePlanResponse.java: does not exist in index error: a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/WMValidateResourcePlanResponse.java: does not exist in index error: a/standalone-metastore/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php: does not exist in index error: a/standalone-metastore/src/gen/thrift/gen-php/metastore/Types.php: does not exist in index error: a/standalone-metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote: does not exist in index error: a/standalone-metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py: does not exist in index error:
[jira] [Commented] (HIVE-20006) Make materializations invalidation cache work with multiple active remote metastores
[ https://issues.apache.org/jira/browse/HIVE-20006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16542060#comment-16542060 ] Jesus Camacho Rodriguez commented on HIVE-20006: [~ashutoshc], done. Updated HIVE-19027 too. > Make materializations invalidation cache work with multiple active remote > metastores > > > Key: HIVE-20006 > URL: https://issues.apache.org/jira/browse/HIVE-20006 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Critical > Attachments: HIVE-19027.01.patch, HIVE-19027.02.patch, > HIVE-19027.03.patch, HIVE-19027.04.patch, HIVE-20006.01.patch, > HIVE-20006.02.patch, HIVE-20006.03.patch, HIVE-20006.04.patch, > HIVE-20006.05.patch, HIVE-20006.06.patch, HIVE-20006.patch > > > The main points: > - Only MVs that use transactional tables and are stored in transactional > tables can have a time window value of 0. Those are the only MVs that can be > guaranteed to not be outdated when a query is executed. > - For MVs that +cannot be outdated+, comparison is based on valid write id > lists. > - For MVs that +can be outdated+: > ** The window for valid outdated MVs can be specified in intervals of 1 > minute. > ** A materialized view is outdated if it was built before that time window > and any source table has been modified since. > A time window of -1 means to always use the materialized view for rewriting > without any checks concerning its validity. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20006) Make materializations invalidation cache work with multiple active remote metastores
[ https://issues.apache.org/jira/browse/HIVE-20006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541994#comment-16541994 ] Ashutosh Chauhan commented on HIVE-20006: - [~jcamachorodriguez] Does the description still accurately reflects the changes in the patch? If not, can you please update it. > Make materializations invalidation cache work with multiple active remote > metastores > > > Key: HIVE-20006 > URL: https://issues.apache.org/jira/browse/HIVE-20006 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Critical > Attachments: HIVE-19027.01.patch, HIVE-19027.02.patch, > HIVE-19027.03.patch, HIVE-19027.04.patch, HIVE-20006.01.patch, > HIVE-20006.02.patch, HIVE-20006.03.patch, HIVE-20006.04.patch, > HIVE-20006.05.patch, HIVE-20006.06.patch, HIVE-20006.patch > > > The main points: > - Only MVs stored in transactional tables can have a time window value of 0. > Those are the only MVs that can be guaranteed to not be outdated when a query > is executed, if we use custom storage handlers to store the materialized > view, we cannot make any promises. > - For MVs that +cannot be outdated+, we do not check the metastore. Instead, > comparison is based on valid write id lists. > - For MVs that +can be outdated+, we still rely on the invalidation cache. > ** The window for valid outdated MVs can be specified in intervals of 1 > minute (less than that, it is difficult to have any guarantees about whether > the MV is actually outdated by less than a minute or not). > ** The async loading is done every interval / 2 (or probably better, we can > make it configurable). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20006) Make materializations invalidation cache work with multiple active remote metastores
[ https://issues.apache.org/jira/browse/HIVE-20006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541402#comment-16541402 ] Hive QA commented on HIVE-20006: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 28s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 53s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 40s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m 40s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 32s{color} | {color:blue} common in master has 64 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m 52s{color} | {color:blue} standalone-metastore in master has 217 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 45s{color} | {color:blue} ql in master has 2287 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m 27s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 21s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 51s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 33s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 14s{color} | {color:red} common: The patch generated 1 new + 422 unchanged - 5 fixed = 423 total (was 427) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 36s{color} | {color:red} standalone-metastore: The patch generated 9 new + 1959 unchanged - 13 fixed = 1968 total (was 1972) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 38s{color} | {color:red} ql: The patch generated 4 new + 380 unchanged - 4 fixed = 384 total (was 384) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m 20s{color} | {color:red} root: The patch generated 14 new + 2761 unchanged - 22 fixed = 2775 total (was 2783) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 6 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m 3s{color} | {color:red} standalone-metastore generated 3 new + 215 unchanged - 2 fixed = 218 total (was 217) {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m 52s{color} | {color:red} ql generated 1 new + 2286 unchanged - 1 fixed = 2287 total (was 2287) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s{color} | {color:green} common in the patch passed. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s{color} | {color:green} standalone-metastore generated 0 new + 53 unchanged - 1 fixed = 53 total (was 54) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s{color} | {color:green} ql in the patch passed. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m 39s{color} | {color:green} root generated 0 new + 370 unchanged - 1 fixed = 370 total (was 371) {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 21s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 68m 47s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:standalone-metastore | | | org.apache.hadoop.hive.metastore.txn.TxnHandler.lockMaterializationRebuild(String, String, long) passes a nonconstant String to an execute or addBatch method on an SQL statement At
[jira] [Commented] (HIVE-20006) Make materializations invalidation cache work with multiple active remote metastores
[ https://issues.apache.org/jira/browse/HIVE-20006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541396#comment-16541396 ] Hive QA commented on HIVE-20006: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12931179/HIVE-20006.05.patch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 14647 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druid_timestamptz] (batchId=192) org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_joins] (batchId=192) org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_masking] (batchId=192) org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query29] (batchId=262) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/12548/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12548/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12548/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 4 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12931179 - PreCommit-HIVE-Build > Make materializations invalidation cache work with multiple active remote > metastores > > > Key: HIVE-20006 > URL: https://issues.apache.org/jira/browse/HIVE-20006 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Critical > Attachments: HIVE-19027.01.patch, HIVE-19027.02.patch, > HIVE-19027.03.patch, HIVE-19027.04.patch, HIVE-20006.01.patch, > HIVE-20006.02.patch, HIVE-20006.03.patch, HIVE-20006.04.patch, > HIVE-20006.05.patch, HIVE-20006.patch > > > The main points: > - Only MVs stored in transactional tables can have a time window value of 0. > Those are the only MVs that can be guaranteed to not be outdated when a query > is executed, if we use custom storage handlers to store the materialized > view, we cannot make any promises. > - For MVs that +cannot be outdated+, we do not check the metastore. Instead, > comparison is based on valid write id lists. > - For MVs that +can be outdated+, we still rely on the invalidation cache. > ** The window for valid outdated MVs can be specified in intervals of 1 > minute (less than that, it is difficult to have any guarantees about whether > the MV is actually outdated by less than a minute or not). > ** The async loading is done every interval / 2 (or probably better, we can > make it configurable). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20006) Make materializations invalidation cache work with multiple active remote metastores
[ https://issues.apache.org/jira/browse/HIVE-20006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16536461#comment-16536461 ] Hive QA commented on HIVE-20006: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 46s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m 30s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 53s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m 8s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 51s{color} | {color:blue} common in master has 64 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m 39s{color} | {color:blue} standalone-metastore in master has 217 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 5m 9s{color} | {color:blue} ql in master has 2287 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m 7s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 20s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 27s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 15s{color} | {color:red} common: The patch generated 1 new + 422 unchanged - 5 fixed = 423 total (was 427) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 42s{color} | {color:red} standalone-metastore: The patch generated 9 new + 1959 unchanged - 13 fixed = 1968 total (was 1972) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 43s{color} | {color:red} ql: The patch generated 4 new + 380 unchanged - 4 fixed = 384 total (was 384) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m 32s{color} | {color:red} root: The patch generated 14 new + 2761 unchanged - 22 fixed = 2775 total (was 2783) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 6 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m 26s{color} | {color:red} standalone-metastore generated 3 new + 215 unchanged - 2 fixed = 218 total (was 217) {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 14s{color} | {color:red} ql generated 1 new + 2286 unchanged - 1 fixed = 2287 total (was 2287) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s{color} | {color:green} common in the patch passed. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s{color} | {color:green} standalone-metastore generated 0 new + 53 unchanged - 1 fixed = 53 total (was 54) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s{color} | {color:green} ql in the patch passed. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m 24s{color} | {color:green} root generated 0 new + 370 unchanged - 1 fixed = 370 total (was 371) {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 12s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 85m 25s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:standalone-metastore | | | org.apache.hadoop.hive.metastore.txn.TxnHandler.lockMaterializationRebuild(String, String, long) passes a nonconstant String to an execute or addBatch method on an SQL statement At
[jira] [Commented] (HIVE-20006) Make materializations invalidation cache work with multiple active remote metastores
[ https://issues.apache.org/jira/browse/HIVE-20006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16536459#comment-16536459 ] Hive QA commented on HIVE-20006: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12930765/HIVE-20006.04.patch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 14629 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druid_timestamptz] (batchId=191) org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_joins] (batchId=191) org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_masking] (batchId=191) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/12478/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12478/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12478/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12930765 - PreCommit-HIVE-Build > Make materializations invalidation cache work with multiple active remote > metastores > > > Key: HIVE-20006 > URL: https://issues.apache.org/jira/browse/HIVE-20006 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Critical > Attachments: HIVE-19027.01.patch, HIVE-19027.02.patch, > HIVE-19027.03.patch, HIVE-19027.04.patch, HIVE-20006.01.patch, > HIVE-20006.02.patch, HIVE-20006.03.patch, HIVE-20006.04.patch, > HIVE-20006.patch > > > The main points: > - Only MVs stored in transactional tables can have a time window value of 0. > Those are the only MVs that can be guaranteed to not be outdated when a query > is executed, if we use custom storage handlers to store the materialized > view, we cannot make any promises. > - For MVs that +cannot be outdated+, we do not check the metastore. Instead, > comparison is based on valid write id lists. > - For MVs that +can be outdated+, we still rely on the invalidation cache. > ** The window for valid outdated MVs can be specified in intervals of 1 > minute (less than that, it is difficult to have any guarantees about whether > the MV is actually outdated by less than a minute or not). > ** The async loading is done every interval / 2 (or probably better, we can > make it configurable). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20006) Make materializations invalidation cache work with multiple active remote metastores
[ https://issues.apache.org/jira/browse/HIVE-20006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16536001#comment-16536001 ] Hive QA commented on HIVE-20006: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 31s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 16s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 12s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 4s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 32s{color} | {color:blue} common in master has 64 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 7s{color} | {color:blue} standalone-metastore in master has 217 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 55s{color} | {color:blue} ql in master has 2287 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m 49s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 22s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 15s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 0s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 14s{color} | {color:red} common: The patch generated 1 new + 422 unchanged - 5 fixed = 423 total (was 427) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 38s{color} | {color:red} standalone-metastore: The patch generated 9 new + 1959 unchanged - 13 fixed = 1968 total (was 1972) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 42s{color} | {color:red} ql: The patch generated 4 new + 380 unchanged - 4 fixed = 384 total (was 384) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m 30s{color} | {color:red} root: The patch generated 14 new + 2761 unchanged - 22 fixed = 2775 total (was 2783) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 6 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m 19s{color} | {color:red} standalone-metastore generated 3 new + 215 unchanged - 2 fixed = 218 total (was 217) {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 1s{color} | {color:red} ql generated 1 new + 2286 unchanged - 1 fixed = 2287 total (was 2287) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s{color} | {color:green} common in the patch passed. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s{color} | {color:green} standalone-metastore generated 0 new + 53 unchanged - 1 fixed = 53 total (was 54) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s{color} | {color:green} ql in the patch passed. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m 19s{color} | {color:green} root generated 0 new + 370 unchanged - 1 fixed = 370 total (was 371) {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 22s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 72m 27s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:standalone-metastore | | | org.apache.hadoop.hive.metastore.txn.TxnHandler.lockMaterializationRebuild(String, String, long) passes a nonconstant String to an execute or addBatch method on an SQL statement At
[jira] [Commented] (HIVE-20006) Make materializations invalidation cache work with multiple active remote metastores
[ https://issues.apache.org/jira/browse/HIVE-20006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16536000#comment-16536000 ] Hive QA commented on HIVE-20006: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12930635/HIVE-20006.03.patch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 14629 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druid_timestamptz] (batchId=191) org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_joins] (batchId=191) org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_masking] (batchId=191) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/12462/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12462/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12462/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12930635 - PreCommit-HIVE-Build > Make materializations invalidation cache work with multiple active remote > metastores > > > Key: HIVE-20006 > URL: https://issues.apache.org/jira/browse/HIVE-20006 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Critical > Attachments: HIVE-19027.01.patch, HIVE-19027.02.patch, > HIVE-19027.03.patch, HIVE-19027.04.patch, HIVE-20006.01.patch, > HIVE-20006.02.patch, HIVE-20006.03.patch, HIVE-20006.patch > > > The main points: > - Only MVs stored in transactional tables can have a time window value of 0. > Those are the only MVs that can be guaranteed to not be outdated when a query > is executed, if we use custom storage handlers to store the materialized > view, we cannot make any promises. > - For MVs that +cannot be outdated+, we do not check the metastore. Instead, > comparison is based on valid write id lists. > - For MVs that +can be outdated+, we still rely on the invalidation cache. > ** The window for valid outdated MVs can be specified in intervals of 1 > minute (less than that, it is difficult to have any guarantees about whether > the MV is actually outdated by less than a minute or not). > ** The async loading is done every interval / 2 (or probably better, we can > make it configurable). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20006) Make materializations invalidation cache work with multiple active remote metastores
[ https://issues.apache.org/jira/browse/HIVE-20006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530578#comment-16530578 ] Hive QA commented on HIVE-20006: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 1s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 39s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 13s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 7s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m 35s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 31s{color} | {color:blue} common in master has 64 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m 58s{color} | {color:blue} standalone-metastore in master has 228 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m 4s{color} | {color:blue} ql in master has 2286 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m 25s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 23s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 8s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 8s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 15s{color} | {color:red} common: The patch generated 1 new + 422 unchanged - 5 fixed = 423 total (was 427) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 35s{color} | {color:red} standalone-metastore: The patch generated 9 new + 1964 unchanged - 13 fixed = 1973 total (was 1977) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 43s{color} | {color:red} ql: The patch generated 4 new + 380 unchanged - 4 fixed = 384 total (was 384) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m 21s{color} | {color:red} root: The patch generated 14 new + 2766 unchanged - 22 fixed = 2780 total (was 2788) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 6 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m 26s{color} | {color:red} standalone-metastore generated 4 new + 226 unchanged - 2 fixed = 230 total (was 228) {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 7s{color} | {color:red} ql generated 1 new + 2285 unchanged - 1 fixed = 2286 total (was 2286) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s{color} | {color:green} common in the patch passed. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s{color} | {color:green} standalone-metastore generated 0 new + 53 unchanged - 1 fixed = 53 total (was 54) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s{color} | {color:green} ql in the patch passed. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 9m 4s{color} | {color:green} root generated 0 new + 370 unchanged - 1 fixed = 370 total (was 371) {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 32s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 76m 7s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:standalone-metastore | | | Dead store to numCompsWritten in org.apache.hadoop.hive.metastore.txn.TxnHandler.commitTxn(CommitTxnRequest) At
[jira] [Commented] (HIVE-20006) Make materializations invalidation cache work with multiple active remote metastores
[ https://issues.apache.org/jira/browse/HIVE-20006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530572#comment-16530572 ] Hive QA commented on HIVE-20006: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12929966/HIVE-20006.02.patch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 14619 tests executed *Failed tests:* {noformat} TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=190) [druidmini_dynamic_partition.q,druidmini_expressions.q,druidmini_test_alter.q,druidmini_test1.q,druidmini_test_insert.q] TestReplicationScenariosAcrossInstances - did not produce a TEST-*.xml file (likely timed out) (batchId=238) org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druid_timestamptz] (batchId=191) org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_joins] (batchId=191) org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_masking] (batchId=191) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/12334/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12334/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12334/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 5 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12929966 - PreCommit-HIVE-Build > Make materializations invalidation cache work with multiple active remote > metastores > > > Key: HIVE-20006 > URL: https://issues.apache.org/jira/browse/HIVE-20006 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Critical > Attachments: HIVE-19027.01.patch, HIVE-19027.02.patch, > HIVE-19027.03.patch, HIVE-19027.04.patch, HIVE-20006.01.patch, > HIVE-20006.02.patch, HIVE-20006.patch > > > The main points: > - Only MVs stored in transactional tables can have a time window value of 0. > Those are the only MVs that can be guaranteed to not be outdated when a query > is executed, if we use custom storage handlers to store the materialized > view, we cannot make any promises. > - For MVs that +cannot be outdated+, we do not check the metastore. Instead, > comparison is based on valid write id lists. > - For MVs that +can be outdated+, we still rely on the invalidation cache. > ** The window for valid outdated MVs can be specified in intervals of 1 > minute (less than that, it is difficult to have any guarantees about whether > the MV is actually outdated by less than a minute or not). > ** The async loading is done every interval / 2 (or probably better, we can > make it configurable). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20006) Make materializations invalidation cache work with multiple active remote metastores
[ https://issues.apache.org/jira/browse/HIVE-20006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526202#comment-16526202 ] Hive QA commented on HIVE-20006: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 34s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 54s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 47s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m 48s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 29s{color} | {color:blue} common in master has 62 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m 54s{color} | {color:blue} standalone-metastore in master has 228 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 41s{color} | {color:blue} ql in master has 2280 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m 44s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 20s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 58s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 39s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 16s{color} | {color:red} common: The patch generated 1 new + 422 unchanged - 5 fixed = 423 total (was 427) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 38s{color} | {color:red} standalone-metastore: The patch generated 9 new + 1962 unchanged - 13 fixed = 1971 total (was 1975) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 40s{color} | {color:red} ql: The patch generated 4 new + 380 unchanged - 4 fixed = 384 total (was 384) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m 18s{color} | {color:red} root: The patch generated 14 new + 2764 unchanged - 22 fixed = 2778 total (was 2786) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 6 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m 2s{color} | {color:red} standalone-metastore generated 4 new + 226 unchanged - 2 fixed = 230 total (was 228) {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m 50s{color} | {color:red} ql generated 1 new + 2279 unchanged - 1 fixed = 2280 total (was 2280) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s{color} | {color:green} common in the patch passed. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 59s{color} | {color:green} standalone-metastore generated 0 new + 53 unchanged - 1 fixed = 53 total (was 54) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s{color} | {color:green} ql in the patch passed. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 39s{color} | {color:green} root generated 0 new + 368 unchanged - 1 fixed = 368 total (was 369) {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 10s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 68m 32s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:standalone-metastore | | | Dead store to numCompsWritten in org.apache.hadoop.hive.metastore.txn.TxnHandler.commitTxn(CommitTxnRequest) At
[jira] [Commented] (HIVE-20006) Make materializations invalidation cache work with multiple active remote metastores
[ https://issues.apache.org/jira/browse/HIVE-20006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526200#comment-16526200 ] Hive QA commented on HIVE-20006: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12929351/HIVE-20006.patch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 14587 tests executed *Failed tests:* {noformat} TestReplicationScenariosAcrossInstances - did not produce a TEST-*.xml file (likely timed out) (batchId=238) org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druid_timestamptz] (batchId=191) org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_joins] (batchId=191) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/12208/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12208/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12208/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12929351 - PreCommit-HIVE-Build > Make materializations invalidation cache work with multiple active remote > metastores > > > Key: HIVE-20006 > URL: https://issues.apache.org/jira/browse/HIVE-20006 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Critical > Attachments: HIVE-19027.01.patch, HIVE-19027.02.patch, > HIVE-19027.03.patch, HIVE-19027.04.patch, HIVE-20006.patch > > > The main points: > - Only MVs stored in transactional tables can have a time window value of 0. > Those are the only MVs that can be guaranteed to not be outdated when a query > is executed, if we use custom storage handlers to store the materialized > view, we cannot make any promises. > - For MVs that +cannot be outdated+, we do not check the metastore. Instead, > comparison is based on valid write id lists. > - For MVs that +can be outdated+, we still rely on the invalidation cache. > ** The window for valid outdated MVs can be specified in intervals of 1 > minute (less than that, it is difficult to have any guarantees about whether > the MV is actually outdated by less than a minute or not). > ** The async loading is done every interval / 2 (or probably better, we can > make it configurable). -- This message was sent by Atlassian JIRA (v7.6.3#76005)