[jira] [Updated] (HIVE-21928) Fix for statistics annotation in nested AND expressions
[ https://issues.apache.org/jira/browse/HIVE-21928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-21928: --- Attachment: HIVE-21928.patch > Fix for statistics annotation in nested AND expressions > --- > > Key: HIVE-21928 > URL: https://issues.apache.org/jira/browse/HIVE-21928 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-21928.patch > > > Discovered while working on HIVE-21867. Having predicates with nested AND > expressions may result in different stats, even if predicates are basically > similar (from stats estimation standpoint). > For instance, stats for {{AND(x=5, true, true)}} are different from {{x=5}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21928) Fix for statistics annotation in nested AND expressions
[ https://issues.apache.org/jira/browse/HIVE-21928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-21928: --- Status: Patch Available (was: In Progress) > Fix for statistics annotation in nested AND expressions > --- > > Key: HIVE-21928 > URL: https://issues.apache.org/jira/browse/HIVE-21928 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > > Discovered while working on HIVE-21867. Having predicates with nested AND > expressions may result in different stats, even if predicates are basically > similar (from stats estimation standpoint). > For instance, stats for {{AND(x=5, true, true)}} are different from {{x=5}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21928) Fix for statistics annotation in nested AND expressions
[ https://issues.apache.org/jira/browse/HIVE-21928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-21928: -- > Fix for statistics annotation in nested AND expressions > --- > > Key: HIVE-21928 > URL: https://issues.apache.org/jira/browse/HIVE-21928 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > > Discovered while working on HIVE-21867. Having predicates with nested AND > expressions may result in different stats, even if predicates are basically > similar (from stats estimation standpoint). > For instance, stats for {{AND(x=5, true, true)}} are different from {{x=5}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-21928) Fix for statistics annotation in nested AND expressions
[ https://issues.apache.org/jira/browse/HIVE-21928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-21928 started by Jesus Camacho Rodriguez. -- > Fix for statistics annotation in nested AND expressions > --- > > Key: HIVE-21928 > URL: https://issues.apache.org/jira/browse/HIVE-21928 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > > Discovered while working on HIVE-21867. Having predicates with nested AND > expressions may result in different stats, even if predicates are basically > similar (from stats estimation standpoint). > For instance, stats for {{AND(x=5, true, true)}} are different from {{x=5}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[ https://issues.apache.org/jira/browse/HIVE-21886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873853#comment-16873853 ] Hive QA commented on HIVE-21886: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 43s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 9s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 46s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 58s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m 7s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 44s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 26s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 29s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 49s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 16s{color} | {color:red} itests/hive-unit: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 5s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 26s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 30m 42s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-17766/dev-support/hive-personality.sh | | git revision | master / 22a5e21 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-17766/yetus/diff-checkstyle-itests_hive-unit.txt | | modules | C: ql itests/hive-unit U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-17766/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > REPL - With table list - Handle rename events during replace policy > --- > > Key: HIVE-21886 > URL: https://issues.apache.org/jira/browse/HIVE-21886 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch, > HIVE-21886.03.patch > > Time Spent: 11h 10m > Remaining Estimate: 0h > > If some rename events are found to be dumped and replayed while replace > policy is getting executed, it needs to take care of the policy inclusion in > both the policy for each table name. > 1. Create a list of tables to be bootstrapped. > 2. During handling of alter table, if the alter type is rename > 1. If the old table name is present in the list of table to be > bo
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[ https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268272&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268272 ] ASF GitHub Bot logged work on HIVE-21886: - Author: ASF GitHub Bot Created on: 27/Jun/19 06:12 Start Date: 27/Jun/19 06:12 Worklog Time Spent: 10m Work Description: sankarh commented on pull request #688: HIVE-21886 : REPL - With table list - Handle rename events during replace policy URL: https://github.com/apache/hive/pull/688#discussion_r298020420 ## File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java ## @@ -118,25 +147,25 @@ private boolean handleForTableLevelReplication(Context withinContext) { // If the old table was in the list of tables to be bootstrapped which is a multi rename case, the old table // is removed from the list of tables to be bootstrapped and new one is added. - if (oldTableIsPresent) { + if (oldTableInBootstrapList) { withinContext.addToListOfTablesForBootstrap(newName); return false; } - // If both old and new table satisfies the filter and old table is present at target, then dump the rename event. - LOG.info("both old and new table satisfies the filter"); + // If both old and new table satisfies the policy and old table is present at target, then dump the rename event. + LOG.info("both old and new table satisfies the policy"); return true; } else { - // if the old table does not satisfies the filter, but the new one satisfies, then the new table should be + // if the old table does not satisfies the policy, but the new one satisfies, then the new table should be // added to the list of tables to be bootstrapped and don't dump the event. if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, newName)) { LOG.info("Table " + newName + " is added for bootstrap " + " during rename from " + oldName); withinContext.addToListOfTablesForBootstrap(newName); return false; } - // if both old and new table does not satisfies the filter, then don't dump the event. - LOG.info("both old and new table not satisfies the filter"); + // if both old and new table does not satisfies the policy, then don't dump the event. + LOG.info("both old and new table not satisfies the policy"); Review comment: Shall log table names also in the log message. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268272) Time Spent: 11h 10m (was: 11h) > REPL - With table list - Handle rename events during replace policy > --- > > Key: HIVE-21886 > URL: https://issues.apache.org/jira/browse/HIVE-21886 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch, > HIVE-21886.03.patch > > Time Spent: 11h 10m > Remaining Estimate: 0h > > If some rename events are found to be dumped and replayed while replace > policy is getting executed, it needs to take care of the policy inclusion in > both the policy for each table name. > 1. Create a list of tables to be bootstrapped. > 2. During handling of alter table, if the alter type is rename > 1. If the old table name is present in the list of table to be > bootstrapped, remove it. > 2. If the new table name, matches the new policy, add it to the list > of tables to be bootstrapped. > 3. If the old table does not match the old policy drop it, even if the > table is not present at target. > 3. During handling of drop table > 1. if the table is in the list of tables to be bootstrapped, then > remove it and ignore the event. > 4. During other event handling > 1. if the table is there in the list of tables to be bootstrapped, > then ignore the event. > 2. If the new policy does not match the table name, then ignore the > event. > > Rename handling during replace policy > # Old name not matching old policy – The old table will not be there at the > target cluster. The table will not be returned by get-all-table. > ## Old name is not matching new policy > ### New name not matching old policy > ##
[jira] [Updated] (HIVE-21927) HiveServer Web UI: Setting the HttpOnly option in the cookies
[ https://issues.apache.org/jira/browse/HIVE-21927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh updated HIVE-21927: -- Attachment: HIVE-21927.patch Status: Patch Available (was: In Progress) > HiveServer Web UI: Setting the HttpOnly option in the cookies > - > > Key: HIVE-21927 > URL: https://issues.apache.org/jira/browse/HIVE-21927 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.1.1 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-21927.patch > > > Intend of this JIRA is to introduce the HttpOnly option in the cookie. > cookie: before change > {code:java} > hdp32bFALSE / FALSE 0 JSESSIONID > 8dkibwayfnrc4y4hvpu3vh74 > {code} > after change: > {code:java} > #HttpOnly_hdp32b FALSE / FALSE 0 JSESSIONID > e1npdkbo3inj1xnd6gdc6ihws > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-21927) HiveServer Web UI: Setting the HttpOnly option in the cookies
[ https://issues.apache.org/jira/browse/HIVE-21927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-21927 started by Rajkumar Singh. - > HiveServer Web UI: Setting the HttpOnly option in the cookies > - > > Key: HIVE-21927 > URL: https://issues.apache.org/jira/browse/HIVE-21927 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.1.1 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Major > > Intend of this JIRA is to introduce the HttpOnly option in the cookie. > cookie: before change > {code:java} > hdp32bFALSE / FALSE 0 JSESSIONID > 8dkibwayfnrc4y4hvpu3vh74 > {code} > after change: > {code:java} > #HttpOnly_hdp32b FALSE / FALSE 0 JSESSIONID > e1npdkbo3inj1xnd6gdc6ihws > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21927) HiveServer Web UI: Setting the HttpOnly option in the cookies
[ https://issues.apache.org/jira/browse/HIVE-21927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh reassigned HIVE-21927: - > HiveServer Web UI: Setting the HttpOnly option in the cookies > - > > Key: HIVE-21927 > URL: https://issues.apache.org/jira/browse/HIVE-21927 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.1.1 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Major > > Intend of this JIRA is to introduce the HttpOnly option in the cookie. > cookie: before change > {code:java} > hdp32bFALSE / FALSE 0 JSESSIONID > 8dkibwayfnrc4y4hvpu3vh74 > {code} > after change: > {code:java} > #HttpOnly_hdp32b FALSE / FALSE 0 JSESSIONID > e1npdkbo3inj1xnd6gdc6ihws > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[ https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268262&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268262 ] ASF GitHub Bot logged work on HIVE-21886: - Author: ASF GitHub Bot Created on: 27/Jun/19 05:52 Start Date: 27/Jun/19 05:52 Worklog Time Spent: 10m Work Description: maheshk114 commented on pull request #688: HIVE-21886 : REPL - With table list - Handle rename events during replace policy URL: https://github.com/apache/hive/pull/688#discussion_r298015828 ## File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java ## @@ -92,21 +93,49 @@ private Scenario scenarioType(org.apache.hadoop.hive.metastore.api.Table before, } } - // return true, if event needs to be dumped, else return false. - private boolean handleForTableLevelReplication(Context withinContext) { -String oldName = before.getTableName(); -String newName = after.getTableName(); + // Return true, if event needs to be dumped, else return false. + private boolean handleRenameForReplacePolicy(Context withinContext, String oldName, String newName) { +// If the table is renamed after being added to the list of tables to be bootstrapped, then remove it from the Review comment: with the addition if withinContext.oldReplScope != null ..before calling ..the assert is not required This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268262) Time Spent: 11h (was: 10h 50m) > REPL - With table list - Handle rename events during replace policy > --- > > Key: HIVE-21886 > URL: https://issues.apache.org/jira/browse/HIVE-21886 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch, > HIVE-21886.03.patch > > Time Spent: 11h > Remaining Estimate: 0h > > If some rename events are found to be dumped and replayed while replace > policy is getting executed, it needs to take care of the policy inclusion in > both the policy for each table name. > 1. Create a list of tables to be bootstrapped. > 2. During handling of alter table, if the alter type is rename > 1. If the old table name is present in the list of table to be > bootstrapped, remove it. > 2. If the new table name, matches the new policy, add it to the list > of tables to be bootstrapped. > 3. If the old table does not match the old policy drop it, even if the > table is not present at target. > 3. During handling of drop table > 1. if the table is in the list of tables to be bootstrapped, then > remove it and ignore the event. > 4. During other event handling > 1. if the table is there in the list of tables to be bootstrapped, > then ignore the event. > 2. If the new policy does not match the table name, then ignore the > event. > > Rename handling during replace policy > # Old name not matching old policy – The old table will not be there at the > target cluster. The table will not be returned by get-all-table. > ## Old name is not matching new policy > ### New name not matching old policy > New name not matching new policy > * Ignore the event, no need to do anything. > New name matching new policy > * The table will be returned by get-all-table. Replace policy handler > will bootstrap this table as its matching new policy and not matching old > policy. > * All the future events will be ignored as part of check added by > replace policy handling. > * All the event with old table name will anyways be ignored as the old > name is not matching the new policy. > ### New name matching old policy > New name not matching new policy > * As the new name is not matching the new policy, the table need not be > replicated. > * As the old name is not matching the new policy, the rename events will > be ignored. > * So nothing to be done for this scenario. > New name matching new policy > * As the new name is matching both old and new policy, replace handler > will not bootstrap the table. > * Add the table to the list of tables to be bootstrapped. > * Ignore all the events with new name. > * If there is a dr
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[ https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268259&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268259 ] ASF GitHub Bot logged work on HIVE-21886: - Author: ASF GitHub Bot Created on: 27/Jun/19 05:49 Start Date: 27/Jun/19 05:49 Worklog Time Spent: 10m Work Description: maheshk114 commented on pull request #688: HIVE-21886 : REPL - With table list - Handle rename events during replace policy URL: https://github.com/apache/hive/pull/688#discussion_r298015157 ## File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java ## @@ -168,10 +205,19 @@ public void handle(Context withinContext) throws Exception { } } -// If the tables are filtered based on name, then needs to handle the rename scenarios. -if (!withinContext.replScope.includeAllTables()) { - if (!handleForTableLevelReplication(withinContext)) { -LOG.info("Alter event for table " + before.getTableName() + " is skipped from dumping"); +// Handle renames for table level replication. +if ((Scenario.RENAME == scenario) +&& (!withinContext.replScope.includeAllTables() || withinContext.oldReplScope != null)) { + String oldName = before.getTableName(); + String newName = after.getTableName(); + boolean needDump; + if (withinContext.oldReplScope == null) { +needDump = handleRenameForTableLevelReplication(withinContext, oldName, newName); + } else { +needDump = handleRenameForReplacePolicy(withinContext, oldName, newName); + } + if (!needDump) { +LOG.info("Rename event for table " + oldName + " is skipped from dumping"); Review comment: done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268259) Time Spent: 10h 50m (was: 10h 40m) > REPL - With table list - Handle rename events during replace policy > --- > > Key: HIVE-21886 > URL: https://issues.apache.org/jira/browse/HIVE-21886 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch, > HIVE-21886.03.patch > > Time Spent: 10h 50m > Remaining Estimate: 0h > > If some rename events are found to be dumped and replayed while replace > policy is getting executed, it needs to take care of the policy inclusion in > both the policy for each table name. > 1. Create a list of tables to be bootstrapped. > 2. During handling of alter table, if the alter type is rename > 1. If the old table name is present in the list of table to be > bootstrapped, remove it. > 2. If the new table name, matches the new policy, add it to the list > of tables to be bootstrapped. > 3. If the old table does not match the old policy drop it, even if the > table is not present at target. > 3. During handling of drop table > 1. if the table is in the list of tables to be bootstrapped, then > remove it and ignore the event. > 4. During other event handling > 1. if the table is there in the list of tables to be bootstrapped, > then ignore the event. > 2. If the new policy does not match the table name, then ignore the > event. > > Rename handling during replace policy > # Old name not matching old policy – The old table will not be there at the > target cluster. The table will not be returned by get-all-table. > ## Old name is not matching new policy > ### New name not matching old policy > New name not matching new policy > * Ignore the event, no need to do anything. > New name matching new policy > * The table will be returned by get-all-table. Replace policy handler > will bootstrap this table as its matching new policy and not matching old > policy. > * All the future events will be ignored as part of check added by > replace policy handling. > * All the event with old table name will anyways be ignored as the old > name is not matching the new policy. > ### New name matching old policy > New name not matching new policy > * As the new name is not matching the new policy, the table need not be > replicated. > * As the old name is not matching the new policy, the rename eve
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[ https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268258&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268258 ] ASF GitHub Bot logged work on HIVE-21886: - Author: ASF GitHub Bot Created on: 27/Jun/19 05:49 Start Date: 27/Jun/19 05:49 Worklog Time Spent: 10m Work Description: maheshk114 commented on pull request #688: HIVE-21886 : REPL - With table list - Handle rename events during replace policy URL: https://github.com/apache/hive/pull/688#discussion_r298015144 ## File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java ## @@ -168,10 +205,19 @@ public void handle(Context withinContext) throws Exception { } } -// If the tables are filtered based on name, then needs to handle the rename scenarios. -if (!withinContext.replScope.includeAllTables()) { - if (!handleForTableLevelReplication(withinContext)) { -LOG.info("Alter event for table " + before.getTableName() + " is skipped from dumping"); +// Handle renames for table level replication. +if ((Scenario.RENAME == scenario) +&& (!withinContext.replScope.includeAllTables() || withinContext.oldReplScope != null)) { Review comment: done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268258) Time Spent: 10h 40m (was: 10.5h) > REPL - With table list - Handle rename events during replace policy > --- > > Key: HIVE-21886 > URL: https://issues.apache.org/jira/browse/HIVE-21886 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch, > HIVE-21886.03.patch > > Time Spent: 10h 40m > Remaining Estimate: 0h > > If some rename events are found to be dumped and replayed while replace > policy is getting executed, it needs to take care of the policy inclusion in > both the policy for each table name. > 1. Create a list of tables to be bootstrapped. > 2. During handling of alter table, if the alter type is rename > 1. If the old table name is present in the list of table to be > bootstrapped, remove it. > 2. If the new table name, matches the new policy, add it to the list > of tables to be bootstrapped. > 3. If the old table does not match the old policy drop it, even if the > table is not present at target. > 3. During handling of drop table > 1. if the table is in the list of tables to be bootstrapped, then > remove it and ignore the event. > 4. During other event handling > 1. if the table is there in the list of tables to be bootstrapped, > then ignore the event. > 2. If the new policy does not match the table name, then ignore the > event. > > Rename handling during replace policy > # Old name not matching old policy – The old table will not be there at the > target cluster. The table will not be returned by get-all-table. > ## Old name is not matching new policy > ### New name not matching old policy > New name not matching new policy > * Ignore the event, no need to do anything. > New name matching new policy > * The table will be returned by get-all-table. Replace policy handler > will bootstrap this table as its matching new policy and not matching old > policy. > * All the future events will be ignored as part of check added by > replace policy handling. > * All the event with old table name will anyways be ignored as the old > name is not matching the new policy. > ### New name matching old policy > New name not matching new policy > * As the new name is not matching the new policy, the table need not be > replicated. > * As the old name is not matching the new policy, the rename events will > be ignored. > * So nothing to be done for this scenario. > New name matching new policy > * As the new name is matching both old and new policy, replace handler > will not bootstrap the table. > * Add the table to the list of tables to be bootstrapped. > * Ignore all the events with new name. > * If there is a drop event for the table (with new name), then remove > the table from the the list of table to be bootstrapped. > * I
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[ https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268251&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268251 ] ASF GitHub Bot logged work on HIVE-21886: - Author: ASF GitHub Bot Created on: 27/Jun/19 05:39 Start Date: 27/Jun/19 05:39 Worklog Time Spent: 10m Work Description: sankarh commented on pull request #688: HIVE-21886 : REPL - With table list - Handle rename events during replace policy URL: https://github.com/apache/hive/pull/688#discussion_r298010710 ## File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java ## @@ -168,10 +205,19 @@ public void handle(Context withinContext) throws Exception { } } -// If the tables are filtered based on name, then needs to handle the rename scenarios. -if (!withinContext.replScope.includeAllTables()) { - if (!handleForTableLevelReplication(withinContext)) { -LOG.info("Alter event for table " + before.getTableName() + " is skipped from dumping"); +// Handle renames for table level replication. +if ((Scenario.RENAME == scenario) +&& (!withinContext.replScope.includeAllTables() || withinContext.oldReplScope != null)) { + String oldName = before.getTableName(); + String newName = after.getTableName(); + boolean needDump; + if (withinContext.oldReplScope == null) { +needDump = handleRenameForTableLevelReplication(withinContext, oldName, newName); + } else { +needDump = handleRenameForReplacePolicy(withinContext, oldName, newName); + } + if (!needDump) { +LOG.info("Rename event for table " + oldName + " is skipped from dumping"); Review comment: Shall change the log to include "newName" as well. ... oldName to newName ... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268251) Time Spent: 10.5h (was: 10h 20m) > REPL - With table list - Handle rename events during replace policy > --- > > Key: HIVE-21886 > URL: https://issues.apache.org/jira/browse/HIVE-21886 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch, > HIVE-21886.03.patch > > Time Spent: 10.5h > Remaining Estimate: 0h > > If some rename events are found to be dumped and replayed while replace > policy is getting executed, it needs to take care of the policy inclusion in > both the policy for each table name. > 1. Create a list of tables to be bootstrapped. > 2. During handling of alter table, if the alter type is rename > 1. If the old table name is present in the list of table to be > bootstrapped, remove it. > 2. If the new table name, matches the new policy, add it to the list > of tables to be bootstrapped. > 3. If the old table does not match the old policy drop it, even if the > table is not present at target. > 3. During handling of drop table > 1. if the table is in the list of tables to be bootstrapped, then > remove it and ignore the event. > 4. During other event handling > 1. if the table is there in the list of tables to be bootstrapped, > then ignore the event. > 2. If the new policy does not match the table name, then ignore the > event. > > Rename handling during replace policy > # Old name not matching old policy – The old table will not be there at the > target cluster. The table will not be returned by get-all-table. > ## Old name is not matching new policy > ### New name not matching old policy > New name not matching new policy > * Ignore the event, no need to do anything. > New name matching new policy > * The table will be returned by get-all-table. Replace policy handler > will bootstrap this table as its matching new policy and not matching old > policy. > * All the future events will be ignored as part of check added by > replace policy handling. > * All the event with old table name will anyways be ignored as the old > name is not matching the new policy. > ### New name matching old policy > New name not matching new policy > * As the new name is not matching the new policy, the table need not be > replicated. > **
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[ https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268252&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268252 ] ASF GitHub Bot logged work on HIVE-21886: - Author: ASF GitHub Bot Created on: 27/Jun/19 05:39 Start Date: 27/Jun/19 05:39 Worklog Time Spent: 10m Work Description: sankarh commented on pull request #688: HIVE-21886 : REPL - With table list - Handle rename events during replace policy URL: https://github.com/apache/hive/pull/688#discussion_r298010206 ## File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java ## @@ -168,10 +205,19 @@ public void handle(Context withinContext) throws Exception { } } -// If the tables are filtered based on name, then needs to handle the rename scenarios. -if (!withinContext.replScope.includeAllTables()) { - if (!handleForTableLevelReplication(withinContext)) { -LOG.info("Alter event for table " + before.getTableName() + " is skipped from dumping"); +// Handle renames for table level replication. +if ((Scenario.RENAME == scenario) +&& (!withinContext.replScope.includeAllTables() || withinContext.oldReplScope != null)) { Review comment: These checks can be moved inside as follows. Will be less complex. if (Scenario.RENAME == scenario) { ... boolean needDump = true; if (withinContext.oldReplScope != null) { needDump = handleRenameForReplacePolicy(); } else if (!withinContext.replScope.includeAllTables()) { needDump = handleRenameForTableLevelReplication(); } if (!needDump) { return; } } This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268252) > REPL - With table list - Handle rename events during replace policy > --- > > Key: HIVE-21886 > URL: https://issues.apache.org/jira/browse/HIVE-21886 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch, > HIVE-21886.03.patch > > Time Spent: 10.5h > Remaining Estimate: 0h > > If some rename events are found to be dumped and replayed while replace > policy is getting executed, it needs to take care of the policy inclusion in > both the policy for each table name. > 1. Create a list of tables to be bootstrapped. > 2. During handling of alter table, if the alter type is rename > 1. If the old table name is present in the list of table to be > bootstrapped, remove it. > 2. If the new table name, matches the new policy, add it to the list > of tables to be bootstrapped. > 3. If the old table does not match the old policy drop it, even if the > table is not present at target. > 3. During handling of drop table > 1. if the table is in the list of tables to be bootstrapped, then > remove it and ignore the event. > 4. During other event handling > 1. if the table is there in the list of tables to be bootstrapped, > then ignore the event. > 2. If the new policy does not match the table name, then ignore the > event. > > Rename handling during replace policy > # Old name not matching old policy – The old table will not be there at the > target cluster. The table will not be returned by get-all-table. > ## Old name is not matching new policy > ### New name not matching old policy > New name not matching new policy > * Ignore the event, no need to do anything. > New name matching new policy > * The table will be returned by get-all-table. Replace policy handler > will bootstrap this table as its matching new policy and not matching old > policy. > * All the future events will be ignored as part of check added by > replace policy handling. > * All the event with old table name will anyways be ignored as the old > name is not matching the new policy. > ### New name matching old policy > New name not matching new policy > * As the new name is not matching the new policy, the table need not be > replicated. > * As the old name is not matching the new policy, the rename events will > be ignored. > * So nothing to be done for this scenario. > N
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[ https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268250&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268250 ] ASF GitHub Bot logged work on HIVE-21886: - Author: ASF GitHub Bot Created on: 27/Jun/19 05:39 Start Date: 27/Jun/19 05:39 Worklog Time Spent: 10m Work Description: sankarh commented on pull request #688: HIVE-21886 : REPL - With table list - Handle rename events during replace policy URL: https://github.com/apache/hive/pull/688#discussion_r298012981 ## File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java ## @@ -92,21 +93,49 @@ private Scenario scenarioType(org.apache.hadoop.hive.metastore.api.Table before, } } - // return true, if event needs to be dumped, else return false. - private boolean handleForTableLevelReplication(Context withinContext) { -String oldName = before.getTableName(); -String newName = after.getTableName(); + // Return true, if event needs to be dumped, else return false. + private boolean handleRenameForReplacePolicy(Context withinContext, String oldName, String newName) { +// If the table is renamed after being added to the list of tables to be bootstrapped, then remove it from the Review comment: Just add an assert that withinContext.oldReplScope != null. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268250) Time Spent: 10h 20m (was: 10h 10m) > REPL - With table list - Handle rename events during replace policy > --- > > Key: HIVE-21886 > URL: https://issues.apache.org/jira/browse/HIVE-21886 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch, > HIVE-21886.03.patch > > Time Spent: 10h 20m > Remaining Estimate: 0h > > If some rename events are found to be dumped and replayed while replace > policy is getting executed, it needs to take care of the policy inclusion in > both the policy for each table name. > 1. Create a list of tables to be bootstrapped. > 2. During handling of alter table, if the alter type is rename > 1. If the old table name is present in the list of table to be > bootstrapped, remove it. > 2. If the new table name, matches the new policy, add it to the list > of tables to be bootstrapped. > 3. If the old table does not match the old policy drop it, even if the > table is not present at target. > 3. During handling of drop table > 1. if the table is in the list of tables to be bootstrapped, then > remove it and ignore the event. > 4. During other event handling > 1. if the table is there in the list of tables to be bootstrapped, > then ignore the event. > 2. If the new policy does not match the table name, then ignore the > event. > > Rename handling during replace policy > # Old name not matching old policy – The old table will not be there at the > target cluster. The table will not be returned by get-all-table. > ## Old name is not matching new policy > ### New name not matching old policy > New name not matching new policy > * Ignore the event, no need to do anything. > New name matching new policy > * The table will be returned by get-all-table. Replace policy handler > will bootstrap this table as its matching new policy and not matching old > policy. > * All the future events will be ignored as part of check added by > replace policy handling. > * All the event with old table name will anyways be ignored as the old > name is not matching the new policy. > ### New name matching old policy > New name not matching new policy > * As the new name is not matching the new policy, the table need not be > replicated. > * As the old name is not matching the new policy, the rename events will > be ignored. > * So nothing to be done for this scenario. > New name matching new policy > * As the new name is matching both old and new policy, replace handler > will not bootstrap the table. > * Add the table to the list of tables to be bootstrapped. > * Ignore all the events with new name. > * If there is a drop event for the table (with new name)
[jira] [Updated] (HIVE-21926) REPL - With table list - "TO" and "FROM" clause should not be allowed along with table filter list
[ https://issues.apache.org/jira/browse/HIVE-21926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-21926: --- Description: If table filter list is provided during dump command, there is a chance that tables will be bootstrapped during rename and during policy replace. If all existing events are not replayed completely, then it may happen that the remaining events in the next replication cycle causes data corruption. So all the existing events should be replayed and "TO" and "FROM" should not be allowed if table filter list is provided. > REPL - With table list - "TO" and "FROM" clause should not be allowed along > with table filter list > -- > > Key: HIVE-21926 > URL: https://issues.apache.org/jira/browse/HIVE-21926 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > > If table filter list is provided during dump command, there is a chance that > tables will be bootstrapped during rename and during policy replace. If all > existing events are not replayed completely, then it may happen that the > remaining events in the next replication cycle causes data corruption. So all > the existing events should be replayed and "TO" and "FROM" should not be > allowed if table filter list is provided. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21926) REPL - With table list - "TO" and "FROM" clause should not be allowed along with table filter list
[ https://issues.apache.org/jira/browse/HIVE-21926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-21926: --- Summary: REPL - With table list - "TO" and "FROM" clause should not be allowed along with table filter list (was: CLONE - REPL - With table list - "TO" and "FROM" clause should not be allowed along with table filter list) > REPL - With table list - "TO" and "FROM" clause should not be allowed along > with table filter list > -- > > Key: HIVE-21926 > URL: https://issues.apache.org/jira/browse/HIVE-21926 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[ https://issues.apache.org/jira/browse/HIVE-21886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-21886: --- Description: If some rename events are found to be dumped and replayed while replace policy is getting executed, it needs to take care of the policy inclusion in both the policy for each table name. 1. Create a list of tables to be bootstrapped. 2. During handling of alter table, if the alter type is rename 1. If the old table name is present in the list of table to be bootstrapped, remove it. 2. If the new table name, matches the new policy, add it to the list of tables to be bootstrapped. 3. If the old table does not match the old policy drop it, even if the table is not present at target. 3. During handling of drop table 1. if the table is in the list of tables to be bootstrapped, then remove it and ignore the event. 4. During other event handling 1. if the table is there in the list of tables to be bootstrapped, then ignore the event. 2. If the new policy does not match the table name, then ignore the event. Rename handling during replace policy # Old name not matching old policy – The old table will not be there at the target cluster. The table will not be returned by get-all-table. ## Old name is not matching new policy ### New name not matching old policy New name not matching new policy * Ignore the event, no need to do anything. New name matching new policy * The table will be returned by get-all-table. Replace policy handler will bootstrap this table as its matching new policy and not matching old policy. * All the future events will be ignored as part of check added by replace policy handling. * All the event with old table name will anyways be ignored as the old name is not matching the new policy. ### New name matching old policy New name not matching new policy * As the new name is not matching the new policy, the table need not be replicated. * As the old name is not matching the new policy, the rename events will be ignored. * So nothing to be done for this scenario. New name matching new policy * As the new name is matching both old and new policy, replace handler will not bootstrap the table. * Add the table to the list of tables to be bootstrapped. * Ignore all the events with new name. * If there is a drop event for the table (with new name), then remove the table from the the list of table to be bootstrapped. * In case of rename event (double rename) ** If the new name satisfies the table pattern, then add the new name to the list of tables to be bootstrapped and remove the old name from the list of tables to be bootstrapped. ** If the new name does not satisfies then just removed the table name from the list of tables to be bootstrapped. ## Old name is matching new policy – As per replace policy handler, which checks based on old table, the table should be bootstrapped and event should be ignored. But rename handler should decide based on new name.The old table name will not be returned by get-all-table, so replace handler will not d anything for the old table. ### New name not matching old policy New name not matching new policy * As the old table is not there at target and new name is not matching new policy. Ignore the event. * No need to add the table to the list of tables to be bootstrapped. * All the subsequent events will be ignored as the new name is not matching the new policy. New name matching new policy * As the new name is not matching old policy but matching new policy, the table will be bootstrapped by replace policy handler. So rename event need not add this table to list of table to be bootstrapped. * All the future events will be ignored by replace policy handler. * For rename event (double rename) ** If there is a rename, the table (with intermittent new name) will not be present and thus replace handler will not bootstrap the table. ** So if the new name (the latest one) is matching the new policy, then add it to the list of table to be bootstrapped. ** And If the new name (the latest one) is not matching the new policy, then just ignore the event as the intermittent new name would not have added to the list of table to be bootstrapped. ### New name matching old policy New name not matching new policy * Dump the event. The table will be dropped by repl load at the target. New name matching new policy * Replace handler will not bootstrap this table as the new name is matching both policies. * As old name is not matching the old policy, the table will not be there at target. The rename event should add the new name to the list of table to be bootstrapped. * Subsequent eve
[jira] [Updated] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[ https://issues.apache.org/jira/browse/HIVE-21886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-21886: --- Description: If some rename events are found to be dumped and replayed while replace policy is getting executed, it needs to take care of the policy inclusion in both the policy for each table name. 1. Create a list of tables to be bootstrapped. 2. During handling of alter table, if the alter type is rename 1. If the old table name is present in the list of table to be bootstrapped, remove it. 2. If the new table name, matches the new policy, add it to the list of tables to be bootstrapped. 3. If the old table does not match the old policy drop it, even if the table is not present at target. 3. During handling of drop table 1. if the table is in the list of tables to be bootstrapped, then remove it and ignore the event. 4. During other event handling 1. if the table is there in the list of tables to be bootstrapped, then ignore the event. Rename handling during replace policy # Old name not matching old policy – The old table will not be there at the target cluster. The table will not be returned by get-all-table. ## Old name is not matching new policy ### New name not matching old policy New name not matching new policy * Ignore the event, no need to do anything. New name matching new policy * The table will be returned by get-all-table. Replace policy handler will bootstrap this table as its matching new policy and not matching old policy. * All the future events will be ignored as part of check added by replace policy handling. * All the event with old table name will anyways be ignored as the old name is not matching the new policy. ### New name matching old policy New name not matching new policy * As the new name is not matching the new policy, the table need not be replicated. * As the old name is not matching the new policy, the rename events will be ignored. * So nothing to be done for this scenario. New name matching new policy * As the new name is matching both old and new policy, replace handler will not bootstrap the table. * Add the table to the list of tables to be bootstrapped. * Ignore all the events with new name. * If there is a drop event for the table (with new name), then remove the table from the the list of table to be bootstrapped. * In case of rename event (double rename) ** If the new name satisfies the table pattern, then add the new name to the list of tables to be bootstrapped and remove the old name from the list of tables to be bootstrapped. ** If the new name does not satisfies then just removed the table name from the list of tables to be bootstrapped. ## Old name is matching new policy – As per replace policy handler, which checks based on old table, the table should be bootstrapped and event should be ignored. But rename handler should decide based on new name.The old table name will not be returned by get-all-table, so replace handler will not d anything for the old table. ### New name not matching old policy New name not matching new policy * As the old table is not there at target and new name is not matching new policy. Ignore the event. * No need to add the table to the list of tables to be bootstrapped. * All the subsequent events will be ignored as the new name is not matching the new policy. New name matching new policy * As the new name is not matching old policy but matching new policy, the table will be bootstrapped by replace policy handler. So rename event need not add this table to list of table to be bootstrapped. * All the future events will be ignored by replace policy handler. * For rename event (double rename) ** If there is a rename, the table (with intermittent new name) will not be present and thus replace handler will not bootstrap the table. ** So if the new name (the latest one) is matching the new policy, then add it to the list of table to be bootstrapped. ** And If the new name (the latest one) is not matching the new policy, then just ignore the event as the intermittent new name would not have added to the list of table to be bootstrapped. ### New name matching old policy New name not matching new policy * Dump the event. The table will be dropped by repl load at the target. New name matching new policy * Replace handler will not bootstrap this table as the new name is matching both policies. * As old name is not matching the old policy, the table will not be there at target. The rename event should add the new name to the list of table to be bootstrapped. * Subsequent events with new table name should be ignored. * Drop events should not be ignored
[jira] [Updated] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[ https://issues.apache.org/jira/browse/HIVE-21886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-21886: --- Status: Patch Available (was: Open) > REPL - With table list - Handle rename events during replace policy > --- > > Key: HIVE-21886 > URL: https://issues.apache.org/jira/browse/HIVE-21886 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch, > HIVE-21886.03.patch > > Time Spent: 10h 10m > Remaining Estimate: 0h > > If some rename events are found to be dumped and replayed while replace > policy is getting executed, it needs to take care of the policy inclusion in > both the policy for each table name. > 1. Create a list of tables to be bootstrapped. > 2. During handling of alter table, if the alter type is rename > 1. If the old table name is present in the list of table to be > bootstrapped, remove it. > 2. If the new table name, matches the new policy, add it to the list > of tables to be bootstrapped. > 3. During handling of drop table > 1. if the table is in the list of tables to be bootstrapped, then > remove it and ignore the event. > 4. During other event handling > 1. if the table is there in the list of tables to be bootstrapped, > then ignore the event. > > Rename handling during replace policy > # Old name not matching old policy – The old table will not be there at the > target cluster. The table will not be returned by get-all-table. > ## Old name is not matching new policy > ### New name not matching old policy > New name not matching new policy > * Ignore the event, no need to do anything. > New name matching new policy > * The table will be returned by get-all-table. Replace policy handler > will bootstrap this table as its matching new policy and not matching old > policy. > * All the future events will be ignored as part of check added by > replace policy handling. > * All the event with old table name will anyways be ignored as the old > name is not matching the new policy. > ### New name matching old policy > New name not matching new policy > * As the new name is not matching the new policy, the table need not be > replicated. > * As the old name is not matching the new policy, the rename events will > be ignored. > * So nothing to be done for this scenario. > New name matching new policy > * As the new name is matching both old and new policy, replace handler > will not bootstrap the table. > * Add the table to the list of tables to be bootstrapped. > * Ignore all the events with new name. > * If there is a drop event for the table (with new name), then remove > the table from the the list of table to be bootstrapped. > * In case of rename event (double rename) > ** If the new name satisfies the table pattern, then add the new name to > the list of tables to be bootstrapped and remove the old name from the list > of tables to be bootstrapped. > ** If the new name does not satisfies then just removed the table name > from the list of tables to be bootstrapped. > ## Old name is matching new policy – As per replace policy handler, which > checks based on old table, the table should be bootstrapped and event should > be ignored. But rename handler should decide based on new name.The old table > name will not be returned by get-all-table, so replace handler will not d > anything for the old table. > ### New name not matching old policy > New name not matching new policy > * As the old table is not there at target and new name is not matching > new policy. Ignore the event. > * No need to add the table to the list of tables to be bootstrapped. > * All the subsequent events will be ignored as the new name is not > matching the new policy. > New name matching new policy > * As the new name is not matching old policy but matching new policy, > the table will be bootstrapped by replace policy handler. So rename event > need not add this table to list of table to be bootstrapped. > * All the future events will be ignored by replace policy handler. > * For rename event (double rename) > ** If there is a rename, the table (with intermittent new name) will not > be present and thus replace handler will not bootstrap the table. > ** So if the new name (the latest one) is matching the new policy, then > add it to the list of table to be bootstrapped. > ** And If the new name (the latest one)
[jira] [Updated] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[ https://issues.apache.org/jira/browse/HIVE-21886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-21886: --- Attachment: HIVE-21886.03.patch > REPL - With table list - Handle rename events during replace policy > --- > > Key: HIVE-21886 > URL: https://issues.apache.org/jira/browse/HIVE-21886 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch, > HIVE-21886.03.patch > > Time Spent: 10h 10m > Remaining Estimate: 0h > > If some rename events are found to be dumped and replayed while replace > policy is getting executed, it needs to take care of the policy inclusion in > both the policy for each table name. > 1. Create a list of tables to be bootstrapped. > 2. During handling of alter table, if the alter type is rename > 1. If the old table name is present in the list of table to be > bootstrapped, remove it. > 2. If the new table name, matches the new policy, add it to the list > of tables to be bootstrapped. > 3. During handling of drop table > 1. if the table is in the list of tables to be bootstrapped, then > remove it and ignore the event. > 4. During other event handling > 1. if the table is there in the list of tables to be bootstrapped, > then ignore the event. > > Rename handling during replace policy > # Old name not matching old policy – The old table will not be there at the > target cluster. The table will not be returned by get-all-table. > ## Old name is not matching new policy > ### New name not matching old policy > New name not matching new policy > * Ignore the event, no need to do anything. > New name matching new policy > * The table will be returned by get-all-table. Replace policy handler > will bootstrap this table as its matching new policy and not matching old > policy. > * All the future events will be ignored as part of check added by > replace policy handling. > * All the event with old table name will anyways be ignored as the old > name is not matching the new policy. > ### New name matching old policy > New name not matching new policy > * As the new name is not matching the new policy, the table need not be > replicated. > * As the old name is not matching the new policy, the rename events will > be ignored. > * So nothing to be done for this scenario. > New name matching new policy > * As the new name is matching both old and new policy, replace handler > will not bootstrap the table. > * Add the table to the list of tables to be bootstrapped. > * Ignore all the events with new name. > * If there is a drop event for the table (with new name), then remove > the table from the the list of table to be bootstrapped. > * In case of rename event (double rename) > ** If the new name satisfies the table pattern, then add the new name to > the list of tables to be bootstrapped and remove the old name from the list > of tables to be bootstrapped. > ** If the new name does not satisfies then just removed the table name > from the list of tables to be bootstrapped. > ## Old name is matching new policy – As per replace policy handler, which > checks based on old table, the table should be bootstrapped and event should > be ignored. But rename handler should decide based on new name.The old table > name will not be returned by get-all-table, so replace handler will not d > anything for the old table. > ### New name not matching old policy > New name not matching new policy > * As the old table is not there at target and new name is not matching > new policy. Ignore the event. > * No need to add the table to the list of tables to be bootstrapped. > * All the subsequent events will be ignored as the new name is not > matching the new policy. > New name matching new policy > * As the new name is not matching old policy but matching new policy, > the table will be bootstrapped by replace policy handler. So rename event > need not add this table to list of table to be bootstrapped. > * All the future events will be ignored by replace policy handler. > * For rename event (double rename) > ** If there is a rename, the table (with intermittent new name) will not > be present and thus replace handler will not bootstrap the table. > ** So if the new name (the latest one) is matching the new policy, then > add it to the list of table to be bootstrapped. > ** And If the new name (the latest one) is no
[jira] [Updated] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[ https://issues.apache.org/jira/browse/HIVE-21886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-21886: --- Status: Open (was: Patch Available) > REPL - With table list - Handle rename events during replace policy > --- > > Key: HIVE-21886 > URL: https://issues.apache.org/jira/browse/HIVE-21886 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch > > Time Spent: 10h 10m > Remaining Estimate: 0h > > If some rename events are found to be dumped and replayed while replace > policy is getting executed, it needs to take care of the policy inclusion in > both the policy for each table name. > 1. Create a list of tables to be bootstrapped. > 2. During handling of alter table, if the alter type is rename > 1. If the old table name is present in the list of table to be > bootstrapped, remove it. > 2. If the new table name, matches the new policy, add it to the list > of tables to be bootstrapped. > 3. During handling of drop table > 1. if the table is in the list of tables to be bootstrapped, then > remove it and ignore the event. > 4. During other event handling > 1. if the table is there in the list of tables to be bootstrapped, > then ignore the event. > > Rename handling during replace policy > # Old name not matching old policy – The old table will not be there at the > target cluster. The table will not be returned by get-all-table. > ## Old name is not matching new policy > ### New name not matching old policy > New name not matching new policy > * Ignore the event, no need to do anything. > New name matching new policy > * The table will be returned by get-all-table. Replace policy handler > will bootstrap this table as its matching new policy and not matching old > policy. > * All the future events will be ignored as part of check added by > replace policy handling. > * All the event with old table name will anyways be ignored as the old > name is not matching the new policy. > ### New name matching old policy > New name not matching new policy > * As the new name is not matching the new policy, the table need not be > replicated. > * As the old name is not matching the new policy, the rename events will > be ignored. > * So nothing to be done for this scenario. > New name matching new policy > * As the new name is matching both old and new policy, replace handler > will not bootstrap the table. > * Add the table to the list of tables to be bootstrapped. > * Ignore all the events with new name. > * If there is a drop event for the table (with new name), then remove > the table from the the list of table to be bootstrapped. > * In case of rename event (double rename) > ** If the new name satisfies the table pattern, then add the new name to > the list of tables to be bootstrapped and remove the old name from the list > of tables to be bootstrapped. > ** If the new name does not satisfies then just removed the table name > from the list of tables to be bootstrapped. > ## Old name is matching new policy – As per replace policy handler, which > checks based on old table, the table should be bootstrapped and event should > be ignored. But rename handler should decide based on new name.The old table > name will not be returned by get-all-table, so replace handler will not d > anything for the old table. > ### New name not matching old policy > New name not matching new policy > * As the old table is not there at target and new name is not matching > new policy. Ignore the event. > * No need to add the table to the list of tables to be bootstrapped. > * All the subsequent events will be ignored as the new name is not > matching the new policy. > New name matching new policy > * As the new name is not matching old policy but matching new policy, > the table will be bootstrapped by replace policy handler. So rename event > need not add this table to list of table to be bootstrapped. > * All the future events will be ignored by replace policy handler. > * For rename event (double rename) > ** If there is a rename, the table (with intermittent new name) will not > be present and thus replace handler will not bootstrap the table. > ** So if the new name (the latest one) is matching the new policy, then > add it to the list of table to be bootstrapped. > ** And If the new name (the latest one) is not matching the new
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[ https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268234&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268234 ] ASF GitHub Bot logged work on HIVE-21886: - Author: ASF GitHub Bot Created on: 27/Jun/19 05:12 Start Date: 27/Jun/19 05:12 Worklog Time Spent: 10m Work Description: maheshk114 commented on pull request #688: HIVE-21886 : REPL - With table list - Handle rename events during replace policy URL: https://github.com/apache/hive/pull/688#discussion_r298008351 ## File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java ## @@ -92,21 +93,46 @@ private Scenario scenarioType(org.apache.hadoop.hive.metastore.api.Table before, } } - // return true, if event needs to be dumped, else return false. - private boolean handleForTableLevelReplication(Context withinContext) { -String oldName = before.getTableName(); -String newName = after.getTableName(); + private boolean isSetForBootstrapByReplaceHandler(Context withinContext, String tblName) { +return (withinContext.oldReplScope != null) +&& !(ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName)) +&& (ReplUtils.tableIncludedInReplScope(withinContext.replScope, tblName)); + } + + // Return true, if event needs to be dumped, else return false. + private boolean handleForTableLevelReplication(Context withinContext, String tblName) { +// For alter, if the table does not satisfy the new policy then ignore the event. In case of replace +// policy, if the table does not satisfy the old policy, then ignore the event. As, if the table satisfy the new +// policy, then the table will be bootstrapped by replace handler anb if the table does not satisfy the new policy, +// then anyways the table should be ignored. The event should be ignored if the table is already in the list of +// tables to be bootstrapped. +if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope, tblName)) { + // In case of replace, it will be dropped during load. In normal case just ignore the alter event. + LOG.debug("Table " + tblName + " does not satisfy the policy"); + return false; +} else if ((withinContext.getTablesForBootstrap().contains(tblName.toLowerCase())) Review comment: done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268234) Time Spent: 10h 10m (was: 10h) > REPL - With table list - Handle rename events during replace policy > --- > > Key: HIVE-21886 > URL: https://issues.apache.org/jira/browse/HIVE-21886 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch > > Time Spent: 10h 10m > Remaining Estimate: 0h > > If some rename events are found to be dumped and replayed while replace > policy is getting executed, it needs to take care of the policy inclusion in > both the policy for each table name. > 1. Create a list of tables to be bootstrapped. > 2. During handling of alter table, if the alter type is rename > 1. If the old table name is present in the list of table to be > bootstrapped, remove it. > 2. If the new table name, matches the new policy, add it to the list > of tables to be bootstrapped. > 3. During handling of drop table > 1. if the table is in the list of tables to be bootstrapped, then > remove it and ignore the event. > 4. During other event handling > 1. if the table is there in the list of tables to be bootstrapped, > then ignore the event. > > Rename handling during replace policy > # Old name not matching old policy – The old table will not be there at the > target cluster. The table will not be returned by get-all-table. > ## Old name is not matching new policy > ### New name not matching old policy > New name not matching new policy > * Ignore the event, no need to do anything. > New name matching new policy > * The table will be returned by get-all-table. Replace policy handler > will bootstrap this table as its matching new policy and not matching old > policy. > * All the future events will be ignored as part of check ad
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[ https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268233&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268233 ] ASF GitHub Bot logged work on HIVE-21886: - Author: ASF GitHub Bot Created on: 27/Jun/19 05:12 Start Date: 27/Jun/19 05:12 Worklog Time Spent: 10m Work Description: maheshk114 commented on pull request #688: HIVE-21886 : REPL - With table list - Handle rename events during replace policy URL: https://github.com/apache/hive/pull/688#discussion_r298008286 ## File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java ## @@ -147,16 +197,19 @@ public void handle(Context withinContext) throws Exception { Table qlMdTableBefore = new Table(before); Set bootstrapTableList; +ReplScope oldReplScope; if (Scenario.RENAME == scenario) { // Handling for table level replication is done in handleForTableLevelReplication method. Review comment: code is removed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268233) Time Spent: 10h (was: 9h 50m) > REPL - With table list - Handle rename events during replace policy > --- > > Key: HIVE-21886 > URL: https://issues.apache.org/jira/browse/HIVE-21886 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch > > Time Spent: 10h > Remaining Estimate: 0h > > If some rename events are found to be dumped and replayed while replace > policy is getting executed, it needs to take care of the policy inclusion in > both the policy for each table name. > 1. Create a list of tables to be bootstrapped. > 2. During handling of alter table, if the alter type is rename > 1. If the old table name is present in the list of table to be > bootstrapped, remove it. > 2. If the new table name, matches the new policy, add it to the list > of tables to be bootstrapped. > 3. During handling of drop table > 1. if the table is in the list of tables to be bootstrapped, then > remove it and ignore the event. > 4. During other event handling > 1. if the table is there in the list of tables to be bootstrapped, > then ignore the event. > > Rename handling during replace policy > # Old name not matching old policy – The old table will not be there at the > target cluster. The table will not be returned by get-all-table. > ## Old name is not matching new policy > ### New name not matching old policy > New name not matching new policy > * Ignore the event, no need to do anything. > New name matching new policy > * The table will be returned by get-all-table. Replace policy handler > will bootstrap this table as its matching new policy and not matching old > policy. > * All the future events will be ignored as part of check added by > replace policy handling. > * All the event with old table name will anyways be ignored as the old > name is not matching the new policy. > ### New name matching old policy > New name not matching new policy > * As the new name is not matching the new policy, the table need not be > replicated. > * As the old name is not matching the new policy, the rename events will > be ignored. > * So nothing to be done for this scenario. > New name matching new policy > * As the new name is matching both old and new policy, replace handler > will not bootstrap the table. > * Add the table to the list of tables to be bootstrapped. > * Ignore all the events with new name. > * If there is a drop event for the table (with new name), then remove > the table from the the list of table to be bootstrapped. > * In case of rename event (double rename) > ** If the new name satisfies the table pattern, then add the new name to > the list of tables to be bootstrapped and remove the old name from the list > of tables to be bootstrapped. > ** If the new name does not satisfies then just removed the table name > from the list of tables to be bootstrapped. > ## Old name is matching new policy – As per replace policy handler, which > checks based on old table, the table should be bootstrapped and
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[ https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268232&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268232 ] ASF GitHub Bot logged work on HIVE-21886: - Author: ASF GitHub Bot Created on: 27/Jun/19 05:11 Start Date: 27/Jun/19 05:11 Worklog Time Spent: 10m Work Description: maheshk114 commented on pull request #688: HIVE-21886 : REPL - With table list - Handle rename events during replace policy URL: https://github.com/apache/hive/pull/688#discussion_r298008125 ## File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java ## @@ -118,13 +144,30 @@ private boolean handleForTableLevelReplication(Context withinContext) { // If the old table was in the list of tables to be bootstrapped which is a multi rename case, the old table // is removed from the list of tables to be bootstrapped and new one is added. - if (oldTableIsPresent) { + if (oldTableInBootstrapList) { +withinContext.addToListOfTablesForBootstrap(newName); +return false; + } + + // All the subsequent events on this table newName are going to be skipped as the table is going to be + // bootstrapped by replace handler, so the rename is also skipped. + if (isSetForBootstrapByReplaceHandler(withinContext, newName)) { +// This addition is not actually required. But is added just to be in safer side. withinContext.addToListOfTablesForBootstrap(newName); + +// If the old table satisfies the new policy and is not in the list of tables to be bootstrapped +// (as per previous check based on oldTableIsPresent), then drop it. +if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) { + scenario = Scenario.DROP; + LOG.info("Table " + oldName + " will be dropped as the table " + newName + " will be bootstrapped."); + return true; +} +LOG.info("Table " + newName + " is set to be bootstrapped by replace policy handler."); return false; } // If both old and new table satisfies the filter and old table is present at target, then dump the rename event. - LOG.info("both old and new table satisfies the filter"); + LOG.info("Both old " + oldName + " and new table " + newName + " satisfies the filter"); Review comment: done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268232) Time Spent: 9h 50m (was: 9h 40m) > REPL - With table list - Handle rename events during replace policy > --- > > Key: HIVE-21886 > URL: https://issues.apache.org/jira/browse/HIVE-21886 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch > > Time Spent: 9h 50m > Remaining Estimate: 0h > > If some rename events are found to be dumped and replayed while replace > policy is getting executed, it needs to take care of the policy inclusion in > both the policy for each table name. > 1. Create a list of tables to be bootstrapped. > 2. During handling of alter table, if the alter type is rename > 1. If the old table name is present in the list of table to be > bootstrapped, remove it. > 2. If the new table name, matches the new policy, add it to the list > of tables to be bootstrapped. > 3. During handling of drop table > 1. if the table is in the list of tables to be bootstrapped, then > remove it and ignore the event. > 4. During other event handling > 1. if the table is there in the list of tables to be bootstrapped, > then ignore the event. > > Rename handling during replace policy > # Old name not matching old policy – The old table will not be there at the > target cluster. The table will not be returned by get-all-table. > ## Old name is not matching new policy > ### New name not matching old policy > New name not matching new policy > * Ignore the event, no need to do anything. > New name matching new policy > * The table will be returned by get-all-table. Replace policy handler > will bootstrap this table as its matching new policy and not matching old > policy. > * All the futu
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[ https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268231&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268231 ] ASF GitHub Bot logged work on HIVE-21886: - Author: ASF GitHub Bot Created on: 27/Jun/19 05:11 Start Date: 27/Jun/19 05:11 Worklog Time Spent: 10m Work Description: maheshk114 commented on pull request #688: HIVE-21886 : REPL - With table list - Handle rename events during replace policy URL: https://github.com/apache/hive/pull/688#discussion_r298008102 ## File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java ## @@ -135,8 +178,15 @@ private boolean handleForTableLevelReplication(Context withinContext) { return false; } + // In case of replace policy, even if the old table matches the new policy, none of the events including create + // table will be replayed as the old table is set of bootstrap by replace handler. So rename event can be skipped. Review comment: code is removed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268231) Time Spent: 9h 40m (was: 9.5h) > REPL - With table list - Handle rename events during replace policy > --- > > Key: HIVE-21886 > URL: https://issues.apache.org/jira/browse/HIVE-21886 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch > > Time Spent: 9h 40m > Remaining Estimate: 0h > > If some rename events are found to be dumped and replayed while replace > policy is getting executed, it needs to take care of the policy inclusion in > both the policy for each table name. > 1. Create a list of tables to be bootstrapped. > 2. During handling of alter table, if the alter type is rename > 1. If the old table name is present in the list of table to be > bootstrapped, remove it. > 2. If the new table name, matches the new policy, add it to the list > of tables to be bootstrapped. > 3. During handling of drop table > 1. if the table is in the list of tables to be bootstrapped, then > remove it and ignore the event. > 4. During other event handling > 1. if the table is there in the list of tables to be bootstrapped, > then ignore the event. > > Rename handling during replace policy > # Old name not matching old policy – The old table will not be there at the > target cluster. The table will not be returned by get-all-table. > ## Old name is not matching new policy > ### New name not matching old policy > New name not matching new policy > * Ignore the event, no need to do anything. > New name matching new policy > * The table will be returned by get-all-table. Replace policy handler > will bootstrap this table as its matching new policy and not matching old > policy. > * All the future events will be ignored as part of check added by > replace policy handling. > * All the event with old table name will anyways be ignored as the old > name is not matching the new policy. > ### New name matching old policy > New name not matching new policy > * As the new name is not matching the new policy, the table need not be > replicated. > * As the old name is not matching the new policy, the rename events will > be ignored. > * So nothing to be done for this scenario. > New name matching new policy > * As the new name is matching both old and new policy, replace handler > will not bootstrap the table. > * Add the table to the list of tables to be bootstrapped. > * Ignore all the events with new name. > * If there is a drop event for the table (with new name), then remove > the table from the the list of table to be bootstrapped. > * In case of rename event (double rename) > ** If the new name satisfies the table pattern, then add the new name to > the list of tables to be bootstrapped and remove the old name from the list > of tables to be bootstrapped. > ** If the new name does not satisfies then just removed the table name > from the list of tables to be bootstrapped. > ## Old name is matching new policy – As per replace policy handler, which > checks based on ol
[jira] [Assigned] (HIVE-21926) CLONE - REPL - With table list - "TO" and "FROM" clause should not be allowed along with table filter list
[ https://issues.apache.org/jira/browse/HIVE-21926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera reassigned HIVE-21926: -- > CLONE - REPL - With table list - "TO" and "FROM" clause should not be allowed > along with table filter list > -- > > Key: HIVE-21926 > URL: https://issues.apache.org/jira/browse/HIVE-21926 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > > If some rename events are found to be dumped and replayed while replace > policy is getting executed, it needs to take care of the policy inclusion in > both the policy for each table name. > 1. Create a list of tables to be bootstrapped. > 2. During handling of alter table, if the alter type is rename > 1. If the old table name is present in the list of table to be > bootstrapped, remove it. > 2. If the new table name, matches the new policy, add it to the list > of tables to be bootstrapped. > 3. During handling of drop table > 1. if the table is in the list of tables to be bootstrapped, then > remove it and ignore the event. > 4. During other event handling > 1. if the table is there in the list of tables to be bootstrapped, > then ignore the event. > > Rename handling during replace policy > # Old name not matching old policy – The old table will not be there at the > target cluster. The table will not be returned by get-all-table. > ## Old name is not matching new policy > ### New name not matching old policy > New name not matching new policy > * Ignore the event, no need to do anything. > New name matching new policy > * The table will be returned by get-all-table. Replace policy handler > will bootstrap this table as its matching new policy and not matching old > policy. > * All the future events will be ignored as part of check added by > replace policy handling. > * All the event with old table name will anyways be ignored as the old > name is not matching the new policy. > ### New name matching old policy > New name not matching new policy > * As the new name is not matching the new policy, the table need not be > replicated. > * As the old name is not matching the new policy, the rename events will > be ignored. > * So nothing to be done for this scenario. > New name matching new policy > * As the new name is matching both old and new policy, replace handler > will not bootstrap the table. > * Add the table to the list of tables to be bootstrapped. > * Ignore all the events with new name. > * If there is a drop event for the table (with new name), then remove > the table from the the list of table to be bootstrapped. > * In case of rename event (double rename) > ** If the new name satisfies the table pattern, then add the new name to > the list of tables to be bootstrapped and remove the old name from the list > of tables to be bootstrapped. > ** If the new name does not satisfies then just removed the table name > from the list of tables to be bootstrapped. > ## Old name is matching new policy – As per replace policy handler, which > checks based on old table, the table should be bootstrapped and event should > be ignored. But rename handler should decide based on new name.The old table > name will not be returned by get-all-table, so replace handler will not d > anything for the old table. > ### New name not matching old policy > New name not matching new policy > * As the old table is not there at target and new name is not matching > new policy. Ignore the event. > * No need to add the table to the list of tables to be bootstrapped. > * All the subsequent events will be ignored as the new name is not > matching the new policy. > New name matching new policy > * As the new name is not matching old policy but matching new policy, > the table will be bootstrapped by replace policy handler. So rename event > need not add this table to list of table to be bootstrapped. > * All the future events will be ignored by replace policy handler. > * For rename event (double rename) > ** If there is a rename, the table (with intermittent new name) will not > be present and thus replace handler will not bootstrap the table. > ** So if the new name (the latest one) is matching the new policy, then > add it to the list of table to be bootstrapped. > ** And If the new name (the latest one) is not matching the new policy, > then just ignore the event as the intermittent new name would no
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[ https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268229&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268229 ] ASF GitHub Bot logged work on HIVE-21886: - Author: ASF GitHub Bot Created on: 27/Jun/19 05:10 Start Date: 27/Jun/19 05:10 Worklog Time Spent: 10m Work Description: maheshk114 commented on pull request #688: HIVE-21886 : REPL - With table list - Handle rename events during replace policy URL: https://github.com/apache/hive/pull/688#discussion_r298008044 ## File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java ## @@ -92,16 +93,40 @@ private Scenario scenarioType(org.apache.hadoop.hive.metastore.api.Table before, } } - // return true, if event needs to be dumped, else return false. - private boolean handleForTableLevelReplication(Context withinContext) { -String oldName = before.getTableName(); -String newName = after.getTableName(); + private boolean isSetForBootstrapByReplaceHandler(Context withinContext, String tblName) { +return (withinContext.oldReplScope != null) +&& !(ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName)) +&& (ReplUtils.tableIncludedInReplScope(withinContext.replScope, tblName)); + } -if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) { - // If the table is renamed after being added to the list of tables to be bootstrapped, then remove it from the - // list of tables to be bootstrapped. - boolean oldTableIsPresent = withinContext.removeFromListOfTablesForBootstrap(before.getTableName()); + // Return true, if event needs to be dumped, else return false. + private boolean handleForTableLevelReplication(Context withinContext, String tblName) { +// For alter, if the table does not satisfy the new policy then ignore the event. In case of replace +// policy, if the table does not satisfy the old policy, then ignore the event. As, if the table satisfy the new +// policy, then the table will be bootstrapped by replace handler anb if the table does not satisfy the new policy, +// then anyways the table should be ignored. +if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope, tblName)) { + // In case of replace, it will be dropped during load. In normal case just ignore the alter event. + LOG.debug("Table " + tblName + " does not satisfy the policy"); + return false; +} else if ((withinContext.oldReplScope != null) +&& (!ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName))) { + LOG.debug("Table " + tblName + " is set for bootstrap"); + return false; +} else { + // Table satisfies both old (if its there) and current policy, dump the alter event. + return true; +} + } + // Return true, if event needs to be dumped, else return false. + private boolean handleForTableLevelReplicationForRename(Context withinContext, String oldName, String newName) { Review comment: new Jira is created This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268229) Time Spent: 9h 20m (was: 9h 10m) > REPL - With table list - Handle rename events during replace policy > --- > > Key: HIVE-21886 > URL: https://issues.apache.org/jira/browse/HIVE-21886 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch > > Time Spent: 9h 20m > Remaining Estimate: 0h > > If some rename events are found to be dumped and replayed while replace > policy is getting executed, it needs to take care of the policy inclusion in > both the policy for each table name. > 1. Create a list of tables to be bootstrapped. > 2. During handling of alter table, if the alter type is rename > 1. If the old table name is present in the list of table to be > bootstrapped, remove it. > 2. If the new table name, matches the new policy, add it to the list > of tables to be bootstrapped. > 3. During handling of drop table > 1. if the table is in the list of tables to be bootstrapped, then > remove it and ignore the event. > 4. During other event handling
[jira] [Updated] (HIVE-21926) CLONE - REPL - With table list - "TO" and "FROM" clause should not be allowed along with table filter list
[ https://issues.apache.org/jira/browse/HIVE-21926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-21926: --- Description: (was: If some rename events are found to be dumped and replayed while replace policy is getting executed, it needs to take care of the policy inclusion in both the policy for each table name. 1. Create a list of tables to be bootstrapped. 2. During handling of alter table, if the alter type is rename 1. If the old table name is present in the list of table to be bootstrapped, remove it. 2. If the new table name, matches the new policy, add it to the list of tables to be bootstrapped. 3. During handling of drop table 1. if the table is in the list of tables to be bootstrapped, then remove it and ignore the event. 4. During other event handling 1. if the table is there in the list of tables to be bootstrapped, then ignore the event. Rename handling during replace policy # Old name not matching old policy – The old table will not be there at the target cluster. The table will not be returned by get-all-table. ## Old name is not matching new policy ### New name not matching old policy New name not matching new policy * Ignore the event, no need to do anything. New name matching new policy * The table will be returned by get-all-table. Replace policy handler will bootstrap this table as its matching new policy and not matching old policy. * All the future events will be ignored as part of check added by replace policy handling. * All the event with old table name will anyways be ignored as the old name is not matching the new policy. ### New name matching old policy New name not matching new policy * As the new name is not matching the new policy, the table need not be replicated. * As the old name is not matching the new policy, the rename events will be ignored. * So nothing to be done for this scenario. New name matching new policy * As the new name is matching both old and new policy, replace handler will not bootstrap the table. * Add the table to the list of tables to be bootstrapped. * Ignore all the events with new name. * If there is a drop event for the table (with new name), then remove the table from the the list of table to be bootstrapped. * In case of rename event (double rename) ** If the new name satisfies the table pattern, then add the new name to the list of tables to be bootstrapped and remove the old name from the list of tables to be bootstrapped. ** If the new name does not satisfies then just removed the table name from the list of tables to be bootstrapped. ## Old name is matching new policy – As per replace policy handler, which checks based on old table, the table should be bootstrapped and event should be ignored. But rename handler should decide based on new name.The old table name will not be returned by get-all-table, so replace handler will not d anything for the old table. ### New name not matching old policy New name not matching new policy * As the old table is not there at target and new name is not matching new policy. Ignore the event. * No need to add the table to the list of tables to be bootstrapped. * All the subsequent events will be ignored as the new name is not matching the new policy. New name matching new policy * As the new name is not matching old policy but matching new policy, the table will be bootstrapped by replace policy handler. So rename event need not add this table to list of table to be bootstrapped. * All the future events will be ignored by replace policy handler. * For rename event (double rename) ** If there is a rename, the table (with intermittent new name) will not be present and thus replace handler will not bootstrap the table. ** So if the new name (the latest one) is matching the new policy, then add it to the list of table to be bootstrapped. ** And If the new name (the latest one) is not matching the new policy, then just ignore the event as the intermittent new name would not have added to the list of table to be bootstrapped. ### New name matching old policy New name not matching new policy * Dump the event. The table will be dropped by repl load at the target. New name matching new policy * Replace handler will not bootstrap this table as the new name is matching both policies. * As old name is not matching the old policy, the table will not be there at target. The rename event should add the new name to the list of table to be bootstrapped. * Subsequent events with new table name should be ignored. * Drop events should not be ignored as if the table is present during bootstrapped, then its a new table and thus should be dropped.
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[ https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268230&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268230 ] ASF GitHub Bot logged work on HIVE-21886: - Author: ASF GitHub Bot Created on: 27/Jun/19 05:10 Start Date: 27/Jun/19 05:10 Worklog Time Spent: 10m Work Description: maheshk114 commented on pull request #688: HIVE-21886 : REPL - With table list - Handle rename events during replace policy URL: https://github.com/apache/hive/pull/688#discussion_r298008064 ## File path: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestTableLevelReplicationScenarios.java ## @@ -196,27 +196,27 @@ private void verifyBootstrapDirInIncrementalDump(String dumpLocation, String[] b @Test public void testBasicBootstrapWithIncludeList() throws Throwable { -String[] originalNonAcidTables = new String[] {"t1", "t2" }; -String[] originalFullAcidTables = new String[] {"t3", "t4" }; -String[] originalMMAcidTables = new String[] {"t5" }; +String[] originalNonAcidTables = new String[] {"t1", "t2"}; +String[] originalFullAcidTables = new String[] {"t3", "t4"}; +String[] originalMMAcidTables = new String[] {"t5"}; createTables(originalNonAcidTables, CreateTableType.NON_ACID); createTables(originalFullAcidTables, CreateTableType.FULL_ACID); createTables(originalMMAcidTables, CreateTableType.MM_ACID); // Replicate and verify if only 2 tables are replicated to target. String replPolicy = primaryDbName + ".['t1', 't4', 't5']"; -String[] replicatedTables = new String[] {"t1", "t4", "t5" }; +String[] replicatedTables = new String[] {"t1", "t4", "t5"}; replicateAndVerify(replPolicy, null, null, null, replicatedTables); } @Test public void testBasicBootstrapWithIncludeAndExcludeList() throws Throwable { -String[] originalTables = new String[] {"t1", "t11", "t2", "t3", "t100" }; +String[] originalTables = new String[] {"t1", "t11", "t2", "t3", "t100"}; Review comment: done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268230) Time Spent: 9.5h (was: 9h 20m) > REPL - With table list - Handle rename events during replace policy > --- > > Key: HIVE-21886 > URL: https://issues.apache.org/jira/browse/HIVE-21886 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch > > Time Spent: 9.5h > Remaining Estimate: 0h > > If some rename events are found to be dumped and replayed while replace > policy is getting executed, it needs to take care of the policy inclusion in > both the policy for each table name. > 1. Create a list of tables to be bootstrapped. > 2. During handling of alter table, if the alter type is rename > 1. If the old table name is present in the list of table to be > bootstrapped, remove it. > 2. If the new table name, matches the new policy, add it to the list > of tables to be bootstrapped. > 3. During handling of drop table > 1. if the table is in the list of tables to be bootstrapped, then > remove it and ignore the event. > 4. During other event handling > 1. if the table is there in the list of tables to be bootstrapped, > then ignore the event. > > Rename handling during replace policy > # Old name not matching old policy – The old table will not be there at the > target cluster. The table will not be returned by get-all-table. > ## Old name is not matching new policy > ### New name not matching old policy > New name not matching new policy > * Ignore the event, no need to do anything. > New name matching new policy > * The table will be returned by get-all-table. Replace policy handler > will bootstrap this table as its matching new policy and not matching old > policy. > * All the future events will be ignored as part of check added by > replace policy handling. > * All the event with old table name will anyways be ignored as the old > name is not matching the new policy. > ### New name matching old policy > New name not matching new policy > * As the new name is not matching the new policy, the table need not be > replicated. >
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[ https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268226&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268226 ] ASF GitHub Bot logged work on HIVE-21886: - Author: ASF GitHub Bot Created on: 27/Jun/19 05:08 Start Date: 27/Jun/19 05:08 Worklog Time Spent: 10m Work Description: maheshk114 commented on pull request #688: HIVE-21886 : REPL - With table list - Handle rename events during replace policy URL: https://github.com/apache/hive/pull/688#discussion_r298007695 ## File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java ## @@ -123,8 +148,21 @@ private boolean handleForTableLevelReplication(Context withinContext) { return false; } + // All the subsequent events on this table newName are going to be skipped, so the rename is also skipped. Review comment: code removed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268226) Time Spent: 9h 10m (was: 9h) > REPL - With table list - Handle rename events during replace policy > --- > > Key: HIVE-21886 > URL: https://issues.apache.org/jira/browse/HIVE-21886 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch > > Time Spent: 9h 10m > Remaining Estimate: 0h > > If some rename events are found to be dumped and replayed while replace > policy is getting executed, it needs to take care of the policy inclusion in > both the policy for each table name. > 1. Create a list of tables to be bootstrapped. > 2. During handling of alter table, if the alter type is rename > 1. If the old table name is present in the list of table to be > bootstrapped, remove it. > 2. If the new table name, matches the new policy, add it to the list > of tables to be bootstrapped. > 3. During handling of drop table > 1. if the table is in the list of tables to be bootstrapped, then > remove it and ignore the event. > 4. During other event handling > 1. if the table is there in the list of tables to be bootstrapped, > then ignore the event. > > Rename handling during replace policy > # Old name not matching old policy – The old table will not be there at the > target cluster. The table will not be returned by get-all-table. > ## Old name is not matching new policy > ### New name not matching old policy > New name not matching new policy > * Ignore the event, no need to do anything. > New name matching new policy > * The table will be returned by get-all-table. Replace policy handler > will bootstrap this table as its matching new policy and not matching old > policy. > * All the future events will be ignored as part of check added by > replace policy handling. > * All the event with old table name will anyways be ignored as the old > name is not matching the new policy. > ### New name matching old policy > New name not matching new policy > * As the new name is not matching the new policy, the table need not be > replicated. > * As the old name is not matching the new policy, the rename events will > be ignored. > * So nothing to be done for this scenario. > New name matching new policy > * As the new name is matching both old and new policy, replace handler > will not bootstrap the table. > * Add the table to the list of tables to be bootstrapped. > * Ignore all the events with new name. > * If there is a drop event for the table (with new name), then remove > the table from the the list of table to be bootstrapped. > * In case of rename event (double rename) > ** If the new name satisfies the table pattern, then add the new name to > the list of tables to be bootstrapped and remove the old name from the list > of tables to be bootstrapped. > ** If the new name does not satisfies then just removed the table name > from the list of tables to be bootstrapped. > ## Old name is matching new policy – As per replace policy handler, which > checks based on old table, the table should be bootstrapped and event should > be ignored. But rename handler should decide based on new name.The old
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[ https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268225&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268225 ] ASF GitHub Bot logged work on HIVE-21886: - Author: ASF GitHub Bot Created on: 27/Jun/19 05:08 Start Date: 27/Jun/19 05:08 Worklog Time Spent: 10m Work Description: maheshk114 commented on pull request #688: HIVE-21886 : REPL - With table list - Handle rename events during replace policy URL: https://github.com/apache/hive/pull/688#discussion_r298007642 ## File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java ## @@ -92,16 +93,40 @@ private Scenario scenarioType(org.apache.hadoop.hive.metastore.api.Table before, } } - // return true, if event needs to be dumped, else return false. - private boolean handleForTableLevelReplication(Context withinContext) { -String oldName = before.getTableName(); -String newName = after.getTableName(); + private boolean isSetForBootstrapByReplaceHandler(Context withinContext, String tblName) { +return (withinContext.oldReplScope != null) +&& !(ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName)) +&& (ReplUtils.tableIncludedInReplScope(withinContext.replScope, tblName)); + } -if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) { - // If the table is renamed after being added to the list of tables to be bootstrapped, then remove it from the - // list of tables to be bootstrapped. - boolean oldTableIsPresent = withinContext.removeFromListOfTablesForBootstrap(before.getTableName()); + // Return true, if event needs to be dumped, else return false. + private boolean handleForTableLevelReplication(Context withinContext, String tblName) { Review comment: code removed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268225) Time Spent: 9h (was: 8h 50m) > REPL - With table list - Handle rename events during replace policy > --- > > Key: HIVE-21886 > URL: https://issues.apache.org/jira/browse/HIVE-21886 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch > > Time Spent: 9h > Remaining Estimate: 0h > > If some rename events are found to be dumped and replayed while replace > policy is getting executed, it needs to take care of the policy inclusion in > both the policy for each table name. > 1. Create a list of tables to be bootstrapped. > 2. During handling of alter table, if the alter type is rename > 1. If the old table name is present in the list of table to be > bootstrapped, remove it. > 2. If the new table name, matches the new policy, add it to the list > of tables to be bootstrapped. > 3. During handling of drop table > 1. if the table is in the list of tables to be bootstrapped, then > remove it and ignore the event. > 4. During other event handling > 1. if the table is there in the list of tables to be bootstrapped, > then ignore the event. > > Rename handling during replace policy > # Old name not matching old policy – The old table will not be there at the > target cluster. The table will not be returned by get-all-table. > ## Old name is not matching new policy > ### New name not matching old policy > New name not matching new policy > * Ignore the event, no need to do anything. > New name matching new policy > * The table will be returned by get-all-table. Replace policy handler > will bootstrap this table as its matching new policy and not matching old > policy. > * All the future events will be ignored as part of check added by > replace policy handling. > * All the event with old table name will anyways be ignored as the old > name is not matching the new policy. > ### New name matching old policy > New name not matching new policy > * As the new name is not matching the new policy, the table need not be > replicated. > * As the old name is not matching the new policy, the rename events will > be ignored. > * So nothing to be done for this scenario. > New name matching new policy > * As
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[ https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268224&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268224 ] ASF GitHub Bot logged work on HIVE-21886: - Author: ASF GitHub Bot Created on: 27/Jun/19 05:07 Start Date: 27/Jun/19 05:07 Worklog Time Spent: 10m Work Description: maheshk114 commented on pull request #688: HIVE-21886 : REPL - With table list - Handle rename events during replace policy URL: https://github.com/apache/hive/pull/688#discussion_r298007570 ## File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java ## @@ -92,16 +93,40 @@ private Scenario scenarioType(org.apache.hadoop.hive.metastore.api.Table before, } } - // return true, if event needs to be dumped, else return false. - private boolean handleForTableLevelReplication(Context withinContext) { -String oldName = before.getTableName(); -String newName = after.getTableName(); + private boolean isSetForBootstrapByReplaceHandler(Context withinContext, String tblName) { Review comment: code removed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268224) Time Spent: 8h 50m (was: 8h 40m) > REPL - With table list - Handle rename events during replace policy > --- > > Key: HIVE-21886 > URL: https://issues.apache.org/jira/browse/HIVE-21886 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch > > Time Spent: 8h 50m > Remaining Estimate: 0h > > If some rename events are found to be dumped and replayed while replace > policy is getting executed, it needs to take care of the policy inclusion in > both the policy for each table name. > 1. Create a list of tables to be bootstrapped. > 2. During handling of alter table, if the alter type is rename > 1. If the old table name is present in the list of table to be > bootstrapped, remove it. > 2. If the new table name, matches the new policy, add it to the list > of tables to be bootstrapped. > 3. During handling of drop table > 1. if the table is in the list of tables to be bootstrapped, then > remove it and ignore the event. > 4. During other event handling > 1. if the table is there in the list of tables to be bootstrapped, > then ignore the event. > > Rename handling during replace policy > # Old name not matching old policy – The old table will not be there at the > target cluster. The table will not be returned by get-all-table. > ## Old name is not matching new policy > ### New name not matching old policy > New name not matching new policy > * Ignore the event, no need to do anything. > New name matching new policy > * The table will be returned by get-all-table. Replace policy handler > will bootstrap this table as its matching new policy and not matching old > policy. > * All the future events will be ignored as part of check added by > replace policy handling. > * All the event with old table name will anyways be ignored as the old > name is not matching the new policy. > ### New name matching old policy > New name not matching new policy > * As the new name is not matching the new policy, the table need not be > replicated. > * As the old name is not matching the new policy, the rename events will > be ignored. > * So nothing to be done for this scenario. > New name matching new policy > * As the new name is matching both old and new policy, replace handler > will not bootstrap the table. > * Add the table to the list of tables to be bootstrapped. > * Ignore all the events with new name. > * If there is a drop event for the table (with new name), then remove > the table from the the list of table to be bootstrapped. > * In case of rename event (double rename) > ** If the new name satisfies the table pattern, then add the new name to > the list of tables to be bootstrapped and remove the old name from the list > of tables to be bootstrapped. > ** If the new name does not satisfies then just removed the table name > from the list of tables to be bootstrapped. > ## Old name is matching n
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[ https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268222&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268222 ] ASF GitHub Bot logged work on HIVE-21886: - Author: ASF GitHub Bot Created on: 27/Jun/19 05:06 Start Date: 27/Jun/19 05:06 Worklog Time Spent: 10m Work Description: maheshk114 commented on pull request #688: HIVE-21886 : REPL - With table list - Handle rename events during replace policy URL: https://github.com/apache/hive/pull/688#discussion_r298007457 ## File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java ## @@ -120,18 +121,18 @@ private boolean handleForTableLevelReplication(Context withinContext, String tbl } // Return true, if event needs to be dumped, else return false. - private boolean handleForTableLevelReplicationForRename(Context withinContext, String oldName, String newName) { + private boolean renameHandlerForTableLevelReplication(Context withinContext, String oldName, String newName) { // If the table is renamed after being added to the list of tables to be bootstrapped, then remove it from the // list of tables to be bootstrapped. -boolean oldTableIsPresent = withinContext.removeFromListOfTablesForBootstrap(oldName); +boolean oldTableInBootstrapList = withinContext.removeFromListOfTablesForBootstrap(oldName); ReplScope oldPolicy = withinContext.oldReplScope == null ? withinContext.replScope : withinContext.oldReplScope; if (ReplUtils.tableIncludedInReplScope(oldPolicy, oldName)) { // If old table satisfies the filter, but the new table does not, then the old table should be dropped. // This should be done, only if the old table is not in the list of tables to be bootstrapped which is a multi // rename case. In case of multi rename, only the first rename should do the drop. if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope, newName)) { -if (oldTableIsPresent) { +if (oldTableInBootstrapList) { // If the old table was present in the list of tables to be bootstrapped, then just ignore the event. return false; } else { Review comment: there is no issue if C doesn't match new policy. As drop is idempotent. But the real issue is when C match new policy. The rename event will get dumped and that will cause data loss as B is not present at target. I have modified the code to avoid all these complexity. Now if old table does not satisfy the old policy then dump drop event(although that is also not required). If new policy matches new name then add it to the bootstrap. So now for rename with replace policy its always bootstrap. Thats will make the code simple and as replace and rename are not very frequently executed operations, it should be fine. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268222) Time Spent: 8h 40m (was: 8.5h) > REPL - With table list - Handle rename events during replace policy > --- > > Key: HIVE-21886 > URL: https://issues.apache.org/jira/browse/HIVE-21886 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch > > Time Spent: 8h 40m > Remaining Estimate: 0h > > If some rename events are found to be dumped and replayed while replace > policy is getting executed, it needs to take care of the policy inclusion in > both the policy for each table name. > 1. Create a list of tables to be bootstrapped. > 2. During handling of alter table, if the alter type is rename > 1. If the old table name is present in the list of table to be > bootstrapped, remove it. > 2. If the new table name, matches the new policy, add it to the list > of tables to be bootstrapped. > 3. During handling of drop table > 1. if the table is in the list of tables to be bootstrapped, then > remove it and ignore the event. > 4. During other event handling > 1. if the table is there in the list of tables to be bootstrapped, > then ignore the event. > > Rename handling during replace policy > # Old name not matching old policy – The old table will not be there at the > target cl
[jira] [Commented] (HIVE-21925) HiveConnection retries should support backoff
[ https://issues.apache.org/jira/browse/HIVE-21925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873795#comment-16873795 ] Hive QA commented on HIVE-21925: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12973018/HIVE-21925.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 16341 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/17764/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17764/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17764/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12973018 - PreCommit-HIVE-Build > HiveConnection retries should support backoff > - > > Key: HIVE-21925 > URL: https://issues.apache.org/jira/browse/HIVE-21925 > Project: Hive > Issue Type: Bug > Components: Clients >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-21925.patch > > > Hive JDBC connection supports retries. In http mode, retries always seem to > happen immediately without any backoff. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-14737) Problem accessing /logs in a Kerberized Hive Server 2 Web UI
[ https://issues.apache.org/jira/browse/HIVE-14737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-14737: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Patch pushed to master. Thanks Rajkumar! > Problem accessing /logs in a Kerberized Hive Server 2 Web UI > > > Key: HIVE-14737 > URL: https://issues.apache.org/jira/browse/HIVE-14737 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0, 3.1.1 >Reporter: Matyas Orhidi >Assignee: Rajkumar Singh >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-14737.01.patch, HIVE-14737.02.patch, > HIVE-14737.03.patch, HIVE-14737.04.patch, HIVE-14737.05.patch, > HIVE-14737.patch > > > The /logs menu fails with error [1] when the cluster is Kerberized. Other > menu items are working properly. > [1] HTTP ERROR: 401 > Problem accessing /logs/. Reason: > Unauthenticated users are not authorized to access this page. > Powered by Jetty:// -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-14737) Problem accessing /logs in a Kerberized Hive Server 2 Web UI
[ https://issues.apache.org/jira/browse/HIVE-14737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh updated HIVE-14737: -- Attachment: HIVE-14737.05.patch Status: Patch Available (was: Open) > Problem accessing /logs in a Kerberized Hive Server 2 Web UI > > > Key: HIVE-14737 > URL: https://issues.apache.org/jira/browse/HIVE-14737 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.1, 1.1.0 >Reporter: Matyas Orhidi >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-14737.01.patch, HIVE-14737.02.patch, > HIVE-14737.03.patch, HIVE-14737.04.patch, HIVE-14737.05.patch, > HIVE-14737.patch > > > The /logs menu fails with error [1] when the cluster is Kerberized. Other > menu items are working properly. > [1] HTTP ERROR: 401 > Problem accessing /logs/. Reason: > Unauthenticated users are not authorized to access this page. > Powered by Jetty:// -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-14737) Problem accessing /logs in a Kerberized Hive Server 2 Web UI
[ https://issues.apache.org/jira/browse/HIVE-14737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh updated HIVE-14737: -- Status: Open (was: Patch Available) > Problem accessing /logs in a Kerberized Hive Server 2 Web UI > > > Key: HIVE-14737 > URL: https://issues.apache.org/jira/browse/HIVE-14737 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.1, 1.1.0 >Reporter: Matyas Orhidi >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-14737.01.patch, HIVE-14737.02.patch, > HIVE-14737.03.patch, HIVE-14737.04.patch, HIVE-14737.patch > > > The /logs menu fails with error [1] when the cluster is Kerberized. Other > menu items are working properly. > [1] HTTP ERROR: 401 > Problem accessing /logs/. Reason: > Unauthenticated users are not authorized to access this page. > Powered by Jetty:// -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21925) HiveConnection retries should support backoff
[ https://issues.apache.org/jira/browse/HIVE-21925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873780#comment-16873780 ] Hive QA commented on HIVE-21925: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 1s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 39s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 12s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 28s{color} | {color:blue} jdbc in master has 16 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 38s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 20s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 12s{color} | {color:red} jdbc: The patch generated 1 new + 34 unchanged - 0 fixed = 35 total (was 34) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 12m 28s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-17764/dev-support/hive-personality.sh | | git revision | master / ee3aeb7 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-17764/yetus/diff-checkstyle-jdbc.txt | | modules | C: jdbc U: jdbc | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-17764/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > HiveConnection retries should support backoff > - > > Key: HIVE-21925 > URL: https://issues.apache.org/jira/browse/HIVE-21925 > Project: Hive > Issue Type: Bug > Components: Clients >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-21925.patch > > > Hive JDBC connection supports retries. In http mode, retries always seem to > happen immediately without any backoff. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21925) HiveConnection retries should support backoff
[ https://issues.apache.org/jira/browse/HIVE-21925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh updated HIVE-21925: -- Status: Open (was: Patch Available) > HiveConnection retries should support backoff > - > > Key: HIVE-21925 > URL: https://issues.apache.org/jira/browse/HIVE-21925 > Project: Hive > Issue Type: Bug > Components: Clients >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-21925.patch > > > Hive JDBC connection supports retries. In http mode, retries always seem to > happen immediately without any backoff. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21925) HiveConnection retries should support backoff
[ https://issues.apache.org/jira/browse/HIVE-21925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh updated HIVE-21925: -- Attachment: (was: HIVE-21925.patch) > HiveConnection retries should support backoff > - > > Key: HIVE-21925 > URL: https://issues.apache.org/jira/browse/HIVE-21925 > Project: Hive > Issue Type: Bug > Components: Clients >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-21925.patch > > > Hive JDBC connection supports retries. In http mode, retries always seem to > happen immediately without any backoff. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21925) HiveConnection retries should support backoff
[ https://issues.apache.org/jira/browse/HIVE-21925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh updated HIVE-21925: -- Attachment: HIVE-21925.patch Status: Patch Available (was: Open) > HiveConnection retries should support backoff > - > > Key: HIVE-21925 > URL: https://issues.apache.org/jira/browse/HIVE-21925 > Project: Hive > Issue Type: Bug > Components: Clients >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-21925.patch > > > Hive JDBC connection supports retries. In http mode, retries always seem to > happen immediately without any backoff. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21925) HiveConnection retries should support backoff
[ https://issues.apache.org/jira/browse/HIVE-21925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873763#comment-16873763 ] Hive QA commented on HIVE-21925: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12973010/HIVE-21925.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/17763/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17763/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17763/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2019-06-27 02:17:18.827 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-17763/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2019-06-27 02:17:18.831 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at ee3aeb7 HIVE-21752 Thread Safety and Memory Leaks in HCatRecordObjectInspectorFactory (Jalpan Randeri via Alan Gates) + git clean -f -d Removing ${project.basedir}/ Removing itests/${project.basedir}/ Removing standalone-metastore/metastore-server/src/gen/ + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at ee3aeb7 HIVE-21752 Thread Safety and Memory Leaks in HCatRecordObjectInspectorFactory (Jalpan Randeri via Alan Gates) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2019-06-27 02:17:19.661 + rm -rf ../yetus_PreCommit-HIVE-Build-17763 + mkdir ../yetus_PreCommit-HIVE-Build-17763 + git gc + cp -R . ../yetus_PreCommit-HIVE-Build-17763 + mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-17763/yetus + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch error: a/jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java: does not exist in index Going to apply patch with: git apply -p1 + [[ maven == \m\a\v\e\n ]] + rm -rf /data/hiveptest/working/maven/org/apache/hive + mvn -B clean install -DskipTests -T 4 -q -Dmaven.repo.local=/data/hiveptest/working/maven protoc-jar: executing: [/tmp/protoc4559357827667831756.exe, --version] libprotoc 2.5.0 protoc-jar: executing: [/tmp/protoc4559357827667831756.exe, -I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore, --java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources, /data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto] ANTLR Parser Generator Version 3.5.2 protoc-jar: executing: [/tmp/protoc7682582718199574526.exe, --version] libprotoc 2.5.0 ANTLR Parser Generator Version 3.5.2 Output file /data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/target/generated-sources/org/apache/hadoop/hive/metastore/parser/FilterParser.java does not exist: must build /data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/parser/Filter.g org/apache/hadoop/hive/metastore/parser/Filter.g log4j:WARN No appenders could be found for logger (DataNucleus.Persistence). log4j:WARN Please initialize the log4j system properly. DataNucleus Enhancer (version 4.1.17) for API "JDO" DataNucleus Enhancer completed with success for 41 classes. ANTLR Parser Generator Version 3.5.2 Output file /data/hiveptest/working/apache-github-source-source/ql/target/generated-sources/ant
[jira] [Commented] (HIVE-14737) Problem accessing /logs in a Kerberized Hive Server 2 Web UI
[ https://issues.apache.org/jira/browse/HIVE-14737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873761#comment-16873761 ] Hive QA commented on HIVE-14737: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12973009/HIVE-14737.04.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/17762/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17762/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17762/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Tests exited with: Exception: Patch URL https://issues.apache.org/jira/secure/attachment/12973009/HIVE-14737.04.patch was found in seen patch url's cache and a test was probably run already on it. Aborting... {noformat} This message is automatically generated. ATTACHMENT ID: 12973009 - PreCommit-HIVE-Build > Problem accessing /logs in a Kerberized Hive Server 2 Web UI > > > Key: HIVE-14737 > URL: https://issues.apache.org/jira/browse/HIVE-14737 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0, 3.1.1 >Reporter: Matyas Orhidi >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-14737.01.patch, HIVE-14737.02.patch, > HIVE-14737.03.patch, HIVE-14737.04.patch, HIVE-14737.patch > > > The /logs menu fails with error [1] when the cluster is Kerberized. Other > menu items are working properly. > [1] HTTP ERROR: 401 > Problem accessing /logs/. Reason: > Unauthenticated users are not authorized to access this page. > Powered by Jetty:// -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-14737) Problem accessing /logs in a Kerberized Hive Server 2 Web UI
[ https://issues.apache.org/jira/browse/HIVE-14737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873760#comment-16873760 ] Hive QA commented on HIVE-14737: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12973009/HIVE-14737.04.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 16341 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/17761/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17761/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17761/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12973009 - PreCommit-HIVE-Build > Problem accessing /logs in a Kerberized Hive Server 2 Web UI > > > Key: HIVE-14737 > URL: https://issues.apache.org/jira/browse/HIVE-14737 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0, 3.1.1 >Reporter: Matyas Orhidi >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-14737.01.patch, HIVE-14737.02.patch, > HIVE-14737.03.patch, HIVE-14737.04.patch, HIVE-14737.patch > > > The /logs menu fails with error [1] when the cluster is Kerberized. Other > menu items are working properly. > [1] HTTP ERROR: 401 > Problem accessing /logs/. Reason: > Unauthenticated users are not authorized to access this page. > Powered by Jetty:// -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-14737) Problem accessing /logs in a Kerberized Hive Server 2 Web UI
[ https://issues.apache.org/jira/browse/HIVE-14737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873737#comment-16873737 ] Hive QA commented on HIVE-14737: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 52s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 13s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 35s{color} | {color:blue} common in master has 62 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 18s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 18s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 18s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 12s{color} | {color:red} common: The patch generated 1 new + 22 unchanged - 0 fixed = 23 total (was 22) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 38s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 12m 30s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-17761/dev-support/hive-personality.sh | | git revision | master / ee3aeb7 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-17761/yetus/diff-checkstyle-common.txt | | modules | C: common U: common | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-17761/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Problem accessing /logs in a Kerberized Hive Server 2 Web UI > > > Key: HIVE-14737 > URL: https://issues.apache.org/jira/browse/HIVE-14737 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0, 3.1.1 >Reporter: Matyas Orhidi >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-14737.01.patch, HIVE-14737.02.patch, > HIVE-14737.03.patch, HIVE-14737.04.patch, HIVE-14737.patch > > > The /logs menu fails with error [1] when the cluster is Kerberized. Other > menu items are working properly. > [1] HTTP ERROR: 401 > Problem accessing /logs/. Reason: > Unauthenticated users are not authorized to access this page. > Powered by Jetty:// -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21921) Support for correlated quantified predicates
[ https://issues.apache.org/jira/browse/HIVE-21921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873728#comment-16873728 ] Hive QA commented on HIVE-21921: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12973002/HIVE-21921.4.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/17760/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17760/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17760/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Tests exited with: Exception: Patch URL https://issues.apache.org/jira/secure/attachment/12973002/HIVE-21921.4.patch was found in seen patch url's cache and a test was probably run already on it. Aborting... {noformat} This message is automatically generated. ATTACHMENT ID: 12973002 - PreCommit-HIVE-Build > Support for correlated quantified predicates > > > Key: HIVE-21921 > URL: https://issues.apache.org/jira/browse/HIVE-21921 > Project: Hive > Issue Type: New Feature > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-21921.1.patch, HIVE-21921.2.patch, > HIVE-21921.3.patch, HIVE-21921.4.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21921) Support for correlated quantified predicates
[ https://issues.apache.org/jira/browse/HIVE-21921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873727#comment-16873727 ] Hive QA commented on HIVE-21921: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12973002/HIVE-21921.4.patch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 16342 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/17759/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17759/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17759/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12973002 - PreCommit-HIVE-Build > Support for correlated quantified predicates > > > Key: HIVE-21921 > URL: https://issues.apache.org/jira/browse/HIVE-21921 > Project: Hive > Issue Type: New Feature > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-21921.1.patch, HIVE-21921.2.patch, > HIVE-21921.3.patch, HIVE-21921.4.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21925) HiveConnection retries should support backoff
[ https://issues.apache.org/jira/browse/HIVE-21925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh updated HIVE-21925: -- Attachment: HIVE-21925.patch Status: Patch Available (was: In Progress) > HiveConnection retries should support backoff > - > > Key: HIVE-21925 > URL: https://issues.apache.org/jira/browse/HIVE-21925 > Project: Hive > Issue Type: Bug > Components: Clients >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-21925.patch > > > Hive JDBC connection supports retries. In http mode, retries always seem to > happen immediately without any backoff. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-14737) Problem accessing /logs in a Kerberized Hive Server 2 Web UI
[ https://issues.apache.org/jira/browse/HIVE-14737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh updated HIVE-14737: -- Affects Version/s: 3.1.1 Attachment: HIVE-14737.04.patch Status: Patch Available (was: Open) > Problem accessing /logs in a Kerberized Hive Server 2 Web UI > > > Key: HIVE-14737 > URL: https://issues.apache.org/jira/browse/HIVE-14737 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.1, 1.1.0 >Reporter: Matyas Orhidi >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-14737.01.patch, HIVE-14737.02.patch, > HIVE-14737.03.patch, HIVE-14737.04.patch, HIVE-14737.patch > > > The /logs menu fails with error [1] when the cluster is Kerberized. Other > menu items are working properly. > [1] HTTP ERROR: 401 > Problem accessing /logs/. Reason: > Unauthenticated users are not authorized to access this page. > Powered by Jetty:// -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21925) HiveConnection retries should support backoff
[ https://issues.apache.org/jira/browse/HIVE-21925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh reassigned HIVE-21925: - Assignee: Rajkumar Singh > HiveConnection retries should support backoff > - > > Key: HIVE-21925 > URL: https://issues.apache.org/jira/browse/HIVE-21925 > Project: Hive > Issue Type: Bug > Components: Clients >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Rajkumar Singh >Priority: Major > > Hive JDBC connection supports retries. In http mode, retries always seem to > happen immediately without any backoff. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-21925) HiveConnection retries should support backoff
[ https://issues.apache.org/jira/browse/HIVE-21925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-21925 started by Rajkumar Singh. - > HiveConnection retries should support backoff > - > > Key: HIVE-21925 > URL: https://issues.apache.org/jira/browse/HIVE-21925 > Project: Hive > Issue Type: Bug > Components: Clients >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Rajkumar Singh >Priority: Major > > Hive JDBC connection supports retries. In http mode, retries always seem to > happen immediately without any backoff. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21921) Support for correlated quantified predicates
[ https://issues.apache.org/jira/browse/HIVE-21921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873715#comment-16873715 ] Hive QA commented on HIVE-21921: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 42s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 6s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 40s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m 5s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 0s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 5s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 5s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 41s{color} | {color:red} ql: The patch generated 19 new + 153 unchanged - 1 fixed = 172 total (was 154) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 18s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 24m 48s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-17759/dev-support/hive-personality.sh | | git revision | master / ee3aeb7 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-17759/yetus/diff-checkstyle-ql.txt | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-17759/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Support for correlated quantified predicates > > > Key: HIVE-21921 > URL: https://issues.apache.org/jira/browse/HIVE-21921 > Project: Hive > Issue Type: New Feature > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-21921.1.patch, HIVE-21921.2.patch, > HIVE-21921.3.patch, HIVE-21921.4.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21888) Set hive.parquet.timestamp.skip.conversion default to true
[ https://issues.apache.org/jira/browse/HIVE-21888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873700#comment-16873700 ] Hive QA commented on HIVE-21888: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12972988/HIVE-21888.02.patch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 16341 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.testMTT[2] (batchId=350) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/17758/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17758/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17758/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12972988 - PreCommit-HIVE-Build > Set hive.parquet.timestamp.skip.conversion default to true > -- > > Key: HIVE-21888 > URL: https://issues.apache.org/jira/browse/HIVE-21888 > Project: Hive > Issue Type: Bug >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-21888.02.patch, HIVE-21888.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21888) Set hive.parquet.timestamp.skip.conversion default to true
[ https://issues.apache.org/jira/browse/HIVE-21888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873689#comment-16873689 ] Hive QA commented on HIVE-21888: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 54s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 13s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 25s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 57s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 35s{color} | {color:blue} common in master has 62 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m 0s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 16s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 28s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 43s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 21s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 21s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 50s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 14s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 28m 51s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-17758/dev-support/hive-personality.sh | | git revision | master / ee3aeb7 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | modules | C: common ql U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-17758/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Set hive.parquet.timestamp.skip.conversion default to true > -- > > Key: HIVE-21888 > URL: https://issues.apache.org/jira/browse/HIVE-21888 > Project: Hive > Issue Type: Bug >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-21888.02.patch, HIVE-21888.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21616) Implement JSON_VALUE, JSON_QUERY, and IS [NOT] JSON
[ https://issues.apache.org/jira/browse/HIVE-21616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873681#comment-16873681 ] Hive QA commented on HIVE-21616: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12972989/HIVE-21616.2.patch {color:green}SUCCESS:{color} +1 due to 12 test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 16639 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/17757/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17757/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17757/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12972989 - PreCommit-HIVE-Build > Implement JSON_VALUE, JSON_QUERY, and IS [NOT] JSON > --- > > Key: HIVE-21616 > URL: https://issues.apache.org/jira/browse/HIVE-21616 > Project: Hive > Issue Type: Sub-task > Components: UDF >Affects Versions: 4.0.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > Attachments: HIVE-21615.patch, HIVE-21616.2.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21616) Implement JSON_VALUE, JSON_QUERY, and IS [NOT] JSON
[ https://issues.apache.org/jira/browse/HIVE-21616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873678#comment-16873678 ] Hive QA commented on HIVE-21616: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 48s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 17s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 16s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 7s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 6m 10s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m 4s{color} | {color:blue} hplsql in master has 157 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m 37s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 27s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 50s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 1s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 52s{color} | {color:red} ql: The patch generated 682 new + 484 unchanged - 0 fixed = 1166 total (was 484) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m 18s{color} | {color:red} root: The patch generated 682 new + 484 unchanged - 0 fixed = 1166 total (was 484) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 4s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 23s{color} | {color:red} ql generated 59 new + 2247 unchanged - 6 fixed = 2306 total (was 2253) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m 49s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 75m 33s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:ql | | | Dead store to LA29_5 in org.apache.hadoop.hive.ql.parse.HiveLexer$DFA29.specialStateTransition(int, IntStream) At HiveLexer.java:org.apache.hadoop.hive.ql.parse.HiveLexer$DFA29.specialStateTransition(int, IntStream) At HiveLexer.java:[line 12962] | | | Dead store to KW_IN244 in org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceSimilarExpressionAtom(CommonTree) At HiveParser_IdentifiersParser.java:org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceSimilarExpressionAtom(CommonTree) At HiveParser_IdentifiersParser.java:[line 9701] | | | Dead store to LPAREN234 in org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.subQueryExpression() At HiveParser_IdentifiersParser.java:org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.subQueryExpression() At HiveParser_IdentifiersParser.java:[line 9203] | | | Dead store to RPAREN236 in org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.subQueryExpression() At HiveParser_IdentifiersParser.java:org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.subQueryExpression() At HiveParser_IdentifiersParser.java:[line 9210] | | | Redundant nullcheck of nonReserved314, which is known to be non-null in org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.identifier() Redundant null ch
[jira] [Updated] (HIVE-21921) Support for correlated quantified predicates
[ https://issues.apache.org/jira/browse/HIVE-21921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-21921: --- Status: Patch Available (was: Open) > Support for correlated quantified predicates > > > Key: HIVE-21921 > URL: https://issues.apache.org/jira/browse/HIVE-21921 > Project: Hive > Issue Type: New Feature > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-21921.1.patch, HIVE-21921.2.patch, > HIVE-21921.3.patch, HIVE-21921.4.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21921) Support for correlated quantified predicates
[ https://issues.apache.org/jira/browse/HIVE-21921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-21921: --- Attachment: HIVE-21921.4.patch > Support for correlated quantified predicates > > > Key: HIVE-21921 > URL: https://issues.apache.org/jira/browse/HIVE-21921 > Project: Hive > Issue Type: New Feature > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-21921.1.patch, HIVE-21921.2.patch, > HIVE-21921.3.patch, HIVE-21921.4.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21921) Support for correlated quantified predicates
[ https://issues.apache.org/jira/browse/HIVE-21921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-21921: --- Status: Open (was: Patch Available) > Support for correlated quantified predicates > > > Key: HIVE-21921 > URL: https://issues.apache.org/jira/browse/HIVE-21921 > Project: Hive > Issue Type: New Feature > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-21921.1.patch, HIVE-21921.2.patch, > HIVE-21921.3.patch, HIVE-21921.4.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21923) Vectorized MapJoin may miss results when only the join key is selected
[ https://issues.apache.org/jira/browse/HIVE-21923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873667#comment-16873667 ] Hive QA commented on HIVE-21923: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 47s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 9s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 33s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 26m 12s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 35s{color} | {color:blue} common in master has 62 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m 1s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m 46s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 26s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 48s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 48s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 34s{color} | {color:red} common: The patch generated 1 new + 1915 unchanged - 0 fixed = 1916 total (was 1915) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 20m 55s{color} | {color:red} root: The patch generated 1 new + 73196 unchanged - 0 fixed = 73197 total (was 73196) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 50s{color} | {color:red} patch/common cannot run setBugDatabaseInfo from findbugs {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 5m 31s{color} | {color:red} patch/ql cannot run setBugDatabaseInfo from findbugs {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 11m 58s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 27s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}122m 32s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-17756/dev-support/hive-personality.sh | | git revision | master / ee3aeb7 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-17756/yetus/diff-checkstyle-common.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-17756/yetus/diff-checkstyle-root.txt | | findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-17756/yetus/patch-findbugs-common.txt | | findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-17756/yetus/patch-findbugs-ql.txt | | modules | C: common ql . U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-17756/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Vectorized MapJoin may miss results when only the join key is selected > -- > > Key: HIVE-21923 > URL: https://issues.apache.org/jira/browse/HIVE-21923 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zolt
[jira] [Commented] (HIVE-21923) Vectorized MapJoin may miss results when only the join key is selected
[ https://issues.apache.org/jira/browse/HIVE-21923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873650#comment-16873650 ] Hive QA commented on HIVE-21923: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12972967/HIVE-21923.01.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 16341 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[correlationoptimizer4] (batchId=162) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[hybridgrace_hashjoin_2] (batchId=110) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join14] (batchId=118) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucket_map_join_tez1] (batchId=151) org.apache.hadoop.hive.ql.TestTxnCommandsWithSplitUpdateAndVectorization.testMergeDeleteUpdate (batchId=322) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/17756/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17756/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17756/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 5 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12972967 - PreCommit-HIVE-Build > Vectorized MapJoin may miss results when only the join key is selected > -- > > Key: HIVE-21923 > URL: https://issues.apache.org/jira/browse/HIVE-21923 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21923.01.patch > > > HIVE-21189 have introduced some resultset changes > in ql/src/test/results/clientpositive/llap/hybridgrace_hashjoin_2.q.out > https://github.com/apache/hive/commit/5799398450c17d06e8ef144ce835a8524f5abec9#diff-56b3ab96b6c90fdbebe2c4f84e8595afL500 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21868) Vectorize CAST...FORMAT
[ https://issues.apache.org/jira/browse/HIVE-21868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873597#comment-16873597 ] Hive QA commented on HIVE-21868: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12972946/HIVE-21868.05.patch {color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 16348 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/17755/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17755/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17755/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12972946 - PreCommit-HIVE-Build > Vectorize CAST...FORMAT > --- > > Key: HIVE-21868 > URL: https://issues.apache.org/jira/browse/HIVE-21868 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-21868.01.patch, HIVE-21868.01.patch, > HIVE-21868.02.patch, HIVE-21868.03.patch, HIVE-21868.04.patch, > HIVE-21868.05.patch > > > Vectorize UDFs for CAST ( AS STRING/CHAR/VARCHAR FORMAT > ) and CAST ( AS TIMESTAMP/DATE FORMAT ). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21868) Vectorize CAST...FORMAT
[ https://issues.apache.org/jira/browse/HIVE-21868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873577#comment-16873577 ] Hive QA commented on HIVE-21868: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 40s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 21s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 23s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 57s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 34s{color} | {color:blue} common in master has 62 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 59s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 12s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 28s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 12s{color} | {color:green} The patch common passed checkstyle {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 44s{color} | {color:green} ql: The patch generated 0 new + 412 unchanged - 1 fixed = 412 total (was 413) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 13s{color} | {color:red} ql generated 2 new + 2253 unchanged - 0 fixed = 2255 total (was 2253) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 12s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 28m 49s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:ql | | | Found reliance on default encoding in org.apache.hadoop.hive.ql.exec.vector.expressions.CastDateToString.sqlFormat(BytesColumnVector, long[], int, HiveSqlDateTimeFormatter):in org.apache.hadoop.hive.ql.exec.vector.expressions.CastDateToString.sqlFormat(BytesColumnVector, long[], int, HiveSqlDateTimeFormatter): String.getBytes() At CastDateToString.java:[line 70] | | | Found reliance on default encoding in org.apache.hadoop.hive.ql.exec.vector.expressions.CastTimestampToString.sqlFormat(BytesColumnVector, TimestampColumnVector, int, HiveSqlDateTimeFormatter):in org.apache.hadoop.hive.ql.exec.vector.expressions.CastTimestampToString.sqlFormat(BytesColumnVector, TimestampColumnVector, int, HiveSqlDateTimeFormatter): String.getBytes() At CastTimestampToString.java:[line 79] | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-17755/dev-support/hive-personality.sh | | git revision | master / ee3aeb7 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-17755/yetus/new-findbugs-ql.html | | modules | C: common ql U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-17755/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Vectorize CAST...FORMAT > --
[jira] [Updated] (HIVE-21921) Support for correlated quantified predicates
[ https://issues.apache.org/jira/browse/HIVE-21921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-21921: --- Status: Patch Available (was: Open) > Support for correlated quantified predicates > > > Key: HIVE-21921 > URL: https://issues.apache.org/jira/browse/HIVE-21921 > Project: Hive > Issue Type: New Feature > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-21921.1.patch, HIVE-21921.2.patch, > HIVE-21921.3.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21921) Support for correlated quantified predicates
[ https://issues.apache.org/jira/browse/HIVE-21921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-21921: --- Status: Open (was: Patch Available) > Support for correlated quantified predicates > > > Key: HIVE-21921 > URL: https://issues.apache.org/jira/browse/HIVE-21921 > Project: Hive > Issue Type: New Feature > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-21921.1.patch, HIVE-21921.2.patch, > HIVE-21921.3.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21921) Support for correlated quantified predicates
[ https://issues.apache.org/jira/browse/HIVE-21921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-21921: --- Attachment: HIVE-21921.3.patch > Support for correlated quantified predicates > > > Key: HIVE-21921 > URL: https://issues.apache.org/jira/browse/HIVE-21921 > Project: Hive > Issue Type: New Feature > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-21921.1.patch, HIVE-21921.2.patch, > HIVE-21921.3.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-14737) Problem accessing /logs in a Kerberized Hive Server 2 Web UI
[ https://issues.apache.org/jira/browse/HIVE-14737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873567#comment-16873567 ] Daniel Dai commented on HIVE-14737: --- +1 pending tests. > Problem accessing /logs in a Kerberized Hive Server 2 Web UI > > > Key: HIVE-14737 > URL: https://issues.apache.org/jira/browse/HIVE-14737 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0 >Reporter: Matyas Orhidi >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-14737.01.patch, HIVE-14737.02.patch, > HIVE-14737.03.patch, HIVE-14737.patch > > > The /logs menu fails with error [1] when the cluster is Kerberized. Other > menu items are working properly. > [1] HTTP ERROR: 401 > Problem accessing /logs/. Reason: > Unauthenticated users are not authorized to access this page. > Powered by Jetty:// -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[ https://issues.apache.org/jira/browse/HIVE-21886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873558#comment-16873558 ] Hive QA commented on HIVE-21886: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12972944/HIVE-21886.02.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 16314 tests executed *Failed tests:* {noformat} TestReplAcrossInstancesWithJsonMessageFormat - did not produce a TEST-*.xml file (likely timed out) (batchId=255) org.apache.hive.service.cli.session.TestSessionManagerMetrics.testAbandonedSessionMetrics (batchId=236) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/17754/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17754/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17754/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12972944 - PreCommit-HIVE-Build > REPL - With table list - Handle rename events during replace policy > --- > > Key: HIVE-21886 > URL: https://issues.apache.org/jira/browse/HIVE-21886 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch > > Time Spent: 8.5h > Remaining Estimate: 0h > > If some rename events are found to be dumped and replayed while replace > policy is getting executed, it needs to take care of the policy inclusion in > both the policy for each table name. > 1. Create a list of tables to be bootstrapped. > 2. During handling of alter table, if the alter type is rename > 1. If the old table name is present in the list of table to be > bootstrapped, remove it. > 2. If the new table name, matches the new policy, add it to the list > of tables to be bootstrapped. > 3. During handling of drop table > 1. if the table is in the list of tables to be bootstrapped, then > remove it and ignore the event. > 4. During other event handling > 1. if the table is there in the list of tables to be bootstrapped, > then ignore the event. > > Rename handling during replace policy > # Old name not matching old policy – The old table will not be there at the > target cluster. The table will not be returned by get-all-table. > ## Old name is not matching new policy > ### New name not matching old policy > New name not matching new policy > * Ignore the event, no need to do anything. > New name matching new policy > * The table will be returned by get-all-table. Replace policy handler > will bootstrap this table as its matching new policy and not matching old > policy. > * All the future events will be ignored as part of check added by > replace policy handling. > * All the event with old table name will anyways be ignored as the old > name is not matching the new policy. > ### New name matching old policy > New name not matching new policy > * As the new name is not matching the new policy, the table need not be > replicated. > * As the old name is not matching the new policy, the rename events will > be ignored. > * So nothing to be done for this scenario. > New name matching new policy > * As the new name is matching both old and new policy, replace handler > will not bootstrap the table. > * Add the table to the list of tables to be bootstrapped. > * Ignore all the events with new name. > * If there is a drop event for the table (with new name), then remove > the table from the the list of table to be bootstrapped. > * In case of rename event (double rename) > ** If the new name satisfies the table pattern, then add the new name to > the list of tables to be bootstrapped and remove the old name from the list > of tables to be bootstrapped. > ** If the new name does not satisfies then just removed the table name > from the list of tables to be bootstrapped. > ## Old name is matching new policy – As per replace policy handler, which > checks based on old table, the table should be bootstrapped and event should > be ignored. But rename handler should decide bas
[jira] [Work logged] (HIVE-21867) Sort semijoin conditions to accelerate query processing
[ https://issues.apache.org/jira/browse/HIVE-21867?focusedWorklogId=267802&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267802 ] ASF GitHub Bot logged work on HIVE-21867: - Author: ASF GitHub Bot Created on: 26/Jun/19 17:29 Start Date: 26/Jun/19 17:29 Worklog Time Spent: 10m Work Description: jcamachor commented on pull request #687: HIVE-21867 URL: https://github.com/apache/hive/pull/687#discussion_r297783212 ## File path: ql/src/test/results/clientpositive/llap/hybridgrace_hashjoin_2.q.out ## @@ -1421,7 +1421,7 @@ STAGE PLANS: outputColumnNames: _col1 input vertices: 1 Map 5 - Statistics: Num rows: 25 Data size: 2225 Basic stats: COMPLETE Column stats: COMPLETE + Statistics: Num rows: 4 Data size: 356 Basic stats: COMPLETE Column stats: COMPLETE Review comment: Good catch, thanks! This is unexpected indeed... Taking a look now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 267802) Time Spent: 50m (was: 40m) > Sort semijoin conditions to accelerate query processing > --- > > Key: HIVE-21867 > URL: https://issues.apache.org/jira/browse/HIVE-21867 > Project: Hive > Issue Type: New Feature > Components: Physical Optimizer >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21867.02.patch, HIVE-21867.03.patch, > HIVE-21867.patch > > Time Spent: 50m > Remaining Estimate: 0h > > The problem was tackled for CBO in HIVE-21857. Semijoin filters are > introduced later in the planning phase. Follow similar approach to sort them, > trying to accelerate filter evaluation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21867) Sort semijoin conditions to accelerate query processing
[ https://issues.apache.org/jira/browse/HIVE-21867?focusedWorklogId=267803&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267803 ] ASF GitHub Bot logged work on HIVE-21867: - Author: ASF GitHub Bot Created on: 26/Jun/19 17:30 Start Date: 26/Jun/19 17:30 Worklog Time Spent: 10m Work Description: jcamachor commented on pull request #687: HIVE-21867 URL: https://github.com/apache/hive/pull/687#discussion_r297783357 ## File path: ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java ## @@ -1766,6 +1774,59 @@ private void removeSemijoinOptimizationByBenefit(OptimizeTezProcContext procCtx) GenTezUtils.removeBranch(rs); GenTezUtils.removeSemiJoinOperator(procCtx.parseContext, rs, ts); } + +for (Entry> e : globalReductionFactorMap.asMap().entrySet()) { Review comment: Good idea, will do. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 267803) Time Spent: 1h (was: 50m) > Sort semijoin conditions to accelerate query processing > --- > > Key: HIVE-21867 > URL: https://issues.apache.org/jira/browse/HIVE-21867 > Project: Hive > Issue Type: New Feature > Components: Physical Optimizer >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21867.02.patch, HIVE-21867.03.patch, > HIVE-21867.patch > > Time Spent: 1h > Remaining Estimate: 0h > > The problem was tackled for CBO in HIVE-21857. Semijoin filters are > introduced later in the planning phase. Follow similar approach to sort them, > trying to accelerate filter evaluation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[ https://issues.apache.org/jira/browse/HIVE-21886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873520#comment-16873520 ] Hive QA commented on HIVE-21886: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 46s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 20s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 47s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 1s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m 5s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 42s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 25s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 29s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 6s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 49s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 17s{color} | {color:red} itests/hive-unit: The patch generated 9 new + 0 unchanged - 0 fixed = 9 total (was 0) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 26s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 15s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 30m 57s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-17754/dev-support/hive-personality.sh | | git revision | master / ee3aeb7 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-17754/yetus/diff-checkstyle-itests_hive-unit.txt | | modules | C: ql itests/hive-unit U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-17754/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > REPL - With table list - Handle rename events during replace policy > --- > > Key: HIVE-21886 > URL: https://issues.apache.org/jira/browse/HIVE-21886 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch > > Time Spent: 8.5h > Remaining Estimate: 0h > > If some rename events are found to be dumped and replayed while replace > policy is getting executed, it needs to take care of the policy inclusion in > both the policy for each table name. > 1. Create a list of tables to be bootstrapped. > 2. During handling of alter table, if the alter type is rename > 1. If the old table name is present in the list of table to be > bootstrapped, remove it. >
[jira] [Updated] (HIVE-21616) Implement JSON_VALUE, JSON_QUERY, and IS [NOT] JSON
[ https://issues.apache.org/jira/browse/HIVE-21616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-21616: -- Attachment: HIVE-21616.2.patch > Implement JSON_VALUE, JSON_QUERY, and IS [NOT] JSON > --- > > Key: HIVE-21616 > URL: https://issues.apache.org/jira/browse/HIVE-21616 > Project: Hive > Issue Type: Sub-task > Components: UDF >Affects Versions: 4.0.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > Attachments: HIVE-21615.patch, HIVE-21616.2.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21888) Set hive.parquet.timestamp.skip.conversion default to true
[ https://issues.apache.org/jira/browse/HIVE-21888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873492#comment-16873492 ] Karen Coppage commented on HIVE-21888: -- Patch 2: removed setting to true in qtests > Set hive.parquet.timestamp.skip.conversion default to true > -- > > Key: HIVE-21888 > URL: https://issues.apache.org/jira/browse/HIVE-21888 > Project: Hive > Issue Type: Bug >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-21888.02.patch, HIVE-21888.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21888) Set hive.parquet.timestamp.skip.conversion default to true
[ https://issues.apache.org/jira/browse/HIVE-21888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-21888: - Status: Open (was: Patch Available) > Set hive.parquet.timestamp.skip.conversion default to true > -- > > Key: HIVE-21888 > URL: https://issues.apache.org/jira/browse/HIVE-21888 > Project: Hive > Issue Type: Bug >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-21888.02.patch, HIVE-21888.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21888) Set hive.parquet.timestamp.skip.conversion default to true
[ https://issues.apache.org/jira/browse/HIVE-21888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-21888: - Attachment: HIVE-21888.02.patch Status: Patch Available (was: Open) > Set hive.parquet.timestamp.skip.conversion default to true > -- > > Key: HIVE-21888 > URL: https://issues.apache.org/jira/browse/HIVE-21888 > Project: Hive > Issue Type: Bug >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-21888.02.patch, HIVE-21888.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21922) Allow keytabs to be reused in LLAP yarn applications through Yarn localization
[ https://issues.apache.org/jira/browse/HIVE-21922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873485#comment-16873485 ] Hive QA commented on HIVE-21922: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12972933/HIVE-21922.2.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/17753/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17753/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17753/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Tests exited with: Exception: Patch URL https://issues.apache.org/jira/secure/attachment/12972933/HIVE-21922.2.patch was found in seen patch url's cache and a test was probably run already on it. Aborting... {noformat} This message is automatically generated. ATTACHMENT ID: 12972933 - PreCommit-HIVE-Build > Allow keytabs to be reused in LLAP yarn applications through Yarn localization > -- > > Key: HIVE-21922 > URL: https://issues.apache.org/jira/browse/HIVE-21922 > Project: Hive > Issue Type: New Feature >Reporter: Adam Szita >Assignee: Adam Szita >Priority: Major > Attachments: HIVE-21922.0.patch, HIVE-21922.1.patch, > HIVE-21922.2.patch > > > In secure clusters LLAP has to be able to reach keytab files for kerberos > login. > Currently _hive.llap.task.scheduler.am.registry.keytab.file_ and > _hive.llap.daemon.keytab.file_ configs are used to define the path of such > keytabs on the Tez AM and LLAP daemon side respectively. Both presume local > file system paths only - hence all nodes in the LLAP cluster (even those that > eventually don't end up executing a daemon...) have to have Hive's keytab > preinstalled on them. > The above is described by this strategy: > [Pre-installed_Keytabs_for_AM_and_containers|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html#Pre-installed_Keytabs_for_AM_and_containers] > Another approach can be > [Keytabs_for_AM_and_containers_distributed_via_YARN|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html#Keytabs_for_AM_and_containers_distributed_via_YARN] > where we rely on HDFS and Yarn resource localization, and no prior keytab > distribution is required. I intend to make this strategy an option for > Hive-LLAP in this jira. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21846) Create a thread in TezAM which periodically fetches LlapDaemon metrics
[ https://issues.apache.org/jira/browse/HIVE-21846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873483#comment-16873483 ] Hive QA commented on HIVE-21846: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12972939/HIVE-21846.03.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/17752/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17752/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17752/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Tests exited with: Exception: Patch URL https://issues.apache.org/jira/secure/attachment/12972939/HIVE-21846.03.patch was found in seen patch url's cache and a test was probably run already on it. Aborting... {noformat} This message is automatically generated. ATTACHMENT ID: 12972939 - PreCommit-HIVE-Build > Create a thread in TezAM which periodically fetches LlapDaemon metrics > -- > > Key: HIVE-21846 > URL: https://issues.apache.org/jira/browse/HIVE-21846 > Project: Hive > Issue Type: Sub-task > Components: llap, Tez >Reporter: Peter Vary >Assignee: Antal Sinkovits >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21846.01.patch, HIVE-21846.02.patch, > HIVE-21846.03.patch > > Time Spent: 1h 50m > Remaining Estimate: 0h > > LlapTaskSchedulerService should start a thread which periodically fetches the > LlapDaemon metrics and stores them in the NodeInfo object. > This should be just the first implementation - later we should find a way > where we do not need NxM requests between N TezAM and M LlapDaemon -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21874) Implement add partitions related methods on temporary table
[ https://issues.apache.org/jira/browse/HIVE-21874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873482#comment-16873482 ] Hive QA commented on HIVE-21874: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12972937/HIVE-21874.04.patch {color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 16598 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.metastore.client.TestDefaultConstraint.addNoSuchTable[Remote] (batchId=224) org.apache.hadoop.hive.metastore.client.TestDefaultConstraint.createGetDrop[Remote] (batchId=224) org.apache.hadoop.hive.metastore.client.TestDefaultConstraint.createTableWithConstraintsPkInOtherCatalog[Remote] (batchId=224) org.apache.hadoop.hive.metastore.client.TestDefaultConstraint.createTableWithConstraintsPk[Remote] (batchId=224) org.apache.hadoop.hive.metastore.client.TestDefaultConstraint.doubleAddUniqueConstraint[Remote] (batchId=224) org.apache.hadoop.hive.metastore.client.TestDefaultConstraint.getNoSuchCatalog[Remote] (batchId=224) org.apache.hadoop.hive.metastore.client.TestDefaultConstraint.getNoSuchDb[Remote] (batchId=224) org.apache.hadoop.hive.metastore.client.TestDefaultConstraint.getNoSuchTable[Remote] (batchId=224) org.apache.hadoop.hive.metastore.client.TestDefaultConstraint.inOtherCatalog[Remote] (batchId=224) org.apache.hive.hcatalog.mapreduce.TestHCatPartitioned.testHCatPartitionedTable[7] (batchId=211) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/17751/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17751/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17751/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 10 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12972937 - PreCommit-HIVE-Build > Implement add partitions related methods on temporary table > --- > > Key: HIVE-21874 > URL: https://issues.apache.org/jira/browse/HIVE-21874 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-21874.01.patch, HIVE-21874.02.patch, > HIVE-21874.03.patch, HIVE-21874.04.patch > > > IMetaStoreClient exposes the following add partition related methods: > {code:java} > Partition add_partition(Partition partition); > int add_partitions(List partitions); > int add_partitions_pspec(PartitionSpecProxy partitionSpec); > List add_partitions(List partitions, boolean > ifNotExists, boolean needResults); > {code} > These methods should be implemented in order to handle addition of partitions > to temporary tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21224) Upgrade tests JUnit3 to JUnit4
[ https://issues.apache.org/jira/browse/HIVE-21224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873463#comment-16873463 ] Alan Gates commented on HIVE-21224: --- Before committing this please make sure the ITests under standalone-metastore execute properly as well. Those are not run as part of the standard nightly build but are needed whenever the data scripts are updated. > Upgrade tests JUnit3 to JUnit4 > -- > > Key: HIVE-21224 > URL: https://issues.apache.org/jira/browse/HIVE-21224 > Project: Hive > Issue Type: Improvement >Reporter: Bruno Pusztahazi >Assignee: Bruno Pusztahazi >Priority: Major > Attachments: HIVE-21224.1.patch, HIVE-21224.2.patch, > HIVE-21224.3.patch, HIVE-21224.4.patch, HIVE-21224.5.patch, > HIVE-21224.6.patch, HIVE-21224.7.patch, HIVE-21224.8.patch, HIVE-21224.9.patch > > > Old JUnit3 tests should be upgraded to JUnit4 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21752) Thread Safety and Memory Leaks in HCatRecordObjectInspectorFactory
[ https://issues.apache.org/jira/browse/HIVE-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-21752: -- Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Patch committed to master. Thanks Jalpan for the patch and sorry it took so long to get it committed. > Thread Safety and Memory Leaks in HCatRecordObjectInspectorFactory > -- > > Key: HIVE-21752 > URL: https://issues.apache.org/jira/browse/HIVE-21752 > Project: Hive > Issue Type: Bug > Components: HCatalog >Reporter: Jalpan Randeri >Assignee: Jalpan Randeri >Priority: Minor > Labels: newbie, patch > Fix For: 4.0.0 > > Attachments: HIVE-21752.patch, HIVE-21752.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > h3. Summary > There are a couple of issues in HCatRecordObjectInspectorFactory[1] because > it uses a static Java HashMap to cache objects: > # Java HashMap is not thread safe. This can lead to data corruptions and > race conditions in multithreaded servers when two threads update the > ObjectInspector. > # There is no eviction policy and as a result, this can result in memory > leaks. If user reads a lot of different schemas, Hive server will start > seeing memory pressure, once it start going to have a lot of cached record > and object inspectors. > This patch propose to replace the cache using a Guava cache which enables > cache evictions and thread safety. Guava cache is already used in Hive > ObjectInspectorFactory [2], so this change is consistent with the rest of > Hive. > Attached is a patch that fixes this issue. > h3. References: > # > [https://github.com/apache/hive/blob/b58d50cb73a1f79a5d079e0a2c5ac33d2efc33a0/hcatalog/core/src/main/java/org/apache/hive/hcatalog/data/HCatRecordObjectInspectorFactory.java#L44-L47] > # > [https://github.com/apache/hive/blob/b58d50cb73a1f79a5d079e0a2c5ac33d2efc33a0/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java#L68-L87] > > h4. Review Board Link: > * [https://reviews.apache.org/r/70674/] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21914) Move Function and Macro related DDL operations into the DDL framework
[ https://issues.apache.org/jira/browse/HIVE-21914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21914: Resolution: Fixed Status: Resolved (was: Patch Available) pushed to master. Thank you [~mgergely]! > Move Function and Macro related DDL operations into the DDL framework > - > > Key: HIVE-21914 > URL: https://issues.apache.org/jira/browse/HIVE-21914 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21914.01.patch, HIVE-21914.02.patch, > HIVE-21914.03.patch > > > Some Function and Macro related operations are handled by FunctionTask, and > FunctionWork while they belong to the DDL framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21896) SHOW FUNCTIONS / SHOW FUNCTIONS LIKE - clarify
[ https://issues.apache.org/jira/browse/HIVE-21896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21896: Resolution: Fixed Status: Resolved (was: Patch Available) pushed to master. Thank you [~mgergely]! > SHOW FUNCTIONS / SHOW FUNCTIONS LIKE - clarify > -- > > Key: HIVE-21896 > URL: https://issues.apache.org/jira/browse/HIVE-21896 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: incompatibleChange, todoc4.0 > Fix For: 4.0.0 > > Attachments: HIVE-21896.01.patch, HIVE-21896.02.patch > > > According to > [https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ShowFunctions] > the currently available functions can be listed like this: > {code:java} > SHOW FUNCTIONS ;{code} > If the user executes this command, they will get the correct list of > functions, but they will also see this on the standard output: > {code:java} > SHOW FUNCTIONS is deprecated, please use SHOW FUNCTIONS LIKE instead.{code} > If the user uses the > {code:java} > SHOW FUNCTIONS LIKE ;{code} > command then they will receive the exact same result (though through > different codes). The only difference is that one can get all the function > names with "SHOW FUNCTIONS;", while "SHOW FUNCTIONS LIKE;" returns an > exception, so in this case the pattern is mandatory. > So there should be a decision if we still accept "SHOW FUNCTIONS" without the > "LIKE". My suggestion is to accept it only if there is no pattern. so "SHOW > FUNCTIONS;" is ok, without deprecation message, but "SHOW FUNCTIONS > " should throw an exception. > Whatever we decide, we should document it appropriately. > cc [~krishahn] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21874) Implement add partitions related methods on temporary table
[ https://issues.apache.org/jira/browse/HIVE-21874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873439#comment-16873439 ] Hive QA commented on HIVE-21874: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 41s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 7s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 28s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 57s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m 6s{color} | {color:blue} standalone-metastore/metastore-server in master has 179 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m 2s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 22s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 29s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 37s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 37s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 40s{color} | {color:red} ql: The patch generated 1 new + 15 unchanged - 0 fixed = 16 total (was 15) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 31s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 23s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 30m 45s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-17751/dev-support/hive-personality.sh | | git revision | master / 967a1cc | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-17751/yetus/diff-checkstyle-ql.txt | | modules | C: standalone-metastore/metastore-server ql U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-17751/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Implement add partitions related methods on temporary table > --- > > Key: HIVE-21874 > URL: https://issues.apache.org/jira/browse/HIVE-21874 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-21874.01.patch, HIVE-21874.02.patch, > HIVE-21874.03.patch, HIVE-21874.04.patch > > > IMetaStoreClient exposes the following add partition related methods: > {code:java} > Partition add_partition(Partition partition); > int add_partitions(List partitions); > int add_partitions_pspec(PartitionSpecProxy partitionSpec); > List add_partitions(List partitions, boolean > ifNotExists, boolean needResults); > {code} > These methods should be implemented in order to handle addition of partitions > to temporary tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21713) Explain consistency Task/Operator
[ https://issues.apache.org/jira/browse/HIVE-21713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manoj Narayanan reassigned HIVE-21713: -- Assignee: Manoj Narayanan > Explain consistency Task/Operator > - > > Key: HIVE-21713 > URL: https://issues.apache.org/jira/browse/HIVE-21713 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Manoj Narayanan >Priority: Major > > there seems to be some inconsistency what gets printed in the explain; > for example "MoveTask" is shown as "Move Operator" in the explain > Becase Tasks and Operators are basically different things - this should be > corrected as it may cause confusion -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21713) Explain consistency Task/Operator
[ https://issues.apache.org/jira/browse/HIVE-21713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873410#comment-16873410 ] Manoj Narayanan commented on HIVE-21713: In the above comment I tried to document each Operator and Task I could find and what their 'ExplainString' is. Also tried to note whether it should be changed (Column Needschange) and if so whether it should 'Add Operator' or 'Change to Operator' or 'Add Task' or 'Change to Task'. > Explain consistency Task/Operator > - > > Key: HIVE-21713 > URL: https://issues.apache.org/jira/browse/HIVE-21713 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Priority: Major > > there seems to be some inconsistency what gets printed in the explain; > for example "MoveTask" is shown as "Move Operator" in the explain > Becase Tasks and Operators are basically different things - this should be > corrected as it may cause confusion -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-21713) Explain consistency Task/Operator
[ https://issues.apache.org/jira/browse/HIVE-21713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873410#comment-16873410 ] Manoj Narayanan edited comment on HIVE-21713 at 6/26/19 3:04 PM: - In the above comment I tried to document each Operator and Task I could find and what their 'ExplainString' is. Also tried to note whether it should be changed (Column Needschange) and if so whether it should 'Add Operator' or 'Change to Operator' or 'Add Task' or 'Change to Task'. Please review [~kgyrtkirk] was (Author: mnarayanan2018): In the above comment I tried to document each Operator and Task I could find and what their 'ExplainString' is. Also tried to note whether it should be changed (Column Needschange) and if so whether it should 'Add Operator' or 'Change to Operator' or 'Add Task' or 'Change to Task'. > Explain consistency Task/Operator > - > > Key: HIVE-21713 > URL: https://issues.apache.org/jira/browse/HIVE-21713 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Priority: Major > > there seems to be some inconsistency what gets printed in the explain; > for example "MoveTask" is shown as "Move Operator" in the explain > Becase Tasks and Operators are basically different things - this should be > corrected as it may cause confusion -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21713) Explain consistency Task/Operator
[ https://issues.apache.org/jira/browse/HIVE-21713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873408#comment-16873408 ] Manoj Narayanan commented on HIVE-21713: |OperatorClass|OfType|ExplainString|hasOperator|hasTask|NeedsChange|ChangeToOperator|AddOperator| |CommonMergeJoinOperator|CommonMergeJoinDesc|Merge Join Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |SMBMapJoinOperator|SMBJoinDesc|Sorted Merge Bucket Map Join Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |ScriptOperator|ScriptDesc|Transform Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |AbstractMapOperator|MapWork|NO_EXPLAIN_STRING|FALSE|FALSE|TRUE|FALSE|TRUE| |AbstractMapJoinOperator|MapJoinDesc|Map Join Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |GroupByOperator|GroupByDesc|Group By Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |LimitOperator|LimitDesc|Limit|FALSE|FALSE|TRUE|FALSE|TRUE| |AbstractFileMergeOperator|FileMergeDesc|NO_EXPLAIN_STRING|FALSE|FALSE|TRUE|FALSE|TRUE| |HashTableDummyOperator|HashTableDummyDesc|HashTable Dummy Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |SelectOperator|SelectDesc|Select Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |LateralViewJoinOperator|LateralViewJoinDesc|Lateral View Join Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |ReduceSinkOperator|ReduceSinkDesc|Reduce Output Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |HashTableSinkOperator|HashTableSinkDesc|HashTable Sink Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |ForwardOperator|ForwardDesc|Forward|FALSE|FALSE|TRUE|FALSE|TRUE| |TableScanOperator|TableScanDesc|TableScan|FALSE|FALSE|TRUE|FALSE|TRUE| |TopNKeyOperator|TopNKeyDesc|Top N Key Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |TopNKeyOperator|TopNKeyDesc|keys|FALSE|FALSE|TRUE|FALSE|TRUE| |FileSinkOperator|FileSinkDesc|File Output Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |JoinOperator|JoinDesc|Join Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |UnionOperator|UnionDesc|Union|FALSE|FALSE|TRUE|FALSE|TRUE| |UDTFOperator|UDTFDesc|UDTF Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |CollectOperator|CollectDesc|Collect|FALSE|FALSE|TRUE|FALSE|TRUE| |FilterOperator|FilterDesc|Filter Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |MapJoinOperator|MapJoinDesc|Map Join Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |DemuxOperator|DemuxDesc|Demux Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |PTFOperator|PTFDesc|PTF Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |AppMasterEventOperator|AppMasterEventDesc|Application Master Event Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |LateralViewForwardOperator|LateralViewForwardDesc|Lateral View Forward|FALSE|FALSE|TRUE|FALSE|TRUE| |CommonJoinOperator|JoinDesc|Join Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |DummyStoreOperator|DummyStoreDesc|Dummy Store|FALSE|FALSE|TRUE|FALSE|TRUE| |MuxOperator|MuxDesc|Mux Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |ListSinkOperator|ListSinkDesc|ListSink|FALSE|FALSE|TRUE|FALSE|TRUE| |FetchOperator|FetchWork|Fetch Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |MapOperator| | | | | | | | |OrcFileMergeOperator|OrcFileMergeDesc|ORC File Merge Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |RCFileMergeOperator|RCFileMergeDesc|RCFile Merge Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |SparkHashTableSinkOperator|SparkHashTableSinkDesc|Spark HashTable Sink Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |TemporaryHashSinkOperator| |HashTable Sink Operator| | | | | | |TezDummyStoreOperator| |Dummy Store| | | | | | | | | | | | | | | |OperatorClass|OfType|ExplainString|hasOperator|hasTask|NeedsChange|ChangeToTask|AddTask| | | | | | | | | | |FunctionTask|FunctionWork|NO_EXPLAIN_STRING|FALSE|FALSE|TRUE|FALSE|TRUE| | | | | | | | | | |DDLTask|DDLWork|NO_EXPLAIN_STRING|FALSE|FALSE|TRUE|FALSE|TRUE| |FetchTask|FetchWork|Fetch Operator|FALSE|TRUE|TRUE|TRUE|FALSE| |ReplCopyTask|ReplCopyWork|Repl Copy|FALSE|FALSE|TRUE|FALSE|TRUE| |ConditionalTask|ConditionalWork|Conditional Operator|FALSE|TRUE|TRUE|TRUE|FALSE| |MaterializedViewTask|MaterializedViewDesc|Materialized View Work|FALSE|FALSE|TRUE|FALSE|TRUE| |ColumnStatsUpdateTask|ColumnStatsUpdateWork|Column Stats Update Work|FALSE|FALSE|TRUE|FALSE|TRUE| |ReplTxnTask|ReplTxnWork|Replication Transaction|FALSE|FALSE|TRUE|FALSE|TRUE| |DependencyCollectionTask|DependencyCollectionWork|Dependency Collection|FALSE|FALSE|TRUE|FALSE|TRUE| |ExplainTask|ExplainWork|NO_EXPLAIN_STRING|FALSE|FALSE|TRUE|FALSE|TRUE| |ExportTask|ExportWork|Export Work|FALSE|FALSE|TRUE|FALSE|TRUE| |CopyTask|CopyWork|Copy|FALSE|FALSE|TRUE|FALSE|TRUE| |MoveTask|MoveWork|Move Operator|FALSE|TRUE|TRUE|TRUE|FALSE| |ExplainSQRewriteTask|ExplainSQRewriteWork|NO_EXPLAIN_STRING|FALSE|FALSE|TRUE|FALSE|TRUE| |StatsTask|StatsWork|Stats Work|FALSE|FALSE|TRUE|FALSE|TRUE| | | | | | | | > Explain consistency Task/Operator > - > > Key: HIVE-21713 > URL: https://issues.apache.org/jira/browse/HIVE-21713 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Priority: Major > > there seems to
[jira] [Commented] (HIVE-18735) Create table like loses transactional attribute
[ https://issues.apache.org/jira/browse/HIVE-18735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873404#comment-16873404 ] Hive QA commented on HIVE-18735: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12972938/HIVE-18735.04.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 16340 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.llap.security.TestLlapSignerImpl.testSigning (batchId=350) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/17750/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17750/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17750/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12972938 - PreCommit-HIVE-Build > Create table like loses transactional attribute > --- > > Key: HIVE-18735 > URL: https://issues.apache.org/jira/browse/HIVE-18735 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 2.0.0 >Reporter: Eugene Koifman >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-18735.01.patch, HIVE-18735.02.patch, > HIVE-18735.03.patch, HIVE-18735.04.patch > > > {noformat} > create table T1(a int, b int) clustered by (a) into 2 buckets stored as orc > TBLPROPERTIES ('transactional'='true')"; > create table T like T1; > show create table T ; > CREATE TABLE `T`( > `a` int, > `b` int) > CLUSTERED BY ( > a) > INTO 2 BUCKETS > ROW FORMAT SERDE > 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' > LOCATION > > 'file:/Users/ekoifman/IdeaProjects/hive/ql/target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands-1518813536099/warehouse/t' > TBLPROPERTIES ( > 'transient_lastDdlTime'='1518813564') > {noformat} > Specifying props explicitly does work > {noformat} > create table T1(a int, b int) clustered by (a) into 2 buckets stored as orc > TBLPROPERTIES ('transactional'='true')"; > create table T like T1 TBLPROPERTIES ('transactional'='true'); > show create table T ; > CREATE TABLE `T`( > `a` int, > `b` int) > CLUSTERED BY ( > a) > INTO 2 BUCKETS > ROW FORMAT SERDE > 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' > LOCATION > > 'file:/Users/ekoifman/IdeaProjects/hive/ql/target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands-1518814098564/warehouse/t' > TBLPROPERTIES ( > 'transactional'='true', > 'transactional_properties'='default', > 'transient_lastDdlTime'='1518814111') > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21923) Vectorized MapJoin may miss results when only the join key is selected
[ https://issues.apache.org/jira/browse/HIVE-21923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21923: Summary: Vectorized MapJoin may miss results when only the join key is selected (was: Disabling n-way joins caused some resultset changes) > Vectorized MapJoin may miss results when only the join key is selected > -- > > Key: HIVE-21923 > URL: https://issues.apache.org/jira/browse/HIVE-21923 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21923.01.patch > > > HIVE-21189 have introduced some resultset changes > in ql/src/test/results/clientpositive/llap/hybridgrace_hashjoin_2.q.out > https://github.com/apache/hive/commit/5799398450c17d06e8ef144ce835a8524f5abec9#diff-56b3ab96b6c90fdbebe2c4f84e8595afL500 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18735) Create table like loses transactional attribute
[ https://issues.apache.org/jira/browse/HIVE-18735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873365#comment-16873365 ] Hive QA commented on HIVE-18735: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 35s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 13s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 24s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 55s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 58s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 32s{color} | {color:blue} hbase-handler in master has 15 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 27s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 58s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 12s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 15s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 28m 38s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-17750/dev-support/hive-personality.sh | | git revision | master / 967a1cc | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | modules | C: ql hbase-handler U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-17750/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Create table like loses transactional attribute > --- > > Key: HIVE-18735 > URL: https://issues.apache.org/jira/browse/HIVE-18735 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 2.0.0 >Reporter: Eugene Koifman >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-18735.01.patch, HIVE-18735.02.patch, > HIVE-18735.03.patch, HIVE-18735.04.patch > > > {noformat} > create table T1(a int, b int) clustered by (a) into 2 buckets stored as orc > TBLPROPERTIES ('transactional'='true')"; > create table T like T1; > show create table T ; > CREATE TABLE `T`( > `a` int, > `b` int) > CLUSTERED BY ( > a) > INTO 2 BUCKETS > ROW FORMAT SERDE > 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' > LOCATION > > 'file:/Users/ekoifman/IdeaProjects/hive/ql/target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands-1518813536099/warehouse/t' > TBLPROPERTIES ( > 'transient_lastDdlTime'='1518813564') > {
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[ https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267667&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267667 ] ASF GitHub Bot logged work on HIVE-21886: - Author: ASF GitHub Bot Created on: 26/Jun/19 14:09 Start Date: 26/Jun/19 14:09 Worklog Time Spent: 10m Work Description: sankarh commented on pull request #688: HIVE-21886 : REPL - With table list - Handle rename events during replace policy URL: https://github.com/apache/hive/pull/688#discussion_r297686895 ## File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java ## @@ -92,21 +93,46 @@ private Scenario scenarioType(org.apache.hadoop.hive.metastore.api.Table before, } } - // return true, if event needs to be dumped, else return false. - private boolean handleForTableLevelReplication(Context withinContext) { -String oldName = before.getTableName(); -String newName = after.getTableName(); + private boolean isSetForBootstrapByReplaceHandler(Context withinContext, String tblName) { +return (withinContext.oldReplScope != null) +&& !(ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName)) +&& (ReplUtils.tableIncludedInReplScope(withinContext.replScope, tblName)); + } + + // Return true, if event needs to be dumped, else return false. + private boolean handleForTableLevelReplication(Context withinContext, String tblName) { +// For alter, if the table does not satisfy the new policy then ignore the event. In case of replace +// policy, if the table does not satisfy the old policy, then ignore the event. As, if the table satisfy the new +// policy, then the table will be bootstrapped by replace handler anb if the table does not satisfy the new policy, +// then anyways the table should be ignored. The event should be ignored if the table is already in the list of +// tables to be bootstrapped. +if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope, tblName)) { + // In case of replace, it will be dropped during load. In normal case just ignore the alter event. + LOG.debug("Table " + tblName + " does not satisfy the policy"); + return false; +} else if ((withinContext.getTablesForBootstrap().contains(tblName.toLowerCase())) Review comment: I think, the else if flow is dead code as we skip this event from caller itself. Utils.shouldReplicate method returns false for this case. Pls check. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 267667) Time Spent: 8.5h (was: 8h 20m) > REPL - With table list - Handle rename events during replace policy > --- > > Key: HIVE-21886 > URL: https://issues.apache.org/jira/browse/HIVE-21886 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch > > Time Spent: 8.5h > Remaining Estimate: 0h > > If some rename events are found to be dumped and replayed while replace > policy is getting executed, it needs to take care of the policy inclusion in > both the policy for each table name. > 1. Create a list of tables to be bootstrapped. > 2. During handling of alter table, if the alter type is rename > 1. If the old table name is present in the list of table to be > bootstrapped, remove it. > 2. If the new table name, matches the new policy, add it to the list > of tables to be bootstrapped. > 3. During handling of drop table > 1. if the table is in the list of tables to be bootstrapped, then > remove it and ignore the event. > 4. During other event handling > 1. if the table is there in the list of tables to be bootstrapped, > then ignore the event. > > Rename handling during replace policy > # Old name not matching old policy – The old table will not be there at the > target cluster. The table will not be returned by get-all-table. > ## Old name is not matching new policy > ### New name not matching old policy > New name not matching new policy > * Ignore the event, no need to do anything. > New name matching new policy > * The table will be returned by get-all-table. Replace policy handler > will bootst
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[ https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267660&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267660 ] ASF GitHub Bot logged work on HIVE-21886: - Author: ASF GitHub Bot Created on: 26/Jun/19 14:03 Start Date: 26/Jun/19 14:03 Worklog Time Spent: 10m Work Description: sankarh commented on pull request #688: HIVE-21886 : REPL - With table list - Handle rename events during replace policy URL: https://github.com/apache/hive/pull/688#discussion_r297683504 ## File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java ## @@ -147,16 +197,19 @@ public void handle(Context withinContext) throws Exception { Table qlMdTableBefore = new Table(before); Set bootstrapTableList; +ReplScope oldReplScope; if (Scenario.RENAME == scenario) { // Handling for table level replication is done in handleForTableLevelReplication method. Review comment: handleForTableLevelReplication doesn't handle rename flow. Incorrect comment. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 267660) Time Spent: 8h 20m (was: 8h 10m) > REPL - With table list - Handle rename events during replace policy > --- > > Key: HIVE-21886 > URL: https://issues.apache.org/jira/browse/HIVE-21886 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch > > Time Spent: 8h 20m > Remaining Estimate: 0h > > If some rename events are found to be dumped and replayed while replace > policy is getting executed, it needs to take care of the policy inclusion in > both the policy for each table name. > 1. Create a list of tables to be bootstrapped. > 2. During handling of alter table, if the alter type is rename > 1. If the old table name is present in the list of table to be > bootstrapped, remove it. > 2. If the new table name, matches the new policy, add it to the list > of tables to be bootstrapped. > 3. During handling of drop table > 1. if the table is in the list of tables to be bootstrapped, then > remove it and ignore the event. > 4. During other event handling > 1. if the table is there in the list of tables to be bootstrapped, > then ignore the event. > > Rename handling during replace policy > # Old name not matching old policy – The old table will not be there at the > target cluster. The table will not be returned by get-all-table. > ## Old name is not matching new policy > ### New name not matching old policy > New name not matching new policy > * Ignore the event, no need to do anything. > New name matching new policy > * The table will be returned by get-all-table. Replace policy handler > will bootstrap this table as its matching new policy and not matching old > policy. > * All the future events will be ignored as part of check added by > replace policy handling. > * All the event with old table name will anyways be ignored as the old > name is not matching the new policy. > ### New name matching old policy > New name not matching new policy > * As the new name is not matching the new policy, the table need not be > replicated. > * As the old name is not matching the new policy, the rename events will > be ignored. > * So nothing to be done for this scenario. > New name matching new policy > * As the new name is matching both old and new policy, replace handler > will not bootstrap the table. > * Add the table to the list of tables to be bootstrapped. > * Ignore all the events with new name. > * If there is a drop event for the table (with new name), then remove > the table from the the list of table to be bootstrapped. > * In case of rename event (double rename) > ** If the new name satisfies the table pattern, then add the new name to > the list of tables to be bootstrapped and remove the old name from the list > of tables to be bootstrapped. > ** If the new name does not satisfies then just removed the table name > from the list of tables to be bootstrapped. > ## Old name is matching new policy – As per replace policy handler, which
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[ https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267656&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267656 ] ASF GitHub Bot logged work on HIVE-21886: - Author: ASF GitHub Bot Created on: 26/Jun/19 14:01 Start Date: 26/Jun/19 14:01 Worklog Time Spent: 10m Work Description: sankarh commented on pull request #688: HIVE-21886 : REPL - With table list - Handle rename events during replace policy URL: https://github.com/apache/hive/pull/688#discussion_r297682711 ## File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java ## @@ -118,13 +144,30 @@ private boolean handleForTableLevelReplication(Context withinContext) { // If the old table was in the list of tables to be bootstrapped which is a multi rename case, the old table // is removed from the list of tables to be bootstrapped and new one is added. - if (oldTableIsPresent) { + if (oldTableInBootstrapList) { +withinContext.addToListOfTablesForBootstrap(newName); +return false; + } + + // All the subsequent events on this table newName are going to be skipped as the table is going to be + // bootstrapped by replace handler, so the rename is also skipped. + if (isSetForBootstrapByReplaceHandler(withinContext, newName)) { +// This addition is not actually required. But is added just to be in safer side. withinContext.addToListOfTablesForBootstrap(newName); + +// If the old table satisfies the new policy and is not in the list of tables to be bootstrapped +// (as per previous check based on oldTableIsPresent), then drop it. +if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) { + scenario = Scenario.DROP; + LOG.info("Table " + oldName + " will be dropped as the table " + newName + " will be bootstrapped."); + return true; +} +LOG.info("Table " + newName + " is set to be bootstrapped by replace policy handler."); return false; } // If both old and new table satisfies the filter and old table is present at target, then dump the rename event. - LOG.info("both old and new table satisfies the filter"); + LOG.info("Both old " + oldName + " and new table " + newName + " satisfies the filter"); Review comment: filter is not the right term. We can say, replication scope/policy. Pls check other places as well. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 267656) Time Spent: 8h 10m (was: 8h) > REPL - With table list - Handle rename events during replace policy > --- > > Key: HIVE-21886 > URL: https://issues.apache.org/jira/browse/HIVE-21886 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch > > Time Spent: 8h 10m > Remaining Estimate: 0h > > If some rename events are found to be dumped and replayed while replace > policy is getting executed, it needs to take care of the policy inclusion in > both the policy for each table name. > 1. Create a list of tables to be bootstrapped. > 2. During handling of alter table, if the alter type is rename > 1. If the old table name is present in the list of table to be > bootstrapped, remove it. > 2. If the new table name, matches the new policy, add it to the list > of tables to be bootstrapped. > 3. During handling of drop table > 1. if the table is in the list of tables to be bootstrapped, then > remove it and ignore the event. > 4. During other event handling > 1. if the table is there in the list of tables to be bootstrapped, > then ignore the event. > > Rename handling during replace policy > # Old name not matching old policy – The old table will not be there at the > target cluster. The table will not be returned by get-all-table. > ## Old name is not matching new policy > ### New name not matching old policy > New name not matching new policy > * Ignore the event, no need to do anything. > New name matching new policy > * The table will be returned by get-all-table. Replace policy handler > will bootstrap th
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[ https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267654&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267654 ] ASF GitHub Bot logged work on HIVE-21886: - Author: ASF GitHub Bot Created on: 26/Jun/19 13:59 Start Date: 26/Jun/19 13:59 Worklog Time Spent: 10m Work Description: sankarh commented on pull request #688: HIVE-21886 : REPL - With table list - Handle rename events during replace policy URL: https://github.com/apache/hive/pull/688#discussion_r297681357 ## File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java ## @@ -135,8 +178,15 @@ private boolean handleForTableLevelReplication(Context withinContext) { return false; } + // In case of replace policy, even if the old table matches the new policy, none of the events including create + // table will be replayed as the old table is set of bootstrap by replace handler. So rename event can be skipped. Review comment: There is nothing called replace handler. So, shall change the comment that "Old table is set for bootstrap due to replace policy." This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 267654) Time Spent: 8h (was: 7h 50m) > REPL - With table list - Handle rename events during replace policy > --- > > Key: HIVE-21886 > URL: https://issues.apache.org/jira/browse/HIVE-21886 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch > > Time Spent: 8h > Remaining Estimate: 0h > > If some rename events are found to be dumped and replayed while replace > policy is getting executed, it needs to take care of the policy inclusion in > both the policy for each table name. > 1. Create a list of tables to be bootstrapped. > 2. During handling of alter table, if the alter type is rename > 1. If the old table name is present in the list of table to be > bootstrapped, remove it. > 2. If the new table name, matches the new policy, add it to the list > of tables to be bootstrapped. > 3. During handling of drop table > 1. if the table is in the list of tables to be bootstrapped, then > remove it and ignore the event. > 4. During other event handling > 1. if the table is there in the list of tables to be bootstrapped, > then ignore the event. > > Rename handling during replace policy > # Old name not matching old policy – The old table will not be there at the > target cluster. The table will not be returned by get-all-table. > ## Old name is not matching new policy > ### New name not matching old policy > New name not matching new policy > * Ignore the event, no need to do anything. > New name matching new policy > * The table will be returned by get-all-table. Replace policy handler > will bootstrap this table as its matching new policy and not matching old > policy. > * All the future events will be ignored as part of check added by > replace policy handling. > * All the event with old table name will anyways be ignored as the old > name is not matching the new policy. > ### New name matching old policy > New name not matching new policy > * As the new name is not matching the new policy, the table need not be > replicated. > * As the old name is not matching the new policy, the rename events will > be ignored. > * So nothing to be done for this scenario. > New name matching new policy > * As the new name is matching both old and new policy, replace handler > will not bootstrap the table. > * Add the table to the list of tables to be bootstrapped. > * Ignore all the events with new name. > * If there is a drop event for the table (with new name), then remove > the table from the the list of table to be bootstrapped. > * In case of rename event (double rename) > ** If the new name satisfies the table pattern, then add the new name to > the list of tables to be bootstrapped and remove the old name from the list > of tables to be bootstrapped. > ** If the new name does not satisfies then just removed the table name > from the list of tables to be boots
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[ https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267648&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267648 ] ASF GitHub Bot logged work on HIVE-21886: - Author: ASF GitHub Bot Created on: 26/Jun/19 13:54 Start Date: 26/Jun/19 13:54 Worklog Time Spent: 10m Work Description: sankarh commented on pull request #688: HIVE-21886 : REPL - With table list - Handle rename events during replace policy URL: https://github.com/apache/hive/pull/688#discussion_r297678333 ## File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java ## @@ -92,16 +93,40 @@ private Scenario scenarioType(org.apache.hadoop.hive.metastore.api.Table before, } } - // return true, if event needs to be dumped, else return false. - private boolean handleForTableLevelReplication(Context withinContext) { -String oldName = before.getTableName(); -String newName = after.getTableName(); + private boolean isSetForBootstrapByReplaceHandler(Context withinContext, String tblName) { +return (withinContext.oldReplScope != null) +&& !(ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName)) +&& (ReplUtils.tableIncludedInReplScope(withinContext.replScope, tblName)); + } -if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) { - // If the table is renamed after being added to the list of tables to be bootstrapped, then remove it from the - // list of tables to be bootstrapped. - boolean oldTableIsPresent = withinContext.removeFromListOfTablesForBootstrap(before.getTableName()); + // Return true, if event needs to be dumped, else return false. + private boolean handleForTableLevelReplication(Context withinContext, String tblName) { +// For alter, if the table does not satisfy the new policy then ignore the event. In case of replace +// policy, if the table does not satisfy the old policy, then ignore the event. As, if the table satisfy the new +// policy, then the table will be bootstrapped by replace handler anb if the table does not satisfy the new policy, +// then anyways the table should be ignored. +if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope, tblName)) { + // In case of replace, it will be dropped during load. In normal case just ignore the alter event. + LOG.debug("Table " + tblName + " does not satisfy the policy"); + return false; +} else if ((withinContext.oldReplScope != null) Review comment: As discussed, let's keep the checks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 267648) Time Spent: 7.5h (was: 7h 20m) > REPL - With table list - Handle rename events during replace policy > --- > > Key: HIVE-21886 > URL: https://issues.apache.org/jira/browse/HIVE-21886 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch > > Time Spent: 7.5h > Remaining Estimate: 0h > > If some rename events are found to be dumped and replayed while replace > policy is getting executed, it needs to take care of the policy inclusion in > both the policy for each table name. > 1. Create a list of tables to be bootstrapped. > 2. During handling of alter table, if the alter type is rename > 1. If the old table name is present in the list of table to be > bootstrapped, remove it. > 2. If the new table name, matches the new policy, add it to the list > of tables to be bootstrapped. > 3. During handling of drop table > 1. if the table is in the list of tables to be bootstrapped, then > remove it and ignore the event. > 4. During other event handling > 1. if the table is there in the list of tables to be bootstrapped, > then ignore the event. > > Rename handling during replace policy > # Old name not matching old policy – The old table will not be there at the > target cluster. The table will not be returned by get-all-table. > ## Old name is not matching new policy > ### New name not matching old policy > New name not matching new policy > * Ignore the event, no need to do anything. > New name matching n
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[ https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267650&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267650 ] ASF GitHub Bot logged work on HIVE-21886: - Author: ASF GitHub Bot Created on: 26/Jun/19 13:54 Start Date: 26/Jun/19 13:54 Worklog Time Spent: 10m Work Description: sankarh commented on pull request #688: HIVE-21886 : REPL - With table list - Handle rename events during replace policy URL: https://github.com/apache/hive/pull/688#discussion_r297678731 ## File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java ## @@ -135,8 +173,15 @@ private boolean handleForTableLevelReplication(Context withinContext) { return false; Review comment: As discussed, let's keep the checks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 267650) Time Spent: 7h 50m (was: 7h 40m) > REPL - With table list - Handle rename events during replace policy > --- > > Key: HIVE-21886 > URL: https://issues.apache.org/jira/browse/HIVE-21886 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch > > Time Spent: 7h 50m > Remaining Estimate: 0h > > If some rename events are found to be dumped and replayed while replace > policy is getting executed, it needs to take care of the policy inclusion in > both the policy for each table name. > 1. Create a list of tables to be bootstrapped. > 2. During handling of alter table, if the alter type is rename > 1. If the old table name is present in the list of table to be > bootstrapped, remove it. > 2. If the new table name, matches the new policy, add it to the list > of tables to be bootstrapped. > 3. During handling of drop table > 1. if the table is in the list of tables to be bootstrapped, then > remove it and ignore the event. > 4. During other event handling > 1. if the table is there in the list of tables to be bootstrapped, > then ignore the event. > > Rename handling during replace policy > # Old name not matching old policy – The old table will not be there at the > target cluster. The table will not be returned by get-all-table. > ## Old name is not matching new policy > ### New name not matching old policy > New name not matching new policy > * Ignore the event, no need to do anything. > New name matching new policy > * The table will be returned by get-all-table. Replace policy handler > will bootstrap this table as its matching new policy and not matching old > policy. > * All the future events will be ignored as part of check added by > replace policy handling. > * All the event with old table name will anyways be ignored as the old > name is not matching the new policy. > ### New name matching old policy > New name not matching new policy > * As the new name is not matching the new policy, the table need not be > replicated. > * As the old name is not matching the new policy, the rename events will > be ignored. > * So nothing to be done for this scenario. > New name matching new policy > * As the new name is matching both old and new policy, replace handler > will not bootstrap the table. > * Add the table to the list of tables to be bootstrapped. > * Ignore all the events with new name. > * If there is a drop event for the table (with new name), then remove > the table from the the list of table to be bootstrapped. > * In case of rename event (double rename) > ** If the new name satisfies the table pattern, then add the new name to > the list of tables to be bootstrapped and remove the old name from the list > of tables to be bootstrapped. > ** If the new name does not satisfies then just removed the table name > from the list of tables to be bootstrapped. > ## Old name is matching new policy – As per replace policy handler, which > checks based on old table, the table should be bootstrapped and event should > be ignored. But rename handler should decide based on new name.The old table > name will not be returned by get-all-table, so replace handler will not d > anything for th
[jira] [Commented] (HIVE-21407) Parquet predicate pushdown is not working correctly for char column types
[ https://issues.apache.org/jira/browse/HIVE-21407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873349#comment-16873349 ] Zoltan Haindrich commented on HIVE-21407: - The format may store it however it want that value; but, it is Hive's type systems responsibility to handle a "rightly" type constant ; for example a Char(n) type if that's what it is - you may ask for the expanded or the non-expanded form...but when you do I think you have to consider what contracts parquet is comforming to. To keep this short; I think the following example might help: {code} select 'a' = 'a ', cast('a' as char(3)) = 'a ', cast('a ' as char(3)) = 'a'; {code} Would it be possible that parquet stores 'a ' somehow? because if that's possible then neither 'a' nor 'a ' will match that... > Parquet predicate pushdown is not working correctly for char column types > - > > Key: HIVE-21407 > URL: https://issues.apache.org/jira/browse/HIVE-21407 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Marta Kuczora >Assignee: Marta Kuczora >Priority: Major > Attachments: HIVE-21407.2.patch, HIVE-21407.3.patch, HIVE-21407.patch > > > If the 'hive.optimize.index.filter' parameter is false, the filter predicate > is not pushed to parquet, so the filtering only happens within Hive. If the > parameter is true, the filter is pushed to parquet, but for a char type, the > value which is pushed to Parquet will be padded with spaces: > {noformat} > @Override > public void setValue(String val, int len) { > super.setValue(HiveBaseChar.getPaddedValue(val, len), -1); > } > {noformat} > So if we have a char(10) column which contains the value "apple" and the > where condition looks like 'where c='apple'', the value pushed to Paquet will > be 'apple' followed by 5 spaces. But the stored values are not padded, so no > rows will be returned from Parquet. > How to reproduce: > {noformat} > $ create table ppd (c char(10), v varchar(10), i int) stored as parquet; > $ insert into ppd values ('apple', 'bee', 1),('apple', 'tree', 2),('hello', > 'world', 1),('hello','vilag',3); > $ set hive.optimize.ppd.storage=true; > $ set hive.vectorized.execution.enabled=true; > $ set hive.vectorized.execution.enabled=false; > $ set hive.optimize.ppd=true; > $ set hive.optimize.index.filter=true; > $ set hive.parquet.timestamp.skip.conversion=false; > $ select * from ppd where c='apple'; > ++++ > | ppd.c | ppd.v | ppd.i | > ++++ > ++++ > $ set hive.optimize.index.filter=false; or set > hive.optimize.ppd.storage=false; > $ select * from ppd where c='apple'; > +-+++ > |ppd.c| ppd.v | ppd.i | > +-+++ > | apple | bee| 1 | > | apple | tree | 2 | > +-+++ > {noformat} > The issue surfaced after uploading the fix for > [HIVE-21327|https://issues.apache.org/jira/browse/HIVE-21327] was uploaded > upstream. Before the HIVE-21327 fix, setting the parameter > 'hive.parquet.timestamp.skip.conversion' to true in the parquet_ppd_char.q > test hid this issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[ https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267649&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267649 ] ASF GitHub Bot logged work on HIVE-21886: - Author: ASF GitHub Bot Created on: 26/Jun/19 13:54 Start Date: 26/Jun/19 13:54 Worklog Time Spent: 10m Work Description: sankarh commented on pull request #688: HIVE-21886 : REPL - With table list - Handle rename events during replace policy URL: https://github.com/apache/hive/pull/688#discussion_r297678557 ## File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java ## @@ -135,8 +173,15 @@ private boolean handleForTableLevelReplication(Context withinContext) { return false; } + // In case of replace policy, even if the old table matches the new policy, none of the events including create + // table will be replayed as the old table is set of bootstrap by replace handler. So rename event can be skipped. + if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) { Review comment: Ok This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 267649) Time Spent: 7h 40m (was: 7.5h) > REPL - With table list - Handle rename events during replace policy > --- > > Key: HIVE-21886 > URL: https://issues.apache.org/jira/browse/HIVE-21886 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch > > Time Spent: 7h 40m > Remaining Estimate: 0h > > If some rename events are found to be dumped and replayed while replace > policy is getting executed, it needs to take care of the policy inclusion in > both the policy for each table name. > 1. Create a list of tables to be bootstrapped. > 2. During handling of alter table, if the alter type is rename > 1. If the old table name is present in the list of table to be > bootstrapped, remove it. > 2. If the new table name, matches the new policy, add it to the list > of tables to be bootstrapped. > 3. During handling of drop table > 1. if the table is in the list of tables to be bootstrapped, then > remove it and ignore the event. > 4. During other event handling > 1. if the table is there in the list of tables to be bootstrapped, > then ignore the event. > > Rename handling during replace policy > # Old name not matching old policy – The old table will not be there at the > target cluster. The table will not be returned by get-all-table. > ## Old name is not matching new policy > ### New name not matching old policy > New name not matching new policy > * Ignore the event, no need to do anything. > New name matching new policy > * The table will be returned by get-all-table. Replace policy handler > will bootstrap this table as its matching new policy and not matching old > policy. > * All the future events will be ignored as part of check added by > replace policy handling. > * All the event with old table name will anyways be ignored as the old > name is not matching the new policy. > ### New name matching old policy > New name not matching new policy > * As the new name is not matching the new policy, the table need not be > replicated. > * As the old name is not matching the new policy, the rename events will > be ignored. > * So nothing to be done for this scenario. > New name matching new policy > * As the new name is matching both old and new policy, replace handler > will not bootstrap the table. > * Add the table to the list of tables to be bootstrapped. > * Ignore all the events with new name. > * If there is a drop event for the table (with new name), then remove > the table from the the list of table to be bootstrapped. > * In case of rename event (double rename) > ** If the new name satisfies the table pattern, then add the new name to > the list of tables to be bootstrapped and remove the old name from the list > of tables to be bootstrapped. > ** If the new name does not satisfies then just removed the table name > from the list of tables to be bootstrapped. > ## Old name is matching new
[jira] [Comment Edited] (HIVE-21407) Parquet predicate pushdown is not working correctly for char column types
[ https://issues.apache.org/jira/browse/HIVE-21407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873349#comment-16873349 ] Zoltan Haindrich edited comment on HIVE-21407 at 6/26/19 1:54 PM: -- The format may store it however it want that value; but, it is Hive's type systems responsibility to handle a "rightly" type constant ; for example a {{Char( n )}} type if that's what it is - you may ask for the expanded or the non-expanded form...but when you do I think you have to consider what contracts parquet is comforming to. To keep this short; I think the following example might help: {code} select 'a' = 'a ', cast('a' as char(3)) = 'a ', cast('a ' as char(3)) = 'a'; {code} Would it be possible that parquet stores 'a ' somehow? because if that's possible then neither 'a' nor 'a ' will match that... was (Author: kgyrtkirk): The format may store it however it want that value; but, it is Hive's type systems responsibility to handle a "rightly" type constant ; for example a Char(n) type if that's what it is - you may ask for the expanded or the non-expanded form...but when you do I think you have to consider what contracts parquet is comforming to. To keep this short; I think the following example might help: {code} select 'a' = 'a ', cast('a' as char(3)) = 'a ', cast('a ' as char(3)) = 'a'; {code} Would it be possible that parquet stores 'a ' somehow? because if that's possible then neither 'a' nor 'a ' will match that... > Parquet predicate pushdown is not working correctly for char column types > - > > Key: HIVE-21407 > URL: https://issues.apache.org/jira/browse/HIVE-21407 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Marta Kuczora >Assignee: Marta Kuczora >Priority: Major > Attachments: HIVE-21407.2.patch, HIVE-21407.3.patch, HIVE-21407.patch > > > If the 'hive.optimize.index.filter' parameter is false, the filter predicate > is not pushed to parquet, so the filtering only happens within Hive. If the > parameter is true, the filter is pushed to parquet, but for a char type, the > value which is pushed to Parquet will be padded with spaces: > {noformat} > @Override > public void setValue(String val, int len) { > super.setValue(HiveBaseChar.getPaddedValue(val, len), -1); > } > {noformat} > So if we have a char(10) column which contains the value "apple" and the > where condition looks like 'where c='apple'', the value pushed to Paquet will > be 'apple' followed by 5 spaces. But the stored values are not padded, so no > rows will be returned from Parquet. > How to reproduce: > {noformat} > $ create table ppd (c char(10), v varchar(10), i int) stored as parquet; > $ insert into ppd values ('apple', 'bee', 1),('apple', 'tree', 2),('hello', > 'world', 1),('hello','vilag',3); > $ set hive.optimize.ppd.storage=true; > $ set hive.vectorized.execution.enabled=true; > $ set hive.vectorized.execution.enabled=false; > $ set hive.optimize.ppd=true; > $ set hive.optimize.index.filter=true; > $ set hive.parquet.timestamp.skip.conversion=false; > $ select * from ppd where c='apple'; > ++++ > | ppd.c | ppd.v | ppd.i | > ++++ > ++++ > $ set hive.optimize.index.filter=false; or set > hive.optimize.ppd.storage=false; > $ select * from ppd where c='apple'; > +-+++ > |ppd.c| ppd.v | ppd.i | > +-+++ > | apple | bee| 1 | > | apple | tree | 2 | > +-+++ > {noformat} > The issue surfaced after uploading the fix for > [HIVE-21327|https://issues.apache.org/jira/browse/HIVE-21327] was uploaded > upstream. Before the HIVE-21327 fix, setting the parameter > 'hive.parquet.timestamp.skip.conversion' to true in the parquet_ppd_char.q > test hid this issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)