[jira] [Commented] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16875379#comment-16875379
]
Hive QA commented on HIVE-21886:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973227/HIVE-21886.05.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16359 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17796/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17796/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17796/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12973227 - PreCommit-HIVE-Build
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch,
> HIVE-21886.03.patch, HIVE-21886.04.patch, HIVE-21886.04.patch,
> HIVE-21886.05.patch
>
> Time Spent: 11h 10m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. If the old table does not match the old policy drop it, even if the
> table is not present at target.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
> 2. If the new policy does not match the table name, then ignore the
> event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> * As the old name is not matching the new policy, the rename events will
> be ignored.
> * So nothing to be done for this scenario.
> New name matching new policy
> * As the new name is matching both old and new policy, replace handler
> will not bootstrap the table.
> * Add the table to the list of tables to be bootstrapped.
> * Ignore all the events with new name.
> * If there is a drop event for the table (with new name), then remove
> the table from the the list of table to be bootstrapped.
> * In case of rename event (double rename)
> ** If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ** If the new name does not satisfies then just removed the table name
> from the list of tables to be bootstrapped.
> ## Old name is matching new policy – As per replace policy handler, which
> checks based on old table, the table should be bootstrapped and event should
> be ignored. But rename handler should decide based on new name.The old table
> name will no
[jira] [Commented] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16875369#comment-16875369
]
Hive QA commented on HIVE-21886:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
52s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
58s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
21s{color} | {color:blue} testutils/ptest2 in master has 24 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
36s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17796/dev-support/hive-personality.sh
|
| git revision | master / b2a265a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17796/yetus/patch-asflicense-problems.txt
|
| modules | C: ql itests/hive-unit testutils/ptest2 U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17796/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch,
> HIVE-21886.03.patch, HIVE-21886.04.patch, HIVE-21886.04.patch,
> HIVE-21886.05.patch
>
> Time Spent: 11h 10m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of t
[jira] [Commented] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[ https://issues.apache.org/jira/browse/HIVE-21886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16874779#comment-16874779 ] Sankar Hariappan commented on HIVE-21886: - +1, patch LGTM, pending tests. > REPL - With table list - Handle rename events during replace policy > --- > > Key: HIVE-21886 > URL: https://issues.apache.org/jira/browse/HIVE-21886 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch, > HIVE-21886.03.patch, HIVE-21886.04.patch, HIVE-21886.04.patch > > Time Spent: 11h 10m > Remaining Estimate: 0h > > If some rename events are found to be dumped and replayed while replace > policy is getting executed, it needs to take care of the policy inclusion in > both the policy for each table name. > 1. Create a list of tables to be bootstrapped. > 2. During handling of alter table, if the alter type is rename > 1. If the old table name is present in the list of table to be > bootstrapped, remove it. > 2. If the new table name, matches the new policy, add it to the list > of tables to be bootstrapped. > 3. If the old table does not match the old policy drop it, even if the > table is not present at target. > 3. During handling of drop table > 1. if the table is in the list of tables to be bootstrapped, then > remove it and ignore the event. > 4. During other event handling > 1. if the table is there in the list of tables to be bootstrapped, > then ignore the event. > 2. If the new policy does not match the table name, then ignore the > event. > > Rename handling during replace policy > # Old name not matching old policy – The old table will not be there at the > target cluster. The table will not be returned by get-all-table. > ## Old name is not matching new policy > ### New name not matching old policy > New name not matching new policy > * Ignore the event, no need to do anything. > New name matching new policy > * The table will be returned by get-all-table. Replace policy handler > will bootstrap this table as its matching new policy and not matching old > policy. > * All the future events will be ignored as part of check added by > replace policy handling. > * All the event with old table name will anyways be ignored as the old > name is not matching the new policy. > ### New name matching old policy > New name not matching new policy > * As the new name is not matching the new policy, the table need not be > replicated. > * As the old name is not matching the new policy, the rename events will > be ignored. > * So nothing to be done for this scenario. > New name matching new policy > * As the new name is matching both old and new policy, replace handler > will not bootstrap the table. > * Add the table to the list of tables to be bootstrapped. > * Ignore all the events with new name. > * If there is a drop event for the table (with new name), then remove > the table from the the list of table to be bootstrapped. > * In case of rename event (double rename) > ** If the new name satisfies the table pattern, then add the new name to > the list of tables to be bootstrapped and remove the old name from the list > of tables to be bootstrapped. > ** If the new name does not satisfies then just removed the table name > from the list of tables to be bootstrapped. > ## Old name is matching new policy – As per replace policy handler, which > checks based on old table, the table should be bootstrapped and event should > be ignored. But rename handler should decide based on new name.The old table > name will not be returned by get-all-table, so replace handler will not d > anything for the old table. > ### New name not matching old policy > New name not matching new policy > * As the old table is not there at target and new name is not matching > new policy. Ignore the event. > * No need to add the table to the list of tables to be bootstrapped. > * All the subsequent events will be ignored as the new name is not > matching the new policy. > New name matching new policy > * As the new name is not matching old policy but matching new policy, > the table will be bootstrapped by replace policy handler. So rename event > need not add this table to list of table to be bootstrapped. > * All the future events will be ignored by replace policy handler. > * For rename event (double rename) > ** If there is a rename, the table
[jira] [Commented] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16874560#comment-16874560
]
Hive QA commented on HIVE-21886:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973103/HIVE-21886.04.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 16327 tests
executed
*Failed tests:*
{noformat}
TestDataSourceProviderFactory - did not produce a TEST-*.xml file (likely timed
out) (batchId=232)
TestObjectStore - did not produce a TEST-*.xml file (likely timed out)
(batchId=232)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/1/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-1/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12973103 - PreCommit-HIVE-Build
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch,
> HIVE-21886.03.patch, HIVE-21886.04.patch, HIVE-21886.04.patch
>
> Time Spent: 11h 10m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. If the old table does not match the old policy drop it, even if the
> table is not present at target.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
> 2. If the new policy does not match the table name, then ignore the
> event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> * As the old name is not matching the new policy, the rename events will
> be ignored.
> * So nothing to be done for this scenario.
> New name matching new policy
> * As the new name is matching both old and new policy, replace handler
> will not bootstrap the table.
> * Add the table to the list of tables to be bootstrapped.
> * Ignore all the events with new name.
> * If there is a drop event for the table (with new name), then remove
> the table from the the list of table to be bootstrapped.
> * In case of rename event (double rename)
> ** If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ** If the new name does not satisfies then just removed the table name
> from the list o
[jira] [Commented] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16874541#comment-16874541
]
Hive QA commented on HIVE-21886:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
41s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
3s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-1/dev-support/hive-personality.sh
|
| git revision | master / e000e2f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-1/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch,
> HIVE-21886.03.patch, HIVE-21886.04.patch, HIVE-21886.04.patch
>
> Time Spent: 11h 10m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
>
[jira] [Commented] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16874201#comment-16874201
]
Hive QA commented on HIVE-21886:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973052/HIVE-21886.04.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 16330 tests
executed
*Failed tests:*
{noformat}
TestReplAcrossInstancesWithJsonMessageFormat - did not produce a TEST-*.xml
file (likely timed out) (batchId=255)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_2]
(batchId=194)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17771/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17771/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17771/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12973052 - PreCommit-HIVE-Build
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch,
> HIVE-21886.03.patch, HIVE-21886.04.patch
>
> Time Spent: 11h 10m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. If the old table does not match the old policy drop it, even if the
> table is not present at target.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
> 2. If the new policy does not match the table name, then ignore the
> event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> * As the old name is not matching the new policy, the rename events will
> be ignored.
> * So nothing to be done for this scenario.
> New name matching new policy
> * As the new name is matching both old and new policy, replace handler
> will not bootstrap the table.
> * Add the table to the list of tables to be bootstrapped.
> * Ignore all the events with new name.
> * If there is a drop event for the table (with new name), then remove
> the table from the the list of table to be bootstrapped.
> * In case of rename event (double rename)
> ** If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ** If the new name does not satisfies then just removed the t
[jira] [Commented] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16874161#comment-16874161
]
Hive QA commented on HIVE-21886:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
52s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
8s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 42s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17771/dev-support/hive-personality.sh
|
| git revision | master / e000e2f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17771/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch,
> HIVE-21886.03.patch, HIVE-21886.04.patch
>
> Time Spent: 11h 10m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. If the old
[jira] [Commented] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873869#comment-16873869
]
Hive QA commented on HIVE-21886:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973033/HIVE-21886.03.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 16343 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestObjectStore.catalogs (batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testDatabaseOps (batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testDeprecatedConfigIsOverwritten
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropParitionsCleanup
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropPartitionsCacheCrossSession
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSqlErrorMetrics
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testEmptyTrustStoreProps
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testMasterKeyOps (batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testMaxEventResponse
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testPartitionOps (batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testQueryCloseOnError
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testRoleOps (batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testTableOps (batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testUseSSLProperty
(batchId=232)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17766/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17766/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17766/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12973033 - PreCommit-HIVE-Build
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch,
> HIVE-21886.03.patch
>
> Time Spent: 11h 10m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. If the old table does not match the old policy drop it, even if the
> table is not present at target.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
> 2. If the new policy does not match the table name, then ignore the
> event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new polic
[jira] [Commented] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873853#comment-16873853
]
Hive QA commented on HIVE-21886:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
43s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
7s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
44s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
26s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
16s{color} | {color:red} itests/hive-unit: The patch generated 2 new + 0
unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 42s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17766/dev-support/hive-personality.sh
|
| git revision | master / 22a5e21 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17766/yetus/diff-checkstyle-itests_hive-unit.txt
|
| modules | C: ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17766/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch,
> HIVE-21886.03.patch
>
> Time Spent: 11h 10m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bo
[jira] [Commented] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873558#comment-16873558
]
Hive QA commented on HIVE-21886:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972944/HIVE-21886.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 16314 tests
executed
*Failed tests:*
{noformat}
TestReplAcrossInstancesWithJsonMessageFormat - did not produce a TEST-*.xml
file (likely timed out) (batchId=255)
org.apache.hive.service.cli.session.TestSessionManagerMetrics.testAbandonedSessionMetrics
(batchId=236)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17754/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17754/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17754/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12972944 - PreCommit-HIVE-Build
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 8.5h
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> * As the old name is not matching the new policy, the rename events will
> be ignored.
> * So nothing to be done for this scenario.
> New name matching new policy
> * As the new name is matching both old and new policy, replace handler
> will not bootstrap the table.
> * Add the table to the list of tables to be bootstrapped.
> * Ignore all the events with new name.
> * If there is a drop event for the table (with new name), then remove
> the table from the the list of table to be bootstrapped.
> * In case of rename event (double rename)
> ** If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ** If the new name does not satisfies then just removed the table name
> from the list of tables to be bootstrapped.
> ## Old name is matching new policy – As per replace policy handler, which
> checks based on old table, the table should be bootstrapped and event should
> be ignored. But rename handler should decide bas
[jira] [Commented] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873520#comment-16873520
]
Hive QA commented on HIVE-21886:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
5s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
42s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
17s{color} | {color:red} itests/hive-unit: The patch generated 9 new + 0
unchanged - 0 fixed = 9 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 57s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17754/dev-support/hive-personality.sh
|
| git revision | master / ee3aeb7 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17754/yetus/diff-checkstyle-itests_hive-unit.txt
|
| modules | C: ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17754/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 8.5h
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
>
[jira] [Commented] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16872984#comment-16872984
]
Hive QA commented on HIVE-21886:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972889/HIVE-21886.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 16284 tests
executed
*Failed tests:*
{noformat}
TestReplAcrossInstancesWithJsonMessageFormat - did not produce a TEST-*.xml
file (likely timed out) (batchId=255)
TestReplicationScenariosExternalTables - did not produce a TEST-*.xml file
(likely timed out) (batchId=255)
TestTableLevelReplicationScenarios - did not produce a TEST-*.xml file (likely
timed out) (batchId=255)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17743/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17743/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17743/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12972889 - PreCommit-HIVE-Build
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> * As the old name is not matching the new policy, the rename events will
> be ignored.
> * So nothing to be done for this scenario.
> New name matching new policy
> * As the new name is matching both old and new policy, replace handler
> will not bootstrap the table.
> * Add the table to the list of tables to be bootstrapped.
> * Ignore all the events with new name.
> * If there is a drop event for the table (with new name), then remove
> the table from the the list of table to be bootstrapped.
> * In case of rename event (double rename)
> ** If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ** If the new name does not satisfies then just removed the table name
> from the list of tables to be bootstrapped.
> ## Old name is matching new policy – As per replace policy handler, which
> checks based on old table, the table shou
[jira] [Commented] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16872942#comment-16872942
]
Hive QA commented on HIVE-21886:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
52s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
1s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 1 new + 8 unchanged - 0 fixed
= 9 total (was 8) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
19s{color} | {color:red} itests/hive-unit: The patch generated 25 new + 0
unchanged - 0 fixed = 25 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17743/dev-support/hive-personality.sh
|
| git revision | master / 967a1cc |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17743/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17743/yetus/diff-checkstyle-itests_hive-unit.txt
|
| modules | C: ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17743/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusio
