[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[ https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17209227#comment-17209227 ] ASF GitHub Bot commented on PHOENIX-5435: - gjacoby126 opened a new pull request #913: URL: https://github.com/apache/phoenix/pull/913 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Annotate HBase WALs with Phoenix Metadata > - > > Key: PHOENIX-5435 > URL: https://issues.apache.org/jira/browse/PHOENIX-5435 > Project: Phoenix > Issue Type: Sub-task >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > > HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase > replication and some HBase backup frameworks. The WALs contain HBase-level > metadata such as table and region, but lack Phoenix-level metadata. That > means that it's quite difficult to build correct logic that needs to know > about Phoenix-level constructs such as multi-tenancy, views, or indexes. > HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate > extra key/value pairs of metadata into the HBase WAL. We should have the > option to annotate the tuple , or > some hashed way to reconstruct that tuple into the WAL. It should have a > feature toggle so operators who don't need it don't bear the slight extra > storage cost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[
https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17209265#comment-17209265
]
Hadoop QA commented on PHOENIX-5435:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 5m
37s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
1s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} 4.x Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
9s{color} | {color:green} 4.x passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} 4.x passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
14s{color} | {color:green} 4.x passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
7s{color} | {color:green} 4.x passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
24s{color} | {color:blue} phoenix-hbase-compat-1.5.0 in 4.x has 3 extant
spotbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
21s{color} | {color:blue} phoenix-hbase-compat-1.4.0 in 4.x has 3 extant
spotbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
20s{color} | {color:blue} phoenix-hbase-compat-1.3.0 in 4.x has 3 extant
spotbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 2m
42s{color} | {color:blue} phoenix-core in 4.x has 957 extant spotbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 3m
42s{color} | {color:blue} root in 4.x has 1011 extant spotbugs warnings.
{color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 3m
22s{color} | {color:red} root: The patch generated 272 new + 6145 unchanged -
41 fixed = 6417 total (was 6186) {color} |
| {color:green}+1{color} | {color:green} prototool {color} | {color:green} 0m
1s{color} | {color:green} There were no new prototool issues. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
40s{color} | {color:red} phoenix-core generated 1 new + 99 unchanged - 1 fixed
= 100 total (was 100) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m
5s{color} | {color:red} root generated 1 new + 131 unchanged - 1 fixed = 132
total (was 132) {color} |
| {color:red}-1{color} | {color:red} spotbugs {color} | {color:red} 3m
6s{color} | {color:red} phoenix-core generated 4 new + 956 unchanged - 1 fixed
= 960 total (was 957) {color} |
| {color:red}-1{color} | {color:red} spotbugs {color} | {color:red} 4m
6s{color} | {color:red} root generated 4 new + 1010 unchanged - 1 fixed = 1014
total (was 1011) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[ https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17209270#comment-17209270 ] ASF GitHub Bot commented on PHOENIX-5435: - stoty commented on pull request #913: URL: https://github.com/apache/phoenix/pull/913#issuecomment-704670166 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 6m 59s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | ||| _ 4.x Compile Tests _ | | +0 :ok: | mvndep | 0m 43s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 9m 16s | 4.x passed | | +1 :green_heart: | compile | 1m 53s | 4.x passed | | +1 :green_heart: | checkstyle | 3m 23s | 4.x passed | | +1 :green_heart: | javadoc | 3m 7s | 4.x passed | | +0 :ok: | spotbugs | 0m 35s | phoenix-hbase-compat-1.5.0 in 4.x has 3 extant spotbugs warnings. | | +0 :ok: | spotbugs | 0m 30s | phoenix-hbase-compat-1.4.0 in 4.x has 3 extant spotbugs warnings. | | +0 :ok: | spotbugs | 0m 28s | phoenix-hbase-compat-1.3.0 in 4.x has 3 extant spotbugs warnings. | | +0 :ok: | spotbugs | 5m 46s | root in 4.x has 1011 extant spotbugs warnings. | | +0 :ok: | spotbugs | 3m 29s | phoenix-core in 4.x has 957 extant spotbugs warnings. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 28s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 56s | the patch passed | | +1 :green_heart: | compile | 1m 48s | the patch passed | | +1 :green_heart: | cc | 1m 48s | the patch passed | | +1 :green_heart: | javac | 1m 48s | the patch passed | | -1 :x: | checkstyle | 3m 35s | root: The patch generated 272 new + 6145 unchanged - 41 fixed = 6417 total (was 6186) | | +1 :green_heart: | prototool | 0m 2s | There were no new prototool issues. | | -1 :x: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | | +1 :green_heart: | javadoc | 3m 5s | the patch passed | | -1 :x: | spotbugs | 4m 0s | phoenix-core generated 4 new + 956 unchanged - 1 fixed = 960 total (was 957) | | -1 :x: | spotbugs | 5m 29s | root generated 4 new + 1010 unchanged - 1 fixed = 1014 total (was 1011) | ||| _ Other Tests _ | | -1 :x: | unit | 143m 30s | root in the patch failed. | | +1 :green_heart: | asflicense | 2m 56s | The patch does not generate ASF License warnings. | | | | 214m 11s | | | Reason | Tests | |---:|:--| | FindBugs | module:phoenix-core | | | org.apache.phoenix.hbase.index.IndexRegionObserver.DEFAULT_PHOENIX_APPEND_METADATA_TO_WAL isn't final but should be At IndexRegionObserver.java:be At IndexRegionObserver.java:[line 125] | | | org.apache.phoenix.hbase.index.IndexRegionObserver.PHOENIX_APPEND_METADATA_TO_WAL isn't final but should be At IndexRegionObserver.java:be At IndexRegionObserver.java:[line 124] | | | Load of known null value in org.apache.phoenix.hbase.index.IndexRegionObserver.prepareIndexMutations(IndexRegionObserver$BatchMutateContext, List, long) At IndexRegionObserver.java:in org.apache.phoenix.hbase.index.IndexRegionObserver.prepareIndexMutations(IndexRegionObserver$BatchMutateContext, List, long) At IndexRegionObserver.java:[line 719] | | | Switch statement found in org.apache.phoenix.hbase.index.scanner.FilteredKeyValueScanner.seekToNextUnfilteredKeyValue() where default case is missing At FilteredKeyValueScanner.java:where default case is missing At FilteredKeyValueScanner.java:[lines 84-101] | | FindBugs | module:root | | | org.apache.phoenix.hbase.index.IndexRegionObserver.DEFAULT_PHOENIX_APPEND_METADATA_TO_WAL isn't final but should be At IndexRegionObserver.java:be At IndexRegionObserver.java:[line 125] | | | org.apache.phoenix.hbase.index.IndexRegionObserver.PHOENIX_APPEND_METADATA_TO_WAL isn't final but should be At IndexRegionObserver.java:be At IndexRegionObserver.java:[line 124] | | | Load of known null value in org.apache.phoenix.hbase.index.IndexRegionObserver.prepareIndexMutations(IndexRegionObserver$BatchMutateContext, List, long) At IndexRegionObserver.jav
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[ https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17212464#comment-17212464 ] ASF GitHub Bot commented on PHOENIX-5435: - stoty commented on pull request #913: URL: https://github.com/apache/phoenix/pull/913#issuecomment-707205451 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 0s | Docker mode activated. | | -1 :x: | docker | 2m 30s | Docker failed to build yetus/phoenix:955047a0b. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/phoenix/pull/913 | | Console output | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-913/2/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Annotate HBase WALs with Phoenix Metadata > - > > Key: PHOENIX-5435 > URL: https://issues.apache.org/jira/browse/PHOENIX-5435 > Project: Phoenix > Issue Type: Sub-task >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > Attachments: PHOENIX-5435-4.x.patch > > > HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase > replication and some HBase backup frameworks. The WALs contain HBase-level > metadata such as table and region, but lack Phoenix-level metadata. That > means that it's quite difficult to build correct logic that needs to know > about Phoenix-level constructs such as multi-tenancy, views, or indexes. > HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate > extra key/value pairs of metadata into the HBase WAL. We should have the > option to annotate the tuple , or > some hashed way to reconstruct that tuple into the WAL. It should have a > feature toggle so operators who don't need it don't bear the slight extra > storage cost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[ https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17212756#comment-17212756 ] ASF GitHub Bot commented on PHOENIX-5435: - stoty commented on pull request #913: URL: https://github.com/apache/phoenix/pull/913#issuecomment-707423341 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 4m 18s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | ||| _ 4.x Compile Tests _ | | +0 :ok: | mvndep | 0m 31s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 26m 3s | 4.x passed | | +1 :green_heart: | compile | 1m 22s | 4.x passed | | +1 :green_heart: | checkstyle | 4m 2s | 4.x passed | | +1 :green_heart: | javadoc | 2m 22s | 4.x passed | | +0 :ok: | spotbugs | 0m 26s | phoenix-hbase-compat-1.5.0 in 4.x has 3 extant spotbugs warnings. | | +0 :ok: | spotbugs | 0m 23s | phoenix-hbase-compat-1.4.0 in 4.x has 3 extant spotbugs warnings. | | +0 :ok: | spotbugs | 0m 21s | phoenix-hbase-compat-1.3.0 in 4.x has 3 extant spotbugs warnings. | | +0 :ok: | spotbugs | 3m 54s | root in 4.x has 1011 extant spotbugs warnings. | | +0 :ok: | spotbugs | 2m 40s | phoenix-core in 4.x has 957 extant spotbugs warnings. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 23m 43s | the patch passed | | +1 :green_heart: | compile | 1m 27s | the patch passed | | +1 :green_heart: | cc | 1m 27s | the patch passed | | +1 :green_heart: | javac | 1m 27s | the patch passed | | -1 :x: | checkstyle | 4m 14s | root: The patch generated 308 new + 8142 unchanged - 61 fixed = 8450 total (was 8203) | | +1 :green_heart: | prototool | 0m 2s | There were no new prototool issues. | | -1 :x: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | | +1 :green_heart: | javadoc | 2m 23s | the patch passed | | -1 :x: | spotbugs | 3m 8s | phoenix-core generated 4 new + 956 unchanged - 1 fixed = 960 total (was 957) | | -1 :x: | spotbugs | 4m 10s | root generated 4 new + 1010 unchanged - 1 fixed = 1014 total (was 1011) | ||| _ Other Tests _ | | -1 :x: | unit | 114m 35s | root in the patch failed. | | +1 :green_heart: | asflicense | 2m 51s | The patch does not generate ASF License warnings. | | | | 209m 17s | | | Reason | Tests | |---:|:--| | FindBugs | module:phoenix-core | | | org.apache.phoenix.hbase.index.IndexRegionObserver.DEFAULT_PHOENIX_APPEND_METADATA_TO_WAL isn't final but should be At IndexRegionObserver.java:be At IndexRegionObserver.java:[line 125] | | | org.apache.phoenix.hbase.index.IndexRegionObserver.PHOENIX_APPEND_METADATA_TO_WAL isn't final but should be At IndexRegionObserver.java:be At IndexRegionObserver.java:[line 124] | | | Load of known null value in org.apache.phoenix.hbase.index.IndexRegionObserver.prepareIndexMutations(IndexRegionObserver$BatchMutateContext, List, long) At IndexRegionObserver.java:in org.apache.phoenix.hbase.index.IndexRegionObserver.prepareIndexMutations(IndexRegionObserver$BatchMutateContext, List, long) At IndexRegionObserver.java:[line 719] | | | Switch statement found in org.apache.phoenix.hbase.index.scanner.FilteredKeyValueScanner.seekToNextUnfilteredKeyValue() where default case is missing At FilteredKeyValueScanner.java:where default case is missing At FilteredKeyValueScanner.java:[lines 84-101] | | FindBugs | module:root | | | org.apache.phoenix.hbase.index.IndexRegionObserver.DEFAULT_PHOENIX_APPEND_METADATA_TO_WAL isn't final but should be At IndexRegionObserver.java:be At IndexRegionObserver.java:[line 125] | | | org.apache.phoenix.hbase.index.IndexRegionObserver.PHOENIX_APPEND_METADATA_TO_WAL isn't final but should be At IndexRegionObserver.java:be At IndexRegionObserver.java:[line 124] | | | Load of known null value in org.apache.phoenix.hbase.index.IndexRegionObserver.prepareIndexMutations(IndexRegionObserver$BatchMutateContext, List, long) At IndexRegionObserver.jav
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[
https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17229577#comment-17229577
]
ASF GitHub Bot commented on PHOENIX-5435:
-
ChinmaySKulkarni commented on a change in pull request #913:
URL: https://github.com/apache/phoenix/pull/913#discussion_r520944121
##
File path:
phoenix-core/src/it/java/org/apache/phoenix/end2end/ImmutableTableIT.java
##
@@ -98,4 +99,13 @@ public void
testQueryWithMultipleColumnFamiliesAndMultipleConditionsForImmutable
}
}
}
+
+@Test
+public void testImmutableTableCreation() throws Exception {
Review comment:
We already have immutable table creation in other tests in this class.
Do we need this test?
##
File path:
phoenix-core/src/it/java/org/apache/phoenix/end2end/WALAnnotationIT.java
##
@@ -0,0 +1,520 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.end2end;
+
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionInfo;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.coprocessor.BaseWALObserver;
+import org.apache.hadoop.hbase.coprocessor.ObserverContext;
+import org.apache.hadoop.hbase.coprocessor.WALCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.wal.WALCoprocessorHost;
+import org.apache.hadoop.hbase.regionserver.wal.WALEdit;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.wal.WAL;
+import org.apache.hadoop.hbase.wal.WALKey;
+import org.apache.phoenix.compat.hbase.HbaseCompatCapabilities;
+import org.apache.phoenix.compat.hbase.coprocessor.CompatIndexRegionObserver;
+import org.apache.phoenix.execute.MutationState;
+import org.apache.phoenix.hbase.index.IndexRegionObserver;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.schema.PTableType;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.MetaDataUtil;
+import org.apache.phoenix.util.PhoenixRuntime;
+import org.apache.phoenix.util.ReadOnlyProps;
+import org.apache.phoenix.util.SchemaUtil;
+import org.apache.phoenix.util.TestDDLUtil;
+import org.apache.phoenix.util.TestUtil;
+import org.junit.Assert;
+import org.junit.Assume;
+import org.junit.BeforeClass;
+import org.junit.Ignore;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+
+import static org.junit.Assert.assertNotNull;
+
+@RunWith(Parameterized.class)
+@Category(NeedsOwnMiniClusterTest.class)
+public class WALAnnotationIT extends BaseUniqueNamesOwnClusterIT {
+private TestDDLUtil ddlUtil;
+private boolean isImmutable;
+private boolean isMultiTenant;
+
+// name is used by failsafe as file name in reports
+@Parameterized.Parameters(name =
"WALAnnotationIT_isImmutable={0}_isMultiTenant={1}")
+public static synchronized Collection data() {
+return Arrays.asList(new Object[]{true, true}, new Object[]{true,
false},
+new Object[]{false, true}, new Object[]{false, false});
+}
+
+public WALAnnotationIT(boolean isImmutable, boolean isMultiTenant) {
+this.isImmutable = isImmutable;
+this.isMultiTenant = isMultiTenant;
+ddlUtil = new TestDDLUtil(false);
+}
+
+@BeforeClass
+public static synchronized void doSetup() throws Exception {
+Map props = new HashMap(2);
+props.put("hbase.coprocessor.wal.classes",
+AnnotatedWALObserver.class.getName());
+props.put(IndexRegionObserver.PHOENIX_APPEND_METADATA_TO_WAL, "true");
+props.put(QueryServices.ENABLE_SERVER_UPSERT_SELECT, "true");
+setUpTestDriver
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[ https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17229592#comment-17229592 ] ASF GitHub Bot commented on PHOENIX-5435: - ChinmaySKulkarni commented on pull request #913: URL: https://github.com/apache/phoenix/pull/913#issuecomment-725042677 @gjacoby126 Some high-level questions: 1. For interactions with old clients, do we still get these WAL annotations (for ex: 4.16 server + 4.14 client) or do we only get this after clients are upgraded? 2. Is it worth-while to maintain a whitelist for tables/views/indices where we may not want to enable this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Annotate HBase WALs with Phoenix Metadata > - > > Key: PHOENIX-5435 > URL: https://issues.apache.org/jira/browse/PHOENIX-5435 > Project: Phoenix > Issue Type: Sub-task >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > Attachments: PHOENIX-5435-4.x.patch > > > HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase > replication and some HBase backup frameworks. The WALs contain HBase-level > metadata such as table and region, but lack Phoenix-level metadata. That > means that it's quite difficult to build correct logic that needs to know > about Phoenix-level constructs such as multi-tenancy, views, or indexes. > HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate > extra key/value pairs of metadata into the HBase WAL. We should have the > option to annotate the tuple , or > some hashed way to reconstruct that tuple into the WAL. It should have a > feature toggle so operators who don't need it don't bear the slight extra > storage cost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[ https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17229614#comment-17229614 ] ASF GitHub Bot commented on PHOENIX-5435: - gjacoby126 commented on pull request #913: URL: https://github.com/apache/phoenix/pull/913#issuecomment-725058655 1. This feature requires a 4.16 client and a 4.16 server. The server code is intended to gracefully no-op and not annotate if the client hasn't provided anything. 2. The question would be how to store the list of tables to annotate, since so much of the logic is client-side where we don't have the server-side hbase-site.xml. It could be Yet Another PTable Attribute, but I'm unsure if it's worth the complexity -- wdyt? The conceptually cleanest way would be to annotate replicated tables and not annotate unreplicated ones, but replication scope is on the TableDescriptor, which we don't usually have access to client-side. A critique I received offline to this PR from @priyankporwal suggested that annotating indexes might not be worth the storage cost, so we could also consider exempting indexes in the next draft (which is waiting on PHOENIX-6186 to populate the DDL timestamps) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Annotate HBase WALs with Phoenix Metadata > - > > Key: PHOENIX-5435 > URL: https://issues.apache.org/jira/browse/PHOENIX-5435 > Project: Phoenix > Issue Type: Sub-task >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > Attachments: PHOENIX-5435-4.x.patch > > > HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase > replication and some HBase backup frameworks. The WALs contain HBase-level > metadata such as table and region, but lack Phoenix-level metadata. That > means that it's quite difficult to build correct logic that needs to know > about Phoenix-level constructs such as multi-tenancy, views, or indexes. > HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate > extra key/value pairs of metadata into the HBase WAL. We should have the > option to annotate the tuple , or > some hashed way to reconstruct that tuple into the WAL. It should have a > feature toggle so operators who don't need it don't bear the slight extra > storage cost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[ https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17236569#comment-17236569 ] ASF GitHub Bot commented on PHOENIX-5435: - stoty commented on pull request #913: URL: https://github.com/apache/phoenix/pull/913#issuecomment-731496674 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 3m 27s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. | ||| _ 4.x Compile Tests _ | | +0 :ok: | mvndep | 0m 42s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 12m 54s | 4.x passed | | +1 :green_heart: | compile | 1m 32s | 4.x passed | | +1 :green_heart: | checkstyle | 2m 53s | 4.x passed | | +1 :green_heart: | javadoc | 2m 23s | 4.x passed | | +0 :ok: | spotbugs | 0m 26s | phoenix-hbase-compat-1.5.0 in 4.x has 3 extant spotbugs warnings. | | +0 :ok: | spotbugs | 0m 23s | phoenix-hbase-compat-1.4.0 in 4.x has 3 extant spotbugs warnings. | | +0 :ok: | spotbugs | 0m 23s | phoenix-hbase-compat-1.3.0 in 4.x has 3 extant spotbugs warnings. | | +0 :ok: | spotbugs | 4m 25s | root in 4.x has 1004 extant spotbugs warnings. | | +0 :ok: | spotbugs | 2m 57s | phoenix-core in 4.x has 950 extant spotbugs warnings. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 6m 57s | the patch passed | | +1 :green_heart: | compile | 1m 36s | the patch passed | | +1 :green_heart: | cc | 1m 36s | the patch passed | | +1 :green_heart: | javac | 1m 36s | the patch passed | | -1 :x: | checkstyle | 3m 0s | root: The patch generated 330 new + 8319 unchanged - 76 fixed = 8649 total (was 8395) | | +1 :green_heart: | prototool | 0m 2s | There were no new prototool issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +1 :green_heart: | javadoc | 2m 28s | the patch passed | | -1 :x: | spotbugs | 3m 29s | phoenix-core generated 4 new + 950 unchanged - 0 fixed = 954 total (was 950) | | -1 :x: | spotbugs | 4m 36s | root generated 4 new + 1004 unchanged - 0 fixed = 1008 total (was 1004) | ||| _ Other Tests _ | | -1 :x: | unit | 195m 56s | root in the patch failed. | | +1 :green_heart: | asflicense | 2m 22s | The patch does not generate ASF License warnings. | | | | 259m 36s | | | Reason | Tests | |---:|:--| | FindBugs | module:phoenix-core | | | org.apache.phoenix.hbase.index.IndexRegionObserver.DEFAULT_PHOENIX_APPEND_METADATA_TO_WAL isn't final but should be At IndexRegionObserver.java:be At IndexRegionObserver.java:[line 126] | | | org.apache.phoenix.hbase.index.IndexRegionObserver.PHOENIX_APPEND_METADATA_TO_WAL isn't final but should be At IndexRegionObserver.java:be At IndexRegionObserver.java:[line 125] | | | Load of known null value in org.apache.phoenix.hbase.index.IndexRegionObserver.prepareIndexMutations(IndexRegionObserver$BatchMutateContext, List, long) At IndexRegionObserver.java:in org.apache.phoenix.hbase.index.IndexRegionObserver.prepareIndexMutations(IndexRegionObserver$BatchMutateContext, List, long) At IndexRegionObserver.java:[line 800] | | | Switch statement found in org.apache.phoenix.hbase.index.scanner.FilteredKeyValueScanner.seekToNextUnfilteredKeyValue() where default case is missing At FilteredKeyValueScanner.java:where default case is missing At FilteredKeyValueScanner.java:[lines 84-101] | | FindBugs | module:root | | | org.apache.phoenix.hbase.index.IndexRegionObserver.DEFAULT_PHOENIX_APPEND_METADATA_TO_WAL isn't final but should be At IndexRegionObserver.java:be At IndexRegionObserver.java:[line 126] | | | org.apache.phoenix.hbase.index.IndexRegionObserver.PHOENIX_APPEND_METADATA_TO_WAL isn't final but should be At IndexRegionObserver.java:be At IndexRegionObserver.java:[line 125] | | | Load of known null value in org.apache.phoenix.hbase.index.IndexRegionObserver.prepareIndexMutations(IndexRegionObserver$BatchMutateContext, List, long) At IndexRegionObserver.java:in org.apache.phoenix.hbase.index.IndexRegionObserver.prepareIndexMutations(
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[ https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17238989#comment-17238989 ] ASF GitHub Bot commented on PHOENIX-5435: - stoty commented on pull request #913: URL: https://github.com/apache/phoenix/pull/913#issuecomment-734013444 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 4m 11s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. | ||| _ 4.x Compile Tests _ | | +0 :ok: | mvndep | 0m 32s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 10m 44s | 4.x passed | | +1 :green_heart: | compile | 1m 25s | 4.x passed | | +1 :green_heart: | checkstyle | 10m 32s | 4.x passed | | +1 :green_heart: | javadoc | 2m 22s | 4.x passed | | +0 :ok: | spotbugs | 0m 27s | phoenix-hbase-compat-1.5.0 in 4.x has 3 extant spotbugs warnings. | | +0 :ok: | spotbugs | 0m 22s | phoenix-hbase-compat-1.4.0 in 4.x has 3 extant spotbugs warnings. | | +0 :ok: | spotbugs | 0m 24s | phoenix-hbase-compat-1.3.0 in 4.x has 3 extant spotbugs warnings. | | +0 :ok: | spotbugs | 3m 59s | root in 4.x has 1004 extant spotbugs warnings. | | +0 :ok: | spotbugs | 2m 38s | phoenix-core in 4.x has 950 extant spotbugs warnings. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 59s | the patch passed | | +1 :green_heart: | compile | 1m 28s | the patch passed | | +1 :green_heart: | cc | 1m 28s | the patch passed | | +1 :green_heart: | javac | 1m 28s | the patch passed | | -1 :x: | checkstyle | 10m 46s | root: The patch generated 541 new + 23086 unchanged - 193 fixed = 23627 total (was 23279) | | +1 :green_heart: | prototool | 0m 1s | There were no new prototool issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | | +1 :green_heart: | javadoc | 2m 26s | the patch passed | | -1 :x: | spotbugs | 3m 11s | phoenix-core generated 5 new + 950 unchanged - 0 fixed = 955 total (was 950) | | -1 :x: | spotbugs | 4m 12s | root generated 5 new + 1004 unchanged - 0 fixed = 1009 total (was 1004) | ||| _ Other Tests _ | | -1 :x: | unit | 143m 37s | root in the patch failed. | | +1 :green_heart: | asflicense | 0m 45s | The patch does not generate ASF License warnings. | | | | 214m 46s | | | Reason | Tests | |---:|:--| | FindBugs | module:phoenix-core | | | org.apache.phoenix.hbase.index.IndexRegionObserver.DEFAULT_PHOENIX_APPEND_METADATA_TO_WAL isn't final but should be At IndexRegionObserver.java:be At IndexRegionObserver.java:[line 126] | | | org.apache.phoenix.hbase.index.IndexRegionObserver.PHOENIX_APPEND_METADATA_TO_WAL isn't final but should be At IndexRegionObserver.java:be At IndexRegionObserver.java:[line 125] | | | Load of known null value in org.apache.phoenix.hbase.index.IndexRegionObserver.prepareIndexMutations(IndexRegionObserver$BatchMutateContext, List, long) At IndexRegionObserver.java:in org.apache.phoenix.hbase.index.IndexRegionObserver.prepareIndexMutations(IndexRegionObserver$BatchMutateContext, List, long) At IndexRegionObserver.java:[line 800] | | | Switch statement found in org.apache.phoenix.hbase.index.scanner.FilteredKeyValueScanner.seekToNextUnfilteredKeyValue() where default case is missing At FilteredKeyValueScanner.java:where default case is missing At FilteredKeyValueScanner.java:[lines 84-101] | | | org.apache.phoenix.jdbc.PhoenixDatabaseMetaData.CHANGE_DETECTION_ENABLED_BYTES is a mutable array At MetaDataEndpointImpl.java: At MetaDataEndpointImpl.java:[line 336] | | FindBugs | module:root | | | org.apache.phoenix.hbase.index.IndexRegionObserver.DEFAULT_PHOENIX_APPEND_METADATA_TO_WAL isn't final but should be At IndexRegionObserver.java:be At IndexRegionObserver.java:[line 126] | | | org.apache.phoenix.hbase.index.IndexRegionObserver.PHOENIX_APPEND_METADATA_TO_WAL isn't final but should be At IndexRegionObserver.java:be At IndexRegionObserver.java:[line 125] | | | Load of known null value in org.apache.phoenix.hbase.index.IndexRegionObs
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[ https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17241037#comment-17241037 ] ASF GitHub Bot commented on PHOENIX-5435: - stoty commented on pull request #913: URL: https://github.com/apache/phoenix/pull/913#issuecomment-736036102 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 33s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. | ||| _ 4.x Compile Tests _ | | +0 :ok: | mvndep | 0m 32s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 10m 53s | 4.x passed | | +1 :green_heart: | compile | 1m 23s | 4.x passed | | +1 :green_heart: | checkstyle | 9m 38s | 4.x passed | | +1 :green_heart: | javadoc | 2m 26s | 4.x passed | | +0 :ok: | spotbugs | 0m 27s | phoenix-hbase-compat-1.5.0 in 4.x has 3 extant spotbugs warnings. | | +0 :ok: | spotbugs | 0m 23s | phoenix-hbase-compat-1.4.0 in 4.x has 3 extant spotbugs warnings. | | +0 :ok: | spotbugs | 0m 22s | phoenix-hbase-compat-1.3.0 in 4.x has 3 extant spotbugs warnings. | | +0 :ok: | spotbugs | 4m 9s | root in 4.x has 1004 extant spotbugs warnings. | | +0 :ok: | spotbugs | 2m 59s | phoenix-core in 4.x has 950 extant spotbugs warnings. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 59s | the patch passed | | +1 :green_heart: | compile | 1m 26s | the patch passed | | +1 :green_heart: | cc | 1m 26s | the patch passed | | +1 :green_heart: | javac | 1m 26s | the patch passed | | -1 :x: | checkstyle | 9m 5s | root: The patch generated 496 new + 23127 unchanged - 152 fixed = 23623 total (was 23279) | | +1 :green_heart: | prototool | 0m 2s | There were no new prototool issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | | +1 :green_heart: | javadoc | 2m 27s | the patch passed | | -1 :x: | spotbugs | 3m 44s | phoenix-core generated 3 new + 950 unchanged - 0 fixed = 953 total (was 950) | | -1 :x: | spotbugs | 5m 25s | root generated 3 new + 1004 unchanged - 0 fixed = 1007 total (was 1004) | ||| _ Other Tests _ | | -1 :x: | unit | 132m 36s | root in the patch failed. | | +1 :green_heart: | asflicense | 2m 37s | The patch does not generate ASF License warnings. | | | | 204m 21s | | | Reason | Tests | |---:|:--| | FindBugs | module:phoenix-core | | | Load of known null value in org.apache.phoenix.hbase.index.IndexRegionObserver.prepareIndexMutations(IndexRegionObserver$BatchMutateContext, List, long) At IndexRegionObserver.java:in org.apache.phoenix.hbase.index.IndexRegionObserver.prepareIndexMutations(IndexRegionObserver$BatchMutateContext, List, long) At IndexRegionObserver.java:[line 800] | | | Switch statement found in org.apache.phoenix.hbase.index.scanner.FilteredKeyValueScanner.seekToNextUnfilteredKeyValue() where default case is missing At FilteredKeyValueScanner.java:where default case is missing At FilteredKeyValueScanner.java:[lines 84-101] | | | org.apache.phoenix.jdbc.PhoenixDatabaseMetaData.CHANGE_DETECTION_ENABLED_BYTES is a mutable array At MetaDataEndpointImpl.java: At MetaDataEndpointImpl.java:[line 336] | | FindBugs | module:root | | | Load of known null value in org.apache.phoenix.hbase.index.IndexRegionObserver.prepareIndexMutations(IndexRegionObserver$BatchMutateContext, List, long) At IndexRegionObserver.java:in org.apache.phoenix.hbase.index.IndexRegionObserver.prepareIndexMutations(IndexRegionObserver$BatchMutateContext, List, long) At IndexRegionObserver.java:[line 800] | | | Switch statement found in org.apache.phoenix.hbase.index.scanner.FilteredKeyValueScanner.seekToNextUnfilteredKeyValue() where default case is missing At FilteredKeyValueScanner.java:where default case is missing At FilteredKeyValueScanner.java:[lines 84-101] | | | org.apache.phoenix.jdbc.PhoenixDatabaseMetaData.CHANGE_DETECTION_ENABLED_BYTES is a mutable array At MetaDataEndpointImpl.java: At MetaDataEndpointImpl.java:[line 336] | | Failed junit tests | phoenix.end2end.ViewTTLIT
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[
https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17241938#comment-17241938
]
ASF GitHub Bot commented on PHOENIX-5435:
-
gjacoby126 commented on a change in pull request #913:
URL: https://github.com/apache/phoenix/pull/913#discussion_r533789672
##
File path: phoenix-core/src/test/java/org/apache/phoenix/util/TestDDLUtil.java
##
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.util;
+
+import java.sql.Connection;
+import java.sql.SQLException;
+
+public class TestDDLUtil {
Review comment:
Switched WALAnnotationIT to use @jpisaac 's SchemaBuilder class (and
enhanced it a bit to fill in a couple feature gaps)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Annotate HBase WALs with Phoenix Metadata
> -
>
> Key: PHOENIX-5435
> URL: https://issues.apache.org/jira/browse/PHOENIX-5435
> Project: Phoenix
> Issue Type: Sub-task
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>Priority: Major
> Attachments: PHOENIX-5435-4.x.patch
>
>
> HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase
> replication and some HBase backup frameworks. The WALs contain HBase-level
> metadata such as table and region, but lack Phoenix-level metadata. That
> means that it's quite difficult to build correct logic that needs to know
> about Phoenix-level constructs such as multi-tenancy, views, or indexes.
> HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate
> extra key/value pairs of metadata into the HBase WAL. We should have the
> option to annotate the tuple , or
> some hashed way to reconstruct that tuple into the WAL. It should have a
> feature toggle so operators who don't need it don't bear the slight extra
> storage cost.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[ https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17241939#comment-17241939 ] ASF GitHub Bot commented on PHOENIX-5435: - gjacoby126 commented on a change in pull request #913: URL: https://github.com/apache/phoenix/pull/913#discussion_r533790379 ## File path: pom.xml ## @@ -1173,7 +1173,7 @@ - phoenix-hbase-compat-1.3.0-default + phoenix-hbase-compat-1.5.0-default Review comment: With 1.3 as the default, it's difficult to run the 1.5-specific tests like WALAnnotationIT in IntelliJ, and it means they won't get exercised in the automated test runs. FYI, @stoty . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Annotate HBase WALs with Phoenix Metadata > - > > Key: PHOENIX-5435 > URL: https://issues.apache.org/jira/browse/PHOENIX-5435 > Project: Phoenix > Issue Type: Sub-task >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > Attachments: PHOENIX-5435-4.x.patch > > > HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase > replication and some HBase backup frameworks. The WALs contain HBase-level > metadata such as table and region, but lack Phoenix-level metadata. That > means that it's quite difficult to build correct logic that needs to know > about Phoenix-level constructs such as multi-tenancy, views, or indexes. > HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate > extra key/value pairs of metadata into the HBase WAL. We should have the > option to annotate the tuple , or > some hashed way to reconstruct that tuple into the WAL. It should have a > feature toggle so operators who don't need it don't bear the slight extra > storage cost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[
https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17241942#comment-17241942
]
ASF GitHub Bot commented on PHOENIX-5435:
-
gjacoby126 commented on a change in pull request #913:
URL: https://github.com/apache/phoenix/pull/913#discussion_r533790609
##
File path:
phoenix-core/src/it/java/org/apache/phoenix/end2end/ImmutableTableIT.java
##
@@ -98,4 +99,13 @@ public void
testQueryWithMultipleColumnFamiliesAndMultipleConditionsForImmutable
}
}
}
+
+@Test
+public void testImmutableTableCreation() throws Exception {
Review comment:
Correct, @ChinmaySKulkarni , doesn't look like we need this. Removed it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Annotate HBase WALs with Phoenix Metadata
> -
>
> Key: PHOENIX-5435
> URL: https://issues.apache.org/jira/browse/PHOENIX-5435
> Project: Phoenix
> Issue Type: Sub-task
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>Priority: Major
> Attachments: PHOENIX-5435-4.x.patch
>
>
> HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase
> replication and some HBase backup frameworks. The WALs contain HBase-level
> metadata such as table and region, but lack Phoenix-level metadata. That
> means that it's quite difficult to build correct logic that needs to know
> about Phoenix-level constructs such as multi-tenancy, views, or indexes.
> HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate
> extra key/value pairs of metadata into the HBase WAL. We should have the
> option to annotate the tuple , or
> some hashed way to reconstruct that tuple into the WAL. It should have a
> feature toggle so operators who don't need it don't bear the slight extra
> storage cost.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[
https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17241940#comment-17241940
]
ASF GitHub Bot commented on PHOENIX-5435:
-
gjacoby126 commented on a change in pull request #913:
URL: https://github.com/apache/phoenix/pull/913#discussion_r533790439
##
File path:
phoenix-core/src/it/java/org/apache/phoenix/end2end/WALAnnotationIT.java
##
@@ -0,0 +1,520 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.end2end;
+
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionInfo;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.coprocessor.BaseWALObserver;
+import org.apache.hadoop.hbase.coprocessor.ObserverContext;
+import org.apache.hadoop.hbase.coprocessor.WALCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.wal.WALCoprocessorHost;
+import org.apache.hadoop.hbase.regionserver.wal.WALEdit;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.wal.WAL;
+import org.apache.hadoop.hbase.wal.WALKey;
+import org.apache.phoenix.compat.hbase.HbaseCompatCapabilities;
+import org.apache.phoenix.compat.hbase.coprocessor.CompatIndexRegionObserver;
+import org.apache.phoenix.execute.MutationState;
+import org.apache.phoenix.hbase.index.IndexRegionObserver;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.schema.PTableType;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.MetaDataUtil;
+import org.apache.phoenix.util.PhoenixRuntime;
+import org.apache.phoenix.util.ReadOnlyProps;
+import org.apache.phoenix.util.SchemaUtil;
+import org.apache.phoenix.util.TestDDLUtil;
+import org.apache.phoenix.util.TestUtil;
+import org.junit.Assert;
+import org.junit.Assume;
+import org.junit.BeforeClass;
+import org.junit.Ignore;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+
+import static org.junit.Assert.assertNotNull;
+
+@RunWith(Parameterized.class)
+@Category(NeedsOwnMiniClusterTest.class)
+public class WALAnnotationIT extends BaseUniqueNamesOwnClusterIT {
+private TestDDLUtil ddlUtil;
+private boolean isImmutable;
+private boolean isMultiTenant;
+
+// name is used by failsafe as file name in reports
+@Parameterized.Parameters(name =
"WALAnnotationIT_isImmutable={0}_isMultiTenant={1}")
+public static synchronized Collection data() {
+return Arrays.asList(new Object[]{true, true}, new Object[]{true,
false},
+new Object[]{false, true}, new Object[]{false, false});
+}
+
+public WALAnnotationIT(boolean isImmutable, boolean isMultiTenant) {
+this.isImmutable = isImmutable;
+this.isMultiTenant = isMultiTenant;
+ddlUtil = new TestDDLUtil(false);
+}
+
+@BeforeClass
+public static synchronized void doSetup() throws Exception {
+Map props = new HashMap(2);
+props.put("hbase.coprocessor.wal.classes",
+AnnotatedWALObserver.class.getName());
+props.put(IndexRegionObserver.PHOENIX_APPEND_METADATA_TO_WAL, "true");
+props.put(QueryServices.ENABLE_SERVER_UPSERT_SELECT, "true");
+setUpTestDriver(new ReadOnlyProps(props.entrySet().iterator()));
+}
+
+@Test
+public void testSimpleUpsertAndDelete() throws Exception {
+Assume.assumeTrue(HbaseCompatCapabilities.hasPreWALAppend());
+TestTableInfo tableInfo = new TestTableInfo();
+long minTimestamp = EnvironmentEdgeManager.currentTimeMillis();
+boolean createGlobalIndex = false;
+upsertAndDeleteHelper(tableInfo, createGlobalIndex);
+
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[ https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17241944#comment-17241944 ] ASF GitHub Bot commented on PHOENIX-5435: - gjacoby126 commented on pull request #913: URL: https://github.com/apache/phoenix/pull/913#issuecomment-736887345 In this latest revision: 1. I added a new boolean flag, CHANGE_DETECTION_ENABLED to System.Catalog which is configured via property on table or view creation. It can be changed via ALTER. Adding it to an index will fail. Only tables and views with this flag set to true (it defaults false) will have their changes annotated into the WAL. Future JIRAs (such as PHOENIX-6227, and the main CDC code which will go in phoenix-connectors) will also use this flag. 2. Indexes are no longer annotated. 3. Shifted WALAnnotationIT to use the SchemaBuilder API introduced with Phoenix TTL I believe the one remaining request left to do is @kadirozde 's suggestion to do a perf test. I will do this over the next couple of days. @ChinmaySKulkarni , @kadirozde , fyi. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Annotate HBase WALs with Phoenix Metadata > - > > Key: PHOENIX-5435 > URL: https://issues.apache.org/jira/browse/PHOENIX-5435 > Project: Phoenix > Issue Type: Sub-task >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > Attachments: PHOENIX-5435-4.x.patch > > > HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase > replication and some HBase backup frameworks. The WALs contain HBase-level > metadata such as table and region, but lack Phoenix-level metadata. That > means that it's quite difficult to build correct logic that needs to know > about Phoenix-level constructs such as multi-tenancy, views, or indexes. > HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate > extra key/value pairs of metadata into the HBase WAL. We should have the > option to annotate the tuple , or > some hashed way to reconstruct that tuple into the WAL. It should have a > feature toggle so operators who don't need it don't bear the slight extra > storage cost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[ https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17242006#comment-17242006 ] ASF GitHub Bot commented on PHOENIX-5435: - stoty commented on pull request #913: URL: https://github.com/apache/phoenix/pull/913#issuecomment-736953930 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. | ||| _ 4.x Compile Tests _ | | +0 :ok: | mvndep | 0m 32s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 10m 57s | 4.x passed | | +1 :green_heart: | compile | 1m 27s | 4.x passed | | +1 :green_heart: | checkstyle | 10m 21s | 4.x passed | | +1 :green_heart: | javadoc | 2m 23s | 4.x passed | | +0 :ok: | spotbugs | 0m 27s | phoenix-hbase-compat-1.5.0 in 4.x has 3 extant spotbugs warnings. | | +0 :ok: | spotbugs | 0m 24s | phoenix-hbase-compat-1.4.0 in 4.x has 3 extant spotbugs warnings. | | +0 :ok: | spotbugs | 0m 21s | phoenix-hbase-compat-1.3.0 in 4.x has 3 extant spotbugs warnings. | | +0 :ok: | spotbugs | 3m 58s | root in 4.x has 1004 extant spotbugs warnings. | | +0 :ok: | spotbugs | 2m 42s | phoenix-core in 4.x has 950 extant spotbugs warnings. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 6m 17s | the patch passed | | +1 :green_heart: | compile | 1m 34s | the patch passed | | +1 :green_heart: | cc | 1m 34s | the patch passed | | +1 :green_heart: | javac | 1m 34s | the patch passed | | -1 :x: | checkstyle | 10m 40s | root: The patch generated 537 new + 23086 unchanged - 193 fixed = 23623 total (was 23279) | | +1 :green_heart: | prototool | 0m 2s | There were no new prototool issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +1 :green_heart: | javadoc | 2m 21s | the patch passed | | -1 :x: | spotbugs | 3m 13s | phoenix-core generated 3 new + 950 unchanged - 0 fixed = 953 total (was 950) | | -1 :x: | spotbugs | 4m 14s | root generated 3 new + 1004 unchanged - 0 fixed = 1007 total (was 1004) | ||| _ Other Tests _ | | -1 :x: | unit | 140m 47s | root in the patch failed. | | +1 :green_heart: | asflicense | 0m 43s | The patch does not generate ASF License warnings. | | | | 208m 41s | | | Reason | Tests | |---:|:--| | FindBugs | module:phoenix-core | | | Load of known null value in org.apache.phoenix.hbase.index.IndexRegionObserver.prepareIndexMutations(IndexRegionObserver$BatchMutateContext, List, long) At IndexRegionObserver.java:in org.apache.phoenix.hbase.index.IndexRegionObserver.prepareIndexMutations(IndexRegionObserver$BatchMutateContext, List, long) At IndexRegionObserver.java:[line 800] | | | Switch statement found in org.apache.phoenix.hbase.index.scanner.FilteredKeyValueScanner.seekToNextUnfilteredKeyValue() where default case is missing At FilteredKeyValueScanner.java:where default case is missing At FilteredKeyValueScanner.java:[lines 84-101] | | | org.apache.phoenix.jdbc.PhoenixDatabaseMetaData.CHANGE_DETECTION_ENABLED_BYTES is a mutable array At MetaDataEndpointImpl.java: At MetaDataEndpointImpl.java:[line 336] | | FindBugs | module:root | | | Load of known null value in org.apache.phoenix.hbase.index.IndexRegionObserver.prepareIndexMutations(IndexRegionObserver$BatchMutateContext, List, long) At IndexRegionObserver.java:in org.apache.phoenix.hbase.index.IndexRegionObserver.prepareIndexMutations(IndexRegionObserver$BatchMutateContext, List, long) At IndexRegionObserver.java:[line 800] | | | Switch statement found in org.apache.phoenix.hbase.index.scanner.FilteredKeyValueScanner.seekToNextUnfilteredKeyValue() where default case is missing At FilteredKeyValueScanner.java:where default case is missing At FilteredKeyValueScanner.java:[lines 84-101] | | | org.apache.phoenix.jdbc.PhoenixDatabaseMetaData.CHANGE_DETECTION_ENABLED_BYTES is a mutable array At MetaDataEndpointImpl.java: At MetaDataEndpointImpl.java:[line 336] | | Subsystem | Report/Notes | |
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[ https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17242022#comment-17242022 ] ASF GitHub Bot commented on PHOENIX-5435: - gjacoby126 commented on pull request #913: URL: https://github.com/apache/phoenix/pull/913#issuecomment-736968410 The test run is odd -- looks like phoenix-core passed but then there was an error in building or shading phoenix-client? I can't reproduce locally -- mvn clean install -DskipTests passes for me. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Annotate HBase WALs with Phoenix Metadata > - > > Key: PHOENIX-5435 > URL: https://issues.apache.org/jira/browse/PHOENIX-5435 > Project: Phoenix > Issue Type: Sub-task >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > Attachments: PHOENIX-5435-4.x.patch > > > HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase > replication and some HBase backup frameworks. The WALs contain HBase-level > metadata such as table and region, but lack Phoenix-level metadata. That > means that it's quite difficult to build correct logic that needs to know > about Phoenix-level constructs such as multi-tenancy, views, or indexes. > HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate > extra key/value pairs of metadata into the HBase WAL. We should have the > option to annotate the tuple , or > some hashed way to reconstruct that tuple into the WAL. It should have a > feature toggle so operators who don't need it don't bear the slight extra > storage cost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[ https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17243713#comment-17243713 ] ASF GitHub Bot commented on PHOENIX-5435: - stoty commented on pull request #913: URL: https://github.com/apache/phoenix/pull/913#issuecomment-738563704 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. | ||| _ 4.x Compile Tests _ | | +0 :ok: | mvndep | 0m 32s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 10m 40s | 4.x passed | | +1 :green_heart: | compile | 1m 22s | 4.x passed | | +1 :green_heart: | checkstyle | 10m 33s | 4.x passed | | +1 :green_heart: | javadoc | 2m 24s | 4.x passed | | +0 :ok: | spotbugs | 0m 26s | phoenix-hbase-compat-1.5.0 in 4.x has 3 extant spotbugs warnings. | | +0 :ok: | spotbugs | 0m 24s | phoenix-hbase-compat-1.4.0 in 4.x has 3 extant spotbugs warnings. | | +0 :ok: | spotbugs | 0m 21s | phoenix-hbase-compat-1.3.0 in 4.x has 3 extant spotbugs warnings. | | +0 :ok: | spotbugs | 3m 59s | root in 4.x has 1004 extant spotbugs warnings. | | +0 :ok: | spotbugs | 2m 39s | phoenix-core in 4.x has 950 extant spotbugs warnings. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 6m 4s | the patch passed | | +1 :green_heart: | compile | 1m 28s | the patch passed | | +1 :green_heart: | cc | 1m 28s | the patch passed | | +1 :green_heart: | javac | 1m 28s | the patch passed | | -1 :x: | checkstyle | 10m 51s | root: The patch generated 496 new + 23126 unchanged - 152 fixed = 23622 total (was 23278) | | +1 :green_heart: | prototool | 0m 1s | There were no new prototool issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +1 :green_heart: | javadoc | 2m 23s | the patch passed | | -1 :x: | spotbugs | 3m 11s | phoenix-core generated 3 new + 950 unchanged - 0 fixed = 953 total (was 950) | | -1 :x: | spotbugs | 4m 13s | root generated 3 new + 1004 unchanged - 0 fixed = 1007 total (was 1004) | ||| _ Other Tests _ | | -1 :x: | unit | 294m 10s | root in the patch failed. | | +1 :green_heart: | asflicense | 2m 31s | The patch does not generate ASF License warnings. | | | | 365m 36s | | | Reason | Tests | |---:|:--| | FindBugs | module:phoenix-core | | | Load of known null value in org.apache.phoenix.hbase.index.IndexRegionObserver.prepareIndexMutations(IndexRegionObserver$BatchMutateContext, List, long) At IndexRegionObserver.java:in org.apache.phoenix.hbase.index.IndexRegionObserver.prepareIndexMutations(IndexRegionObserver$BatchMutateContext, List, long) At IndexRegionObserver.java:[line 800] | | | Switch statement found in org.apache.phoenix.hbase.index.scanner.FilteredKeyValueScanner.seekToNextUnfilteredKeyValue() where default case is missing At FilteredKeyValueScanner.java:where default case is missing At FilteredKeyValueScanner.java:[lines 84-101] | | | org.apache.phoenix.jdbc.PhoenixDatabaseMetaData.CHANGE_DETECTION_ENABLED_BYTES is a mutable array At MetaDataEndpointImpl.java: At MetaDataEndpointImpl.java:[line 336] | | FindBugs | module:root | | | Load of known null value in org.apache.phoenix.hbase.index.IndexRegionObserver.prepareIndexMutations(IndexRegionObserver$BatchMutateContext, List, long) At IndexRegionObserver.java:in org.apache.phoenix.hbase.index.IndexRegionObserver.prepareIndexMutations(IndexRegionObserver$BatchMutateContext, List, long) At IndexRegionObserver.java:[line 800] | | | Switch statement found in org.apache.phoenix.hbase.index.scanner.FilteredKeyValueScanner.seekToNextUnfilteredKeyValue() where default case is missing At FilteredKeyValueScanner.java:where default case is missing At FilteredKeyValueScanner.java:[lines 84-101] | | | org.apache.phoenix.jdbc.PhoenixDatabaseMetaData.CHANGE_DETECTION_ENABLED_BYTES is a mutable array At MetaDataEndpointImpl.java: At MetaDataEndpointImpl.java:[line 336] | | Failed junit tests | phoenix.end2end.Permissio
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[ https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17246140#comment-17246140 ] ASF GitHub Bot commented on PHOENIX-5435: - stoty commented on pull request #913: URL: https://github.com/apache/phoenix/pull/913#issuecomment-741008245 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. | ||| _ 4.x Compile Tests _ | | +0 :ok: | mvndep | 0m 33s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 11m 14s | 4.x passed | | +1 :green_heart: | compile | 1m 22s | 4.x passed | | +1 :green_heart: | checkstyle | 8m 47s | 4.x passed | | +1 :green_heart: | javadoc | 2m 20s | 4.x passed | | +0 :ok: | spotbugs | 0m 29s | phoenix-hbase-compat-1.5.0 in 4.x has 3 extant spotbugs warnings. | | +0 :ok: | spotbugs | 0m 24s | phoenix-hbase-compat-1.4.0 in 4.x has 3 extant spotbugs warnings. | | +0 :ok: | spotbugs | 0m 21s | phoenix-hbase-compat-1.3.0 in 4.x has 3 extant spotbugs warnings. | | +0 :ok: | spotbugs | 4m 4s | root in 4.x has 1004 extant spotbugs warnings. | | +0 :ok: | spotbugs | 2m 43s | phoenix-core in 4.x has 950 extant spotbugs warnings. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 59s | the patch passed | | +1 :green_heart: | compile | 1m 28s | the patch passed | | +1 :green_heart: | cc | 1m 28s | the patch passed | | +1 :green_heart: | javac | 1m 28s | the patch passed | | -1 :x: | checkstyle | 9m 1s | root: The patch generated 536 new + 23080 unchanged - 192 fixed = 23616 total (was 23272) | | +1 :green_heart: | prototool | 0m 2s | There were no new prototool issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +1 :green_heart: | javadoc | 2m 24s | the patch passed | | -1 :x: | spotbugs | 3m 14s | phoenix-core generated 3 new + 950 unchanged - 0 fixed = 953 total (was 950) | | -1 :x: | spotbugs | 4m 17s | root generated 3 new + 1004 unchanged - 0 fixed = 1007 total (was 1004) | ||| _ Other Tests _ | | -1 :x: | unit | 148m 12s | root in the patch failed. | | +1 :green_heart: | asflicense | 0m 47s | The patch does not generate ASF License warnings. | | | | 212m 54s | | | Reason | Tests | |---:|:--| | FindBugs | module:phoenix-core | | | Load of known null value in org.apache.phoenix.hbase.index.IndexRegionObserver.prepareIndexMutations(IndexRegionObserver$BatchMutateContext, List, long) At IndexRegionObserver.java:in org.apache.phoenix.hbase.index.IndexRegionObserver.prepareIndexMutations(IndexRegionObserver$BatchMutateContext, List, long) At IndexRegionObserver.java:[line 800] | | | Switch statement found in org.apache.phoenix.hbase.index.scanner.FilteredKeyValueScanner.seekToNextUnfilteredKeyValue() where default case is missing At FilteredKeyValueScanner.java:where default case is missing At FilteredKeyValueScanner.java:[lines 84-101] | | | org.apache.phoenix.jdbc.PhoenixDatabaseMetaData.CHANGE_DETECTION_ENABLED_BYTES is a mutable array At MetaDataEndpointImpl.java: At MetaDataEndpointImpl.java:[line 336] | | FindBugs | module:root | | | Load of known null value in org.apache.phoenix.hbase.index.IndexRegionObserver.prepareIndexMutations(IndexRegionObserver$BatchMutateContext, List, long) At IndexRegionObserver.java:in org.apache.phoenix.hbase.index.IndexRegionObserver.prepareIndexMutations(IndexRegionObserver$BatchMutateContext, List, long) At IndexRegionObserver.java:[line 800] | | | Switch statement found in org.apache.phoenix.hbase.index.scanner.FilteredKeyValueScanner.seekToNextUnfilteredKeyValue() where default case is missing At FilteredKeyValueScanner.java:where default case is missing At FilteredKeyValueScanner.java:[lines 84-101] | | | org.apache.phoenix.jdbc.PhoenixDatabaseMetaData.CHANGE_DETECTION_ENABLED_BYTES is a mutable array At MetaDataEndpointImpl.java: At MetaDataEndpointImpl.java:[line 336] | | Subsystem | Report/Notes | |
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[
https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17246657#comment-17246657
]
ASF GitHub Bot commented on PHOENIX-5435:
-
virajjasani commented on a change in pull request #913:
URL: https://github.com/apache/phoenix/pull/913#discussion_r539235012
##
File path:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/UngroupedAggregateRegionScanner.java
##
@@ -644,4 +651,33 @@ public boolean next(List resultsToReturn) throws
IOException {
public long getMaxResultSize() {
return scan.getMaxResultSize();
}
+
+private void
annotateDataMutations(UngroupedAggregateRegionObserver.MutationList
mutationsList,
+ Scan scan) {
+byte[] tenantId = null;
Review comment:
nit: we are assigning these variables representing MutationMetadataType
twice (`null` and `scan.getAttribute()`), should we assign all of them just
once by removing null assignments?
##
File path:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/UngroupedAggregateRegionScanner.java
##
@@ -64,6 +64,7 @@
import org.apache.phoenix.cache.GlobalCache;
import org.apache.phoenix.cache.TenantCache;
import org.apache.phoenix.exception.DataExceedsCapacityException;
+import org.apache.phoenix.execute.MutationState;
Review comment:
nit: redundant?
##
File path:
phoenix-core/src/main/java/org/apache/phoenix/schema/MetaDataClient.java
##
@@ -2102,6 +2106,17 @@ private PTable createTableInternal(CreateTableStatement
statement, byte[][] spli
}
}
+Boolean isChangeDetectionEnabledProp =
+(Boolean)
TableProperty.CHANGE_DETECTION_ENABLED.getValue(tableProps);
+if (isChangeDetectionEnabledProp != null) {
+if (tableType != TABLE && tableType != VIEW) {
+throw new SQLExceptionInfo.Builder(
+
SQLExceptionCode.CHANGE_DETECTION_SUPPORTED_FOR_TABLES_AND_VIEWS_ONLY)
Review comment:
If possible, we can refactor this validation in a small util method and
it can be used by both workflows: createTable/createIndex and
addColumn/alterIndex?
##
File path:
phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java
##
@@ -682,6 +687,49 @@ private void generateMutations(final TableRef tableRef,
final long mutationTimes
values.putAll(modifiedValues);
}
+public static void annotateMutationsWithMetadata(PTable table,
List rowMutations) {
+//only annotate if the change detection flag is on the table and HBase
supports
+// preWALAppend coprocs server-side
+if (table == null || !table.isChangeDetectionEnabled()
+|| !HbaseCompatCapabilities.hasPreWALAppend()) {
+return;
+}
+//annotate each mutation with enough metadata so that anyone
interested can
+// deterministically figure out exactly what Phoenix schema object
created the mutation
+// Server-side we can annotate the HBase WAL with these.
+for (Mutation mutation : rowMutations) {
+annotateMutationWithMetadata(table, mutation);
+}
+
+}
+
+public static void annotateMutationWithMetadata(PTable table, Mutation
mutation) {
Review comment:
nit: we can keep both util methods `private`?
##
File path:
phoenix-core/src/main/java/org/apache/phoenix/util/WALAnnotationUtil.java
##
@@ -0,0 +1,49 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.util;
+
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.wal.WALKey;
+import org.apache.phoenix.compat.hbase.coprocessor.CompatIndexRegionObserver;
+import org.apache.phoenix.execute.MutationState;
+import org.apache.phoenix.hbase.index.IndexRegionObserver;
+
+import java.util.Map;
+
+/**
+ * Utility functions shared between IndexRegionObserver and GlobalIndexChecker
for annotating the
+ *
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[
https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17246659#comment-17246659
]
ASF GitHub Bot commented on PHOENIX-5435:
-
virajjasani commented on a change in pull request #913:
URL: https://github.com/apache/phoenix/pull/913#discussion_r539466637
##
File path:
phoenix-core/src/main/java/org/apache/phoenix/util/WALAnnotationUtil.java
##
@@ -0,0 +1,49 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.util;
+
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.wal.WALKey;
+import org.apache.phoenix.compat.hbase.coprocessor.CompatIndexRegionObserver;
+import org.apache.phoenix.execute.MutationState;
+import org.apache.phoenix.hbase.index.IndexRegionObserver;
+
+import java.util.Map;
+
+/**
+ * Utility functions shared between IndexRegionObserver and GlobalIndexChecker
for annotating the
+ * HBase WAL with Phoenix-level metadata about mutations.
+ */
+public class WALAnnotationUtil {
+
+public static void appendMutationAttributesToWALKey(WALKey key,
+IndexRegionObserver.BatchMutateContext
context) {
+if (context != null && context.getOriginalMutations().size() > 0) {
+Mutation firstMutation = context.getOriginalMutations().get(0);
+Map attrMap = firstMutation.getAttributesMap();
+for (MutationState.MutationMetadataType metadataType :
+MutationState.MutationMetadataType.values()) {
+String metadataTypeKey = metadataType.toString();
+if (attrMap.containsKey(metadataTypeKey)) {
+CompatIndexRegionObserver.appendToWALKey(key,
metadataTypeKey,
Review comment:
Moreover, 1.3/1.4 compact classes don't have it as `static`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Annotate HBase WALs with Phoenix Metadata
> -
>
> Key: PHOENIX-5435
> URL: https://issues.apache.org/jira/browse/PHOENIX-5435
> Project: Phoenix
> Issue Type: Sub-task
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>Priority: Major
> Attachments: PHOENIX-5435-4.x.patch
>
>
> HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase
> replication and some HBase backup frameworks. The WALs contain HBase-level
> metadata such as table and region, but lack Phoenix-level metadata. That
> means that it's quite difficult to build correct logic that needs to know
> about Phoenix-level constructs such as multi-tenancy, views, or indexes.
> HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate
> extra key/value pairs of metadata into the HBase WAL. We should have the
> option to annotate the tuple , or
> some hashed way to reconstruct that tuple into the WAL. It should have a
> feature toggle so operators who don't need it don't bear the slight extra
> storage cost.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[ https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17246813#comment-17246813 ] ASF GitHub Bot commented on PHOENIX-5435: - gjacoby126 commented on a change in pull request #913: URL: https://github.com/apache/phoenix/pull/913#discussion_r539637405 ## File path: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java ## @@ -1160,6 +1169,12 @@ private PTable getTable(RegionScanner scanner, long clientTimeStamp, long tableT null : PLong.INSTANCE.getCodec().decodeLong(lastDDLTimestampKv.getValueArray(), lastDDLTimestampKv.getValueOffset(), SortOrder.getDefault()); +Cell changeDetectionEnabledKv = tableKeyValues[CHANGE_DETECTION_ENABLED_INDEX]; +boolean isChangeDetectionEnabled = changeDetectionEnabledKv != null Review comment: So long as some other client hasn't set CHANGE_DETECTION_ENABLED=true on the table via an ALTER statement. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Annotate HBase WALs with Phoenix Metadata > - > > Key: PHOENIX-5435 > URL: https://issues.apache.org/jira/browse/PHOENIX-5435 > Project: Phoenix > Issue Type: Sub-task >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > Attachments: PHOENIX-5435-4.x.patch > > > HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase > replication and some HBase backup frameworks. The WALs contain HBase-level > metadata such as table and region, but lack Phoenix-level metadata. That > means that it's quite difficult to build correct logic that needs to know > about Phoenix-level constructs such as multi-tenancy, views, or indexes. > HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate > extra key/value pairs of metadata into the HBase WAL. We should have the > option to annotate the tuple , or > some hashed way to reconstruct that tuple into the WAL. It should have a > feature toggle so operators who don't need it don't bear the slight extra > storage cost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[
https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17246818#comment-17246818
]
ASF GitHub Bot commented on PHOENIX-5435:
-
gjacoby126 commented on a change in pull request #913:
URL: https://github.com/apache/phoenix/pull/913#discussion_r539639738
##
File path:
phoenix-hbase-compat-1.3.0/src/main/java/org/apache/phoenix/compat/hbase/coprocessor/CompatIndexRegionObserver.java
##
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.phoenix.compat.hbase.coprocessor;
+
+import org.apache.hadoop.hbase.coprocessor.BaseRegionObserver;
+import org.apache.hadoop.hbase.coprocessor.ObserverContext;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.wal.WALEdit;
+import org.apache.hadoop.hbase.wal.WALKey;
+
+import java.util.HashMap;
+import java.util.Map;
+
+public class CompatIndexRegionObserver extends BaseRegionObserver {
+public static String PHOENIX_APPEND_METADATA_TO_WAL =
"phoenix.append.metadata.to.wal";
+public static boolean DEFAULT_PHOENIX_APPEND_METADATA_TO_WAL = false;
Review comment:
These aren't used, will remove.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Annotate HBase WALs with Phoenix Metadata
> -
>
> Key: PHOENIX-5435
> URL: https://issues.apache.org/jira/browse/PHOENIX-5435
> Project: Phoenix
> Issue Type: Sub-task
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>Priority: Major
> Attachments: PHOENIX-5435-4.x.patch
>
>
> HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase
> replication and some HBase backup frameworks. The WALs contain HBase-level
> metadata such as table and region, but lack Phoenix-level metadata. That
> means that it's quite difficult to build correct logic that needs to know
> about Phoenix-level constructs such as multi-tenancy, views, or indexes.
> HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate
> extra key/value pairs of metadata into the HBase WAL. We should have the
> option to annotate the tuple , or
> some hashed way to reconstruct that tuple into the WAL. It should have a
> feature toggle so operators who don't need it don't bear the slight extra
> storage cost.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[
https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17246819#comment-17246819
]
ASF GitHub Bot commented on PHOENIX-5435:
-
gjacoby126 commented on a change in pull request #913:
URL: https://github.com/apache/phoenix/pull/913#discussion_r539642721
##
File path:
phoenix-core/src/main/java/org/apache/phoenix/util/WALAnnotationUtil.java
##
@@ -0,0 +1,49 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.util;
+
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.wal.WALKey;
+import org.apache.phoenix.compat.hbase.coprocessor.CompatIndexRegionObserver;
+import org.apache.phoenix.execute.MutationState;
+import org.apache.phoenix.hbase.index.IndexRegionObserver;
+
+import java.util.Map;
+
+/**
+ * Utility functions shared between IndexRegionObserver and GlobalIndexChecker
for annotating the
+ * HBase WAL with Phoenix-level metadata about mutations.
+ */
+public class WALAnnotationUtil {
+
+public static void appendMutationAttributesToWALKey(WALKey key,
+IndexRegionObserver.BatchMutateContext
context) {
+if (context != null && context.getOriginalMutations().size() > 0) {
+Mutation firstMutation = context.getOriginalMutations().get(0);
+Map attrMap = firstMutation.getAttributesMap();
+for (MutationState.MutationMetadataType metadataType :
+MutationState.MutationMetadataType.values()) {
+String metadataTypeKey = metadataType.toString();
+if (attrMap.containsKey(metadataTypeKey)) {
+CompatIndexRegionObserver.appendToWALKey(key,
metadataTypeKey,
Review comment:
Oops, thanks for the catch.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Annotate HBase WALs with Phoenix Metadata
> -
>
> Key: PHOENIX-5435
> URL: https://issues.apache.org/jira/browse/PHOENIX-5435
> Project: Phoenix
> Issue Type: Sub-task
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>Priority: Major
> Attachments: PHOENIX-5435-4.x.patch
>
>
> HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase
> replication and some HBase backup frameworks. The WALs contain HBase-level
> metadata such as table and region, but lack Phoenix-level metadata. That
> means that it's quite difficult to build correct logic that needs to know
> about Phoenix-level constructs such as multi-tenancy, views, or indexes.
> HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate
> extra key/value pairs of metadata into the HBase WAL. We should have the
> option to annotate the tuple , or
> some hashed way to reconstruct that tuple into the WAL. It should have a
> feature toggle so operators who don't need it don't bear the slight extra
> storage cost.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[
https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17246849#comment-17246849
]
ASF GitHub Bot commented on PHOENIX-5435:
-
gokceni commented on a change in pull request #913:
URL: https://github.com/apache/phoenix/pull/913#discussion_r539564620
##
File path:
phoenix-core/src/it/java/org/apache/phoenix/end2end/index/ViewIndexIT.java
##
@@ -72,6 +72,7 @@
import org.apache.phoenix.util.PropertiesUtil;
import org.apache.phoenix.util.QueryUtil;
import org.apache.phoenix.util.SchemaUtil;
+import org.apache.phoenix.util.TestDDLUtil;
Review comment:
ditto
##
File path:
phoenix-core/src/it/java/org/apache/phoenix/end2end/ImmutableTableIT.java
##
@@ -17,6 +17,7 @@
*/
package org.apache.phoenix.end2end;
+import org.apache.phoenix.util.TestDDLUtil;
Review comment:
nit: is the import necessary? Seems like the only change on this file is
this import
##
File path:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/GroupedAggregateRegionObserver.java
##
@@ -400,7 +400,6 @@ protected boolean isRegionObserverFor(Scan scan) {
* Used for an aggregate query in which the key order does not necessarily
match the group by
* key order. In this case, we must collect all distinct groups within a
region into a map,
* aggregating as we go.
- * @param limit TODO
Review comment:
TODO seems to be dropped but not implemented?
##
File path:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/UngroupedAggregateRegionObserver.java
##
@@ -73,6 +73,7 @@
import org.apache.hadoop.io.WritableUtils;
import org.apache.phoenix.coprocessor.generated.PTableProtos;
import org.apache.phoenix.exception.SQLExceptionCode;
+import org.apache.phoenix.execute.MutationState;
Review comment:
nit: lonely import
##
File path:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/UngroupedAggregateRegionScanner.java
##
@@ -609,6 +613,9 @@ public boolean next(List resultsToReturn) throws
IOException {
} while (hasMore &&
(EnvironmentEdgeManager.currentTimeMillis() - startTime) < pageSizeInMs);
if (!mutations.isEmpty()) {
+if (isDelete || isUpsert) {
+annotateDataMutations(mutations, scan);
+}
Review comment:
It would be nice to refactor this perhaps to a new method? Lines 596 to
602 is same as 616 to 622
##
File path:
phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java
##
@@ -682,6 +687,49 @@ private void generateMutations(final TableRef tableRef,
final long mutationTimes
values.putAll(modifiedValues);
}
+private void annotateMutationsWithMetadata(PTable table, List
rowMutations) {
+//only annotate if the change detection flag is on the table and HBase
supports
+// preWALAppend coprocs server-side
+if (table == null || !table.isChangeDetectionEnabled()
+|| !HbaseCompatCapabilities.hasPreWALAppend()) {
+return;
+}
+//annotate each mutation with enough metadata so that anyone
interested can
+// deterministically figure out exactly what Phoenix schema object
created the mutation
+// Server-side we can annotate the HBase WAL with these.
+for (Mutation mutation : rowMutations) {
+annotateMutationWithMetadata(table, mutation);
+}
+
+}
+
+public static void annotateMutationWithMetadata(PTable table, Mutation
mutation) {
Review comment:
Actually why not use the this method in the Ungroupped..Scanner in that
for loop that goes over mutations? Also mutation durability was not checked
there I think
##
File path:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/generated/ServerCachingProtos.java
##
@@ -2192,6 +2192,21 @@ public Builder setColumnNameBytes(
*/
com.google.protobuf.ByteString
getParentTableTypeBytes();
+
+// optional string logicalIndexName = 25;
+/**
+ * optional string logicalIndexName = 25;
+ */
+boolean hasLogicalIndexName();
+/**
+ * optional string logicalIndexName = 25;
+ */
+java.lang.String getLogicalIndexName();
Review comment:
Would this be populated for both view indexes and regular indexes and
local indexes?
##
File path:
phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java
##
@@ -674,6 +678,7 @@ private void generateMutations(final TableRef tableRef,
final long mutationTimes
mutation.setAttribute(PhoenixIndexBuilder.ATOMIC_OP_ATTRIB, onDupKeyBytes);
}
}
+annotateMutationsWithMetadata(table, rowMutations);
Review c
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[
https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17246896#comment-17246896
]
ASF GitHub Bot commented on PHOENIX-5435:
-
ChinmaySKulkarni commented on a change in pull request #913:
URL: https://github.com/apache/phoenix/pull/913#discussion_r539644075
##
File path:
phoenix-core/src/it/java/org/apache/phoenix/end2end/ImmutableTableIT.java
##
@@ -17,6 +17,7 @@
*/
package org.apache.phoenix.end2end;
+import org.apache.phoenix.util.TestDDLUtil;
Review comment:
nit: unused import
##
File path:
phoenix-core/src/it/java/org/apache/phoenix/end2end/WALAnnotationIT.java
##
@@ -0,0 +1,550 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.end2end;
+
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionInfo;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.coprocessor.BaseWALObserver;
+import org.apache.hadoop.hbase.coprocessor.ObserverContext;
+import org.apache.hadoop.hbase.coprocessor.WALCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.wal.WALCoprocessorHost;
+import org.apache.hadoop.hbase.regionserver.wal.WALEdit;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.wal.WAL;
+import org.apache.hadoop.hbase.wal.WALKey;
+import org.apache.phoenix.compat.hbase.HbaseCompatCapabilities;
+import org.apache.phoenix.compat.hbase.coprocessor.CompatIndexRegionObserver;
+import org.apache.phoenix.exception.SQLExceptionCode;
+import org.apache.phoenix.execute.MutationState;
+import org.apache.phoenix.hbase.index.IndexRegionObserver;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.PhoenixTestBuilder;
+import org.apache.phoenix.query.PhoenixTestBuilder.SchemaBuilder;
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.PTableType;
+import org.apache.phoenix.util.MetaDataUtil;
+import org.apache.phoenix.util.PhoenixRuntime;
+import org.apache.phoenix.util.ReadOnlyProps;
+import org.apache.phoenix.util.SchemaUtil;
+import org.apache.phoenix.util.TestUtil;
+import org.junit.Assert;
+import org.junit.Assume;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+
+import static
org.apache.phoenix.jdbc.PhoenixDatabaseMetaData.CHANGE_DETECTION_ENABLED;
+import static org.junit.Assert.assertNotNull;
+
+@RunWith(Parameterized.class)
+@Category(NeedsOwnMiniClusterTest.class)
+public class WALAnnotationIT extends BaseUniqueNamesOwnClusterIT {
+private final boolean isImmutable;
+private final boolean isMultiTenant;
+
+// name is used by failsafe as file name in reports
+@Parameterized.Parameters(name =
"WALAnnotationIT_isImmutable={0}_isMultiTenant={1}")
+public static synchronized Collection data() {
+return Arrays.asList(new Object[]{true, true}, new Object[]{true,
false},
+new Object[]{false, true}, new Object[]{false, false});
+}
+
+public WALAnnotationIT(boolean isImmutable, boolean isMultiTenant) {
+this.isImmutable = isImmutable;
+this.isMultiTenant = isMultiTenant;
+}
+
+@BeforeClass
+public static synchronized void doSetup() throws Exception {
+Map props = new HashMap<>(2);
+props.put("hbase.coprocessor.wal.classes",
+AnnotatedWALObserver.class.getName());
+props.put(IndexRegionObserver.PHOENIX_APPEND_METADATA_TO_WAL, "true");
+props.put(QueryServices.ENABLE_SERVER_UPSERT_SELECT, "true");
+setUpTestDriver(new ReadOnlyProps(props.entrySet().iterator()));
+}
+
+
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[
https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17246898#comment-17246898
]
ASF GitHub Bot commented on PHOENIX-5435:
-
ChinmaySKulkarni commented on a change in pull request #913:
URL: https://github.com/apache/phoenix/pull/913#discussion_r539689792
##
File path:
phoenix-core/src/it/java/org/apache/phoenix/end2end/WALAnnotationIT.java
##
@@ -0,0 +1,550 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.end2end;
+
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionInfo;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.coprocessor.BaseWALObserver;
+import org.apache.hadoop.hbase.coprocessor.ObserverContext;
+import org.apache.hadoop.hbase.coprocessor.WALCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.wal.WALCoprocessorHost;
+import org.apache.hadoop.hbase.regionserver.wal.WALEdit;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.wal.WAL;
+import org.apache.hadoop.hbase.wal.WALKey;
+import org.apache.phoenix.compat.hbase.HbaseCompatCapabilities;
+import org.apache.phoenix.compat.hbase.coprocessor.CompatIndexRegionObserver;
+import org.apache.phoenix.exception.SQLExceptionCode;
+import org.apache.phoenix.execute.MutationState;
+import org.apache.phoenix.hbase.index.IndexRegionObserver;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.PhoenixTestBuilder;
+import org.apache.phoenix.query.PhoenixTestBuilder.SchemaBuilder;
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.PTableType;
+import org.apache.phoenix.util.MetaDataUtil;
+import org.apache.phoenix.util.PhoenixRuntime;
+import org.apache.phoenix.util.ReadOnlyProps;
+import org.apache.phoenix.util.SchemaUtil;
+import org.apache.phoenix.util.TestUtil;
+import org.junit.Assert;
+import org.junit.Assume;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+
+import static
org.apache.phoenix.jdbc.PhoenixDatabaseMetaData.CHANGE_DETECTION_ENABLED;
+import static org.junit.Assert.assertNotNull;
+
+@RunWith(Parameterized.class)
+@Category(NeedsOwnMiniClusterTest.class)
+public class WALAnnotationIT extends BaseUniqueNamesOwnClusterIT {
+private final boolean isImmutable;
+private final boolean isMultiTenant;
+
+// name is used by failsafe as file name in reports
+@Parameterized.Parameters(name =
"WALAnnotationIT_isImmutable={0}_isMultiTenant={1}")
+public static synchronized Collection data() {
+return Arrays.asList(new Object[]{true, true}, new Object[]{true,
false},
+new Object[]{false, true}, new Object[]{false, false});
+}
+
+public WALAnnotationIT(boolean isImmutable, boolean isMultiTenant) {
+this.isImmutable = isImmutable;
+this.isMultiTenant = isMultiTenant;
+}
+
+@BeforeClass
+public static synchronized void doSetup() throws Exception {
+Map props = new HashMap<>(2);
+props.put("hbase.coprocessor.wal.classes",
+AnnotatedWALObserver.class.getName());
+props.put(IndexRegionObserver.PHOENIX_APPEND_METADATA_TO_WAL, "true");
+props.put(QueryServices.ENABLE_SERVER_UPSERT_SELECT, "true");
+setUpTestDriver(new ReadOnlyProps(props.entrySet().iterator()));
+}
+
+@Test
+public void testSimpleUpsertAndDelete() throws Exception {
+Assume.assumeTrue(HbaseCompatCapabilities.hasPreWALAppend());
+SchemaBuilder builder = new SchemaBuilder(getUrl());
+boolean createGlobalIndex = false;
+lo
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[
https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17246900#comment-17246900
]
ASF GitHub Bot commented on PHOENIX-5435:
-
ChinmaySKulkarni commented on a change in pull request #913:
URL: https://github.com/apache/phoenix/pull/913#discussion_r539718201
##
File path: phoenix-core/src/main/java/org/apache/phoenix/util/ScanUtil.java
##
@@ -1239,4 +1239,21 @@ public static boolean isDummy(List result) {
Cell cell = result.get(0);
return CellUtil.matchingColumn(cell, EMPTY_BYTE_ARRAY,
EMPTY_BYTE_ARRAY);
}
+
+public static void setWALAnnotationAttributes(PTable table, Scan scan) {
Review comment:
@gjacoby126 Here is my understanding overall, please correct me if I'm
wrong. There are 2 basic paths:
1. (Client-side delete and upsert-values) Client issues Puts/Deletes
directly: This is handled in `MutationState. annotateMutationsWithMetadata()`
2. (Server-side delete and upsert-selects) Client issues scans, the server
intercepts this and issues Put/Delete mutations based on the results read: This
is what this method handles right? My understanding is we set the scan
attributes in UpsertCompiler and DeleteCompiler and this is intercepted in
`UngroupedAggregateRegionScanner` where we read the scan attribute and set it
on the mutations.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Annotate HBase WALs with Phoenix Metadata
> -
>
> Key: PHOENIX-5435
> URL: https://issues.apache.org/jira/browse/PHOENIX-5435
> Project: Phoenix
> Issue Type: Sub-task
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>Priority: Major
> Attachments: PHOENIX-5435-4.x.patch
>
>
> HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase
> replication and some HBase backup frameworks. The WALs contain HBase-level
> metadata such as table and region, but lack Phoenix-level metadata. That
> means that it's quite difficult to build correct logic that needs to know
> about Phoenix-level constructs such as multi-tenancy, views, or indexes.
> HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate
> extra key/value pairs of metadata into the HBase WAL. We should have the
> option to annotate the tuple , or
> some hashed way to reconstruct that tuple into the WAL. It should have a
> feature toggle so operators who don't need it don't bear the slight extra
> storage cost.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[
https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17246904#comment-17246904
]
ASF GitHub Bot commented on PHOENIX-5435:
-
gjacoby126 commented on a change in pull request #913:
URL: https://github.com/apache/phoenix/pull/913#discussion_r539739219
##
File path:
phoenix-core/src/it/java/org/apache/phoenix/end2end/WALAnnotationIT.java
##
@@ -0,0 +1,550 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.end2end;
+
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionInfo;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.coprocessor.BaseWALObserver;
+import org.apache.hadoop.hbase.coprocessor.ObserverContext;
+import org.apache.hadoop.hbase.coprocessor.WALCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.wal.WALCoprocessorHost;
+import org.apache.hadoop.hbase.regionserver.wal.WALEdit;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.wal.WAL;
+import org.apache.hadoop.hbase.wal.WALKey;
+import org.apache.phoenix.compat.hbase.HbaseCompatCapabilities;
+import org.apache.phoenix.compat.hbase.coprocessor.CompatIndexRegionObserver;
+import org.apache.phoenix.exception.SQLExceptionCode;
+import org.apache.phoenix.execute.MutationState;
+import org.apache.phoenix.hbase.index.IndexRegionObserver;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.PhoenixTestBuilder;
+import org.apache.phoenix.query.PhoenixTestBuilder.SchemaBuilder;
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.PTableType;
+import org.apache.phoenix.util.MetaDataUtil;
+import org.apache.phoenix.util.PhoenixRuntime;
+import org.apache.phoenix.util.ReadOnlyProps;
+import org.apache.phoenix.util.SchemaUtil;
+import org.apache.phoenix.util.TestUtil;
+import org.junit.Assert;
+import org.junit.Assume;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+
+import static
org.apache.phoenix.jdbc.PhoenixDatabaseMetaData.CHANGE_DETECTION_ENABLED;
+import static org.junit.Assert.assertNotNull;
+
+@RunWith(Parameterized.class)
+@Category(NeedsOwnMiniClusterTest.class)
+public class WALAnnotationIT extends BaseUniqueNamesOwnClusterIT {
+private final boolean isImmutable;
+private final boolean isMultiTenant;
+
+// name is used by failsafe as file name in reports
+@Parameterized.Parameters(name =
"WALAnnotationIT_isImmutable={0}_isMultiTenant={1}")
+public static synchronized Collection data() {
+return Arrays.asList(new Object[]{true, true}, new Object[]{true,
false},
+new Object[]{false, true}, new Object[]{false, false});
+}
+
+public WALAnnotationIT(boolean isImmutable, boolean isMultiTenant) {
+this.isImmutable = isImmutable;
+this.isMultiTenant = isMultiTenant;
+}
+
+@BeforeClass
+public static synchronized void doSetup() throws Exception {
+Map props = new HashMap<>(2);
+props.put("hbase.coprocessor.wal.classes",
+AnnotatedWALObserver.class.getName());
+props.put(IndexRegionObserver.PHOENIX_APPEND_METADATA_TO_WAL, "true");
+props.put(QueryServices.ENABLE_SERVER_UPSERT_SELECT, "true");
+setUpTestDriver(new ReadOnlyProps(props.entrySet().iterator()));
+}
+
+@Test
+public void testSimpleUpsertAndDelete() throws Exception {
+Assume.assumeTrue(HbaseCompatCapabilities.hasPreWALAppend());
+SchemaBuilder builder = new SchemaBuilder(getUrl());
+boolean createGlobalIndex = false;
+long ddl
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[
https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17246906#comment-17246906
]
ASF GitHub Bot commented on PHOENIX-5435:
-
gjacoby126 commented on a change in pull request #913:
URL: https://github.com/apache/phoenix/pull/913#discussion_r539739591
##
File path:
phoenix-core/src/it/java/org/apache/phoenix/end2end/WALAnnotationIT.java
##
@@ -0,0 +1,550 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.end2end;
+
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionInfo;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.coprocessor.BaseWALObserver;
+import org.apache.hadoop.hbase.coprocessor.ObserverContext;
+import org.apache.hadoop.hbase.coprocessor.WALCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.wal.WALCoprocessorHost;
+import org.apache.hadoop.hbase.regionserver.wal.WALEdit;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.wal.WAL;
+import org.apache.hadoop.hbase.wal.WALKey;
+import org.apache.phoenix.compat.hbase.HbaseCompatCapabilities;
+import org.apache.phoenix.compat.hbase.coprocessor.CompatIndexRegionObserver;
+import org.apache.phoenix.exception.SQLExceptionCode;
+import org.apache.phoenix.execute.MutationState;
+import org.apache.phoenix.hbase.index.IndexRegionObserver;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.PhoenixTestBuilder;
+import org.apache.phoenix.query.PhoenixTestBuilder.SchemaBuilder;
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.PTableType;
+import org.apache.phoenix.util.MetaDataUtil;
+import org.apache.phoenix.util.PhoenixRuntime;
+import org.apache.phoenix.util.ReadOnlyProps;
+import org.apache.phoenix.util.SchemaUtil;
+import org.apache.phoenix.util.TestUtil;
+import org.junit.Assert;
+import org.junit.Assume;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+
+import static
org.apache.phoenix.jdbc.PhoenixDatabaseMetaData.CHANGE_DETECTION_ENABLED;
+import static org.junit.Assert.assertNotNull;
+
+@RunWith(Parameterized.class)
+@Category(NeedsOwnMiniClusterTest.class)
+public class WALAnnotationIT extends BaseUniqueNamesOwnClusterIT {
+private final boolean isImmutable;
+private final boolean isMultiTenant;
+
+// name is used by failsafe as file name in reports
+@Parameterized.Parameters(name =
"WALAnnotationIT_isImmutable={0}_isMultiTenant={1}")
+public static synchronized Collection data() {
+return Arrays.asList(new Object[]{true, true}, new Object[]{true,
false},
+new Object[]{false, true}, new Object[]{false, false});
+}
+
+public WALAnnotationIT(boolean isImmutable, boolean isMultiTenant) {
+this.isImmutable = isImmutable;
+this.isMultiTenant = isMultiTenant;
+}
+
+@BeforeClass
+public static synchronized void doSetup() throws Exception {
+Map props = new HashMap<>(2);
+props.put("hbase.coprocessor.wal.classes",
+AnnotatedWALObserver.class.getName());
+props.put(IndexRegionObserver.PHOENIX_APPEND_METADATA_TO_WAL, "true");
+props.put(QueryServices.ENABLE_SERVER_UPSERT_SELECT, "true");
+setUpTestDriver(new ReadOnlyProps(props.entrySet().iterator()));
+}
+
+@Test
+public void testSimpleUpsertAndDelete() throws Exception {
+Assume.assumeTrue(HbaseCompatCapabilities.hasPreWALAppend());
+SchemaBuilder builder = new SchemaBuilder(getUrl());
+boolean createGlobalIndex = false;
+long ddl
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[
https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17246907#comment-17246907
]
ASF GitHub Bot commented on PHOENIX-5435:
-
gjacoby126 commented on a change in pull request #913:
URL: https://github.com/apache/phoenix/pull/913#discussion_r539739771
##
File path:
phoenix-core/src/it/java/org/apache/phoenix/end2end/WALAnnotationIT.java
##
@@ -0,0 +1,550 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.end2end;
+
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionInfo;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.coprocessor.BaseWALObserver;
+import org.apache.hadoop.hbase.coprocessor.ObserverContext;
+import org.apache.hadoop.hbase.coprocessor.WALCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.wal.WALCoprocessorHost;
+import org.apache.hadoop.hbase.regionserver.wal.WALEdit;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.wal.WAL;
+import org.apache.hadoop.hbase.wal.WALKey;
+import org.apache.phoenix.compat.hbase.HbaseCompatCapabilities;
+import org.apache.phoenix.compat.hbase.coprocessor.CompatIndexRegionObserver;
+import org.apache.phoenix.exception.SQLExceptionCode;
+import org.apache.phoenix.execute.MutationState;
+import org.apache.phoenix.hbase.index.IndexRegionObserver;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.PhoenixTestBuilder;
+import org.apache.phoenix.query.PhoenixTestBuilder.SchemaBuilder;
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.PTableType;
+import org.apache.phoenix.util.MetaDataUtil;
+import org.apache.phoenix.util.PhoenixRuntime;
+import org.apache.phoenix.util.ReadOnlyProps;
+import org.apache.phoenix.util.SchemaUtil;
+import org.apache.phoenix.util.TestUtil;
+import org.junit.Assert;
+import org.junit.Assume;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+
+import static
org.apache.phoenix.jdbc.PhoenixDatabaseMetaData.CHANGE_DETECTION_ENABLED;
+import static org.junit.Assert.assertNotNull;
+
+@RunWith(Parameterized.class)
+@Category(NeedsOwnMiniClusterTest.class)
+public class WALAnnotationIT extends BaseUniqueNamesOwnClusterIT {
+private final boolean isImmutable;
+private final boolean isMultiTenant;
+
+// name is used by failsafe as file name in reports
+@Parameterized.Parameters(name =
"WALAnnotationIT_isImmutable={0}_isMultiTenant={1}")
+public static synchronized Collection data() {
+return Arrays.asList(new Object[]{true, true}, new Object[]{true,
false},
+new Object[]{false, true}, new Object[]{false, false});
+}
+
+public WALAnnotationIT(boolean isImmutable, boolean isMultiTenant) {
+this.isImmutable = isImmutable;
+this.isMultiTenant = isMultiTenant;
+}
+
+@BeforeClass
+public static synchronized void doSetup() throws Exception {
+Map props = new HashMap<>(2);
+props.put("hbase.coprocessor.wal.classes",
+AnnotatedWALObserver.class.getName());
+props.put(IndexRegionObserver.PHOENIX_APPEND_METADATA_TO_WAL, "true");
+props.put(QueryServices.ENABLE_SERVER_UPSERT_SELECT, "true");
+setUpTestDriver(new ReadOnlyProps(props.entrySet().iterator()));
+}
+
+@Test
+public void testSimpleUpsertAndDelete() throws Exception {
+Assume.assumeTrue(HbaseCompatCapabilities.hasPreWALAppend());
+SchemaBuilder builder = new SchemaBuilder(getUrl());
+boolean createGlobalIndex = false;
+long ddl
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[
https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17246908#comment-17246908
]
ASF GitHub Bot commented on PHOENIX-5435:
-
gjacoby126 commented on a change in pull request #913:
URL: https://github.com/apache/phoenix/pull/913#discussion_r539740063
##
File path:
phoenix-core/src/it/java/org/apache/phoenix/end2end/WALAnnotationIT.java
##
@@ -0,0 +1,550 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.end2end;
+
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionInfo;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.coprocessor.BaseWALObserver;
+import org.apache.hadoop.hbase.coprocessor.ObserverContext;
+import org.apache.hadoop.hbase.coprocessor.WALCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.wal.WALCoprocessorHost;
+import org.apache.hadoop.hbase.regionserver.wal.WALEdit;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.wal.WAL;
+import org.apache.hadoop.hbase.wal.WALKey;
+import org.apache.phoenix.compat.hbase.HbaseCompatCapabilities;
+import org.apache.phoenix.compat.hbase.coprocessor.CompatIndexRegionObserver;
+import org.apache.phoenix.exception.SQLExceptionCode;
+import org.apache.phoenix.execute.MutationState;
+import org.apache.phoenix.hbase.index.IndexRegionObserver;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.PhoenixTestBuilder;
+import org.apache.phoenix.query.PhoenixTestBuilder.SchemaBuilder;
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.PTableType;
+import org.apache.phoenix.util.MetaDataUtil;
+import org.apache.phoenix.util.PhoenixRuntime;
+import org.apache.phoenix.util.ReadOnlyProps;
+import org.apache.phoenix.util.SchemaUtil;
+import org.apache.phoenix.util.TestUtil;
+import org.junit.Assert;
+import org.junit.Assume;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+
+import static
org.apache.phoenix.jdbc.PhoenixDatabaseMetaData.CHANGE_DETECTION_ENABLED;
+import static org.junit.Assert.assertNotNull;
+
+@RunWith(Parameterized.class)
+@Category(NeedsOwnMiniClusterTest.class)
+public class WALAnnotationIT extends BaseUniqueNamesOwnClusterIT {
+private final boolean isImmutable;
+private final boolean isMultiTenant;
+
+// name is used by failsafe as file name in reports
+@Parameterized.Parameters(name =
"WALAnnotationIT_isImmutable={0}_isMultiTenant={1}")
+public static synchronized Collection data() {
+return Arrays.asList(new Object[]{true, true}, new Object[]{true,
false},
+new Object[]{false, true}, new Object[]{false, false});
+}
+
+public WALAnnotationIT(boolean isImmutable, boolean isMultiTenant) {
+this.isImmutable = isImmutable;
+this.isMultiTenant = isMultiTenant;
+}
+
+@BeforeClass
+public static synchronized void doSetup() throws Exception {
+Map props = new HashMap<>(2);
+props.put("hbase.coprocessor.wal.classes",
+AnnotatedWALObserver.class.getName());
+props.put(IndexRegionObserver.PHOENIX_APPEND_METADATA_TO_WAL, "true");
+props.put(QueryServices.ENABLE_SERVER_UPSERT_SELECT, "true");
+setUpTestDriver(new ReadOnlyProps(props.entrySet().iterator()));
+}
+
+@Test
+public void testSimpleUpsertAndDelete() throws Exception {
+Assume.assumeTrue(HbaseCompatCapabilities.hasPreWALAppend());
+SchemaBuilder builder = new SchemaBuilder(getUrl());
+boolean createGlobalIndex = false;
+long ddl
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[
https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17246912#comment-17246912
]
ASF GitHub Bot commented on PHOENIX-5435:
-
gjacoby126 commented on a change in pull request #913:
URL: https://github.com/apache/phoenix/pull/913#discussion_r539742326
##
File path:
phoenix-core/src/it/java/org/apache/phoenix/end2end/WALAnnotationIT.java
##
@@ -0,0 +1,550 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.end2end;
+
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionInfo;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.coprocessor.BaseWALObserver;
+import org.apache.hadoop.hbase.coprocessor.ObserverContext;
+import org.apache.hadoop.hbase.coprocessor.WALCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.wal.WALCoprocessorHost;
+import org.apache.hadoop.hbase.regionserver.wal.WALEdit;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.wal.WAL;
+import org.apache.hadoop.hbase.wal.WALKey;
+import org.apache.phoenix.compat.hbase.HbaseCompatCapabilities;
+import org.apache.phoenix.compat.hbase.coprocessor.CompatIndexRegionObserver;
+import org.apache.phoenix.exception.SQLExceptionCode;
+import org.apache.phoenix.execute.MutationState;
+import org.apache.phoenix.hbase.index.IndexRegionObserver;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.PhoenixTestBuilder;
+import org.apache.phoenix.query.PhoenixTestBuilder.SchemaBuilder;
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.PTableType;
+import org.apache.phoenix.util.MetaDataUtil;
+import org.apache.phoenix.util.PhoenixRuntime;
+import org.apache.phoenix.util.ReadOnlyProps;
+import org.apache.phoenix.util.SchemaUtil;
+import org.apache.phoenix.util.TestUtil;
+import org.junit.Assert;
+import org.junit.Assume;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+
+import static
org.apache.phoenix.jdbc.PhoenixDatabaseMetaData.CHANGE_DETECTION_ENABLED;
+import static org.junit.Assert.assertNotNull;
+
+@RunWith(Parameterized.class)
+@Category(NeedsOwnMiniClusterTest.class)
+public class WALAnnotationIT extends BaseUniqueNamesOwnClusterIT {
+private final boolean isImmutable;
+private final boolean isMultiTenant;
+
+// name is used by failsafe as file name in reports
+@Parameterized.Parameters(name =
"WALAnnotationIT_isImmutable={0}_isMultiTenant={1}")
+public static synchronized Collection data() {
+return Arrays.asList(new Object[]{true, true}, new Object[]{true,
false},
+new Object[]{false, true}, new Object[]{false, false});
+}
+
+public WALAnnotationIT(boolean isImmutable, boolean isMultiTenant) {
+this.isImmutable = isImmutable;
+this.isMultiTenant = isMultiTenant;
+}
+
+@BeforeClass
+public static synchronized void doSetup() throws Exception {
+Map props = new HashMap<>(2);
+props.put("hbase.coprocessor.wal.classes",
+AnnotatedWALObserver.class.getName());
+props.put(IndexRegionObserver.PHOENIX_APPEND_METADATA_TO_WAL, "true");
+props.put(QueryServices.ENABLE_SERVER_UPSERT_SELECT, "true");
+setUpTestDriver(new ReadOnlyProps(props.entrySet().iterator()));
+}
+
+@Test
+public void testSimpleUpsertAndDelete() throws Exception {
+Assume.assumeTrue(HbaseCompatCapabilities.hasPreWALAppend());
+SchemaBuilder builder = new SchemaBuilder(getUrl());
+boolean createGlobalIndex = false;
+long ddl
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[
https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17246915#comment-17246915
]
ASF GitHub Bot commented on PHOENIX-5435:
-
gjacoby126 commented on a change in pull request #913:
URL: https://github.com/apache/phoenix/pull/913#discussion_r539742976
##
File path:
phoenix-core/src/main/java/org/apache/phoenix/hbase/index/IndexRegionObserver.java
##
@@ -1034,6 +1066,15 @@ private void
removeBatchMutateContext(ObserverContext c,
WALKey key,
+ WALEdit edit) {
+if (HbaseCompatCapabilities.hasPreWALAppend() && shouldWALAppend) {
Review comment:
Despite the name, IndexRegionObserver is actually the coproc that lives
on _base tables_, not indexes. (GlobalIndexChecker is the index coproc)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Annotate HBase WALs with Phoenix Metadata
> -
>
> Key: PHOENIX-5435
> URL: https://issues.apache.org/jira/browse/PHOENIX-5435
> Project: Phoenix
> Issue Type: Sub-task
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>Priority: Major
> Attachments: PHOENIX-5435-4.x.patch
>
>
> HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase
> replication and some HBase backup frameworks. The WALs contain HBase-level
> metadata such as table and region, but lack Phoenix-level metadata. That
> means that it's quite difficult to build correct logic that needs to know
> about Phoenix-level constructs such as multi-tenancy, views, or indexes.
> HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate
> extra key/value pairs of metadata into the HBase WAL. We should have the
> option to annotate the tuple , or
> some hashed way to reconstruct that tuple into the WAL. It should have a
> feature toggle so operators who don't need it don't bear the slight extra
> storage cost.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[ https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17246969#comment-17246969 ] ASF GitHub Bot commented on PHOENIX-5435: - stoty commented on pull request #913: URL: https://github.com/apache/phoenix/pull/913#issuecomment-742188973 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 10s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. | ||| _ 4.x Compile Tests _ | | +0 :ok: | mvndep | 0m 31s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 12m 3s | 4.x passed | | +1 :green_heart: | compile | 1m 36s | 4.x passed | | +1 :green_heart: | checkstyle | 6m 24s | 4.x passed | | +1 :green_heart: | javadoc | 2m 47s | 4.x passed | | +0 :ok: | spotbugs | 0m 29s | phoenix-hbase-compat-1.5.0 in 4.x has 3 extant spotbugs warnings. | | +0 :ok: | spotbugs | 0m 25s | phoenix-hbase-compat-1.4.0 in 4.x has 3 extant spotbugs warnings. | | +0 :ok: | spotbugs | 0m 23s | phoenix-hbase-compat-1.3.0 in 4.x has 3 extant spotbugs warnings. | | +0 :ok: | spotbugs | 4m 43s | root in 4.x has 1005 extant spotbugs warnings. | | +0 :ok: | spotbugs | 3m 11s | phoenix-core in 4.x has 951 extant spotbugs warnings. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 7m 22s | the patch passed | | +1 :green_heart: | compile | 2m 4s | the patch passed | | +1 :green_heart: | cc | 2m 4s | the patch passed | | +1 :green_heart: | javac | 2m 4s | the patch passed | | -1 :x: | checkstyle | 8m 10s | root: The patch generated 506 new + 23134 unchanged - 191 fixed = 23640 total (was 23325) | | +1 :green_heart: | prototool | 0m 2s | There were no new prototool issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | | +1 :green_heart: | javadoc | 3m 5s | the patch passed | | -1 :x: | spotbugs | 3m 59s | phoenix-core generated 2 new + 951 unchanged - 0 fixed = 953 total (was 951) | | -1 :x: | spotbugs | 5m 32s | root generated 2 new + 1005 unchanged - 0 fixed = 1007 total (was 1005) | ||| _ Other Tests _ | | -1 :x: | unit | 198m 10s | root in the patch failed. | | +1 :green_heart: | asflicense | 2m 8s | The patch does not generate ASF License warnings. | | | | 271m 24s | | | Reason | Tests | |---:|:--| | FindBugs | module:phoenix-core | | | Switch statement found in org.apache.phoenix.hbase.index.scanner.FilteredKeyValueScanner.seekToNextUnfilteredKeyValue() where default case is missing At FilteredKeyValueScanner.java:where default case is missing At FilteredKeyValueScanner.java:[lines 84-101] | | | org.apache.phoenix.jdbc.PhoenixDatabaseMetaData.CHANGE_DETECTION_ENABLED_BYTES is a mutable array At MetaDataEndpointImpl.java: At MetaDataEndpointImpl.java:[line 336] | | FindBugs | module:root | | | Switch statement found in org.apache.phoenix.hbase.index.scanner.FilteredKeyValueScanner.seekToNextUnfilteredKeyValue() where default case is missing At FilteredKeyValueScanner.java:where default case is missing At FilteredKeyValueScanner.java:[lines 84-101] | | | org.apache.phoenix.jdbc.PhoenixDatabaseMetaData.CHANGE_DETECTION_ENABLED_BYTES is a mutable array At MetaDataEndpointImpl.java: At MetaDataEndpointImpl.java:[line 336] | | Failed junit tests | phoenix.end2end.IndexExtendedIT | | | phoenix.util.IndexScrutinyIT | | | phoenix.end2end.IndexScrutinyToolIT | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-913/11/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/phoenix/pull/913 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile cc prototool xml | | uname | Linux b75c8bf09267 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev/phoenix-personali
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[
https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17247445#comment-17247445
]
ASF GitHub Bot commented on PHOENIX-5435:
-
ChinmaySKulkarni commented on a change in pull request #913:
URL: https://github.com/apache/phoenix/pull/913#discussion_r540433388
##
File path:
phoenix-core/src/it/java/org/apache/phoenix/end2end/WALAnnotationIT.java
##
@@ -0,0 +1,550 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.end2end;
+
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionInfo;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.coprocessor.BaseWALObserver;
+import org.apache.hadoop.hbase.coprocessor.ObserverContext;
+import org.apache.hadoop.hbase.coprocessor.WALCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.wal.WALCoprocessorHost;
+import org.apache.hadoop.hbase.regionserver.wal.WALEdit;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.wal.WAL;
+import org.apache.hadoop.hbase.wal.WALKey;
+import org.apache.phoenix.compat.hbase.HbaseCompatCapabilities;
+import org.apache.phoenix.compat.hbase.coprocessor.CompatIndexRegionObserver;
+import org.apache.phoenix.exception.SQLExceptionCode;
+import org.apache.phoenix.execute.MutationState;
+import org.apache.phoenix.hbase.index.IndexRegionObserver;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.PhoenixTestBuilder;
+import org.apache.phoenix.query.PhoenixTestBuilder.SchemaBuilder;
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.PTableType;
+import org.apache.phoenix.util.MetaDataUtil;
+import org.apache.phoenix.util.PhoenixRuntime;
+import org.apache.phoenix.util.ReadOnlyProps;
+import org.apache.phoenix.util.SchemaUtil;
+import org.apache.phoenix.util.TestUtil;
+import org.junit.Assert;
+import org.junit.Assume;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+
+import static
org.apache.phoenix.jdbc.PhoenixDatabaseMetaData.CHANGE_DETECTION_ENABLED;
+import static org.junit.Assert.assertNotNull;
+
+@RunWith(Parameterized.class)
+@Category(NeedsOwnMiniClusterTest.class)
+public class WALAnnotationIT extends BaseUniqueNamesOwnClusterIT {
+private final boolean isImmutable;
+private final boolean isMultiTenant;
+
+// name is used by failsafe as file name in reports
+@Parameterized.Parameters(name =
"WALAnnotationIT_isImmutable={0}_isMultiTenant={1}")
+public static synchronized Collection data() {
+return Arrays.asList(new Object[]{true, true}, new Object[]{true,
false},
+new Object[]{false, true}, new Object[]{false, false});
+}
+
+public WALAnnotationIT(boolean isImmutable, boolean isMultiTenant) {
+this.isImmutable = isImmutable;
+this.isMultiTenant = isMultiTenant;
+}
+
+@BeforeClass
+public static synchronized void doSetup() throws Exception {
+Map props = new HashMap<>(2);
+props.put("hbase.coprocessor.wal.classes",
+AnnotatedWALObserver.class.getName());
+props.put(IndexRegionObserver.PHOENIX_APPEND_METADATA_TO_WAL, "true");
+props.put(QueryServices.ENABLE_SERVER_UPSERT_SELECT, "true");
+setUpTestDriver(new ReadOnlyProps(props.entrySet().iterator()));
+}
+
+@Test
+public void testSimpleUpsertAndDelete() throws Exception {
+Assume.assumeTrue(HbaseCompatCapabilities.hasPreWALAppend());
+SchemaBuilder builder = new SchemaBuilder(getUrl());
+boolean createGlobalIndex = false;
+lo
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[
https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17248209#comment-17248209
]
ASF GitHub Bot commented on PHOENIX-5435:
-
gjacoby126 commented on a change in pull request #913:
URL: https://github.com/apache/phoenix/pull/913#discussion_r541345592
##
File path:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/GroupedAggregateRegionObserver.java
##
@@ -400,7 +400,6 @@ protected boolean isRegionObserverFor(Scan scan) {
* Used for an aggregate query in which the key order does not necessarily
match the group by
* key order. In this case, we must collect all distinct groups within a
region into a map,
* aggregating as we go.
- * @param limit TODO
Review comment:
Removed the params from the Javadoc as they were Javadoc errors.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Annotate HBase WALs with Phoenix Metadata
> -
>
> Key: PHOENIX-5435
> URL: https://issues.apache.org/jira/browse/PHOENIX-5435
> Project: Phoenix
> Issue Type: Sub-task
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>Priority: Major
> Attachments: PHOENIX-5435-4.x.patch
>
>
> HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase
> replication and some HBase backup frameworks. The WALs contain HBase-level
> metadata such as table and region, but lack Phoenix-level metadata. That
> means that it's quite difficult to build correct logic that needs to know
> about Phoenix-level constructs such as multi-tenancy, views, or indexes.
> HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate
> extra key/value pairs of metadata into the HBase WAL. We should have the
> option to annotate the tuple , or
> some hashed way to reconstruct that tuple into the WAL. It should have a
> feature toggle so operators who don't need it don't bear the slight extra
> storage cost.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[ https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17248211#comment-17248211 ] ASF GitHub Bot commented on PHOENIX-5435: - gjacoby126 commented on a change in pull request #913: URL: https://github.com/apache/phoenix/pull/913#discussion_r541346794 ## File path: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java ## @@ -1160,6 +1169,12 @@ private PTable getTable(RegionScanner scanner, long clientTimeStamp, long tableT null : PLong.INSTANCE.getCodec().decodeLong(lastDDLTimestampKv.getValueArray(), lastDDLTimestampKv.getValueOffset(), SortOrder.getDefault()); +Cell changeDetectionEnabledKv = tableKeyValues[CHANGE_DETECTION_ENABLED_INDEX]; +boolean isChangeDetectionEnabled = changeDetectionEnabledKv != null Review comment: Since it's an optional field in the protobuf and the old client won't even know to deserialize it I don't think it would cause a problem. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Annotate HBase WALs with Phoenix Metadata > - > > Key: PHOENIX-5435 > URL: https://issues.apache.org/jira/browse/PHOENIX-5435 > Project: Phoenix > Issue Type: Sub-task >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > Attachments: PHOENIX-5435-4.x.patch > > > HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase > replication and some HBase backup frameworks. The WALs contain HBase-level > metadata such as table and region, but lack Phoenix-level metadata. That > means that it's quite difficult to build correct logic that needs to know > about Phoenix-level constructs such as multi-tenancy, views, or indexes. > HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate > extra key/value pairs of metadata into the HBase WAL. We should have the > option to annotate the tuple , or > some hashed way to reconstruct that tuple into the WAL. It should have a > feature toggle so operators who don't need it don't bear the slight extra > storage cost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[ https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17248212#comment-17248212 ] ASF GitHub Bot commented on PHOENIX-5435: - gjacoby126 commented on a change in pull request #913: URL: https://github.com/apache/phoenix/pull/913#discussion_r541347335 ## File path: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/UngroupedAggregateRegionScanner.java ## @@ -64,6 +64,7 @@ import org.apache.phoenix.cache.GlobalCache; import org.apache.phoenix.cache.TenantCache; import org.apache.phoenix.exception.DataExceedsCapacityException; +import org.apache.phoenix.execute.MutationState; Review comment: removed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Annotate HBase WALs with Phoenix Metadata > - > > Key: PHOENIX-5435 > URL: https://issues.apache.org/jira/browse/PHOENIX-5435 > Project: Phoenix > Issue Type: Sub-task >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > Attachments: PHOENIX-5435-4.x.patch > > > HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase > replication and some HBase backup frameworks. The WALs contain HBase-level > metadata such as table and region, but lack Phoenix-level metadata. That > means that it's quite difficult to build correct logic that needs to know > about Phoenix-level constructs such as multi-tenancy, views, or indexes. > HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate > extra key/value pairs of metadata into the HBase WAL. We should have the > option to annotate the tuple , or > some hashed way to reconstruct that tuple into the WAL. It should have a > feature toggle so operators who don't need it don't bear the slight extra > storage cost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[
https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17248214#comment-17248214
]
ASF GitHub Bot commented on PHOENIX-5435:
-
gjacoby126 commented on a change in pull request #913:
URL: https://github.com/apache/phoenix/pull/913#discussion_r541349997
##
File path:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/UngroupedAggregateRegionScanner.java
##
@@ -609,6 +613,9 @@ public boolean next(List resultsToReturn) throws
IOException {
} while (hasMore &&
(EnvironmentEdgeManager.currentTimeMillis() - startTime) < pageSizeInMs);
if (!mutations.isEmpty()) {
+if (isDelete || isUpsert) {
+annotateDataMutations(mutations, scan);
+}
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Annotate HBase WALs with Phoenix Metadata
> -
>
> Key: PHOENIX-5435
> URL: https://issues.apache.org/jira/browse/PHOENIX-5435
> Project: Phoenix
> Issue Type: Sub-task
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>Priority: Major
> Attachments: PHOENIX-5435-4.x.patch
>
>
> HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase
> replication and some HBase backup frameworks. The WALs contain HBase-level
> metadata such as table and region, but lack Phoenix-level metadata. That
> means that it's quite difficult to build correct logic that needs to know
> about Phoenix-level constructs such as multi-tenancy, views, or indexes.
> HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate
> extra key/value pairs of metadata into the HBase WAL. We should have the
> option to annotate the tuple , or
> some hashed way to reconstruct that tuple into the WAL. It should have a
> feature toggle so operators who don't need it don't bear the slight extra
> storage cost.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[
https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17248215#comment-17248215
]
ASF GitHub Bot commented on PHOENIX-5435:
-
gjacoby126 commented on a change in pull request #913:
URL: https://github.com/apache/phoenix/pull/913#discussion_r541350748
##
File path:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/UngroupedAggregateRegionScanner.java
##
@@ -644,4 +651,33 @@ public boolean next(List resultsToReturn) throws
IOException {
public long getMaxResultSize() {
return scan.getMaxResultSize();
}
+
+private void
annotateDataMutations(UngroupedAggregateRegionObserver.MutationList
mutationsList,
+ Scan scan) {
+byte[] tenantId = null;
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Annotate HBase WALs with Phoenix Metadata
> -
>
> Key: PHOENIX-5435
> URL: https://issues.apache.org/jira/browse/PHOENIX-5435
> Project: Phoenix
> Issue Type: Sub-task
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>Priority: Major
> Attachments: PHOENIX-5435-4.x.patch
>
>
> HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase
> replication and some HBase backup frameworks. The WALs contain HBase-level
> metadata such as table and region, but lack Phoenix-level metadata. That
> means that it's quite difficult to build correct logic that needs to know
> about Phoenix-level constructs such as multi-tenancy, views, or indexes.
> HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate
> extra key/value pairs of metadata into the HBase WAL. We should have the
> option to annotate the tuple , or
> some hashed way to reconstruct that tuple into the WAL. It should have a
> feature toggle so operators who don't need it don't bear the slight extra
> storage cost.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[
https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17248221#comment-17248221
]
ASF GitHub Bot commented on PHOENIX-5435:
-
gjacoby126 commented on a change in pull request #913:
URL: https://github.com/apache/phoenix/pull/913#discussion_r541362377
##
File path:
phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java
##
@@ -682,6 +687,49 @@ private void generateMutations(final TableRef tableRef,
final long mutationTimes
values.putAll(modifiedValues);
}
+public static void annotateMutationsWithMetadata(PTable table,
List rowMutations) {
+//only annotate if the change detection flag is on the table and HBase
supports
+// preWALAppend coprocs server-side
+if (table == null || !table.isChangeDetectionEnabled()
+|| !HbaseCompatCapabilities.hasPreWALAppend()) {
+return;
+}
+//annotate each mutation with enough metadata so that anyone
interested can
+// deterministically figure out exactly what Phoenix schema object
created the mutation
+// Server-side we can annotate the HBase WAL with these.
+for (Mutation mutation : rowMutations) {
+annotateMutationWithMetadata(table, mutation);
+}
+
+}
+
+public static void annotateMutationWithMetadata(PTable table, Mutation
mutation) {
Review comment:
Switched to private
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Annotate HBase WALs with Phoenix Metadata
> -
>
> Key: PHOENIX-5435
> URL: https://issues.apache.org/jira/browse/PHOENIX-5435
> Project: Phoenix
> Issue Type: Sub-task
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>Priority: Major
> Attachments: PHOENIX-5435-4.x.patch
>
>
> HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase
> replication and some HBase backup frameworks. The WALs contain HBase-level
> metadata such as table and region, but lack Phoenix-level metadata. That
> means that it's quite difficult to build correct logic that needs to know
> about Phoenix-level constructs such as multi-tenancy, views, or indexes.
> HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate
> extra key/value pairs of metadata into the HBase WAL. We should have the
> option to annotate the tuple , or
> some hashed way to reconstruct that tuple into the WAL. It should have a
> feature toggle so operators who don't need it don't bear the slight extra
> storage cost.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[
https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17248222#comment-17248222
]
ASF GitHub Bot commented on PHOENIX-5435:
-
gjacoby126 commented on a change in pull request #913:
URL: https://github.com/apache/phoenix/pull/913#discussion_r541362722
##
File path:
phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java
##
@@ -682,6 +687,49 @@ private void generateMutations(final TableRef tableRef,
final long mutationTimes
values.putAll(modifiedValues);
}
+private void annotateMutationsWithMetadata(PTable table, List
rowMutations) {
+//only annotate if the change detection flag is on the table and HBase
supports
+// preWALAppend coprocs server-side
+if (table == null || !table.isChangeDetectionEnabled()
+|| !HbaseCompatCapabilities.hasPreWALAppend()) {
+return;
+}
+//annotate each mutation with enough metadata so that anyone
interested can
+// deterministically figure out exactly what Phoenix schema object
created the mutation
+// Server-side we can annotate the HBase WAL with these.
+for (Mutation mutation : rowMutations) {
+annotateMutationWithMetadata(table, mutation);
+}
+
+}
+
+public static void annotateMutationWithMetadata(PTable table, Mutation
mutation) {
Review comment:
Consolidated in WALAnnotationUtil as @ChinmaySKulkarni suggested.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Annotate HBase WALs with Phoenix Metadata
> -
>
> Key: PHOENIX-5435
> URL: https://issues.apache.org/jira/browse/PHOENIX-5435
> Project: Phoenix
> Issue Type: Sub-task
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>Priority: Major
> Attachments: PHOENIX-5435-4.x.patch
>
>
> HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase
> replication and some HBase backup frameworks. The WALs contain HBase-level
> metadata such as table and region, but lack Phoenix-level metadata. That
> means that it's quite difficult to build correct logic that needs to know
> about Phoenix-level constructs such as multi-tenancy, views, or indexes.
> HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate
> extra key/value pairs of metadata into the HBase WAL. We should have the
> option to annotate the tuple , or
> some hashed way to reconstruct that tuple into the WAL. It should have a
> feature toggle so operators who don't need it don't bear the slight extra
> storage cost.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[
https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17248224#comment-17248224
]
ASF GitHub Bot commented on PHOENIX-5435:
-
gjacoby126 commented on a change in pull request #913:
URL: https://github.com/apache/phoenix/pull/913#discussion_r541364364
##
File path:
phoenix-core/src/main/java/org/apache/phoenix/index/GlobalIndexChecker.java
##
@@ -93,10 +106,11 @@
* the verified version that is masked by the unverified version(s).
*
*/
-public class GlobalIndexChecker extends BaseRegionObserver {
+public class GlobalIndexChecker extends CompatGlobalIndexChecker {
Review comment:
We don't need CompatGlobalIndexChecker now that we don't annotate
indexes so I removed it
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Annotate HBase WALs with Phoenix Metadata
> -
>
> Key: PHOENIX-5435
> URL: https://issues.apache.org/jira/browse/PHOENIX-5435
> Project: Phoenix
> Issue Type: Sub-task
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>Priority: Major
> Attachments: PHOENIX-5435-4.x.patch
>
>
> HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase
> replication and some HBase backup frameworks. The WALs contain HBase-level
> metadata such as table and region, but lack Phoenix-level metadata. That
> means that it's quite difficult to build correct logic that needs to know
> about Phoenix-level constructs such as multi-tenancy, views, or indexes.
> HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate
> extra key/value pairs of metadata into the HBase WAL. We should have the
> option to annotate the tuple , or
> some hashed way to reconstruct that tuple into the WAL. It should have a
> feature toggle so operators who don't need it don't bear the slight extra
> storage cost.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[ https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17248223#comment-17248223 ] ASF GitHub Bot commented on PHOENIX-5435: - gjacoby126 commented on a change in pull request #913: URL: https://github.com/apache/phoenix/pull/913#discussion_r541363956 ## File path: phoenix-core/src/main/java/org/apache/phoenix/index/GlobalIndexChecker.java ## @@ -50,11 +53,19 @@ import org.apache.hadoop.hbase.filter.FilterList; import org.apache.hadoop.hbase.filter.PageFilter; import org.apache.hadoop.hbase.io.ImmutableBytesWritable; +import org.apache.hadoop.hbase.regionserver.MiniBatchOperationInProgress; import org.apache.hadoop.hbase.regionserver.Region; import org.apache.hadoop.hbase.regionserver.RegionScanner; import org.apache.hadoop.hbase.regionserver.ScannerContext; +import org.apache.hadoop.hbase.regionserver.wal.WALEdit; import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.wal.WALKey; +import org.apache.phoenix.compat.hbase.HbaseCompatCapabilities; +import org.apache.phoenix.compat.hbase.coprocessor.CompatGlobalIndexChecker; +import org.apache.phoenix.compat.hbase.coprocessor.CompatIndexRegionObserver; import org.apache.phoenix.coprocessor.BaseScannerRegionObserver; +import org.apache.phoenix.execute.MutationState; +import org.apache.phoenix.hbase.index.IndexRegionObserver; Review comment: Fixed -- lots were remnants from when we annotated indexes This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Annotate HBase WALs with Phoenix Metadata > - > > Key: PHOENIX-5435 > URL: https://issues.apache.org/jira/browse/PHOENIX-5435 > Project: Phoenix > Issue Type: Sub-task >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > Attachments: PHOENIX-5435-4.x.patch > > > HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase > replication and some HBase backup frameworks. The WALs contain HBase-level > metadata such as table and region, but lack Phoenix-level metadata. That > means that it's quite difficult to build correct logic that needs to know > about Phoenix-level constructs such as multi-tenancy, views, or indexes. > HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate > extra key/value pairs of metadata into the HBase WAL. We should have the > option to annotate the tuple , or > some hashed way to reconstruct that tuple into the WAL. It should have a > feature toggle so operators who don't need it don't bear the slight extra > storage cost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[
https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17248225#comment-17248225
]
ASF GitHub Bot commented on PHOENIX-5435:
-
gjacoby126 commented on a change in pull request #913:
URL: https://github.com/apache/phoenix/pull/913#discussion_r541364685
##
File path:
phoenix-core/src/main/java/org/apache/phoenix/index/IndexMaintainer.java
##
@@ -1257,7 +1261,11 @@ public ImmutableBytesPtr getEmptyKeyValueFamily() {
// If if there are no covered columns, we know it's our default name
return emptyKeyValueCFPtr;
}
-
+
+public String getLogicalIndexName() {
+return logicalIndexName;
Review comment:
Correct. Will add a comment.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Annotate HBase WALs with Phoenix Metadata
> -
>
> Key: PHOENIX-5435
> URL: https://issues.apache.org/jira/browse/PHOENIX-5435
> Project: Phoenix
> Issue Type: Sub-task
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>Priority: Major
> Attachments: PHOENIX-5435-4.x.patch
>
>
> HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase
> replication and some HBase backup frameworks. The WALs contain HBase-level
> metadata such as table and region, but lack Phoenix-level metadata. That
> means that it's quite difficult to build correct logic that needs to know
> about Phoenix-level constructs such as multi-tenancy, views, or indexes.
> HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate
> extra key/value pairs of metadata into the HBase WAL. We should have the
> option to annotate the tuple , or
> some hashed way to reconstruct that tuple into the WAL. It should have a
> feature toggle so operators who don't need it don't bear the slight extra
> storage cost.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[
https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17248226#comment-17248226
]
ASF GitHub Bot commented on PHOENIX-5435:
-
gjacoby126 commented on a change in pull request #913:
URL: https://github.com/apache/phoenix/pull/913#discussion_r541367097
##
File path:
phoenix-core/src/main/java/org/apache/phoenix/util/WALAnnotationUtil.java
##
@@ -0,0 +1,49 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.util;
+
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.wal.WALKey;
+import org.apache.phoenix.compat.hbase.coprocessor.CompatIndexRegionObserver;
+import org.apache.phoenix.execute.MutationState;
+import org.apache.phoenix.hbase.index.IndexRegionObserver;
+
+import java.util.Map;
+
+/**
+ * Utility functions shared between IndexRegionObserver and GlobalIndexChecker
for annotating the
+ * HBase WAL with Phoenix-level metadata about mutations.
+ */
+public class WALAnnotationUtil {
Review comment:
It's not specific to indexing.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Annotate HBase WALs with Phoenix Metadata
> -
>
> Key: PHOENIX-5435
> URL: https://issues.apache.org/jira/browse/PHOENIX-5435
> Project: Phoenix
> Issue Type: Sub-task
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>Priority: Major
> Attachments: PHOENIX-5435-4.x.patch
>
>
> HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase
> replication and some HBase backup frameworks. The WALs contain HBase-level
> metadata such as table and region, but lack Phoenix-level metadata. That
> means that it's quite difficult to build correct logic that needs to know
> about Phoenix-level constructs such as multi-tenancy, views, or indexes.
> HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate
> extra key/value pairs of metadata into the HBase WAL. We should have the
> option to annotate the tuple , or
> some hashed way to reconstruct that tuple into the WAL. It should have a
> feature toggle so operators who don't need it don't bear the slight extra
> storage cost.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[
https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17248232#comment-17248232
]
ASF GitHub Bot commented on PHOENIX-5435:
-
gjacoby126 commented on a change in pull request #913:
URL: https://github.com/apache/phoenix/pull/913#discussion_r541382620
##
File path: phoenix-core/src/main/java/org/apache/phoenix/util/ScanUtil.java
##
@@ -1239,4 +1239,21 @@ public static boolean isDummy(List result) {
Cell cell = result.get(0);
return CellUtil.matchingColumn(cell, EMPTY_BYTE_ARRAY,
EMPTY_BYTE_ARRAY);
}
+
+public static void setWALAnnotationAttributes(PTable table, Scan scan) {
Review comment:
Added comments to the Utility methods in ScanUtil and WALAnnotationUtil
to avoid duplication
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Annotate HBase WALs with Phoenix Metadata
> -
>
> Key: PHOENIX-5435
> URL: https://issues.apache.org/jira/browse/PHOENIX-5435
> Project: Phoenix
> Issue Type: Sub-task
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>Priority: Major
> Attachments: PHOENIX-5435-4.x.patch
>
>
> HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase
> replication and some HBase backup frameworks. The WALs contain HBase-level
> metadata such as table and region, but lack Phoenix-level metadata. That
> means that it's quite difficult to build correct logic that needs to know
> about Phoenix-level constructs such as multi-tenancy, views, or indexes.
> HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate
> extra key/value pairs of metadata into the HBase WAL. We should have the
> option to annotate the tuple , or
> some hashed way to reconstruct that tuple into the WAL. It should have a
> feature toggle so operators who don't need it don't bear the slight extra
> storage cost.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[
https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17248234#comment-17248234
]
ASF GitHub Bot commented on PHOENIX-5435:
-
gjacoby126 commented on a change in pull request #913:
URL: https://github.com/apache/phoenix/pull/913#discussion_r541383107
##
File path:
phoenix-hbase-compat-1.5.0/src/main/java/org/apache/phoenix/compat/hbase/coprocessor/CompatGlobalIndexChecker.java
##
@@ -0,0 +1,27 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.compat.hbase.coprocessor;
+
+import org.apache.hadoop.hbase.coprocessor.BaseRegionObserver;
+
+public class CompatGlobalIndexChecker extends BaseRegionObserver {
+//HBase 1.5+ has preWALAppend already declared in BaseRegionObserver, so
this class is only
Review comment:
This particular file no longer exists, but I fixed some of these as I
found them. The reason why is just that I usually don't type a space when I
start a one-line comment, and when I find I need to make it multi-line the IDE
fixes the spacing on subsequent lines but not the first one. :-)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Annotate HBase WALs with Phoenix Metadata
> -
>
> Key: PHOENIX-5435
> URL: https://issues.apache.org/jira/browse/PHOENIX-5435
> Project: Phoenix
> Issue Type: Sub-task
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>Priority: Major
> Attachments: PHOENIX-5435-4.x.patch
>
>
> HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase
> replication and some HBase backup frameworks. The WALs contain HBase-level
> metadata such as table and region, but lack Phoenix-level metadata. That
> means that it's quite difficult to build correct logic that needs to know
> about Phoenix-level constructs such as multi-tenancy, views, or indexes.
> HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate
> extra key/value pairs of metadata into the HBase WAL. We should have the
> option to annotate the tuple , or
> some hashed way to reconstruct that tuple into the WAL. It should have a
> feature toggle so operators who don't need it don't bear the slight extra
> storage cost.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[ https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17248278#comment-17248278 ] ASF GitHub Bot commented on PHOENIX-5435: - stoty commented on pull request #913: URL: https://github.com/apache/phoenix/pull/913#issuecomment-743673616 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. | ||| _ 4.x Compile Tests _ | | +0 :ok: | mvndep | 0m 31s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 10m 58s | 4.x passed | | +1 :green_heart: | compile | 1m 21s | 4.x passed | | +1 :green_heart: | checkstyle | 9m 47s | 4.x passed | | +1 :green_heart: | javadoc | 2m 24s | 4.x passed | | +0 :ok: | spotbugs | 0m 27s | phoenix-hbase-compat-1.5.0 in 4.x has 3 extant spotbugs warnings. | | +0 :ok: | spotbugs | 0m 24s | phoenix-hbase-compat-1.4.0 in 4.x has 3 extant spotbugs warnings. | | +0 :ok: | spotbugs | 0m 23s | phoenix-hbase-compat-1.3.0 in 4.x has 3 extant spotbugs warnings. | | +0 :ok: | spotbugs | 4m 9s | root in 4.x has 1005 extant spotbugs warnings. | | +0 :ok: | spotbugs | 2m 44s | phoenix-core in 4.x has 951 extant spotbugs warnings. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 6m 13s | the patch passed | | +1 :green_heart: | compile | 1m 30s | the patch passed | | +1 :green_heart: | cc | 1m 30s | the patch passed | | +1 :green_heart: | javac | 1m 30s | the patch passed | | -1 :x: | checkstyle | 9m 15s | root: The patch generated 474 new + 23163 unchanged - 157 fixed = 23637 total (was 23320) | | +1 :green_heart: | prototool | 0m 1s | There were no new prototool issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +1 :green_heart: | javadoc | 2m 35s | the patch passed | | -1 :x: | spotbugs | 3m 19s | phoenix-core generated 2 new + 951 unchanged - 0 fixed = 953 total (was 951) | | -1 :x: | spotbugs | 4m 30s | root generated 2 new + 1005 unchanged - 0 fixed = 1007 total (was 1005) | ||| _ Other Tests _ | | -1 :x: | unit | 128m 30s | root in the patch failed. | | +1 :green_heart: | asflicense | 2m 30s | The patch does not generate ASF License warnings. | | | | 198m 43s | | | Reason | Tests | |---:|:--| | FindBugs | module:phoenix-core | | | Switch statement found in org.apache.phoenix.hbase.index.scanner.FilteredKeyValueScanner.seekToNextUnfilteredKeyValue() where default case is missing At FilteredKeyValueScanner.java:where default case is missing At FilteredKeyValueScanner.java:[lines 84-101] | | | org.apache.phoenix.jdbc.PhoenixDatabaseMetaData.CHANGE_DETECTION_ENABLED_BYTES is a mutable array At MetaDataEndpointImpl.java: At MetaDataEndpointImpl.java:[line 336] | | FindBugs | module:root | | | Switch statement found in org.apache.phoenix.hbase.index.scanner.FilteredKeyValueScanner.seekToNextUnfilteredKeyValue() where default case is missing At FilteredKeyValueScanner.java:where default case is missing At FilteredKeyValueScanner.java:[lines 84-101] | | | org.apache.phoenix.jdbc.PhoenixDatabaseMetaData.CHANGE_DETECTION_ENABLED_BYTES is a mutable array At MetaDataEndpointImpl.java: At MetaDataEndpointImpl.java:[line 336] | | Failed junit tests | phoenix.end2end.index.ViewIndexIT | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-913/12/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/phoenix/pull/913 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile cc prototool xml | | uname | Linux fbce87f802cf 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev/phoenix-personality.sh | | git revision | 4.x / d43c362 | | Default Java | Private Build-1.8.0_2
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[
https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17249149#comment-17249149
]
ASF GitHub Bot commented on PHOENIX-5435:
-
gjacoby126 commented on a change in pull request #913:
URL: https://github.com/apache/phoenix/pull/913#discussion_r542572032
##
File path:
phoenix-core/src/test/java/org/apache/phoenix/coprocessor/TaskMetaDataEndpointTest.java
##
@@ -139,7 +140,12 @@ public RegionServerServices getRegionServerServices() {
public ConcurrentMap getSharedData() {
return null;
}
-};
+
+@Override
Review comment:
A previous version of this patch had PHOENIX-6155 but not its addendum
that fixed compilation on certain HBase versions. The current version has the
addendum and so this file shouldn't be altered in this patch anymore.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Annotate HBase WALs with Phoenix Metadata
> -
>
> Key: PHOENIX-5435
> URL: https://issues.apache.org/jira/browse/PHOENIX-5435
> Project: Phoenix
> Issue Type: Sub-task
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>Priority: Major
> Attachments: PHOENIX-5435-4.x.patch
>
>
> HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase
> replication and some HBase backup frameworks. The WALs contain HBase-level
> metadata such as table and region, but lack Phoenix-level metadata. That
> means that it's quite difficult to build correct logic that needs to know
> about Phoenix-level constructs such as multi-tenancy, views, or indexes.
> HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate
> extra key/value pairs of metadata into the HBase WAL. We should have the
> option to annotate the tuple , or
> some hashed way to reconstruct that tuple into the WAL. It should have a
> feature toggle so operators who don't need it don't bear the slight extra
> storage cost.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[ https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17249151#comment-17249151 ] ASF GitHub Bot commented on PHOENIX-5435: - gjacoby126 commented on a change in pull request #913: URL: https://github.com/apache/phoenix/pull/913#discussion_r542572942 ## File path: phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java ## @@ -674,6 +678,7 @@ private void generateMutations(final TableRef tableRef, final long mutationTimes mutation.setAttribute(PhoenixIndexBuilder.ATOMIC_OP_ATTRIB, onDupKeyBytes); } } +annotateMutationsWithMetadata(table, rowMutations); Review comment: Done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Annotate HBase WALs with Phoenix Metadata > - > > Key: PHOENIX-5435 > URL: https://issues.apache.org/jira/browse/PHOENIX-5435 > Project: Phoenix > Issue Type: Sub-task >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > Attachments: PHOENIX-5435-4.x.patch > > > HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase > replication and some HBase backup frameworks. The WALs contain HBase-level > metadata such as table and region, but lack Phoenix-level metadata. That > means that it's quite difficult to build correct logic that needs to know > about Phoenix-level constructs such as multi-tenancy, views, or indexes. > HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate > extra key/value pairs of metadata into the HBase WAL. We should have the > option to annotate the tuple , or > some hashed way to reconstruct that tuple into the WAL. It should have a > feature toggle so operators who don't need it don't bear the slight extra > storage cost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[ https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17249152#comment-17249152 ] ASF GitHub Bot commented on PHOENIX-5435: - gjacoby126 commented on a change in pull request #913: URL: https://github.com/apache/phoenix/pull/913#discussion_r542573315 ## File path: phoenix-core/src/it/java/org/apache/phoenix/end2end/ImmutableTableIT.java ## @@ -17,6 +17,7 @@ */ package org.apache.phoenix.end2end; +import org.apache.phoenix.util.TestDDLUtil; Review comment: Removed ## File path: phoenix-core/src/it/java/org/apache/phoenix/end2end/index/ViewIndexIT.java ## @@ -72,6 +72,7 @@ import org.apache.phoenix.util.PropertiesUtil; import org.apache.phoenix.util.QueryUtil; import org.apache.phoenix.util.SchemaUtil; +import org.apache.phoenix.util.TestDDLUtil; Review comment: Removed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Annotate HBase WALs with Phoenix Metadata > - > > Key: PHOENIX-5435 > URL: https://issues.apache.org/jira/browse/PHOENIX-5435 > Project: Phoenix > Issue Type: Sub-task >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > Attachments: PHOENIX-5435-4.x.patch > > > HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase > replication and some HBase backup frameworks. The WALs contain HBase-level > metadata such as table and region, but lack Phoenix-level metadata. That > means that it's quite difficult to build correct logic that needs to know > about Phoenix-level constructs such as multi-tenancy, views, or indexes. > HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate > extra key/value pairs of metadata into the HBase WAL. We should have the > option to annotate the tuple , or > some hashed way to reconstruct that tuple into the WAL. It should have a > feature toggle so operators who don't need it don't bear the slight extra > storage cost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[ https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17249155#comment-17249155 ] ASF GitHub Bot commented on PHOENIX-5435: - gjacoby126 commented on pull request #913: URL: https://github.com/apache/phoenix/pull/913#issuecomment-744599398 @gokceni , @ChinmaySKulkarni , @virajjasani , I believe I have addressed all outstanding comments (and also pushed up a fix for an NPE in ViewIndexIT the last test run found.) Could you please give it another look? Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Annotate HBase WALs with Phoenix Metadata > - > > Key: PHOENIX-5435 > URL: https://issues.apache.org/jira/browse/PHOENIX-5435 > Project: Phoenix > Issue Type: Sub-task >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > Attachments: PHOENIX-5435-4.x.patch > > > HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase > replication and some HBase backup frameworks. The WALs contain HBase-level > metadata such as table and region, but lack Phoenix-level metadata. That > means that it's quite difficult to build correct logic that needs to know > about Phoenix-level constructs such as multi-tenancy, views, or indexes. > HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate > extra key/value pairs of metadata into the HBase WAL. We should have the > option to annotate the tuple , or > some hashed way to reconstruct that tuple into the WAL. It should have a > feature toggle so operators who don't need it don't bear the slight extra > storage cost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[
https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17249153#comment-17249153
]
ASF GitHub Bot commented on PHOENIX-5435:
-
gjacoby126 commented on a change in pull request #913:
URL: https://github.com/apache/phoenix/pull/913#discussion_r542574493
##
File path:
phoenix-core/src/main/java/org/apache/phoenix/schema/MetaDataClient.java
##
@@ -2102,6 +2106,17 @@ private PTable createTableInternal(CreateTableStatement
statement, byte[][] spli
}
}
+Boolean isChangeDetectionEnabledProp =
+(Boolean)
TableProperty.CHANGE_DETECTION_ENABLED.getValue(tableProps);
+if (isChangeDetectionEnabledProp != null) {
+if (tableType != TABLE && tableType != VIEW) {
+throw new SQLExceptionInfo.Builder(
+
SQLExceptionCode.CHANGE_DETECTION_SUPPORTED_FOR_TABLES_AND_VIEWS_ONLY)
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Annotate HBase WALs with Phoenix Metadata
> -
>
> Key: PHOENIX-5435
> URL: https://issues.apache.org/jira/browse/PHOENIX-5435
> Project: Phoenix
> Issue Type: Sub-task
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>Priority: Major
> Attachments: PHOENIX-5435-4.x.patch
>
>
> HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase
> replication and some HBase backup frameworks. The WALs contain HBase-level
> metadata such as table and region, but lack Phoenix-level metadata. That
> means that it's quite difficult to build correct logic that needs to know
> about Phoenix-level constructs such as multi-tenancy, views, or indexes.
> HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate
> extra key/value pairs of metadata into the HBase WAL. We should have the
> option to annotate the tuple , or
> some hashed way to reconstruct that tuple into the WAL. It should have a
> feature toggle so operators who don't need it don't bear the slight extra
> storage cost.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[ https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17249158#comment-17249158 ] ASF GitHub Bot commented on PHOENIX-5435: - gokceni commented on a change in pull request #913: URL: https://github.com/apache/phoenix/pull/913#discussion_r542579426 ## File path: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/generated/ServerCachingProtos.java ## @@ -2192,6 +2192,21 @@ public Builder setColumnNameBytes( */ com.google.protobuf.ByteString getParentTableTypeBytes(); + +// optional string logicalIndexName = 25; +/** + * optional string logicalIndexName = 25; + */ +boolean hasLogicalIndexName(); +/** + * optional string logicalIndexName = 25; + */ +java.lang.String getLogicalIndexName(); Review comment: @gjacoby126 thanks for addressing my comments. There is just this question remaining. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Annotate HBase WALs with Phoenix Metadata > - > > Key: PHOENIX-5435 > URL: https://issues.apache.org/jira/browse/PHOENIX-5435 > Project: Phoenix > Issue Type: Sub-task >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > Attachments: PHOENIX-5435-4.x.patch > > > HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase > replication and some HBase backup frameworks. The WALs contain HBase-level > metadata such as table and region, but lack Phoenix-level metadata. That > means that it's quite difficult to build correct logic that needs to know > about Phoenix-level constructs such as multi-tenancy, views, or indexes. > HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate > extra key/value pairs of metadata into the HBase WAL. We should have the > option to annotate the tuple , or > some hashed way to reconstruct that tuple into the WAL. It should have a > feature toggle so operators who don't need it don't bear the slight extra > storage cost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[ https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17249169#comment-17249169 ] ASF GitHub Bot commented on PHOENIX-5435: - virajjasani commented on pull request #913: URL: https://github.com/apache/phoenix/pull/913#issuecomment-744607510 @gjacoby126 btw for the past couple of days, all precommit PR builds (that I came across) are timing out and precommit Jira builds for 4.x patch are passing by ~1/4 probability (that I am aware of) so you might also want to upload 4.x patch to Jira to trigger 4.x precommit Jira build. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Annotate HBase WALs with Phoenix Metadata > - > > Key: PHOENIX-5435 > URL: https://issues.apache.org/jira/browse/PHOENIX-5435 > Project: Phoenix > Issue Type: Sub-task >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > Attachments: PHOENIX-5435-4.x.patch > > > HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase > replication and some HBase backup frameworks. The WALs contain HBase-level > metadata such as table and region, but lack Phoenix-level metadata. That > means that it's quite difficult to build correct logic that needs to know > about Phoenix-level constructs such as multi-tenancy, views, or indexes. > HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate > extra key/value pairs of metadata into the HBase WAL. We should have the > option to annotate the tuple , or > some hashed way to reconstruct that tuple into the WAL. It should have a > feature toggle so operators who don't need it don't bear the slight extra > storage cost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[ https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17249170#comment-17249170 ] ASF GitHub Bot commented on PHOENIX-5435: - virajjasani edited a comment on pull request #913: URL: https://github.com/apache/phoenix/pull/913#issuecomment-744607510 @gjacoby126 btw for the past couple of days, all precommit PR builds for 4.x (that I came across) are timing out and precommit Jira builds for 4.x patch are passing by ~1/4 probability (that I am aware of) so you might also want to upload 4.x patch to Jira to trigger 4.x precommit Jira build. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Annotate HBase WALs with Phoenix Metadata > - > > Key: PHOENIX-5435 > URL: https://issues.apache.org/jira/browse/PHOENIX-5435 > Project: Phoenix > Issue Type: Sub-task >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > Attachments: PHOENIX-5435-4.x.patch > > > HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase > replication and some HBase backup frameworks. The WALs contain HBase-level > metadata such as table and region, but lack Phoenix-level metadata. That > means that it's quite difficult to build correct logic that needs to know > about Phoenix-level constructs such as multi-tenancy, views, or indexes. > HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate > extra key/value pairs of metadata into the HBase WAL. We should have the > option to annotate the tuple , or > some hashed way to reconstruct that tuple into the WAL. It should have a > feature toggle so operators who don't need it don't bear the slight extra > storage cost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[ https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17249281#comment-17249281 ] ASF GitHub Bot commented on PHOENIX-5435: - stoty commented on pull request #913: URL: https://github.com/apache/phoenix/pull/913#issuecomment-744693604 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 41s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. | ||| _ 4.x Compile Tests _ | | +0 :ok: | mvndep | 0m 38s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 11m 7s | 4.x passed | | +1 :green_heart: | compile | 1m 28s | 4.x passed | | +1 :green_heart: | checkstyle | 7m 31s | 4.x passed | | +1 :green_heart: | javadoc | 2m 25s | 4.x passed | | +0 :ok: | spotbugs | 0m 28s | phoenix-hbase-compat-1.5.0 in 4.x has 3 extant spotbugs warnings. | | +0 :ok: | spotbugs | 0m 23s | phoenix-hbase-compat-1.4.0 in 4.x has 3 extant spotbugs warnings. | | +0 :ok: | spotbugs | 0m 24s | phoenix-hbase-compat-1.3.0 in 4.x has 3 extant spotbugs warnings. | | +0 :ok: | spotbugs | 4m 18s | root in 4.x has 1005 extant spotbugs warnings. | | +0 :ok: | spotbugs | 2m 55s | phoenix-core in 4.x has 951 extant spotbugs warnings. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 6m 35s | the patch passed | | +1 :green_heart: | compile | 1m 37s | the patch passed | | +1 :green_heart: | cc | 1m 37s | the patch passed | | +1 :green_heart: | javac | 1m 37s | the patch passed | | -1 :x: | checkstyle | 7m 36s | root: The patch generated 459 new + 23220 unchanged - 157 fixed = 23679 total (was 23377) | | +1 :green_heart: | prototool | 0m 2s | There were no new prototool issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | | +1 :green_heart: | javadoc | 2m 31s | the patch passed | | -1 :x: | spotbugs | 3m 29s | phoenix-core generated 2 new + 951 unchanged - 0 fixed = 953 total (was 951) | | -1 :x: | spotbugs | 4m 36s | root generated 2 new + 1005 unchanged - 0 fixed = 1007 total (was 1005) | ||| _ Other Tests _ | | -1 :x: | unit | 127m 59s | root in the patch failed. | | +1 :green_heart: | asflicense | 2m 33s | The patch does not generate ASF License warnings. | | | | 195m 44s | | | Reason | Tests | |---:|:--| | FindBugs | module:phoenix-core | | | Switch statement found in org.apache.phoenix.hbase.index.scanner.FilteredKeyValueScanner.seekToNextUnfilteredKeyValue() where default case is missing At FilteredKeyValueScanner.java:where default case is missing At FilteredKeyValueScanner.java:[lines 84-101] | | | org.apache.phoenix.jdbc.PhoenixDatabaseMetaData.CHANGE_DETECTION_ENABLED_BYTES is a mutable array At MetaDataEndpointImpl.java: At MetaDataEndpointImpl.java:[line 336] | | FindBugs | module:root | | | Switch statement found in org.apache.phoenix.hbase.index.scanner.FilteredKeyValueScanner.seekToNextUnfilteredKeyValue() where default case is missing At FilteredKeyValueScanner.java:where default case is missing At FilteredKeyValueScanner.java:[lines 84-101] | | | org.apache.phoenix.jdbc.PhoenixDatabaseMetaData.CHANGE_DETECTION_ENABLED_BYTES is a mutable array At MetaDataEndpointImpl.java: At MetaDataEndpointImpl.java:[line 336] | | Failed junit tests | phoenix.end2end.ViewMetadataIT | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-913/13/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/phoenix/pull/913 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile cc prototool xml | | uname | Linux ea827aee166e 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev/phoenix-personality.sh | | git revision | 4.x / 4b272b1 | | Default Java | Private Build-1.8.0_242-8u
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[ https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17249308#comment-17249308 ] ASF GitHub Bot commented on PHOENIX-5435: - gjacoby126 commented on a change in pull request #913: URL: https://github.com/apache/phoenix/pull/913#discussion_r542791748 ## File path: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/generated/ServerCachingProtos.java ## @@ -2192,6 +2192,21 @@ public Builder setColumnNameBytes( */ com.google.protobuf.ByteString getParentTableTypeBytes(); + +// optional string logicalIndexName = 25; +/** + * optional string logicalIndexName = 25; + */ +boolean hasLogicalIndexName(); +/** + * optional string logicalIndexName = 25; + */ +java.lang.String getLogicalIndexName(); Review comment: @gokceni Correct, should be populated for all indexes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Annotate HBase WALs with Phoenix Metadata > - > > Key: PHOENIX-5435 > URL: https://issues.apache.org/jira/browse/PHOENIX-5435 > Project: Phoenix > Issue Type: Sub-task >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > Attachments: PHOENIX-5435-4.x.patch > > > HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase > replication and some HBase backup frameworks. The WALs contain HBase-level > metadata such as table and region, but lack Phoenix-level metadata. That > means that it's quite difficult to build correct logic that needs to know > about Phoenix-level constructs such as multi-tenancy, views, or indexes. > HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate > extra key/value pairs of metadata into the HBase WAL. We should have the > option to annotate the tuple , or > some hashed way to reconstruct that tuple into the WAL. It should have a > feature toggle so operators who don't need it don't bear the slight extra > storage cost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[ https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17249307#comment-17249307 ] ASF GitHub Bot commented on PHOENIX-5435: - gjacoby126 commented on a change in pull request #913: URL: https://github.com/apache/phoenix/pull/913#discussion_r542791748 ## File path: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/generated/ServerCachingProtos.java ## @@ -2192,6 +2192,21 @@ public Builder setColumnNameBytes( */ com.google.protobuf.ByteString getParentTableTypeBytes(); + +// optional string logicalIndexName = 25; +/** + * optional string logicalIndexName = 25; + */ +boolean hasLogicalIndexName(); +/** + * optional string logicalIndexName = 25; + */ +java.lang.String getLogicalIndexName(); Review comment: Correct, should be populated for all indexes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Annotate HBase WALs with Phoenix Metadata > - > > Key: PHOENIX-5435 > URL: https://issues.apache.org/jira/browse/PHOENIX-5435 > Project: Phoenix > Issue Type: Sub-task >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > Attachments: PHOENIX-5435-4.x.patch > > > HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase > replication and some HBase backup frameworks. The WALs contain HBase-level > metadata such as table and region, but lack Phoenix-level metadata. That > means that it's quite difficult to build correct logic that needs to know > about Phoenix-level constructs such as multi-tenancy, views, or indexes. > HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate > extra key/value pairs of metadata into the HBase WAL. We should have the > option to annotate the tuple , or > some hashed way to reconstruct that tuple into the WAL. It should have a > feature toggle so operators who don't need it don't bear the slight extra > storage cost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[ https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17249675#comment-17249675 ] ASF GitHub Bot commented on PHOENIX-5435: - virajjasani commented on a change in pull request #913: URL: https://github.com/apache/phoenix/pull/913#discussion_r543322994 ## File path: pom.xml ## @@ -1173,7 +1173,7 @@ - phoenix-hbase-compat-1.3.0-default + phoenix-hbase-compat-1.5.0-default Review comment: Since 4.x multibranch keep running all tests with all supported versions including 1.5, pom.xml changes will be reverted before merging this PR? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Annotate HBase WALs with Phoenix Metadata > - > > Key: PHOENIX-5435 > URL: https://issues.apache.org/jira/browse/PHOENIX-5435 > Project: Phoenix > Issue Type: Sub-task >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > Attachments: PHOENIX-5435-4.x.patch > > > HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase > replication and some HBase backup frameworks. The WALs contain HBase-level > metadata such as table and region, but lack Phoenix-level metadata. That > means that it's quite difficult to build correct logic that needs to know > about Phoenix-level constructs such as multi-tenancy, views, or indexes. > HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate > extra key/value pairs of metadata into the HBase WAL. We should have the > option to annotate the tuple , or > some hashed way to reconstruct that tuple into the WAL. It should have a > feature toggle so operators who don't need it don't bear the slight extra > storage cost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[ https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17249795#comment-17249795 ] ASF GitHub Bot commented on PHOENIX-5435: - gokceni commented on pull request #913: URL: https://github.com/apache/phoenix/pull/913#issuecomment-745426803 @gjacoby126 +1 Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Annotate HBase WALs with Phoenix Metadata > - > > Key: PHOENIX-5435 > URL: https://issues.apache.org/jira/browse/PHOENIX-5435 > Project: Phoenix > Issue Type: Sub-task >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > Attachments: PHOENIX-5435-4.x.patch > > > HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase > replication and some HBase backup frameworks. The WALs contain HBase-level > metadata such as table and region, but lack Phoenix-level metadata. That > means that it's quite difficult to build correct logic that needs to know > about Phoenix-level constructs such as multi-tenancy, views, or indexes. > HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate > extra key/value pairs of metadata into the HBase WAL. We should have the > option to annotate the tuple , or > some hashed way to reconstruct that tuple into the WAL. It should have a > feature toggle so operators who don't need it don't bear the slight extra > storage cost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[ https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17249796#comment-17249796 ] ASF GitHub Bot commented on PHOENIX-5435: - gjacoby126 commented on pull request #913: URL: https://github.com/apache/phoenix/pull/913#issuecomment-745429251 Thanks for the reviews, @ChinmaySKulkarni @gokceni and @virajjasani ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Annotate HBase WALs with Phoenix Metadata > - > > Key: PHOENIX-5435 > URL: https://issues.apache.org/jira/browse/PHOENIX-5435 > Project: Phoenix > Issue Type: Sub-task >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > Attachments: PHOENIX-5435-4.x.patch > > > HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase > replication and some HBase backup frameworks. The WALs contain HBase-level > metadata such as table and region, but lack Phoenix-level metadata. That > means that it's quite difficult to build correct logic that needs to know > about Phoenix-level constructs such as multi-tenancy, views, or indexes. > HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate > extra key/value pairs of metadata into the HBase WAL. We should have the > option to annotate the tuple , or > some hashed way to reconstruct that tuple into the WAL. It should have a > feature toggle so operators who don't need it don't bear the slight extra > storage cost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[ https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16904271#comment-16904271 ] Andrew Purtell commented on PHOENIX-5435: - bq. It should have a feature toggle so operators who don't need it don't bear the slight extra storage cost. Not arguing with the idea. Slight is probably the right characterization, will be interesting to see how that bears out. I've found the HBase WAL is already really inefficient as a storage format - that's not its intent - but that can be mitigated with WAL compression if you turn it on. For example if I Avro encode a 100 column row (where the data in cells is 8 byte longs, randomly set) the per record encoding is 100x more efficient in Avro form than the WAL. That would drop considerably with WAL compression enabled although I did not measure it. > Annotate HBase WALs with Phoenix Metadata > - > > Key: PHOENIX-5435 > URL: https://issues.apache.org/jira/browse/PHOENIX-5435 > Project: Phoenix > Issue Type: New Feature >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > > HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase > replication and some HBase backup frameworks. The WALs contain HBase-level > metadata such as table and region, but lack Phoenix-level metadata. That > means that it's quite difficult to build correct logic that needs to know > about Phoenix-level constructs such as multi-tenancy, views, or indexes. > HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate > extra key/value pairs of metadata into the HBase WAL. We should have the > option to annotate the tuple , or > some hashed way to reconstruct that tuple into the WAL. It should have a > feature toggle so operators who don't need it don't bear the slight extra > storage cost. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[ https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17250154#comment-17250154 ] Istvan Toth commented on PHOENIX-5435: -- [~gjacoby] , you have switched the default HBase profile to 1.5 with this patch. We're using the oldest supported HBase version by default to maximise the compatibility of the binary release and the public maven artifacts, and this change means that neither will work with 1.4 or 1.5. Was this your intention, or is this just a development setting that accidentally got committed ? > Annotate HBase WALs with Phoenix Metadata > - > > Key: PHOENIX-5435 > URL: https://issues.apache.org/jira/browse/PHOENIX-5435 > Project: Phoenix > Issue Type: Sub-task >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > Attachments: PHOENIX-5435-4.x.patch > > > HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase > replication and some HBase backup frameworks. The WALs contain HBase-level > metadata such as table and region, but lack Phoenix-level metadata. That > means that it's quite difficult to build correct logic that needs to know > about Phoenix-level constructs such as multi-tenancy, views, or indexes. > HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate > extra key/value pairs of metadata into the HBase WAL. We should have the > option to annotate the tuple , or > some hashed way to reconstruct that tuple into the WAL. It should have a > feature toggle so operators who don't need it don't bear the slight extra > storage cost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[ https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17250817#comment-17250817 ] Geoffrey Jacoby commented on PHOENIX-5435: -- [~stoty] - yes, switching to default 1.5 was my intention. Been meaning to bring up a discussion about this on the dev list, which I hope to do tomorrow, to evaluate options. The problem with defaulting to the lowest HBase version when doing test runs (either locally or in Jenkins) is that there are now quite a few tests that don't get exercised on earlier versions of HBase. (This is particularly true on the master branch, where many secondary index tests no-op when run against HBase 2.1 or 2.2.) I made the switch as part of this JIRA because WAL annotations are only supported in newer versions of branch-1 (1.5+ I think? and only 2.3 and up when I port to master in the next few days), so the tests would never get executed at default settings. > Annotate HBase WALs with Phoenix Metadata > - > > Key: PHOENIX-5435 > URL: https://issues.apache.org/jira/browse/PHOENIX-5435 > Project: Phoenix > Issue Type: Sub-task >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > Fix For: 5.1.0, 4.16.0 > > Attachments: PHOENIX-5435-4.x.patch > > > HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase > replication and some HBase backup frameworks. The WALs contain HBase-level > metadata such as table and region, but lack Phoenix-level metadata. That > means that it's quite difficult to build correct logic that needs to know > about Phoenix-level constructs such as multi-tenancy, views, or indexes. > HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate > extra key/value pairs of metadata into the HBase WAL. We should have the > option to annotate the tuple , or > some hashed way to reconstruct that tuple into the WAL. It should have a > feature toggle so operators who don't need it don't bear the slight extra > storage cost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[ https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17253171#comment-17253171 ] ASF GitHub Bot commented on PHOENIX-5435: - gjacoby126 opened a new pull request #1030: URL: https://github.com/apache/phoenix/pull/1030 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Annotate HBase WALs with Phoenix Metadata > - > > Key: PHOENIX-5435 > URL: https://issues.apache.org/jira/browse/PHOENIX-5435 > Project: Phoenix > Issue Type: Sub-task >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > Fix For: 5.1.0, 4.16.0 > > Attachments: PHOENIX-5435-4.x.patch > > > HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase > replication and some HBase backup frameworks. The WALs contain HBase-level > metadata such as table and region, but lack Phoenix-level metadata. That > means that it's quite difficult to build correct logic that needs to know > about Phoenix-level constructs such as multi-tenancy, views, or indexes. > HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate > extra key/value pairs of metadata into the HBase WAL. We should have the > option to annotate the tuple , or > some hashed way to reconstruct that tuple into the WAL. It should have a > feature toggle so operators who don't need it don't bear the slight extra > storage cost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[ https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17253242#comment-17253242 ] ASF GitHub Bot commented on PHOENIX-5435: - stoty commented on pull request #1030: URL: https://github.com/apache/phoenix/pull/1030#issuecomment-749314328 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 5m 43s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 4m 55s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 9m 57s | master passed | | +1 :green_heart: | compile | 1m 17s | master passed | | +1 :green_heart: | checkstyle | 10m 2s | master passed | | +1 :green_heart: | javadoc | 2m 29s | master passed | | +0 :ok: | spotbugs | 0m 30s | phoenix-hbase-compat-2.3.0 in master has 1 extant spotbugs warnings. | | +0 :ok: | spotbugs | 0m 26s | phoenix-hbase-compat-2.2.1 in master has 1 extant spotbugs warnings. | | +0 :ok: | spotbugs | 0m 28s | phoenix-hbase-compat-2.1.6 in master has 1 extant spotbugs warnings. | | +0 :ok: | spotbugs | 2m 53s | phoenix-core in master has 972 extant spotbugs warnings. | | +0 :ok: | spotbugs | 3m 42s | root in master has 1021 extant spotbugs warnings. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 9s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 7m 24s | the patch passed | | +1 :green_heart: | compile | 1m 15s | the patch passed | | +1 :green_heart: | cc | 1m 15s | the patch passed | | +1 :green_heart: | javac | 1m 15s | the patch passed | | -1 :x: | checkstyle | 10m 21s | root: The patch generated 552 new + 22558 unchanged - 232 fixed = 23110 total (was 22790) | | +1 :green_heart: | prototool | 0m 2s | There were no new prototool issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | javadoc | 2m 28s | the patch passed | | -1 :x: | spotbugs | 3m 10s | phoenix-core generated 1 new + 972 unchanged - 0 fixed = 973 total (was 972) | | -1 :x: | spotbugs | 4m 6s | root generated 1 new + 1021 unchanged - 0 fixed = 1022 total (was 1021) | ||| _ Other Tests _ | | -1 :x: | unit | 106m 27s | root in the patch failed. | | -1 :x: | asflicense | 1m 53s | The patch generated 1 ASF License warnings. | | | | 185m 54s | | | Reason | Tests | |---:|:--| | FindBugs | module:phoenix-core | | | org.apache.phoenix.jdbc.PhoenixDatabaseMetaData.CHANGE_DETECTION_ENABLED_BYTES is a mutable array At MetaDataEndpointImpl.java: At MetaDataEndpointImpl.java:[line 343] | | FindBugs | module:root | | | org.apache.phoenix.jdbc.PhoenixDatabaseMetaData.CHANGE_DETECTION_ENABLED_BYTES is a mutable array At MetaDataEndpointImpl.java: At MetaDataEndpointImpl.java:[line 343] | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1030/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/phoenix/pull/1030 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile cc prototool | | uname | Linux 5d600763565e 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev/phoenix-personality.sh | | git revision | master / 1dcb749 | | Default Java | Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08 | | checkstyle | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1030/1/artifact/yetus-general-check/output/diff-checkstyle-root.txt | | spotbugs | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1030/1/artifact/yetus-general-check/output/new-spotbugs-phoenix-core.html | | spotbugs | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1030/1/artifact/yetus-general-check/output/new-spotbugs-root.html | | unit | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1030/1/artifact/yetus-general-check/output/pa
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[ https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17253647#comment-17253647 ] ASF GitHub Bot commented on PHOENIX-5435: - gjacoby126 commented on pull request #1030: URL: https://github.com/apache/phoenix/pull/1030#issuecomment-749679998 Merging as this is a port of a previously reviewed PR (#913 ) to the master branch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Annotate HBase WALs with Phoenix Metadata > - > > Key: PHOENIX-5435 > URL: https://issues.apache.org/jira/browse/PHOENIX-5435 > Project: Phoenix > Issue Type: Sub-task >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > Fix For: 5.1.0, 4.16.0 > > Attachments: PHOENIX-5435-4.x.patch > > > HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase > replication and some HBase backup frameworks. The WALs contain HBase-level > metadata such as table and region, but lack Phoenix-level metadata. That > means that it's quite difficult to build correct logic that needs to know > about Phoenix-level constructs such as multi-tenancy, views, or indexes. > HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate > extra key/value pairs of metadata into the HBase WAL. We should have the > option to annotate the tuple , or > some hashed way to reconstruct that tuple into the WAL. It should have a > feature toggle so operators who don't need it don't bear the slight extra > storage cost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[ https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17253648#comment-17253648 ] ASF GitHub Bot commented on PHOENIX-5435: - gjacoby126 merged pull request #1030: URL: https://github.com/apache/phoenix/pull/1030 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Annotate HBase WALs with Phoenix Metadata > - > > Key: PHOENIX-5435 > URL: https://issues.apache.org/jira/browse/PHOENIX-5435 > Project: Phoenix > Issue Type: Sub-task >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > Fix For: 5.1.0, 4.16.0 > > Attachments: PHOENIX-5435-4.x.patch > > > HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase > replication and some HBase backup frameworks. The WALs contain HBase-level > metadata such as table and region, but lack Phoenix-level metadata. That > means that it's quite difficult to build correct logic that needs to know > about Phoenix-level constructs such as multi-tenancy, views, or indexes. > HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate > extra key/value pairs of metadata into the HBase WAL. We should have the > option to annotate the tuple , or > some hashed way to reconstruct that tuple into the WAL. It should have a > feature toggle so operators who don't need it don't bear the slight extra > storage cost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
