[jira] [Updated] (PHOENIX-3383) Comparison between descending row keys used in RVC is reverse
[ https://issues.apache.org/jira/browse/PHOENIX-3383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated PHOENIX-3383: -- Attachment: (was: PHOENIX-3383_v13.patch) > Comparison between descending row keys used in RVC is reverse > - > > Key: PHOENIX-3383 > URL: https://issues.apache.org/jira/browse/PHOENIX-3383 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: James Taylor >Priority: Major > Labels: DESC > Fix For: 4.15.0 > > Attachments: PHOENIX-3383-wip1.patch, PHOENIX-3383-wip5.patch, > PHOENIX-3383-wip6.patch, PHOENIX-3383-wip7.patch, PHOENIX-3383_v1.patch, > PHOENIX-3383_v10.patch, PHOENIX-3383_v11.patch, PHOENIX-3383_v12.patch, > PHOENIX-3383_v2.patch, PHOENIX-3383_v3.patch, PHOENIX-3383_v4.patch, > PHOENIX-3383_v5.patch, PHOENIX-3383_v6.patch, PHOENIX-3383_v7.patch, > PHOENIX-3383_v8.patch, PHOENIX-3383_v9.patch, PHOENIX-3383_wip.patch, > PHOENIX-3383_wip2.patch, PHOENIX-3383_wip3.patch > > > See PHOENIX-3382, but the comparison for RVC with descending row key columns > is the reverse of what it should be. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PHOENIX-3383) Comparison between descending row keys used in RVC is reverse
[ https://issues.apache.org/jira/browse/PHOENIX-3383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated PHOENIX-3383: -- Attachment: PHOENIX-3383_v13.patch > Comparison between descending row keys used in RVC is reverse > - > > Key: PHOENIX-3383 > URL: https://issues.apache.org/jira/browse/PHOENIX-3383 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: James Taylor >Priority: Major > Labels: DESC > Fix For: 4.15.0 > > Attachments: PHOENIX-3383-wip1.patch, PHOENIX-3383-wip5.patch, > PHOENIX-3383-wip6.patch, PHOENIX-3383-wip7.patch, PHOENIX-3383_v1.patch, > PHOENIX-3383_v10.patch, PHOENIX-3383_v11.patch, PHOENIX-3383_v12.patch, > PHOENIX-3383_v13.patch, PHOENIX-3383_v2.patch, PHOENIX-3383_v3.patch, > PHOENIX-3383_v4.patch, PHOENIX-3383_v5.patch, PHOENIX-3383_v6.patch, > PHOENIX-3383_v7.patch, PHOENIX-3383_v8.patch, PHOENIX-3383_v9.patch, > PHOENIX-3383_wip.patch, PHOENIX-3383_wip2.patch, PHOENIX-3383_wip3.patch > > > See PHOENIX-3382, but the comparison for RVC with descending row key columns > is the reverse of what it should be. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-3383) Comparison between descending row keys used in RVC is reverse
[ https://issues.apache.org/jira/browse/PHOENIX-3383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540165#comment-16540165 ] James Taylor commented on PHOENIX-3383: --- Thanks for the feedback, [~tdsilva]. I've incorporated that and fixed the test failures with v13 patch: [https://builds.apache.org/job/PreCommit-PHOENIX-Build/1921/.|https://builds.apache.org/job/PreCommit-PHOENIX-Build/1921/] Will go ahead and commit that version. > Comparison between descending row keys used in RVC is reverse > - > > Key: PHOENIX-3383 > URL: https://issues.apache.org/jira/browse/PHOENIX-3383 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: James Taylor >Priority: Major > Labels: DESC > Fix For: 4.15.0 > > Attachments: PHOENIX-3383-wip1.patch, PHOENIX-3383-wip5.patch, > PHOENIX-3383-wip6.patch, PHOENIX-3383-wip7.patch, PHOENIX-3383_v1.patch, > PHOENIX-3383_v10.patch, PHOENIX-3383_v11.patch, PHOENIX-3383_v12.patch, > PHOENIX-3383_v13.patch, PHOENIX-3383_v2.patch, PHOENIX-3383_v3.patch, > PHOENIX-3383_v4.patch, PHOENIX-3383_v5.patch, PHOENIX-3383_v6.patch, > PHOENIX-3383_v7.patch, PHOENIX-3383_v8.patch, PHOENIX-3383_v9.patch, > PHOENIX-3383_wip.patch, PHOENIX-3383_wip2.patch, PHOENIX-3383_wip3.patch > > > See PHOENIX-3382, but the comparison for RVC with descending row key columns > is the reverse of what it should be. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] phoenix pull request #307: PHOENIX-4688 Kerberize python phoenixdb
Github user pu239ppy commented on a diff in the pull request: https://github.com/apache/phoenix/pull/307#discussion_r201716375 --- Diff: python/phoenixdb-module/phoenixdb/__init__.py --- @@ -1,11 +1,10 @@ -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at +# Copyright 2015 Lukas Lalinsky --- End diff -- I am not sure how that went back in, it is possible that I may have copied __init__.py from the time I was doing this work on my own before I found out that this has been moved to phoenix. I will change the header ---
[jira] [Commented] (PHOENIX-4688) Add kerberos authentication to python-phoenixdb
[ https://issues.apache.org/jira/browse/PHOENIX-4688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540181#comment-16540181 ] ASF GitHub Bot commented on PHOENIX-4688: - Github user pu239ppy commented on a diff in the pull request: https://github.com/apache/phoenix/pull/307#discussion_r201716375 --- Diff: python/phoenixdb-module/phoenixdb/__init__.py --- @@ -1,11 +1,10 @@ -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at +# Copyright 2015 Lukas Lalinsky --- End diff -- I am not sure how that went back in, it is possible that I may have copied __init__.py from the time I was doing this work on my own before I found out that this has been moved to phoenix. I will change the header > Add kerberos authentication to python-phoenixdb > --- > > Key: PHOENIX-4688 > URL: https://issues.apache.org/jira/browse/PHOENIX-4688 > Project: Phoenix > Issue Type: Improvement >Reporter: Lev Bronshtein >Priority: Minor > > In its current state python-phoenixdv does not support support kerberos > authentication. Using a modern python http library such as requests or > urllib it would be simple (if not trivial) to add this support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] phoenix pull request #307: PHOENIX-4688 Kerberize python phoenixdb
Github user pu239ppy commented on a diff in the pull request: https://github.com/apache/phoenix/pull/307#discussion_r201717820 --- Diff: phoenix-queryserver/src/it/java/org/apache/phoenix/end2end/SecureQueryServerPhoenixDBIT.java --- @@ -0,0 +1,423 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to you under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.phoenix.end2end; + +import com.google.common.base.Preconditions; +import com.google.common.collect.Maps; +import org.apache.commons.io.FileUtils; +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.LocalHBaseCluster; +import org.apache.hadoop.hbase.coprocessor.CoprocessorHost; +import org.apache.hadoop.hbase.http.ssl.KeyStoreTestUtil; +import org.apache.hadoop.hbase.security.HBaseKerberosUtils; +import org.apache.hadoop.hbase.security.token.TokenProvider; +import org.apache.hadoop.hbase.util.FSUtils; +import org.apache.hadoop.hdfs.DFSConfigKeys; +import org.apache.hadoop.http.HttpConfig; +import org.apache.hadoop.minikdc.MiniKdc; +import org.apache.hadoop.security.UserGroupInformation; +import org.apache.hadoop.security.authentication.util.KerberosName; +import org.apache.phoenix.query.ConfigurationFactory; +import org.apache.phoenix.query.QueryServices; +import org.apache.phoenix.queryserver.client.ThinClientUtil; +import org.apache.phoenix.queryserver.server.QueryServer; +import org.apache.phoenix.util.InstanceResolver; +import org.junit.AfterClass; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +import java.io.*; +import java.lang.reflect.Field; +import java.security.PrivilegedAction; +import java.security.PrivilegedExceptionAction; +import java.sql.DriverManager; +import java.sql.ResultSet; +import java.sql.Statement; +import java.util.ArrayList; +import java.util.List; +import java.util.Map.Entry; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.TimeUnit; + +import java.nio.file.Paths; +import java.util.Map; + +import static org.junit.Assert.*; + +@Category(NeedsOwnMiniClusterTest.class) +public class SecureQueryServerPhoenixDBIT { +private static final Log LOG = LogFactory.getLog(SecureQueryServerPhoenixDBIT.class); + +private static final File TEMP_DIR = new File(getTempDirForClass()); +private static final File KEYTAB_DIR = new File(TEMP_DIR, "keytabs"); +private static final List USER_KEYTAB_FILES = new ArrayList<>(); + +private static final String SPNEGO_PRINCIPAL = "HTTP/localhost"; +private static final String PQS_PRINCIPAL = "phoenixqs/localhost"; +private static final String SERVICE_PRINCIPAL = "securecluster/localhost"; +private static File KEYTAB; + +private static MiniKdc KDC; +private static HBaseTestingUtility UTIL = new HBaseTestingUtility(); +private static LocalHBaseCluster HBASE_CLUSTER; +private static int NUM_CREATED_USERS; + +private static ExecutorService PQS_EXECUTOR; +private static QueryServer PQS; +private static int PQS_PORT; +private static String PQS_URL; + +private static String getTempDirForClass() { +StringBuilder sb = new StringBuilder(32); +sb.append(System.getProperty("user.dir")).append(File.separator); +sb.append("target").append(File.separator); +sb.append(SecureQueryServerPhoenixDBIT.class.getSimpleName()); +return sb.toString(); +} + +private static void updateDefaultRealm() throws Exception { +// (at least) one other phoenix test triggers the caching of this field before the KDC is up +
[jira] [Updated] (PHOENIX-4790) Simplify check for client side delete
[ https://issues.apache.org/jira/browse/PHOENIX-4790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated PHOENIX-4790: -- Fix Version/s: 5.1.0 4.15.0 > Simplify check for client side delete > - > > Key: PHOENIX-4790 > URL: https://issues.apache.org/jira/browse/PHOENIX-4790 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: James Taylor >Priority: Minor > Fix For: 4.15.0, 5.1.0 > > Attachments: PHOENIX-4790.patch > > > We don't need to check every query plan for the existence of a where clause > to determine if we can do a client-side delete. Instead, we can simply check > if any non PK columns are projected. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (PHOENIX-4790) Simplify check for client side delete
[ https://issues.apache.org/jira/browse/PHOENIX-4790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor resolved PHOENIX-4790. --- Resolution: Fixed > Simplify check for client side delete > - > > Key: PHOENIX-4790 > URL: https://issues.apache.org/jira/browse/PHOENIX-4790 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: James Taylor >Priority: Minor > Fix For: 4.15.0, 5.1.0 > > Attachments: PHOENIX-4790.patch > > > We don't need to check every query plan for the existence of a where clause > to determine if we can do a client-side delete. Instead, we can simply check > if any non PK columns are projected. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-4688) Add kerberos authentication to python-phoenixdb
[ https://issues.apache.org/jira/browse/PHOENIX-4688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540185#comment-16540185 ] ASF GitHub Bot commented on PHOENIX-4688: - Github user pu239ppy commented on a diff in the pull request: https://github.com/apache/phoenix/pull/307#discussion_r201717820 --- Diff: phoenix-queryserver/src/it/java/org/apache/phoenix/end2end/SecureQueryServerPhoenixDBIT.java --- @@ -0,0 +1,423 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to you under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.phoenix.end2end; + +import com.google.common.base.Preconditions; +import com.google.common.collect.Maps; +import org.apache.commons.io.FileUtils; +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.LocalHBaseCluster; +import org.apache.hadoop.hbase.coprocessor.CoprocessorHost; +import org.apache.hadoop.hbase.http.ssl.KeyStoreTestUtil; +import org.apache.hadoop.hbase.security.HBaseKerberosUtils; +import org.apache.hadoop.hbase.security.token.TokenProvider; +import org.apache.hadoop.hbase.util.FSUtils; +import org.apache.hadoop.hdfs.DFSConfigKeys; +import org.apache.hadoop.http.HttpConfig; +import org.apache.hadoop.minikdc.MiniKdc; +import org.apache.hadoop.security.UserGroupInformation; +import org.apache.hadoop.security.authentication.util.KerberosName; +import org.apache.phoenix.query.ConfigurationFactory; +import org.apache.phoenix.query.QueryServices; +import org.apache.phoenix.queryserver.client.ThinClientUtil; +import org.apache.phoenix.queryserver.server.QueryServer; +import org.apache.phoenix.util.InstanceResolver; +import org.junit.AfterClass; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +import java.io.*; +import java.lang.reflect.Field; +import java.security.PrivilegedAction; +import java.security.PrivilegedExceptionAction; +import java.sql.DriverManager; +import java.sql.ResultSet; +import java.sql.Statement; +import java.util.ArrayList; +import java.util.List; +import java.util.Map.Entry; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.TimeUnit; + +import java.nio.file.Paths; +import java.util.Map; + +import static org.junit.Assert.*; + +@Category(NeedsOwnMiniClusterTest.class) +public class SecureQueryServerPhoenixDBIT { +private static final Log LOG = LogFactory.getLog(SecureQueryServerPhoenixDBIT.class); + +private static final File TEMP_DIR = new File(getTempDirForClass()); +private static final File KEYTAB_DIR = new File(TEMP_DIR, "keytabs"); +private static final List USER_KEYTAB_FILES = new ArrayList<>(); + +private static final String SPNEGO_PRINCIPAL = "HTTP/localhost"; +private static final String PQS_PRINCIPAL = "phoenixqs/localhost"; +private static final String SERVICE_PRINCIPAL = "securecluster/localhost"; +private static File KEYTAB; + +private static MiniKdc KDC; +private static HBaseTestingUtility UTIL = new HBaseTestingUtility(); +private static LocalHBaseCluster HBASE_CLUSTER; +private static int NUM_CREATED_USERS; + +private static ExecutorService PQS_EXECUTOR; +private static QueryServer PQS; +private static int PQS_PORT; +private static String PQS_URL; + +private static String getTempDirForClass() { +StringBuilder sb = new StringBuilder(32); +sb.append(System.getProperty("user.dir")).append(File.separator); +sb.append("target").append(File.separator); +sb.append(SecureQueryServerPhoenixDBIT.
Re: Help: setting hbase row timestamp in phoenix upserts ?
I think the answer is PHOENIX-4552. There's an outline of the work involved on the JIRA. I think passing through data like that for hints would get unwieldy quickly. On Tue, Jul 10, 2018 at 1:31 PM, Pedro Boado wrote: > Hi guys, just a refloat from the @user list. > > May it be of interest having this functionality for defining HBase > timestamps in a per row basis as part of an UPSERT VALUES? > > For a table defined as > CREATE TABLE T0001 ( k VARCHAR PRIMARY KEY, v INTEGER) > > Allow a hint to extract and override hbase put timestamp through a > "virtual" column? > UPSERT /*+ ROW_TIMESTAMP(ts) */ INTO T0001(k,v,ts) VALUES > ('a',1, 1531253959043) > > If the column existed and had appropiate type it would also be populated > with the same value. > > Thanks, > Pedro. > > > On Fri, 1 Dec 2017 at 07:15, James Taylor wrote: > > > The only way I can think of accomplishing this is by using the raw HBase > > APIs to write the data but using our utilities to write it in a Phoenix > > compatible manner. For example, you could run an UPSERT VALUES statement, > > use the PhoenixRuntime.getUncommittedDataIterator()method to get the > Cells > > that would have been written, update the Cell timestamp as needed, and do > > an htable.batch() call to commit them. > > > > On Wed, Nov 29, 2017 at 11:46 AM Pedro Boado > > wrote: > > > >> Hi, > >> > >> I'm looking for a little bit of help trying to get some light over > >> ROW_TIMESTAMP. > >> > >> Some background over the problem ( simplified ) : I'm working in a > >> project that needs to create a "enriched" replica of a RBDMS table > based on > >> a stream of cdc changes off that table. > >> > >> Each cdc event contains the timestamp of the change plus all the column > >> values 'before' and 'after' the change . And each event is pushed to a > >> kafka topic. Because of certain "non-negotiable" design decisions kafka > >> guarantees delivering each event at least once, but doesn't guarantee > >> ordering for changes over the same row in the source table. > >> > >> The final step of the kafka-based flow is sinking the information into > >> HBase/Phoenix. > >> > >> As I cannot get in order delivery guarantee from Kafka I need to use the > >> cdc event timestamp to ensure that HBase keeps the latest change over a > row. > >> > >> This fits perfectly well with an HBase table design with VERSIONS=1 and > >> using the source event timestamp as HBase row/cells timestamp > >> > >> The thing is that I cannot find a way to define the value of the HBase > >> cell from a Phoenix upsert. > >> > >> I came across the ROW_TIMESTAMP functionality, but I've just found ( I'm > >> devastated now ) that the ROW_TIMESTAMP columns store the date in both > >> hbase's cell timestamp and in the primary key, meaning that I cannot > >> leverage that functionality to keep only the latest change. > >> > >> Is there a way of defining hbase's row timestamp when doing the UPSERT - > >> even by setting it through some obscure hidden jdbc property - ? > >> > >> I want to avoid by all means doing a checkAndPut as the volume of > changes > >> is going to be quite bug. > >> > >> > >> > >> -- > >> Un saludo. > >> Pedro Boado. > >> > > > > -- > Un saludo. > Pedro Boado. >
Re: Help: setting hbase row timestamp in phoenix upserts ?
I hadn't seen this Jira. Yes that is essentially it. On Wed, 11 Jul 2018, 15:49 James Taylor, wrote: > I think the answer is PHOENIX-4552. There's an outline of the work involved > on the JIRA. I think passing through data like that for hints would get > unwieldy quickly. > > On Tue, Jul 10, 2018 at 1:31 PM, Pedro Boado > wrote: > > > Hi guys, just a refloat from the @user list. > > > > May it be of interest having this functionality for defining HBase > > timestamps in a per row basis as part of an UPSERT VALUES? > > > > For a table defined as > > CREATE TABLE T0001 ( k VARCHAR PRIMARY KEY, v INTEGER) > > > > Allow a hint to extract and override hbase put timestamp through a > > "virtual" column? > > UPSERT /*+ ROW_TIMESTAMP(ts) */ INTO T0001(k,v,ts) VALUES > > ('a',1, 1531253959043) > > > > If the column existed and had appropiate type it would also be populated > > with the same value. > > > > Thanks, > > Pedro. > > > > > > On Fri, 1 Dec 2017 at 07:15, James Taylor > wrote: > > > > > The only way I can think of accomplishing this is by using the raw > HBase > > > APIs to write the data but using our utilities to write it in a Phoenix > > > compatible manner. For example, you could run an UPSERT VALUES > statement, > > > use the PhoenixRuntime.getUncommittedDataIterator()method to get the > > Cells > > > that would have been written, update the Cell timestamp as needed, and > do > > > an htable.batch() call to commit them. > > > > > > On Wed, Nov 29, 2017 at 11:46 AM Pedro Boado > > > wrote: > > > > > >> Hi, > > >> > > >> I'm looking for a little bit of help trying to get some light over > > >> ROW_TIMESTAMP. > > >> > > >> Some background over the problem ( simplified ) : I'm working in a > > >> project that needs to create a "enriched" replica of a RBDMS table > > based on > > >> a stream of cdc changes off that table. > > >> > > >> Each cdc event contains the timestamp of the change plus all the > column > > >> values 'before' and 'after' the change . And each event is pushed to a > > >> kafka topic. Because of certain "non-negotiable" design decisions > kafka > > >> guarantees delivering each event at least once, but doesn't guarantee > > >> ordering for changes over the same row in the source table. > > >> > > >> The final step of the kafka-based flow is sinking the information into > > >> HBase/Phoenix. > > >> > > >> As I cannot get in order delivery guarantee from Kafka I need to use > the > > >> cdc event timestamp to ensure that HBase keeps the latest change over > a > > row. > > >> > > >> This fits perfectly well with an HBase table design with VERSIONS=1 > and > > >> using the source event timestamp as HBase row/cells timestamp > > >> > > >> The thing is that I cannot find a way to define the value of the HBase > > >> cell from a Phoenix upsert. > > >> > > >> I came across the ROW_TIMESTAMP functionality, but I've just found ( > I'm > > >> devastated now ) that the ROW_TIMESTAMP columns store the date in both > > >> hbase's cell timestamp and in the primary key, meaning that I cannot > > >> leverage that functionality to keep only the latest change. > > >> > > >> Is there a way of defining hbase's row timestamp when doing the > UPSERT - > > >> even by setting it through some obscure hidden jdbc property - ? > > >> > > >> I want to avoid by all means doing a checkAndPut as the volume of > > changes > > >> is going to be quite bug. > > >> > > >> > > >> > > >> -- > > >> Un saludo. > > >> Pedro Boado. > > >> > > > > > > > -- > > Un saludo. > > Pedro Boado. > > >
[jira] [Commented] (PHOENIX-3955) Ensure KEEP_DELETED_CELLS, REPLICATION_SCOPE, and TTL properties stay in sync between the physical data table and index tables
[ https://issues.apache.org/jira/browse/PHOENIX-3955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540238#comment-16540238 ] James Taylor commented on PHOENIX-3955: --- You've made great progress on this one, [~ckulkarni]. Will you be able to see this one through? It's pretty high on the prioritized list here: https://docs.google.com/document/d/16BWU73qlnCQdxCflM3LScyIxE_OGi76ZYT2WHUfewiE > Ensure KEEP_DELETED_CELLS, REPLICATION_SCOPE, and TTL properties stay in sync > between the physical data table and index tables > -- > > Key: PHOENIX-3955 > URL: https://issues.apache.org/jira/browse/PHOENIX-3955 > Project: Phoenix > Issue Type: Bug >Reporter: Samarth Jain >Assignee: Chinmay Kulkarni >Priority: Major > > We need to make sure that indexes inherit the REPLICATION_SCOPE, > KEEP_DELETED_CELLS and TTL properties from the base table. Otherwise we can > run into situations where the data was removed (or not removed) from the data > table but was removed (or not removed) from the index. Or vice-versa. We also > need to make sure that any ALTER TABLE SET TTL or ALTER TABLE SET > KEEP_DELETED_CELLS statements propagate the properties to the indexes too. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (PHOENIX-4587) Store child links in separate table from system catalog
[ https://issues.apache.org/jira/browse/PHOENIX-4587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas D'Silva resolved PHOENIX-4587. - Resolution: Duplicate > Store child links in separate table from system catalog > --- > > Key: PHOENIX-4587 > URL: https://issues.apache.org/jira/browse/PHOENIX-4587 > Project: Phoenix > Issue Type: Improvement >Reporter: James Taylor >Priority: Major > > Because there can be so many child links (in particular from a global table > or view), we should store them in a separate table without a split policy > that keeps them together. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] phoenix issue #307: PHOENIX-4688 Kerberize python phoenixdb
Github user pu239ppy commented on the issue: https://github.com/apache/phoenix/pull/307 @joshelser If you are still unable to get .deactivate to work, remove it from the shell script for now and we can revisit it. Again I am being overly pedantic here, but a the shell exits, we really do not need to clean up any environment. ---
[jira] [Commented] (PHOENIX-4688) Add kerberos authentication to python-phoenixdb
[ https://issues.apache.org/jira/browse/PHOENIX-4688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540255#comment-16540255 ] ASF GitHub Bot commented on PHOENIX-4688: - Github user pu239ppy commented on the issue: https://github.com/apache/phoenix/pull/307 @joshelser If you are still unable to get .deactivate to work, remove it from the shell script for now and we can revisit it. Again I am being overly pedantic here, but a the shell exits, we really do not need to clean up any environment. > Add kerberos authentication to python-phoenixdb > --- > > Key: PHOENIX-4688 > URL: https://issues.apache.org/jira/browse/PHOENIX-4688 > Project: Phoenix > Issue Type: Improvement >Reporter: Lev Bronshtein >Priority: Minor > > In its current state python-phoenixdv does not support support kerberos > authentication. Using a modern python http library such as requests or > urllib it would be simple (if not trivial) to add this support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] phoenix pull request #303: PHOENIX-3534 Support multi region SYSTEM.CATALOG ...
Github user twdsilva commented on a diff in the pull request: https://github.com/apache/phoenix/pull/303#discussion_r201739480 --- Diff: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java --- @@ -1809,180 +2256,97 @@ public void createTable(RpcController controller, CreateTableRequest request, } catch (Throwable t) { logger.error("createTable failed", t); ProtobufUtil.setControllerException(controller, - ServerUtil.createIOException(SchemaUtil.getTableName(schemaName, tableName), t)); +ServerUtil.createIOException(fullTableName, t)); } } + private void dropChildMetadata(byte[] schemaName, byte[] tableName, byte[] tenantIdBytes) + throws IOException, SQLException, ClassNotFoundException { + TableViewFinderResult childViewsResult = new TableViewFinderResult(); + findAllChildViews(tenantIdBytes, schemaName, tableName, childViewsResult); + if (childViewsResult.hasViews()) { + for (TableInfo viewInfo : childViewsResult.getResults()) { + byte[] viewTenantId = viewInfo.getTenantId(); + byte[] viewSchemaName = viewInfo.getSchemaName(); + byte[] viewName = viewInfo.getTableName(); + Properties props = new Properties(); + if (viewTenantId != null && viewTenantId.length != 0) + props.setProperty(PhoenixRuntime.TENANT_ID_ATTRIB, Bytes.toString(viewTenantId)); + try (PhoenixConnection connection = QueryUtil.getConnectionOnServer(env.getConfiguration()) + .unwrap(PhoenixConnection.class)) { + MetaDataClient client = new MetaDataClient(connection); + org.apache.phoenix.parse.TableName viewTableName = org.apache.phoenix.parse.TableName + .create(Bytes.toString(viewSchemaName), Bytes.toString(viewName)); + client.dropTable( + new DropTableStatement(viewTableName, PTableType.VIEW, false, true, true)); + } + } + } + } + private boolean execeededIndexQuota(PTableType tableType, PTable parentTable) { return PTableType.INDEX == tableType && parentTable.getIndexes().size() >= maxIndexesPerTable; } - -private void findAllChildViews(Region region, byte[] tenantId, PTable table, -TableViewFinder result, long clientTimeStamp, int clientVersion) throws IOException, SQLException { -TableViewFinder currResult = findChildViews(region, tenantId, table, clientVersion, false); -result.addResult(currResult); -for (ViewInfo viewInfo : currResult.getViewInfoList()) { -byte[] viewtenantId = viewInfo.getTenantId(); -byte[] viewSchema = viewInfo.getSchemaName(); -byte[] viewTable = viewInfo.getViewName(); -byte[] tableKey = SchemaUtil.getTableKey(viewtenantId, viewSchema, viewTable); -ImmutableBytesPtr cacheKey = new ImmutableBytesPtr(tableKey); -PTable view = loadTable(env, tableKey, cacheKey, clientTimeStamp, clientTimeStamp, clientVersion); -if (view == null) { -logger.warn("Found orphan tenant view row in SYSTEM.CATALOG with tenantId:" -+ Bytes.toString(tenantId) + ", schema:" -+ Bytes.toString(viewSchema) + ", table:" -+ Bytes.toString(viewTable)); -continue; -} -findAllChildViews(region, viewtenantId, view, result, clientTimeStamp, clientVersion); -} -} - -// TODO use child link instead once splittable system catalog (PHOENIX-3534) is implemented -// and we have a separate table for links. -private TableViewFinder findChildViews_deprecated(Region region, byte[] tenantId, PTable table, byte[] linkTypeBytes, boolean stopAfterFirst) throws IOException { -byte[] schemaName = table.getSchemaName().getBytes(); -byte[] tableName = table.getTableName().getBytes(); -boolean isMultiTenant = table.isMultiTenant(); -Scan scan = new Scan(); -// If the table is multi-tenant, we need to check across all tenant_ids, -// so we can't constrain the row key. Otherwise, any views would have -// the same tenantId. -
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540256#comment-16540256 ] ASF GitHub Bot commented on PHOENIX-3534: - Github user twdsilva commented on a diff in the pull request: https://github.com/apache/phoenix/pull/303#discussion_r201739480 --- Diff: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java --- @@ -1809,180 +2256,97 @@ public void createTable(RpcController controller, CreateTableRequest request, } catch (Throwable t) { logger.error("createTable failed", t); ProtobufUtil.setControllerException(controller, - ServerUtil.createIOException(SchemaUtil.getTableName(schemaName, tableName), t)); +ServerUtil.createIOException(fullTableName, t)); } } + private void dropChildMetadata(byte[] schemaName, byte[] tableName, byte[] tenantIdBytes) + throws IOException, SQLException, ClassNotFoundException { + TableViewFinderResult childViewsResult = new TableViewFinderResult(); + findAllChildViews(tenantIdBytes, schemaName, tableName, childViewsResult); + if (childViewsResult.hasViews()) { + for (TableInfo viewInfo : childViewsResult.getResults()) { + byte[] viewTenantId = viewInfo.getTenantId(); + byte[] viewSchemaName = viewInfo.getSchemaName(); + byte[] viewName = viewInfo.getTableName(); + Properties props = new Properties(); + if (viewTenantId != null && viewTenantId.length != 0) + props.setProperty(PhoenixRuntime.TENANT_ID_ATTRIB, Bytes.toString(viewTenantId)); + try (PhoenixConnection connection = QueryUtil.getConnectionOnServer(env.getConfiguration()) + .unwrap(PhoenixConnection.class)) { + MetaDataClient client = new MetaDataClient(connection); + org.apache.phoenix.parse.TableName viewTableName = org.apache.phoenix.parse.TableName + .create(Bytes.toString(viewSchemaName), Bytes.toString(viewName)); + client.dropTable( + new DropTableStatement(viewTableName, PTableType.VIEW, false, true, true)); + } + } + } + } + private boolean execeededIndexQuota(PTableType tableType, PTable parentTable) { return PTableType.INDEX == tableType && parentTable.getIndexes().size() >= maxIndexesPerTable; } - -private void findAllChildViews(Region region, byte[] tenantId, PTable table, -TableViewFinder result, long clientTimeStamp, int clientVersion) throws IOException, SQLException { -TableViewFinder currResult = findChildViews(region, tenantId, table, clientVersion, false); -result.addResult(currResult); -for (ViewInfo viewInfo : currResult.getViewInfoList()) { -byte[] viewtenantId = viewInfo.getTenantId(); -byte[] viewSchema = viewInfo.getSchemaName(); -byte[] viewTable = viewInfo.getViewName(); -byte[] tableKey = SchemaUtil.getTableKey(viewtenantId, viewSchema, viewTable); -ImmutableBytesPtr cacheKey = new ImmutableBytesPtr(tableKey); -PTable view = loadTable(env, tableKey, cacheKey, clientTimeStamp, clientTimeStamp, clientVersion); -if (view == null) { -logger.warn("Found orphan tenant view row in SYSTEM.CATALOG with tenantId:" -+ Bytes.toString(tenantId) + ", schema:" -+ Bytes.toString(viewSchema) + ", table:" -+ Bytes.toString(viewTable)); -continue; -} -findAllChildViews(region, viewtenantId, view, result, clientTimeStamp, clientVersion); -} -} - -// TODO use child link instead once splittable system catalog (PHOENIX-3534) is implemented -// and we have a separate table for links. -private TableViewFinder findChildViews_deprecated(Region region, byte[] tenantId, PTable table, byte[] linkTypeBytes, boolean stopAfterFirst) throws IOException { -byte[] schemaName = table.getSchemaName().getBytes(); -byte[] tableName = table.getTableName().getBytes(); -boolean isMultiTenant = table.isMu
[jira] [Updated] (PHOENIX-4799) Write cells using checkAndMutate to prevent conflicting changes
[ https://issues.apache.org/jira/browse/PHOENIX-4799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas D'Silva updated PHOENIX-4799: Description: In order to prevent race conditions when multiple client try to mutate the same column before sending the request to mutate the column to the server do a checkAndMutate with the column name being added/dropped. Also: 1. When a view is created do a checkAndMutate with the columns in the view where clause. 2. When an index on a view is created do a checkAndMutate with the indexed columns. To prevent a race condition in the DROP TABLE CASCADE case, when a table is dropped do a checkAndMutate with the rowkey of the base table name. If a view is created it also does a checkAndMutate with the same rowkey. was: In order to prevent race conditions when multiple client try to mutate the same column before sending the request to mutate the column to the server do a checkAndMutate with the column name being added/dropped. Also: 1. When a view is created do a checkAndMutate with the columns in the view where clause. 2. When an index on a view is created do a checkAndMutate with the indexed columns. > Write cells using checkAndMutate to prevent conflicting changes > --- > > Key: PHOENIX-4799 > URL: https://issues.apache.org/jira/browse/PHOENIX-4799 > Project: Phoenix > Issue Type: Sub-task >Reporter: Thomas D'Silva >Priority: Major > > In order to prevent race conditions when multiple client try to mutate the > same column before sending the request to mutate the column to the server do > a checkAndMutate with the column name being added/dropped. Also: > 1. When a view is created do a checkAndMutate with the columns in the view > where clause. > 2. When an index on a view is created do a checkAndMutate with the indexed > columns. > > To prevent a race condition in the DROP TABLE CASCADE case, when a table is > dropped do a checkAndMutate with the rowkey of the base table name. If a view > is created it also does a checkAndMutate with the same rowkey. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] phoenix pull request #303: PHOENIX-3534 Support multi region SYSTEM.CATALOG ...
Github user twdsilva commented on a diff in the pull request: https://github.com/apache/phoenix/pull/303#discussion_r201741221 --- Diff: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java --- @@ -2147,46 +2566,29 @@ private MetaDataMutationResult doDropTable(byte[] key, byte[] tenantId, byte[] s } if (tableType == PTableType.TABLE || tableType == PTableType.SYSTEM) { -// Handle any child views that exist -TableViewFinder tableViewFinderResult = findChildViews(region, tenantId, table, clientVersion, !isCascade); -if (tableViewFinderResult.hasViews()) { -if (isCascade) { -if (tableViewFinderResult.allViewsInMultipleRegions()) { -// We don't yet support deleting a table with views where SYSTEM.CATALOG has split and the -// view metadata spans multiple regions -return new MetaDataMutationResult(MutationCode.UNALLOWED_TABLE_MUTATION, - EnvironmentEdgeManager.currentTimeMillis(), null); -} else if (tableViewFinderResult.allViewsInSingleRegion()) { -// Recursively delete views - safe as all the views as all in the same region -for (ViewInfo viewInfo : tableViewFinderResult.getViewInfoList()) { -byte[] viewTenantId = viewInfo.getTenantId(); -byte[] viewSchemaName = viewInfo.getSchemaName(); -byte[] viewName = viewInfo.getViewName(); -byte[] viewKey = SchemaUtil.getTableKey(viewTenantId, viewSchemaName, viewName); -Delete delete = new Delete(viewKey, clientTimeStamp); -rowsToDelete.add(delete); -acquireLock(region, viewKey, locks); -MetaDataMutationResult result = doDropTable(viewKey, viewTenantId, viewSchemaName, -viewName, null, PTableType.VIEW, rowsToDelete, invalidateList, locks, -tableNamesToDelete, sharedTablesToDelete, false, clientVersion); -if (result.getMutationCode() != MutationCode.TABLE_ALREADY_EXISTS) { return result; } -} -} -} else { +// check to see if the table has any child views +try (Table hTable = +env.getTable(SchemaUtil.getPhysicalTableName( + PhoenixDatabaseMetaData.SYSTEM_CHILD_LINK_NAME_BYTES, +env.getConfiguration( { +boolean hasChildViews = +ViewFinder.hasChildViews(hTable, tenantId, schemaName, tableName, +clientTimeStamp); +if (hasChildViews && !isCascade) { --- End diff -- I think we can handle the race condition in a similar way to how we handle conflicting columns using checkAndMutate. I have updated PHOENIX-4799 to include this case. ---
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540261#comment-16540261 ] ASF GitHub Bot commented on PHOENIX-3534: - Github user twdsilva commented on a diff in the pull request: https://github.com/apache/phoenix/pull/303#discussion_r201741221 --- Diff: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java --- @@ -2147,46 +2566,29 @@ private MetaDataMutationResult doDropTable(byte[] key, byte[] tenantId, byte[] s } if (tableType == PTableType.TABLE || tableType == PTableType.SYSTEM) { -// Handle any child views that exist -TableViewFinder tableViewFinderResult = findChildViews(region, tenantId, table, clientVersion, !isCascade); -if (tableViewFinderResult.hasViews()) { -if (isCascade) { -if (tableViewFinderResult.allViewsInMultipleRegions()) { -// We don't yet support deleting a table with views where SYSTEM.CATALOG has split and the -// view metadata spans multiple regions -return new MetaDataMutationResult(MutationCode.UNALLOWED_TABLE_MUTATION, - EnvironmentEdgeManager.currentTimeMillis(), null); -} else if (tableViewFinderResult.allViewsInSingleRegion()) { -// Recursively delete views - safe as all the views as all in the same region -for (ViewInfo viewInfo : tableViewFinderResult.getViewInfoList()) { -byte[] viewTenantId = viewInfo.getTenantId(); -byte[] viewSchemaName = viewInfo.getSchemaName(); -byte[] viewName = viewInfo.getViewName(); -byte[] viewKey = SchemaUtil.getTableKey(viewTenantId, viewSchemaName, viewName); -Delete delete = new Delete(viewKey, clientTimeStamp); -rowsToDelete.add(delete); -acquireLock(region, viewKey, locks); -MetaDataMutationResult result = doDropTable(viewKey, viewTenantId, viewSchemaName, -viewName, null, PTableType.VIEW, rowsToDelete, invalidateList, locks, -tableNamesToDelete, sharedTablesToDelete, false, clientVersion); -if (result.getMutationCode() != MutationCode.TABLE_ALREADY_EXISTS) { return result; } -} -} -} else { +// check to see if the table has any child views +try (Table hTable = +env.getTable(SchemaUtil.getPhysicalTableName( + PhoenixDatabaseMetaData.SYSTEM_CHILD_LINK_NAME_BYTES, +env.getConfiguration( { +boolean hasChildViews = +ViewFinder.hasChildViews(hTable, tenantId, schemaName, tableName, +clientTimeStamp); +if (hasChildViews && !isCascade) { --- End diff -- I think we can handle the race condition in a similar way to how we handle conflicting columns using checkAndMutate. I have updated PHOENIX-4799 to include this case. > Support multi region SYSTEM.CATALOG table > - > > Key: PHOENIX-3534 > URL: https://issues.apache.org/jira/browse/PHOENIX-3534 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: Thomas D'Silva >Priority: Major > Fix For: 5.0.0, 4.15.0 > > Attachments: PHOENIX-3534.patch > > > Currently Phoenix requires that the SYSTEM.CATALOG table is single region > based on the server-side row locks being held for operations that impact a > table and all of it's views. For example, adding/removing a column from a > base table pushes this change to all views. > As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), > when a new table is created we can do a lazy cleanup of any rows that may be > left over from a failed DDL call (kudos to [~lhofhansl] for coming up with > this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 > so that we can efficiently find derived views. > The implementation would rely on an optimistic concurrency model based on > checking our sequence numbers for each table/view b
[GitHub] phoenix pull request #303: PHOENIX-3534 Support multi region SYSTEM.CATALOG ...
Github user twdsilva commented on a diff in the pull request: https://github.com/apache/phoenix/pull/303#discussion_r201744675 --- Diff: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java --- @@ -2573,307 +2897,139 @@ else if (pkCount <= COLUMN_NAME_INDEX return new MetaDataMutationResult(MutationCode.UNALLOWED_TABLE_MUTATION, EnvironmentEdgeManager.currentTimeMillis(), basePhysicalTable); } -ColumnOrdinalPositionUpdateList ordinalPositionList = new ColumnOrdinalPositionUpdateList(); +//add the new columns to the child view List viewPkCols = new ArrayList<>(view.getPKColumns()); boolean addingExistingPkCol = false; -int numCols = view.getColumns().size(); -// add the new columns to the child view -for (PutWithOrdinalPosition p : columnPutsForBaseTable) { -Put baseTableColumnPut = p.put; +for (Put columnToBeAdded : columnPutsForBaseTable) { PColumn existingViewColumn = null; byte[][] rkmd = new byte[5][]; -getVarChars(baseTableColumnPut.getRow(), rkmd); +getVarChars(columnToBeAdded.getRow(), rkmd); String columnName = Bytes.toString(rkmd[COLUMN_NAME_INDEX]); -String columnFamily = rkmd[FAMILY_NAME_INDEX] == null ? null : Bytes.toString(rkmd[FAMILY_NAME_INDEX]); +String columnFamily = +rkmd[FAMILY_NAME_INDEX] == null ? null +: Bytes.toString(rkmd[FAMILY_NAME_INDEX]); try { -existingViewColumn = columnFamily == null ? view.getColumnForColumnName(columnName) : view.getColumnFamily( - columnFamily).getPColumnForColumnName(columnName); +existingViewColumn = +columnFamily == null ? view.getColumnForColumnName(columnName) +: view.getColumnFamily(columnFamily) + .getPColumnForColumnName(columnName); } catch (ColumnFamilyNotFoundException e) { -// ignore since it means that the column family is not present for the column to be added. +// ignore since it means that the column family is not present for the column to +// be added. } catch (ColumnNotFoundException e) { // ignore since it means the column is not present in the view } - -boolean isPkCol = columnFamily == null; -byte[] columnKey = getColumnKey(viewKey, columnName, columnFamily); + +boolean isColumnToBeAddPkCol = columnFamily == null; if (existingViewColumn != null) { -MetaDataMutationResult result = validateColumnForAddToBaseTable(existingViewColumn, baseTableColumnPut, basePhysicalTable, isPkCol, view); -if (result != null) { -return result; +if (EncodedColumnsUtil.usesEncodedColumnNames(basePhysicalTable) +&& !SchemaUtil.isPKColumn(existingViewColumn)) { --- End diff -- The race condition with adding the same column to the base table and a child view will be covered in PHOENIX-4799. ---
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540272#comment-16540272 ] ASF GitHub Bot commented on PHOENIX-3534: - Github user twdsilva commented on a diff in the pull request: https://github.com/apache/phoenix/pull/303#discussion_r201744675 --- Diff: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java --- @@ -2573,307 +2897,139 @@ else if (pkCount <= COLUMN_NAME_INDEX return new MetaDataMutationResult(MutationCode.UNALLOWED_TABLE_MUTATION, EnvironmentEdgeManager.currentTimeMillis(), basePhysicalTable); } -ColumnOrdinalPositionUpdateList ordinalPositionList = new ColumnOrdinalPositionUpdateList(); +//add the new columns to the child view List viewPkCols = new ArrayList<>(view.getPKColumns()); boolean addingExistingPkCol = false; -int numCols = view.getColumns().size(); -// add the new columns to the child view -for (PutWithOrdinalPosition p : columnPutsForBaseTable) { -Put baseTableColumnPut = p.put; +for (Put columnToBeAdded : columnPutsForBaseTable) { PColumn existingViewColumn = null; byte[][] rkmd = new byte[5][]; -getVarChars(baseTableColumnPut.getRow(), rkmd); +getVarChars(columnToBeAdded.getRow(), rkmd); String columnName = Bytes.toString(rkmd[COLUMN_NAME_INDEX]); -String columnFamily = rkmd[FAMILY_NAME_INDEX] == null ? null : Bytes.toString(rkmd[FAMILY_NAME_INDEX]); +String columnFamily = +rkmd[FAMILY_NAME_INDEX] == null ? null +: Bytes.toString(rkmd[FAMILY_NAME_INDEX]); try { -existingViewColumn = columnFamily == null ? view.getColumnForColumnName(columnName) : view.getColumnFamily( - columnFamily).getPColumnForColumnName(columnName); +existingViewColumn = +columnFamily == null ? view.getColumnForColumnName(columnName) +: view.getColumnFamily(columnFamily) + .getPColumnForColumnName(columnName); } catch (ColumnFamilyNotFoundException e) { -// ignore since it means that the column family is not present for the column to be added. +// ignore since it means that the column family is not present for the column to +// be added. } catch (ColumnNotFoundException e) { // ignore since it means the column is not present in the view } - -boolean isPkCol = columnFamily == null; -byte[] columnKey = getColumnKey(viewKey, columnName, columnFamily); + +boolean isColumnToBeAddPkCol = columnFamily == null; if (existingViewColumn != null) { -MetaDataMutationResult result = validateColumnForAddToBaseTable(existingViewColumn, baseTableColumnPut, basePhysicalTable, isPkCol, view); -if (result != null) { -return result; +if (EncodedColumnsUtil.usesEncodedColumnNames(basePhysicalTable) +&& !SchemaUtil.isPKColumn(existingViewColumn)) { --- End diff -- The race condition with adding the same column to the base table and a child view will be covered in PHOENIX-4799. > Support multi region SYSTEM.CATALOG table > - > > Key: PHOENIX-3534 > URL: https://issues.apache.org/jira/browse/PHOENIX-3534 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: Thomas D'Silva >Priority: Major > Fix For: 5.0.0, 4.15.0 > > Attachments: PHOENIX-3534.patch > > > Currently Phoenix requires that the SYSTEM.CATALOG table is single region > based on the server-side row locks being held for operations that impact a > table and all of it's views. For example, adding/removing a column from a > base table pushes this change to all views. > As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), > when a new table is created we can do a lazy cleanup of any rows that may be > left over from a failed DDL call (kudos to [~lhofhansl] for coming up with > this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
[GitHub] phoenix pull request #303: PHOENIX-3534 Support multi region SYSTEM.CATALOG ...
Github user twdsilva commented on a diff in the pull request: https://github.com/apache/phoenix/pull/303#discussion_r201745438 --- Diff: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java --- @@ -3642,30 +3596,62 @@ private void dropIndexes(PTable table, Region region, List in boolean isCoveredColumn = indexMaintainer.getCoveredColumns().contains(colDropRef); // If index requires this column for its pk, then drop it if (isColumnIndexed) { -// Since we're dropping the index, lock it to ensure -// that a change in index state doesn't -// occur while we're dropping it. -acquireLock(region, indexKey, locks); // Drop the index table. The doDropTable will expand // this to all of the table rows and invalidate the // index table -additionalTableMetaData.add(new Delete(indexKey, clientTimeStamp)); +Delete delete = new Delete(indexKey, clientTimeStamp); byte[] linkKey = MetaDataUtil.getParentLinkKey(tenantId, schemaName, tableName, index .getTableName().getBytes()); -// Drop the link between the data table and the +// Drop the link between the parent table and the // index table -additionalTableMetaData.add(new Delete(linkKey, clientTimeStamp)); -doDropTable(indexKey, tenantId, index.getSchemaName().getBytes(), index -.getTableName().getBytes(), tableName, index.getType(), -additionalTableMetaData, invalidateList, locks, tableNamesToDelete, sharedTablesToDelete, false, clientVersion); -invalidateList.add(new ImmutableBytesPtr(indexKey)); +Delete linkDelete = new Delete(linkKey, clientTimeStamp); +List tableMetaData = Lists.newArrayListWithExpectedSize(2); +Delete tableDelete = delete; +tableMetaData.add(tableDelete); +tableMetaData.add(linkDelete); +// if the index is not present on the current region make an rpc to drop it --- End diff -- This is to handle dropping indexes on views. We could drop a column on a base table which is used by an index on view which could be on a separate region. ---
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540273#comment-16540273 ] ASF GitHub Bot commented on PHOENIX-3534: - Github user twdsilva commented on a diff in the pull request: https://github.com/apache/phoenix/pull/303#discussion_r201745438 --- Diff: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java --- @@ -3642,30 +3596,62 @@ private void dropIndexes(PTable table, Region region, List in boolean isCoveredColumn = indexMaintainer.getCoveredColumns().contains(colDropRef); // If index requires this column for its pk, then drop it if (isColumnIndexed) { -// Since we're dropping the index, lock it to ensure -// that a change in index state doesn't -// occur while we're dropping it. -acquireLock(region, indexKey, locks); // Drop the index table. The doDropTable will expand // this to all of the table rows and invalidate the // index table -additionalTableMetaData.add(new Delete(indexKey, clientTimeStamp)); +Delete delete = new Delete(indexKey, clientTimeStamp); byte[] linkKey = MetaDataUtil.getParentLinkKey(tenantId, schemaName, tableName, index .getTableName().getBytes()); -// Drop the link between the data table and the +// Drop the link between the parent table and the // index table -additionalTableMetaData.add(new Delete(linkKey, clientTimeStamp)); -doDropTable(indexKey, tenantId, index.getSchemaName().getBytes(), index -.getTableName().getBytes(), tableName, index.getType(), -additionalTableMetaData, invalidateList, locks, tableNamesToDelete, sharedTablesToDelete, false, clientVersion); -invalidateList.add(new ImmutableBytesPtr(indexKey)); +Delete linkDelete = new Delete(linkKey, clientTimeStamp); +List tableMetaData = Lists.newArrayListWithExpectedSize(2); +Delete tableDelete = delete; +tableMetaData.add(tableDelete); +tableMetaData.add(linkDelete); +// if the index is not present on the current region make an rpc to drop it --- End diff -- This is to handle dropping indexes on views. We could drop a column on a base table which is used by an index on view which could be on a separate region. > Support multi region SYSTEM.CATALOG table > - > > Key: PHOENIX-3534 > URL: https://issues.apache.org/jira/browse/PHOENIX-3534 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: Thomas D'Silva >Priority: Major > Fix For: 5.0.0, 4.15.0 > > Attachments: PHOENIX-3534.patch > > > Currently Phoenix requires that the SYSTEM.CATALOG table is single region > based on the server-side row locks being held for operations that impact a > table and all of it's views. For example, adding/removing a column from a > base table pushes this change to all views. > As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), > when a new table is created we can do a lazy cleanup of any rows that may be > left over from a failed DDL call (kudos to [~lhofhansl] for coming up with > this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 > so that we can efficiently find derived views. > The implementation would rely on an optimistic concurrency model based on > checking our sequence numbers for each table/view before/after updating. Each > table/view row would be individually locked for their change (metadata for a > view or table cannot span regions due to our split policy), with the sequence > number being incremented under lock and then returned to the client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] phoenix pull request #303: PHOENIX-3534 Support multi region SYSTEM.CATALOG ...
Github user twdsilva commented on a diff in the pull request: https://github.com/apache/phoenix/pull/303#discussion_r201748315 --- Diff: phoenix-core/src/main/java/org/apache/phoenix/schema/MetaDataClient.java --- @@ -1957,6 +1968,17 @@ private PTable createTableInternal(CreateTableStatement statement, byte[][] spli linkStatement.setLong(6, parent.getSequenceNumber()); linkStatement.setString(7, PTableType.INDEX.getSerializedValue()); linkStatement.execute(); + +// Add row linking index table to parent table for indexes on views +if (parent.getType() == PTableType.VIEW) { + linkStatement = connection.prepareStatement(CREATE_VIEW_INDEX_PARENT_LINK); + linkStatement.setString(1, tenantIdStr); + linkStatement.setString(2, schemaName); + linkStatement.setString(3, tableName); + linkStatement.setString(4, parent.getName().getString()); + linkStatement.setByte(5, LinkType.VIEW_INDEX_PARENT_TABLE.getSerializedValue()); + linkStatement.execute(); +} --- End diff -- I created PHOENIX-4766 for this, I will add a comment referencing this JIRA in createTableInternal(). ---
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540284#comment-16540284 ] ASF GitHub Bot commented on PHOENIX-3534: - Github user twdsilva commented on a diff in the pull request: https://github.com/apache/phoenix/pull/303#discussion_r201748315 --- Diff: phoenix-core/src/main/java/org/apache/phoenix/schema/MetaDataClient.java --- @@ -1957,6 +1968,17 @@ private PTable createTableInternal(CreateTableStatement statement, byte[][] spli linkStatement.setLong(6, parent.getSequenceNumber()); linkStatement.setString(7, PTableType.INDEX.getSerializedValue()); linkStatement.execute(); + +// Add row linking index table to parent table for indexes on views +if (parent.getType() == PTableType.VIEW) { + linkStatement = connection.prepareStatement(CREATE_VIEW_INDEX_PARENT_LINK); + linkStatement.setString(1, tenantIdStr); + linkStatement.setString(2, schemaName); + linkStatement.setString(3, tableName); + linkStatement.setString(4, parent.getName().getString()); + linkStatement.setByte(5, LinkType.VIEW_INDEX_PARENT_TABLE.getSerializedValue()); + linkStatement.execute(); +} --- End diff -- I created PHOENIX-4766 for this, I will add a comment referencing this JIRA in createTableInternal(). > Support multi region SYSTEM.CATALOG table > - > > Key: PHOENIX-3534 > URL: https://issues.apache.org/jira/browse/PHOENIX-3534 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: Thomas D'Silva >Priority: Major > Fix For: 5.0.0, 4.15.0 > > Attachments: PHOENIX-3534.patch > > > Currently Phoenix requires that the SYSTEM.CATALOG table is single region > based on the server-side row locks being held for operations that impact a > table and all of it's views. For example, adding/removing a column from a > base table pushes this change to all views. > As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), > when a new table is created we can do a lazy cleanup of any rows that may be > left over from a failed DDL call (kudos to [~lhofhansl] for coming up with > this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 > so that we can efficiently find derived views. > The implementation would rely on an optimistic concurrency model based on > checking our sequence numbers for each table/view before/after updating. Each > table/view row would be individually locked for their change (metadata for a > view or table cannot span regions due to our split policy), with the sequence > number being incremented under lock and then returned to the client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] phoenix pull request #303: PHOENIX-3534 Support multi region SYSTEM.CATALOG ...
Github user twdsilva commented on a diff in the pull request: https://github.com/apache/phoenix/pull/303#discussion_r201776089 --- Diff: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java --- @@ -1457,28 +1791,110 @@ private static void getSchemaTableNames(Mutation row, byte[][] schemaTableNames) schemaTableNames[2] = tName; } } - + @Override public void createTable(RpcController controller, CreateTableRequest request, RpcCallback done) { MetaDataResponse.Builder builder = MetaDataResponse.newBuilder(); byte[][] rowKeyMetaData = new byte[3][]; byte[] schemaName = null; byte[] tableName = null; +String fullTableName = null; try { int clientVersion = request.getClientVersion(); List tableMetadata = ProtobufUtil.getMutations(request); MetaDataUtil.getTenantIdAndSchemaAndTableName(tableMetadata, rowKeyMetaData); byte[] tenantIdBytes = rowKeyMetaData[PhoenixDatabaseMetaData.TENANT_ID_INDEX]; schemaName = rowKeyMetaData[PhoenixDatabaseMetaData.SCHEMA_NAME_INDEX]; tableName = rowKeyMetaData[PhoenixDatabaseMetaData.TABLE_NAME_INDEX]; +fullTableName = SchemaUtil.getTableName(schemaName, tableName); +// TODO before creating a table we need to see if the table was previously created and then dropped +// and clean up any parent->child links or child views --- End diff -- Done. ---
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540414#comment-16540414 ] ASF GitHub Bot commented on PHOENIX-3534: - Github user twdsilva commented on a diff in the pull request: https://github.com/apache/phoenix/pull/303#discussion_r201776009 --- Diff: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java --- @@ -586,48 +590,359 @@ public void getTable(RpcController controller, GetTableRequest request, builder.setMutationTime(minNonZerodisableIndexTimestamp - 1); } } - -if (table.getTimeStamp() != tableTimeStamp) { +// the PTable of views and indexes on views might get updated because a column is added to one of +// their parents (this won't change the timestamp) +if (table.getType()!=PTableType.TABLE || table.getTimeStamp() != tableTimeStamp) { builder.setTable(PTableImpl.toProto(table)); } done.run(builder.build()); -return; } catch (Throwable t) { logger.error("getTable failed", t); ProtobufUtil.setControllerException(controller, ServerUtil.createIOException(SchemaUtil.getTableName(schemaName, tableName), t)); } } +/** + * Used to add the columns present the ancestor hierarchy to the PTable of the given view or + * view index + * @param table PTable of the view or view index + * @param skipAddingIndexes if true the returned PTable won't include indexes + * @param skipAddingParentColumns if true the returned PTable won't include columns derived from + *ancestor tables + * @param lockedAncestorTable ancestor table table that is being mutated (as we won't be able to + *resolve this table as its locked) + */ +private Pair combineColumns(PTable table, long timestamp, +int clientVersion, boolean skipAddingIndexes, boolean skipAddingParentColumns, +PTable lockedAncestorTable) throws SQLException, IOException { +boolean hasIndexId = table.getViewIndexId() != null; +if (table.getType() != PTableType.VIEW && !hasIndexId) { +return new Pair(table, +MetaDataProtos.MutationCode.TABLE_ALREADY_EXISTS); +} +if (!skipAddingParentColumns) { +table = +addDerivedColumnsFromAncestors(table, timestamp, clientVersion, +lockedAncestorTable); +if (table==null) { +return new Pair(table, +MetaDataProtos.MutationCode.TABLE_NOT_FOUND); +} +// we need to resolve the indexes of views (to get ensure they also have all the columns +// derived from their ancestors) +if (!skipAddingIndexes && !table.getIndexes().isEmpty()) { +List indexes = Lists.newArrayListWithExpectedSize(table.getIndexes().size()); +for (PTable index : table.getIndexes()) { +byte[] tenantIdBytes = +index.getTenantId() == null ? ByteUtil.EMPTY_BYTE_ARRAY +: index.getTenantId().getBytes(); +PTable latestIndex = +doGetTable(tenantIdBytes, index.getSchemaName().getBytes(), +index.getTableName().getBytes(), timestamp, null, clientVersion, true, +false, lockedAncestorTable); +if (latestIndex == null) { +throw new TableNotFoundException( +"Could not find index table while combining columns " ++ index.getTableName().getString() + " with tenant id " ++ index.getTenantId()); +} +indexes.add(latestIndex); +} +table = PTableImpl.makePTable(table, table.getTimeStamp(), indexes); +} +} + +MetaDataProtos.MutationCode mutationCode = +table != null ? MetaDataProtos.MutationCode.TABLE_ALREADY_EXISTS +: MetaDataProtos.MutationCode.TABLE_NOT_FOUND; +return new Pair(table, mutationCode); +} + + +private PTable addDerivedColumnsFromAncestors(PTable table, long timestamp, +int clientVersion,
[GitHub] phoenix pull request #303: PHOENIX-3534 Support multi region SYSTEM.CATALOG ...
Github user twdsilva commented on a diff in the pull request: https://github.com/apache/phoenix/pull/303#discussion_r201776009 --- Diff: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java --- @@ -586,48 +590,359 @@ public void getTable(RpcController controller, GetTableRequest request, builder.setMutationTime(minNonZerodisableIndexTimestamp - 1); } } - -if (table.getTimeStamp() != tableTimeStamp) { +// the PTable of views and indexes on views might get updated because a column is added to one of +// their parents (this won't change the timestamp) +if (table.getType()!=PTableType.TABLE || table.getTimeStamp() != tableTimeStamp) { builder.setTable(PTableImpl.toProto(table)); } done.run(builder.build()); -return; } catch (Throwable t) { logger.error("getTable failed", t); ProtobufUtil.setControllerException(controller, ServerUtil.createIOException(SchemaUtil.getTableName(schemaName, tableName), t)); } } +/** + * Used to add the columns present the ancestor hierarchy to the PTable of the given view or + * view index + * @param table PTable of the view or view index + * @param skipAddingIndexes if true the returned PTable won't include indexes + * @param skipAddingParentColumns if true the returned PTable won't include columns derived from + *ancestor tables + * @param lockedAncestorTable ancestor table table that is being mutated (as we won't be able to + *resolve this table as its locked) + */ +private Pair combineColumns(PTable table, long timestamp, +int clientVersion, boolean skipAddingIndexes, boolean skipAddingParentColumns, +PTable lockedAncestorTable) throws SQLException, IOException { +boolean hasIndexId = table.getViewIndexId() != null; +if (table.getType() != PTableType.VIEW && !hasIndexId) { +return new Pair(table, +MetaDataProtos.MutationCode.TABLE_ALREADY_EXISTS); +} +if (!skipAddingParentColumns) { +table = +addDerivedColumnsFromAncestors(table, timestamp, clientVersion, +lockedAncestorTable); +if (table==null) { +return new Pair(table, +MetaDataProtos.MutationCode.TABLE_NOT_FOUND); +} +// we need to resolve the indexes of views (to get ensure they also have all the columns +// derived from their ancestors) +if (!skipAddingIndexes && !table.getIndexes().isEmpty()) { +List indexes = Lists.newArrayListWithExpectedSize(table.getIndexes().size()); +for (PTable index : table.getIndexes()) { +byte[] tenantIdBytes = +index.getTenantId() == null ? ByteUtil.EMPTY_BYTE_ARRAY +: index.getTenantId().getBytes(); +PTable latestIndex = +doGetTable(tenantIdBytes, index.getSchemaName().getBytes(), +index.getTableName().getBytes(), timestamp, null, clientVersion, true, +false, lockedAncestorTable); +if (latestIndex == null) { +throw new TableNotFoundException( +"Could not find index table while combining columns " ++ index.getTableName().getString() + " with tenant id " ++ index.getTenantId()); +} +indexes.add(latestIndex); +} +table = PTableImpl.makePTable(table, table.getTimeStamp(), indexes); +} +} + +MetaDataProtos.MutationCode mutationCode = +table != null ? MetaDataProtos.MutationCode.TABLE_ALREADY_EXISTS +: MetaDataProtos.MutationCode.TABLE_NOT_FOUND; +return new Pair(table, mutationCode); +} + + +private PTable addDerivedColumnsFromAncestors(PTable table, long timestamp, +int clientVersion, PTable lockedAncestorTable) throws IOException, SQLException, TableNotFoundException { +// combine columns for view and view indexes +byte[] tenantId = +table.getTenantId() != null ? table.getTenantId().getBytes()
[GitHub] phoenix pull request #303: PHOENIX-3534 Support multi region SYSTEM.CATALOG ...
Github user twdsilva commented on a diff in the pull request: https://github.com/apache/phoenix/pull/303#discussion_r201776136 --- Diff: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java --- @@ -1457,28 +1791,110 @@ private static void getSchemaTableNames(Mutation row, byte[][] schemaTableNames) schemaTableNames[2] = tName; } } - + @Override public void createTable(RpcController controller, CreateTableRequest request, RpcCallback done) { MetaDataResponse.Builder builder = MetaDataResponse.newBuilder(); byte[][] rowKeyMetaData = new byte[3][]; byte[] schemaName = null; byte[] tableName = null; +String fullTableName = null; try { int clientVersion = request.getClientVersion(); List tableMetadata = ProtobufUtil.getMutations(request); MetaDataUtil.getTenantIdAndSchemaAndTableName(tableMetadata, rowKeyMetaData); byte[] tenantIdBytes = rowKeyMetaData[PhoenixDatabaseMetaData.TENANT_ID_INDEX]; schemaName = rowKeyMetaData[PhoenixDatabaseMetaData.SCHEMA_NAME_INDEX]; tableName = rowKeyMetaData[PhoenixDatabaseMetaData.TABLE_NAME_INDEX]; +fullTableName = SchemaUtil.getTableName(schemaName, tableName); +// TODO before creating a table we need to see if the table was previously created and then dropped +// and clean up any parent->child links or child views boolean isNamespaceMapped = MetaDataUtil.isNameSpaceMapped(tableMetadata, GenericKeyValueBuilder.INSTANCE, new ImmutableBytesWritable()); final IndexType indexType = MetaDataUtil.getIndexType(tableMetadata, GenericKeyValueBuilder.INSTANCE, new ImmutableBytesWritable()); +byte[] parentTenantId = null; byte[] parentSchemaName = null; byte[] parentTableName = null; PTableType tableType = MetaDataUtil.getTableType(tableMetadata, GenericKeyValueBuilder.INSTANCE, new ImmutableBytesWritable()); +ViewType viewType = MetaDataUtil.getViewType(tableMetadata, GenericKeyValueBuilder.INSTANCE, new ImmutableBytesWritable()); + +// Load table to see if it already exists +byte[] tableKey = SchemaUtil.getTableKey(tenantIdBytes, schemaName, tableName); +ImmutableBytesPtr cacheKey = new ImmutableBytesPtr(tableKey); +long clientTimeStamp = MetaDataUtil.getClientTimeStamp(tableMetadata); +PTable table = null; + try { + // Get as of latest timestamp so we can detect if we have a newer table that already + // exists without making an additional query + table = loadTable(env, tableKey, cacheKey, clientTimeStamp, HConstants.LATEST_TIMESTAMP, + clientVersion); + } catch (ParentTableNotFoundException e) { + dropChildMetadata(e.getParentSchemaName(), e.getParentTableName(), e.getParentTenantId()); + } +if (table != null) { +if (table.getTimeStamp() < clientTimeStamp) { +// If the table is older than the client time stamp and it's deleted, +// continue +if (!isTableDeleted(table)) { + builder.setReturnCode(MetaDataProtos.MutationCode.TABLE_ALREADY_EXISTS); + builder.setMutationTime(EnvironmentEdgeManager.currentTimeMillis()); +builder.setTable(PTableImpl.toProto(table)); +done.run(builder.build()); +return; +} +} else { + builder.setReturnCode(MetaDataProtos.MutationCode.NEWER_TABLE_FOUND); + builder.setMutationTime(EnvironmentEdgeManager.currentTimeMillis()); +builder.setTable(PTableImpl.toProto(table)); +done.run(builder.build()); +return; +} +} + + // check if the table was dropped, but had child views that were have not yet + // been cleaned up by compaction + if (!Bytes.toString(schemaName).equals(QueryConstants.SYSTEM_SCHEMA_NAME)) { + dropChildMetadata(schemaName, tableName, tenantIdBytes); + } --- End diff -- Fixed. ---
[GitHub] phoenix pull request #303: PHOENIX-3534 Support multi region SYSTEM.CATALOG ...
Github user twdsilva commented on a diff in the pull request: https://github.com/apache/phoenix/pull/303#discussion_r201776046 --- Diff: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java --- @@ -586,48 +590,359 @@ public void getTable(RpcController controller, GetTableRequest request, builder.setMutationTime(minNonZerodisableIndexTimestamp - 1); } } - -if (table.getTimeStamp() != tableTimeStamp) { +// the PTable of views and indexes on views might get updated because a column is added to one of +// their parents (this won't change the timestamp) +if (table.getType()!=PTableType.TABLE || table.getTimeStamp() != tableTimeStamp) { builder.setTable(PTableImpl.toProto(table)); } done.run(builder.build()); -return; } catch (Throwable t) { logger.error("getTable failed", t); ProtobufUtil.setControllerException(controller, ServerUtil.createIOException(SchemaUtil.getTableName(schemaName, tableName), t)); } } +/** + * Used to add the columns present the ancestor hierarchy to the PTable of the given view or + * view index + * @param table PTable of the view or view index + * @param skipAddingIndexes if true the returned PTable won't include indexes + * @param skipAddingParentColumns if true the returned PTable won't include columns derived from + *ancestor tables + * @param lockedAncestorTable ancestor table table that is being mutated (as we won't be able to + *resolve this table as its locked) + */ +private Pair combineColumns(PTable table, long timestamp, +int clientVersion, boolean skipAddingIndexes, boolean skipAddingParentColumns, +PTable lockedAncestorTable) throws SQLException, IOException { +boolean hasIndexId = table.getViewIndexId() != null; +if (table.getType() != PTableType.VIEW && !hasIndexId) { +return new Pair(table, +MetaDataProtos.MutationCode.TABLE_ALREADY_EXISTS); +} +if (!skipAddingParentColumns) { +table = +addDerivedColumnsFromAncestors(table, timestamp, clientVersion, +lockedAncestorTable); +if (table==null) { +return new Pair(table, +MetaDataProtos.MutationCode.TABLE_NOT_FOUND); +} +// we need to resolve the indexes of views (to get ensure they also have all the columns +// derived from their ancestors) +if (!skipAddingIndexes && !table.getIndexes().isEmpty()) { +List indexes = Lists.newArrayListWithExpectedSize(table.getIndexes().size()); +for (PTable index : table.getIndexes()) { +byte[] tenantIdBytes = +index.getTenantId() == null ? ByteUtil.EMPTY_BYTE_ARRAY +: index.getTenantId().getBytes(); +PTable latestIndex = +doGetTable(tenantIdBytes, index.getSchemaName().getBytes(), +index.getTableName().getBytes(), timestamp, null, clientVersion, true, +false, lockedAncestorTable); +if (latestIndex == null) { +throw new TableNotFoundException( +"Could not find index table while combining columns " ++ index.getTableName().getString() + " with tenant id " ++ index.getTenantId()); +} +indexes.add(latestIndex); +} +table = PTableImpl.makePTable(table, table.getTimeStamp(), indexes); +} +} + +MetaDataProtos.MutationCode mutationCode = +table != null ? MetaDataProtos.MutationCode.TABLE_ALREADY_EXISTS +: MetaDataProtos.MutationCode.TABLE_NOT_FOUND; +return new Pair(table, mutationCode); +} + + +private PTable addDerivedColumnsFromAncestors(PTable table, long timestamp, +int clientVersion, PTable lockedAncestorTable) throws IOException, SQLException, TableNotFoundException { +// combine columns for view and view indexes +byte[] tenantId = +table.getTenantId() != null ? table.getTenantId().getBytes()
[GitHub] phoenix pull request #303: PHOENIX-3534 Support multi region SYSTEM.CATALOG ...
Github user twdsilva commented on a diff in the pull request: https://github.com/apache/phoenix/pull/303#discussion_r201776174 --- Diff: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java --- @@ -1488,20 +1904,19 @@ public void createTable(RpcController controller, CreateTableRequest request, if (tableType == PTableType.VIEW) { byte[][] parentSchemaTableNames = new byte[3][]; byte[][] parentPhysicalSchemaTableNames = new byte[3][]; -/* - * For a view, we lock the base physical table row. For a mapped view, there is - * no link present to the physical table. So the viewPhysicalTableRow is null - * in that case. - */ + /* +* For a mapped view, there is no link present to the physical table. So the +* viewPhysicalTableRow is null in that case. +*/ --- End diff -- Fixed. ---
[GitHub] phoenix pull request #303: PHOENIX-3534 Support multi region SYSTEM.CATALOG ...
Github user twdsilva commented on a diff in the pull request: https://github.com/apache/phoenix/pull/303#discussion_r201776204 --- Diff: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java --- @@ -1809,180 +2256,97 @@ public void createTable(RpcController controller, CreateTableRequest request, } catch (Throwable t) { logger.error("createTable failed", t); ProtobufUtil.setControllerException(controller, - ServerUtil.createIOException(SchemaUtil.getTableName(schemaName, tableName), t)); +ServerUtil.createIOException(fullTableName, t)); } } + private void dropChildMetadata(byte[] schemaName, byte[] tableName, byte[] tenantIdBytes) + throws IOException, SQLException, ClassNotFoundException { + TableViewFinderResult childViewsResult = new TableViewFinderResult(); + findAllChildViews(tenantIdBytes, schemaName, tableName, childViewsResult); + if (childViewsResult.hasViews()) { + for (TableInfo viewInfo : childViewsResult.getResults()) { + byte[] viewTenantId = viewInfo.getTenantId(); + byte[] viewSchemaName = viewInfo.getSchemaName(); + byte[] viewName = viewInfo.getTableName(); + Properties props = new Properties(); + if (viewTenantId != null && viewTenantId.length != 0) + props.setProperty(PhoenixRuntime.TENANT_ID_ATTRIB, Bytes.toString(viewTenantId)); + try (PhoenixConnection connection = QueryUtil.getConnectionOnServer(env.getConfiguration()) + .unwrap(PhoenixConnection.class)) { + MetaDataClient client = new MetaDataClient(connection); + org.apache.phoenix.parse.TableName viewTableName = org.apache.phoenix.parse.TableName + .create(Bytes.toString(viewSchemaName), Bytes.toString(viewName)); + client.dropTable( + new DropTableStatement(viewTableName, PTableType.VIEW, false, true, true)); + } + } + } + } + private boolean execeededIndexQuota(PTableType tableType, PTable parentTable) { return PTableType.INDEX == tableType && parentTable.getIndexes().size() >= maxIndexesPerTable; } - -private void findAllChildViews(Region region, byte[] tenantId, PTable table, -TableViewFinder result, long clientTimeStamp, int clientVersion) throws IOException, SQLException { -TableViewFinder currResult = findChildViews(region, tenantId, table, clientVersion, false); -result.addResult(currResult); -for (ViewInfo viewInfo : currResult.getViewInfoList()) { -byte[] viewtenantId = viewInfo.getTenantId(); -byte[] viewSchema = viewInfo.getSchemaName(); -byte[] viewTable = viewInfo.getViewName(); -byte[] tableKey = SchemaUtil.getTableKey(viewtenantId, viewSchema, viewTable); -ImmutableBytesPtr cacheKey = new ImmutableBytesPtr(tableKey); -PTable view = loadTable(env, tableKey, cacheKey, clientTimeStamp, clientTimeStamp, clientVersion); -if (view == null) { -logger.warn("Found orphan tenant view row in SYSTEM.CATALOG with tenantId:" -+ Bytes.toString(tenantId) + ", schema:" -+ Bytes.toString(viewSchema) + ", table:" -+ Bytes.toString(viewTable)); -continue; -} -findAllChildViews(region, viewtenantId, view, result, clientTimeStamp, clientVersion); -} -} - -// TODO use child link instead once splittable system catalog (PHOENIX-3534) is implemented -// and we have a separate table for links. -private TableViewFinder findChildViews_deprecated(Region region, byte[] tenantId, PTable table, byte[] linkTypeBytes, boolean stopAfterFirst) throws IOException { -byte[] schemaName = table.getSchemaName().getBytes(); -byte[] tableName = table.getTableName().getBytes(); -boolean isMultiTenant = table.isMultiTenant(); -Scan scan = new Scan(); -// If the table is multi-tenant, we need to check across all tenant_ids, -// so we can't constrain the row key. Otherwise, any views would have -// the same tenantId. -
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540417#comment-16540417 ] ASF GitHub Bot commented on PHOENIX-3534: - Github user twdsilva commented on a diff in the pull request: https://github.com/apache/phoenix/pull/303#discussion_r201776136 --- Diff: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java --- @@ -1457,28 +1791,110 @@ private static void getSchemaTableNames(Mutation row, byte[][] schemaTableNames) schemaTableNames[2] = tName; } } - + @Override public void createTable(RpcController controller, CreateTableRequest request, RpcCallback done) { MetaDataResponse.Builder builder = MetaDataResponse.newBuilder(); byte[][] rowKeyMetaData = new byte[3][]; byte[] schemaName = null; byte[] tableName = null; +String fullTableName = null; try { int clientVersion = request.getClientVersion(); List tableMetadata = ProtobufUtil.getMutations(request); MetaDataUtil.getTenantIdAndSchemaAndTableName(tableMetadata, rowKeyMetaData); byte[] tenantIdBytes = rowKeyMetaData[PhoenixDatabaseMetaData.TENANT_ID_INDEX]; schemaName = rowKeyMetaData[PhoenixDatabaseMetaData.SCHEMA_NAME_INDEX]; tableName = rowKeyMetaData[PhoenixDatabaseMetaData.TABLE_NAME_INDEX]; +fullTableName = SchemaUtil.getTableName(schemaName, tableName); +// TODO before creating a table we need to see if the table was previously created and then dropped +// and clean up any parent->child links or child views boolean isNamespaceMapped = MetaDataUtil.isNameSpaceMapped(tableMetadata, GenericKeyValueBuilder.INSTANCE, new ImmutableBytesWritable()); final IndexType indexType = MetaDataUtil.getIndexType(tableMetadata, GenericKeyValueBuilder.INSTANCE, new ImmutableBytesWritable()); +byte[] parentTenantId = null; byte[] parentSchemaName = null; byte[] parentTableName = null; PTableType tableType = MetaDataUtil.getTableType(tableMetadata, GenericKeyValueBuilder.INSTANCE, new ImmutableBytesWritable()); +ViewType viewType = MetaDataUtil.getViewType(tableMetadata, GenericKeyValueBuilder.INSTANCE, new ImmutableBytesWritable()); + +// Load table to see if it already exists +byte[] tableKey = SchemaUtil.getTableKey(tenantIdBytes, schemaName, tableName); +ImmutableBytesPtr cacheKey = new ImmutableBytesPtr(tableKey); +long clientTimeStamp = MetaDataUtil.getClientTimeStamp(tableMetadata); +PTable table = null; + try { + // Get as of latest timestamp so we can detect if we have a newer table that already + // exists without making an additional query + table = loadTable(env, tableKey, cacheKey, clientTimeStamp, HConstants.LATEST_TIMESTAMP, + clientVersion); + } catch (ParentTableNotFoundException e) { + dropChildMetadata(e.getParentSchemaName(), e.getParentTableName(), e.getParentTenantId()); + } +if (table != null) { +if (table.getTimeStamp() < clientTimeStamp) { +// If the table is older than the client time stamp and it's deleted, +// continue +if (!isTableDeleted(table)) { + builder.setReturnCode(MetaDataProtos.MutationCode.TABLE_ALREADY_EXISTS); + builder.setMutationTime(EnvironmentEdgeManager.currentTimeMillis()); +builder.setTable(PTableImpl.toProto(table)); +done.run(builder.build()); +return; +} +} else { + builder.setReturnCode(MetaDataProtos.MutationCode.NEWER_TABLE_FOUND); + builder.setMutationTime(EnvironmentEdgeManager.currentTimeMillis()); +builder.setTable(PTableImpl.toProto(table)); +done.run(builder.build()); +return; +} +} + + // check if the table was dropped, but had child views that were have not yet + // been cleaned up by compaction + if (!Bytes.toSt
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540418#comment-16540418 ] ASF GitHub Bot commented on PHOENIX-3534: - Github user twdsilva commented on a diff in the pull request: https://github.com/apache/phoenix/pull/303#discussion_r201776174 --- Diff: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java --- @@ -1488,20 +1904,19 @@ public void createTable(RpcController controller, CreateTableRequest request, if (tableType == PTableType.VIEW) { byte[][] parentSchemaTableNames = new byte[3][]; byte[][] parentPhysicalSchemaTableNames = new byte[3][]; -/* - * For a view, we lock the base physical table row. For a mapped view, there is - * no link present to the physical table. So the viewPhysicalTableRow is null - * in that case. - */ + /* +* For a mapped view, there is no link present to the physical table. So the +* viewPhysicalTableRow is null in that case. +*/ --- End diff -- Fixed. > Support multi region SYSTEM.CATALOG table > - > > Key: PHOENIX-3534 > URL: https://issues.apache.org/jira/browse/PHOENIX-3534 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: Thomas D'Silva >Priority: Major > Fix For: 5.0.0, 4.15.0 > > Attachments: PHOENIX-3534.patch > > > Currently Phoenix requires that the SYSTEM.CATALOG table is single region > based on the server-side row locks being held for operations that impact a > table and all of it's views. For example, adding/removing a column from a > base table pushes this change to all views. > As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), > when a new table is created we can do a lazy cleanup of any rows that may be > left over from a failed DDL call (kudos to [~lhofhansl] for coming up with > this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 > so that we can efficiently find derived views. > The implementation would rely on an optimistic concurrency model based on > checking our sequence numbers for each table/view before/after updating. Each > table/view row would be individually locked for their change (metadata for a > view or table cannot span regions due to our split policy), with the sequence > number being incremented under lock and then returned to the client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] phoenix pull request #303: PHOENIX-3534 Support multi region SYSTEM.CATALOG ...
Github user twdsilva commented on a diff in the pull request: https://github.com/apache/phoenix/pull/303#discussion_r201776288 --- Diff: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java --- @@ -447,7 +447,7 @@ static { Collections.sort(FUNCTION_KV_COLUMNS, KeyValue.COMPARATOR); } - + --- End diff -- I modified the class level comment. ---
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540416#comment-16540416 ] ASF GitHub Bot commented on PHOENIX-3534: - Github user twdsilva commented on a diff in the pull request: https://github.com/apache/phoenix/pull/303#discussion_r201776089 --- Diff: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java --- @@ -1457,28 +1791,110 @@ private static void getSchemaTableNames(Mutation row, byte[][] schemaTableNames) schemaTableNames[2] = tName; } } - + @Override public void createTable(RpcController controller, CreateTableRequest request, RpcCallback done) { MetaDataResponse.Builder builder = MetaDataResponse.newBuilder(); byte[][] rowKeyMetaData = new byte[3][]; byte[] schemaName = null; byte[] tableName = null; +String fullTableName = null; try { int clientVersion = request.getClientVersion(); List tableMetadata = ProtobufUtil.getMutations(request); MetaDataUtil.getTenantIdAndSchemaAndTableName(tableMetadata, rowKeyMetaData); byte[] tenantIdBytes = rowKeyMetaData[PhoenixDatabaseMetaData.TENANT_ID_INDEX]; schemaName = rowKeyMetaData[PhoenixDatabaseMetaData.SCHEMA_NAME_INDEX]; tableName = rowKeyMetaData[PhoenixDatabaseMetaData.TABLE_NAME_INDEX]; +fullTableName = SchemaUtil.getTableName(schemaName, tableName); +// TODO before creating a table we need to see if the table was previously created and then dropped +// and clean up any parent->child links or child views --- End diff -- Done. > Support multi region SYSTEM.CATALOG table > - > > Key: PHOENIX-3534 > URL: https://issues.apache.org/jira/browse/PHOENIX-3534 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: Thomas D'Silva >Priority: Major > Fix For: 5.0.0, 4.15.0 > > Attachments: PHOENIX-3534.patch > > > Currently Phoenix requires that the SYSTEM.CATALOG table is single region > based on the server-side row locks being held for operations that impact a > table and all of it's views. For example, adding/removing a column from a > base table pushes this change to all views. > As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), > when a new table is created we can do a lazy cleanup of any rows that may be > left over from a failed DDL call (kudos to [~lhofhansl] for coming up with > this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 > so that we can efficiently find derived views. > The implementation would rely on an optimistic concurrency model based on > checking our sequence numbers for each table/view before/after updating. Each > table/view row would be individually locked for their change (metadata for a > view or table cannot span regions due to our split policy), with the sequence > number being incremented under lock and then returned to the client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540415#comment-16540415 ] ASF GitHub Bot commented on PHOENIX-3534: - Github user twdsilva commented on a diff in the pull request: https://github.com/apache/phoenix/pull/303#discussion_r201776046 --- Diff: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java --- @@ -586,48 +590,359 @@ public void getTable(RpcController controller, GetTableRequest request, builder.setMutationTime(minNonZerodisableIndexTimestamp - 1); } } - -if (table.getTimeStamp() != tableTimeStamp) { +// the PTable of views and indexes on views might get updated because a column is added to one of +// their parents (this won't change the timestamp) +if (table.getType()!=PTableType.TABLE || table.getTimeStamp() != tableTimeStamp) { builder.setTable(PTableImpl.toProto(table)); } done.run(builder.build()); -return; } catch (Throwable t) { logger.error("getTable failed", t); ProtobufUtil.setControllerException(controller, ServerUtil.createIOException(SchemaUtil.getTableName(schemaName, tableName), t)); } } +/** + * Used to add the columns present the ancestor hierarchy to the PTable of the given view or + * view index + * @param table PTable of the view or view index + * @param skipAddingIndexes if true the returned PTable won't include indexes + * @param skipAddingParentColumns if true the returned PTable won't include columns derived from + *ancestor tables + * @param lockedAncestorTable ancestor table table that is being mutated (as we won't be able to + *resolve this table as its locked) + */ +private Pair combineColumns(PTable table, long timestamp, +int clientVersion, boolean skipAddingIndexes, boolean skipAddingParentColumns, +PTable lockedAncestorTable) throws SQLException, IOException { +boolean hasIndexId = table.getViewIndexId() != null; +if (table.getType() != PTableType.VIEW && !hasIndexId) { +return new Pair(table, +MetaDataProtos.MutationCode.TABLE_ALREADY_EXISTS); +} +if (!skipAddingParentColumns) { +table = +addDerivedColumnsFromAncestors(table, timestamp, clientVersion, +lockedAncestorTable); +if (table==null) { +return new Pair(table, +MetaDataProtos.MutationCode.TABLE_NOT_FOUND); +} +// we need to resolve the indexes of views (to get ensure they also have all the columns +// derived from their ancestors) +if (!skipAddingIndexes && !table.getIndexes().isEmpty()) { +List indexes = Lists.newArrayListWithExpectedSize(table.getIndexes().size()); +for (PTable index : table.getIndexes()) { +byte[] tenantIdBytes = +index.getTenantId() == null ? ByteUtil.EMPTY_BYTE_ARRAY +: index.getTenantId().getBytes(); +PTable latestIndex = +doGetTable(tenantIdBytes, index.getSchemaName().getBytes(), +index.getTableName().getBytes(), timestamp, null, clientVersion, true, +false, lockedAncestorTable); +if (latestIndex == null) { +throw new TableNotFoundException( +"Could not find index table while combining columns " ++ index.getTableName().getString() + " with tenant id " ++ index.getTenantId()); +} +indexes.add(latestIndex); +} +table = PTableImpl.makePTable(table, table.getTimeStamp(), indexes); +} +} + +MetaDataProtos.MutationCode mutationCode = +table != null ? MetaDataProtos.MutationCode.TABLE_ALREADY_EXISTS +: MetaDataProtos.MutationCode.TABLE_NOT_FOUND; +return new Pair(table, mutationCode); +} + + +private PTable addDerivedColumnsFromAncestors(PTable table, long timestamp, +int clientVersion,
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540419#comment-16540419 ] ASF GitHub Bot commented on PHOENIX-3534: - Github user twdsilva commented on a diff in the pull request: https://github.com/apache/phoenix/pull/303#discussion_r201776204 --- Diff: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java --- @@ -1809,180 +2256,97 @@ public void createTable(RpcController controller, CreateTableRequest request, } catch (Throwable t) { logger.error("createTable failed", t); ProtobufUtil.setControllerException(controller, - ServerUtil.createIOException(SchemaUtil.getTableName(schemaName, tableName), t)); +ServerUtil.createIOException(fullTableName, t)); } } + private void dropChildMetadata(byte[] schemaName, byte[] tableName, byte[] tenantIdBytes) + throws IOException, SQLException, ClassNotFoundException { + TableViewFinderResult childViewsResult = new TableViewFinderResult(); + findAllChildViews(tenantIdBytes, schemaName, tableName, childViewsResult); + if (childViewsResult.hasViews()) { + for (TableInfo viewInfo : childViewsResult.getResults()) { + byte[] viewTenantId = viewInfo.getTenantId(); + byte[] viewSchemaName = viewInfo.getSchemaName(); + byte[] viewName = viewInfo.getTableName(); + Properties props = new Properties(); + if (viewTenantId != null && viewTenantId.length != 0) + props.setProperty(PhoenixRuntime.TENANT_ID_ATTRIB, Bytes.toString(viewTenantId)); + try (PhoenixConnection connection = QueryUtil.getConnectionOnServer(env.getConfiguration()) + .unwrap(PhoenixConnection.class)) { + MetaDataClient client = new MetaDataClient(connection); + org.apache.phoenix.parse.TableName viewTableName = org.apache.phoenix.parse.TableName + .create(Bytes.toString(viewSchemaName), Bytes.toString(viewName)); + client.dropTable( + new DropTableStatement(viewTableName, PTableType.VIEW, false, true, true)); + } + } + } + } + private boolean execeededIndexQuota(PTableType tableType, PTable parentTable) { return PTableType.INDEX == tableType && parentTable.getIndexes().size() >= maxIndexesPerTable; } - -private void findAllChildViews(Region region, byte[] tenantId, PTable table, -TableViewFinder result, long clientTimeStamp, int clientVersion) throws IOException, SQLException { -TableViewFinder currResult = findChildViews(region, tenantId, table, clientVersion, false); -result.addResult(currResult); -for (ViewInfo viewInfo : currResult.getViewInfoList()) { -byte[] viewtenantId = viewInfo.getTenantId(); -byte[] viewSchema = viewInfo.getSchemaName(); -byte[] viewTable = viewInfo.getViewName(); -byte[] tableKey = SchemaUtil.getTableKey(viewtenantId, viewSchema, viewTable); -ImmutableBytesPtr cacheKey = new ImmutableBytesPtr(tableKey); -PTable view = loadTable(env, tableKey, cacheKey, clientTimeStamp, clientTimeStamp, clientVersion); -if (view == null) { -logger.warn("Found orphan tenant view row in SYSTEM.CATALOG with tenantId:" -+ Bytes.toString(tenantId) + ", schema:" -+ Bytes.toString(viewSchema) + ", table:" -+ Bytes.toString(viewTable)); -continue; -} -findAllChildViews(region, viewtenantId, view, result, clientTimeStamp, clientVersion); -} -} - -// TODO use child link instead once splittable system catalog (PHOENIX-3534) is implemented -// and we have a separate table for links. -private TableViewFinder findChildViews_deprecated(Region region, byte[] tenantId, PTable table, byte[] linkTypeBytes, boolean stopAfterFirst) throws IOException { -byte[] schemaName = table.getSchemaName().getBytes(); -byte[] tableName = table.getTableName().getBytes(); -boolean isMultiTenant = table.isMu
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540420#comment-16540420 ] ASF GitHub Bot commented on PHOENIX-3534: - Github user twdsilva commented on a diff in the pull request: https://github.com/apache/phoenix/pull/303#discussion_r201776288 --- Diff: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java --- @@ -447,7 +447,7 @@ static { Collections.sort(FUNCTION_KV_COLUMNS, KeyValue.COMPARATOR); } - + --- End diff -- I modified the class level comment. > Support multi region SYSTEM.CATALOG table > - > > Key: PHOENIX-3534 > URL: https://issues.apache.org/jira/browse/PHOENIX-3534 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: Thomas D'Silva >Priority: Major > Fix For: 5.0.0, 4.15.0 > > Attachments: PHOENIX-3534.patch > > > Currently Phoenix requires that the SYSTEM.CATALOG table is single region > based on the server-side row locks being held for operations that impact a > table and all of it's views. For example, adding/removing a column from a > base table pushes this change to all views. > As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), > when a new table is created we can do a lazy cleanup of any rows that may be > left over from a failed DDL call (kudos to [~lhofhansl] for coming up with > this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 > so that we can efficiently find derived views. > The implementation would rely on an optimistic concurrency model based on > checking our sequence numbers for each table/view before/after updating. Each > table/view row would be individually locked for their change (metadata for a > view or table cannot span regions due to our split policy), with the sequence > number being incremented under lock and then returned to the client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (PHOENIX-4808) Phoenix Spark Do not verify zKUrl in DefaultSource.scala
Lev Bronshtein created PHOENIX-4808: --- Summary: Phoenix Spark Do not verify zKUrl in DefaultSource.scala Key: PHOENIX-4808 URL: https://issues.apache.org/jira/browse/PHOENIX-4808 Project: Phoenix Issue Type: Improvement Reporter: Lev Bronshtein Currently verifyParameters will throw an exception _[zkUrl check|https://github.com/apache/phoenix/blob/master/phoenix-spark/src/main/scala/org/apache/phoenix/spark/DefaultSource.scala#L56]_ _if (parameters.get("zkUrl").isEmpty)_ however this is unnecessary as ultimately new PhoenixRelation(...) [PhoenixRelation|https://github.com/apache/phoenix/blob/master/phoenix-spark/src/main/scala/org/apache/phoenix/spark/DefaultSource.scala#L29] is called (see PhoenixRelation.scala) which in turn calls new PhoenixRDD(...) (See PhoenixRDD.scala). PhoenixRDD constructor declares zkUrl as optional. PhoenixRDD will try to use zkUrl and fall back on HBase Configuration {code:java} // Override the Zookeeper URL if present. Throw exception if no address given. zkUrl match { case Some(url) => ConfigurationUtil.setZookeeperURL(config, url) case _ => { if(ConfigurationUtil.getZookeeperURL(config).isEmpty) { throw new UnsupportedOperationException( s"One of zkUrl or '${HConstants.ZOOKEEPER_QUORUM}' config property must be provided" ) } } {code} I propose that this check is unnecessary and should be removed. As we will still get an exception, but a little later -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PHOENIX-4808) Phoenix Spark Do not verify zKUrl in DefaultSource.scala
[ https://issues.apache.org/jira/browse/PHOENIX-4808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lev Bronshtein updated PHOENIX-4808: Description: Currently verifyParameters will throw an exception _[zkUrl check|https://github.com/apache/phoenix/blob/master/phoenix-spark/src/main/scala/org/apache/phoenix/spark/DefaultSource.scala#L56]_ _if (parameters.get("zkUrl").isEmpty)_ however this is unnecessary as ultimately new PhoenixRelation(...) [PhoenixRelation|https://github.com/apache/phoenix/blob/master/phoenix-spark/src/main/scala/org/apache/phoenix/spark/DefaultSource.scala#L29] is called (see PhoenixRelation.scala) which in turn calls new PhoenixRDD(...) (See PhoenixRDD.scala). PhoenixRDD constructor declares zkUrl as optional. PhoenixRDD will try to use zkUrl and fall back on HBase Configuration {code:java} // Override the Zookeeper URL if present. Throw exception if no address given. zkUrl match { case Some(url) => ConfigurationUtil.setZookeeperURL(config, url) case _ => { if(ConfigurationUtil.getZookeeperURL(config).isEmpty) { throw new UnsupportedOperationException( s"One of zkUrl or '${HConstants.ZOOKEEPER_QUORUM}' config property must be provided" ) } } {code} I propose that this check is unnecessary and should be removed. As we will still get an exception, but a little later was: Currently verifyParameters will throw an exception _[zkUrl check|https://github.com/apache/phoenix/blob/master/phoenix-spark/src/main/scala/org/apache/phoenix/spark/DefaultSource.scala#L56]_ _if (parameters.get("zkUrl").isEmpty)_ however this is unnecessary as ultimately new PhoenixRelation(...) [PhoenixRelation|https://github.com/apache/phoenix/blob/master/phoenix-spark/src/main/scala/org/apache/phoenix/spark/DefaultSource.scala#L29] is called (see PhoenixRelation.scala) which in turn calls new PhoenixRDD(...) (See PhoenixRDD.scala). PhoenixRDD constructor declares zkUrl as optional. PhoenixRDD will try to use zkUrl and fall back on HBase Configuration {code:java} // Override the Zookeeper URL if present. Throw exception if no address given. zkUrl match { case Some(url) => ConfigurationUtil.setZookeeperURL(config, url) case _ => { if(ConfigurationUtil.getZookeeperURL(config).isEmpty) { throw new UnsupportedOperationException( s"One of zkUrl or '${HConstants.ZOOKEEPER_QUORUM}' config property must be provided" ) } } {code} I propose that this check is unnecessary and should be removed. As we will still get an exception, but a little later > Phoenix Spark Do not verify zKUrl in DefaultSource.scala > > > Key: PHOENIX-4808 > URL: https://issues.apache.org/jira/browse/PHOENIX-4808 > Project: Phoenix > Issue Type: Improvement >Reporter: Lev Bronshtein >Priority: Major > > Currently verifyParameters will throw an exception > _[zkUrl > check|https://github.com/apache/phoenix/blob/master/phoenix-spark/src/main/scala/org/apache/phoenix/spark/DefaultSource.scala#L56]_ > _if (parameters.get("zkUrl").isEmpty)_ however this is unnecessary as > ultimately new PhoenixRelation(...) > [PhoenixRelation|https://github.com/apache/phoenix/blob/master/phoenix-spark/src/main/scala/org/apache/phoenix/spark/DefaultSource.scala#L29] > is called (see PhoenixRelation.scala) which in turn calls new PhoenixRDD(...) > (See PhoenixRDD.scala). PhoenixRDD constructor declares zkUrl as optional. > PhoenixRDD will try to use zkUrl and fall back on HBase Configuration > > {code:java} > // Override the Zookeeper URL if present. Throw exception if no address given. > zkUrl match { > case Some(url) => ConfigurationUtil.setZookeeperURL(config, url) > case _ => { > if(ConfigurationUtil.getZookeeperURL(config).isEmpty) { > throw new UnsupportedOperationException( > s"One of zkUrl or '${HConstants.ZOOKEEPER_QUORUM}' config > property must be provided" > ) > } > } > {code} > > I propose that this check is unnecessary and should be removed. As we will > still get an exception, but a little later -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (PHOENIX-4809) connectionQueue never cleared in ConnectionQueryServicesImpl
Josh Elser created PHOENIX-4809: --- Summary: connectionQueue never cleared in ConnectionQueryServicesImpl Key: PHOENIX-4809 URL: https://issues.apache.org/jira/browse/PHOENIX-4809 Project: Phoenix Issue Type: Bug Reporter: Josh Elser Assignee: Josh Elser Fix For: 4.15.0, 5.1.0 When we create a new {{PhoenixConnection}}, we update {{connectionQueues}} in CQSI: {code:java} @Override public void addConnection(PhoenixConnection connection) throws SQLException { connectionQueues.get(getQueueIndex(connection)).add(new WeakReference(connection)); if (returnSequenceValues) { synchronized (connectionCountLock) { connectionCount++; } } }{code} We use connectionQueues to determine what needs lease renewal done. However, when the user closes a connection, this datastructure is never cleaned up. {code:java} @Override public void removeConnection(PhoenixConnection connection) throws SQLException { if (returnSequenceValues) { ConcurrentMap formerSequenceMap = null; synchronized (connectionCountLock) { if (--connectionCount <= 0) { if (!this.sequenceMap.isEmpty()) { formerSequenceMap = this.sequenceMap; this.sequenceMap = Maps.newConcurrentMap(); } } if (connectionCount < 0) { connectionCount = 0; } } // Since we're using the former sequenceMap, we can do this outside // the lock. if (formerSequenceMap != null) { // When there are no more connections, attempt to return any sequences returnAllSequences(formerSequenceMap); } } else if (shouldThrottleNumConnections){ //still need to decrement connection count synchronized (connectionCountLock) { if (connectionCount > 0) { --connectionCount; } } } }{code} Running a test now, but seems to be the case on master. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (PHOENIX-4810) Send parent->child link mutations to SYSTEM.CHILD_LINK table in MetdataClient.createTableInternal
Thomas D'Silva created PHOENIX-4810: --- Summary: Send parent->child link mutations to SYSTEM.CHILD_LINK table in MetdataClient.createTableInternal Key: PHOENIX-4810 URL: https://issues.apache.org/jira/browse/PHOENIX-4810 Project: Phoenix Issue Type: Sub-task Reporter: Thomas D'Silva Instead of sending the parent->child link mutations to the MetadataEndpointImpl.createTable write them directly to SYSTEM.CHILD_LINK -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] phoenix pull request #303: PHOENIX-3534 Support multi region SYSTEM.CATALOG ...
Github user twdsilva commented on a diff in the pull request: https://github.com/apache/phoenix/pull/303#discussion_r201799117 --- Diff: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java --- @@ -1779,13 +2174,65 @@ public void createTable(RpcController controller, CreateTableRequest request, } } +// The mutations to create a table are written in the following order: +// 1. Write the child link as if the next two steps fail we +// ignore missing children while processing a parent +// 2. Update the encoded column qualifier for the parent table if its on a +// different region server (for tables that use column qualifier encoding) +// if the next step fails we end up wasting a few col qualifiers +// 3. Finally write the mutations to create the table + +// From 4.15 the parent->child links are stored in a separate table SYSTEM.CHILD_LINK +List childLinkMutations = MetaDataUtil.removeChildLinks(tableMetadata); --- End diff -- I filed PHOENIX-4810 and added a comment to reference this jira. ---
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540498#comment-16540498 ] ASF GitHub Bot commented on PHOENIX-3534: - Github user twdsilva commented on a diff in the pull request: https://github.com/apache/phoenix/pull/303#discussion_r201799117 --- Diff: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java --- @@ -1779,13 +2174,65 @@ public void createTable(RpcController controller, CreateTableRequest request, } } +// The mutations to create a table are written in the following order: +// 1. Write the child link as if the next two steps fail we +// ignore missing children while processing a parent +// 2. Update the encoded column qualifier for the parent table if its on a +// different region server (for tables that use column qualifier encoding) +// if the next step fails we end up wasting a few col qualifiers +// 3. Finally write the mutations to create the table + +// From 4.15 the parent->child links are stored in a separate table SYSTEM.CHILD_LINK +List childLinkMutations = MetaDataUtil.removeChildLinks(tableMetadata); --- End diff -- I filed PHOENIX-4810 and added a comment to reference this jira. > Support multi region SYSTEM.CATALOG table > - > > Key: PHOENIX-3534 > URL: https://issues.apache.org/jira/browse/PHOENIX-3534 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: Thomas D'Silva >Priority: Major > Fix For: 5.0.0, 4.15.0 > > Attachments: PHOENIX-3534.patch > > > Currently Phoenix requires that the SYSTEM.CATALOG table is single region > based on the server-side row locks being held for operations that impact a > table and all of it's views. For example, adding/removing a column from a > base table pushes this change to all views. > As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), > when a new table is created we can do a lazy cleanup of any rows that may be > left over from a failed DDL call (kudos to [~lhofhansl] for coming up with > this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 > so that we can efficiently find derived views. > The implementation would rely on an optimistic concurrency model based on > checking our sequence numbers for each table/view before/after updating. Each > table/view row would be individually locked for their change (metadata for a > view or table cannot span regions due to our split policy), with the sequence > number being incremented under lock and then returned to the client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] phoenix pull request #303: PHOENIX-3534 Support multi region SYSTEM.CATALOG ...
Github user twdsilva commented on a diff in the pull request: https://github.com/apache/phoenix/pull/303#discussion_r201800952 --- Diff: phoenix-core/src/main/java/org/apache/phoenix/util/UpgradeUtil.java --- @@ -1893,26 +1981,45 @@ public static void upgradeTable(PhoenixConnection conn, String srcTable) throws conn.commit(); } conn.getQueryServices().clearTableFromCache( -conn.getTenantId() == null ? ByteUtil.EMPTY_BYTE_ARRAY : conn.getTenantId().getBytes(), +tenantIdBytes, index.getSchemaName().getBytes(), index.getTableName().getBytes(), PhoenixRuntime.getCurrentScn(readOnlyProps)); } updateIndexesSequenceIfPresent(conn, table); conn.commit(); - } else { throw new RuntimeException("Error: problem occured during upgrade. Table is not upgraded successfully"); } if (table.getType() == PTableType.VIEW) { logger.info(String.format("Updating link information for view '%s' ..", table.getTableName())); updateLink(conn, oldPhysicalName, newPhysicalTablename,table.getSchemaName(),table.getTableName()); conn.commit(); - + +// if the view is a first level child, then we need to create the PARENT_TABLE link +// that was overwritten by the PHYSICAL_TABLE link --- End diff -- Yes this will make it so that the parent link row will be created correctly when upgrading tables to be namespace mapped. ---
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540504#comment-16540504 ] ASF GitHub Bot commented on PHOENIX-3534: - Github user twdsilva commented on a diff in the pull request: https://github.com/apache/phoenix/pull/303#discussion_r201800952 --- Diff: phoenix-core/src/main/java/org/apache/phoenix/util/UpgradeUtil.java --- @@ -1893,26 +1981,45 @@ public static void upgradeTable(PhoenixConnection conn, String srcTable) throws conn.commit(); } conn.getQueryServices().clearTableFromCache( -conn.getTenantId() == null ? ByteUtil.EMPTY_BYTE_ARRAY : conn.getTenantId().getBytes(), +tenantIdBytes, index.getSchemaName().getBytes(), index.getTableName().getBytes(), PhoenixRuntime.getCurrentScn(readOnlyProps)); } updateIndexesSequenceIfPresent(conn, table); conn.commit(); - } else { throw new RuntimeException("Error: problem occured during upgrade. Table is not upgraded successfully"); } if (table.getType() == PTableType.VIEW) { logger.info(String.format("Updating link information for view '%s' ..", table.getTableName())); updateLink(conn, oldPhysicalName, newPhysicalTablename,table.getSchemaName(),table.getTableName()); conn.commit(); - + +// if the view is a first level child, then we need to create the PARENT_TABLE link +// that was overwritten by the PHYSICAL_TABLE link --- End diff -- Yes this will make it so that the parent link row will be created correctly when upgrading tables to be namespace mapped. > Support multi region SYSTEM.CATALOG table > - > > Key: PHOENIX-3534 > URL: https://issues.apache.org/jira/browse/PHOENIX-3534 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: Thomas D'Silva >Priority: Major > Fix For: 5.0.0, 4.15.0 > > Attachments: PHOENIX-3534.patch > > > Currently Phoenix requires that the SYSTEM.CATALOG table is single region > based on the server-side row locks being held for operations that impact a > table and all of it's views. For example, adding/removing a column from a > base table pushes this change to all views. > As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), > when a new table is created we can do a lazy cleanup of any rows that may be > left over from a failed DDL call (kudos to [~lhofhansl] for coming up with > this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 > so that we can efficiently find derived views. > The implementation would rely on an optimistic concurrency model based on > checking our sequence numbers for each table/view before/after updating. Each > table/view row would be individually locked for their change (metadata for a > view or table cannot span regions due to our split policy), with the sequence > number being incremented under lock and then returned to the client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-4809) connectionQueue never cleared in ConnectionQueryServicesImpl
[ https://issues.apache.org/jira/browse/PHOENIX-4809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540555#comment-16540555 ] Josh Elser commented on PHOENIX-4809: - I ran a quick test locally, the odd thing is that I see fewer {{WeakReference}} objects than I'd expect.. {code:java} public class PhoenixQuery { private static final String URL = "jdbc:phoenix:localhost:2181:/hbase-1.4"; public static void main(String[] args) throws Exception { ExecutorService svc = Executors.newFixedThreadPool(8); try (Connection conn = DriverManager.getConnection(URL, "", "")) { PhoenixConnection phxConn = conn.unwrap(PhoenixConnection.class); ConnectionQueryServicesImpl queryServices = (ConnectionQueryServicesImpl) phxConn.getQueryServices(); System.out.println(numConnectionsCached(queryServices)); } CountDownLatch latch = new CountDownLatch(8); for (int i = 0; i < 8; i++) { svc.submit(new Runnable() { @Override public void run() { for (int j = 0; j < 1_000_000; j++) { try (Connection conn = DriverManager.getConnection(URL, "", "")) { conn.close(); } catch (Exception e) { System.out.println("Caught error"); } } latch.countDown(); } }); } latch.await(); try (Connection conn = DriverManager.getConnection(URL, "", "")) { PhoenixConnection phxConn = conn.unwrap(PhoenixConnection.class); ConnectionQueryServicesImpl queryServices = (ConnectionQueryServicesImpl) phxConn.getQueryServices(); System.out.println(numConnectionsCached(queryServices)); } } private static long numConnectionsCached(ConnectionQueryServicesImpl cqs) { try { Field f = ConnectionQueryServicesImpl.class.getDeclaredField("connectionQueues"); f.setAccessible(true); List>> list = (List>>) f.get(cqs); return list.stream().collect(Collectors.summingLong(LinkedBlockingQueue::size)); } catch (Exception e) { e.printStackTrace(System.err); throw new RuntimeException(e); } } }{code} I'd expect to see 8M objects cached, but I'm actually seeing 615,696. Something must clean this up somehow.. Maybe related to lease renewal not being on? > connectionQueue never cleared in ConnectionQueryServicesImpl > > > Key: PHOENIX-4809 > URL: https://issues.apache.org/jira/browse/PHOENIX-4809 > Project: Phoenix > Issue Type: Bug >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: 4.15.0, 5.1.0 > > > When we create a new {{PhoenixConnection}}, we update {{connectionQueues}} in > CQSI: > {code:java} > @Override > public void addConnection(PhoenixConnection connection) throws > SQLException { > connectionQueues.get(getQueueIndex(connection)).add(new > WeakReference(connection)); > if (returnSequenceValues) { > synchronized (connectionCountLock) { > connectionCount++; > } > } > }{code} > We use connectionQueues to determine what needs lease renewal done. > However, when the user closes a connection, this datastructure is never > cleaned up. > {code:java} > @Override > public void removeConnection(PhoenixConnection connection) throws > SQLException { > if (returnSequenceValues) { > ConcurrentMap formerSequenceMap = null; > synchronized (connectionCountLock) { > if (--connectionCount <= 0) { > if (!this.sequenceMap.isEmpty()) { > formerSequenceMap = this.sequenceMap; > this.sequenceMap = Maps.newConcurrentMap(); > } > } > if (connectionCount < 0) { > connectionCount = 0; > } > } > // Since we're using the former sequenceMap, we can do this > outside > // the lock. > if (formerSequenceMap != null) { > // When there are no more connections, attempt to return any > sequences > returnAllSequences(formerSequenceMap); > } > } else if (shouldThrottleNumConnections){ //still need to decrement > connection count > synchronized (connectionCountLock) { > if (connectionCount > 0) { > --connectionCount; > } > } > } > }{code} > Running a test now, but seems to be the case on master. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-4809) connectionQueue never cleared in ConnectionQueryServicesImpl
[ https://issues.apache.org/jira/browse/PHOENIX-4809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540557#comment-16540557 ] Josh Elser commented on PHOENIX-4809: - {quote}Maybe related to lease renewal not being on? {quote} Ah, yeah, that's the rub. If the RenewLeaseTask doesn't run, nothing drains these queues. > connectionQueue never cleared in ConnectionQueryServicesImpl > > > Key: PHOENIX-4809 > URL: https://issues.apache.org/jira/browse/PHOENIX-4809 > Project: Phoenix > Issue Type: Bug >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: 4.15.0, 5.1.0 > > > When we create a new {{PhoenixConnection}}, we update {{connectionQueues}} in > CQSI: > {code:java} > @Override > public void addConnection(PhoenixConnection connection) throws > SQLException { > connectionQueues.get(getQueueIndex(connection)).add(new > WeakReference(connection)); > if (returnSequenceValues) { > synchronized (connectionCountLock) { > connectionCount++; > } > } > }{code} > We use connectionQueues to determine what needs lease renewal done. > However, when the user closes a connection, this datastructure is never > cleaned up. > {code:java} > @Override > public void removeConnection(PhoenixConnection connection) throws > SQLException { > if (returnSequenceValues) { > ConcurrentMap formerSequenceMap = null; > synchronized (connectionCountLock) { > if (--connectionCount <= 0) { > if (!this.sequenceMap.isEmpty()) { > formerSequenceMap = this.sequenceMap; > this.sequenceMap = Maps.newConcurrentMap(); > } > } > if (connectionCount < 0) { > connectionCount = 0; > } > } > // Since we're using the former sequenceMap, we can do this > outside > // the lock. > if (formerSequenceMap != null) { > // When there are no more connections, attempt to return any > sequences > returnAllSequences(formerSequenceMap); > } > } else if (shouldThrottleNumConnections){ //still need to decrement > connection count > synchronized (connectionCountLock) { > if (connectionCount > 0) { > --connectionCount; > } > } > } > }{code} > Running a test now, but seems to be the case on master. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PHOENIX-4809) connectionQueue never cleared in ConnectionQueryServicesImpl when lease renewal is disabled/unsupported
[ https://issues.apache.org/jira/browse/PHOENIX-4809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated PHOENIX-4809: Summary: connectionQueue never cleared in ConnectionQueryServicesImpl when lease renewal is disabled/unsupported (was: connectionQueue never cleared in ConnectionQueryServicesImpl) > connectionQueue never cleared in ConnectionQueryServicesImpl when lease > renewal is disabled/unsupported > --- > > Key: PHOENIX-4809 > URL: https://issues.apache.org/jira/browse/PHOENIX-4809 > Project: Phoenix > Issue Type: Bug >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: 4.15.0, 5.1.0 > > > When we create a new {{PhoenixConnection}}, we update {{connectionQueues}} in > CQSI: > {code:java} > @Override > public void addConnection(PhoenixConnection connection) throws > SQLException { > connectionQueues.get(getQueueIndex(connection)).add(new > WeakReference(connection)); > if (returnSequenceValues) { > synchronized (connectionCountLock) { > connectionCount++; > } > } > }{code} > We use connectionQueues to determine what needs lease renewal done. > However, when the user closes a connection, this datastructure is never > cleaned up. > {code:java} > @Override > public void removeConnection(PhoenixConnection connection) throws > SQLException { > if (returnSequenceValues) { > ConcurrentMap formerSequenceMap = null; > synchronized (connectionCountLock) { > if (--connectionCount <= 0) { > if (!this.sequenceMap.isEmpty()) { > formerSequenceMap = this.sequenceMap; > this.sequenceMap = Maps.newConcurrentMap(); > } > } > if (connectionCount < 0) { > connectionCount = 0; > } > } > // Since we're using the former sequenceMap, we can do this > outside > // the lock. > if (formerSequenceMap != null) { > // When there are no more connections, attempt to return any > sequences > returnAllSequences(formerSequenceMap); > } > } else if (shouldThrottleNumConnections){ //still need to decrement > connection count > synchronized (connectionCountLock) { > if (connectionCount > 0) { > --connectionCount; > } > } > } > }{code} > Running a test now, but seems to be the case on master. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Videos from HBaseCon and PhoenixCon 2018
Looks like we missed one the first time around (sorry, Scott!) https://www.youtube.com/watch?v=hbLYt4dUHp8 On 7/6/18 1:36 PM, Josh Elser wrote: Hi all, As enabled by one of the event sponsors (Bloomberg -- big thanks to them), we have some professional recordings of all of the talks that were given on June 18th in San Jose, CA. I've compiled them and uploaded them to Youtube: https://www.youtube.com/playlist?list=PLdOhKvENNn6WH71iHXRNbEd1pTN2xY6P3 Each video is the presentation view and the audio from the speaker. We did not have cameras on each speaker during the talk. We also did not curate the slides that accompanied each talks -- please follow up with the speaker if you'd like a copy. I mention that I personally compiled them to outline that I may have fat-fingered something :). Please let me know if you see any typos or similar. I'll be working on updating the HBase and Phoenix websites to reflect the content later. - JOsh
[jira] [Commented] (PHOENIX-4793) Backport PHOENIX-4303, PHOENIX-4305, PHOENIX-4321 to 4.x branches.
[ https://issues.apache.org/jira/browse/PHOENIX-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540609#comment-16540609 ] James Taylor commented on PHOENIX-4793: --- Would it be possible for you to do this one after PHOENIX-3534 makes it in in the next couple of days, [~chrajeshbab...@gmail.com] ? > Backport PHOENIX-4303, PHOENIX-4305, PHOENIX-4321 to 4.x branches. > -- > > Key: PHOENIX-4793 > URL: https://issues.apache.org/jira/browse/PHOENIX-4793 > Project: Phoenix > Issue Type: Bug >Reporter: Rajeshbabu Chintaguntla >Assignee: Rajeshbabu Chintaguntla >Priority: Major > Fix For: 4.15.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540649#comment-16540649 ] ASF GitHub Bot commented on PHOENIX-3534: - Github user twdsilva commented on a diff in the pull request: https://github.com/apache/phoenix/pull/303#discussion_r201835494 --- Diff: phoenix-core/src/it/java/org/apache/phoenix/end2end/ViewIT.java --- @@ -372,6 +378,31 @@ public void testViewAndTableAndDrop() throws Exception { // drop table cascade should succeed conn.createStatement().execute("DROP TABLE " + fullTableName + " CASCADE"); +validateViewDoesNotExist(conn, fullViewName1); +validateViewDoesNotExist(conn, fullViewName2); + +} + +@Test +public void testRecreateDroppedTableWithChildViews() throws Exception { --- End diff -- We write the parent->child link first, then if the table uses column encoding we update the encoded column qualifiers on the parent table, and finally use mutateRowsWithLocks to write the view metadata atomically. We ignore views that can't be found (in case writing the child view metadata fails). If the metadata write fails and the table uses column encoding then we will lose a few column qualifiers. I'll add a test for this. > Support multi region SYSTEM.CATALOG table > - > > Key: PHOENIX-3534 > URL: https://issues.apache.org/jira/browse/PHOENIX-3534 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: Thomas D'Silva >Priority: Major > Fix For: 5.0.0, 4.15.0 > > Attachments: PHOENIX-3534.patch > > > Currently Phoenix requires that the SYSTEM.CATALOG table is single region > based on the server-side row locks being held for operations that impact a > table and all of it's views. For example, adding/removing a column from a > base table pushes this change to all views. > As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), > when a new table is created we can do a lazy cleanup of any rows that may be > left over from a failed DDL call (kudos to [~lhofhansl] for coming up with > this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 > so that we can efficiently find derived views. > The implementation would rely on an optimistic concurrency model based on > checking our sequence numbers for each table/view before/after updating. Each > table/view row would be individually locked for their change (metadata for a > view or table cannot span regions due to our split policy), with the sequence > number being incremented under lock and then returned to the client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] phoenix pull request #303: PHOENIX-3534 Support multi region SYSTEM.CATALOG ...
Github user twdsilva commented on a diff in the pull request: https://github.com/apache/phoenix/pull/303#discussion_r201835494 --- Diff: phoenix-core/src/it/java/org/apache/phoenix/end2end/ViewIT.java --- @@ -372,6 +378,31 @@ public void testViewAndTableAndDrop() throws Exception { // drop table cascade should succeed conn.createStatement().execute("DROP TABLE " + fullTableName + " CASCADE"); +validateViewDoesNotExist(conn, fullViewName1); +validateViewDoesNotExist(conn, fullViewName2); + +} + +@Test +public void testRecreateDroppedTableWithChildViews() throws Exception { --- End diff -- We write the parent->child link first, then if the table uses column encoding we update the encoded column qualifiers on the parent table, and finally use mutateRowsWithLocks to write the view metadata atomically. We ignore views that can't be found (in case writing the child view metadata fails). If the metadata write fails and the table uses column encoding then we will lose a few column qualifiers. I'll add a test for this. ---
[jira] [Updated] (PHOENIX-4809) connectionQueue never cleared in ConnectionQueryServicesImpl when lease renewal is disabled/unsupported
[ https://issues.apache.org/jira/browse/PHOENIX-4809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated PHOENIX-4809: Attachment: PHOENIX-4809.001.patch > connectionQueue never cleared in ConnectionQueryServicesImpl when lease > renewal is disabled/unsupported > --- > > Key: PHOENIX-4809 > URL: https://issues.apache.org/jira/browse/PHOENIX-4809 > Project: Phoenix > Issue Type: Bug >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: 4.15.0, 5.1.0 > > Attachments: PHOENIX-4809.001.patch > > > When we create a new {{PhoenixConnection}}, we update {{connectionQueues}} in > CQSI: > {code:java} > @Override > public void addConnection(PhoenixConnection connection) throws > SQLException { > connectionQueues.get(getQueueIndex(connection)).add(new > WeakReference(connection)); > if (returnSequenceValues) { > synchronized (connectionCountLock) { > connectionCount++; > } > } > }{code} > We use connectionQueues to determine what needs lease renewal done. > However, when the user closes a connection, this datastructure is never > cleaned up. > {code:java} > @Override > public void removeConnection(PhoenixConnection connection) throws > SQLException { > if (returnSequenceValues) { > ConcurrentMap formerSequenceMap = null; > synchronized (connectionCountLock) { > if (--connectionCount <= 0) { > if (!this.sequenceMap.isEmpty()) { > formerSequenceMap = this.sequenceMap; > this.sequenceMap = Maps.newConcurrentMap(); > } > } > if (connectionCount < 0) { > connectionCount = 0; > } > } > // Since we're using the former sequenceMap, we can do this > outside > // the lock. > if (formerSequenceMap != null) { > // When there are no more connections, attempt to return any > sequences > returnAllSequences(formerSequenceMap); > } > } else if (shouldThrottleNumConnections){ //still need to decrement > connection count > synchronized (connectionCountLock) { > if (connectionCount > 0) { > --connectionCount; > } > } > } > }{code} > Running a test now, but seems to be the case on master. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (PHOENIX-4807) Document query behavior when statistics unavailable for some table regions.
[ https://issues.apache.org/jira/browse/PHOENIX-4807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Soldatov resolved PHOENIX-4807. -- Resolution: Fixed > Document query behavior when statistics unavailable for some table regions. > --- > > Key: PHOENIX-4807 > URL: https://issues.apache.org/jira/browse/PHOENIX-4807 > Project: Phoenix > Issue Type: Task >Reporter: Guru Prateek Pinnadhari >Assignee: Sergey Soldatov >Priority: Major > > When troubleshooting an issue with retrieval of duplicate records for a > select query on same PK, the behavior seen was: query plan's collection of > Scan objects were such that they led to generation of overlapping scans on a > salted table's regions. > A contributing factor was statistics being unavailable for the terminal > regions of the table. Filing Jira to add a note about this on > [https://phoenix.apache.org/update_statistics.html] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-4807) Document query behavior when statistics unavailable for some table regions.
[ https://issues.apache.org/jira/browse/PHOENIX-4807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540663#comment-16540663 ] Sergey Soldatov commented on PHOENIX-4807: -- I've updated the page with a known issues section. Not sure whether a real example is required, because the end user usually doesn't care about how it works under the hood. > Document query behavior when statistics unavailable for some table regions. > --- > > Key: PHOENIX-4807 > URL: https://issues.apache.org/jira/browse/PHOENIX-4807 > Project: Phoenix > Issue Type: Task >Reporter: Guru Prateek Pinnadhari >Assignee: Sergey Soldatov >Priority: Major > > When troubleshooting an issue with retrieval of duplicate records for a > select query on same PK, the behavior seen was: query plan's collection of > Scan objects were such that they led to generation of overlapping scans on a > salted table's regions. > A contributing factor was statistics being unavailable for the terminal > regions of the table. Filing Jira to add a note about this on > [https://phoenix.apache.org/update_statistics.html] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-4809) connectionQueue never cleared in ConnectionQueryServicesImpl when lease renewal is disabled/unsupported
[ https://issues.apache.org/jira/browse/PHOENIX-4809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540666#comment-16540666 ] Ankit Singhal commented on PHOENIX-4809: +1 [~elserj], Thanks for the test as well :) > connectionQueue never cleared in ConnectionQueryServicesImpl when lease > renewal is disabled/unsupported > --- > > Key: PHOENIX-4809 > URL: https://issues.apache.org/jira/browse/PHOENIX-4809 > Project: Phoenix > Issue Type: Bug >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: 4.15.0, 5.1.0 > > Attachments: PHOENIX-4809.001.patch > > > When we create a new {{PhoenixConnection}}, we update {{connectionQueues}} in > CQSI: > {code:java} > @Override > public void addConnection(PhoenixConnection connection) throws > SQLException { > connectionQueues.get(getQueueIndex(connection)).add(new > WeakReference(connection)); > if (returnSequenceValues) { > synchronized (connectionCountLock) { > connectionCount++; > } > } > }{code} > We use connectionQueues to determine what needs lease renewal done. > However, when the user closes a connection, this datastructure is never > cleaned up. > {code:java} > @Override > public void removeConnection(PhoenixConnection connection) throws > SQLException { > if (returnSequenceValues) { > ConcurrentMap formerSequenceMap = null; > synchronized (connectionCountLock) { > if (--connectionCount <= 0) { > if (!this.sequenceMap.isEmpty()) { > formerSequenceMap = this.sequenceMap; > this.sequenceMap = Maps.newConcurrentMap(); > } > } > if (connectionCount < 0) { > connectionCount = 0; > } > } > // Since we're using the former sequenceMap, we can do this > outside > // the lock. > if (formerSequenceMap != null) { > // When there are no more connections, attempt to return any > sequences > returnAllSequences(formerSequenceMap); > } > } else if (shouldThrottleNumConnections){ //still need to decrement > connection count > synchronized (connectionCountLock) { > if (connectionCount > 0) { > --connectionCount; > } > } > } > }{code} > Running a test now, but seems to be the case on master. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-4805) Move Avatica version to 1.12 for PQS
[ https://issues.apache.org/jira/browse/PHOENIX-4805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540881#comment-16540881 ] Karan Mehta commented on PHOENIX-4805: -- Looks like [https://builds.apache.org/job/PreCommit-PHOENIX-Build/1917/console] has completed the tests successfully, buy it failed when trying to post Jira comment for it. Will commit to 4.x and master branches. [~elserj] any thing else over here? > Move Avatica version to 1.12 for PQS > > > Key: PHOENIX-4805 > URL: https://issues.apache.org/jira/browse/PHOENIX-4805 > Project: Phoenix > Issue Type: Improvement >Reporter: Karan Mehta >Assignee: Karan Mehta >Priority: Major > Attachments: PHOENIX-4805.001.patch > > > CALCITE-2330 is now resolved. We can use the new version for supporting new > features such as MTLS authentication and server customization in PQS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hadoop QA updated PHOENIX-3534: --- Attachment: PHOENIX-3534-v2.patch > Support multi region SYSTEM.CATALOG table > - > > Key: PHOENIX-3534 > URL: https://issues.apache.org/jira/browse/PHOENIX-3534 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: Thomas D'Silva >Priority: Major > Fix For: 5.0.0, 4.15.0 > > Attachments: PHOENIX-3534-v2.patch, PHOENIX-3534.patch > > > Currently Phoenix requires that the SYSTEM.CATALOG table is single region > based on the server-side row locks being held for operations that impact a > table and all of it's views. For example, adding/removing a column from a > base table pushes this change to all views. > As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), > when a new table is created we can do a lazy cleanup of any rows that may be > left over from a failed DDL call (kudos to [~lhofhansl] for coming up with > this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 > so that we can efficiently find derived views. > The implementation would rely on an optimistic concurrency model based on > checking our sequence numbers for each table/view before/after updating. Each > table/view row would be individually locked for their change (metadata for a > view or table cannot span regions due to our split policy), with the sequence > number being incremented under lock and then returned to the client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: [VOTE] stop minor releases for 0.98 and 1.1
+1 On Tue, Jul 10, 2018 at 1:12 PM, Ankit Singhal wrote: > +1 > > On Tue, Jul 10, 2018 at 12:22 PM Josh Elser wrote: > > > +1 > > > > On 7/9/18 7:03 PM, James Taylor wrote: > > > +1 > > > > > > On Mon, Jul 9, 2018 at 3:55 PM, Vincent Poon < > vincent.poon...@gmail.com> > > > wrote: > > > > > >> This is a call to vote on stopping maintenance of both the > > 4.x-HBase-0.98 > > >> and 4.x-HBase-1.1 branches. > > >> HBase 0.98 and 1.1 have been EOLed. To commit currently, it's usually > > 7 or > > >> more branches, which is a burden on devs. > > >> > > >> Vote will be open for at least 72 hours. Please vote: > > >> > > >> [ ] +1 approve > > >> [ ] +0 no opinion > > >> [ ] -1 disapprove (and reason why) > > >> > > >> Thanks, > > >> The Apache Phoenix Team > > >> > > > > > >
[jira] [Commented] (PHOENIX-4790) Simplify check for client side delete
[ https://issues.apache.org/jira/browse/PHOENIX-4790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540953#comment-16540953 ] Thomas D'Silva commented on PHOENIX-4790: - [~jamestaylor] I think this patch is causing tests to fail : [BinaryRowKeyIT|eclipse-javadoc:%E2%98%82=phoenix-core/src%5C/it%5C/java%3Corg.apache.phoenix.end2end%7BBinaryRowKeyIT.java%E2%98%83BinaryRowKeyIT].testInsertPaddedBinaryValue() and [ArithmeticQueryIT|eclipse-javadoc:%E2%98%82=phoenix-core/src%5C/it%5C/java%3Corg.apache.phoenix.end2end%7BArithmeticQueryIT.java%E2%98%83ArithmeticQueryIT].testRandomFunction() > Simplify check for client side delete > - > > Key: PHOENIX-4790 > URL: https://issues.apache.org/jira/browse/PHOENIX-4790 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: James Taylor >Priority: Minor > Fix For: 4.15.0, 5.1.0 > > Attachments: PHOENIX-4790.patch > > > We don't need to check every query plan for the existence of a where clause > to determine if we can do a client-side delete. Instead, we can simply check > if any non PK columns are projected. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (PHOENIX-4790) Simplify check for client side delete
[ https://issues.apache.org/jira/browse/PHOENIX-4790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540953#comment-16540953 ] Thomas D'Silva edited comment on PHOENIX-4790 at 7/12/18 12:59 AM: --- [~jamestaylor] I think this patch is causing tests to fail : BinaryRowKeyIT.testInsertPaddedBinaryValue() ArithmeticQueryIT.testRandomFunction() was (Author: tdsilva): [~jamestaylor] I think this patch is causing tests to fail : [BinaryRowKeyIT|eclipse-javadoc:%E2%98%82=phoenix-core/src%5C/it%5C/java%3Corg.apache.phoenix.end2end%7BBinaryRowKeyIT.java%E2%98%83BinaryRowKeyIT].testInsertPaddedBinaryValue() and [ArithmeticQueryIT|eclipse-javadoc:%E2%98%82=phoenix-core/src%5C/it%5C/java%3Corg.apache.phoenix.end2end%7BArithmeticQueryIT.java%E2%98%83ArithmeticQueryIT].testRandomFunction() > Simplify check for client side delete > - > > Key: PHOENIX-4790 > URL: https://issues.apache.org/jira/browse/PHOENIX-4790 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: James Taylor >Priority: Minor > Fix For: 4.15.0, 5.1.0 > > Attachments: PHOENIX-4790.patch > > > We don't need to check every query plan for the existence of a where clause > to determine if we can do a client-side delete. Instead, we can simply check > if any non PK columns are projected. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas D'Silva updated PHOENIX-3534: Attachment: PHOENIX-3534-v3.patch > Support multi region SYSTEM.CATALOG table > - > > Key: PHOENIX-3534 > URL: https://issues.apache.org/jira/browse/PHOENIX-3534 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: Thomas D'Silva >Priority: Major > Fix For: 5.0.0, 4.15.0 > > Attachments: PHOENIX-3534-v2.patch, PHOENIX-3534-v3.patch, > PHOENIX-3534.patch > > > Currently Phoenix requires that the SYSTEM.CATALOG table is single region > based on the server-side row locks being held for operations that impact a > table and all of it's views. For example, adding/removing a column from a > base table pushes this change to all views. > As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), > when a new table is created we can do a lazy cleanup of any rows that may be > left over from a failed DDL call (kudos to [~lhofhansl] for coming up with > this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 > so that we can efficiently find derived views. > The implementation would rely on an optimistic concurrency model based on > checking our sequence numbers for each table/view before/after updating. Each > table/view row would be individually locked for their change (metadata for a > view or table cannot span regions due to our split policy), with the sequence > number being incremented under lock and then returned to the client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[DISCUSS] reduce notifications to phoenix-dev list
I think we should reduce the number of notifications that are sent to the phoenix-dev list. The notification scheme we use is located here : https://issues.apache.org/ jira/plugins/servlet/project-config/PHOENIX/notifications Maybe we don't need to notify the dev list when an issue is updated, or a comment is created/edited/deleted. People watching a particular issue would still see all the changes. This would be similar to how other projects like HBase and Calcite operate. Thanks, Thomas
[jira] [Created] (PHOENIX-4811) Phoenix Secondary Index isn't be used
张延召 created PHOENIX-4811: Summary: Phoenix Secondary Index isn't be used Key: PHOENIX-4811 URL: https://issues.apache.org/jira/browse/PHOENIX-4811 Project: Phoenix Issue Type: Bug Affects Versions: 4.13.1 Environment: Operating System : centos6.9 ; Phoenix version : 4.13.1; HBase version : 1.3.1; Reporter: 张延召 I created two indexes (not Covered Indexes): ^create table test(id varchar not null primary key,name varchar,age varchar ,sex varchar);^ ^create local index if not exists val_test_name on test(name);^ ^create local index if not exists val_test_age on test(age);^ ^upsert into test(id,name,age,sex)values('myid','name','age','sex');^ It is work to use one index: ^explain select name from test where name = 'name';^ But why can't hit index when using multiple indexes? ^explain select name,age from test where name = 'name' and age = 'age';^ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (PHOENIX-4811) Phoenix Secondary Index isn't be used
[ https://issues.apache.org/jira/browse/PHOENIX-4811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas D'Silva resolved PHOENIX-4811. - Resolution: Not A Problem The index stores data sorted on the indexed columns. If your query filters on name and age and you want to use the index create the index with (name, age) or (age, name) > Phoenix Secondary Index isn't be used > - > > Key: PHOENIX-4811 > URL: https://issues.apache.org/jira/browse/PHOENIX-4811 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.13.1 > Environment: Operating System : centos6.9 ; > Phoenix version : 4.13.1; > HBase version : 1.3.1; >Reporter: 张延召 >Priority: Major > Labels: SecondaryIndex > Original Estimate: 48h > Remaining Estimate: 48h > > I created two indexes (not Covered Indexes): > ^create table test(id varchar not null primary key,name varchar,age varchar > ,sex varchar);^ > ^create local index if not exists val_test_name on test(name);^ > ^create local index if not exists val_test_age on test(age);^ > ^upsert into test(id,name,age,sex)values('myid','name','age','sex');^ > It is work to use one index: > ^explain select name from test where name = 'name';^ > But why can't hit index when using multiple indexes? > ^explain select name,age from test where name = 'name' and age = 'age';^ > > > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-4790) Simplify check for client side delete
[ https://issues.apache.org/jira/browse/PHOENIX-4790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541149#comment-16541149 ] James Taylor commented on PHOENIX-4790: --- Sorry about that, [~tdsilva]. I've attached an addendum which I'll commit unless I hear objections. > Simplify check for client side delete > - > > Key: PHOENIX-4790 > URL: https://issues.apache.org/jira/browse/PHOENIX-4790 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: James Taylor >Priority: Minor > Fix For: 4.15.0, 5.1.0 > > Attachments: PHOENIX-4790.patch, PHOENIX-4790_addendum1.patch > > > We don't need to check every query plan for the existence of a where clause > to determine if we can do a client-side delete. Instead, we can simply check > if any non PK columns are projected. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PHOENIX-4790) Simplify check for client side delete
[ https://issues.apache.org/jira/browse/PHOENIX-4790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated PHOENIX-4790: -- Attachment: PHOENIX-4790_addendum1.patch > Simplify check for client side delete > - > > Key: PHOENIX-4790 > URL: https://issues.apache.org/jira/browse/PHOENIX-4790 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: James Taylor >Priority: Minor > Fix For: 4.15.0, 5.1.0 > > Attachments: PHOENIX-4790.patch, PHOENIX-4790_addendum1.patch > > > We don't need to check every query plan for the existence of a where clause > to determine if we can do a client-side delete. Instead, we can simply check > if any non PK columns are projected. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541179#comment-16541179 ] Hadoop QA commented on PHOENIX-3534: {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12931252/PHOENIX-3534-v3.patch against master branch at commit aee568beb02cdf983bb10889902c338ea016e6c9. ATTACHMENT ID: 12931252 {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 29 new or modified tests. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:red}-1 release audit{color}. The applied patch generated 3 release audit warnings (more than the master's current 0 warnings). {color:red}-1 lineLengths{color}. The patch introduces the following lines longer than 100: +conn.createStatement().execute("CREATE LOCAL INDEX " + generateUniqueName() + " ON " + tableName + "(KV1)"); +assertEquals(4, getIndexOfPkColumn(phxConn, IndexUtil.getIndexColumnName(null, "k2"), fullView2IndexName)); +assertEquals(5, getIndexOfPkColumn(phxConn, IndexUtil.getIndexColumnName(null, "k3"), fullView2IndexName)); +assertEquals(4, getIndexOfPkColumn(phxConn, IndexUtil.getIndexColumnName(null, "k2"), fullView3IndexName)); +assertEquals(5, getIndexOfPkColumn(phxConn, IndexUtil.getIndexColumnName(null, "k3"), fullView3IndexName)); +String view1DDL = "CREATE VIEW " + view1 + " ( VIEW_COL1 DECIMAL(10,2), VIEW_COL2 CHAR(256)) AS SELECT * FROM " + baseTable; +String divergedViewDDL = "CREATE VIEW " + divergedView + " ( VIEW_COL1 DECIMAL(10,2), VIEW_COL2 CHAR(256)) AS SELECT * FROM " + baseTable; +String indexDDL = "CREATE INDEX " + divergedViewIndex + " ON " + divergedView + " (V1) include (V3)"; +assertTableDefinition(tenant1Conn, view1, PTableType.VIEW, baseTable, 0, 7, 5, "PK1", "V1", "V2", "V3", "KV", "PK2", "VIEW_COL1", "VIEW_COL2"); +assertTableDefinition(tenant2Conn, divergedView, PTableType.VIEW, baseTable, 1, 6, DIVERGED_VIEW_BASE_COLUMN_COUNT, "PK1", "V1", "V3", "PK2", "VIEW_COL1", "VIEW_COL2"); {color:red}-1 core tests{color}. The patch failed these unit tests: ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.index.MutableIndexIT {color:red}-1 core zombie tests{color}. There are 4 zombie test(s): at org.apache.phoenix.end2end.DeleteIT.testPointDeleteRowFromTableWithImmutableIndex(DeleteIT.java:403) at org.apache.phoenix.end2end.DeleteIT.testPointDeleteRowFromTableWithImmutableIndex(DeleteIT.java:376) at org.apache.phoenix.end2end.DefaultColumnValueIT.testDefaultColumnValue(DefaultColumnValueIT.java:66) Test results: https://builds.apache.org/job/PreCommit-PHOENIX-Build/1926//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-PHOENIX-Build/1926//artifact/patchprocess/patchReleaseAuditWarnings.txt Console output: https://builds.apache.org/job/PreCommit-PHOENIX-Build/1926//console This message is automatically generated. > Support multi region SYSTEM.CATALOG table > - > > Key: PHOENIX-3534 > URL: https://issues.apache.org/jira/browse/PHOENIX-3534 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: Thomas D'Silva >Priority: Major > Fix For: 5.0.0, 4.15.0 > > Attachments: PHOENIX-3534-v2.patch, PHOENIX-3534-v3.patch, > PHOENIX-3534.patch > > > Currently Phoenix requires that the SYSTEM.CATALOG table is single region > based on the server-side row locks being held for operations that impact a > table and all of it's views. For example, adding/removing a column from a > base table pushes this change to all views. > As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), > when a new table is created we can do a lazy cleanup of any rows that may be > left over from a failed DDL call (kudos to [~lhofhansl] for coming up with > this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 > so that we can efficiently find derived views. > The implementation would rely on an optimistic concurrency model based on > checking our sequence numbers for each table/view before/after updating. Each > table/view row would be individually locked for their change (metadata for a > view or table cannot span regions due to our split policy), with the sequence > number being incremented under lock and then returned to the client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)