[jira] [Created] (PHOENIX-4317) Update RPC controller to use the updated APIs
Ankit Singhal created PHOENIX-4317: -- Summary: Update RPC controller to use the updated APIs Key: PHOENIX-4317 URL: https://issues.apache.org/jira/browse/PHOENIX-4317 Project: Phoenix Issue Type: Sub-task Reporter: Ankit Singhal Assignee: Ankit Singhal Fix For: 4.13.0 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (PHOENIX-4317) Update RPC controller to use the updated APIs
[ https://issues.apache.org/jira/browse/PHOENIX-4317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Singhal updated PHOENIX-4317: --- Description: PayloadCarryingRpcController to HBaseRpcController DelegatingPayloadCarryingRpcController to DelegatingHBaseRpcController Update BalancedQueueRpcExecutor constructor > Update RPC controller to use the updated APIs > - > > Key: PHOENIX-4317 > URL: https://issues.apache.org/jira/browse/PHOENIX-4317 > Project: Phoenix > Issue Type: Sub-task >Reporter: Ankit Singhal >Assignee: Ankit Singhal > Labels: HBase-2.0 > Fix For: 4.13.0 > > > PayloadCarryingRpcController to HBaseRpcController > DelegatingPayloadCarryingRpcController to DelegatingHBaseRpcController > Update BalancedQueueRpcExecutor constructor -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (PHOENIX-4318) Fix IndexHalfStoreFileReader and related classes
Ankit Singhal created PHOENIX-4318: -- Summary: Fix IndexHalfStoreFileReader and related classes Key: PHOENIX-4318 URL: https://issues.apache.org/jira/browse/PHOENIX-4318 Project: Phoenix Issue Type: Sub-task Reporter: Ankit Singhal Assignee: Ankit Singhal Fix For: 4.13.0 These classes use the internals of HBase.(And most of them are not accessible in HBase 2.0) phoenix-core/src/main/java/org/apache/hadoop/hbase/regionserver/LocalIndexStoreFileScanner.java phoenix-core/src/main/java/org/apache/hadoop/hbase/regionserver/IndexHalfStoreFileReader.java phoenix-core/src/main/java/org/apache/hadoop/hbase/regionserver/IndexHalfStoreFileReaderGenerator.java phoenix-core/src/main/java/org/apache/phoenix/coprocessor/DelegateRegionScanner.java phoenix-core/src/main/java/org/apache/phoenix/util/IndexUtil.java phoenix-core/src/main/java/org/apache/phoenix/coprocessor/DelegateRegionObserver.java -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (PHOENIX-4317) Update RPC controller to use the updated APIs
[ https://issues.apache.org/jira/browse/PHOENIX-4317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217108#comment-16217108 ] James Taylor commented on PHOENIX-4317: --- For 2.0, we should use the setPriority method. > Update RPC controller to use the updated APIs > - > > Key: PHOENIX-4317 > URL: https://issues.apache.org/jira/browse/PHOENIX-4317 > Project: Phoenix > Issue Type: Sub-task >Reporter: Ankit Singhal >Assignee: Ankit Singhal > Labels: HBase-2.0 > Fix For: 4.13.0 > > > PayloadCarryingRpcController to HBaseRpcController > DelegatingPayloadCarryingRpcController to DelegatingHBaseRpcController > Update BalancedQueueRpcExecutor constructor -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] phoenix issue #277: PHOENIX-3757 System mutex table not being created in SYS...
Github user twdsilva commented on the issue: https://github.com/apache/phoenix/pull/277 @ankitsinghal @samarthjain @joshelser in case you guys want to take a look as well ---
[jira] [Commented] (PHOENIX-3757) System mutex table not being created in SYSTEM namespace when namespace mapping is enabled
[
https://issues.apache.org/jira/browse/PHOENIX-3757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217448#comment-16217448
]
ASF GitHub Bot commented on PHOENIX-3757:
-
Github user twdsilva commented on the issue:
https://github.com/apache/phoenix/pull/277
@ankitsinghal @samarthjain @joshelser
in case you guys want to take a look as well
> System mutex table not being created in SYSTEM namespace when namespace
> mapping is enabled
> --
>
> Key: PHOENIX-3757
> URL: https://issues.apache.org/jira/browse/PHOENIX-3757
> Project: Phoenix
> Issue Type: Bug
>Reporter: Josh Elser
>Assignee: Karan Mehta
>Priority: Critical
> Labels: namespaces
> Fix For: 4.13.0
>
> Attachments: PHOENIX-3757.001.patch, PHOENIX-3757.002.patch
>
>
> Noticed this issue while writing a test for PHOENIX-3756:
> The SYSTEM.MUTEX table is always created in the default namespace, even when
> {{phoenix.schema.isNamespaceMappingEnabled=true}}. At a glance, it looks like
> the logic for the other system tables isn't applied to the mutex table.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] phoenix pull request #277: PHOENIX-3757 System mutex table not being created...
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/277#discussion_r146659434
--- Diff:
phoenix-core/src/it/java/org/apache/phoenix/end2end/MigrateSystemTablesToSystemNamespaceIT.java
---
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.end2end;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HBaseTestingUtility;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.phoenix.coprocessor.MetaDataProtocol;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.BaseTest;
+import org.apache.phoenix.query.ConnectionQueryServices;
+import org.apache.phoenix.query.ConnectionQueryServicesImpl;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.schema.PTableType;
+import org.apache.phoenix.util.ReadOnlyProps;
+import org.apache.phoenix.util.SchemaUtil;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+import java.io.IOException;
+import java.security.PrivilegedExceptionAction;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+
+import static org.junit.Assert.*;
+
+@Category(NeedsOwnMiniClusterTest.class)
+public class MigrateSystemTablesToSystemNamespaceIT extends BaseTest {
+
+private static final Set PHOENIX_SYSTEM_TABLES = new
HashSet<>(Arrays.asList(
+"SYSTEM.CATALOG", "SYSTEM.SEQUENCE", "SYSTEM.STATS",
"SYSTEM.FUNCTION",
+"SYSTEM.MUTEX"));
+private static final Set
PHOENIX_NAMESPACE_MAPPED_SYSTEM_TABLES = new HashSet<>(
+Arrays.asList("SYSTEM:CATALOG", "SYSTEM:SEQUENCE",
"SYSTEM:STATS", "SYSTEM:FUNCTION",
+"SYSTEM:MUTEX"));
+private static final String SCHEMA_NAME = "MIGRATETEST";
+private static final String TABLE_NAME =
+SCHEMA_NAME + "." +
MigrateSystemTablesToSystemNamespaceIT.class.getSimpleName().toUpperCase();
+private static final int NUM_RECORDS = 5;
+
+private HBaseTestingUtility testUtil = null;
+private Set hbaseTables;
+
+// Create Multiple users since Phoenix caches the connection per user
+// Migration or upgrade code will run every time for each user.
+final UserGroupInformation user1 =
+UserGroupInformation.createUserForTesting("user1", new
String[0]);
+final UserGroupInformation user2 =
+UserGroupInformation.createUserForTesting("user2", new
String[0]);
+final UserGroupInformation user3 =
+UserGroupInformation.createUserForTesting("user3", new
String[0]);
+final UserGroupInformation user4 =
+UserGroupInformation.createUserForTesting("user4", new
String[0]);
+
+
+@Before
+public final void doSetup() throws Exception {
+testUtil = new HBaseTestingUtility();
+Configuration conf = testUtil.getConfiguration();
+enableNamespacesOnServer(conf);
+testUtil.startMiniCluster(1);
+}
+
+@After
+public void tearDownMiniCluster() {
+try {
+if (testUtil != null) {
+testUtil.shutdownMiniCluster();
+testUtil = null;
+}
+} catch (Exception e) {
+// ignore
+}
+}
+
+// Tests that client can create and read tables on a fresh HBase
clu
[jira] [Commented] (PHOENIX-3757) System mutex table not being created in SYSTEM namespace when namespace mapping is enabled
[
https://issues.apache.org/jira/browse/PHOENIX-3757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217454#comment-16217454
]
ASF GitHub Bot commented on PHOENIX-3757:
-
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/277#discussion_r146659434
--- Diff:
phoenix-core/src/it/java/org/apache/phoenix/end2end/MigrateSystemTablesToSystemNamespaceIT.java
---
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.end2end;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HBaseTestingUtility;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.phoenix.coprocessor.MetaDataProtocol;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.BaseTest;
+import org.apache.phoenix.query.ConnectionQueryServices;
+import org.apache.phoenix.query.ConnectionQueryServicesImpl;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.schema.PTableType;
+import org.apache.phoenix.util.ReadOnlyProps;
+import org.apache.phoenix.util.SchemaUtil;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+import java.io.IOException;
+import java.security.PrivilegedExceptionAction;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+
+import static org.junit.Assert.*;
+
+@Category(NeedsOwnMiniClusterTest.class)
+public class MigrateSystemTablesToSystemNamespaceIT extends BaseTest {
+
+private static final Set PHOENIX_SYSTEM_TABLES = new
HashSet<>(Arrays.asList(
+"SYSTEM.CATALOG", "SYSTEM.SEQUENCE", "SYSTEM.STATS",
"SYSTEM.FUNCTION",
+"SYSTEM.MUTEX"));
+private static final Set
PHOENIX_NAMESPACE_MAPPED_SYSTEM_TABLES = new HashSet<>(
+Arrays.asList("SYSTEM:CATALOG", "SYSTEM:SEQUENCE",
"SYSTEM:STATS", "SYSTEM:FUNCTION",
+"SYSTEM:MUTEX"));
+private static final String SCHEMA_NAME = "MIGRATETEST";
+private static final String TABLE_NAME =
+SCHEMA_NAME + "." +
MigrateSystemTablesToSystemNamespaceIT.class.getSimpleName().toUpperCase();
+private static final int NUM_RECORDS = 5;
+
+private HBaseTestingUtility testUtil = null;
+private Set hbaseTables;
+
+// Create Multiple users since Phoenix caches the connection per user
+// Migration or upgrade code will run every time for each user.
+final UserGroupInformation user1 =
+UserGroupInformation.createUserForTesting("user1", new
String[0]);
+final UserGroupInformation user2 =
+UserGroupInformation.createUserForTesting("user2", new
String[0]);
+final UserGroupInformation user3 =
+UserGroupInformation.createUserForTesting("user3", new
String[0]);
+final UserGroupInformation user4 =
+UserGroupInformation.createUserForTesting("user4", new
String[0]);
+
+
+@Before
+public final void doSetup() throws Exception {
+testUtil = new HBaseTestingUtility();
+Configuration conf = testUtil.getConfiguration();
+enableNamespacesOnServer(conf);
+testUtil.startMiniCluster(1);
+}
+
+@After
+public void tearDownMiniCluster() {
+try {
+if (testUtil != null) {
+t
[GitHub] phoenix pull request #277: PHOENIX-3757 System mutex table not being created...
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/277#discussion_r146660595
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataProtocol.java
---
@@ -95,6 +97,7 @@
// Key is the SYSTEM.CATALOG timestamp for the version and value is
the version string.
private static final NavigableMap TIMESTAMP_VERSION_MAP
= new TreeMap<>();
static {
+TIMESTAMP_VERSION_MAP.put(MIN_SYSTEM_TABLE_MIGRATION_TIMESTAMP,
MIGRATION_IN_PROGRESS);
--- End diff --
Is this only used in a test? If so can you change the test to use
MIN_SYSTEM_TABLE_TIMESTAMP?
---
[jira] [Commented] (PHOENIX-3757) System mutex table not being created in SYSTEM namespace when namespace mapping is enabled
[
https://issues.apache.org/jira/browse/PHOENIX-3757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217469#comment-16217469
]
ASF GitHub Bot commented on PHOENIX-3757:
-
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/277#discussion_r146660595
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataProtocol.java
---
@@ -95,6 +97,7 @@
// Key is the SYSTEM.CATALOG timestamp for the version and value is
the version string.
private static final NavigableMap TIMESTAMP_VERSION_MAP
= new TreeMap<>();
static {
+TIMESTAMP_VERSION_MAP.put(MIN_SYSTEM_TABLE_MIGRATION_TIMESTAMP,
MIGRATION_IN_PROGRESS);
--- End diff --
Is this only used in a test? If so can you change the test to use
MIN_SYSTEM_TABLE_TIMESTAMP?
> System mutex table not being created in SYSTEM namespace when namespace
> mapping is enabled
> --
>
> Key: PHOENIX-3757
> URL: https://issues.apache.org/jira/browse/PHOENIX-3757
> Project: Phoenix
> Issue Type: Bug
>Reporter: Josh Elser
>Assignee: Karan Mehta
>Priority: Critical
> Labels: namespaces
> Fix For: 4.13.0
>
> Attachments: PHOENIX-3757.001.patch, PHOENIX-3757.002.patch
>
>
> Noticed this issue while writing a test for PHOENIX-3756:
> The SYSTEM.MUTEX table is always created in the default namespace, even when
> {{phoenix.schema.isNamespaceMappingEnabled=true}}. At a glance, it looks like
> the logic for the other system tables isn't applied to the mutex table.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (PHOENIX-4198) Remove the need for users to have access to the Phoenix SYSTEM tables to create tables
[
https://issues.apache.org/jira/browse/PHOENIX-4198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217473#comment-16217473
]
Josh Elser commented on PHOENIX-4198:
-
{code}
+try {
+Call rpcContext = RpcUtil.getRpcContext();
+//Setting RPC context as null so that user can be
resetted
+RpcUtil.setRpcContext(null);
+region.mutateRowsWithLocks(mutations, rowsToLock,
nonceGroup, nonce);
+//Setting RPC context back to original context of the
RPC
+RpcUtil.setRpcContext(rpcContext);
+} catch (Throwable e) {
+throw new IOException(e);
+}
{code}
[~ankit.singhal], would suggest you tweak this to use a try/finally block just
in case. e.g.
{code}
final Call rpcContext = RpcUtil.getRpcContext();
try {
//Setting RPC context as null so that user can be
resetted
RpcUtil.setRpcContext(null);
region.mutateRowsWithLocks(mutations, rowsToLock,
nonceGroup, nonce);
} catch (Throwable e) {
throw new IOException(e);
} finally {
//Setting RPC context back to original context of the
RPC
RpcUtil.setRpcContext(rpcContext);
}
{code}
> Remove the need for users to have access to the Phoenix SYSTEM tables to
> create tables
> --
>
> Key: PHOENIX-4198
> URL: https://issues.apache.org/jira/browse/PHOENIX-4198
> Project: Phoenix
> Issue Type: Bug
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
> Labels: namespaces, security
> Fix For: 4.13.0
>
> Attachments: PHOENIX-4198.patch, PHOENIX-4198_v2.patch,
> PHOENIX-4198_v3.patch, PHOENIX-4198_v4.patch
>
>
> Problem statement:-
> A user who doesn't have access to a table should also not be able to modify
> Phoenix Metadata. Currently, every user required to have a write permission
> to SYSTEM tables which is a security concern as they can
> create/alter/drop/corrupt meta data of any other table without proper access
> to the corresponding physical tables.
> [~devaraj] recommended a solution as below.
> 1. A coprocessor endpoint would be implemented and all write accesses to the
> catalog table would have to necessarily go through that. The 'hbase' user
> would own that table. Today, there is MetaDataEndpointImpl that's run on the
> RS where the catalog is hosted, and that could be enhanced to serve the
> purpose we need.
> 2. The regionserver hosting the catalog table would do the needful for all
> catalog updates - creating the mutations as needed, that is.
> 3. The coprocessor endpoint could use Ranger to do necessary authorization
> checks before updating the catalog table. So for example, if a user doesn't
> have authorization to create a table in a certain namespace, or update the
> schema, etc., it can reject such requests outright. Only after successful
> validations, does it perform the operations (physical operations to do with
> creating the table, and updating the catalog table with the necessary
> mutations).
> 4. In essence, the code that implements dealing with DDLs, would be hosted in
> the catalog table endpoint. The client code would be really thin, and it
> would just invoke the endpoint with the necessary info. The additional thing
> that needs to be done in the endpoint is the validation of authorization to
> prevent unauthorized users from making changes to someone else's
> tables/schemas/etc. For example, one should be able to create a view on a
> table if he has read access on the base table. That mutation on the catalog
> table would be permitted. For changing the schema (adding a new column for
> example), the said user would need write permission on the table... etc etc.
> Thanks [~elserj] for the write-up.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] phoenix pull request #277: PHOENIX-3757 System mutex table not being created...
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/277#discussion_r146662118
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
---
@@ -2905,14 +2918,18 @@ public void upgradeSystemTables(final String url,
final Properties props) throws
}
} finally {
if (acquiredMutexLock) {
-releaseUpgradeMutex(mutexRowKey);
+try {
+releaseUpgradeMutex(mutexRowKey);
+} catch (IOException e) {
+logger.warn("Release of upgrade mutex failed
", e);
--- End diff --
Is it ok to continue on if we fail to release the upgrade mutex, or should
we just let the exception be thrown?
---
[jira] [Commented] (PHOENIX-3757) System mutex table not being created in SYSTEM namespace when namespace mapping is enabled
[
https://issues.apache.org/jira/browse/PHOENIX-3757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217480#comment-16217480
]
ASF GitHub Bot commented on PHOENIX-3757:
-
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/277#discussion_r146662118
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
---
@@ -2905,14 +2918,18 @@ public void upgradeSystemTables(final String url,
final Properties props) throws
}
} finally {
if (acquiredMutexLock) {
-releaseUpgradeMutex(mutexRowKey);
+try {
+releaseUpgradeMutex(mutexRowKey);
+} catch (IOException e) {
+logger.warn("Release of upgrade mutex failed
", e);
--- End diff --
Is it ok to continue on if we fail to release the upgrade mutex, or should
we just let the exception be thrown?
> System mutex table not being created in SYSTEM namespace when namespace
> mapping is enabled
> --
>
> Key: PHOENIX-3757
> URL: https://issues.apache.org/jira/browse/PHOENIX-3757
> Project: Phoenix
> Issue Type: Bug
>Reporter: Josh Elser
>Assignee: Karan Mehta
>Priority: Critical
> Labels: namespaces
> Fix For: 4.13.0
>
> Attachments: PHOENIX-3757.001.patch, PHOENIX-3757.002.patch
>
>
> Noticed this issue while writing a test for PHOENIX-3756:
> The SYSTEM.MUTEX table is always created in the default namespace, even when
> {{phoenix.schema.isNamespaceMappingEnabled=true}}. At a glance, it looks like
> the logic for the other system tables isn't applied to the mutex table.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] phoenix pull request #277: PHOENIX-3757 System mutex table not being created...
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/277#discussion_r146663164
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
---
@@ -3101,49 +3124,68 @@ void ensureSystemTablesUpgraded(ReadOnlyProps props)
// Regardless of the case 1 or 2, if the NS does not
exist, we will error expectedly
// below. If the NS does exist and is mapped, the below
check will exit gracefully.
}
-
+
List tableNames = getSystemTableNames(admin);
// No tables exist matching "SYSTEM\..*", they are all already
in "SYSTEM:.*"
if (tableNames.size() == 0) { return; }
// Try to move any remaining tables matching "SYSTEM\..*" into
"SYSTEM:"
if (tableNames.size() > 5) {
logger.warn("Expected 5 system tables but found " +
tableNames.size() + ":" + tableNames);
}
+
+// Try acquiring a lock in SYSMUTEX table before migrating the
tables since it involves disabling the table
+// If we cannot acquire lock, it means some old client is
either migrating SYSCAT or trying to upgrade the
+// schema of SYSCAT table and hence it should not be
interrupted
+acquiredMutexLock = acquireUpgradeMutex(0, mutexRowKey);
+if(acquiredMutexLock) {
+logger.debug("Acquired lock in SYSMUTEX table for
migrating SYSTEM tables to SYSTEM namespace");
+}
+// We will not reach here if we fail to acquire the lock,
since it throws UpgradeInProgressException
+
+// Handle the upgrade of SYSMUTEX table separately since it
doesn't have any entries in SYSCAT
+String sysMutexSrcTableName =
PhoenixDatabaseMetaData.SYSTEM_MUTEX_NAME;
+String sysMutexDestTableName =
SchemaUtil.getPhysicalName(sysMutexSrcTableName.getBytes(),
props).getNameAsString();
+UpgradeUtil.mapTableToNamespace(admin, sysMutexSrcTableName,
sysMutexDestTableName, PTableType.SYSTEM);
+
byte[] mappedSystemTable = SchemaUtil
.getPhysicalName(PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME_BYTES,
props).getName();
metatable = getTable(mappedSystemTable);
if
(tableNames.contains(PhoenixDatabaseMetaData.SYSTEM_CATALOG_HBASE_TABLE_NAME)) {
if (!admin.tableExists(mappedSystemTable)) {
+// Actual migration of SYSCAT table
UpgradeUtil.mapTableToNamespace(admin, metatable,
PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME,
props, null, PTableType.SYSTEM,
null);
+// Invalidate the client-side metadataCache
ConnectionQueryServicesImpl.this.removeTable(null,
PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME,
null,
MetaDataProtocol.MIN_SYSTEM_TABLE_TIMESTAMP_4_1_0);
}
tableNames.remove(PhoenixDatabaseMetaData.SYSTEM_CATALOG_HBASE_TABLE_NAME);
}
-
tableNames.remove(PhoenixDatabaseMetaData.SYSTEM_MUTEX_HBASE_TABLE_NAME);
--- End diff --
I think you still need to remove SYSTEM.MUTEX from tableNames.
---
[jira] [Commented] (PHOENIX-3757) System mutex table not being created in SYSTEM namespace when namespace mapping is enabled
[
https://issues.apache.org/jira/browse/PHOENIX-3757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217491#comment-16217491
]
ASF GitHub Bot commented on PHOENIX-3757:
-
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/277#discussion_r146663164
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
---
@@ -3101,49 +3124,68 @@ void ensureSystemTablesUpgraded(ReadOnlyProps props)
// Regardless of the case 1 or 2, if the NS does not
exist, we will error expectedly
// below. If the NS does exist and is mapped, the below
check will exit gracefully.
}
-
+
List tableNames = getSystemTableNames(admin);
// No tables exist matching "SYSTEM\..*", they are all already
in "SYSTEM:.*"
if (tableNames.size() == 0) { return; }
// Try to move any remaining tables matching "SYSTEM\..*" into
"SYSTEM:"
if (tableNames.size() > 5) {
logger.warn("Expected 5 system tables but found " +
tableNames.size() + ":" + tableNames);
}
+
+// Try acquiring a lock in SYSMUTEX table before migrating the
tables since it involves disabling the table
+// If we cannot acquire lock, it means some old client is
either migrating SYSCAT or trying to upgrade the
+// schema of SYSCAT table and hence it should not be
interrupted
+acquiredMutexLock = acquireUpgradeMutex(0, mutexRowKey);
+if(acquiredMutexLock) {
+logger.debug("Acquired lock in SYSMUTEX table for
migrating SYSTEM tables to SYSTEM namespace");
+}
+// We will not reach here if we fail to acquire the lock,
since it throws UpgradeInProgressException
+
+// Handle the upgrade of SYSMUTEX table separately since it
doesn't have any entries in SYSCAT
+String sysMutexSrcTableName =
PhoenixDatabaseMetaData.SYSTEM_MUTEX_NAME;
+String sysMutexDestTableName =
SchemaUtil.getPhysicalName(sysMutexSrcTableName.getBytes(),
props).getNameAsString();
+UpgradeUtil.mapTableToNamespace(admin, sysMutexSrcTableName,
sysMutexDestTableName, PTableType.SYSTEM);
+
byte[] mappedSystemTable = SchemaUtil
.getPhysicalName(PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME_BYTES,
props).getName();
metatable = getTable(mappedSystemTable);
if
(tableNames.contains(PhoenixDatabaseMetaData.SYSTEM_CATALOG_HBASE_TABLE_NAME)) {
if (!admin.tableExists(mappedSystemTable)) {
+// Actual migration of SYSCAT table
UpgradeUtil.mapTableToNamespace(admin, metatable,
PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME,
props, null, PTableType.SYSTEM,
null);
+// Invalidate the client-side metadataCache
ConnectionQueryServicesImpl.this.removeTable(null,
PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME,
null,
MetaDataProtocol.MIN_SYSTEM_TABLE_TIMESTAMP_4_1_0);
}
tableNames.remove(PhoenixDatabaseMetaData.SYSTEM_CATALOG_HBASE_TABLE_NAME);
}
-
tableNames.remove(PhoenixDatabaseMetaData.SYSTEM_MUTEX_HBASE_TABLE_NAME);
--- End diff --
I think you still need to remove SYSTEM.MUTEX from tableNames.
> System mutex table not being created in SYSTEM namespace when namespace
> mapping is enabled
> --
>
> Key: PHOENIX-3757
> URL: https://issues.apache.org/jira/browse/PHOENIX-3757
> Project: Phoenix
> Issue Type: Bug
>Reporter: Josh Elser
>Assignee: Karan Mehta
>Priority: Critical
> Labels: namespaces
> Fix For: 4.13.0
>
> Attachments: PHOENIX-3757.001.patch, PHOENIX-3757.002.patch
>
>
> Noticed this issue while writing a test for PHOENIX-3756:
> The SYSTEM.MUTEX table is always created in the default namespace, even when
> {{phoenix.schema.isNamespaceMappingEnabled=true}}. At a glance, it looks like
> the logic for the other system tables isn't applied to the mutex table.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] phoenix pull request #277: PHOENIX-3757 System mutex table not being created...
Github user aertoria commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/277#discussion_r146682331
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
---
@@ -3101,49 +3124,68 @@ void ensureSystemTablesUpgraded(ReadOnlyProps props)
// Regardless of the case 1 or 2, if the NS does not
exist, we will error expectedly
// below. If the NS does exist and is mapped, the below
check will exit gracefully.
}
-
+
List tableNames = getSystemTableNames(admin);
// No tables exist matching "SYSTEM\..*", they are all already
in "SYSTEM:.*"
if (tableNames.size() == 0) { return; }
// Try to move any remaining tables matching "SYSTEM\..*" into
"SYSTEM:"
if (tableNames.size() > 5) {
logger.warn("Expected 5 system tables but found " +
tableNames.size() + ":" + tableNames);
}
+
+// Try acquiring a lock in SYSMUTEX table before migrating the
tables since it involves disabling the table
+// If we cannot acquire lock, it means some old client is
either migrating SYSCAT or trying to upgrade the
+// schema of SYSCAT table and hence it should not be
interrupted
+acquiredMutexLock = acquireUpgradeMutex(0, mutexRowKey);
+if(acquiredMutexLock) {
+logger.debug("Acquired lock in SYSMUTEX table for
migrating SYSTEM tables to SYSTEM namespace");
+}
+// We will not reach here if we fail to acquire the lock,
since it throws UpgradeInProgressException
+
+// Handle the upgrade of SYSMUTEX table separately since it
doesn't have any entries in SYSCAT
+String sysMutexSrcTableName =
PhoenixDatabaseMetaData.SYSTEM_MUTEX_NAME;
+String sysMutexDestTableName =
SchemaUtil.getPhysicalName(sysMutexSrcTableName.getBytes(),
props).getNameAsString();
+UpgradeUtil.mapTableToNamespace(admin, sysMutexSrcTableName,
sysMutexDestTableName, PTableType.SYSTEM);
+
byte[] mappedSystemTable = SchemaUtil
.getPhysicalName(PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME_BYTES,
props).getName();
metatable = getTable(mappedSystemTable);
if
(tableNames.contains(PhoenixDatabaseMetaData.SYSTEM_CATALOG_HBASE_TABLE_NAME)) {
if (!admin.tableExists(mappedSystemTable)) {
+// Actual migration of SYSCAT table
UpgradeUtil.mapTableToNamespace(admin, metatable,
PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME,
props, null, PTableType.SYSTEM,
null);
+// Invalidate the client-side metadataCache
ConnectionQueryServicesImpl.this.removeTable(null,
PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME,
null,
MetaDataProtocol.MIN_SYSTEM_TABLE_TIMESTAMP_4_1_0);
}
tableNames.remove(PhoenixDatabaseMetaData.SYSTEM_CATALOG_HBASE_TABLE_NAME);
}
-
tableNames.remove(PhoenixDatabaseMetaData.SYSTEM_MUTEX_HBASE_TABLE_NAME);
for (TableName table : tableNames) {
UpgradeUtil.mapTableToNamespace(admin, metatable,
table.getNameAsString(), props, null, PTableType.SYSTEM,
null);
ConnectionQueryServicesImpl.this.removeTable(null,
table.getNameAsString(), null,
MetaDataProtocol.MIN_SYSTEM_TABLE_TIMESTAMP_4_1_0);
}
-if (!tableNames.isEmpty()) {
-clearCache();
-}
+
+// Clear the server-side metadataCache when all tables are
migrated so that the new PTable can be loaded with NS mapping
+clearCache();
--- End diff --
What does clearCache() here do? Does it clear PMeta cache at the server
side?
---
[GitHub] phoenix pull request #277: PHOENIX-3757 System mutex table not being created...
Github user aertoria commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/277#discussion_r146687243
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
---
@@ -3101,49 +3124,68 @@ void ensureSystemTablesUpgraded(ReadOnlyProps props)
// Regardless of the case 1 or 2, if the NS does not
exist, we will error expectedly
// below. If the NS does exist and is mapped, the below
check will exit gracefully.
}
-
+
List tableNames = getSystemTableNames(admin);
// No tables exist matching "SYSTEM\..*", they are all already
in "SYSTEM:.*"
if (tableNames.size() == 0) { return; }
// Try to move any remaining tables matching "SYSTEM\..*" into
"SYSTEM:"
if (tableNames.size() > 5) {
logger.warn("Expected 5 system tables but found " +
tableNames.size() + ":" + tableNames);
}
+
+// Try acquiring a lock in SYSMUTEX table before migrating the
tables since it involves disabling the table
+// If we cannot acquire lock, it means some old client is
either migrating SYSCAT or trying to upgrade the
+// schema of SYSCAT table and hence it should not be
interrupted
+acquiredMutexLock = acquireUpgradeMutex(0, mutexRowKey);
+if(acquiredMutexLock) {
+logger.debug("Acquired lock in SYSMUTEX table for
migrating SYSTEM tables to SYSTEM namespace");
+}
+// We will not reach here if we fail to acquire the lock,
since it throws UpgradeInProgressException
+
+// Handle the upgrade of SYSMUTEX table separately since it
doesn't have any entries in SYSCAT
+String sysMutexSrcTableName =
PhoenixDatabaseMetaData.SYSTEM_MUTEX_NAME;
+String sysMutexDestTableName =
SchemaUtil.getPhysicalName(sysMutexSrcTableName.getBytes(),
props).getNameAsString();
+UpgradeUtil.mapTableToNamespace(admin, sysMutexSrcTableName,
sysMutexDestTableName, PTableType.SYSTEM);
+
byte[] mappedSystemTable = SchemaUtil
.getPhysicalName(PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME_BYTES,
props).getName();
metatable = getTable(mappedSystemTable);
if
(tableNames.contains(PhoenixDatabaseMetaData.SYSTEM_CATALOG_HBASE_TABLE_NAME)) {
if (!admin.tableExists(mappedSystemTable)) {
+// Actual migration of SYSCAT table
UpgradeUtil.mapTableToNamespace(admin, metatable,
PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME,
props, null, PTableType.SYSTEM,
null);
+// Invalidate the client-side metadataCache
ConnectionQueryServicesImpl.this.removeTable(null,
PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME,
null,
MetaDataProtocol.MIN_SYSTEM_TABLE_TIMESTAMP_4_1_0);
}
tableNames.remove(PhoenixDatabaseMetaData.SYSTEM_CATALOG_HBASE_TABLE_NAME);
}
-
tableNames.remove(PhoenixDatabaseMetaData.SYSTEM_MUTEX_HBASE_TABLE_NAME);
for (TableName table : tableNames) {
UpgradeUtil.mapTableToNamespace(admin, metatable,
table.getNameAsString(), props, null, PTableType.SYSTEM,
null);
ConnectionQueryServicesImpl.this.removeTable(null,
table.getNameAsString(), null,
MetaDataProtocol.MIN_SYSTEM_TABLE_TIMESTAMP_4_1_0);
}
-if (!tableNames.isEmpty()) {
-clearCache();
-}
+
+// Clear the server-side metadataCache when all tables are
migrated so that the new PTable can be loaded with NS mapping
+clearCache();
} finally {
if (metatable != null) {
metatable.close();
}
+if(acquiredMutexLock) {
--- End diff --
Why wasn't this "UpgradeUtil.mapTableToNamespace" synchronized in the
existing code?
---
[jira] [Commented] (PHOENIX-3757) System mutex table not being created in SYSTEM namespace when namespace mapping is enabled
[
https://issues.apache.org/jira/browse/PHOENIX-3757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217660#comment-16217660
]
ASF GitHub Bot commented on PHOENIX-3757:
-
Github user aertoria commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/277#discussion_r146687243
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
---
@@ -3101,49 +3124,68 @@ void ensureSystemTablesUpgraded(ReadOnlyProps props)
// Regardless of the case 1 or 2, if the NS does not
exist, we will error expectedly
// below. If the NS does exist and is mapped, the below
check will exit gracefully.
}
-
+
List tableNames = getSystemTableNames(admin);
// No tables exist matching "SYSTEM\..*", they are all already
in "SYSTEM:.*"
if (tableNames.size() == 0) { return; }
// Try to move any remaining tables matching "SYSTEM\..*" into
"SYSTEM:"
if (tableNames.size() > 5) {
logger.warn("Expected 5 system tables but found " +
tableNames.size() + ":" + tableNames);
}
+
+// Try acquiring a lock in SYSMUTEX table before migrating the
tables since it involves disabling the table
+// If we cannot acquire lock, it means some old client is
either migrating SYSCAT or trying to upgrade the
+// schema of SYSCAT table and hence it should not be
interrupted
+acquiredMutexLock = acquireUpgradeMutex(0, mutexRowKey);
+if(acquiredMutexLock) {
+logger.debug("Acquired lock in SYSMUTEX table for
migrating SYSTEM tables to SYSTEM namespace");
+}
+// We will not reach here if we fail to acquire the lock,
since it throws UpgradeInProgressException
+
+// Handle the upgrade of SYSMUTEX table separately since it
doesn't have any entries in SYSCAT
+String sysMutexSrcTableName =
PhoenixDatabaseMetaData.SYSTEM_MUTEX_NAME;
+String sysMutexDestTableName =
SchemaUtil.getPhysicalName(sysMutexSrcTableName.getBytes(),
props).getNameAsString();
+UpgradeUtil.mapTableToNamespace(admin, sysMutexSrcTableName,
sysMutexDestTableName, PTableType.SYSTEM);
+
byte[] mappedSystemTable = SchemaUtil
.getPhysicalName(PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME_BYTES,
props).getName();
metatable = getTable(mappedSystemTable);
if
(tableNames.contains(PhoenixDatabaseMetaData.SYSTEM_CATALOG_HBASE_TABLE_NAME)) {
if (!admin.tableExists(mappedSystemTable)) {
+// Actual migration of SYSCAT table
UpgradeUtil.mapTableToNamespace(admin, metatable,
PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME,
props, null, PTableType.SYSTEM,
null);
+// Invalidate the client-side metadataCache
ConnectionQueryServicesImpl.this.removeTable(null,
PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME,
null,
MetaDataProtocol.MIN_SYSTEM_TABLE_TIMESTAMP_4_1_0);
}
tableNames.remove(PhoenixDatabaseMetaData.SYSTEM_CATALOG_HBASE_TABLE_NAME);
}
-
tableNames.remove(PhoenixDatabaseMetaData.SYSTEM_MUTEX_HBASE_TABLE_NAME);
for (TableName table : tableNames) {
UpgradeUtil.mapTableToNamespace(admin, metatable,
table.getNameAsString(), props, null, PTableType.SYSTEM,
null);
ConnectionQueryServicesImpl.this.removeTable(null,
table.getNameAsString(), null,
MetaDataProtocol.MIN_SYSTEM_TABLE_TIMESTAMP_4_1_0);
}
-if (!tableNames.isEmpty()) {
-clearCache();
-}
+
+// Clear the server-side metadataCache when all tables are
migrated so that the new PTable can be loaded with NS mapping
+clearCache();
} finally {
if (metatable != null) {
metatable.close();
}
+if(acquiredMutexLock) {
--- End diff --
Why wasn't this "UpgradeUtil.mapTableToNamespace" synchronized in the
existing code?
> System mutex table not being created in SYSTEM namespace when namespace
> mapping is enabled
> --
>
> Key: PHOENIX-3757
> URL: https://issues.apache.org/
[jira] [Commented] (PHOENIX-3757) System mutex table not being created in SYSTEM namespace when namespace mapping is enabled
[
https://issues.apache.org/jira/browse/PHOENIX-3757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217661#comment-16217661
]
ASF GitHub Bot commented on PHOENIX-3757:
-
Github user aertoria commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/277#discussion_r146682331
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
---
@@ -3101,49 +3124,68 @@ void ensureSystemTablesUpgraded(ReadOnlyProps props)
// Regardless of the case 1 or 2, if the NS does not
exist, we will error expectedly
// below. If the NS does exist and is mapped, the below
check will exit gracefully.
}
-
+
List tableNames = getSystemTableNames(admin);
// No tables exist matching "SYSTEM\..*", they are all already
in "SYSTEM:.*"
if (tableNames.size() == 0) { return; }
// Try to move any remaining tables matching "SYSTEM\..*" into
"SYSTEM:"
if (tableNames.size() > 5) {
logger.warn("Expected 5 system tables but found " +
tableNames.size() + ":" + tableNames);
}
+
+// Try acquiring a lock in SYSMUTEX table before migrating the
tables since it involves disabling the table
+// If we cannot acquire lock, it means some old client is
either migrating SYSCAT or trying to upgrade the
+// schema of SYSCAT table and hence it should not be
interrupted
+acquiredMutexLock = acquireUpgradeMutex(0, mutexRowKey);
+if(acquiredMutexLock) {
+logger.debug("Acquired lock in SYSMUTEX table for
migrating SYSTEM tables to SYSTEM namespace");
+}
+// We will not reach here if we fail to acquire the lock,
since it throws UpgradeInProgressException
+
+// Handle the upgrade of SYSMUTEX table separately since it
doesn't have any entries in SYSCAT
+String sysMutexSrcTableName =
PhoenixDatabaseMetaData.SYSTEM_MUTEX_NAME;
+String sysMutexDestTableName =
SchemaUtil.getPhysicalName(sysMutexSrcTableName.getBytes(),
props).getNameAsString();
+UpgradeUtil.mapTableToNamespace(admin, sysMutexSrcTableName,
sysMutexDestTableName, PTableType.SYSTEM);
+
byte[] mappedSystemTable = SchemaUtil
.getPhysicalName(PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME_BYTES,
props).getName();
metatable = getTable(mappedSystemTable);
if
(tableNames.contains(PhoenixDatabaseMetaData.SYSTEM_CATALOG_HBASE_TABLE_NAME)) {
if (!admin.tableExists(mappedSystemTable)) {
+// Actual migration of SYSCAT table
UpgradeUtil.mapTableToNamespace(admin, metatable,
PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME,
props, null, PTableType.SYSTEM,
null);
+// Invalidate the client-side metadataCache
ConnectionQueryServicesImpl.this.removeTable(null,
PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME,
null,
MetaDataProtocol.MIN_SYSTEM_TABLE_TIMESTAMP_4_1_0);
}
tableNames.remove(PhoenixDatabaseMetaData.SYSTEM_CATALOG_HBASE_TABLE_NAME);
}
-
tableNames.remove(PhoenixDatabaseMetaData.SYSTEM_MUTEX_HBASE_TABLE_NAME);
for (TableName table : tableNames) {
UpgradeUtil.mapTableToNamespace(admin, metatable,
table.getNameAsString(), props, null, PTableType.SYSTEM,
null);
ConnectionQueryServicesImpl.this.removeTable(null,
table.getNameAsString(), null,
MetaDataProtocol.MIN_SYSTEM_TABLE_TIMESTAMP_4_1_0);
}
-if (!tableNames.isEmpty()) {
-clearCache();
-}
+
+// Clear the server-side metadataCache when all tables are
migrated so that the new PTable can be loaded with NS mapping
+clearCache();
--- End diff --
What does clearCache() here do? Does it clear PMeta cache at the server
side?
> System mutex table not being created in SYSTEM namespace when namespace
> mapping is enabled
> --
>
> Key: PHOENIX-3757
> URL: https://issues.apache.org/jira/browse/PHOENIX-3757
> Project: Phoenix
> Issue Type: Bug
>Reporter: Josh Elser
>Assignee: Karan Mehta
>Priority:
[jira] [Updated] (PHOENIX-3757) System mutex table not being created in SYSTEM namespace when namespace mapping is enabled
[
https://issues.apache.org/jira/browse/PHOENIX-3757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karan Mehta updated PHOENIX-3757:
-
Attachment: PHOENIX-3757.003.patch
> System mutex table not being created in SYSTEM namespace when namespace
> mapping is enabled
> --
>
> Key: PHOENIX-3757
> URL: https://issues.apache.org/jira/browse/PHOENIX-3757
> Project: Phoenix
> Issue Type: Bug
>Reporter: Josh Elser
>Assignee: Karan Mehta
>Priority: Critical
> Labels: namespaces
> Fix For: 4.13.0
>
> Attachments: PHOENIX-3757.001.patch, PHOENIX-3757.002.patch,
> PHOENIX-3757.003.patch
>
>
> Noticed this issue while writing a test for PHOENIX-3756:
> The SYSTEM.MUTEX table is always created in the default namespace, even when
> {{phoenix.schema.isNamespaceMappingEnabled=true}}. At a glance, it looks like
> the logic for the other system tables isn't applied to the mutex table.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] phoenix pull request #277: PHOENIX-3757 System mutex table not being created...
Github user karanmehta93 commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/277#discussion_r146695278
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataProtocol.java
---
@@ -95,6 +97,7 @@
// Key is the SYSTEM.CATALOG timestamp for the version and value is
the version string.
private static final NavigableMap TIMESTAMP_VERSION_MAP
= new TreeMap<>();
static {
+TIMESTAMP_VERSION_MAP.put(MIN_SYSTEM_TABLE_MIGRATION_TIMESTAMP,
MIGRATION_IN_PROGRESS);
--- End diff --
Nope, this one is not only for test.
When you acquire a lock on SYSMUTEX, the method requires you to pass in a
server side timestamp of SYSCAT table (Since it was meant for that earlier) and
it checks if the current timestamp is less than the MIN_SYSTEM_TABLE_TIMESTAMP
before upgrading it.
I can't use MIN_SYSTEM_TABLE_TIMESTAMP because then the code for
`acquireUpgradeMutex()` wont allow me to acquire the mutex lock.
I use the same `UpgradeInProgressException` to prevent multiple clients
migrating the system tables together. I use this value in
`UpgradeInProgressException` to change the error message displayed to the user.
---
[jira] [Commented] (PHOENIX-3757) System mutex table not being created in SYSTEM namespace when namespace mapping is enabled
[
https://issues.apache.org/jira/browse/PHOENIX-3757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217714#comment-16217714
]
ASF GitHub Bot commented on PHOENIX-3757:
-
Github user karanmehta93 commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/277#discussion_r146695278
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataProtocol.java
---
@@ -95,6 +97,7 @@
// Key is the SYSTEM.CATALOG timestamp for the version and value is
the version string.
private static final NavigableMap TIMESTAMP_VERSION_MAP
= new TreeMap<>();
static {
+TIMESTAMP_VERSION_MAP.put(MIN_SYSTEM_TABLE_MIGRATION_TIMESTAMP,
MIGRATION_IN_PROGRESS);
--- End diff --
Nope, this one is not only for test.
When you acquire a lock on SYSMUTEX, the method requires you to pass in a
server side timestamp of SYSCAT table (Since it was meant for that earlier) and
it checks if the current timestamp is less than the MIN_SYSTEM_TABLE_TIMESTAMP
before upgrading it.
I can't use MIN_SYSTEM_TABLE_TIMESTAMP because then the code for
`acquireUpgradeMutex()` wont allow me to acquire the mutex lock.
I use the same `UpgradeInProgressException` to prevent multiple clients
migrating the system tables together. I use this value in
`UpgradeInProgressException` to change the error message displayed to the user.
> System mutex table not being created in SYSTEM namespace when namespace
> mapping is enabled
> --
>
> Key: PHOENIX-3757
> URL: https://issues.apache.org/jira/browse/PHOENIX-3757
> Project: Phoenix
> Issue Type: Bug
>Reporter: Josh Elser
>Assignee: Karan Mehta
>Priority: Critical
> Labels: namespaces
> Fix For: 4.13.0
>
> Attachments: PHOENIX-3757.001.patch, PHOENIX-3757.002.patch,
> PHOENIX-3757.003.patch
>
>
> Noticed this issue while writing a test for PHOENIX-3756:
> The SYSTEM.MUTEX table is always created in the default namespace, even when
> {{phoenix.schema.isNamespaceMappingEnabled=true}}. At a glance, it looks like
> the logic for the other system tables isn't applied to the mutex table.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] phoenix pull request #277: PHOENIX-3757 System mutex table not being created...
Github user karanmehta93 commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/277#discussion_r146695798
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
---
@@ -2905,14 +2918,18 @@ public void upgradeSystemTables(final String url,
final Properties props) throws
}
} finally {
if (acquiredMutexLock) {
-releaseUpgradeMutex(mutexRowKey);
+try {
+releaseUpgradeMutex(mutexRowKey);
+} catch (IOException e) {
+logger.warn("Release of upgrade mutex failed
", e);
--- End diff --
I think it is fine not to throw the exception because we have a TTL on the
table. Typically it should not matter to the client as such. Moreover, similar
behavior is also defined the code that Upgrades the SYSCAT table.
---
[jira] [Commented] (PHOENIX-3757) System mutex table not being created in SYSTEM namespace when namespace mapping is enabled
[
https://issues.apache.org/jira/browse/PHOENIX-3757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217716#comment-16217716
]
Hadoop QA commented on PHOENIX-3757:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12893812/PHOENIX-3757.003.patch
against master branch at commit 7cdcb2313b08d2eaeb775f0c989642f8d416cfb6.
ATTACHMENT ID: 12893812
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1568//console
This message is automatically generated.
> System mutex table not being created in SYSTEM namespace when namespace
> mapping is enabled
> --
>
> Key: PHOENIX-3757
> URL: https://issues.apache.org/jira/browse/PHOENIX-3757
> Project: Phoenix
> Issue Type: Bug
>Reporter: Josh Elser
>Assignee: Karan Mehta
>Priority: Critical
> Labels: namespaces
> Fix For: 4.13.0
>
> Attachments: PHOENIX-3757.001.patch, PHOENIX-3757.002.patch,
> PHOENIX-3757.003.patch
>
>
> Noticed this issue while writing a test for PHOENIX-3756:
> The SYSTEM.MUTEX table is always created in the default namespace, even when
> {{phoenix.schema.isNamespaceMappingEnabled=true}}. At a glance, it looks like
> the logic for the other system tables isn't applied to the mutex table.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (PHOENIX-3757) System mutex table not being created in SYSTEM namespace when namespace mapping is enabled
[
https://issues.apache.org/jira/browse/PHOENIX-3757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217717#comment-16217717
]
ASF GitHub Bot commented on PHOENIX-3757:
-
Github user karanmehta93 commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/277#discussion_r146695798
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
---
@@ -2905,14 +2918,18 @@ public void upgradeSystemTables(final String url,
final Properties props) throws
}
} finally {
if (acquiredMutexLock) {
-releaseUpgradeMutex(mutexRowKey);
+try {
+releaseUpgradeMutex(mutexRowKey);
+} catch (IOException e) {
+logger.warn("Release of upgrade mutex failed
", e);
--- End diff --
I think it is fine not to throw the exception because we have a TTL on the
table. Typically it should not matter to the client as such. Moreover, similar
behavior is also defined the code that Upgrades the SYSCAT table.
> System mutex table not being created in SYSTEM namespace when namespace
> mapping is enabled
> --
>
> Key: PHOENIX-3757
> URL: https://issues.apache.org/jira/browse/PHOENIX-3757
> Project: Phoenix
> Issue Type: Bug
>Reporter: Josh Elser
>Assignee: Karan Mehta
>Priority: Critical
> Labels: namespaces
> Fix For: 4.13.0
>
> Attachments: PHOENIX-3757.001.patch, PHOENIX-3757.002.patch,
> PHOENIX-3757.003.patch
>
>
> Noticed this issue while writing a test for PHOENIX-3756:
> The SYSTEM.MUTEX table is always created in the default namespace, even when
> {{phoenix.schema.isNamespaceMappingEnabled=true}}. At a glance, it looks like
> the logic for the other system tables isn't applied to the mutex table.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] phoenix pull request #277: PHOENIX-3757 System mutex table not being created...
Github user karanmehta93 commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/277#discussion_r146697994
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
---
@@ -3101,49 +3124,68 @@ void ensureSystemTablesUpgraded(ReadOnlyProps props)
// Regardless of the case 1 or 2, if the NS does not
exist, we will error expectedly
// below. If the NS does exist and is mapped, the below
check will exit gracefully.
}
-
+
List tableNames = getSystemTableNames(admin);
// No tables exist matching "SYSTEM\..*", they are all already
in "SYSTEM:.*"
if (tableNames.size() == 0) { return; }
// Try to move any remaining tables matching "SYSTEM\..*" into
"SYSTEM:"
if (tableNames.size() > 5) {
logger.warn("Expected 5 system tables but found " +
tableNames.size() + ":" + tableNames);
}
+
+// Try acquiring a lock in SYSMUTEX table before migrating the
tables since it involves disabling the table
+// If we cannot acquire lock, it means some old client is
either migrating SYSCAT or trying to upgrade the
+// schema of SYSCAT table and hence it should not be
interrupted
+acquiredMutexLock = acquireUpgradeMutex(0, mutexRowKey);
+if(acquiredMutexLock) {
+logger.debug("Acquired lock in SYSMUTEX table for
migrating SYSTEM tables to SYSTEM namespace");
+}
+// We will not reach here if we fail to acquire the lock,
since it throws UpgradeInProgressException
+
+// Handle the upgrade of SYSMUTEX table separately since it
doesn't have any entries in SYSCAT
+String sysMutexSrcTableName =
PhoenixDatabaseMetaData.SYSTEM_MUTEX_NAME;
+String sysMutexDestTableName =
SchemaUtil.getPhysicalName(sysMutexSrcTableName.getBytes(),
props).getNameAsString();
+UpgradeUtil.mapTableToNamespace(admin, sysMutexSrcTableName,
sysMutexDestTableName, PTableType.SYSTEM);
+
byte[] mappedSystemTable = SchemaUtil
.getPhysicalName(PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME_BYTES,
props).getName();
metatable = getTable(mappedSystemTable);
if
(tableNames.contains(PhoenixDatabaseMetaData.SYSTEM_CATALOG_HBASE_TABLE_NAME)) {
if (!admin.tableExists(mappedSystemTable)) {
+// Actual migration of SYSCAT table
UpgradeUtil.mapTableToNamespace(admin, metatable,
PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME,
props, null, PTableType.SYSTEM,
null);
+// Invalidate the client-side metadataCache
ConnectionQueryServicesImpl.this.removeTable(null,
PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME,
null,
MetaDataProtocol.MIN_SYSTEM_TABLE_TIMESTAMP_4_1_0);
}
tableNames.remove(PhoenixDatabaseMetaData.SYSTEM_CATALOG_HBASE_TABLE_NAME);
}
-
tableNames.remove(PhoenixDatabaseMetaData.SYSTEM_MUTEX_HBASE_TABLE_NAME);
for (TableName table : tableNames) {
UpgradeUtil.mapTableToNamespace(admin, metatable,
table.getNameAsString(), props, null, PTableType.SYSTEM,
null);
ConnectionQueryServicesImpl.this.removeTable(null,
table.getNameAsString(), null,
MetaDataProtocol.MIN_SYSTEM_TABLE_TIMESTAMP_4_1_0);
}
-if (!tableNames.isEmpty()) {
-clearCache();
-}
+
+// Clear the server-side metadataCache when all tables are
migrated so that the new PTable can be loaded with NS mapping
+clearCache();
--- End diff --
if (!tableNames.isEmpty()) {
clearCache();
}
I don't really understand why it was done this way, but since we are not
removing entries from the `tableNames` variable, this piece of code will always
get executed and clear the server side cache.
Also, we clear server side cache anytime a SYSTEM table is disabled, using
a coprocessor hook.
So I decided to simplify the logic and clear the cache every time. Its good
to do that because the server side cache will go out of sync with the SYSCAT
table once the migration happens.
For example, if the cached Ptable represented SYSTEM.CATALOG
(IS_NAMESPACE_MAPPED=false), the new Ptable should represent SYSTEM:CATALOG
(IS_NAMESPACE_MAPPED=true). This can onl
[jira] [Commented] (PHOENIX-3757) System mutex table not being created in SYSTEM namespace when namespace mapping is enabled
[
https://issues.apache.org/jira/browse/PHOENIX-3757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217730#comment-16217730
]
ASF GitHub Bot commented on PHOENIX-3757:
-
Github user karanmehta93 commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/277#discussion_r146697994
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
---
@@ -3101,49 +3124,68 @@ void ensureSystemTablesUpgraded(ReadOnlyProps props)
// Regardless of the case 1 or 2, if the NS does not
exist, we will error expectedly
// below. If the NS does exist and is mapped, the below
check will exit gracefully.
}
-
+
List tableNames = getSystemTableNames(admin);
// No tables exist matching "SYSTEM\..*", they are all already
in "SYSTEM:.*"
if (tableNames.size() == 0) { return; }
// Try to move any remaining tables matching "SYSTEM\..*" into
"SYSTEM:"
if (tableNames.size() > 5) {
logger.warn("Expected 5 system tables but found " +
tableNames.size() + ":" + tableNames);
}
+
+// Try acquiring a lock in SYSMUTEX table before migrating the
tables since it involves disabling the table
+// If we cannot acquire lock, it means some old client is
either migrating SYSCAT or trying to upgrade the
+// schema of SYSCAT table and hence it should not be
interrupted
+acquiredMutexLock = acquireUpgradeMutex(0, mutexRowKey);
+if(acquiredMutexLock) {
+logger.debug("Acquired lock in SYSMUTEX table for
migrating SYSTEM tables to SYSTEM namespace");
+}
+// We will not reach here if we fail to acquire the lock,
since it throws UpgradeInProgressException
+
+// Handle the upgrade of SYSMUTEX table separately since it
doesn't have any entries in SYSCAT
+String sysMutexSrcTableName =
PhoenixDatabaseMetaData.SYSTEM_MUTEX_NAME;
+String sysMutexDestTableName =
SchemaUtil.getPhysicalName(sysMutexSrcTableName.getBytes(),
props).getNameAsString();
+UpgradeUtil.mapTableToNamespace(admin, sysMutexSrcTableName,
sysMutexDestTableName, PTableType.SYSTEM);
+
byte[] mappedSystemTable = SchemaUtil
.getPhysicalName(PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME_BYTES,
props).getName();
metatable = getTable(mappedSystemTable);
if
(tableNames.contains(PhoenixDatabaseMetaData.SYSTEM_CATALOG_HBASE_TABLE_NAME)) {
if (!admin.tableExists(mappedSystemTable)) {
+// Actual migration of SYSCAT table
UpgradeUtil.mapTableToNamespace(admin, metatable,
PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME,
props, null, PTableType.SYSTEM,
null);
+// Invalidate the client-side metadataCache
ConnectionQueryServicesImpl.this.removeTable(null,
PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME,
null,
MetaDataProtocol.MIN_SYSTEM_TABLE_TIMESTAMP_4_1_0);
}
tableNames.remove(PhoenixDatabaseMetaData.SYSTEM_CATALOG_HBASE_TABLE_NAME);
}
-
tableNames.remove(PhoenixDatabaseMetaData.SYSTEM_MUTEX_HBASE_TABLE_NAME);
for (TableName table : tableNames) {
UpgradeUtil.mapTableToNamespace(admin, metatable,
table.getNameAsString(), props, null, PTableType.SYSTEM,
null);
ConnectionQueryServicesImpl.this.removeTable(null,
table.getNameAsString(), null,
MetaDataProtocol.MIN_SYSTEM_TABLE_TIMESTAMP_4_1_0);
}
-if (!tableNames.isEmpty()) {
-clearCache();
-}
+
+// Clear the server-side metadataCache when all tables are
migrated so that the new PTable can be loaded with NS mapping
+clearCache();
--- End diff --
if (!tableNames.isEmpty()) {
clearCache();
}
I don't really understand why it was done this way, but since we are not
removing entries from the `tableNames` variable, this piece of code will always
get executed and clear the server side cache.
Also, we clear server side cache anytime a SYSTEM table is disabled, using
a coprocessor hook.
So I decided to simplify the logic and clear the cache every time. Its good
to do that because the server side cach
[GitHub] phoenix pull request #277: PHOENIX-3757 System mutex table not being created...
Github user karanmehta93 commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/277#discussion_r146698425
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
---
@@ -3101,49 +3124,68 @@ void ensureSystemTablesUpgraded(ReadOnlyProps props)
// Regardless of the case 1 or 2, if the NS does not
exist, we will error expectedly
// below. If the NS does exist and is mapped, the below
check will exit gracefully.
}
-
+
List tableNames = getSystemTableNames(admin);
// No tables exist matching "SYSTEM\..*", they are all already
in "SYSTEM:.*"
if (tableNames.size() == 0) { return; }
// Try to move any remaining tables matching "SYSTEM\..*" into
"SYSTEM:"
if (tableNames.size() > 5) {
logger.warn("Expected 5 system tables but found " +
tableNames.size() + ":" + tableNames);
}
+
+// Try acquiring a lock in SYSMUTEX table before migrating the
tables since it involves disabling the table
+// If we cannot acquire lock, it means some old client is
either migrating SYSCAT or trying to upgrade the
+// schema of SYSCAT table and hence it should not be
interrupted
+acquiredMutexLock = acquireUpgradeMutex(0, mutexRowKey);
+if(acquiredMutexLock) {
+logger.debug("Acquired lock in SYSMUTEX table for
migrating SYSTEM tables to SYSTEM namespace");
+}
+// We will not reach here if we fail to acquire the lock,
since it throws UpgradeInProgressException
+
+// Handle the upgrade of SYSMUTEX table separately since it
doesn't have any entries in SYSCAT
+String sysMutexSrcTableName =
PhoenixDatabaseMetaData.SYSTEM_MUTEX_NAME;
+String sysMutexDestTableName =
SchemaUtil.getPhysicalName(sysMutexSrcTableName.getBytes(),
props).getNameAsString();
+UpgradeUtil.mapTableToNamespace(admin, sysMutexSrcTableName,
sysMutexDestTableName, PTableType.SYSTEM);
+
byte[] mappedSystemTable = SchemaUtil
.getPhysicalName(PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME_BYTES,
props).getName();
metatable = getTable(mappedSystemTable);
if
(tableNames.contains(PhoenixDatabaseMetaData.SYSTEM_CATALOG_HBASE_TABLE_NAME)) {
if (!admin.tableExists(mappedSystemTable)) {
+// Actual migration of SYSCAT table
UpgradeUtil.mapTableToNamespace(admin, metatable,
PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME,
props, null, PTableType.SYSTEM,
null);
+// Invalidate the client-side metadataCache
ConnectionQueryServicesImpl.this.removeTable(null,
PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME,
null,
MetaDataProtocol.MIN_SYSTEM_TABLE_TIMESTAMP_4_1_0);
}
tableNames.remove(PhoenixDatabaseMetaData.SYSTEM_CATALOG_HBASE_TABLE_NAME);
}
-
tableNames.remove(PhoenixDatabaseMetaData.SYSTEM_MUTEX_HBASE_TABLE_NAME);
for (TableName table : tableNames) {
UpgradeUtil.mapTableToNamespace(admin, metatable,
table.getNameAsString(), props, null, PTableType.SYSTEM,
null);
ConnectionQueryServicesImpl.this.removeTable(null,
table.getNameAsString(), null,
MetaDataProtocol.MIN_SYSTEM_TABLE_TIMESTAMP_4_1_0);
}
-if (!tableNames.isEmpty()) {
-clearCache();
-}
+
+// Clear the server-side metadataCache when all tables are
migrated so that the new PTable can be loaded with NS mapping
+clearCache();
} finally {
if (metatable != null) {
metatable.close();
}
+if(acquiredMutexLock) {
--- End diff --
@joshelser @ankitsinghal Can probably comment on this one.
---
[jira] [Commented] (PHOENIX-3757) System mutex table not being created in SYSTEM namespace when namespace mapping is enabled
[
https://issues.apache.org/jira/browse/PHOENIX-3757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217734#comment-16217734
]
ASF GitHub Bot commented on PHOENIX-3757:
-
Github user karanmehta93 commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/277#discussion_r146698425
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
---
@@ -3101,49 +3124,68 @@ void ensureSystemTablesUpgraded(ReadOnlyProps props)
// Regardless of the case 1 or 2, if the NS does not
exist, we will error expectedly
// below. If the NS does exist and is mapped, the below
check will exit gracefully.
}
-
+
List tableNames = getSystemTableNames(admin);
// No tables exist matching "SYSTEM\..*", they are all already
in "SYSTEM:.*"
if (tableNames.size() == 0) { return; }
// Try to move any remaining tables matching "SYSTEM\..*" into
"SYSTEM:"
if (tableNames.size() > 5) {
logger.warn("Expected 5 system tables but found " +
tableNames.size() + ":" + tableNames);
}
+
+// Try acquiring a lock in SYSMUTEX table before migrating the
tables since it involves disabling the table
+// If we cannot acquire lock, it means some old client is
either migrating SYSCAT or trying to upgrade the
+// schema of SYSCAT table and hence it should not be
interrupted
+acquiredMutexLock = acquireUpgradeMutex(0, mutexRowKey);
+if(acquiredMutexLock) {
+logger.debug("Acquired lock in SYSMUTEX table for
migrating SYSTEM tables to SYSTEM namespace");
+}
+// We will not reach here if we fail to acquire the lock,
since it throws UpgradeInProgressException
+
+// Handle the upgrade of SYSMUTEX table separately since it
doesn't have any entries in SYSCAT
+String sysMutexSrcTableName =
PhoenixDatabaseMetaData.SYSTEM_MUTEX_NAME;
+String sysMutexDestTableName =
SchemaUtil.getPhysicalName(sysMutexSrcTableName.getBytes(),
props).getNameAsString();
+UpgradeUtil.mapTableToNamespace(admin, sysMutexSrcTableName,
sysMutexDestTableName, PTableType.SYSTEM);
+
byte[] mappedSystemTable = SchemaUtil
.getPhysicalName(PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME_BYTES,
props).getName();
metatable = getTable(mappedSystemTable);
if
(tableNames.contains(PhoenixDatabaseMetaData.SYSTEM_CATALOG_HBASE_TABLE_NAME)) {
if (!admin.tableExists(mappedSystemTable)) {
+// Actual migration of SYSCAT table
UpgradeUtil.mapTableToNamespace(admin, metatable,
PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME,
props, null, PTableType.SYSTEM,
null);
+// Invalidate the client-side metadataCache
ConnectionQueryServicesImpl.this.removeTable(null,
PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME,
null,
MetaDataProtocol.MIN_SYSTEM_TABLE_TIMESTAMP_4_1_0);
}
tableNames.remove(PhoenixDatabaseMetaData.SYSTEM_CATALOG_HBASE_TABLE_NAME);
}
-
tableNames.remove(PhoenixDatabaseMetaData.SYSTEM_MUTEX_HBASE_TABLE_NAME);
for (TableName table : tableNames) {
UpgradeUtil.mapTableToNamespace(admin, metatable,
table.getNameAsString(), props, null, PTableType.SYSTEM,
null);
ConnectionQueryServicesImpl.this.removeTable(null,
table.getNameAsString(), null,
MetaDataProtocol.MIN_SYSTEM_TABLE_TIMESTAMP_4_1_0);
}
-if (!tableNames.isEmpty()) {
-clearCache();
-}
+
+// Clear the server-side metadataCache when all tables are
migrated so that the new PTable can be loaded with NS mapping
+clearCache();
} finally {
if (metatable != null) {
metatable.close();
}
+if(acquiredMutexLock) {
--- End diff --
@joshelser @ankitsinghal Can probably comment on this one.
> System mutex table not being created in SYSTEM namespace when namespace
> mapping is enabled
> --
>
> Key: PHOENIX-3757
> URL: https://issues.apache.org/jira/browse/PHOENIX-3757
[jira] [Updated] (PHOENIX-4292) Filters on Tables and Views with composite PK of VARCHAR fields with sort direction DESC do not work
[
https://issues.apache.org/jira/browse/PHOENIX-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas D'Silva updated PHOENIX-4292:
Attachment: PHOENIX-4292-addendum.patch
Attaching addendum that also tests views that pk formats that were working
correctly.
> Filters on Tables and Views with composite PK of VARCHAR fields with sort
> direction DESC do not work
>
>
> Key: PHOENIX-4292
> URL: https://issues.apache.org/jira/browse/PHOENIX-4292
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.10.0
>Reporter: Jan Fernando
>Assignee: Thomas D'Silva
> Fix For: 4.13.0, 4.12.1
>
> Attachments: PHOENIX-4292-addendum.patch, PHOENIX-4292.patch
>
>
> We noticed that in certain instances on tables and views that were defined
> with a Composite PK and where the elements of the PK were all DESC that
> queries exhibited strange behavior and did not return results when expected.
> A simple query on the first element of the PK returned 0 results e.g SELECT *
> FROM MY_TABLE WHERE PK1 = 'myvaluethatexists' would return 0 results.
> After some investigation it appears that querying tables and views with a
> Composite PK that :
> a) have multiple VARCHAR columns in the PK
> b) the sort direction of all the VARCHAR columns is defined as DESC
> does not work correctly and the filters are not honored and SQL appears
> broken to the end user.
> Detailed repro steps:
> ---
> -- 1. Create Global Base Table
> CREATE TABLE IF NOT EXISTS TEST.BASETABLE (
> TENANT_ID CHAR(15) NOT NULL,
> KEY_PREFIX CHAR(3) NOT NULL,
> CREATED_DATE DATE,
> CREATED_BY CHAR(15),
> SYSTEM_MODSTAMP DATE
> CONSTRAINT PK PRIMARY KEY (
> TENANT_ID,
> KEY_PREFIX
> )
> ) VERSIONS=1, MULTI_TENANT=true, IMMUTABLE_ROWS=TRUE, REPLICATION_SCOPE=1
> -- 2. Create various TENANT SPECIFIC VIEWS i.e. use a tenant specific
> connection
> CREATE VIEW IF NOT EXISTS TEST."abc" (pk1 VARCHAR(10) NOT NULL, pk2
> VARCHAR(10) NOT NULL, col1 DATE, col3 DECIMAL CONSTRAINT PK PRIMARY KEY (pk1
> DESC, pk2 DESC)) AS SELECT * FROM TEST.BASETABLE WHERE KEY_PREFIX = 'abc';
> CREATE VIEW IF NOT EXISTS TEST."ab2" (pk1 VARCHAR(10) NOT NULL, pk2
> VARCHAR(10) NOT NULL, col1 DATE, col3 DECIMAL CONSTRAINT PK PRIMARY KEY (pk1
> DESC, pk2 ASC)) AS SELECT * FROM TEST.BASETABLE WHERE KEY_PREFIX = 'ab2';
> CREATE VIEW IF NOT EXISTS TEST."ab3" (pk1 DATE(10) NOT NULL, pk2 DATE(10) NOT
> NULL, col1 VARCHAR(10), col3 DECIMAL CONSTRAINT PK PRIMARY KEY (pk1 DESC,
> pk2 DESC)) AS SELECT * FROM TEST.BASETABLE WHERE KEY_PREFIX = 'ab3';
> CREATE VIEW IF NOT EXISTS TEST."ab4" (pk1 DATE(10) NOT NULL, pk2 DECIMAL NOT
> NULL, pk3 VARCHAR(10) NOT NULL, col3 DECIMAL CONSTRAINT PK PRIMARY KEY (pk1
> DESC, pk2 DESC, pk3 DESC)) AS SELECT * FROM TEST.BASETABLE WHERE KEY_PREFIX =
> 'ab4';
> -- 3. Test cases that exhibit this issues
> -- SIMULATE EQUALITY QUERY PROBLEMS: View with composite PK with multiple PK
> values of VARCHAR values DESC
> upsert into TEST."abc" (pk1, pk2, col1, col3) VALUES ('testa', 'testb',
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10);
> upsert into TEST."abc" (pk1, pk2, col1, col3) VALUES ('testa', 'testc',
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10);
> upsert into TEST."abc" (pk1, pk2, col1, col3) VALUES ('testa', 'testd',
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10);
> upsert into TEST."abc" (pk1, pk2, col1, col3) VALUES ('testa', 'teste',
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10);
> upsert into TEST."abc" (pk1, pk2, col1, col3) VALUES ('testb', 'testa',
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10);
> SELECT * FROM TEST."abc" WHERE pk1 = 'testa'; -- INCORRECT RESULT: This query
> returns no records, expected to return 4
> SELECT * FROM TEST."abc"; -- Returns 5 rows as expected
> SELECT * FROM TEST."abc" WHERE pk1 >= 'testa'; -- INCORRECT RESULT: This
> query returns 1 record, expected to return 5
> SELECT * FROM TEST."abc" WHERE pk1 <= 'testa'; -- Returns 4 rows as expected
> SELECT * FROM TEST."abc" WHERE pk1 > 'testa'; -- Returns 1 row as expected
> SELECT * FROM TEST."abc" WHERE pk1 < 'testa'; -- INCORRECT RESULT: This query
> returns 1 record, expected to return 0
> -- The following are cases where everything works as expected and which don't
> have composite VARCHAR PKs
> -- DEMONOSTRATE NO QUERY PROBLEMS WHEN HAVE VIEW WITH SINGLE DESC TEXT PK:
> View with composite PK with single pk value DESC
> upsert into TEST."ab2" (pk1, pk2, col1, col3) VALUES ('testa', 'testb',
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10);
> upsert into TEST."ab2" (pk1, pk2, col1, col3) VALUES ('testa', 'testc',
> TO_DATE(
[jira] [Comment Edited] (PHOENIX-4292) Filters on Tables and Views with composite PK of VARCHAR fields with sort direction DESC do not work
[
https://issues.apache.org/jira/browse/PHOENIX-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217845#comment-16217845
]
Thomas D'Silva edited comment on PHOENIX-4292 at 10/24/17 10:56 PM:
Attaching addendum that also tests views with pk formats that were working
correctly.
was (Author: tdsilva):
Attaching addendum that also tests views that pk formats that were working
correctly.
> Filters on Tables and Views with composite PK of VARCHAR fields with sort
> direction DESC do not work
>
>
> Key: PHOENIX-4292
> URL: https://issues.apache.org/jira/browse/PHOENIX-4292
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.10.0
>Reporter: Jan Fernando
>Assignee: Thomas D'Silva
> Fix For: 4.13.0, 4.12.1
>
> Attachments: PHOENIX-4292-addendum.patch, PHOENIX-4292.patch
>
>
> We noticed that in certain instances on tables and views that were defined
> with a Composite PK and where the elements of the PK were all DESC that
> queries exhibited strange behavior and did not return results when expected.
> A simple query on the first element of the PK returned 0 results e.g SELECT *
> FROM MY_TABLE WHERE PK1 = 'myvaluethatexists' would return 0 results.
> After some investigation it appears that querying tables and views with a
> Composite PK that :
> a) have multiple VARCHAR columns in the PK
> b) the sort direction of all the VARCHAR columns is defined as DESC
> does not work correctly and the filters are not honored and SQL appears
> broken to the end user.
> Detailed repro steps:
> ---
> -- 1. Create Global Base Table
> CREATE TABLE IF NOT EXISTS TEST.BASETABLE (
> TENANT_ID CHAR(15) NOT NULL,
> KEY_PREFIX CHAR(3) NOT NULL,
> CREATED_DATE DATE,
> CREATED_BY CHAR(15),
> SYSTEM_MODSTAMP DATE
> CONSTRAINT PK PRIMARY KEY (
> TENANT_ID,
> KEY_PREFIX
> )
> ) VERSIONS=1, MULTI_TENANT=true, IMMUTABLE_ROWS=TRUE, REPLICATION_SCOPE=1
> -- 2. Create various TENANT SPECIFIC VIEWS i.e. use a tenant specific
> connection
> CREATE VIEW IF NOT EXISTS TEST."abc" (pk1 VARCHAR(10) NOT NULL, pk2
> VARCHAR(10) NOT NULL, col1 DATE, col3 DECIMAL CONSTRAINT PK PRIMARY KEY (pk1
> DESC, pk2 DESC)) AS SELECT * FROM TEST.BASETABLE WHERE KEY_PREFIX = 'abc';
> CREATE VIEW IF NOT EXISTS TEST."ab2" (pk1 VARCHAR(10) NOT NULL, pk2
> VARCHAR(10) NOT NULL, col1 DATE, col3 DECIMAL CONSTRAINT PK PRIMARY KEY (pk1
> DESC, pk2 ASC)) AS SELECT * FROM TEST.BASETABLE WHERE KEY_PREFIX = 'ab2';
> CREATE VIEW IF NOT EXISTS TEST."ab3" (pk1 DATE(10) NOT NULL, pk2 DATE(10) NOT
> NULL, col1 VARCHAR(10), col3 DECIMAL CONSTRAINT PK PRIMARY KEY (pk1 DESC,
> pk2 DESC)) AS SELECT * FROM TEST.BASETABLE WHERE KEY_PREFIX = 'ab3';
> CREATE VIEW IF NOT EXISTS TEST."ab4" (pk1 DATE(10) NOT NULL, pk2 DECIMAL NOT
> NULL, pk3 VARCHAR(10) NOT NULL, col3 DECIMAL CONSTRAINT PK PRIMARY KEY (pk1
> DESC, pk2 DESC, pk3 DESC)) AS SELECT * FROM TEST.BASETABLE WHERE KEY_PREFIX =
> 'ab4';
> -- 3. Test cases that exhibit this issues
> -- SIMULATE EQUALITY QUERY PROBLEMS: View with composite PK with multiple PK
> values of VARCHAR values DESC
> upsert into TEST."abc" (pk1, pk2, col1, col3) VALUES ('testa', 'testb',
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10);
> upsert into TEST."abc" (pk1, pk2, col1, col3) VALUES ('testa', 'testc',
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10);
> upsert into TEST."abc" (pk1, pk2, col1, col3) VALUES ('testa', 'testd',
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10);
> upsert into TEST."abc" (pk1, pk2, col1, col3) VALUES ('testa', 'teste',
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10);
> upsert into TEST."abc" (pk1, pk2, col1, col3) VALUES ('testb', 'testa',
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10);
> SELECT * FROM TEST."abc" WHERE pk1 = 'testa'; -- INCORRECT RESULT: This query
> returns no records, expected to return 4
> SELECT * FROM TEST."abc"; -- Returns 5 rows as expected
> SELECT * FROM TEST."abc" WHERE pk1 >= 'testa'; -- INCORRECT RESULT: This
> query returns 1 record, expected to return 5
> SELECT * FROM TEST."abc" WHERE pk1 <= 'testa'; -- Returns 4 rows as expected
> SELECT * FROM TEST."abc" WHERE pk1 > 'testa'; -- Returns 1 row as expected
> SELECT * FROM TEST."abc" WHERE pk1 < 'testa'; -- INCORRECT RESULT: This query
> returns 1 record, expected to return 0
> -- The following are cases where everything works as expected and which don't
> have composite VARCHAR PKs
> -- DEMONOSTRATE NO QUERY PROBLEMS WHEN HAVE VIEW WITH SINGLE DESC TEXT PK:
> View with composite PK with single pk value DESC
> upsert into TEST."ab2" (pk1, pk2, col1, col
[jira] [Updated] (PHOENIX-4290) Full table scan performed for DELETE with table having immutable indexes
[ https://issues.apache.org/jira/browse/PHOENIX-4290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated PHOENIX-4290: -- Attachment: PHOENIX-4290_wip1.patch > Full table scan performed for DELETE with table having immutable indexes > > > Key: PHOENIX-4290 > URL: https://issues.apache.org/jira/browse/PHOENIX-4290 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: James Taylor > Fix For: 4.13.0, 4.12.1 > > Attachments: PHOENIX-4290_wip1.patch > > > If a DELETE command is issued with a partial match for the leading part of > the primary key, instead of using the data table, when the table has > immutable indexes, a full scan will occur against the index. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (PHOENIX-4290) Full table scan performed for DELETE with table having immutable indexes
[ https://issues.apache.org/jira/browse/PHOENIX-4290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated PHOENIX-4290: -- Fix Version/s: 4.13.0 > Full table scan performed for DELETE with table having immutable indexes > > > Key: PHOENIX-4290 > URL: https://issues.apache.org/jira/browse/PHOENIX-4290 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: James Taylor > Fix For: 4.13.0, 4.12.1 > > Attachments: PHOENIX-4290_wip1.patch > > > If a DELETE command is issued with a partial match for the leading part of > the primary key, instead of using the data table, when the table has > immutable indexes, a full scan will occur against the index. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (PHOENIX-4292) Filters on Tables and Views with composite PK of VARCHAR fields with sort direction DESC do not work
[
https://issues.apache.org/jira/browse/PHOENIX-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16218069#comment-16218069
]
Hudson commented on PHOENIX-4292:
-
FAILURE: Integrated in Jenkins build Phoenix-master #1847 (See
[https://builds.apache.org/job/Phoenix-master/1847/])
PHOENIX-4292 Filters on Tables and Views with composite PK of VARCHAR (tdsilva:
rev fe13b257e5dfe29581b1c3265d79596f194954cd)
* (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/ViewIT.java
> Filters on Tables and Views with composite PK of VARCHAR fields with sort
> direction DESC do not work
>
>
> Key: PHOENIX-4292
> URL: https://issues.apache.org/jira/browse/PHOENIX-4292
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.10.0
>Reporter: Jan Fernando
>Assignee: Thomas D'Silva
> Fix For: 4.13.0, 4.12.1
>
> Attachments: PHOENIX-4292-addendum.patch, PHOENIX-4292.patch
>
>
> We noticed that in certain instances on tables and views that were defined
> with a Composite PK and where the elements of the PK were all DESC that
> queries exhibited strange behavior and did not return results when expected.
> A simple query on the first element of the PK returned 0 results e.g SELECT *
> FROM MY_TABLE WHERE PK1 = 'myvaluethatexists' would return 0 results.
> After some investigation it appears that querying tables and views with a
> Composite PK that :
> a) have multiple VARCHAR columns in the PK
> b) the sort direction of all the VARCHAR columns is defined as DESC
> does not work correctly and the filters are not honored and SQL appears
> broken to the end user.
> Detailed repro steps:
> ---
> -- 1. Create Global Base Table
> CREATE TABLE IF NOT EXISTS TEST.BASETABLE (
> TENANT_ID CHAR(15) NOT NULL,
> KEY_PREFIX CHAR(3) NOT NULL,
> CREATED_DATE DATE,
> CREATED_BY CHAR(15),
> SYSTEM_MODSTAMP DATE
> CONSTRAINT PK PRIMARY KEY (
> TENANT_ID,
> KEY_PREFIX
> )
> ) VERSIONS=1, MULTI_TENANT=true, IMMUTABLE_ROWS=TRUE, REPLICATION_SCOPE=1
> -- 2. Create various TENANT SPECIFIC VIEWS i.e. use a tenant specific
> connection
> CREATE VIEW IF NOT EXISTS TEST."abc" (pk1 VARCHAR(10) NOT NULL, pk2
> VARCHAR(10) NOT NULL, col1 DATE, col3 DECIMAL CONSTRAINT PK PRIMARY KEY (pk1
> DESC, pk2 DESC)) AS SELECT * FROM TEST.BASETABLE WHERE KEY_PREFIX = 'abc';
> CREATE VIEW IF NOT EXISTS TEST."ab2" (pk1 VARCHAR(10) NOT NULL, pk2
> VARCHAR(10) NOT NULL, col1 DATE, col3 DECIMAL CONSTRAINT PK PRIMARY KEY (pk1
> DESC, pk2 ASC)) AS SELECT * FROM TEST.BASETABLE WHERE KEY_PREFIX = 'ab2';
> CREATE VIEW IF NOT EXISTS TEST."ab3" (pk1 DATE(10) NOT NULL, pk2 DATE(10) NOT
> NULL, col1 VARCHAR(10), col3 DECIMAL CONSTRAINT PK PRIMARY KEY (pk1 DESC,
> pk2 DESC)) AS SELECT * FROM TEST.BASETABLE WHERE KEY_PREFIX = 'ab3';
> CREATE VIEW IF NOT EXISTS TEST."ab4" (pk1 DATE(10) NOT NULL, pk2 DECIMAL NOT
> NULL, pk3 VARCHAR(10) NOT NULL, col3 DECIMAL CONSTRAINT PK PRIMARY KEY (pk1
> DESC, pk2 DESC, pk3 DESC)) AS SELECT * FROM TEST.BASETABLE WHERE KEY_PREFIX =
> 'ab4';
> -- 3. Test cases that exhibit this issues
> -- SIMULATE EQUALITY QUERY PROBLEMS: View with composite PK with multiple PK
> values of VARCHAR values DESC
> upsert into TEST."abc" (pk1, pk2, col1, col3) VALUES ('testa', 'testb',
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10);
> upsert into TEST."abc" (pk1, pk2, col1, col3) VALUES ('testa', 'testc',
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10);
> upsert into TEST."abc" (pk1, pk2, col1, col3) VALUES ('testa', 'testd',
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10);
> upsert into TEST."abc" (pk1, pk2, col1, col3) VALUES ('testa', 'teste',
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10);
> upsert into TEST."abc" (pk1, pk2, col1, col3) VALUES ('testb', 'testa',
> TO_DATE('2017-10-16 22:00:00', '-MM-dd HH:mm:ss'), 10);
> SELECT * FROM TEST."abc" WHERE pk1 = 'testa'; -- INCORRECT RESULT: This query
> returns no records, expected to return 4
> SELECT * FROM TEST."abc"; -- Returns 5 rows as expected
> SELECT * FROM TEST."abc" WHERE pk1 >= 'testa'; -- INCORRECT RESULT: This
> query returns 1 record, expected to return 5
> SELECT * FROM TEST."abc" WHERE pk1 <= 'testa'; -- Returns 4 rows as expected
> SELECT * FROM TEST."abc" WHERE pk1 > 'testa'; -- Returns 1 row as expected
> SELECT * FROM TEST."abc" WHERE pk1 < 'testa'; -- INCORRECT RESULT: This query
> returns 1 record, expected to return 0
> -- The following are cases where everything works as expected and which don't
> have composite VARCHAR PKs
> -- DEMONOSTRATE NO QUERY PROBLEMS WHEN HAVE VIEW WITH SINGLE DESC TEXT PK:
> View with composite PK with single pk value
[jira] [Commented] (PHOENIX-4290) Full table scan performed for DELETE with table having immutable indexes
[
https://issues.apache.org/jira/browse/PHOENIX-4290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16218081#comment-16218081
]
Hadoop QA commented on PHOENIX-4290:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12893857/PHOENIX-4290_wip1.patch
against master branch at commit 7cdcb2313b08d2eaeb775f0c989642f8d416cfb6.
ATTACHMENT ID: 12893857
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+PreparedStatement psDelete = con.prepareStatement("DELETE FROM
" + tableName + " WHERE (HOST, DOMAIN, FEATURE, \"DATE\") = (?,?,?,?)");
+rs = con.createStatement().executeQuery("SELECT /*+ NO_INDEX */
count(*) FROM " + tableName);
+private static MutationState deleteRows(StatementContext context,
QueryPlan dataPlan, List indexTableRefs, ResultIterator iterator,
+RowProjector projector, TableRef sourceTableRef, final
MapvalueGetterMap) throws SQLException {
+public ImmutableBytesWritable getLatestValue(ColumnReference
ref, long ts) throws IOException {
+valuePtr.set(cell.getValueArray(), cell.getValueOffset(),
cell.getValueLength());
+// Create IndexMaintainer based on projected table
(i.e. SELECT expressions) so that client-side
+IndexMaintainer maintainer =
IndexMaintainer.create(projectedTable, indexTableRefs.get(i).getTable(),
connection);
+indexPtr.set(maintainer.buildRowKey(getter, indexPtr,
null, null, HConstants.LATEST_TIMESTAMP));
+MutationState state = deleteRows(ctx, dataQueryPlan,
indexTableRefs, iterator, projector, sourceTableRef, valueGetterMap);
{color:red}-1 core tests{color}. The patch failed these unit tests:
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.join.SubqueryIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.DeleteIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.index.ViewIndexIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.index.GlobalImmutableNonTxIndexIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.tx.TxCheckpointIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.index.GlobalImmutableTxIndexIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.index.IndexMaintenanceIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.ConcurrentMutationsIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.index.ImmutableIndexIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.IndexExtendedIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.TenantSpecificTablesDMLIT
Test results:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1569//testReport/
Console output:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1569//console
This message is automatically generated.
> Full table scan performed for DELETE with table having immutable indexes
>
>
> Key: PHOENIX-4290
> URL: https://issues.apache.org/jira/browse/PHOENIX-4290
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: James Taylor
> Fix For: 4.13.0, 4.12.1
>
> Attachments: PHOENIX-4290_wip1.patch
>
>
> If a DELETE command is issued with a partial match for the leading part of
> the primary key, instead of using the data table, when the table has
> immutable indexes, a full scan will occur against the index.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
