[GitHub] phoenix pull request #317: PHOENIX-3547 Supporting more number of indices pe...
GitHub user m2je opened a pull request: https://github.com/apache/phoenix/pull/317 PHOENIX-3547 Supporting more number of indices per table. Currently the number of indices per Phoenix table is bound to maximum of 65535 (java.lang.Short) which is a limitation for applications requiring to have unlimited number of indices. This change will consider any new table created in Phoenix to support view index ids to be in the range of -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807 (java.lang.Long) which is undoubtably big enough to cover this requirement. Any existing Phoenix table will still continue to support only maximum of 65535 of indices. A new boolean column (USE_LONG_VIEW_INDEX BOOLEAN DEFAULT FALSE) is added to SYSTEM.CATALOG to specify each Phoenix table's support for large number of indices. On each new Phoenix table creation the value for USE_LONG_VIEW_INDEX will be set to `true` while this value would be false for any existing table. You can merge this pull request into a Git repository by running: $ git pull https://github.com/m2je/phoenix PHOENIX-3547 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/phoenix/pull/317.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #317 commit f1ffcf6b76a0a764df5529a03ba8dc42112dc5a9 Author: Mahdi Salarkia Date: 2018-07-28T05:27:46Z PHOENIX-3547 Supporting more number of indices per table. Currently the number of indices per Phoenix table is bound to maximum of 65535 (java.lang.Short) which is a limitation for applications requiring to have unlimited number of indices. This change will consider any new table created in Phoenix to support view index ids to be in the range of -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807 (java.lang.Long) which is undoubtably big enough to cover this requirement. Any existing Phoenix table will still continue to support only maximum of 65535 of indices. A new boolean column (USE_LONG_VIEW_INDEX BOOLEAN DEFAULT FALSE) is added to SYSTEM.CATALOG to specify each Phoenix table's support for large number of indices. On each new Phoenix table creation the value for USE_LONG_VIEW_INDEX will be set to `true` while this value would be false for any existing table. ---
[GitHub] phoenix pull request #313: PHOENIX-4799 Write cells using checkAndMutate to ...
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/313#discussion_r205950104
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/schema/MetaDataClient.java ---
@@ -3034,6 +3088,11 @@ MutationState dropTable(String schemaName, String
tableName, String parentTableN
return new MutationState(0, 0, connection);
} finally {
connection.setAutoCommit(wasAutoCommit);
+// acquire a mutex on the table to prevent creating views
while concurrently
+// dropping the base table
+if (tableType == PTableType.TABLE) {
+deleteCell(null, schemaName, tableName, null);
+}
--- End diff --
Done.
---
[GitHub] phoenix pull request #313: PHOENIX-4799 Write cells using checkAndMutate to ...
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/313#discussion_r205950165
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/schema/MetaDataClient.java ---
@@ -3604,6 +3675,17 @@ public MutationState addColumn(PTable table,
List origColumnDefs,
}
} finally {
connection.setAutoCommit(wasAutoCommit);
+if (!columns.isEmpty()) {
+for (PColumn pColumn : columns) {
+PName physicalName = table.getPhysicalName();
+String physicalSchemaName =
+
SchemaUtil.getSchemaNameFromFullName(physicalName.getString());
+String physicalTableName =
+
SchemaUtil.getTableNameFromFullName(physicalName.getString());
+deleteCell(null, physicalSchemaName, physicalTableName,
--- End diff --
I changed the code to only delete the cells if we were able to successfully
do the checkAndPut.
When we add a column we write a cell per column that we are creating.
When we drop a base table or create a view, we write a single cell with the
rowkey of the physical table (to prevent a view being created while we are
dropping the base table).
---
[GitHub] phoenix issue #313: PHOENIX-4799 Write cells using checkAndMutate to prevent...
Github user twdsilva commented on the issue: https://github.com/apache/phoenix/pull/313 @ankitsinghal Thanks for the review. I have updated the PR to based on the feedback. Can you please take a look? ---
[jira] [Updated] (PHOENIX-4799) Write cells using checkAndMutate to prevent conflicting changes
[ https://issues.apache.org/jira/browse/PHOENIX-4799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas D'Silva updated PHOENIX-4799: Attachment: PHOENIX-4799-v3.patch > Write cells using checkAndMutate to prevent conflicting changes > --- > > Key: PHOENIX-4799 > URL: https://issues.apache.org/jira/browse/PHOENIX-4799 > Project: Phoenix > Issue Type: Sub-task >Affects Versions: 4.15.0, 5.1.0 >Reporter: Thomas D'Silva >Assignee: Thomas D'Silva >Priority: Major > Attachments: PHOENIX-4799-v2.patch, PHOENIX-4799-v3.patch > > > In order to prevent race conditions when multiple client try to mutate the > same column before sending the request to mutate the column to the server do > a checkAndMutate with the column name being added/dropped. Also: > 1. When a view is created do a checkAndMutate with the columns in the view > where clause. > 2. When an index on a view is created do a checkAndMutate with the indexed > columns. > > To prevent a race condition in the DROP TABLE CASCADE case, when a table is > dropped do a checkAndMutate with the rowkey of the base table name. If a view > is created it also does a checkAndMutate with the same rowkey. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (PHOENIX-4824) Update BRANCH_NAMES in dev/test-patch.properties
Thomas D'Silva created PHOENIX-4824: --- Summary: Update BRANCH_NAMES in dev/test-patch.properties Key: PHOENIX-4824 URL: https://issues.apache.org/jira/browse/PHOENIX-4824 Project: Phoenix Issue Type: Bug Reporter: Thomas D'Silva Assignee: Thomas D'Silva -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PHOENIX-4824) Update BRANCH_NAMES in dev/test-patch.properties
[ https://issues.apache.org/jira/browse/PHOENIX-4824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas D'Silva updated PHOENIX-4824: Attachment: PHOENIX-4824.patch > Update BRANCH_NAMES in dev/test-patch.properties > > > Key: PHOENIX-4824 > URL: https://issues.apache.org/jira/browse/PHOENIX-4824 > Project: Phoenix > Issue Type: Bug >Reporter: Thomas D'Silva >Assignee: Thomas D'Silva >Priority: Major > Attachments: PHOENIX-4824.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (PHOENIX-3547) Promote CATALOG.VIEW_INDEX_ID to an int
[ https://issues.apache.org/jira/browse/PHOENIX-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas D'Silva reassigned PHOENIX-3547: --- Assignee: Mehdi Salarkia > Promote CATALOG.VIEW_INDEX_ID to an int > --- > > Key: PHOENIX-3547 > URL: https://issues.apache.org/jira/browse/PHOENIX-3547 > Project: Phoenix > Issue Type: New Feature >Reporter: Jeremy Huffman >Assignee: Mehdi Salarkia >Priority: Minor > > Increase the size of CATALOG.VIEW_INDEX_ID from smallint to int to support a > large number of indexed views on a single table. > Per James: "The code would just need to be tolerant when reading the data if > the length is two byte short versus four byte int. At write time, we'd just > always write an int." > See: > https://lists.apache.org/thread.html/22849e4fc73452cee3bea763cf6d5af7164dedcb44573ba6b9f452a2@%3Cuser.phoenix.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PHOENIX-4824) Update BRANCH_NAMES in dev/test-patch.properties
[ https://issues.apache.org/jira/browse/PHOENIX-4824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas D'Silva updated PHOENIX-4824: Attachment: PHOENIX-4824-v2.patch > Update BRANCH_NAMES in dev/test-patch.properties > > > Key: PHOENIX-4824 > URL: https://issues.apache.org/jira/browse/PHOENIX-4824 > Project: Phoenix > Issue Type: Bug >Reporter: Thomas D'Silva >Assignee: Thomas D'Silva >Priority: Major > Attachments: PHOENIX-4824-v2.patch, PHOENIX-4824.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] phoenix pull request #317: PHOENIX-3547 Supporting more number of indices pe...
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/317#discussion_r205952155
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/index/IndexMaintainer.java ---
@@ -1223,7 +1225,9 @@ public void readFields(DataInput input) throws
IOException {
boolean hasViewIndexId = encodedIndexedColumnsAndViewId < 0;
if (hasViewIndexId) {
// Fixed length
-viewIndexId = new
byte[MetaDataUtil.getViewIndexIdDataType().getByteSize()];
+//Use leacy viewIndexIdType for clients older than 4.10 release
--- End diff --
nit: typo
---
[GitHub] phoenix pull request #317: PHOENIX-3547 Supporting more number of indices pe...
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/317#discussion_r205952450
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
---
@@ -2185,7 +2223,7 @@ public void createTable(RpcController controller,
CreateTableRequest request,
throw sqlExceptions[0];
}
long seqValue = seqValues[0];
-if (seqValue > Short.MAX_VALUE) {
+if (seqValue > Long.MAX_VALUE) {
--- End diff --
This check is no longer needed since the indexId is now a Long. When you
call incrementSequences if the current sequence value is LONG.MAX_VALUE you
will get an exception.
---
[GitHub] phoenix pull request #317: PHOENIX-3547 Supporting more number of indices pe...
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/317#discussion_r205953961
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/iterate/ExplainTable.java ---
@@ -204,14 +205,19 @@ private void appendPKColumnValue(StringBuilder buf,
byte[] range, Boolean isNull
range = ptr.get();
}
if (changeViewIndexId) {
-Short s = (Short) type.toObject(range);
-s = (short) (s + (-Short.MAX_VALUE));
-buf.append(s.toString());
+PDataType viewIndexDataType =
tableRef.getTable().getViewIndexType();
+buf.append(getViewIndexValue(type, range,
viewIndexDataType).toString());
} else {
Format formatter = context.getConnection().getFormatter(type);
buf.append(type.toStringLiteral(range, formatter));
}
}
+
+private Long getViewIndexValue(PDataType type, byte[] range, PDataType
viewIndexDataType){
+boolean useLongViewIndex =
MetaDataUtil.getViewIndexIdDataType().equals(viewIndexDataType);
+Object s = type.toObject(range);
+return (useLongViewIndex ? (Long) s : (Short) s) -
(useLongViewIndex ? Long.MAX_VALUE : Short.MAX_VALUE);
--- End diff --
@JamesRTaylor or @chrajeshbabu do you know if this change is correct?
---
[GitHub] phoenix pull request #317: PHOENIX-3547 Supporting more number of indices pe...
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/317#discussion_r205954264
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
---
@@ -1431,11 +1438,42 @@ private PTable getTable(RegionScanner scanner, long
clientTimeStamp, long tableT
// server while holding this lock is a bad idea and likely to
cause contention.
return PTableImpl.makePTable(tenantId, schemaName, tableName,
tableType, indexState, timeStamp, tableSeqNum,
pkName, saltBucketNum, columns, parentSchemaName,
parentTableName, indexes, isImmutableRows, physicalTables, defaultFamilyName,
-viewStatement, disableWAL, multiTenant, storeNulls,
viewType, viewIndexId, indexType,
+viewStatement, disableWAL, multiTenant, storeNulls,
viewType, viewIndexType, viewIndexId, indexType,
rowKeyOrderOptimizable, transactionProvider,
updateCacheFrequency, baseColumnCount,
indexDisableTimestamp, isNamespaceMapped,
autoPartitionSeq, isAppendOnlySchema, storageScheme, encodingScheme, cqCounter,
useStatsForParallelization);
}
+private Long getViewIndexId(Cell[] tableKeyValues, PDataType
viewIndexType) {
+Cell viewIndexIdKv = tableKeyValues[VIEW_INDEX_ID_INDEX];
+return viewIndexIdKv == null ? null :
+decodeViewIndexId(viewIndexIdKv, viewIndexType);
+}
+/**
+ * check the value for {@value USE_LONG_VIEW_INDEX} and if its present
consider viewIndexId as long otherwise
+ * read as short and convert it to long
+ *
+ * @param tableKeyValues
+ * @param viewIndexType
+ * @return
+ */
+private Long decodeViewIndexId(Cell viewIndexIdKv, PDataType
viewIndexType) {
+boolean useLongViewIndex =
MetaDataUtil.getViewIndexIdDataType().equals(viewIndexType);
+ return new Long(
+ useLongViewIndex
+ ?
viewIndexType.getCodec().decodeLong(viewIndexIdKv.getValueArray(),
+ viewIndexIdKv.getValueOffset(),
SortOrder.getDefault())
+ :
MetaDataUtil.getLegacyViewIndexIdDataType().getCodec().decodeShort(viewIndexIdKv.getValueArray(),
+ viewIndexIdKv.getValueOffset(),
SortOrder.getDefault())
+ );
+}
+
+private PDataType getViewIndexType(Cell[] tableKeyValues) {
+Cell useLongViewIndexKv = tableKeyValues[USE_LONG_VIEW_INDEX];
--- End diff --
For existing tables the VIEW_INDEX_ID column will store the index as a
short while for new tables the value will be stored as a long.
Would it be cleaner to create a new column that stores the view index id as
a long. In QueryServicesConnectionImpl.upgradeSystemTables() we would set the
new column based on the existing value. Finally we can remove the existing
VIEW_INDEX_ID in the next release.
WDYT @JamesRTaylor ?
---
[jira] [Assigned] (PHOENIX-4597) Do not initalize phoenixTransactionContext in MutationState if transactions are not enabled.
[ https://issues.apache.org/jira/browse/PHOENIX-4597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xu Cang reassigned PHOENIX-4597: Assignee: Xu Cang > Do not initalize phoenixTransactionContext in MutationState if transactions > are not enabled. > > > Key: PHOENIX-4597 > URL: https://issues.apache.org/jira/browse/PHOENIX-4597 > Project: Phoenix > Issue Type: Improvement >Reporter: Thomas D'Silva >Assignee: Xu Cang >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PHOENIX-4597) Do not initalize phoenixTransactionContext in MutationState if transactions are not enabled.
[ https://issues.apache.org/jira/browse/PHOENIX-4597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xu Cang updated PHOENIX-4597: - Attachment: PHOENIX-4597.master.001.patch > Do not initalize phoenixTransactionContext in MutationState if transactions > are not enabled. > > > Key: PHOENIX-4597 > URL: https://issues.apache.org/jira/browse/PHOENIX-4597 > Project: Phoenix > Issue Type: Improvement >Reporter: Thomas D'Silva >Assignee: Xu Cang >Priority: Major > Attachments: PHOENIX-4597.master.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
