[
https://issues.apache.org/jira/browse/PHOENIX-1634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14321610#comment-14321610
]
James Taylor edited comment on PHOENIX-1634 at 2/14/15 6:25 PM:
----------------------------------------------------------------
[~rajeshbabu] - what's your ETA for the patch, as we'll roll a new RC when this
comes in? For the updates to existing local index tables, you can take the same
approach we do when the SYSTEM.CATALOG has a column added to it (which
minimizes the upgrade code to only be potentially run on first connection to a
cluster, and then only if it hasn't been run before):
- bump up the MIN_SYSTEM_TABLE_TIMESTAMP and create a new static for
MIN_SYSTEM_TABLE_TIMESTAMP_4_3_0 in MetaDataProtocol
- in ConnectionQueryServicesImpl.init(), in the catch
TableAlreadyExistsException, create an if block for if
(currentServerSideTableTimeStamp < MIN_SYSTEM_TABLE_TIMESTAMP_4_3_0). You can
move the setting of columnsToAdd += PhoenixDatabaseMetaData.STORE_NULLS + " " +
PBoolean.INSTANCE.getSqlTypeName(); into that block as well.
- within that if block, iterate through all index tables. You can do that with
code like this:
{code}
ResultSet rs = metaConnection.getTables(null, null, null, Arrays.asList(new
String[] {PTableType.INDEX.toString()}));
while (rs.next()) {
if (IndexType.LOCAL.name().equals(resultSet.getString("INDEX_TYPE"))) {
String tableName = SchemaUtil.getTableName(rs.getString(2),
rs.getString(3));
PTable ptable = PhoenixRuntime.getTable(metaConnection, tableName);
byte[] htableName = ptable.getPhysicalName().getBytes();
HTableInterface htable = getTable(htableName);
... // Add missing local index properties to local index HTable
}
{code}
- for the other change, delaying the creation of the local index, I think
you'll just need to change a couple of lines in
ConnectionQueryServicesImpl.createTable(): 1) remove the call to
ensureLocalIndexTableCreated() on line 1227 so that it's not created when a
multi tenant table is created, and 2) remove the !isMultiTenant() check in the
if on line 1200, so that the local index is not created on demand for multi
tenant tables. Move that isMultiTenant() check to inside the else block
surrounding the ensureViewIndexTableCreated call instead:
{code}
@Override
public MetaDataMutationResult createTable(final List<Mutation>
tableMetaData, byte[] physicalTableName, PTableType tableType,
Map<String,Object> tableProps, final
List<Pair<byte[],Map<String,Object>>> families, byte[][] splits) throws
SQLException {
byte[][] rowKeyMetadata = new byte[3][];
Mutation m = MetaDataUtil.getPutOnlyTableHeaderRow(tableMetaData);
byte[] key = m.getRow();
SchemaUtil.getVarChars(key, rowKeyMetadata);
byte[] tenantIdBytes =
rowKeyMetadata[PhoenixDatabaseMetaData.TENANT_ID_INDEX];
byte[] schemaBytes =
rowKeyMetadata[PhoenixDatabaseMetaData.SCHEMA_NAME_INDEX];
byte[] tableBytes =
rowKeyMetadata[PhoenixDatabaseMetaData.TABLE_NAME_INDEX];
byte[] tableName = physicalTableName != null ? physicalTableName :
SchemaUtil.getTableNameAsBytes(schemaBytes, tableBytes);
boolean localIndexTable =
Boolean.TRUE.equals(tableProps.remove(MetaDataUtil.IS_LOCAL_INDEX_TABLE_PROP_NAME));
if ((tableType == PTableType.VIEW && physicalTableName != null) ||
(tableType != PTableType.VIEW && physicalTableName == null)) {
// For views this will ensure that metadata already exists
// For tables and indexes, this will create the metadata if it
doesn't already exist
ensureTableCreated(tableName, tableType, tableProps, families,
splits, true);
}
ImmutableBytesWritable ptr = new ImmutableBytesWritable();
if (tableType == PTableType.INDEX) { // Index on view
// Physical index table created up front for multi tenant
// TODO: if viewIndexId is Short.MIN_VALUE, then we don't need to
attempt to create it
if (physicalTableName != null && !MetaDataUtil.isMultiTenant(m,
kvBuilder, ptr)) {
if (localIndexTable) {
ensureLocalIndexTableCreated(tableName, tableProps,
families, splits, MetaDataUtil.getClientTimeStamp(m));
} else {
ensureViewIndexTableCreated(tenantIdBytes.length == 0 ?
null : PNameFactory.newName(tenantIdBytes), physicalTableName,
MetaDataUtil.getClientTimeStamp(m));
}
}
} else if (tableType == PTableType.TABLE &&
MetaDataUtil.isMultiTenant(m, kvBuilder, ptr)) { // Create view index table up
front for multi tenant tables
{code}
was (Author: jamestaylor):
[~rajeshbabu] - what's your ETA for the patch, as we'll roll a new RC when this
comes in? For the updates to existing local index tables, you can take the same
approach we do when the SYSTEM.CATALOG has a column added to it (which
minimizes the upgrade code to only be potentially run on first connection to a
cluster, and then only if it hasn't been run before):
- bump up the MIN_SYSTEM_TABLE_TIMESTAMP and create a new static for
MIN_SYSTEM_TABLE_TIMESTAMP_4_3_0 in MetaDataProtocol
- in ConnectionQueryServicesImpl.init(), in the catch
TableAlreadyExistsException, create an if block for if
(currentServerSideTableTimeStamp < MIN_SYSTEM_TABLE_TIMESTAMP_4_3_0). You can
move the setting of columnsToAdd += PhoenixDatabaseMetaData.STORE_NULLS + " " +
PBoolean.INSTANCE.getSqlTypeName(); into that block as well.
- within that if block, iterate through all index tables. You can do that with
the following command: metaConnection.getTables(null, null, null,
Arrays.asList(new String[] {PTableType.INDEX.toString()})). Then you can
iterate through each row and if
(IndexType.LOCAL.name().equals(resultSet.getString("INDEX_TYPE"))) then call
PhoenixRuntime.getTable(metaConnection, SchemaUtil.getTableDisplayName());
Then, from the returned PTable, call ptable.getPhysicalName().getBytes() to get
the byte[] name of the local index table so you can add the missing property.
- for the other change, delaying the creation of the local index, I think if
you just remove the !isMultiTenant() check in the if on line 1200 of
ConnectionQueryServicesImpl, then the local index will be created on demand for
multi tenant tables. You can just move that if check to inside the else block
surrounding the ensureViewIndexTableCreated call instead:
{code}
@Override
public MetaDataMutationResult createTable(final List<Mutation>
tableMetaData, byte[] physicalTableName, PTableType tableType,
Map<String,Object> tableProps, final
List<Pair<byte[],Map<String,Object>>> families, byte[][] splits) throws
SQLException {
byte[][] rowKeyMetadata = new byte[3][];
Mutation m = MetaDataUtil.getPutOnlyTableHeaderRow(tableMetaData);
byte[] key = m.getRow();
SchemaUtil.getVarChars(key, rowKeyMetadata);
byte[] tenantIdBytes =
rowKeyMetadata[PhoenixDatabaseMetaData.TENANT_ID_INDEX];
byte[] schemaBytes =
rowKeyMetadata[PhoenixDatabaseMetaData.SCHEMA_NAME_INDEX];
byte[] tableBytes =
rowKeyMetadata[PhoenixDatabaseMetaData.TABLE_NAME_INDEX];
byte[] tableName = physicalTableName != null ? physicalTableName :
SchemaUtil.getTableNameAsBytes(schemaBytes, tableBytes);
boolean localIndexTable =
Boolean.TRUE.equals(tableProps.remove(MetaDataUtil.IS_LOCAL_INDEX_TABLE_PROP_NAME));
if ((tableType == PTableType.VIEW && physicalTableName != null) ||
(tableType != PTableType.VIEW && physicalTableName == null)) {
// For views this will ensure that metadata already exists
// For tables and indexes, this will create the metadata if it
doesn't already exist
ensureTableCreated(tableName, tableType, tableProps, families,
splits, true);
}
ImmutableBytesWritable ptr = new ImmutableBytesWritable();
if (tableType == PTableType.INDEX) { // Index on view
// Physical index table created up front for multi tenant
// TODO: if viewIndexId is Short.MIN_VALUE, then we don't need to
attempt to create it
if (physicalTableName != null && !MetaDataUtil.isMultiTenant(m,
kvBuilder, ptr)) {
if (localIndexTable) {
ensureLocalIndexTableCreated(tableName, tableProps,
families, splits, MetaDataUtil.getClientTimeStamp(m));
} else {
ensureViewIndexTableCreated(tenantIdBytes.length == 0 ?
null : PNameFactory.newName(tenantIdBytes), physicalTableName,
MetaDataUtil.getClientTimeStamp(m));
}
}
} else if (tableType == PTableType.TABLE &&
MetaDataUtil.isMultiTenant(m, kvBuilder, ptr)) { // Create view index table up
front for multi tenant tables
{code}
> LocalIndexSplitter prevents region from auto split
> --------------------------------------------------
>
> Key: PHOENIX-1634
> URL: https://issues.apache.org/jira/browse/PHOENIX-1634
> Project: Phoenix
> Issue Type: Bug
> Affects Versions: 5.0.0, 4.3
> Reporter: Mujtaba Chohan
> Assignee: Rajeshbabu Chintaguntla
> Attachments: PHOENIX-1634.patch, logs.zip, performance.py
>
>
> Local index is *not* created for a multi-tenant table however empty HBase
> table is created in advance for local index. With data upserted in the
> multi-tenant table, after multiple successive auto-splits when region tries
> to split again on another region server, LocalIndexSplitter prevents
> auto-split from happening. [~rajesh23] Please see the log below. Thanks
> [~apurtell] and [~jamestaylor] for narrowing down this issue.
> {code}
> WARN org.apache.hadoop.hbase.regionserver.LocalIndexSplitter: Index region
> corresponindg to data region
> MYSCHEMA.MY_MULTITENANT_TABLE,,1422663910075.db3861e02b58e21b5383704375539ee5.
> not in the same server. So skipping the split.
> 2015-01-31 04:48:53,532 INFO
> org.apache.hadoop.hbase.regionserver.SplitRequest: Running rollback/cleanup
> of failed split of
> MYSCHEMA.MY_MULTITENANT_TABLE,,1422663910075.db3861e02b58e21b5383704375539ee5.;
> Coprocessor bypassing region
> MYSCHEMA.MY_MULTITENANT_TABLE,,1422663910075.db3861e02b58e21b5383704375539ee5.
> split.
> java.io.IOException: Coprocessor bypassing region
> MYSCHEMA.MY_MULTITENANT_TABLE,,1422663910075.db3861e02b58e21b5383704375539ee5.
> split.
> at
> org.apache.hadoop.hbase.regionserver.SplitTransaction.createDaughters(SplitTransaction.java:309)
> at
> org.apache.hadoop.hbase.regionserver.SplitTransaction.execute(SplitTransaction.java:655)
> at org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:84)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
