[
https://issues.apache.org/jira/browse/PHOENIX-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294592#comment-15294592
]
ASF GitHub Bot commented on PHOENIX-1734:
-----------------------------------------
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/168#discussion_r64124199
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/util/UpgradeUtil.java ---
@@ -292,6 +294,119 @@ private static void
preSplitSequenceTable(PhoenixConnection conn, int nSaltBucke
}
}
}
+
+ public static void upgradeLocalIndexes(PhoenixConnection
metaConnection) throws SQLException,
+ IOException, org.apache.hadoop.hbase.TableNotFoundException {
+ HBaseAdmin admin = null;
+ try {
+ admin = metaConnection.getQueryServices().getAdmin();
+ ResultSet rs =
metaConnection.createStatement().executeQuery("SELECT TABLE_SCHEM, TABLE_NAME,
DATA_TABLE_NAME FROM SYSTEM.CATALOG "
+ + " WHERE COLUMN_NAME IS NULL"
+ + " AND COLUMN_FAMILY IS NULL"
+ + " AND INDEX_TYPE=2");
+ boolean droppedLocalIndexes = false;
+ while (rs.next()) {
+ if(!droppedLocalIndexes) {
+ HTableDescriptor[] localIndexTables =
admin.listTables(MetaDataUtil.LOCAL_INDEX_TABLE_PREFIX+".*");
+ String localIndexSplitter =
LocalIndexSplitter.class.getName();
+ for (HTableDescriptor table : localIndexTables) {
+ HTableDescriptor dataTableDesc =
admin.getTableDescriptor(TableName.valueOf(MetaDataUtil.getUserTableName(table.getNameAsString())));
+ HColumnDescriptor[] columnFamilies =
dataTableDesc.getColumnFamilies();
+ boolean modifyTable = false;
+ for(HColumnDescriptor cf : columnFamilies) {
+ String localIndexCf =
QueryConstants.LOCAL_INDEX_COLUMN_FAMILY_PREFIX+cf.getNameAsString();
+
if(dataTableDesc.getFamily(Bytes.toBytes(localIndexCf))==null){
+ HColumnDescriptor colDef =
+ new
HColumnDescriptor(localIndexCf);
+ for(Entry<ImmutableBytesWritable,
ImmutableBytesWritable>keyValue: cf.getValues().entrySet()){
+
colDef.setValue(keyValue.getKey().copyBytes(), keyValue.getValue().copyBytes());
+ }
+ dataTableDesc.addFamily(colDef);
+ modifyTable = true;
+ }
+ }
+ List<String> coprocessors =
dataTableDesc.getCoprocessors();
+ for(String coprocessor: coprocessors) {
+ if(coprocessor.equals(localIndexSplitter)) {
+
dataTableDesc.removeCoprocessor(localIndexSplitter);
+ modifyTable = true;
+ }
+ }
+ if(modifyTable) {
+ admin.modifyTable(dataTableDesc.getName(),
dataTableDesc);
+ }
+ }
+
admin.disableTables(MetaDataUtil.LOCAL_INDEX_TABLE_PREFIX+".*");
+
admin.deleteTables(MetaDataUtil.LOCAL_INDEX_TABLE_PREFIX+".*");
+ droppedLocalIndexes = true;
+ }
+ String getColumns =
+ "SELECT COLUMN_NAME, COLUMN_FAMILY FROM
SYSTEM.CATALOG WHERE TABLE_SCHEM "
+ + (rs.getString(1) == null ? "IS NULL " :
"='" + rs.getString(1)
+ + "'") + " and TABLE_NAME='" +
rs.getString(2)
+ + "' AND COLUMN_NAME IS NOT NULL";
+ ResultSet getColumnsRs =
metaConnection.createStatement().executeQuery(getColumns);
+ List<String> indexedColumns = new ArrayList<String>(1);
+ List<String> coveredColumns = new ArrayList<String>(1);
+
+ while (getColumnsRs.next()) {
+ String column = getColumnsRs.getString(1);
+ String columnName =
IndexUtil.getDataColumnName(column);
+ if
(columnName.equals(MetaDataUtil.getViewIndexIdColumnName())) {
+ continue;
+ }
+ String columnFamily =
IndexUtil.getDataColumnFamilyName(column);
+ if (getColumnsRs.getString(2) == null) {
+ if (columnFamily != null &&
!columnFamily.isEmpty()) {
+ if
(columnFamily.equals(QueryConstants.DEFAULT_COLUMN_FAMILY)) {
+ indexedColumns.add(columnName);
+ } else {
+
indexedColumns.add(SchemaUtil.getColumnName(columnFamily,
+ columnName));
+ }
+ }
+ } else {
+
coveredColumns.add(SchemaUtil.getColumnName(columnFamily, columnName));
+ }
+ }
+ StringBuilder createIndex = new StringBuilder("CREATE
LOCAL INDEX ");
+ createIndex.append(rs.getString(2));
+ createIndex.append(" ON ");
+
createIndex.append(SchemaUtil.getTableName(rs.getString(1), rs.getString(3)));
+ createIndex.append("(");
+ for (int i = 0; i < indexedColumns.size(); i++) {
+ createIndex.append(indexedColumns.get(i));
+ if (i < indexedColumns.size() - 1) {
+ createIndex.append(",");
+ }
+ }
+ createIndex.append(")");
+
+ if (!coveredColumns.isEmpty()) {
+ createIndex.append(" INCLUDE(");
+ for (int i = 0; i < coveredColumns.size(); i++) {
+ createIndex.append(coveredColumns.get(i));
+ if (i < coveredColumns.size() - 1) {
+ createIndex.append(",");
+ }
+ }
+ createIndex.append(")");
--- End diff --
Can we just add the ASYNC keyword here so that the rebuilds are
asynchronous, can be started when the user is ready, and will run through MR?
> Local index improvements
> ------------------------
>
> Key: PHOENIX-1734
> URL: https://issues.apache.org/jira/browse/PHOENIX-1734
> Project: Phoenix
> Issue Type: Improvement
> Reporter: Rajeshbabu Chintaguntla
> Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENI-1734-WIP.patch, PHOENIX-1734_v1.patch,
> PHOENIX-1734_v4.patch, PHOENIX-1734_v5.patch, TestAtomicLocalIndex.java
>
>
> Local index design considerations:
> 1. Colocation: We need to co-locate regions of local index regions and data
> regions. The co-location can be a hard guarantee or a soft (best approach)
> guarantee. The co-location is a performance requirement, and also maybe
> needed for consistency(2). Hard co-location means that either both the data
> region and index region are opened atomically, or neither of them open for
> serving.
> 2. Index consistency : Ideally we want the index region and data region to
> have atomic updates. This means that they should either (a)use transactions,
> or they should (b)share the same WALEdit and also MVCC for visibility. (b) is
> only applicable if there is hard colocation guarantee.
> 3. Local index clients : How the local index will be accessed from clients.
> In case of the local index being managed in a table, the HBase client can be
> used for doing scans, etc. If the local index is hidden inside the data
> regions, there has to be a different mechanism to access the data through the
> data region.
> With the above considerations, we imagine three possible implementation for
> the local index solution, each detailed below.
> APPROACH 1: Current approach
> (1) Current approach uses balancer as a soft guarantee. Because of this, in
> some rare cases, colocation might not happen.
> (2) The index and data regions do not share the same WALEdits. Meaning
> consistency cannot be achieved. Also there are two WAL writes per write from
> client.
> (3) Regular Hbase client can be used to access index data since index is just
> another table.
> APPROACH 2: Shadow regions + shared WAL & MVCC
> (1) Introduce a shadow regions concept in HBase. Shadow regions are not
> assigned by AM. Phoenix implements atomic open (and split/merge) of region
> opening for data regions and index regions so that hard co-location is
> guaranteed.
> (2) For consistency requirements, the index regions and data regions will
> share the same WALEdit (and thus recovery) and they will also share the same
> MVCC mechanics so that index update and data update is visible atomically.
> (3) Regular Hbase client can be used to access index data since index is just
> another table.
> APPROACH 3: Storing index data in separate column families in the table.
> (1) Regions will have store files for cfs, which is sorted using the primary
> sort order. Regions may also maintain stores, sorted in secondary sort
> orders. This approach is similar in vein how a RDBMS keeps data (a B-TREE in
> primary sort order and multiple B-TREEs in secondary sort orders with
> pointers to primary key). That means store the index data in separate column
> families in the data region. This way a region is extended to be more similar
> to a RDBMS (but LSM instead of BTree). This is sometimes called shadow cf’s
> as well. This approach guarantees hard co-location.
> (2) Since everything is in a single region, they automatically share the
> same WALEdit and MVCC numbers. Atomicity is easily achieved.
> (3) Current Phoenix implementation need to change in such a way that column
> families selection in read/write path is based data table/index table(logical
> table in phoenix).
> I think that APPROACH 3 is the best one for long term, since it does not
> require to change anything in HBase, mainly we don't need to muck around with
> the split/merge stuff in HBase. It will be win-win.
> However, APPROACH 2 still needs a “shadow regions” concept to be implemented
> in HBase itself, and also a way to share WALEdits and MVCCs from multiple
> regions.
> APPROACH 1 is a good start for local indexes, but I think we are not getting
> the full benefits for the feature. We can support this for the short term,
> and decide on the next steps for a longer term implementation.
> we won't be able to get to implementing it immediately, and want to start a
> brainstorm.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
