[
https://issues.apache.org/jira/browse/HIVE-21344?focusedWorklogId=326610&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-326610
]
ASF GitHub Bot logged work on HIVE-21344:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 10/Oct/19 22:32
Start Date: 10/Oct/19 22:32
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #749: HIVE-21344
URL: https://github.com/apache/hive/pull/749#discussion_r333763093
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
##########
@@ -1612,6 +1612,150 @@ public Table
apply(org.apache.hadoop.hive.metastore.api.Table table) {
}
}
+ /**
+ * Get the materialized views that have been enabled for rewriting from the
+ * cache (registry). It will preprocess them to discard those that are
+ * outdated and augment those that need to be augmented, e.g., if incremental
+ * rewriting is enabled.
+ *
+ * @return the list of materialized views available for rewriting from the
registry
+ * @throws HiveException
+ */
+ public List<RelOptMaterialization>
getPreprocessedMaterializedViewsFromRegistry(
+ List<String> tablesUsed, HiveTxnManager txnMgr) throws HiveException {
+ // From cache
+ List<RelOptMaterialization> materializedViews =
+ HiveMaterializedViewsRegistry.get().getRewritingMaterializedViews();
+ if (materializedViews.isEmpty()) {
+ // Bail out: empty list
+ return new ArrayList<>();
+ }
+ // Add to final result
+ return filterAugmentMaterializedViews(materializedViews, tablesUsed,
txnMgr);
+ }
+
+ private List<RelOptMaterialization>
filterAugmentMaterializedViews(List<RelOptMaterialization> materializedViews,
+ List<String> tablesUsed, HiveTxnManager txnMgr) throws HiveException {
+ final String validTxnsList = conf.get(ValidTxnList.VALID_TXNS_KEY);
+ final ValidTxnWriteIdList currentTxnWriteIds =
txnMgr.getValidWriteIds(tablesUsed, validTxnsList);
+ final boolean tryIncrementalRewriting =
+ HiveConf.getBoolVar(conf,
HiveConf.ConfVars.HIVE_MATERIALIZED_VIEW_REWRITING_INCREMENTAL);
+ final long defaultTimeWindow =
+ HiveConf.getTimeVar(conf,
HiveConf.ConfVars.HIVE_MATERIALIZED_VIEW_REWRITING_TIME_WINDOW,
+ TimeUnit.MILLISECONDS);
+ try {
+ // Final result
+ List<RelOptMaterialization> result = new ArrayList<>();
+ for (RelOptMaterialization materialization : materializedViews) {
+ final RelNode viewScan = materialization.tableRel;
+ final Table materializedViewTable;
+ if (viewScan instanceof Project) {
+ // There is a Project on top (due to nullability)
+ materializedViewTable = ((RelOptHiveTable)
viewScan.getInput(0).getTable()).getHiveTableMD();
+ } else {
+ materializedViewTable = ((RelOptHiveTable)
viewScan.getTable()).getHiveTableMD();
+ }
+ final Boolean outdated =
isOutdatedMaterializedView(materializedViewTable, currentTxnWriteIds,
+ defaultTimeWindow, tablesUsed, false);
+ if (outdated == null) {
+ continue;
+ }
+
+ final CreationMetadata creationMetadata =
materializedViewTable.getCreationMetadata();
+ if (outdated) {
+ // The MV is outdated, see whether we should consider it for
rewriting or not
+ if (!tryIncrementalRewriting) {
+ LOG.debug("Materialized view " +
materializedViewTable.getFullyQualifiedName() +
+ " ignored for rewriting as its contents are outdated");
+ continue;
+ }
+ // We will rewrite it to include the filters on transaction list
+ // so we can produce partial rewritings.
+ // This would be costly since we are doing it for every materialized
view
+ // that is outdated, but it only happens for more than one
materialized view
+ // if rewriting with outdated materialized views is enabled
(currently
+ // disabled by default).
+ materialization = augmentMaterializationWithTimeInformation(
+ materialization, validTxnsList, new ValidTxnWriteIdList(
+ creationMetadata.getValidTxnList()));
+ }
+ result.add(materialization);
+ }
+ return result;
+ } catch (Exception e) {
+ throw new HiveException(e);
+ }
+ }
+
+ /**
+ * Validate that the materialized views retrieved from registry are still
up-to-date.
+ * For those that are not, the method loads them from the metastore into the
registry.
+ *
+ * @return true if they are up-to-date, otherwise false
+ * @throws HiveException
+ */
+ public boolean validateMaterializedViewsFromRegistry(List<Table>
cachedMaterializedViewTables,
+ List<String> tablesUsed, HiveTxnManager txnMgr) throws HiveException {
+ final long defaultTimeWindow =
+ HiveConf.getTimeVar(conf,
HiveConf.ConfVars.HIVE_MATERIALIZED_VIEW_REWRITING_TIME_WINDOW,
+ TimeUnit.MILLISECONDS);
+ final String validTxnsList = conf.get(ValidTxnList.VALID_TXNS_KEY);
+ final ValidTxnWriteIdList currentTxnWriteIds =
txnMgr.getValidWriteIds(tablesUsed, validTxnsList);
+ try {
+ // Final result
+ boolean result = true;
+ for (Table cachedMaterializedViewTable : cachedMaterializedViewTables) {
+ // Retrieve the materialized view table from the metastore
+ final Table materializedViewTable = getTable(
+ cachedMaterializedViewTable.getDbName(),
cachedMaterializedViewTable.getTableName());
+ if (materializedViewTable == null ||
!materializedViewTable.isRewriteEnabled()) {
+ // This could happen if materialized view has been deleted or
rewriting has been disabled.
+ // We remove it from the registry and set result to false.
+
HiveMaterializedViewsRegistry.get().dropMaterializedView(cachedMaterializedViewTable);
+ result = false;
Review comment:
We are not done. We do try to validate all materialized views that we were
going to use and we reload from metastore those that are not up-to-date, hence
we need to iterate through all that were introduced by the rewriting. However,
if any of them is not up-to-date, we return `false` to cancel the rewriting.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 326610)
Time Spent: 0.5h (was: 20m)
> CBO: Reduce compilation time in presence of materialized views
> --------------------------------------------------------------
>
> Key: HIVE-21344
> URL: https://issues.apache.org/jira/browse/HIVE-21344
> Project: Hive
> Issue Type: Bug
> Components: Materialized views
> Affects Versions: 4.0.0
> Reporter: Gopal Vijayaraghavan
> Assignee: Jesus Camacho Rodriguez
> Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21344.01.patch, HIVE-21344.02.patch,
> HIVE-21344.03.patch, HIVE-21344.04.patch, HIVE-21344.patch,
> calcite-planner-after-fix.svg.zip, mv-get-from-remote.png
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> For every query, {{getAllValidMaterializedViews}} still requires a call to
> metastore to verify that the materializations exist, whether they are
> outdated or not, etc. Since this is only useful for active-active HS2
> deployments, we could take a less aggressive approach and check this
> information only after rewriting has been triggered. In addition, we could
> refresh the information in the HS2 registry periodically in a background
> thread.
> {code}
> // This is not a rebuild, we retrieve all the materializations. In turn, we
> do not need
> // to force the materialization contents to be up-to-date, as this is not a
> rebuild, and
> // we apply the user parameters
> (HIVE_MATERIALIZED_VIEW_REWRITING_TIME_WINDOW) instead.
> materializations = db.getAllValidMaterializedViews(getTablesUsed(basePlan),
> false, getTxnMgr());
> {code}
> !mv-get-from-remote.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
