[
https://issues.apache.org/jira/browse/HIVE-25656?focusedWorklogId=681501&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-681501
]
ASF GitHub Bot logged work on HIVE-25656:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 15/Nov/21 13:47
Start Date: 15/Nov/21 13:47
Worklog Time Spent: 10m
Work Description: kasakrisz commented on a change in pull request #2756:
URL: https://github.com/apache/hive/pull/2756#discussion_r749340621
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/stats/BasicStatsTask.java
##########
@@ -235,6 +223,46 @@ private void updateStats(StatsAggregator statsAggregator,
Map<String, String> pa
}
+ private static class TransactionalStatsProcessor {
+ private final HiveTxnManager txnManager;
+ private final Partish partish;
+
+ private TransactionalStatsProcessor(HiveTxnManager txnManager, Partish
partish) {
+ this.txnManager = txnManager;
+ this.partish = partish;
+ }
+
+ private long toLong(String value) {
+ if (value == null || value.isEmpty()) {
+ return 0;
+ }
+
+ return Long.parseLong(value);
+ }
+
+ public void process(StatsAggregator statsAggregator) throws HiveException,
MetaException {
+ if (statsAggregator == null) {
+ return;
+ }
+
+ if (partish.isTransactionalTable()) {
+ String prefix = getAggregationPrefix(partish.getTable(),
partish.getPartition());
+ long insertCount = toLong(statsAggregator.aggregateStats(prefix,
INSERT_COUNT));
+ long updateCount = toLong(statsAggregator.aggregateStats(prefix,
UPDATE_COUNT));
+ long deleteCount = toLong(statsAggregator.aggregateStats(prefix,
DELETE_COUNT));
+
+ if (insertCount > 0 || updateCount > 0 || deleteCount > 0) {
+ AffectedRowCount affectedRowCount = new AffectedRowCount();
+ affectedRowCount.setTableId(partish.getTable().getTTable().getId());
+ affectedRowCount.setInsertCount(insertCount);
+ affectedRowCount.setUpdatedCount(updateCount);
+ affectedRowCount.setDeletedCount(deleteCount);
+
+ txnManager.addAffectedRowCount(affectedRowCount);
Review comment:
I changed this part to save the stats from `BasicStatsTask.java` by
introducing a new thrift api method:
```
void update_transaction_statistics(1:UpdateTransactionalStatsRequest req)
```
However instead of passing all the delta in one call to HMS and updating the
stats in the backend db by one call per table it will cause several HMS and
backend DB calls. Example: a merge statement performs 3+3 calls.
Maybe we should optimize this later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 681501)
Time Spent: 1.5h (was: 1h 20m)
> Get materialized view state based on number of affected rows of transactions
> ----------------------------------------------------------------------------
>
> Key: HIVE-25656
> URL: https://issues.apache.org/jira/browse/HIVE-25656
> Project: Hive
> Issue Type: Improvement
> Components: Materialized views, Transactions
> Reporter: Krisztian Kasa
> Assignee: Krisztian Kasa
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> To enable the faster incremental rebuild of materialized views presence of
> update/delete operations on the source tables of the view since the last
> rebuild must be checked. Based on the outcome different plan is generated for
> scenarios in presence of update/delete and insert only operations.
> Currently this is done by querying the COMPLETED_TXN_COMPONENTS table however
> the records from this table is cleaned when MV source tables are compacted.
> This reduces the chances of incremental MV rebuild.
> The goal of this patch is to find an alternative way to store and retrieve
> this information.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
