[
https://issues.apache.org/jira/browse/HIVE-18747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16420187#comment-16420187
]
Sankar Hariappan commented on HIVE-18747:
-----------------------------------------
Thanks for the review [~ekoifman]!
I'll submit another patch with fixes for check style issues.
TxnHandler.getValidWriteIdsForTable() is changed to correct the logic to get
writeIdHwm.
In base code, we always depend on TXN_TO_WRITE_ID to get it using "writeIdHwm=
max(txn_to_write_id.t2w_write_id) where t2i_txn_id <= txnHwm". But if entries
are cleaned-up, it may not work as we may empty TXN_TO_WRITE_ID table which may
result in 0 for writeIdHwm.
So, in this patch, modified this logic as writeIdHwm =
min(next_write_id.nwi_next, max(txn_to_write_id.twi_write_id where t2i_txn_id
<= txnHwm)). This works for all cases of clean-up.
> Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL.
> ------------------------------------------------------------------
>
> Key: HIVE-18747
> URL: https://issues.apache.org/jira/browse/HIVE-18747
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
> Affects Versions: 3.0.0
> Reporter: Sankar Hariappan
> Assignee: Sankar Hariappan
> Priority: Minor
> Labels: ACID, pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18747.01.patch, HIVE-18747.02.patch,
> HIVE-18747.03.patch
>
>
> Per table write ID implementation (HIVE-18192) maintains a map between txn ID
> and table write ID in TXN_TO_WRITE_ID meta table.
> The entries in this table is used to generate ValidWriteIdList for the given
> ValidTxnList to ensure snapshot isolation.
> When table or database is dropped, then these entries are cleaned-up. But, it
> is necessary to clean-up for active tables too for better performance.
> TXN_TO_WRITE_ID table keeps a mapping of Transaction ID to Write ID. The
> state of each Write ID (open, committed, aborted) is determined by the state
> of the parent transaction. In order to be able to get a WriteIdList that is
> accurate wrt ValidTxnList that is locked in at the start of the transaction,
> we have to retain txnid<->writeid mapping even after the transaction ends.
> This is because a reader at Snapshot Isolation that started when transaction
> X was open, should continue to ignore the data written by X even after X
> commits.
> So we need a mechanism to know when it is safe to remove TXN_TO_WRITE_ID.
> There are 2 parts to it. When txn X is opened, it records Y=select
> min(txn_id) from TXNS where txn_state=’o’ in MIN_HISTORY(txnid,opentxnid)
> table, i.e. it adds (X, Y) to MIN_HISTORY. On commit (and abort) of X, it
> removes its own entry from MIN_HISTORY. In the absence of Aborted
> transactions, MIN_HISTORY gives us the smallest open txnid across all active
> reader snapshots. Let Z=select min(opentxnid) from MIN_HISTORY. We can
> delete entries from TXN_TO_WRITE_ID once TXN_TO_WRITE_ID.T2W_TXNID < Z since
> every active reader sees txns < Z as committed.
> If S is aborted txns, we retain the metadata about it in TXNS as long as any
> data written S may be visible to some reader in the system so that the reader
> knows to skip this data. The rules for when that is are complex but wrt to
> TXN_TO_WRITE_ID, if A=select min(TXN_ID) from TXNS where TXN_STATE=’a’, then
> it’s safe to delete from TXN_TO_WRITE_ID when TXN_TO_WRITE_ID.T2W_TXNID <
> min(Z,A).
> If no open or aborted txns exist in the system, then we need to enable
> cleanup using latest allocated value of NEXT_TXN_ID table. Delete condition
> would be TXN_TO_WRITE_ID.T2W_TXNID < min(Z,A,NEXT_TXN_ID.ntxn_next).
> Also, it is proposed to trigger cleanup on TXN_TO_WRITE_ID from initiator
> immediately after cleaning up aborted txns metadata from TXNS table.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
