[
https://issues.apache.org/jira/browse/HIVE-18192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16329176#comment-16329176
]
Sankar Hariappan commented on HIVE-18192:
-----------------------------------------
Attached 02.patch with below changes.
{quote}1. Metastore tables for write id management. (Added only for derby now)
2. ValidWriteIdList generation. Used Read and Write entities to build the
tables list.
3. ACID write and read flows.
4. Compactor based on write id instead of txn id.
5. Working well for queries with single table. Join queries won’t work. 6.
6. Streaming ingest to use table write id.{quote}
Pending changes relevant to this Jira which will be available in subsequent
patches.
{quote}1. Support ValidWriteIdList config string with multiple tables. Need to
change the format to include table name and separator as
follows.\{<table_name>:hwm:minWriteId:open1,open2:abort1,abort2:}{<table_name>:hwm:…}
2. Pass appropriate ValidWriteIdList string to ORCInputFormat without need to
get table name there. (As per Gopal’s suggestion)
3. Store ValidTxnList in the TxnManager to be re-used for multi-statement txns.
Also, need to pass ValidTxnList while getting ValidWriteIdList instead of
getting open txns list in metastore.
4. Cleaner for TXNS_TO_WRITE_ID table entries. Also, need to maintain the LWM
for each table write id.
5. Update classes to use WriteID instead of TxnId as methods and variable names.
6. Correct test cases for new changes.
7. FetchTask optimisation to refer ValidTxnList seems to be not valid if query
involves multiple tables. Need to check it.
8. Metastore tables for write id management. Add for other databases too.
9. Remove entries from TXNS_TO_WRITE_ID table when drop a table/database.{quote}
Request [~ekoifman]/[~anishek] to review this patch and provide your feedback.
cc [~thejas]
> Introduce WriteID per table rather than using global transaction ID
> -------------------------------------------------------------------
>
> Key: HIVE-18192
> URL: https://issues.apache.org/jira/browse/HIVE-18192
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2, Transactions
> Affects Versions: 3.0.0
> Reporter: anishek
> Assignee: Sankar Hariappan
> Priority: Major
> Labels: ACID, DR, pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18192.01.patch, HIVE-18192.02.patch
>
>
> To support ACID replication, we will be introducing a per table write Id
> which will replace the transaction id in the primary key for each row in a
> ACID table.
> The current primary key is determined via
> <original transaction id, bucketid , rowid>
> which will move to
> <write id, bucketid, rowid>
> a persistable map of global txn id -> to table -> write id for that table has
> to be maintained to now allow Snapshot isolation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
