[
https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17107128#comment-17107128
]
Peter Vary commented on HIVE-21052:
-----------------------------------
[~asomani]: You are right, that this could still be a problem.
The cleaner task removes every aborted TXN without components attached to it.
See:
https://github.com/apache/hive/blob/ba1e5f057eca9fc61eb1283fb1873593c70c1685/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java#L577-L580
With non-dynamic partitioning this is not a problem (where the components are
registered), since the compactor should remove the files before removing the
components from the table. With dynamic partitioning, if the components are not
registered but the files are there, then nothing guarantees that the partitions
are compacted before the aborted transaction is removed, and the next query
could read the delta directories.
This might have been a problem before direct inserts if the MoveTask moved the
files before registering the partitions, but definitely will be a problem with
direct inserts. CC: [~kuczoram].
[~asomani]: Do you plan to take over this jira? I or [~dkuzmenko] can help with
the review.
Thanks,
Peter
> Make sure transactions get cleaned if they are aborted before addPartitions
> is called
> -------------------------------------------------------------------------------------
>
> Key: HIVE-21052
> URL: https://issues.apache.org/jira/browse/HIVE-21052
> Project: Hive
> Issue Type: Bug
> Components: Transactions
> Affects Versions: 3.0.0, 3.1.1
> Reporter: Jaume M
> Assignee: Jaume M
> Priority: Critical
> Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch,
> HIVE-21052.10.patch, HIVE-21052.11.patch, HIVE-21052.12.patch,
> HIVE-21052.2.patch, HIVE-21052.3.patch, HIVE-21052.4.patch,
> HIVE-21052.5.patch, HIVE-21052.6.patch, HIVE-21052.7.patch,
> HIVE-21052.8.patch, HIVE-21052.9.patch
>
>
> If the transaction is aborted between openTxn and addPartitions and data has
> been written on the table the transaction manager will think it's an empty
> transaction and no cleaning will be done.
> This is currently an issue in the streaming API and in micromanaged tables.
> As proposed by [~ekoifman] this can be solved by:
> * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and
> when addPartitions is called remove this entry from TXN_COMPONENTS and add
> the corresponding partition entry to TXN_COMPONENTS.
> * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that
> specifies that a transaction was opened and it was aborted it must generate
> jobs for the worker for every possible partition available.
> cc [~ewohlstadter]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
