[
https://issues.apache.org/jira/browse/HIVE-12353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15111519#comment-15111519
]
Alan Gates commented on HIVE-12353:
-----------------------------------
bq. ResultSet.getLong() returns 0 when value was actually NULL so I thought it
would be more consistent with other places in the code that use 0 as invalid
txn id
Makes sense
bq. PreparedStatement cc_meta_info is a binary field - I don't think there is
way to set it w/o prepared statement. (Even though it isn't set yet but will be
soon)
Ok.
bq. A concern on the history stuff my thought here was that if I'm looking at
10 failed compactions, it would be useful for me to know when the last
successful run was. Allowing retention value to be set per type allows that.
Makes sense. It's unfortunate that users could easily get confused that there
were actually 50 successful compactions since the last failed one, but maybe we
can fix that in the future.
bq. checkFailedCompactions The idea in this loop is to look at the last
"failedThreshold" elements. If they are all "failed" - don't start another one.
If not all "failed" - that's the intervening success.
Ok, I missed that the loop only goes through "allowed failed transactions"
number of records, so you'll never reach that value unless they are all
transactions. Weird style, but it works.
bq. It also seems users could unwittingly mess things
bq. Yes, that is why there is
CompactionTxnHandler.getFailedCompactionRetention() to make sure this doesn't
happen
Ok, missed that.
I'm +1 on the patch.
> When Compactor fails it calls CompactionTxnHandler.markedCleaned(). it
> should not.
> -----------------------------------------------------------------------------------
>
> Key: HIVE-12353
> URL: https://issues.apache.org/jira/browse/HIVE-12353
> Project: Hive
> Issue Type: Bug
> Components: Transactions
> Affects Versions: 1.0.0
> Reporter: Eugene Koifman
> Assignee: Eugene Koifman
> Priority: Blocker
> Attachments: HIVE-12353.2.patch, HIVE-12353.3.patch,
> HIVE-12353.4.patch, HIVE-12353.6.patch, HIVE-12353.7.patch,
> HIVE-12353.8.patch, HIVE-12353.patch
>
>
> One of the things that this method does is delete entries from TXN_COMPONENTS
> for partition that it was trying to compact.
> This causes Aborted transactions in TXNS to become empty according to
> CompactionTxnHandler.cleanEmptyAbortedTxns() which means they can now be
> deleted.
> Once they are deleted, data that belongs to these txns is deemed committed...
> We should extend COMPACTION_QUEUE state with 'f' and 's' (failed, success)
> states. We should also not delete then entry from markedCleaned()
> We'll have separate process that cleans 'f' and 's' records after X minutes
> (or after > N records for a given partition exist).
> This allows SHOW COMPACTIONS to show some history info and how many times
> compaction failed on a given partition (subject to retention interval) so
> that we don't have to call markCleaned() on Compactor failures at the same
> time preventing Compactor to constantly getting stuck on the same bad
> partition/table.
> Ideally we'd want to include END_TIME field.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
