[
https://issues.apache.org/jira/browse/HIVE-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alan Gates updated HIVE-8966:
-----------------------------
Attachment: HIVE-8966.4.patch
This patch takes a new approach. Rather than changing AcidUtils.getAcidState
(as previous 2 attempts) this patch gives a new implementation of ValidTxnList
that only returns isTxnRangeValid ALL or NONE, and gives NONE if there are any
open transactions <= the max transaction in the range (even if it's below the
range). This new implementation is used only by the compactor so that it's
understanding of what files it should compact are different than what files a
reader views as available for reading.
I've also added tests to TestCompactor to test compaction during streaming and
compaction after a streamer has aborted and died without cleaning up.
> Delta files created by hive hcatalog streaming cannot be compacted
> ------------------------------------------------------------------
>
> Key: HIVE-8966
> URL: https://issues.apache.org/jira/browse/HIVE-8966
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
> Affects Versions: 0.14.0
> Environment: hive
> Reporter: Jihong Liu
> Assignee: Alan Gates
> Priority: Critical
> Fix For: 0.14.1
>
> Attachments: HIVE-8966.2.patch, HIVE-8966.3.patch, HIVE-8966.4.patch,
> HIVE-8966.patch
>
>
> hive hcatalog streaming will also create a file like bucket_n_flush_length in
> each delta directory. Where "n" is the bucket number. But the
> compactor.CompactorMR think this file also needs to compact. However this
> file of course cannot be compacted, so compactor.CompactorMR will not
> continue to do the compaction.
> Did a test, after removed the bucket_n_flush_length file, then the "alter
> table partition compact" finished successfully. If don't delete that file,
> nothing will be compacted.
> This is probably a very severity bug. Both 0.13 and 0.14 have this issue
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
