Hi David, You do not really need to run compaction every time. Is it possible to wait for the compaction to start automatically next time?
Thanks, Peter > On Jun 2, 2020, at 12:51, David Morin <morin.david....@gmail.com> wrote: > > Thanks Peter, > > Any workaround on HDP 2.6.x with Hive 2 ? > Otherwise, the only way is to reduce time it takes for this "merge" queries > in order to cancel locks and related transactions. Am I right ? > > Le mar. 2 juin 2020 à 11:52, Peter Vary <pv...@cloudera.com > <mailto:pv...@cloudera.com>> a écrit : > Hi David, > > I think this jira describes your situation: > https://issues.apache.org/jira/browse/HIVE-16360 > <https://issues.apache.org/jira/browse/HIVE-16360> > > "The reason is that compactor won't compact anything above the level of an > open transaction. So if there is a very long running txn, it may filter out > all deltas in a partition. This looks very confusing when looking at the > logs." > > Thanks, > Peter > >> On Jun 2, 2020, at 11:44, David Morin <morin.david....@gmail.com >> <mailto:morin.david....@gmail.com>> wrote: >> >> I don't get it. >> The transaction id in the error message "No delta files or original files >> found to compact in hdfs://... with min Open txnid:6704143" is related to a >> current transaction (transaction + lock) for another user, another database >> and table. But still it blocks the compaction for the current database/table >> >> On 2020/06/01 20:13:08, David Morin <morin.david....@gmail.com >> <mailto:morin.david....@gmail.com>> wrote: >>> Hi, >>> >>> I have a compaction issue on my cluster. When I force a compaction (major) >>> on one table I get this error in Metastore logs: >>> >>> 2020-06-01 19:49:35,512 ERROR [XXXX-78]: compactor.CompactorMR >>> (CompactorMR.java:run(264)) - No delta files or original files found to >>> compact in hdfs://...hive/warehouse/my.db/test <> for compactionId=57944651 >>> with min Open txnid:6704143. Compaction cannot compact above this txnid >>> >>> Any idea about the root cause and how to fix it ? >>> thx >>> >>> Current env: HDP 2.6.5 >>> Hive table in transactional mode (ORC) >>> >
