Hi David, Maybe this can help: https://docs.cloudera.com/HDPDocuments/HDP2/HDP-2.6.5/bk_data-access/content/understanding-administering-compactions.html <https://docs.cloudera.com/HDPDocuments/HDP2/HDP-2.6.5/bk_data-access/content/understanding-administering-compactions.html>
Also if everything else fails, you can still issue the ALTER TABLE command periodically using crontab. Running extra compaction will not hurt that much. Thanks, Peter > On Jun 2, 2020, at 14:25, David Morin <morin.david....@gmail.com> wrote: > > Yes Peter. We can wait a little. > We use streaming so we have to merge files regularly. > If we can reduce the duration for some big Merge queries and make auto > compaction works properly it should be ok. > Problem: at the moment, compactions are not triggered automatically. Have you > an idea ? > > Le mar. 2 juin 2020 à 12:57, Peter Vary <pv...@cloudera.com > <mailto:pv...@cloudera.com>> a écrit : > Hi David, > > You do not really need to run compaction every time. > Is it possible to wait for the compaction to start automatically next time? > > Thanks, > Peter > >> On Jun 2, 2020, at 12:51, David Morin <morin.david....@gmail.com >> <mailto:morin.david....@gmail.com>> wrote: >> >> Thanks Peter, >> >> Any workaround on HDP 2.6.x with Hive 2 ? >> Otherwise, the only way is to reduce time it takes for this "merge" queries >> in order to cancel locks and related transactions. Am I right ? >> >> Le mar. 2 juin 2020 à 11:52, Peter Vary <pv...@cloudera.com >> <mailto:pv...@cloudera.com>> a écrit : >> Hi David, >> >> I think this jira describes your situation: >> https://issues.apache.org/jira/browse/HIVE-16360 >> <https://issues.apache.org/jira/browse/HIVE-16360> >> >> "The reason is that compactor won't compact anything above the level of an >> open transaction. So if there is a very long running txn, it may filter out >> all deltas in a partition. This looks very confusing when looking at the >> logs." >> >> Thanks, >> Peter >> >>> On Jun 2, 2020, at 11:44, David Morin <morin.david....@gmail.com >>> <mailto:morin.david....@gmail.com>> wrote: >>> >>> I don't get it. >>> The transaction id in the error message "No delta files or original files >>> found to compact in hdfs://... with min Open txnid:6704143" is related to a >>> current transaction (transaction + lock) for another user, another database >>> and table. But still it blocks the compaction for the current database/table >>> >>> On 2020/06/01 20:13:08, David Morin <morin.david....@gmail.com >>> <mailto:morin.david....@gmail.com>> wrote: >>>> Hi, >>>> >>>> I have a compaction issue on my cluster. When I force a compaction (major) >>>> on one table I get this error in Metastore logs: >>>> >>>> 2020-06-01 19:49:35,512 ERROR [XXXX-78]: compactor.CompactorMR >>>> (CompactorMR.java:run(264)) - No delta files or original files found to >>>> compact in hdfs://...hive/warehouse/my.db/test <> for >>>> compactionId=57944651 with min Open txnid:6704143. Compaction cannot >>>> compact above this txnid >>>> >>>> Any idea about the root cause and how to fix it ? >>>> thx >>>> >>>> Current env: HDP 2.6.5 >>>> Hive table in transactional mode (ORC) >>>> >> >
