Yes Peter. We can wait a little. We use streaming so we have to merge files regularly. If we can reduce the duration for some big Merge queries and make auto compaction works properly it should be ok. Problem: at the moment, compactions are not triggered automatically. Have you an idea ?
Le mar. 2 juin 2020 à 12:57, Peter Vary <pv...@cloudera.com> a écrit : > Hi David, > > You do not really need to run compaction every time. > Is it possible to wait for the compaction to start automatically next time? > > Thanks, > Peter > > On Jun 2, 2020, at 12:51, David Morin <morin.david....@gmail.com> wrote: > > Thanks Peter, > > Any workaround on HDP 2.6.x with Hive 2 ? > Otherwise, the only way is to reduce time it takes for this "merge" > queries in order to cancel locks and related transactions. Am I right ? > > Le mar. 2 juin 2020 à 11:52, Peter Vary <pv...@cloudera.com> a écrit : > >> Hi David, >> >> I think this jira describes your situation: >> https://issues.apache.org/jira/browse/HIVE-16360 >> >> "The reason is that compactor won't compact anything above the level of >> an open transaction. So if there is a very long running txn, it may filter >> out all deltas in a partition. This looks very confusing when looking at >> the logs." >> >> Thanks, >> Peter >> >> On Jun 2, 2020, at 11:44, David Morin <morin.david....@gmail.com> wrote: >> >> I don't get it. >> The transaction id in the error message "No delta files or original files >> found to compact in hdfs://... with min Open txnid:6704143" is related to a >> current transaction (transaction + lock) for another user, another database >> and table. But still it blocks the compaction for the current database/table >> >> On 2020/06/01 20:13:08, David Morin <morin.david....@gmail.com> wrote: >> >> Hi, >> >> I have a compaction issue on my cluster. When I force a compaction >> (major) on one table I get this error in Metastore logs: >> >> 2020-06-01 19:49:35,512 ERROR [XXXX-78]: compactor.CompactorMR >> (CompactorMR.java:run(264)) - No delta files or original files found to >> compact in hdfs://...hive/warehouse/my.db/test for compactionId=57944651 >> with min Open txnid:6704143. Compaction cannot compact above this txnid >> >> Any idea about the root cause and how to fix it ? >> thx >> >> Current env: HDP 2.6.5 >> Hive table in transactional mode (ORC) >> >> >> >
