Re: compaction issue: Compaction cannot compact above this txnid

[David Morin]

Yes Peter, we're working on it. We try to make compaction work
automatically. With crontab otherwise.
Thanks for your help

David

Le mar. 2 juin 2020 à 14:48, Peter Vary  a écrit :

> Hi David,
>
> Maybe this can help:
> https://docs.cloudera.com/HDPDocuments/HDP2/HDP-2.6.5/bk_data-access/content/understanding-administering-compactions.html
>
> Also if everything else fails, you can still issue the ALTER TABLE command
> periodically using crontab. Running extra compaction will not hurt that
> much.
>
> Thanks,
> Peter
>
> On Jun 2, 2020, at 14:25, David Morin  wrote:
>
> Yes Peter. We can wait a little.
> We use streaming so we have to merge files regularly.
> If we can reduce the duration for some big Merge queries and make auto
> compaction works properly it should be ok.
> Problem: at the moment, compactions are not triggered automatically. Have
> you an idea ?
>
> Le mar. 2 juin 2020 à 12:57, Peter Vary  a écrit :
>
>> Hi David,
>>
>> You do not really need to run compaction every time.
>> Is it possible to wait for the compaction to start automatically next
>> time?
>>
>> Thanks,
>> Peter
>>
>> On Jun 2, 2020, at 12:51, David Morin  wrote:
>>
>> Thanks Peter,
>>
>> Any workaround on HDP 2.6.x with Hive 2 ?
>> Otherwise, the only way is to reduce time it takes for this "merge"
>> queries in order to cancel locks and related transactions. Am I right ?
>>
>> Le mar. 2 juin 2020 à 11:52, Peter Vary  a écrit :
>>
>>> Hi David,
>>>
>>> I think this jira describes your situation:
>>> https://issues.apache.org/jira/browse/HIVE-16360
>>>
>>> "The reason is that compactor won't compact anything above the level of
>>> an open transaction.  So if there is a very long running txn, it may filter
>>> out all deltas in a partition.  This looks very confusing when looking at
>>> the logs."
>>>
>>> Thanks,
>>> Peter
>>>
>>> On Jun 2, 2020, at 11:44, David Morin  wrote:
>>>
>>> I don't get it.
>>> The transaction id in the error message "No delta files or original
>>> files found to compact in hdfs://... with min Open txnid:6704143" is
>>> related to a current transaction (transaction + lock) for another user,
>>> another database and table. But still it blocks the compaction for the
>>> current database/table
>>>
>>> On 2020/06/01 20:13:08, David Morin  wrote:
>>>
>>> Hi,
>>>
>>> I have a compaction issue on my cluster. When I force a compaction
>>> (major) on one table I get this error in Metastore logs:
>>>
>>> 2020-06-01 19:49:35,512 ERROR [-78]: compactor.CompactorMR
>>> (CompactorMR.java:run(264)) - No delta files or original files found to
>>> compact in hdfs://...hive/warehouse/my.db/test for
>>> compactionId=57944651 with min Open txnid:6704143.  Compaction cannot
>>> compact above this txnid
>>>
>>> Any idea about the root cause and how to fix it ?
>>> thx
>>>
>>> Current env: HDP 2.6.5
>>> Hive table in transactional mode (ORC)
>>>
>>>
>>>
>>
>

Re: compaction issue: Compaction cannot compact above this txnid

[Peter Vary]

Hi David,

Maybe this can help: 
https://docs.cloudera.com/HDPDocuments/HDP2/HDP-2.6.5/bk_data-access/content/understanding-administering-compactions.html
 


Also if everything else fails, you can still issue the ALTER TABLE command 
periodically using crontab. Running extra compaction will not hurt that much.

Thanks,
Peter

> On Jun 2, 2020, at 14:25, David Morin  wrote:
> 
> Yes Peter. We can wait a little.
> We use streaming so we have to merge files regularly.
> If we can reduce the duration for some big Merge queries and make auto 
> compaction works properly it should be ok.
> Problem: at the moment, compactions are not triggered automatically. Have you 
> an idea ?
> 
> Le mar. 2 juin 2020 à 12:57, Peter Vary  > a écrit :
> Hi David,
> 
> You do not really need to run compaction every time.
> Is it possible to wait for the compaction to start automatically next time?
> 
> Thanks,
> Peter
> 
>> On Jun 2, 2020, at 12:51, David Morin > > wrote:
>> 
>> Thanks Peter,
>> 
>> Any workaround on HDP 2.6.x with Hive 2 ?
>> Otherwise, the only way is to reduce time it takes for this "merge" queries 
>> in order to cancel locks and related transactions. Am I right ?
>> 
>> Le mar. 2 juin 2020 à 11:52, Peter Vary > > a écrit :
>> Hi David,
>> 
>> I think this jira describes your situation:
>> https://issues.apache.org/jira/browse/HIVE-16360 
>> 
>> 
>> "The reason is that compactor won't compact anything above the level of an 
>> open transaction.  So if there is a very long running txn, it may filter out 
>> all deltas in a partition.  This looks very confusing when looking at the 
>> logs."
>> 
>> Thanks,
>> Peter
>> 
>>> On Jun 2, 2020, at 11:44, David Morin >> > wrote:
>>> 
>>> I don't get it.
>>> The transaction id in the error message "No delta files or original files 
>>> found to compact in hdfs://... with min Open txnid:6704143" is related to a 
>>> current transaction (transaction + lock) for another user, another database 
>>> and table. But still it blocks the compaction for the current database/table
>>> 
>>> On 2020/06/01 20:13:08, David Morin >> > wrote: 
 Hi,
 
 I have a compaction issue on my cluster. When I force a compaction (major) 
 on one table I get this error in Metastore logs:
 
 2020-06-01 19:49:35,512 ERROR [-78]: compactor.CompactorMR 
 (CompactorMR.java:run(264)) - No delta files or original files found to 
 compact in hdfs://...hive/warehouse/my.db/test <> for 
 compactionId=57944651 with min Open txnid:6704143.  Compaction cannot 
 compact above this txnid
 
 Any idea about the root cause and how to fix it ?
 thx
 
 Current env: HDP 2.6.5
 Hive table in transactional mode (ORC)
 
>> 
> 

Re: compaction issue: Compaction cannot compact above this txnid

[David Morin]

Yes Peter. We can wait a little.
We use streaming so we have to merge files regularly.
If we can reduce the duration for some big Merge queries and make auto
compaction works properly it should be ok.
Problem: at the moment, compactions are not triggered automatically. Have
you an idea ?

Le mar. 2 juin 2020 à 12:57, Peter Vary  a écrit :

> Hi David,
>
> You do not really need to run compaction every time.
> Is it possible to wait for the compaction to start automatically next time?
>
> Thanks,
> Peter
>
> On Jun 2, 2020, at 12:51, David Morin  wrote:
>
> Thanks Peter,
>
> Any workaround on HDP 2.6.x with Hive 2 ?
> Otherwise, the only way is to reduce time it takes for this "merge"
> queries in order to cancel locks and related transactions. Am I right ?
>
> Le mar. 2 juin 2020 à 11:52, Peter Vary  a écrit :
>
>> Hi David,
>>
>> I think this jira describes your situation:
>> https://issues.apache.org/jira/browse/HIVE-16360
>>
>> "The reason is that compactor won't compact anything above the level of
>> an open transaction.  So if there is a very long running txn, it may filter
>> out all deltas in a partition.  This looks very confusing when looking at
>> the logs."
>>
>> Thanks,
>> Peter
>>
>> On Jun 2, 2020, at 11:44, David Morin  wrote:
>>
>> I don't get it.
>> The transaction id in the error message "No delta files or original files
>> found to compact in hdfs://... with min Open txnid:6704143" is related to a
>> current transaction (transaction + lock) for another user, another database
>> and table. But still it blocks the compaction for the current database/table
>>
>> On 2020/06/01 20:13:08, David Morin  wrote:
>>
>> Hi,
>>
>> I have a compaction issue on my cluster. When I force a compaction
>> (major) on one table I get this error in Metastore logs:
>>
>> 2020-06-01 19:49:35,512 ERROR [-78]: compactor.CompactorMR
>> (CompactorMR.java:run(264)) - No delta files or original files found to
>> compact in hdfs://...hive/warehouse/my.db/test for compactionId=57944651
>> with min Open txnid:6704143.  Compaction cannot compact above this txnid
>>
>> Any idea about the root cause and how to fix it ?
>> thx
>>
>> Current env: HDP 2.6.5
>> Hive table in transactional mode (ORC)
>>
>>
>>
>

Re: compaction issue: Compaction cannot compact above this txnid

[Peter Vary]

Hi David,

You do not really need to run compaction every time.
Is it possible to wait for the compaction to start automatically next time?

Thanks,
Peter

> On Jun 2, 2020, at 12:51, David Morin  wrote:
> 
> Thanks Peter,
> 
> Any workaround on HDP 2.6.x with Hive 2 ?
> Otherwise, the only way is to reduce time it takes for this "merge" queries 
> in order to cancel locks and related transactions. Am I right ?
> 
> Le mar. 2 juin 2020 à 11:52, Peter Vary  > a écrit :
> Hi David,
> 
> I think this jira describes your situation:
> https://issues.apache.org/jira/browse/HIVE-16360 
> 
> 
> "The reason is that compactor won't compact anything above the level of an 
> open transaction.  So if there is a very long running txn, it may filter out 
> all deltas in a partition.  This looks very confusing when looking at the 
> logs."
> 
> Thanks,
> Peter
> 
>> On Jun 2, 2020, at 11:44, David Morin > > wrote:
>> 
>> I don't get it.
>> The transaction id in the error message "No delta files or original files 
>> found to compact in hdfs://... with min Open txnid:6704143" is related to a 
>> current transaction (transaction + lock) for another user, another database 
>> and table. But still it blocks the compaction for the current database/table
>> 
>> On 2020/06/01 20:13:08, David Morin > > wrote: 
>>> Hi,
>>> 
>>> I have a compaction issue on my cluster. When I force a compaction (major) 
>>> on one table I get this error in Metastore logs:
>>> 
>>> 2020-06-01 19:49:35,512 ERROR [-78]: compactor.CompactorMR 
>>> (CompactorMR.java:run(264)) - No delta files or original files found to 
>>> compact in hdfs://...hive/warehouse/my.db/test <> for compactionId=57944651 
>>> with min Open txnid:6704143.  Compaction cannot compact above this txnid
>>> 
>>> Any idea about the root cause and how to fix it ?
>>> thx
>>> 
>>> Current env: HDP 2.6.5
>>> Hive table in transactional mode (ORC)
>>> 
> 

Re: compaction issue: Compaction cannot compact above this txnid

[David Morin]

Thanks Peter,

Any workaround on HDP 2.6.x with Hive 2 ?
Otherwise, the only way is to reduce time it takes for this "merge" queries
in order to cancel locks and related transactions. Am I right ?

Le mar. 2 juin 2020 à 11:52, Peter Vary  a écrit :

> Hi David,
>
> I think this jira describes your situation:
> https://issues.apache.org/jira/browse/HIVE-16360
>
> "The reason is that compactor won't compact anything above the level of an
> open transaction.  So if there is a very long running txn, it may filter
> out all deltas in a partition.  This looks very confusing when looking at
> the logs."
>
> Thanks,
> Peter
>
> On Jun 2, 2020, at 11:44, David Morin  wrote:
>
> I don't get it.
> The transaction id in the error message "No delta files or original files
> found to compact in hdfs://... with min Open txnid:6704143" is related to a
> current transaction (transaction + lock) for another user, another database
> and table. But still it blocks the compaction for the current database/table
>
> On 2020/06/01 20:13:08, David Morin  wrote:
>
> Hi,
>
> I have a compaction issue on my cluster. When I force a compaction (major)
> on one table I get this error in Metastore logs:
>
> 2020-06-01 19:49:35,512 ERROR [-78]: compactor.CompactorMR
> (CompactorMR.java:run(264)) - No delta files or original files found to
> compact in hdfs://...hive/warehouse/my.db/test for compactionId=57944651
> with min Open txnid:6704143.  Compaction cannot compact above this txnid
>
> Any idea about the root cause and how to fix it ?
> thx
>
> Current env: HDP 2.6.5
> Hive table in transactional mode (ORC)
>
>
>

Re: compaction issue: Compaction cannot compact above this txnid

[Peter Vary]

Hi David,

I think this jira describes your situation:
https://issues.apache.org/jira/browse/HIVE-16360 


"The reason is that compactor won't compact anything above the level of an open 
transaction.  So if there is a very long running txn, it may filter out all 
deltas in a partition.  This looks very confusing when looking at the logs."

Thanks,
Peter

> On Jun 2, 2020, at 11:44, David Morin  wrote:
> 
> I don't get it.
> The transaction id in the error message "No delta files or original files 
> found to compact in hdfs://... with min Open txnid:6704143" is related to a 
> current transaction (transaction + lock) for another user, another database 
> and table. But still it blocks the compaction for the current database/table
> 
> On 2020/06/01 20:13:08, David Morin  wrote: 
>> Hi,
>> 
>> I have a compaction issue on my cluster. When I force a compaction (major) 
>> on one table I get this error in Metastore logs:
>> 
>> 2020-06-01 19:49:35,512 ERROR [-78]: compactor.CompactorMR 
>> (CompactorMR.java:run(264)) - No delta files or original files found to 
>> compact in hdfs://...hive/warehouse/my.db/test for compactionId=57944651 
>> with min Open txnid:6704143.  Compaction cannot compact above this txnid
>> 
>> Any idea about the root cause and how to fix it ?
>> thx
>> 
>> Current env: HDP 2.6.5
>> Hive table in transactional mode (ORC)
>> 

Re: compaction issue: Compaction cannot compact above this txnid

[David Morin]

I don't get it.
The transaction id in the error message "No delta files or original files found 
to compact in hdfs://... with min Open txnid:6704143" is related to a current 
transaction (transaction + lock) for another user, another database and table. 
But still it blocks the compaction for the current database/table

On 2020/06/01 20:13:08, David Morin  wrote: 
> Hi,
> 
> I have a compaction issue on my cluster. When I force a compaction (major) on 
> one table I get this error in Metastore logs:
> 
> 2020-06-01 19:49:35,512 ERROR [-78]: compactor.CompactorMR 
> (CompactorMR.java:run(264)) - No delta files or original files found to 
> compact in hdfs://...hive/warehouse/my.db/test for compactionId=57944651 with 
> min Open txnid:6704143.  Compaction cannot compact above this txnid
> 
> Any idea about the root cause and how to fix it ?
> thx
> 
> Current env: HDP 2.6.5
> Hive table in transactional mode (ORC)
>