Re: Odp: Re: hive 4.0.0-alpha2 external table locks

[Denys Kuzmenko]

DbTxnManager doesn't have support for explicit locking. 
You could use HMSClient to issue an explicit lock request or create a lock 
record manually in the backend DB `HIVE_LOCKS` table.
Maybe you could temporarily restrict access to those tables via Ranger or drop 
them from the HMS while performing maintenance?

Re: Odp: Re: hive 4.0.0-alpha2 external table locks

[Wojtek Meler]

Any other options with DbTxnManager to prevent the access from hive queries to 
files on hdfs while performing maintenance of external tables ?

Re: Odp: Re: hive 4.0.0-alpha2 external table locks

[Denys Kuzmenko]

Unfortunately no. DummyTxnManager doesn't support DB locks, only Zookeeper and 
in-memory. So you won't be able to transparently lock the external table from 
other non-DummyTxtManager's.   

Re: Odp: Re: hive 4.0.0-alpha2 external table locks

[Wojtek Meler]

Would it force non-DummyTxtManager instances to not access external tables 
during exclusive locks?   Dnia 1 lutego 2023 15:55 Denys Kuzmenko 
 napisał(a):  DbTxnManager comes with DbLockManager 
implementation that doesn't support explicit lock/unlock functionality.  
That's something we plan to address in a future. For now you can configure 
a dedicated HS2 instance with DummyTxnManager to have explicit locking 
functionality for ext tables.

Re: Odp: Re: hive 4.0.0-alpha2 external table locks

[Denys Kuzmenko]

DbTxnManager comes with DbLockManager implementation that doesn't support 
explicit lock/unlock functionality. 
It was designed to be used only for ACID tables, however, due to code refactor 
challenges is reused for external tables as well. This comes with few drawbacks 
like suboptimal behavior (opens a txn for an ext table) and reduced 
functionality.
That's something we plan to address in a future. For now you can configure a 
dedicated HS2 instance with DummyTxnManager to have explicit locking 
functionality for ext tables.  

Re: Odp: Re: hive 4.0.0-alpha2 external table locks

[Wojtek Meler]

We are using external tables on early stages of ETL. Next we switch to managed 
tables as we don't need to worry about partition management.   With 
DummyTxnManager we had problems dealing with managed tables.   Regards,  Wojtek 
  Dnia 1 lutego 2023 10:31 Denys Kuzmenko  
napisał(a):  If you are using external tables only, you could try switching to 
DummyTxnManager.  It supports explicit lock requests.

Re: Odp: Re: hive 4.0.0-alpha2 external table locks

[Denys Kuzmenko]

If you are using external tables only, you could try switching to 
DummyTxnManager.  It supports explicit lock requests.

Odp: Re: hive 4.0.0-alpha2 external table locks

[Wojtek Meler]

My settings regarding transactions are:   hive.txn.acid.dir.cache.duration=120  
 |  | hive.txn.acid.metrics.delta.num.threshold=100  |  | 
hive.txn.acid.metrics.delta.pct.threshold=0.01 |  | 
hive.txn.acid.metrics.max.cache.size=100   |  | 
hive.txn.acid.metrics.obsolete.delta.num.threshold=100 |  | 
hive.txn.acid.metrics.reporting.interval=30s   |  | 
hive.txn.ext.locking.enabled=true  |  | 
hive.txn.filter.delete.events=true |  | 
hive.txn.heartbeat.threadpool.size=5   |  | 
hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager |  | 
hive.txn.manager.dump.lock.state.on.acquire.timeout=false |  | 
hive.txn.max.open.batch=1000   |  | 
hive.txn.nonacid.read.locks=true   |  | 
hive.txn.operational.properties=1  |  | 
hive.txn.read.locks=true   |  | 
hive.txn.readonly.enabled=false    |  | 
hive.txn.retrysnapshot.max.count=5 |  | 
hive.txn.stats.enabled=true    |  | 
hive.txn.strict.locking.mode=false |  | hive.txn.timeout=300s   
   |  | hive.txn.write.acid.version.file=true   
   |  | hive.txn.xlock.ctas=true   |  | 
hive.txn.xlock.iow=true    |  | 
hive.txn.xlock.mergeinsert=false   |  | 
hive.txn.xlock.write=true  |   after   LOCK TABLE 
example EXCLUSIVE;   I get: org.apache.hadoop.hive.ql.metadata.HiveException: 
Current transaction manager does not support explicit lock requests.  
Transaction manager:   org.apache.hadoop.hive.ql.lockmgr.DbTxnManager   It 
seams that HIVE-26258 did not enabled explicit locking with HQL with that 
transaction manager.   Any other options to exclusively lock partition?  Dnia 
30 stycznia 2023 23:29 Sai Hemanth Gantasala  
napisał(a):  Hi Wojtek Meler,  Have you tried setting the below config before 
running the queries?  set hive.txn.ext.locking.enabled=true      On Fri, Jan 
27, 2023 at 1:48 PM Wojtek Meler <   wme...@wp.pl > wrote:  I've 
enabled external tables locking with  issues.apache.org [HIVE-26258] Provide an 
option for enable locking of external tables - ASF JIRA (apache.org)  but still 
locking with HQL is not posible:   
org.apache.hadoop.hive.ql.metadata.HiveException: Current transaction manager 
does not support explicit lock requests.  Transaction manager:   
org.apache.hadoop.hive.ql.lockmgr.DbTxnManager   How to lock external table 
partition in hive4 to perform any maintenance on it?    Regards,  Wojtek