GitHub user guangyy opened a pull request:

    https://github.com/apache/hive/pull/453

    Fix the leak of lock during concurrent partition drop

    We have seen a leaked lock on hive metastore DB which caused all
    PARTITION insertion failed on timeout waiting for lock until the
    metastore service is restarted.
    
    A transaction dump on the DB shows there is a thread that is Sleep which
    potentiall holds the the lock, like:
      trx_id: 33603171058
                     trx_state: RUNNING
                   trx_started: 2018-10-23 06:43:22
         trx_requested_lock_id: NULL
              trx_wait_started: NULL
                    trx_weight: 70298
           trx_mysql_thread_id: 275402202
                     trx_query: NULL
           trx_operation_state: NULL
             trx_tables_in_use: 0
             trx_tables_locked: 0
              trx_lock_structs: 21286
         trx_lock_memory_bytes: 2881064
               trx_rows_locked: 98810
             trx_rows_modified: 49012
       trx_concurrency_tickets: 0
           trx_isolation_level: READ COMMITTED
             trx_unique_checks: 1
        trx_foreign_key_checks: 1
    trx_last_foreign_key_error: NULL
     trx_adaptive_hash_latched: 0
     trx_adaptive_hash_timeout: 0
              trx_is_read_only: 0
    trx_autocommit_non_locking: 0
                            ID: 275402202
                          USER: metastore_gold
                          HOST: 10.37.182.82:36684
                            DB: metastoregold
                       COMMAND: Sleep
                          TIME: 1
                         STATE:
                          INFO: NULL
                      duration: 1316
    
    Given the HOST ip, we trace back to the hive metastore instance and found 
the following exceptions:
    2018-10-23 06:43:22,805 WARN DataNucleus.Persistence: Exception thrown by 
StateManager.isLoaded
    No such database row
    org.datanucleus.exceptions.NucleusObjectNotFoundException: No such database 
row
            at 
org.datanucleus.store.rdbms.request.FetchRequest.execute(FetchRequest.java:357)
            at 
org.datanucleus.store.rdbms.RDBMSPersistenceHandler.fetchObject(RDBMSPersistenceHandler.java:324)
            at 
org.datanucleus.state.AbstractStateManager.loadFieldsFromDatastore(AbstractStateManager.java:1120)
            at 
org.datanucleus.state.JDOStateManager.loadSpecifiedFields(JDOStateManager.java:2916)
            at 
org.datanucleus.state.JDOStateManager.isLoaded(JDOStateManager.java:3219)
    
    The problem is that the caller expects a NULL if the partition does not 
exist, however, the convertToPart function would throw
    an exception which lead to the leak.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/guangyy/hive guang--fix-db-lock-leak

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/hive/pull/453.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #453
    
----
commit 7f6ace1146c32b7e6b8f175cc9c18489119c7613
Author: Guang Yang <guang.yang@...>
Date:   2018-10-24T22:28:09Z

    Fix the leak of lock during concurrent partition drop
    
    We have seen a leaked lock on hive metastore DB which caused all
    PARTITION insertion failed on timeout waiting for lock until the
    metastore service is restarted.
    
    A transaction dump on the DB shows there is a thread that is Sleep which
    potentiall holds the the lock, like:
      trx_id: 33603171058
                     trx_state: RUNNING
                   trx_started: 2018-10-23 06:43:22
         trx_requested_lock_id: NULL
              trx_wait_started: NULL
                    trx_weight: 70298
           trx_mysql_thread_id: 275402202
                     trx_query: NULL
           trx_operation_state: NULL
             trx_tables_in_use: 0
             trx_tables_locked: 0
              trx_lock_structs: 21286
         trx_lock_memory_bytes: 2881064
               trx_rows_locked: 98810
             trx_rows_modified: 49012
       trx_concurrency_tickets: 0
           trx_isolation_level: READ COMMITTED
             trx_unique_checks: 1
        trx_foreign_key_checks: 1
    trx_last_foreign_key_error: NULL
     trx_adaptive_hash_latched: 0
     trx_adaptive_hash_timeout: 0
              trx_is_read_only: 0
    trx_autocommit_non_locking: 0
                            ID: 275402202
                          USER: metastore_gold
                          HOST: 10.37.182.82:36684
                            DB: metastoregold
                       COMMAND: Sleep
                          TIME: 1
                         STATE:
                          INFO: NULL
                      duration: 1316
    
    Given the HOST ip, we trace back to the hive metastore instance and found 
the following exceptions:
    2018-10-23 06:43:22,805 WARN DataNucleus.Persistence: Exception thrown by 
StateManager.isLoaded
    No such database row
    org.datanucleus.exceptions.NucleusObjectNotFoundException: No such database 
row
            at 
org.datanucleus.store.rdbms.request.FetchRequest.execute(FetchRequest.java:357)
            at 
org.datanucleus.store.rdbms.RDBMSPersistenceHandler.fetchObject(RDBMSPersistenceHandler.java:324)
            at 
org.datanucleus.state.AbstractStateManager.loadFieldsFromDatastore(AbstractStateManager.java:1120)
            at 
org.datanucleus.state.JDOStateManager.loadSpecifiedFields(JDOStateManager.java:2916)
            at 
org.datanucleus.state.JDOStateManager.isLoaded(JDOStateManager.java:3219)
    
    The problem is that the caller expects a NULL if the partition does not 
exist, however, the convertToPart function would throw
    an exception which lead to the leak.

----


---

Reply via email to