Samrat Deb created HIVE-26024:
---------------------------------
Summary: Metastore RDS CPU spike in DDL commands ()
Key: HIVE-26024
URL: https://issues.apache.org/jira/browse/HIVE-26024
Project: Hive
Issue Type: Bug
Reporter: Samrat Deb
We are experiencing sudden spike in the cpu utilization of hive-metastore rds,
whenever there is `ALTER TABLE ` queries in hive.
There is sudden spike in the rds cpu (attached screenshot )
This leads to non-responsive metastore service , and effects other jobs/queries
Meta Information
-> Hive version - 3.1.2
-> Hive metastore version - 3.1.2
-> Hadoop version - 3.2.2
-> DB used for metastore - Postgres 42.2.18
Observations :-
1) On running DDL queries like "ALTER TABLE " commands it fails in around 120s
approx with the error message
```while processing statement: FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask. Unable to alter table. null```
though it has returned error , we observed that
even if the query has returned an error, the query still keeps running on the
metastore via querying pg_stat_activity table .
2) the spike in metastore CPU is mostly related to partition stats collection,
Tried running the queries via a hive cli in debug mode and noticed these logs
when the CPU was spiking
logs
```
2022-02-21T08:33:22,876 DEBUG [<object_id> main] metastore.ObjectStore:
Open transaction: count = 3, isActive = true at:
org.apache.hadoop.hive.metastore.ObjectStore.removeUnusedColumnDescriptor(ObjectStore.java:4232)
org.apache.hadoop.hive.metastore.ObjectStore.alterPartition(ObjectStore.java:4129)
sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
2022-02-21T08:33:22,876 DEBUG [<object_id> main] metastore.ObjectStore: execute
removeUnusedColumnDescriptor
2022-02-21T08:33:24,482 DEBUG [70618618-a5a0-497a-abc3-6aacf390024e main]
metastore.ObjectStore: Commit transaction: count = 2, isactive true at:
org.apache.hadoop.hive.metastore.ObjectStore.removeUnusedColumnDescriptor(ObjectStore.java:4245)
org.apache.hadoop.hive.metastore.ObjectStore.alterPartition(ObjectStore.java:4129)
sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
2022-02-21T08:33:24,482 DEBUG [<object_id> main] metastore.ObjectStore:
successfully deleted a CD in removeUnusedColumnDescriptor
2022-02-21T08:33:24,483 DEBUG [<object_id> main] metastore.ObjectStore: Commit
transaction: count = 1, isactive true at:
org.apache.hadoop.hive.metastore.ObjectStore.alterPartition(ObjectStore.java:4131)
sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2022-02-21T08:33:24,483 DEBUG [<object_id> main] metastore.ObjectStore: Open
transaction: count = 2, isActive = true at:
org.apache.hadoop.hive.metastore.ObjectStore.alterPartition(ObjectStore.java:4127)
sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2022-02-21T08:33:24,483 DEBUG [<object_id> main] metastore.ObjectStore: Open
transaction: count = 3, isActive = true at:```
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
