Jeongdae Kim created HIVE-25472:
-----------------------------------
Summary: Prevent hive-server2 from getting OOM(Compressed space
size) (Backport HIVE-18920)
Key: HIVE-25472
URL: https://issues.apache.org/jira/browse/HIVE-25472
Project: Hive
Issue Type: Bug
Affects Versions: 2.3.8
Reporter: Jeongdae Kim
Assignee: Jeongdae Kim
Our hive servers are getting shutdown regularly by OOM.
{code:java}
Terminating due to java.lang.OutOfMemoryError: Compressed class space {code}
We found out a lot of classes generated by janino compiler exist from heap dump,
(about 98% of all classes loaded)
and those generated classes are cached (in JaninoRelMetadataProvider)
This cache has no expiration, and hive server makes new metadata providers, one
of keys for caching, every query, which means hive servers make metadata
classes generated in runtime every query and we can't utilize the cache, and
finally those classes can't be loaded due to lack of meta space.
By this issue, hive servers are getting slow down because it takes too much
time to load classes, until OOM, as below flame graph.
I think we can fix this issue by either
a) maintain a static metadata provider (HIVE-18920)
or
b) add an option to make constant sizing cache
(https://issues.apache.org/jira/browse/CALCITE-1808)
to appy b), we need to upgrade calcite version to 1.15, but this includes lots
of changes.
it may be inappropriate for patch releases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
