JinsuKim created HIVE-13665: ------------------------------- Summary: HS2 memory leak When multiple queries are running with get_json_object Key: HIVE-13665 URL: https://issues.apache.org/jira/browse/HIVE-13665 Project: Hive Issue Type: Bug Affects Versions: 1.1.0 Reporter: JinsuKim
The extractObjectCache in UDFJson is increased over limitation(CACHE_SIZE = 16). When multiple queries are running concurrently on HS2 local(not mr/tez) with get_json_object or get_json_tuple {code:java|title=HS2 heap_dump} Object at 0x515ab18f8 instance of org.apache.hadoop.hive.ql.udf.UDFJson$HashCache@0x515ab18f8 (77 bytes) Class: class org.apache.hadoop.hive.ql.udf.UDFJson$HashCache Instance data members: accessOrder (Z) : false entrySet (L) : <null> hashSeed (I) : 0 header (L) : java.util.LinkedHashMap$Entry@0x515a577d0 (60 bytes) keySet (L) : <null> loadFactor (F) : 0.6 modCount (I) : 4741146 size (I) : 2733158 <========== here!! table (L) : [Ljava.util.HashMap$Entry;@0x7163d8b70 (67108880 bytes) threshold (I) : 5033165 values (L) : <null> References to this object: {code} I think that this problem be caused by the LinkedHashMap object is not thread-safe {code} * <p><strong>Note that this implementation is not synchronized.</strong> * If multiple threads access a linked hash map concurrently, and at least * one of the threads modifies the map structurally, it <em>must</em> be * synchronized externally. This is typically accomplished by * synchronizing on some object that naturally encapsulates the map. {code} Reproduce : # Multiple queries are running with get_json_object and small input data(for execution on hs2 local mode) # jvm heap dump & analyze {code:title=test scenario} Multiple queries are running with get_json_object and small input data(for execute on hs2 local mode) 1.hql : SELECT get_json_object(body, '$.fileSize'), get_json_object(body, '$.ps_totalTimeSeconds'), get_json_object(body, '$.totalTimeSeconds') FROM xxx.tttt WHERE part_hour='2016040105' 2.hql : SELECT get_json_object(body, '$.fileSize'), get_json_object(body, '$.ps_totalTimeSeconds'), get_json_object(body, '$.totalTimeSeconds') FROM xxx.tttt WHERE part_hour='2016040106' 3.hql : SELECT get_json_object(body, '$.fileSize'), get_json_object(body, '$.ps_totalTimeSeconds'), get_json_object(body, '$.totalTimeSeconds') FROM xxx.tttt WHERE part_hour='2016040107' 4.hql : SELECT get_json_object(body, '$.fileSize'), get_json_object(body, '$.ps_totalTimeSeconds'), get_json_object(body, '$.totalTimeSeconds') FROM xxx.tttt WHERE part_hour='2016040108' run.sh : t_cnt=0 while true do echo "query executing..." for i in 1 2 3 4 do beeline -u jdbc:hive2://localhost:10000 -n hive --silent=true -f $i.hql > $i.log 2>&1 & done wait t_cnt=`expr $t_cnt + 1` echo "query count : $t_cnt" sleep 2 done jvm heap dump & analyze : jmap -dump:format=b,file=hive.dmp $PID jhat -J-mx48000m -port 8080 hive.dmp & {code} Finally I have attached our patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)