[ https://issues.apache.org/jira/browse/HIVE-16079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15975932#comment-15975932 ]
Hive QA commented on HIVE-16079: -------------------------------- Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12864068/HIVE-16079.02.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10581 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr] (batchId=143) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4772/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4772/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4772/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12864068 - PreCommit-HIVE-Build > HS2: high memory pressure due to duplicate Properties objects > ------------------------------------------------------------- > > Key: HIVE-16079 > URL: https://issues.apache.org/jira/browse/HIVE-16079 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 > Reporter: Misha Dmitriev > Assignee: Misha Dmitriev > Attachments: HIVE-16079.01.patch, HIVE-16079.02.patch, > hs2-crash-2000p-500m-50q.txt > > > I've created a Hive table with 2000 partitions, each backed by two files, > with one row in each file. When I execute some number of concurrent queries > against this table, e.g. as follows > {code} > for i in `seq 1 50`; do beeline -u jdbc:hive2://localhost:10000 -n admin -p > admin -e "select count(i_f_1) from misha_table;" & done > {code} > it results in a big memory spike. With 20 queries I caused an OOM in a HS2 > server with -Xmx200m and with 50 queries - in the one with -Xmx500m. > I am attaching the results of jxray (www.jxray.com) analysis of a heap dump > that was generated in the 50queries/500m heap scenario. It suggests that > there are several opportunities to reduce memory pressure with not very > invasive changes to the code. One (duplicate strings) has been addressed in > https://issues.apache.org/jira/browse/HIVE-15882 In this ticket, I am going > to address the fact that almost 20% of memory is used by instances of > java.util.Properties. These objects are highly duplicate, since for each > partition each concurrently running query creates its own copy of Partion, > PartitionDesc and Properties. Thus we have nearly 100,000 (50 queries * 2,000 > partitions) Properties in memory. By interning/deduplicating these objects we > may be able to save perhaps 15% of memory. > Note, however, that if there are queries that mutate partitions, the > corresponding Properties would be mutated as well. Thus we cannot simply use > a single "canonicalized" Properties object at all times for all Partition > objects representing the same DB partition. Instead, I am going to introduce > a special CopyOnFirstWriteProperties class. Such an object initially > internally references a canonicalized Properties object, and keeps doing so > while only read methods are called. However, once any mutating method is > called, the given CopyOnFirstWriteProperties copies the data into its own > table from the canonicalized table, and uses it ever after. -- This message was sent by Atlassian JIRA (v6.3.15#6346)