Daniel Voros created HIVE-22901:
-----------------------------------
Summary: Variable substitution can lead to OOM on circular
references
Key: HIVE-22901
URL: https://issues.apache.org/jira/browse/HIVE-22901
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 3.1.2
Reporter: Daniel Voros
Assignee: Daniel Voros
{{SystemVariables#substitute()}} is dealing with circular references between
variables by only doing the substitution 40 times by default. If the
substituted part is sufficiently large though, it's possible that the
substitution will produce a string bigger than the heap size within the 40
executions.
Take the following test case that fails with OOM in current master (third round
of execution would need 10G heap, while running with only 2G):
{code}
@Test
public void testSubstitute() {
String randomPart = RandomStringUtils.random(100_000);
String reference = "${hiveconf:myTestVariable}";
StringBuilder longStringWithReferences = new StringBuilder();
for(int i = 0; i < 10; i ++) {
longStringWithReferences.append(randomPart).append(reference);
}
SystemVariables uut = new SystemVariables();
HiveConf conf = new HiveConf();
conf.set("myTestVariable", longStringWithReferences.toString());
uut.substitute(conf, longStringWithReferences.toString(), 40);
}
{code}
Produces:
{code}
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3332)
at
java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
at
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
at java.lang.StringBuilder.append(StringBuilder.java:136)
at
org.apache.hadoop.hive.conf.SystemVariables.substitute(SystemVariables.java:110)
at
org.apache.hadoop.hive.conf.SystemVariablesTest.testSubstitute(SystemVariablesTest.java:27)
{code}
We should check the size of the substituted query and bail out earlier.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
