[
https://issues.apache.org/jira/browse/TEZ-3402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated TEZ-3402:
-------------------------
Description:
Bad configs triggers integer overflow. This is a 5Tb query which tries to group
by max-size of 4096.
{code}
// splits too small to work. Need to override with size.
int newDesiredNumSplits = (int)(totalLength/minLengthPerGroup) + 1;
{code}
{code}
diagnostics=[Vertex vertex_1470081722620_0072_3_00 [Map 2] killed/failed due
to:ROOT_INPUT_INIT_FAILURE, Vertex Input: srvc_fee initializer failed,
vertex=vertex_1470081722620_0072_3_00 [Map 2],
java.lang.IllegalArgumentException: Illegal Capacity: -1401168103
at java.util.ArrayList.<init>(ArrayList.java:156)
at
org.apache.hadoop.mapred.split.TezMapredSplitsGrouper.getGroupedSplits(TezMapredSplitsGrouper.java:230)
at
org.apache.hadoop.hive.ql.exec.tez.SplitGrouper.group(SplitGrouper.java:89)
at
org.apache.hadoop.hive.ql.exec.tez.SplitGrouper.generateGroupedSplits(SplitGrouper.java:168)
at
org.apache.hadoop.hive.ql.exec.tez.SplitGrouper.generateGroupedSplits(SplitGrouper.java:138)
at
org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:159)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:273)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:266)
{code}
was:
Bad configs triggers integer overflow. This is a 5Tb query which tries to group
by max-size of 4096.
{code}
groupedSplits = new ArrayList<GroupedSplitContainer>(desiredNumSplits);
{code}
{code}
diagnostics=[Vertex vertex_1470081722620_0072_3_00 [Map 2] killed/failed due
to:ROOT_INPUT_INIT_FAILURE, Vertex Input: srvc_fee initializer failed,
vertex=vertex_1470081722620_0072_3_00 [Map 2],
java.lang.IllegalArgumentException: Illegal Capacity: -1401168103
at java.util.ArrayList.<init>(ArrayList.java:156)
at
org.apache.hadoop.mapred.split.TezMapredSplitsGrouper.getGroupedSplits(TezMapredSplitsGrouper.java:230)
at
org.apache.hadoop.hive.ql.exec.tez.SplitGrouper.group(SplitGrouper.java:89)
at
org.apache.hadoop.hive.ql.exec.tez.SplitGrouper.generateGroupedSplits(SplitGrouper.java:168)
at
org.apache.hadoop.hive.ql.exec.tez.SplitGrouper.generateGroupedSplits(SplitGrouper.java:138)
at
org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:159)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:273)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:266)
{code}
> SplitGrouper: Integer overflow
> ------------------------------
>
> Key: TEZ-3402
> URL: https://issues.apache.org/jira/browse/TEZ-3402
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.8.4
> Reporter: Gopal V
>
> Bad configs triggers integer overflow. This is a 5Tb query which tries to
> group by max-size of 4096.
> {code}
> // splits too small to work. Need to override with size.
> int newDesiredNumSplits = (int)(totalLength/minLengthPerGroup) + 1;
> {code}
> {code}
> diagnostics=[Vertex vertex_1470081722620_0072_3_00 [Map 2] killed/failed due
> to:ROOT_INPUT_INIT_FAILURE, Vertex Input: srvc_fee initializer failed,
> vertex=vertex_1470081722620_0072_3_00 [Map 2],
> java.lang.IllegalArgumentException: Illegal Capacity: -1401168103
> at java.util.ArrayList.<init>(ArrayList.java:156)
> at
> org.apache.hadoop.mapred.split.TezMapredSplitsGrouper.getGroupedSplits(TezMapredSplitsGrouper.java:230)
> at
> org.apache.hadoop.hive.ql.exec.tez.SplitGrouper.group(SplitGrouper.java:89)
> at
> org.apache.hadoop.hive.ql.exec.tez.SplitGrouper.generateGroupedSplits(SplitGrouper.java:168)
> at
> org.apache.hadoop.hive.ql.exec.tez.SplitGrouper.generateGroupedSplits(SplitGrouper.java:138)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:159)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:273)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:266)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
