[jira] [Commented] (TEZ-3212) IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB
[ https://issues.apache.org/jira/browse/TEZ-3212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105858#comment-16105858 ] Rajesh Balamohan commented on TEZ-3212: --- Thanks [~samirkhan], [~jeagles]. +1. No other comments from my side. > IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB > --- > > Key: TEZ-3212 > URL: https://issues.apache.org/jira/browse/TEZ-3212 > Project: Apache Tez > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Muhammad Samir Khan > Attachments: tez-3212.002.patch, tez-3212.003.patch, > tez-3212.004.patch, tez-3212.005.patch, TEZ-3212.1.patch > > > This is not a regression with respect to MR, just an issue that was > encountered with a job whose IFile record values (which can be of max size > 2GB) which can be successfully written but not successfully read. > Failure while running task:java.lang.NegativeArraySizeException > at > org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.nextRawValue(IFile.java:765) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (TEZ-3212) IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB
[ https://issues.apache.org/jira/browse/TEZ-3212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105671#comment-16105671 ] Muhammad Samir Khan commented on TEZ-3212: -- The failing tests pass locally. > IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB > --- > > Key: TEZ-3212 > URL: https://issues.apache.org/jira/browse/TEZ-3212 > Project: Apache Tez > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Muhammad Samir Khan > Attachments: tez-3212.002.patch, tez-3212.003.patch, > tez-3212.004.patch, tez-3212.005.patch, TEZ-3212.1.patch > > > This is not a regression with respect to MR, just an issue that was > encountered with a job whose IFile record values (which can be of max size > 2GB) which can be successfully written but not successfully read. > Failure while running task:java.lang.NegativeArraySizeException > at > org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.nextRawValue(IFile.java:765) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (TEZ-3212) IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB
[
https://issues.apache.org/jira/browse/TEZ-3212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105518#comment-16105518
]
TezQA commented on TEZ-3212:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12879387/tez-3212.005.patch
against master revision 6cbfd1e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.history.TestHistoryParser
org.apache.tez.dag.history.ats.acls.TestATSHistoryWithACLs
org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2592//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2592//console
This message is automatically generated.
> IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB
> ---
>
> Key: TEZ-3212
> URL: https://issues.apache.org/jira/browse/TEZ-3212
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Muhammad Samir Khan
> Attachments: tez-3212.002.patch, tez-3212.003.patch,
> tez-3212.004.patch, tez-3212.005.patch, TEZ-3212.1.patch
>
>
> This is not a regression with respect to MR, just an issue that was
> encountered with a job whose IFile record values (which can be of max size
> 2GB) which can be successfully written but not successfully read.
> Failure while running task:java.lang.NegativeArraySizeException
> at
> org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.nextRawValue(IFile.java:765)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (TEZ-3212) IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB
[ https://issues.apache.org/jira/browse/TEZ-3212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105461#comment-16105461 ] Jonathan Eagles commented on TEZ-3212: -- I'm happy with this change. [~rajesh.balamohan], do you have any more comments? > IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB > --- > > Key: TEZ-3212 > URL: https://issues.apache.org/jira/browse/TEZ-3212 > Project: Apache Tez > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Muhammad Samir Khan > Attachments: tez-3212.002.patch, tez-3212.003.patch, > tez-3212.004.patch, tez-3212.005.patch, TEZ-3212.1.patch > > > This is not a regression with respect to MR, just an issue that was > encountered with a job whose IFile record values (which can be of max size > 2GB) which can be successfully written but not successfully read. > Failure while running task:java.lang.NegativeArraySizeException > at > org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.nextRawValue(IFile.java:765) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (TEZ-3212) IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB
[ https://issues.apache.org/jira/browse/TEZ-3212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105330#comment-16105330 ] Jonathan Eagles commented on TEZ-3212: -- Can we also make the separate method for growing the array static? > IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB > --- > > Key: TEZ-3212 > URL: https://issues.apache.org/jira/browse/TEZ-3212 > Project: Apache Tez > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Muhammad Samir Khan > Attachments: tez-3212.002.patch, tez-3212.003.patch, > tez-3212.004.patch, TEZ-3212.1.patch > > > This is not a regression with respect to MR, just an issue that was > encountered with a job whose IFile record values (which can be of max size > 2GB) which can be successfully written but not successfully read. > Failure while running task:java.lang.NegativeArraySizeException > at > org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.nextRawValue(IFile.java:765) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (TEZ-3212) IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB
[
https://issues.apache.org/jira/browse/TEZ-3212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105255#comment-16105255
]
TezQA commented on TEZ-3212:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12879361/tez-3212.004.patch
against master revision 46e7706.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2591//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2591//console
This message is automatically generated.
> IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB
> ---
>
> Key: TEZ-3212
> URL: https://issues.apache.org/jira/browse/TEZ-3212
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Muhammad Samir Khan
> Attachments: tez-3212.002.patch, tez-3212.003.patch,
> tez-3212.004.patch, TEZ-3212.1.patch
>
>
> This is not a regression with respect to MR, just an issue that was
> encountered with a job whose IFile record values (which can be of max size
> 2GB) which can be successfully written but not successfully read.
> Failure while running task:java.lang.NegativeArraySizeException
> at
> org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.nextRawValue(IFile.java:765)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (TEZ-3212) IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB
[
https://issues.apache.org/jira/browse/TEZ-3212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16104018#comment-16104018
]
Rajesh Balamohan commented on TEZ-3212:
---
Thanks for sharing the patch [~samirkhan].
Very minor comment on the patch: May be the common code for computing new array
length in {{nextRawValue, readRawKey}} can be moved to separate method. E.g can
be changed to {{newLength = computeArraySize(currentKeyLength); . newLength
= computeArraySize(currentValueLength);}}
> IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB
> ---
>
> Key: TEZ-3212
> URL: https://issues.apache.org/jira/browse/TEZ-3212
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Muhammad Samir Khan
> Attachments: tez-3212.002.patch, tez-3212.003.patch, TEZ-3212.1.patch
>
>
> This is not a regression with respect to MR, just an issue that was
> encountered with a job whose IFile record values (which can be of max size
> 2GB) which can be successfully written but not successfully read.
> Failure while running task:java.lang.NegativeArraySizeException
> at
> org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.nextRawValue(IFile.java:765)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (TEZ-3212) IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB
[
https://issues.apache.org/jira/browse/TEZ-3212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16103921#comment-16103921
]
TezQA commented on TEZ-3212:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12879234/tez-3212.003.patch
against master revision 4b5448d.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2589//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2589//console
This message is automatically generated.
> IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB
> ---
>
> Key: TEZ-3212
> URL: https://issues.apache.org/jira/browse/TEZ-3212
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Muhammad Samir Khan
> Attachments: tez-3212.002.patch, tez-3212.003.patch, TEZ-3212.1.patch
>
>
> This is not a regression with respect to MR, just an issue that was
> encountered with a job whose IFile record values (which can be of max size
> 2GB) which can be successfully written but not successfully read.
> Failure while running task:java.lang.NegativeArraySizeException
> at
> org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.nextRawValue(IFile.java:765)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (TEZ-3212) IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB
[ https://issues.apache.org/jira/browse/TEZ-3212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096906#comment-16096906 ] Muhammad Samir Khan commented on TEZ-3212: -- Added unit test and throw an error message when the length exceeds the max. > IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB > --- > > Key: TEZ-3212 > URL: https://issues.apache.org/jira/browse/TEZ-3212 > Project: Apache Tez > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Muhammad Samir Khan > Attachments: tez-3212.002.patch, TEZ-3212.1.patch > > > This is not a regression with respect to MR, just an issue that was > encountered with a job whose IFile record values (which can be of max size > 2GB) which can be successfully written but not successfully read. > Failure while running task:java.lang.NegativeArraySizeException > at > org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.nextRawValue(IFile.java:765) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (TEZ-3212) IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB
[
https://issues.apache.org/jira/browse/TEZ-3212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093815#comment-16093815
]
Rohini Palaniswamy commented on TEZ-3212:
-
Based on
https://stackoverflow.com/questions/3038392/do-java-arrays-have-a-maximum-size
we might want to set Integer.MAX_VALUE - 8 as max array size to support more
jvms.
bq. In such instances (i.e 2 GB), should valBytes allocation be restricted only
to current key/value length? (i.e Math.min(currentKeyLength, MAX_ARRAY_LENGTH)
I like this idea as it might save lot of space. But instead of doing
Math.min, we should throw an error if data size exceeds the max array size
limit. If we don't do that we might be losing data. i.e
{code}
if (currentValueLength > MAX_ARRAY_LENGTH) {
throw new NegativeArraySizeException("Size of data " +
currentValueLength + " is greater than java maximum byte array size");
}
int newLength = currentValueLength << 1;
if (newLength < 0) {
newLength = currentValueLength;
}
{code}
> IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB
> ---
>
> Key: TEZ-3212
> URL: https://issues.apache.org/jira/browse/TEZ-3212
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: TEZ-3212.1.patch
>
>
> This is not a regression with respect to MR, just an issue that was
> encountered with a job whose IFile record values (which can be of max size
> 2GB) which can be successfully written but not successfully read.
> Failure while running task:java.lang.NegativeArraySizeException
> at
> org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.nextRawValue(IFile.java:765)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (TEZ-3212) IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB
[
https://issues.apache.org/jira/browse/TEZ-3212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15682025#comment-15682025
]
Rajesh Balamohan commented on TEZ-3212:
---
Even for a key/value size 1.3 GB, it would have thrown
{{NegativeArrayException}} in earlier case, as {{valBytes}} was trying to
allocate 2x the capacity.
With the patch, it would restrict the allocation to {{Integer.MAX_VALUE}}. In
such instances (i.e 2 GB), should {{valBytes}} allocation be restricted only to
current key/value length? (i.e {{Math.min(currentKeyLength,
MAX_ARRAY_LENGTH)}}. For e.g, in corner cases, it is possible that JVM was
able to allocate 1.4 GB instead of 2 GB and the min condition will help in such
instances.
It could throw OOM, if it is unable to allocate the space. But that is expected
behavior.
> IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB
> ---
>
> Key: TEZ-3212
> URL: https://issues.apache.org/jira/browse/TEZ-3212
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: TEZ-3212.1.patch
>
>
> This is not a regression with respect to MR, just an issue that was
> encountered with a job whose IFile record values (which can be of max size
> 2GB) which can be successfully written but not successfully read.
> Failure while running task:java.lang.NegativeArraySizeException
> at
> org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.nextRawValue(IFile.java:765)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3212) IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB
[
https://issues.apache.org/jira/browse/TEZ-3212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15678016#comment-15678016
]
TezQA commented on TEZ-3212:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12798372/TEZ-3212.1.patch
against master revision b71ea4a.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
See
https://builds.apache.org/job/PreCommit-TEZ-Build/2112//artifact/patchprocess/diffJavadocWarnings.txt
for details.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.runtime.library.common.sort.impl.TestPipelinedSorter
org.apache.tez.runtime.library.common.sort.impl.TestIFile
org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter
org.apache.tez.dag.app.TestDAGAppMaster
org.apache.tez.mapreduce.processor.map.TestMapProcessor
org.apache.tez.mapreduce.processor.reduce.TestReduceProcessor
org.apache.tez.mapreduce.TestMRRJobsDAGApi
org.apache.tez.dag.history.logging.ats.TestATSHistoryWithMiniCluster
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2112//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2112//console
This message is automatically generated.
> IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB
> ---
>
> Key: TEZ-3212
> URL: https://issues.apache.org/jira/browse/TEZ-3212
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: TEZ-3212.1.patch
>
>
> This is not a regression with respect to MR, just an issue that was
> encountered with a job whose IFile record values (which can be of max size
> 2GB) which can be successfully written but not successfully read.
> Failure while running task:java.lang.NegativeArraySizeException
> at
> org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.nextRawValue(IFile.java:765)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3212) IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB
[ https://issues.apache.org/jira/browse/TEZ-3212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15677795#comment-15677795 ] Hitesh Shah commented on TEZ-3212: -- \cc [~rajesh.balamohan] for review > IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB > --- > > Key: TEZ-3212 > URL: https://issues.apache.org/jira/browse/TEZ-3212 > Project: Apache Tez > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: TEZ-3212.1.patch > > > This is not a regression with respect to MR, just an issue that was > encountered with a job whose IFile record values (which can be of max size > 2GB) which can be successfully written but not successfully read. > Failure while running task:java.lang.NegativeArraySizeException > at > org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.nextRawValue(IFile.java:765) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3212) IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB
[ https://issues.apache.org/jira/browse/TEZ-3212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15246506#comment-15246506 ] Siddharth Seth commented on TEZ-3212: - [~jeagles] - this is a single key being extremely large ? I think there'll be additional places where this may need to be looked at, and possibly move to disk instead of trying to handle everything via memory. The sorters do that - writing a single record out per file. > IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB > --- > > Key: TEZ-3212 > URL: https://issues.apache.org/jira/browse/TEZ-3212 > Project: Apache Tez > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: TEZ-3212.1.patch > > > This is not a regression with respect to MR, just an issue that was > encountered with a job whose IFile record values (which can be of max size > 2GB) which can be successfully written but not successfully read. > Failure while running task:java.lang.NegativeArraySizeException > at > org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.nextRawValue(IFile.java:765) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3212) IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB
[
https://issues.apache.org/jira/browse/TEZ-3212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15238281#comment-15238281
]
TezQA commented on TEZ-3212:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12798372/TEZ-3212.1.patch
against master revision 53981d4.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.test.TestFaultTolerance
org.apache.tez.dag.app.rm.TestContainerReuse
The following test timeouts occurred in :
org.apache.tez.dag.app.dag.impl.TestCommit
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1653//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1653//console
This message is automatically generated.
> IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB
> ---
>
> Key: TEZ-3212
> URL: https://issues.apache.org/jira/browse/TEZ-3212
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: TEZ-3212.1.patch
>
>
> This is not a regression with respect to MR, just an issue that was
> encountered with a job whose IFile record values (which can be of max size
> 2GB) which can be successfully written but not successfully read.
> Failure while running task:java.lang.NegativeArraySizeException
> at
> org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.nextRawValue(IFile.java:765)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3212) IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB
[ https://issues.apache.org/jira/browse/TEZ-3212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15237994#comment-15237994 ] Jonathan Eagles commented on TEZ-3212: -- Of course the job above was behaving badly and was corrected to not do that. However, perhaps there are valid use cases for this working. > IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB > --- > > Key: TEZ-3212 > URL: https://issues.apache.org/jira/browse/TEZ-3212 > Project: Apache Tez > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: TEZ-3212.1.patch > > > This is not a regression with respect to MR, just an issue that was > encountered with a job whose IFile record values (which can be of max size > 2GB) which can be successfully written but not successfully read. > Failure while running task:java.lang.NegativeArraySizeException > at > org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.nextRawValue(IFile.java:765) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3212) IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB
[ https://issues.apache.org/jira/browse/TEZ-3212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15237992#comment-15237992 ] Jonathan Eagles commented on TEZ-3212: -- This patch is really just to start a discussion on what to do with really large values. It is arguable that IFile should be able to read what was written. MAX_ARRAY_LENGTH is due to the limitation where byte array of Integer.MAX_VALUE cannot be allocated due to jvm limits https://plumbr.eu/outofmemoryerror/requested-array-size-exceeds-vm-limit > IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB > --- > > Key: TEZ-3212 > URL: https://issues.apache.org/jira/browse/TEZ-3212 > Project: Apache Tez > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: TEZ-3212.1.patch > > > This is not a regression with respect to MR, just an issue that was > encountered with a job whose IFile record values (which can be of max size > 2GB) which can be successfully written but not successfully read. > Failure while running task:java.lang.NegativeArraySizeException > at > org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.nextRawValue(IFile.java:765) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
