[jira] [Commented] (HADOOP-9230) TestUniformSizeInputFormat fails intermittently
[
https://issues.apache.org/jira/browse/HADOOP-9230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13590412#comment-13590412
]
Hudson commented on HADOOP-9230:
Integrated in Hadoop-Yarn-trunk #142 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/142/])
HADOOP-9230. TestUniformSizeInputFormat fails intermittently. (kkambatl via
tucu) (Revision 1451291)
Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1451291
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/mapred/TestUniformSizeInputFormat.java
TestUniformSizeInputFormat fails intermittently
---
Key: HADOOP-9230
URL: https://issues.apache.org/jira/browse/HADOOP-9230
Project: Hadoop Common
Issue Type: Bug
Components: test
Affects Versions: 2.0.2-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Labels: distcp
Fix For: 2.0.4-beta
Attachments: hadoop-9230.patch
TestUniformSizeFileInputFormat fails intermittently. I ran the test 50 times
and noticed 5 failures.
Haven't noticed any particular pattern to which runs fail.
A sample stack trace is as follows:
{noformat}
java.lang.AssertionError: expected:1944 but was:1820
at org.junit.Assert.fail(Assert.java:91)
at org.junit.Assert.failNotEquals(Assert.java:645)
at org.junit.Assert.assertEquals(Assert.java:126)
at org.junit.Assert.assertEquals(Assert.java:470)
at org.junit.Assert.assertEquals(Assert.java:454)
at
org.apache.hadoop.tools.mapred.TestUniformSizeInputFormat.checkAgainstLegacy(TestUniformSizeInputFormat.java:244)
at
org.apache.hadoop.tools.mapred.TestUniformSizeInputFormat.testGetSplits(TestUniformSizeInputFormat.java:126)
at
org.apache.hadoop.tools.mapred.TestUniformSizeInputFormat.testGetSplits(TestUniformSizeInputFormat.java:252)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9230) TestUniformSizeInputFormat fails intermittently
[
https://issues.apache.org/jira/browse/HADOOP-9230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13590483#comment-13590483
]
Hudson commented on HADOOP-9230:
Integrated in Hadoop-Hdfs-trunk #1331 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1331/])
HADOOP-9230. TestUniformSizeInputFormat fails intermittently. (kkambatl via
tucu) (Revision 1451291)
Result = FAILURE
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1451291
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/mapred/TestUniformSizeInputFormat.java
TestUniformSizeInputFormat fails intermittently
---
Key: HADOOP-9230
URL: https://issues.apache.org/jira/browse/HADOOP-9230
Project: Hadoop Common
Issue Type: Bug
Components: test
Affects Versions: 2.0.2-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Labels: distcp
Fix For: 2.0.4-beta
Attachments: hadoop-9230.patch
TestUniformSizeFileInputFormat fails intermittently. I ran the test 50 times
and noticed 5 failures.
Haven't noticed any particular pattern to which runs fail.
A sample stack trace is as follows:
{noformat}
java.lang.AssertionError: expected:1944 but was:1820
at org.junit.Assert.fail(Assert.java:91)
at org.junit.Assert.failNotEquals(Assert.java:645)
at org.junit.Assert.assertEquals(Assert.java:126)
at org.junit.Assert.assertEquals(Assert.java:470)
at org.junit.Assert.assertEquals(Assert.java:454)
at
org.apache.hadoop.tools.mapred.TestUniformSizeInputFormat.checkAgainstLegacy(TestUniformSizeInputFormat.java:244)
at
org.apache.hadoop.tools.mapred.TestUniformSizeInputFormat.testGetSplits(TestUniformSizeInputFormat.java:126)
at
org.apache.hadoop.tools.mapred.TestUniformSizeInputFormat.testGetSplits(TestUniformSizeInputFormat.java:252)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9230) TestUniformSizeInputFormat fails intermittently
[
https://issues.apache.org/jira/browse/HADOOP-9230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13590532#comment-13590532
]
Hudson commented on HADOOP-9230:
Integrated in Hadoop-Mapreduce-trunk #1359 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1359/])
HADOOP-9230. TestUniformSizeInputFormat fails intermittently. (kkambatl via
tucu) (Revision 1451291)
Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1451291
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/mapred/TestUniformSizeInputFormat.java
TestUniformSizeInputFormat fails intermittently
---
Key: HADOOP-9230
URL: https://issues.apache.org/jira/browse/HADOOP-9230
Project: Hadoop Common
Issue Type: Bug
Components: test
Affects Versions: 2.0.2-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Labels: distcp
Fix For: 2.0.4-beta
Attachments: hadoop-9230.patch
TestUniformSizeFileInputFormat fails intermittently. I ran the test 50 times
and noticed 5 failures.
Haven't noticed any particular pattern to which runs fail.
A sample stack trace is as follows:
{noformat}
java.lang.AssertionError: expected:1944 but was:1820
at org.junit.Assert.fail(Assert.java:91)
at org.junit.Assert.failNotEquals(Assert.java:645)
at org.junit.Assert.assertEquals(Assert.java:126)
at org.junit.Assert.assertEquals(Assert.java:470)
at org.junit.Assert.assertEquals(Assert.java:454)
at
org.apache.hadoop.tools.mapred.TestUniformSizeInputFormat.checkAgainstLegacy(TestUniformSizeInputFormat.java:244)
at
org.apache.hadoop.tools.mapred.TestUniformSizeInputFormat.testGetSplits(TestUniformSizeInputFormat.java:126)
at
org.apache.hadoop.tools.mapred.TestUniformSizeInputFormat.testGetSplits(TestUniformSizeInputFormat.java:252)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9230) TestUniformSizeInputFormat fails intermittently
[
https://issues.apache.org/jira/browse/HADOOP-9230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13589790#comment-13589790
]
Alejandro Abdelnur commented on HADOOP-9230:
Agree, committing patch.
TestUniformSizeInputFormat fails intermittently
---
Key: HADOOP-9230
URL: https://issues.apache.org/jira/browse/HADOOP-9230
Project: Hadoop Common
Issue Type: Bug
Components: test
Affects Versions: 2.0.2-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Labels: distcp
Attachments: hadoop-9230.patch
TestUniformSizeFileInputFormat fails intermittently. I ran the test 50 times
and noticed 5 failures.
Haven't noticed any particular pattern to which runs fail.
A sample stack trace is as follows:
{noformat}
java.lang.AssertionError: expected:1944 but was:1820
at org.junit.Assert.fail(Assert.java:91)
at org.junit.Assert.failNotEquals(Assert.java:645)
at org.junit.Assert.assertEquals(Assert.java:126)
at org.junit.Assert.assertEquals(Assert.java:470)
at org.junit.Assert.assertEquals(Assert.java:454)
at
org.apache.hadoop.tools.mapred.TestUniformSizeInputFormat.checkAgainstLegacy(TestUniformSizeInputFormat.java:244)
at
org.apache.hadoop.tools.mapred.TestUniformSizeInputFormat.testGetSplits(TestUniformSizeInputFormat.java:126)
at
org.apache.hadoop.tools.mapred.TestUniformSizeInputFormat.testGetSplits(TestUniformSizeInputFormat.java:252)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9230) TestUniformSizeInputFormat fails intermittently
[
https://issues.apache.org/jira/browse/HADOOP-9230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13589809#comment-13589809
]
Hudson commented on HADOOP-9230:
Integrated in Hadoop-trunk-Commit #3396 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/3396/])
HADOOP-9230. TestUniformSizeInputFormat fails intermittently. (kkambatl via
tucu) (Revision 1451291)
Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1451291
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/mapred/TestUniformSizeInputFormat.java
TestUniformSizeInputFormat fails intermittently
---
Key: HADOOP-9230
URL: https://issues.apache.org/jira/browse/HADOOP-9230
Project: Hadoop Common
Issue Type: Bug
Components: test
Affects Versions: 2.0.2-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Labels: distcp
Fix For: 2.0.4-beta
Attachments: hadoop-9230.patch
TestUniformSizeFileInputFormat fails intermittently. I ran the test 50 times
and noticed 5 failures.
Haven't noticed any particular pattern to which runs fail.
A sample stack trace is as follows:
{noformat}
java.lang.AssertionError: expected:1944 but was:1820
at org.junit.Assert.fail(Assert.java:91)
at org.junit.Assert.failNotEquals(Assert.java:645)
at org.junit.Assert.assertEquals(Assert.java:126)
at org.junit.Assert.assertEquals(Assert.java:470)
at org.junit.Assert.assertEquals(Assert.java:454)
at
org.apache.hadoop.tools.mapred.TestUniformSizeInputFormat.checkAgainstLegacy(TestUniformSizeInputFormat.java:244)
at
org.apache.hadoop.tools.mapred.TestUniformSizeInputFormat.testGetSplits(TestUniformSizeInputFormat.java:126)
at
org.apache.hadoop.tools.mapred.TestUniformSizeInputFormat.testGetSplits(TestUniformSizeInputFormat.java:252)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9230) TestUniformSizeInputFormat fails intermittently
[
https://issues.apache.org/jira/browse/HADOOP-9230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13582033#comment-13582033
]
Karthik Kambatla commented on HADOOP-9230:
--
Thanks for the investigation, Tom. As tailoring the test just to make it pass
doesn't make sense, I think we should get rid of the test. Pinged [~mithun]
about 10 days ago to see if he has any additional insights on this.
I think it is safe to commit the patch.
TestUniformSizeInputFormat fails intermittently
---
Key: HADOOP-9230
URL: https://issues.apache.org/jira/browse/HADOOP-9230
Project: Hadoop Common
Issue Type: Bug
Components: test
Affects Versions: 2.0.2-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Labels: distcp
Attachments: hadoop-9230.patch
TestUniformSizeFileInputFormat fails intermittently. I ran the test 50 times
and noticed 5 failures.
Haven't noticed any particular pattern to which runs fail.
A sample stack trace is as follows:
{noformat}
java.lang.AssertionError: expected:1944 but was:1820
at org.junit.Assert.fail(Assert.java:91)
at org.junit.Assert.failNotEquals(Assert.java:645)
at org.junit.Assert.assertEquals(Assert.java:126)
at org.junit.Assert.assertEquals(Assert.java:470)
at org.junit.Assert.assertEquals(Assert.java:454)
at
org.apache.hadoop.tools.mapred.TestUniformSizeInputFormat.checkAgainstLegacy(TestUniformSizeInputFormat.java:244)
at
org.apache.hadoop.tools.mapred.TestUniformSizeInputFormat.testGetSplits(TestUniformSizeInputFormat.java:126)
at
org.apache.hadoop.tools.mapred.TestUniformSizeInputFormat.testGetSplits(TestUniformSizeInputFormat.java:252)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9230) TestUniformSizeInputFormat fails intermittently
[
https://issues.apache.org/jira/browse/HADOOP-9230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573423#comment-13573423
]
Tom White commented on HADOOP-9230:
---
I guess that the legacy split check was included as a sanity check. Since the
split calculation is slightly different, the occasional difference should not
be too surprising. Nor should it be a problem for correctness, since the aim of
the calculation is to produce roughly equal-sized splits for distcp. Therefore
I think we can remove the legacy check in order to ensure that the test always
passes. +1 for the patch.
bq. I would expect the math discrepancy to lead to more than 10% failure rate
though.
Although the difference between floor and ceiling will almost always produce
different target split sizes differing by 1 (whenever the number of splits
doesn't exactly divide the total size), only very rarely will this affect
whether a file is or is not included in any particular split. This is why the
failures are relatively rare.
E.g. on one failure I got the file sizes were 1482, 2012, 1860, ... (making a
total size of 31439) and for 9 maps the legacy target size works out at 3493,
while the new target size is 3494. The first legacy split only included 1482
(since 1482 + 2012 3493) while the first split for the new code includes both
1482 and 2012.
TestUniformSizeInputFormat fails intermittently
---
Key: HADOOP-9230
URL: https://issues.apache.org/jira/browse/HADOOP-9230
Project: Hadoop Common
Issue Type: Bug
Components: test
Affects Versions: 2.0.2-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Labels: distcp
Attachments: hadoop-9230.patch
TestUniformSizeFileInputFormat fails intermittently. I ran the test 50 times
and noticed 5 failures.
Haven't noticed any particular pattern to which runs fail.
A sample stack trace is as follows:
{noformat}
java.lang.AssertionError: expected:1944 but was:1820
at org.junit.Assert.fail(Assert.java:91)
at org.junit.Assert.failNotEquals(Assert.java:645)
at org.junit.Assert.assertEquals(Assert.java:126)
at org.junit.Assert.assertEquals(Assert.java:470)
at org.junit.Assert.assertEquals(Assert.java:454)
at
org.apache.hadoop.tools.mapred.TestUniformSizeInputFormat.checkAgainstLegacy(TestUniformSizeInputFormat.java:244)
at
org.apache.hadoop.tools.mapred.TestUniformSizeInputFormat.testGetSplits(TestUniformSizeInputFormat.java:126)
at
org.apache.hadoop.tools.mapred.TestUniformSizeInputFormat.testGetSplits(TestUniformSizeInputFormat.java:252)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9230) TestUniformSizeInputFormat fails intermittently
[
https://issues.apache.org/jira/browse/HADOOP-9230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13570898#comment-13570898
]
Hadoop QA commented on HADOOP-9230:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12567942/hadoop-9230.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-tools/hadoop-distcp.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2147//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2147//console
This message is automatically generated.
TestUniformSizeInputFormat fails intermittently
---
Key: HADOOP-9230
URL: https://issues.apache.org/jira/browse/HADOOP-9230
Project: Hadoop Common
Issue Type: Bug
Components: test
Affects Versions: 2.0.2-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Labels: distcp
Attachments: hadoop-9230.patch
TestUniformSizeFileInputFormat fails intermittently. I ran the test 50 times
and noticed 5 failures.
Haven't noticed any particular pattern to which runs fail.
A sample stack trace is as follows:
{noformat}
java.lang.AssertionError: expected:1944 but was:1820
at org.junit.Assert.fail(Assert.java:91)
at org.junit.Assert.failNotEquals(Assert.java:645)
at org.junit.Assert.assertEquals(Assert.java:126)
at org.junit.Assert.assertEquals(Assert.java:470)
at org.junit.Assert.assertEquals(Assert.java:454)
at
org.apache.hadoop.tools.mapred.TestUniformSizeInputFormat.checkAgainstLegacy(TestUniformSizeInputFormat.java:244)
at
org.apache.hadoop.tools.mapred.TestUniformSizeInputFormat.testGetSplits(TestUniformSizeInputFormat.java:126)
at
org.apache.hadoop.tools.mapred.TestUniformSizeInputFormat.testGetSplits(TestUniformSizeInputFormat.java:252)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9230) TestUniformSizeInputFormat fails intermittently
[
https://issues.apache.org/jira/browse/HADOOP-9230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13557502#comment-13557502
]
Karthik Kambatla commented on HADOOP-9230:
--
checkAgainstLegacy() compares the generated splits against a legacy split
generation. I don't quite understand the purpose behind this check. Can anyone
who knows this better throw some light on why we need the test.
I noticed a conflict in the math between UniformSizeInputFormat split
generation and the legacy generation:
Current:
{code}
long nBytesPerSplit = (long) Math.ceil(totalSizeBytes * 1.0 / numSplits);
{code}
Legacy:
{code}
final long targetsize = totalFileSize / numSplits;
{code}
I would expect the math discrepancy to lead to more than 10% failure rate
though.
TestUniformSizeInputFormat fails intermittently
---
Key: HADOOP-9230
URL: https://issues.apache.org/jira/browse/HADOOP-9230
Project: Hadoop Common
Issue Type: Bug
Components: test
Affects Versions: 2.0.2-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Labels: distcp
TestUniformSizeFileInputFormat fails intermittently. I ran the test 50 times
and noticed 5 failures.
Haven't noticed any particular pattern to which runs fail.
A sample stack trace is as follows:
{noformat}
java.lang.AssertionError: expected:1944 but was:1820
at org.junit.Assert.fail(Assert.java:91)
at org.junit.Assert.failNotEquals(Assert.java:645)
at org.junit.Assert.assertEquals(Assert.java:126)
at org.junit.Assert.assertEquals(Assert.java:470)
at org.junit.Assert.assertEquals(Assert.java:454)
at
org.apache.hadoop.tools.mapred.TestUniformSizeInputFormat.checkAgainstLegacy(TestUniformSizeInputFormat.java:244)
at
org.apache.hadoop.tools.mapred.TestUniformSizeInputFormat.testGetSplits(TestUniformSizeInputFormat.java:126)
at
org.apache.hadoop.tools.mapred.TestUniformSizeInputFormat.testGetSplits(TestUniformSizeInputFormat.java:252)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
