[jira] Commented: (PIG-1231) Default DataBagIterator.hasNext() should be idempotent in all cases
[ https://issues.apache.org/jira/browse/PIG-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12832904#action_12832904 ] Hadoop QA commented on PIG-1231: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12435645/PIG-1231-2.patch against trunk revision 909210. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/202/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/202/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/202/console This message is automatically generated. Default DataBagIterator.hasNext() should be idempotent in all cases --- Key: PIG-1231 URL: https://issues.apache.org/jira/browse/PIG-1231 Project: Pig Issue Type: Bug Components: impl Affects Versions: 0.6.0 Reporter: Daniel Dai Assignee: Daniel Dai Fix For: 0.6.0 Attachments: PIG-1231-1.patch, PIG-1231-2.patch DefaultDataBagIterator.hasNext() is not repeatable when the below conditions met: 1. There is no more tuple in the last spill file 2. There is no tuples in memory (all contents are spilled to files) This is not acceptable cuz the name hasNext() implies that it is idempotent. In BagFormat, we do misuse DataBagIterator.hasNext() because of the assumption that hasNext() is always idempotent, which leads to some mysterious errors. Condition 2 seems to be very restrictive, but when the databag is really big, the memory can hold less than a couple of tuples, the chance to hit 2. is high enough. Here is one error we saw: Caused by: java.io.IOException: Stream closed at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:145) at java.io.BufferedInputStream.fill(BufferedInputStream.java:189) at java.io.BufferedInputStream.read(BufferedInputStream.java:237) at java.io.DataInputStream.readByte(DataInputStream.java:248) at org.apache.pig.data.DefaultTuple.readFields(DefaultTuple.java:278) at org.apache.pig.data.DefaultDataBag$DefaultDataBagIterator.readFromFile(DefaultDataBag.java:237) ... 20 more This happens because: we call hasNext(), which reach EOF and we close the file. Then we call hasNext() again in the assumption that it is idempotent. However, the stream is closed so we get this error message. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1231) Default DataBagIterator.hasNext() should be idempotent in all cases
[ https://issues.apache.org/jira/browse/PIG-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12833100#action_12833100 ] Alan Gates commented on PIG-1231: - +1 for patch 2. Default DataBagIterator.hasNext() should be idempotent in all cases --- Key: PIG-1231 URL: https://issues.apache.org/jira/browse/PIG-1231 Project: Pig Issue Type: Bug Components: impl Affects Versions: 0.6.0 Reporter: Daniel Dai Assignee: Daniel Dai Fix For: 0.6.0 Attachments: PIG-1231-1.patch, PIG-1231-2.patch DefaultDataBagIterator.hasNext() is not repeatable when the below conditions met: 1. There is no more tuple in the last spill file 2. There is no tuples in memory (all contents are spilled to files) This is not acceptable cuz the name hasNext() implies that it is idempotent. In BagFormat, we do misuse DataBagIterator.hasNext() because of the assumption that hasNext() is always idempotent, which leads to some mysterious errors. Condition 2 seems to be very restrictive, but when the databag is really big, the memory can hold less than a couple of tuples, the chance to hit 2. is high enough. Here is one error we saw: Caused by: java.io.IOException: Stream closed at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:145) at java.io.BufferedInputStream.fill(BufferedInputStream.java:189) at java.io.BufferedInputStream.read(BufferedInputStream.java:237) at java.io.DataInputStream.readByte(DataInputStream.java:248) at org.apache.pig.data.DefaultTuple.readFields(DefaultTuple.java:278) at org.apache.pig.data.DefaultDataBag$DefaultDataBagIterator.readFromFile(DefaultDataBag.java:237) ... 20 more This happens because: we call hasNext(), which reach EOF and we close the file. Then we call hasNext() again in the assumption that it is idempotent. However, the stream is closed so we get this error message. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1231) Default DataBagIterator.hasNext() should be idempotent in all cases
[ https://issues.apache.org/jira/browse/PIG-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12833129#action_12833129 ] Daniel Dai commented on PIG-1231: - Seems hudson is not running the testing process at all. Manual test success in both trunk and 0.6 branch. Did not include new testcase since it is a fix to existing testcase. Default DataBagIterator.hasNext() should be idempotent in all cases --- Key: PIG-1231 URL: https://issues.apache.org/jira/browse/PIG-1231 Project: Pig Issue Type: Bug Components: impl Affects Versions: 0.6.0 Reporter: Daniel Dai Assignee: Daniel Dai Fix For: 0.6.0 Attachments: PIG-1231-1.patch, PIG-1231-2.patch DefaultDataBagIterator.hasNext() is not repeatable when the below conditions met: 1. There is no more tuple in the last spill file 2. There is no tuples in memory (all contents are spilled to files) This is not acceptable cuz the name hasNext() implies that it is idempotent. In BagFormat, we do misuse DataBagIterator.hasNext() because of the assumption that hasNext() is always idempotent, which leads to some mysterious errors. Condition 2 seems to be very restrictive, but when the databag is really big, the memory can hold less than a couple of tuples, the chance to hit 2. is high enough. Here is one error we saw: Caused by: java.io.IOException: Stream closed at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:145) at java.io.BufferedInputStream.fill(BufferedInputStream.java:189) at java.io.BufferedInputStream.read(BufferedInputStream.java:237) at java.io.DataInputStream.readByte(DataInputStream.java:248) at org.apache.pig.data.DefaultTuple.readFields(DefaultTuple.java:278) at org.apache.pig.data.DefaultDataBag$DefaultDataBagIterator.readFromFile(DefaultDataBag.java:237) ... 20 more This happens because: we call hasNext(), which reach EOF and we close the file. Then we call hasNext() again in the assumption that it is idempotent. However, the stream is closed so we get this error message. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1231) Default DataBagIterator.hasNext() should be idempotent in all cases
[ https://issues.apache.org/jira/browse/PIG-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12831400#action_12831400 ] Hadoop QA commented on PIG-1231: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12435230/PIG-1231-1.patch against trunk revision 907760. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/206/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/206/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/206/console This message is automatically generated. Default DataBagIterator.hasNext() should be idempotent in all cases --- Key: PIG-1231 URL: https://issues.apache.org/jira/browse/PIG-1231 Project: Pig Issue Type: Bug Components: impl Affects Versions: 0.6.0 Reporter: Daniel Dai Assignee: Daniel Dai Fix For: 0.6.0 Attachments: PIG-1231-1.patch DefaultDataBagIterator.hasNext() is not repeatable when the below conditions met: 1. There is no more tuple in the last spill file 2. There is no tuples in memory (all contents are spilled to files) This is not acceptable cuz the name hasNext() implies that it is idempotent. In BagFormat, we do misuse DataBagIterator.hasNext() because of the assumption that hasNext() is always idempotent, which leads to some mysterious errors. Condition 2 seems to be very restrictive, but when the databag is really big, the memory can hold less than a couple of tuples, the chance to hit 2. is high enough. Here is one error we saw: Caused by: java.io.IOException: Stream closed at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:145) at java.io.BufferedInputStream.fill(BufferedInputStream.java:189) at java.io.BufferedInputStream.read(BufferedInputStream.java:237) at java.io.DataInputStream.readByte(DataInputStream.java:248) at org.apache.pig.data.DefaultTuple.readFields(DefaultTuple.java:278) at org.apache.pig.data.DefaultDataBag$DefaultDataBagIterator.readFromFile(DefaultDataBag.java:237) ... 20 more This happens because: we call hasNext(), which reach EOF and we close the file. Then we call hasNext() again in the assumption that it is idempotent. However, the stream is closed so we get this error message. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1231) Default DataBagIterator.hasNext() should be idempotent in all cases
[ https://issues.apache.org/jira/browse/PIG-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12831558#action_12831558 ] Daniel Dai commented on PIG-1231: - testCompressed1: java.lang.IllegalArgumentException: port out of range:-1. Not a real problem. Manual test passes. Default DataBagIterator.hasNext() should be idempotent in all cases --- Key: PIG-1231 URL: https://issues.apache.org/jira/browse/PIG-1231 Project: Pig Issue Type: Bug Components: impl Affects Versions: 0.6.0 Reporter: Daniel Dai Assignee: Daniel Dai Fix For: 0.6.0 Attachments: PIG-1231-1.patch DefaultDataBagIterator.hasNext() is not repeatable when the below conditions met: 1. There is no more tuple in the last spill file 2. There is no tuples in memory (all contents are spilled to files) This is not acceptable cuz the name hasNext() implies that it is idempotent. In BagFormat, we do misuse DataBagIterator.hasNext() because of the assumption that hasNext() is always idempotent, which leads to some mysterious errors. Condition 2 seems to be very restrictive, but when the databag is really big, the memory can hold less than a couple of tuples, the chance to hit 2. is high enough. Here is one error we saw: Caused by: java.io.IOException: Stream closed at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:145) at java.io.BufferedInputStream.fill(BufferedInputStream.java:189) at java.io.BufferedInputStream.read(BufferedInputStream.java:237) at java.io.DataInputStream.readByte(DataInputStream.java:248) at org.apache.pig.data.DefaultTuple.readFields(DefaultTuple.java:278) at org.apache.pig.data.DefaultDataBag$DefaultDataBagIterator.readFromFile(DefaultDataBag.java:237) ... 20 more This happens because: we call hasNext(), which reach EOF and we close the file. Then we call hasNext() again in the assumption that it is idempotent. However, the stream is closed so we get this error message. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1231) Default DataBagIterator.hasNext() should be idempotent in all cases
[ https://issues.apache.org/jira/browse/PIG-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12831572#action_12831572 ] Alan Gates commented on PIG-1231: - +1 Changes look good. Default DataBagIterator.hasNext() should be idempotent in all cases --- Key: PIG-1231 URL: https://issues.apache.org/jira/browse/PIG-1231 Project: Pig Issue Type: Bug Components: impl Affects Versions: 0.6.0 Reporter: Daniel Dai Assignee: Daniel Dai Fix For: 0.6.0 Attachments: PIG-1231-1.patch DefaultDataBagIterator.hasNext() is not repeatable when the below conditions met: 1. There is no more tuple in the last spill file 2. There is no tuples in memory (all contents are spilled to files) This is not acceptable cuz the name hasNext() implies that it is idempotent. In BagFormat, we do misuse DataBagIterator.hasNext() because of the assumption that hasNext() is always idempotent, which leads to some mysterious errors. Condition 2 seems to be very restrictive, but when the databag is really big, the memory can hold less than a couple of tuples, the chance to hit 2. is high enough. Here is one error we saw: Caused by: java.io.IOException: Stream closed at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:145) at java.io.BufferedInputStream.fill(BufferedInputStream.java:189) at java.io.BufferedInputStream.read(BufferedInputStream.java:237) at java.io.DataInputStream.readByte(DataInputStream.java:248) at org.apache.pig.data.DefaultTuple.readFields(DefaultTuple.java:278) at org.apache.pig.data.DefaultDataBag$DefaultDataBagIterator.readFromFile(DefaultDataBag.java:237) ... 20 more This happens because: we call hasNext(), which reach EOF and we close the file. Then we call hasNext() again in the assumption that it is idempotent. However, the stream is closed so we get this error message. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
