[jira] Created: (PIG-1450) TestAlgebraicEvalLocal failures due to OOM
TestAlgebraicEvalLocal failures due to OOM -- Key: PIG-1450 URL: https://issues.apache.org/jira/browse/PIG-1450 Project: Pig Issue Type: Test Affects Versions: 0.7.0, 0.8.0 Reporter: Eli Collins Fix For: 0.8.0 6 test cases in TestAlgebraicEvalLocal fail on trunk and release 0.7 across a number of different machines. Example failure: Unable to open iterator for alias myid org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias myid at org.apache.pig.PigServer.openIterator(PigServer.java:521) at org.apache.pig.test.TestAlgebraicEvalLocal.testGroupUniqueColumnCount(TestAlgebraicEvalLocal.java:236) Caused by: java.io.IOException: Job terminated with anomalous status FAILED at org.apache.pig.PigServer.openIterator(PigServer.java:515) Probably due to OOMs in the log: 10/06/14 19:38:43 WARN mapred.LocalJobRunner: job_local_0002 java.lang.OutOfMemoryError: Java heap space at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:781) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:524) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1450) TestAlgebraicEvalLocal failures due to OOM
[ https://issues.apache.org/jira/browse/PIG-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated PIG-1450: - Attachment: TEST-org.apache.pig.test.TestAlgebraicEvalLocal.txt Full test log attached. > TestAlgebraicEvalLocal failures due to OOM > -- > > Key: PIG-1450 > URL: https://issues.apache.org/jira/browse/PIG-1450 > Project: Pig > Issue Type: Test >Affects Versions: 0.7.0, 0.8.0 >Reporter: Eli Collins > Fix For: 0.8.0 > > Attachments: TEST-org.apache.pig.test.TestAlgebraicEvalLocal.txt > > > 6 test cases in TestAlgebraicEvalLocal fail on trunk and release 0.7 across a > number of different machines. > Example failure: > Unable to open iterator for alias myid > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to > open iterator for alias myid > at org.apache.pig.PigServer.openIterator(PigServer.java:521) > at > org.apache.pig.test.TestAlgebraicEvalLocal.testGroupUniqueColumnCount(TestAlgebraicEvalLocal.java:236) > Caused by: java.io.IOException: Job terminated with anomalous status FAILED > at org.apache.pig.PigServer.openIterator(PigServer.java:515) > Probably due to OOMs in the log: > 10/06/14 19:38:43 WARN mapred.LocalJobRunner: job_local_0002 > java.lang.OutOfMemoryError: Java heap space > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:781) > at > org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:524) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) > at > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1450) TestAlgebraicEvalLocal failures due to OOM
[ https://issues.apache.org/jira/browse/PIG-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879830#action_12879830 ] Eli Collins commented on PIG-1450: -- Could this be due to the switch to Hadoop's local mode? > TestAlgebraicEvalLocal failures due to OOM > -- > > Key: PIG-1450 > URL: https://issues.apache.org/jira/browse/PIG-1450 > Project: Pig > Issue Type: Test >Affects Versions: 0.7.0, 0.8.0 >Reporter: Eli Collins > Fix For: 0.8.0 > > Attachments: TEST-org.apache.pig.test.TestAlgebraicEvalLocal.txt > > > 6 test cases in TestAlgebraicEvalLocal fail on trunk and release 0.7 across a > number of different machines. > Example failure: > Unable to open iterator for alias myid > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to > open iterator for alias myid > at org.apache.pig.PigServer.openIterator(PigServer.java:521) > at > org.apache.pig.test.TestAlgebraicEvalLocal.testGroupUniqueColumnCount(TestAlgebraicEvalLocal.java:236) > Caused by: java.io.IOException: Job terminated with anomalous status FAILED > at org.apache.pig.PigServer.openIterator(PigServer.java:515) > Probably due to OOMs in the log: > 10/06/14 19:38:43 WARN mapred.LocalJobRunner: job_local_0002 > java.lang.OutOfMemoryError: Java heap space > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:781) > at > org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:524) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) > at > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1450) TestAlgebraicEvalLocal failures due to OOM
[ https://issues.apache.org/jira/browse/PIG-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879940#action_12879940 ] Eli Collins commented on PIG-1450: -- I confirmed this patch fixed the test failures. Thanks Daniel. > TestAlgebraicEvalLocal failures due to OOM > -- > > Key: PIG-1450 > URL: https://issues.apache.org/jira/browse/PIG-1450 > Project: Pig > Issue Type: Test >Affects Versions: 0.7.0, 0.8.0 >Reporter: Eli Collins > Fix For: 0.8.0 > > Attachments: PIG-1450.patch, > TEST-org.apache.pig.test.TestAlgebraicEvalLocal.txt > > > 6 test cases in TestAlgebraicEvalLocal fail on trunk and release 0.7 across a > number of different machines. > Example failure: > Unable to open iterator for alias myid > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to > open iterator for alias myid > at org.apache.pig.PigServer.openIterator(PigServer.java:521) > at > org.apache.pig.test.TestAlgebraicEvalLocal.testGroupUniqueColumnCount(TestAlgebraicEvalLocal.java:236) > Caused by: java.io.IOException: Job terminated with anomalous status FAILED > at org.apache.pig.PigServer.openIterator(PigServer.java:515) > Probably due to OOMs in the log: > 10/06/14 19:38:43 WARN mapred.LocalJobRunner: job_local_0002 > java.lang.OutOfMemoryError: Java heap space > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:781) > at > org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:524) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) > at > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (PIG-1632) The core jar in the tarball contains the kitchen sink
The core jar in the tarball contains the kitchen sink
--
Key: PIG-1632
URL: https://issues.apache.org/jira/browse/PIG-1632
Project: Pig
Issue Type: Bug
Components: build
Affects Versions: 0.8.0, 0.9.0
Reporter: Eli Collins
Fix For: site, 0.9.0
The core jar in the tarball contains the kitchen sink, it's not the same core
jar built by ant jar. This is problematic since other projects that want to
depend on the pig core jar just want pig core, but pig-0.8.0-SNAPSHOT-core.jar
in the tarball contains a bunch of other stuff (hadoop, com.google, commons,
etc) that may conflict with the packages also on a user's classpath.
{noformat}
pig1 (trunk)$ jar tvf build/pig-0.8.0-SNAPSHOT-core.jar |grep -v pig|wc -l
12
pig1 (trunk)$ tar xvzf build/pig-0.8.0-SNAPSHOT.tar.gz
...
pig1 (trunk)$ jar tvf pig-0.8.0-SNAPSHOT/pig-0.8.0-SNAPSHOT-core.jar |grep -v
pig|wc -l
4819
{noformat}
How about restricting the core jar to just Pig classes?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1632) The core jar in the tarball contains the kitchen sink
[
https://issues.apache.org/jira/browse/PIG-1632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Collins updated PIG-1632:
-
Attachment: pig-1632-1.patch
Attached patch updates the package target so that the tarball, and therefore
Pig release, just contain the Pig core jar. If a Pig release needs to bundle
Hadoop and a bunch of other stuff perhaps we could put those jars in lib
instead of the core jar.
Running things like the tests out of a tarball that just includes the core jar
works as these come in via ivy, anything else that needs to be tested?
> The core jar in the tarball contains the kitchen sink
> --
>
> Key: PIG-1632
> URL: https://issues.apache.org/jira/browse/PIG-1632
> Project: Pig
> Issue Type: Bug
> Components: build
>Affects Versions: 0.8.0, 0.9.0
>Reporter: Eli Collins
> Fix For: site, 0.9.0
>
> Attachments: pig-1632-1.patch
>
>
> The core jar in the tarball contains the kitchen sink, it's not the same core
> jar built by ant jar. This is problematic since other projects that want to
> depend on the pig core jar just want pig core, but
> pig-0.8.0-SNAPSHOT-core.jar in the tarball contains a bunch of other stuff
> (hadoop, com.google, commons, etc) that may conflict with the packages also
> on a user's classpath.
> {noformat}
> pig1 (trunk)$ jar tvf build/pig-0.8.0-SNAPSHOT-core.jar |grep -v pig|wc -l
> 12
> pig1 (trunk)$ tar xvzf build/pig-0.8.0-SNAPSHOT.tar.gz
> ...
> pig1 (trunk)$ jar tvf pig-0.8.0-SNAPSHOT/pig-0.8.0-SNAPSHOT-core.jar |grep -v
> pig|wc -l
> 4819
> {noformat}
> How about restricting the core jar to just Pig classes?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1632) The core jar in the tarball contains the kitchen sink
[
https://issues.apache.org/jira/browse/PIG-1632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Collins updated PIG-1632:
-
Status: Patch Available (was: Open)
> The core jar in the tarball contains the kitchen sink
> --
>
> Key: PIG-1632
> URL: https://issues.apache.org/jira/browse/PIG-1632
> Project: Pig
> Issue Type: Bug
> Components: build
>Affects Versions: 0.8.0, 0.9.0
>Reporter: Eli Collins
> Fix For: site, 0.9.0
>
> Attachments: pig-1632-1.patch
>
>
> The core jar in the tarball contains the kitchen sink, it's not the same core
> jar built by ant jar. This is problematic since other projects that want to
> depend on the pig core jar just want pig core, but
> pig-0.8.0-SNAPSHOT-core.jar in the tarball contains a bunch of other stuff
> (hadoop, com.google, commons, etc) that may conflict with the packages also
> on a user's classpath.
> {noformat}
> pig1 (trunk)$ jar tvf build/pig-0.8.0-SNAPSHOT-core.jar |grep -v pig|wc -l
> 12
> pig1 (trunk)$ tar xvzf build/pig-0.8.0-SNAPSHOT.tar.gz
> ...
> pig1 (trunk)$ jar tvf pig-0.8.0-SNAPSHOT/pig-0.8.0-SNAPSHOT-core.jar |grep -v
> pig|wc -l
> 4819
> {noformat}
> How about restricting the core jar to just Pig classes?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1632) The core jar in the tarball contains the kitchen sink
[
https://issues.apache.org/jira/browse/PIG-1632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913738#action_12913738
]
Eli Collins commented on PIG-1632:
--
Hey Olga,
Thanks for the feedback.Agree that we want the out of box experience to use
the same versions of other jars we've been testing with, but shouldn't that
happen by bundling the necessary jars in eg the lib directory rather than
embedding all the jars inside the core pig jar?
If people want all the dependencies bundled into a single jar, how about I
update the patch so the release has two jars: a pig.jar which is like the
current one (has all the other jars bundled in) and a pig-core.jar which just
has pig?
Thanks,
Eli
> The core jar in the tarball contains the kitchen sink
> --
>
> Key: PIG-1632
> URL: https://issues.apache.org/jira/browse/PIG-1632
> Project: Pig
> Issue Type: Bug
> Components: build
>Affects Versions: 0.8.0, 0.9.0
>Reporter: Eli Collins
> Fix For: site, 0.9.0
>
> Attachments: pig-1632-1.patch
>
>
> The core jar in the tarball contains the kitchen sink, it's not the same core
> jar built by ant jar. This is problematic since other projects that want to
> depend on the pig core jar just want pig core, but
> pig-0.8.0-SNAPSHOT-core.jar in the tarball contains a bunch of other stuff
> (hadoop, com.google, commons, etc) that may conflict with the packages also
> on a user's classpath.
> {noformat}
> pig1 (trunk)$ jar tvf build/pig-0.8.0-SNAPSHOT-core.jar |grep -v pig|wc -l
> 12
> pig1 (trunk)$ tar xvzf build/pig-0.8.0-SNAPSHOT.tar.gz
> ...
> pig1 (trunk)$ jar tvf pig-0.8.0-SNAPSHOT/pig-0.8.0-SNAPSHOT-core.jar |grep -v
> pig|wc -l
> 4819
> {noformat}
> How about restricting the core jar to just Pig classes?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1632) The core jar in the tarball contains the kitchen sink
[
https://issues.apache.org/jira/browse/PIG-1632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Collins updated PIG-1632:
-
Attachment: pig-1632-2.patch
Great. Patch attached. I verified the tarball produced by ant tar includes both
a core jar that is just pig core and a pig jar that has everything.
> The core jar in the tarball contains the kitchen sink
> --
>
> Key: PIG-1632
> URL: https://issues.apache.org/jira/browse/PIG-1632
> Project: Pig
> Issue Type: Bug
> Components: build
>Affects Versions: 0.8.0, 0.9.0
>Reporter: Eli Collins
> Fix For: site, 0.9.0
>
> Attachments: pig-1632-1.patch, pig-1632-2.patch
>
>
> The core jar in the tarball contains the kitchen sink, it's not the same core
> jar built by ant jar. This is problematic since other projects that want to
> depend on the pig core jar just want pig core, but
> pig-0.8.0-SNAPSHOT-core.jar in the tarball contains a bunch of other stuff
> (hadoop, com.google, commons, etc) that may conflict with the packages also
> on a user's classpath.
> {noformat}
> pig1 (trunk)$ jar tvf build/pig-0.8.0-SNAPSHOT-core.jar |grep -v pig|wc -l
> 12
> pig1 (trunk)$ tar xvzf build/pig-0.8.0-SNAPSHOT.tar.gz
> ...
> pig1 (trunk)$ jar tvf pig-0.8.0-SNAPSHOT/pig-0.8.0-SNAPSHOT-core.jar |grep -v
> pig|wc -l
> 4819
> {noformat}
> How about restricting the core jar to just Pig classes?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
