[ https://issues.apache.org/jira/browse/PIG-3181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13604062#comment-13604062 ]
Johnny Zhang commented on PIG-3181: ----------------------------------- Hi, do you mean below right? {noformat} a = load '/input' as (f1, f2); a = group a by f1; logs = foreach a { generate group, a.f2; } store logs into '/output/' using org.apache.pig.piggybank.storage.MultiStorage('/output/', '0'); {noformat} can you share how large is your /input file? so that I can try to reproduce it. > MultiStorage - java.lang.OutOfMemoryError: Java heap space > ---------------------------------------------------------- > > Key: PIG-3181 > URL: https://issues.apache.org/jira/browse/PIG-3181 > Project: Pig > Issue Type: Bug > Components: piggybank > Affects Versions: 0.10.0 > Reporter: Fabian Alenius > > Hi, I have a script that looks like this: > a = load '/input' as (f1, f2); > > a = group a by f1; > > > a = foreach logs { > > > generate group, a.f2; > > > } > store logs into '/output/' using > org.apache.pig.piggybank.storage.MultiStorage('/output/', '0'); > But for some reason it fails with: > FATAL org.apache.hadoop.mapred.Child: Error running child : > java.lang.OutOfMemoryError: Java heap space > at java.util.Arrays.copyOf(Arrays.java:2786) > at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94) > at java.io.OutputStream.write(OutputStream.java:58) > at org.apache.pig.impl.util.StorageUtil.putField(StorageUtil.java:145) > at org.apache.pig.impl.util.StorageUtil.putField(StorageUtil.java:176) > at org.apache.pig.impl.util.StorageUtil.putField(StorageUtil.java:194) > at > org.apache.pig.piggybank.storage.MultiStorage$MultiStorageOutputFormat$1.write(MultiStorage.java:208) > at > org.apache.pig.piggybank.storage.MultiStorage$MultiStorageOutputFormat$1.write(MultiStorage.java:187) > at > org.apache.pig.piggybank.storage.MultiStorage.putNext(MultiStorage.java:138) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:139) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:98) > at > org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:537) > at > org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:88) > at > org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.write(WrappedReducer.java:99) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.runPipeline(PigGenericMapReduce.java:463) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.processOnePackageOutput(PigGenericMapReduce.java:428) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:408) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:262) > at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:164) > at > org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:595) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:433) > at org.apache.hadoop.mapred.Child$4.run(Child.java:268) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) > at org.apache.hadoop.mapred.Child.main(Child.java:262) > *stderr logs* > java.lang.RuntimeException: InternalCachedBag.spill() should not be called > at > org.apache.pig.data.InternalCachedBag.spill(InternalCachedBag.java:159) > at > org.apache.pig.impl.util.SpillableMemoryManager.handleNotification(SpillableMemoryManager.java:243) > at > sun.management.NotificationEmitterSupport.sendNotification(NotificationEmitterSupport.java:138) > at sun.management.MemoryImpl.createNotification(MemoryImpl.java:171) > at > sun.management.MemoryPoolImpl$PoolSensor.triggerAction(MemoryPoolImpl.java:272) > at sun.management.Sensor.trigger(Sensor.java:120) > Exception in thread "Low Memory Detector" java.lang.InternalError: Error in > invoking listener > at > sun.management.NotificationEmitterSupport.sendNotification(NotificationEmitterSupport.java:141) > at sun.management.MemoryImpl.createNotification(MemoryImpl.java:171) > at > sun.management.MemoryPoolImpl$PoolSensor.triggerAction(MemoryPoolImpl.java:272) > at sun.management.Sensor.trigger(Sensor.java:120) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira