I have a process which is loading data into hive hourly. Loading data
hourly isn't a problem however when I load historical data say 24-48
hours I receive the below error msg. In googling I've come across some
suggestions that jvm memory needs to be increased. Are there any other
options or is that pretty much it?
I appreciate any help with this one. To get around the problem I've
loaded fewer historical logs which works great but isn't what I had in
mind :-)
Thanks a bunch!
---
2011-04-05 15:33:38,400 INFO
org.apache.hadoop.hive.ql.exec.SelectOperator: 7 finished. closing...
2011-04-05 15:33:38,400 INFO
org.apache.hadoop.hive.ql.exec.SelectOperator: 7 forwarded 0 rows
2011-04-05 15:33:38,400 INFO
org.apache.hadoop.hive.ql.exec.GroupByOperator: 8 finished. closing...
2011-04-05 15:33:38,400 INFO
org.apache.hadoop.hive.ql.exec.GroupByOperator: 8 forwarded 0 rows
2011-04-05 15:33:38,401 WARN
org.apache.hadoop.hive.ql.exec.GroupByOperator: Begin Hash Table flush
at close: size = 0
2011-04-05 15:33:38,401 INFO
org.apache.hadoop.hive.ql.exec.FileSinkOperator: 9 finished. closing...
2011-04-05 15:33:38,401 INFO
org.apache.hadoop.hive.ql.exec.FileSinkOperator: 9 forwarded 0 rows
2011-04-05 15:33:38,401 INFO
org.apache.hadoop.hive.ql.exec.FileSinkOperator: Final Path: FS
hdfs://namenoden1:9000
/tmp/hive-etl/hive_2011-04-05_15-25-32_126_7118636463039801851/_tmp.-mr-10004/000049_0
2011-04-05 15:33:38,401 INFO
org.apache.hadoop.hive.ql.exec.FileSinkOperator: Writing to temp file: FS
hdfs://namenoden1:9000/tmp/hive-etl/hive_2011-04-05_15-25-32_126_7118636463039801851/_tmp.-mr-10004/_tmp.000049_0
2011-04-05 15:33:38,401 INFO
org.apache.hadoop.hive.ql.exec.FileSinkOperator: New Final Path: FS
hdfs://namenoden1:9000
/tmp/hive-etl/hive_2011-04-05_15-25-32_126_7118636463039801851/_tmp.-mr-10004/000049_0
2011-04-05 15:33:38,448 INFO org.apache.hadoop.io.compress.CodecPool:
Got brand-new compressor
2011-04-05 15:33:38,495 INFO
org.apache.hadoop.hive.ql.exec.GroupByOperator: 8 Close done
2011-04-05 15:33:38,495 INFO
org.apache.hadoop.hive.ql.exec.SelectOperator: 7 Close done
2011-04-05 15:33:38,495 INFO
org.apache.hadoop.hive.ql.exec.FilterOperator: 6 Close done
2011-04-05 15:33:38,495 INFO
org.apache.hadoop.hive.ql.exec.SelectOperator: 1 Close done
2011-04-05 15:33:38,495 INFO
org.apache.hadoop.hive.ql.exec.TableScanOperator: 0 Close done
2011-04-05 15:33:38,495 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
13 Close done
2011-04-05 15:33:38,495 INFO ExecMapper: ExecMapper: processed 188614
rows: used memory = 720968432
2011-04-05 15:33:38,501 FATAL org.apache.hadoop.mapred.Child: Error
running child : java.lang.OutOfMemoryError: Java
heap space
at org.apache.hadoop.io.Text.setCapacity(Text.java:240)
at org.apache.hadoop.io.Text.append(Text.java:216)
at
org.apache.hadoop.util.LineReader.readLine(LineReader.java:159)
at
org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:136)
at
org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:40)
at
org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:66)
at
org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:32)
at
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:67)
at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:202)
at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:186)
at
org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:383)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:317)
at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
at java.security.AccessController.doPrivileged(Native
Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
at org.apache.hadoop.mapred.Child.main(Child.java:211)
