Hi, I got a query that's producing about 3000 partitions which we load dynamically (On Hive 2.3.5). At the end of this query (running on M/R which runs fine) the M/R job is finished and we see this on the hive cli:
Loading data to table my_db.temp__v1_2019_12_03_182627 partition (c_date=null, c_hour=null, c_b=null, c_p=null) Time taken to load dynamic partitions: 540.025 seconds Time taken for adding to write entity : 0.329 seconds # # java.lang.OutOfMemoryError: Java heap space # -XX:OnOutOfMemoryError="kill -9 %p" # Executing /bin/sh -c "kill -9 19644"... os::fork_and_exec failed: Cannot allocate memory (12) MapReduce Jobs Launched: Stage-Stage-1: Map: 387 Reduce: 486 Cumulative CPU: 110521.05 sec HDFS Read: 533411354 HDFS Write: 262054898296 SUCCESS Stage-Stage-2: Map: 973 Reduce: 1009 Cumulative CPU: 48710.45 sec HDFS Read: 262126094987 HDFS Write: 70666472011 SUCCESS Total MapReduce CPU Time Spent: 1 days 20 hours 13 minutes 51 seconds 500 msec OK Where is this OutOfMemoryError coming from which heap space am I supposed to increase. We've tried increasing 'yarn.app.mapreduce.am.resource.mb' but that didn't seem to help. I know we should probably not have this many partitions but this is a one off would like this to just work. Thanks for any pointers, Patrick