> SET hive.exec.orc.memory.pool=1.0; Might be a bad idea in general, this causes more OOMs than less.
> SET mapred.map.child.java.opts=-Xmx2048M; > SET mapred.child.java.opts=-Xmx2048M; ... > Container >[pid=6278,containerID=container_e26_1460661845156_49295_01_000244] is >running beyond physical memory limits. Current usage: 2.2 GB of 2 GB >physical memory used; If you use 2Gb YARN containers, Xmx can only be -Xmx1600M or so - approx ~300Mb is used by the JVM for keeping track of loaded jars, created native code etc. In addition, NIO, Zlib, Snappy, x86_64 Crc32c etc use off-heap memory buffers, which come outside of the JVM heaps (Xmx). > I'd also appreciate any suggestions for other tuning I could do to fix >the memory management when using ORC. set hive.optimize.sort.dynamic.partition=true; is the one that prevents OOMs usually, but slows down 1-partition inserts (disabled by default). It is possible it might not kick in for your case due to one column being a constant (HIVE-12893). Cheers, Gopal
