[ https://issues.apache.org/jira/browse/HIVE-11414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HIVE-11414: ------------------------------ Description: MapTask hit OOM in the following situation in our production environment: * src: 2048 partitions, each with 1 file of about 2MB using RCFile format * query: INSERT OVERWRITE TABLE tgt SELECT * FROM src * Hadoop version: Both on CDH 4.7 using MR1 and CDH 5.4.1 using YARN. * MapTask memory Xmx: 1.5GB By analyzing the heap dump using jhat, we realized that the problem is: * One single mapper is processing many partitions (because of CombineHiveInputFormat) * Each input path (equivalent to partition here) will construct its own SerDe * Each SerDe will do its own caching of deserialized object (and try to reuse it), but will never release it (in this case, the serde2.columnar.ColumnarSerDeBase has a field cachedLazyStruct which can take a lot of space - pretty much the last N rows of a file where N is the number of rows in a columnar block). * This problem may exist in other SerDe as well, but columnar file format are affected the most because they need bigger cache for the last N rows instead of 1 row. Proposed solution: * Remove cachedLazyStruct in serde2.columnar.ColumnarSerDeBase. The cost saving of not recreating a single object is too small compared to processing N rows. Alternative solutions: * We can also free up the whole SerDe after processing a block/file. The problem with that is that the input splits may contain multiple blocks/files that maps to the same SerDe, and recreating a SerDe is a much bigger change to the code. * We can also move the SerDe creation/free-up to the place when input file changes. But that requires a much bigger change to the code. * We can also add a "cleanup()" method to SerDe interface that release the cached object, but that change is not backward compatible with many SerDes that people have wrote. * We can make cachedLazyStruct in serde2.columnar.ColumnarSerDeBase a weakly referenced object, but that feels like an overkill. was: MapTask hit OOM in the following situation in our production environment: * src: 2048 partitions, each with 1 file of about 2MB using RCFile format * query: INSERT OVERWRITE TABLE tgt SELECT * FROM src * Hadoop version: Both on CDH 4.7 using MR1 and CDH 5.4.1 using YARN. * MapTask memory Xmx: 1.5GB By analyzing the heap dump using jhat, we realized that the problem is: * One single mapper is processing many partitions (because of CombineHiveInputFormat) * Each input path (equivalent to partition here) will construct its own SerDe * Each SerDe will do its own caching of deserialized object (and try to reuse it), but will never release it (in this case, the serde2.columnar.ColumnarSerDeBase has a field cachedLazyStruct which can take a lot of space - pretty much the last N rows of a file where N is the number of rows in a columnar block). * This problem may exist in other SerDe as well, but columnar file format are affected the most because they need bigger cache for the last N rows instead of 1 row. Proposed solution: * Remove cachedLazyStruct in serde2.columnar.ColumnarSerDeBase. The cost saving of not recreating a single object is too small compared to processing N rows. Alternative solutions: * We can also free up the whole SerDe after processing a block/file. The problem with that is that the input splits may contain multiple blocks/files that maps to the same SerDe, and recreating a SerDe is just more work. * We can also move the SerDe creation/free-up to the place when input file changes. But that requires a much bigger change to the code. * We can also add a "cleanup()" method to SerDe interface that release the cached object, but that change is not backward compatible with many SerDes that people have wrote. * We can make cachedLazyStruct in serde2.columnar.ColumnarSerDeBase a weakly referenced object, but that feels like an overkill. > Fix OOM in MapTask with many input partitions with RCFile > --------------------------------------------------------- > > Key: HIVE-11414 > URL: https://issues.apache.org/jira/browse/HIVE-11414 > Project: Hive > Issue Type: Improvement > Components: File Formats, Serializers/Deserializers > Affects Versions: 0.11.0, 0.12.0, 0.14.0, 0.13.1, 1.2.0 > Reporter: Zheng Shao > Priority: Minor > > MapTask hit OOM in the following situation in our production environment: > * src: 2048 partitions, each with 1 file of about 2MB using RCFile format > * query: INSERT OVERWRITE TABLE tgt SELECT * FROM src > * Hadoop version: Both on CDH 4.7 using MR1 and CDH 5.4.1 using YARN. > * MapTask memory Xmx: 1.5GB > By analyzing the heap dump using jhat, we realized that the problem is: > * One single mapper is processing many partitions (because of > CombineHiveInputFormat) > * Each input path (equivalent to partition here) will construct its own SerDe > * Each SerDe will do its own caching of deserialized object (and try to reuse > it), but will never release it (in this case, the > serde2.columnar.ColumnarSerDeBase has a field cachedLazyStruct which can take > a lot of space - pretty much the last N rows of a file where N is the number > of rows in a columnar block). > * This problem may exist in other SerDe as well, but columnar file format are > affected the most because they need bigger cache for the last N rows instead > of 1 row. > Proposed solution: > * Remove cachedLazyStruct in serde2.columnar.ColumnarSerDeBase. The cost > saving of not recreating a single object is too small compared to processing > N rows. > Alternative solutions: > * We can also free up the whole SerDe after processing a block/file. The > problem with that is that the input splits may contain multiple blocks/files > that maps to the same SerDe, and recreating a SerDe is a much bigger change > to the code. > * We can also move the SerDe creation/free-up to the place when input file > changes. But that requires a much bigger change to the code. > * We can also add a "cleanup()" method to SerDe interface that release the > cached object, but that change is not backward compatible with many SerDes > that people have wrote. > * We can make cachedLazyStruct in serde2.columnar.ColumnarSerDeBase a weakly > referenced object, but that feels like an overkill. -- This message was sent by Atlassian JIRA (v6.3.4#6332)