Chao Sun created HIVE-15659: ------------------------------- Summary: StackOverflowError when ClassLoader.loadClass for Spark Key: HIVE-15659 URL: https://issues.apache.org/jira/browse/HIVE-15659 Project: Hive Issue Type: Bug Components: Spark Affects Versions: 2.2.0 Reporter: Chao Sun
Sometimes a query needs to process a large number of input files, which could cause the following error: {code} 17/01/15 09:31:52 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, hadoopworker1344-sjc1.prod.uber.internal): java.lang.StackOverflowError at java.util.concurrent.ConcurrentHashMap.putIfAbsent(ConcurrentHashMap.java:1535) at java.lang.ClassLoader.getClassLoadingLock(ClassLoader.java:463) at java.lang.ClassLoader.loadClass(ClassLoader.java:404) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) {code} The cause, I think, is that for each input file we may need to load additional jars to the class loader of the current thread. This accumulates with the number of input files. When adding a new class loader, the old class loader will be used as the parent of the new one. See [Utilities#getBaseWork|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L388] for more details. One possible solution is to detect duplicated jar paths before creating the new class loader. -- This message was sent by Atlassian JIRA (v6.3.4#6332)