[jira] [Comment Edited] (SPARK-22373) Intermittent NullPointerException in org.codehaus.janino.IClass.isAssignableFrom
[ https://issues.apache.org/jira/browse/SPARK-22373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269354#comment-16269354 ] Leigh Klotz edited comment on SPARK-22373 at 11/28/17 7:51 PM: This happens to me regularly enough using 2.1.1.20, Avro, and more than one executor core that I have abandoned use of multiple cores with Avro. I've attempted to make a 100% reproducible test case but failed, so I'm reporting these factors here. 1. set --conf spark.executor.cores=2 (or any higher number) 2. reading in a certain large Avro file 3. spark 2.1.1.20 4. spark.read.avro(fn).cache.count or other action involving writes; just count doesn't do it. 5. The Avro file contains a key of type Map[String->Array[Byte]], though the values can all be empty arrays. The cardinality of they keyspace is high and the number of keys per map is tens to hundreds. 6. Multiple partitions are necessary to trigger the error. 7. Before the stack trace reported above, I see "ERROR CodeGenerator: failed to compile: java.lang.NullPointerException" followed by a dump of generated Java code. The nodes that fail have "INFO CodeGenerator: Code generated in ##.# ms" messages and my theory is that the code generator being used here has a thread safety issue. was (Author: leighklotz): This happens to me regularly enough using 2.1.1.20, Avro, and more than one executor core that I have abandoned use of multiple cores with Avro. I've attempted to make a 100% reproducible test case but failed, so I'm reporting these factors here. 1. set --conf spark.executor.cores=2 (or any higher number) 2. reading in a certain large Avro file 3. spark 2.1.1.20 4. spark.read.avro(fn).cache.count or other action involving writes; just count doesn't do it. 5. The Avro file contains a key of type Map[String->Array[Byte]], though the values cab all be empty arrays. The cardinality of they keyspace is high and the number of keys per map is tens to hundreds. 6. Multiple partitions are necessary to trigger the error. 7. Before the stack trace reported above, I see "ERROR CodeGenerator: failed to compile: java.lang.NullPointerException" followed by a dump of generated Java code. The nodes that fail have "INFO CodeGenerator: Code generated in ##.# ms" messages and my theory is that the code generator being used here has a thread safety issue. > Intermittent NullPointerException in > org.codehaus.janino.IClass.isAssignableFrom > > > Key: SPARK-22373 > URL: https://issues.apache.org/jira/browse/SPARK-22373 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.1.1 > Environment: Hortonworks distribution: HDP 2.6.2.0-205 , > /usr/hdp/current/spark2-client/jars/spark-core_2.11-2.1.1.2.6.2.0-205.jar >Reporter: Dan Meany >Priority: Minor > > Very occasional and retry works. > Full stack: > 17/10/27 21:06:15 ERROR Executor: Exception in task 29.0 in stage 12.0 (TID > 758) > java.lang.NullPointerException > at org.codehaus.janino.IClass.isAssignableFrom(IClass.java:569) > at > org.codehaus.janino.UnitCompiler.isWideningReferenceConvertible(UnitCompiler.java:10347) > at > org.codehaus.janino.UnitCompiler.isMethodInvocationConvertible(UnitCompiler.java:8636) > at > org.codehaus.janino.UnitCompiler.findMostSpecificIInvocable(UnitCompiler.java:8427) > at > org.codehaus.janino.UnitCompiler.findMostSpecificIInvocable(UnitCompiler.java:8285) > at org.codehaus.janino.UnitCompiler.findIMethod(UnitCompiler.java:8169) > at org.codehaus.janino.UnitCompiler.findIMethod(UnitCompiler.java:8071) > at org.codehaus.janino.UnitCompiler.compileGet2(UnitCompiler.java:4421) > at org.codehaus.janino.UnitCompiler.access$7500(UnitCompiler.java:206) > at > org.codehaus.janino.UnitCompiler$12.visitMethodInvocation(UnitCompiler.java:3774) > at > org.codehaus.janino.UnitCompiler$12.visitMethodInvocation(UnitCompiler.java:3762) > at org.codehaus.janino.Java$MethodInvocation.accept(Java.java:4328) > at org.codehaus.janino.UnitCompiler.compileGet(UnitCompiler.java:3762) > at > org.codehaus.janino.UnitCompiler.compileGetValue(UnitCompiler.java:4933) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:3180) > at org.codehaus.janino.UnitCompiler.access$5000(UnitCompiler.java:206) > at > org.codehaus.janino.UnitCompiler$9.visitMethodInvocation(UnitCompiler.java:3151) > at > org.codehaus.janino.UnitCompiler$9.visitMethodInvocation(UnitCompiler.java:3139) > at org.codehaus.janino.Java$MethodInvocation.accept(Java.java:4328) > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:3139) > at org.co
[jira] [Comment Edited] (SPARK-22373) Intermittent NullPointerException in org.codehaus.janino.IClass.isAssignableFrom
[ https://issues.apache.org/jira/browse/SPARK-22373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269354#comment-16269354 ] Leigh Klotz edited comment on SPARK-22373 at 11/28/17 7:51 PM: This happens to me regularly enough using 2.1.1.20, Avro, and more than one executor core that I have abandoned use of multiple cores with Avro. I've attempted to make a 100% reproducible test case but failed, so I'm reporting these factors here. 1. set --conf spark.executor.cores=2 (or any higher number) 2. reading in a certain large Avro file 3. spark 2.1.1.20 4. spark.read.avro(fn).cache.count or other action involving writes; just count doesn't do it. 5. The Avro file contains a key of type Map[String->Array[Byte]], though the values cab all be empty arrays. The cardinality of they keyspace is high and the number of keys per map is tens to hundreds. 6. Multiple partitions are necessary to trigger the error. 7. Before the stack trace reported above, I see "ERROR CodeGenerator: failed to compile: java.lang.NullPointerException" followed by a dump of generated Java code. The nodes that fail have "INFO CodeGenerator: Code generated in ##.# ms" messages and my theory is that the code generator being used here has a thread safety issue. was (Author: leighklotz): This happens to me regularly enough using 2.1.1.20, Avro, and more than one executor core that I have abandoned use of multiple cores with Avro. I've attempted to make a 100% reproducible test case but failed, so I'm reporting these factors here. 1. set --conf spark.executor.cores=2 (or any higher number) 2. reading in a certain large Avro file 3. spark 2.1.1.20 4. spark.read.avro(fn).cache.count 5. The Avro file contains a key of type Map[String->Array[Byte]], though the values cab all be empty arrays. The cardinality of they keyspace is high and the number of keys per map is tens to hundreds. 6. Multiple partitions are necessary to trigger the error. 7. Before the stack trace reported above, I see "ERROR CodeGenerator: failed to compile: java.lang.NullPointerException" followed by a dump of generated Java code. The nodes that fail have "INFO CodeGenerator: Code generated in ##.# ms" messages and my theory is that the code generator being used here has a thread safety issue. > Intermittent NullPointerException in > org.codehaus.janino.IClass.isAssignableFrom > > > Key: SPARK-22373 > URL: https://issues.apache.org/jira/browse/SPARK-22373 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.1.1 > Environment: Hortonworks distribution: HDP 2.6.2.0-205 , > /usr/hdp/current/spark2-client/jars/spark-core_2.11-2.1.1.2.6.2.0-205.jar >Reporter: Dan Meany >Priority: Minor > > Very occasional and retry works. > Full stack: > 17/10/27 21:06:15 ERROR Executor: Exception in task 29.0 in stage 12.0 (TID > 758) > java.lang.NullPointerException > at org.codehaus.janino.IClass.isAssignableFrom(IClass.java:569) > at > org.codehaus.janino.UnitCompiler.isWideningReferenceConvertible(UnitCompiler.java:10347) > at > org.codehaus.janino.UnitCompiler.isMethodInvocationConvertible(UnitCompiler.java:8636) > at > org.codehaus.janino.UnitCompiler.findMostSpecificIInvocable(UnitCompiler.java:8427) > at > org.codehaus.janino.UnitCompiler.findMostSpecificIInvocable(UnitCompiler.java:8285) > at org.codehaus.janino.UnitCompiler.findIMethod(UnitCompiler.java:8169) > at org.codehaus.janino.UnitCompiler.findIMethod(UnitCompiler.java:8071) > at org.codehaus.janino.UnitCompiler.compileGet2(UnitCompiler.java:4421) > at org.codehaus.janino.UnitCompiler.access$7500(UnitCompiler.java:206) > at > org.codehaus.janino.UnitCompiler$12.visitMethodInvocation(UnitCompiler.java:3774) > at > org.codehaus.janino.UnitCompiler$12.visitMethodInvocation(UnitCompiler.java:3762) > at org.codehaus.janino.Java$MethodInvocation.accept(Java.java:4328) > at org.codehaus.janino.UnitCompiler.compileGet(UnitCompiler.java:3762) > at > org.codehaus.janino.UnitCompiler.compileGetValue(UnitCompiler.java:4933) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:3180) > at org.codehaus.janino.UnitCompiler.access$5000(UnitCompiler.java:206) > at > org.codehaus.janino.UnitCompiler$9.visitMethodInvocation(UnitCompiler.java:3151) > at > org.codehaus.janino.UnitCompiler$9.visitMethodInvocation(UnitCompiler.java:3139) > at org.codehaus.janino.Java$MethodInvocation.accept(Java.java:4328) > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:3139) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:2112)
[jira] [Comment Edited] (SPARK-22373) Intermittent NullPointerException in org.codehaus.janino.IClass.isAssignableFrom
[ https://issues.apache.org/jira/browse/SPARK-22373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269354#comment-16269354 ] Leigh Klotz edited comment on SPARK-22373 at 11/28/17 7:50 PM: This happens to me regularly enough using 2.1.1.20, Avro, and more than one executor core that I have abandoned use of multiple cores with Avro. I've attempted to make a 100% reproducible test case but failed, so I'm reporting these factors here. 1. set --conf spark.executor.cores=2 (or any higher number) 2. reading in a certain large Avro file 3. spark 2.1.1.20 4. spark.read.avro(fn).cache.count 5. The Avro file contains a key of type Map[String->Array[Byte]], though the values cab all be empty arrays. The cardinality of they keyspace is high and the number of keys per map is tens to hundreds. 6. Multiple partitions are necessary to trigger the error. 7. Before the stack trace reported above, I see "ERROR CodeGenerator: failed to compile: java.lang.NullPointerException" followed by a dump of generated Java code. The nodes that fail have "INFO CodeGenerator: Code generated in ##.# ms" messages and my theory is that the code generator being used here has a thread safety issue. was (Author: leighklotz): This happens to me regularly enough using 2.1.1.20, Avro, and more than one executor core that I have abandoned use of multiple cores with Avro. I've attempted to make a 100% reproducible test case but failed, so I'm reporting these factors here. 1. set --conf spark.executor.cores=2 (or any higher number) 2. reading in a certain large Avro file with 3. spark 2.1.1.20 4. spark.read.avro(fn).cache.count 5. The Avro file contains a key of type Map[String->Array[Byte]], though the values cab all be empty arrays. The cardinality of they keyspace is high and the number of keys per map is tens to hundreds. 6. Multiple partitions are necessary to trigger the error. 7. Before the stack trace reported above, I see "ERROR CodeGenerator: failed to compile: java.lang.NullPointerException" followed by a dump of generated Java code. The nodes that fail have "INFO CodeGenerator: Code generated in ##.# ms" messages and my theory is that the code generator being used here has a thread safety issue. > Intermittent NullPointerException in > org.codehaus.janino.IClass.isAssignableFrom > > > Key: SPARK-22373 > URL: https://issues.apache.org/jira/browse/SPARK-22373 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.1.1 > Environment: Hortonworks distribution: HDP 2.6.2.0-205 , > /usr/hdp/current/spark2-client/jars/spark-core_2.11-2.1.1.2.6.2.0-205.jar >Reporter: Dan Meany >Priority: Minor > > Very occasional and retry works. > Full stack: > 17/10/27 21:06:15 ERROR Executor: Exception in task 29.0 in stage 12.0 (TID > 758) > java.lang.NullPointerException > at org.codehaus.janino.IClass.isAssignableFrom(IClass.java:569) > at > org.codehaus.janino.UnitCompiler.isWideningReferenceConvertible(UnitCompiler.java:10347) > at > org.codehaus.janino.UnitCompiler.isMethodInvocationConvertible(UnitCompiler.java:8636) > at > org.codehaus.janino.UnitCompiler.findMostSpecificIInvocable(UnitCompiler.java:8427) > at > org.codehaus.janino.UnitCompiler.findMostSpecificIInvocable(UnitCompiler.java:8285) > at org.codehaus.janino.UnitCompiler.findIMethod(UnitCompiler.java:8169) > at org.codehaus.janino.UnitCompiler.findIMethod(UnitCompiler.java:8071) > at org.codehaus.janino.UnitCompiler.compileGet2(UnitCompiler.java:4421) > at org.codehaus.janino.UnitCompiler.access$7500(UnitCompiler.java:206) > at > org.codehaus.janino.UnitCompiler$12.visitMethodInvocation(UnitCompiler.java:3774) > at > org.codehaus.janino.UnitCompiler$12.visitMethodInvocation(UnitCompiler.java:3762) > at org.codehaus.janino.Java$MethodInvocation.accept(Java.java:4328) > at org.codehaus.janino.UnitCompiler.compileGet(UnitCompiler.java:3762) > at > org.codehaus.janino.UnitCompiler.compileGetValue(UnitCompiler.java:4933) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:3180) > at org.codehaus.janino.UnitCompiler.access$5000(UnitCompiler.java:206) > at > org.codehaus.janino.UnitCompiler$9.visitMethodInvocation(UnitCompiler.java:3151) > at > org.codehaus.janino.UnitCompiler$9.visitMethodInvocation(UnitCompiler.java:3139) > at org.codehaus.janino.Java$MethodInvocation.accept(Java.java:4328) > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:3139) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:2112) > at org.codehaus.janino.UnitCompiler.access$1700