[
https://issues.apache.org/jira/browse/SPARK-26819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16760799#comment-16760799
]
M. Le Bihan commented on SPARK-26819:
-------------------------------------
You made me search a lot...
In this stacktrace part that ends with the bug, classical jacksson
"com.fasterxml.jackson.databind.introspect" is left to enter mostly methods of
"com.fasterxml.jackson.module.scala.introspect.BeanIntrospector" and from
"scala.collection" package :
{code:java}
java.lang.ArrayIndexOutOfBoundsException: 10582
{{ at
com.thoughtworks.paranamer.BytecodeReadingParanamer$ClassReader.accept(BytecodeReadingParanamer.java:563)}}
{{ at
com.thoughtworks.paranamer.BytecodeReadingParanamer$ClassReader.access$200(BytecodeReadingParanamer.java:338)}}
{{ at
com.thoughtworks.paranamer.BytecodeReadingParanamer.lookupParameterNames(BytecodeReadingParanamer.java:103)}}
{{ at
com.thoughtworks.paranamer.CachingParanamer.lookupParameterNames(CachingParanamer.java:90)}}
{{ at
com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.getCtorParams(BeanIntrospector.scala:44)}}
{{ at
com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$1(BeanIntrospector.scala:58)}}
{{ at
com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$1$adapted(BeanIntrospector.scala:58)}}
{{ at
scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:240)}}
{{ at scala.collection.Iterator.foreach(Iterator.scala:937)}}
{{ at scala.collection.Iterator.foreach$(Iterator.scala:937)}}
{{ at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)}}
{{ at scala.collection.IterableLike.foreach(IterableLike.scala:70)}}
{{ at scala.collection.IterableLike.foreach$(IterableLike.scala:69)}}
{{ at scala.collection.AbstractIterable.foreach(Iterable.scala:54)}}
{{ at scala.collection.TraversableLike.flatMap(TraversableLike.scala:240)}}
{{ at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:237)}}
{{ at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)}}
{{ at
com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.findConstructorParam$1(BeanIntrospector.scala:58)}}
{{ at
com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$19(BeanIntrospector.scala:176)}}
{{ at
scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:233)}}
{{ at
scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:32)}}
{{ at
scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:29)}}
{{ at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:194)}}
{{ at scala.collection.TraversableLike.map(TraversableLike.scala:233)}}
{{ at scala.collection.TraversableLike.map$(TraversableLike.scala:226)}}
{{ at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:194)}}
{{ at
com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$14(BeanIntrospector.scala:170)}}
{{ at
com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$14$adapted(BeanIntrospector.scala:169)}}
{{ at
scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:240)}}
{{ at scala.collection.immutable.List.foreach(List.scala:388)}}
{{ at scala.collection.TraversableLike.flatMap(TraversableLike.scala:240)}}
{{ at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:237)}}
{{ at scala.collection.immutable.List.flatMap(List.scala:351)}}
{{ at
com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.apply(BeanIntrospector.scala:169)}}
{{ at
com.fasterxml.jackson.module.scala.introspect.ScalaAnnotationIntrospector$._descriptorFor(ScalaAnnotationIntrospectorModule.scala:21)}}
{{ at
com.fasterxml.jackson.module.scala.introspect.ScalaAnnotationIntrospector$.fieldName(ScalaAnnotationIntrospectorModule.scala:29)}}
{{ at
com.fasterxml.jackson.module.scala.introspect.ScalaAnnotationIntrospector$.findImplicitPropertyName(ScalaAnnotationIntrospectorModule.scala:77)}}
{{ at
com.fasterxml.jackson.databind.introspect.AnnotationIntrospectorPair.findImplicitPropertyName(AnnotationIntrospectorPair.java:490)}}
{code}
_BeanInstrospector.java_ seems to be the same in
[2.9.7:2_11|https://jar-download.com/artifacts/com.fasterxml.jackson.module/jackson-module-scala_2.11/2.9.7/source-code/com/fasterxml/jackson/module/scala/introspect/BeanIntrospector.scala]
and
[2.9.7:2_12|https://jar-download.com/artifacts/com.fasterxml.jackson.module/jackson-module-scala_2.12/2.9.7/source-code/com/fasterxml/jackson/module/scala/introspect/BeanIntrospector.scala]
[com.thoughtworks.paranamer|https://mvnrepository.com/artifact/com.thoughtworks.paranamer/paranamer]
seems to be stable, in 2.8, and didn't change since year 2015, and that
version 2.8 was taken into account by Spark 2.3.0 in 2017 :
[https://jira.apache.org/jira/browse/SPARK-22128]
I'd rather put the source of the problem inside "scala" packages.
But I agree it's hard to be sure. Myself, I avoid scala as much I can to stay
in plain Java when I can : it's a source of unstability, in my mind.
> ArrayIndexOutOfBoundsException while loading a CSV to a Dataset with
> dependencies spark-core_2.12 and spark-sql_2.12 (with spark-core_2.11 and
> spark-sql_2.11 : working fine)
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-26819
> URL: https://issues.apache.org/jira/browse/SPARK-26819
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, SQL
> Affects Versions: 2.4.0
> Environment: Java 8, Windows 7.
> Reporter: M. Le Bihan
> Priority: Major
> Attachments: CompteResultatCSV.java, ComptesResultatsIT.java,
> comptes-communes-Entr'Allier.csv
>
>
> A simple CSV reading to a Dataset fails if Spark 2.4.0 is associated to
> dependencies spark-spark-core_2.12 and spark-sql_2.12, but works fine with
> spark-core_2.11 and spark-sql_2.11.
>
> With _2.12, I encounter this stacktrace :
>
> {{java.lang.ArrayIndexOutOfBoundsException: 10582}}
> {{ at
> com.thoughtworks.paranamer.BytecodeReadingParanamer$ClassReader.accept(BytecodeReadingParanamer.java:563)}}
> {{ at
> com.thoughtworks.paranamer.BytecodeReadingParanamer$ClassReader.access$200(BytecodeReadingParanamer.java:338)}}
> {{ at
> com.thoughtworks.paranamer.BytecodeReadingParanamer.lookupParameterNames(BytecodeReadingParanamer.java:103)}}
> {{ at
> com.thoughtworks.paranamer.CachingParanamer.lookupParameterNames(CachingParanamer.java:90)}}
> {{ at
> com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.getCtorParams(BeanIntrospector.scala:44)}}
> {{ at
> com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$1(BeanIntrospector.scala:58)}}
> {{ at
> com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$1$adapted(BeanIntrospector.scala:58)}}
> {{ at
> scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:240)}}
> {{ at scala.collection.Iterator.foreach(Iterator.scala:937)}}
> {{ at scala.collection.Iterator.foreach$(Iterator.scala:937)}}
> {{ at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)}}
> {{ at scala.collection.IterableLike.foreach(IterableLike.scala:70)}}
> {{ at scala.collection.IterableLike.foreach$(IterableLike.scala:69)}}
> {{ at scala.collection.AbstractIterable.foreach(Iterable.scala:54)}}
> {{ at scala.collection.TraversableLike.flatMap(TraversableLike.scala:240)}}
> {{ at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:237)}}
> {{ at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)}}
> {{ at
> com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.findConstructorParam$1(BeanIntrospector.scala:58)}}
> {{ at
> com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$19(BeanIntrospector.scala:176)}}
> {{ at
> scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:233)}}
> {{ at
> scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:32)}}
> {{ at
> scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:29)}}
> {{ at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:194)}}
> {{ at scala.collection.TraversableLike.map(TraversableLike.scala:233)}}
> {{ at scala.collection.TraversableLike.map$(TraversableLike.scala:226)}}
> {{ at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:194)}}
> {{ at
> com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$14(BeanIntrospector.scala:170)}}
> {{ at
> com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$14$adapted(BeanIntrospector.scala:169)}}
> {{ at
> scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:240)}}
> {{ at scala.collection.immutable.List.foreach(List.scala:388)}}
> {{ at scala.collection.TraversableLike.flatMap(TraversableLike.scala:240)}}
> {{ at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:237)}}
> {{ at scala.collection.immutable.List.flatMap(List.scala:351)}}
> {{ at
> com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.apply(BeanIntrospector.scala:169)}}
> {{ at
> com.fasterxml.jackson.module.scala.introspect.ScalaAnnotationIntrospector$._descriptorFor(ScalaAnnotationIntrospectorModule.scala:21)}}
> {{ at
> com.fasterxml.jackson.module.scala.introspect.ScalaAnnotationIntrospector$.fieldName(ScalaAnnotationIntrospectorModule.scala:29)}}
> {{ at
> com.fasterxml.jackson.module.scala.introspect.ScalaAnnotationIntrospector$.findImplicitPropertyName(ScalaAnnotationIntrospectorModule.scala:77)}}
> {{ at
> com.fasterxml.jackson.databind.introspect.AnnotationIntrospectorPair.findImplicitPropertyName(AnnotationIntrospectorPair.java:490)}}
> {{ at
> com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector._addFields(POJOPropertiesCollector.java:380)}}
> {{ at
> com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.collectAll(POJOPropertiesCollector.java:308)}}
> {{ at
> com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.getJsonValueAccessor(POJOPropertiesCollector.java:196)}}
> {{ at
> com.fasterxml.jackson.databind.introspect.BasicBeanDescription.findJsonValueAccessor(BasicBeanDescription.java:251)}}
> {{ at
> com.fasterxml.jackson.databind.ser.BasicSerializerFactory.findSerializerByAnnotations(BasicSerializerFactory.java:346)}}
> {{ at
> com.fasterxml.jackson.databind.ser.BeanSerializerFactory._createSerializer2(BeanSerializerFactory.java:216)}}
> {{ at
> com.fasterxml.jackson.databind.ser.BeanSerializerFactory.createSerializer(BeanSerializerFactory.java:165)}}
> {{ at
> com.fasterxml.jackson.databind.SerializerProvider._createUntypedSerializer(SerializerProvider.java:1388)}}
> {{ at
> com.fasterxml.jackson.databind.SerializerProvider._createAndCacheUntypedSerializer(SerializerProvider.java:1336)}}
> {{ at
> com.fasterxml.jackson.databind.SerializerProvider.findValueSerializer(SerializerProvider.java:510)}}
> {{ at
> com.fasterxml.jackson.databind.SerializerProvider.findTypedValueSerializer(SerializerProvider.java:713)}}
> {{ at
> com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:308)}}
> {{ at
> com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3905)}}
> {{ at
> com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:3219)}}
> {{ at
> org.apache.spark.rdd.RDDOperationScope.toJson(RDDOperationScope.scala:52)}}
> {{ at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:142)}}
> {{ at
> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)}}
> {{ at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)}}
> {{ at
> org.apache.spark.sql.execution.SparkPlan.getByteArrayRdd(SparkPlan.scala:247)}}
> {{ at
> org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:339)}}
> {{ at
> org.apache.spark.sql.execution.CollectLimitExec.executeCollect(limit.scala:38)}}
> {{ at org.apache.spark.sql.Dataset.collectFromPlan(Dataset.scala:3384)}}
> {{ at org.apache.spark.sql.Dataset.$anonfun$head$1(Dataset.scala:2545)}}
> {{ at org.apache.spark.sql.Dataset.$anonfun$withAction$2(Dataset.scala:3365)}}
> {{ at
> org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:78)}}
> {{ at
> org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)}}
> {{ at
> org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)}}
> {{ at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3365)}}
> {{ at org.apache.spark.sql.Dataset.head(Dataset.scala:2545)}}
> {{ at org.apache.spark.sql.Dataset.take(Dataset.scala:2759)}}
> {{ at
> org.apache.spark.sql.execution.datasources.csv.TextInputCSVDataSource$.infer(CSVDataSource.scala:232)}}
> {{ at
> org.apache.spark.sql.execution.datasources.csv.CSVDataSource.inferSchema(CSVDataSource.scala:68)}}
> {{ at
> org.apache.spark.sql.execution.datasources.csv.CSVFileFormat.inferSchema(CSVFileFormat.scala:63)}}
> {{ at
> org.apache.spark.sql.execution.datasources.DataSource.$anonfun$getOrInferFileFormatSchema$12(DataSource.scala:183)}}
> {{ at scala.Option.orElse(Option.scala:289)}}
> {{ at
> org.apache.spark.sql.execution.datasources.DataSource.getOrInferFileFormatSchema(DataSource.scala:180)}}
> {{ at
> org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:373)}}
> {{ at
> org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)}}
> {{ at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)}}
> {{ at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:178)}}
> {{ at
> test.fr.comptes.france.metier.application.spark.ComptesResultatsIT.datasetComptesResultats(ComptesResultatsIT.java:44)}}
> {{ at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)}}
> {{ at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}}
> {{ at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}}
> {{ at java.lang.reflect.Method.invoke(Method.java:498)}}
> {{ at
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:532)}}
> {{ at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:115)}}
> {{ at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:171)}}
> {{ at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:72)}}
> {{ at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:167)}}
> {{ at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:114)}}
> {{ at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:59)}}
> {{ at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$4(NodeTestTask.java:108)}}
> {{ at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:72)}}
> {{ at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:98)}}
> {{ at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:74)}}
> {{ at java.util.ArrayList.forEach(ArrayList.java:1249)}}
> {{ at
> org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)}}
> {{ at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$4(NodeTestTask.java:112)}}
> {{ at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:72)}}
> {{ at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:98)}}
> {{ at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:74)}}
> {{ at java.util.ArrayList.forEach(ArrayList.java:1249)}}
> {{ at
> org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)}}
> {{ at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$4(NodeTestTask.java:112)}}
> {{ at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:72)}}
> {{ at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:98)}}
> {{ at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:74)}}
> {{ at
> org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:32)}}
> {{ at
> org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)}}
> {{ at
> org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:51)}}
> {{ at
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:170)}}
> {{ at
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:154)}}
> {{ at
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:90)}}
> {{ at
> org.eclipse.jdt.internal.junit5.runner.JUnit5TestReference.run(JUnit5TestReference.java:89)}}
> {{ at
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:41)}}
> {{ at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:541)}}
> {{ at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:763)}}
> {{ at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:463)}}
> {{ at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:209)}}{{
> }}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
