[ https://issues.apache.org/jira/browse/SPARK-26819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16763419#comment-16763419 ]
Dongjoon Hyun edited comment on SPARK-26819 at 2/8/19 8:48 AM: --------------------------------------------------------------- Hi, [~mlebihan]. Yes. This is a known issue with Scala-2.12 artifacts and fixed at SPARK-22128 in Spark itself and SPARK-26583 for Spark-dependent application for Apache Spark 2.4.1 and 3.0.0. So, you are suffering due to SPARK-26583. One workaround for your application is adding `paranamer-2.8.jar` dependency. was (Author: dongjoon): Hi, [~mlebihan]. Yes. This is a known issue with Scala-2.12 artifacts and fixed at SPARK-22128 in Spark itself and SPARK-26583 for Spark-dependent application for Apache Spark 2.4.1 and 3.0.0. One workaround for your application is adding `paranamer-2.8.jar` dependency. > ArrayIndexOutOfBoundsException while loading a CSV to a Dataset with > dependencies spark-core_2.12 and spark-sql_2.12 (with spark-core_2.11 and > spark-sql_2.11 : working fine) > ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > Key: SPARK-26819 > URL: https://issues.apache.org/jira/browse/SPARK-26819 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL > Affects Versions: 2.4.0 > Environment: Java 8, Windows 7. > Reporter: M. Le Bihan > Priority: Major > Attachments: CompteResultatCSV.java, ComptesResultatsIT.java, > comptes-communes-Entr'Allier.csv > > > A simple CSV reading to a Dataset fails if Spark 2.4.0 is associated to > dependencies spark-spark-core_2.12 and spark-sql_2.12, but works fine with > spark-core_2.11 and spark-sql_2.11. > > With _2.12, I encounter this stacktrace : > > {{java.lang.ArrayIndexOutOfBoundsException: 10582}} > {{ at > com.thoughtworks.paranamer.BytecodeReadingParanamer$ClassReader.accept(BytecodeReadingParanamer.java:563)}} > {{ at > com.thoughtworks.paranamer.BytecodeReadingParanamer$ClassReader.access$200(BytecodeReadingParanamer.java:338)}} > {{ at > com.thoughtworks.paranamer.BytecodeReadingParanamer.lookupParameterNames(BytecodeReadingParanamer.java:103)}} > {{ at > com.thoughtworks.paranamer.CachingParanamer.lookupParameterNames(CachingParanamer.java:90)}} > {{ at > com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.getCtorParams(BeanIntrospector.scala:44)}} > {{ at > com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$1(BeanIntrospector.scala:58)}} > {{ at > com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$1$adapted(BeanIntrospector.scala:58)}} > {{ at > scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:240)}} > {{ at scala.collection.Iterator.foreach(Iterator.scala:937)}} > {{ at scala.collection.Iterator.foreach$(Iterator.scala:937)}} > {{ at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)}} > {{ at scala.collection.IterableLike.foreach(IterableLike.scala:70)}} > {{ at scala.collection.IterableLike.foreach$(IterableLike.scala:69)}} > {{ at scala.collection.AbstractIterable.foreach(Iterable.scala:54)}} > {{ at scala.collection.TraversableLike.flatMap(TraversableLike.scala:240)}} > {{ at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:237)}} > {{ at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)}} > {{ at > com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.findConstructorParam$1(BeanIntrospector.scala:58)}} > {{ at > com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$19(BeanIntrospector.scala:176)}} > {{ at > scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:233)}} > {{ at > scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:32)}} > {{ at > scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:29)}} > {{ at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:194)}} > {{ at scala.collection.TraversableLike.map(TraversableLike.scala:233)}} > {{ at scala.collection.TraversableLike.map$(TraversableLike.scala:226)}} > {{ at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:194)}} > {{ at > com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$14(BeanIntrospector.scala:170)}} > {{ at > com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$14$adapted(BeanIntrospector.scala:169)}} > {{ at > scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:240)}} > {{ at scala.collection.immutable.List.foreach(List.scala:388)}} > {{ at scala.collection.TraversableLike.flatMap(TraversableLike.scala:240)}} > {{ at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:237)}} > {{ at scala.collection.immutable.List.flatMap(List.scala:351)}} > {{ at > com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.apply(BeanIntrospector.scala:169)}} > {{ at > com.fasterxml.jackson.module.scala.introspect.ScalaAnnotationIntrospector$._descriptorFor(ScalaAnnotationIntrospectorModule.scala:21)}} > {{ at > com.fasterxml.jackson.module.scala.introspect.ScalaAnnotationIntrospector$.fieldName(ScalaAnnotationIntrospectorModule.scala:29)}} > {{ at > com.fasterxml.jackson.module.scala.introspect.ScalaAnnotationIntrospector$.findImplicitPropertyName(ScalaAnnotationIntrospectorModule.scala:77)}} > {{ at > com.fasterxml.jackson.databind.introspect.AnnotationIntrospectorPair.findImplicitPropertyName(AnnotationIntrospectorPair.java:490)}} > {{ at > com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector._addFields(POJOPropertiesCollector.java:380)}} > {{ at > com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.collectAll(POJOPropertiesCollector.java:308)}} > {{ at > com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.getJsonValueAccessor(POJOPropertiesCollector.java:196)}} > {{ at > com.fasterxml.jackson.databind.introspect.BasicBeanDescription.findJsonValueAccessor(BasicBeanDescription.java:251)}} > {{ at > com.fasterxml.jackson.databind.ser.BasicSerializerFactory.findSerializerByAnnotations(BasicSerializerFactory.java:346)}} > {{ at > com.fasterxml.jackson.databind.ser.BeanSerializerFactory._createSerializer2(BeanSerializerFactory.java:216)}} > {{ at > com.fasterxml.jackson.databind.ser.BeanSerializerFactory.createSerializer(BeanSerializerFactory.java:165)}} > {{ at > com.fasterxml.jackson.databind.SerializerProvider._createUntypedSerializer(SerializerProvider.java:1388)}} > {{ at > com.fasterxml.jackson.databind.SerializerProvider._createAndCacheUntypedSerializer(SerializerProvider.java:1336)}} > {{ at > com.fasterxml.jackson.databind.SerializerProvider.findValueSerializer(SerializerProvider.java:510)}} > {{ at > com.fasterxml.jackson.databind.SerializerProvider.findTypedValueSerializer(SerializerProvider.java:713)}} > {{ at > com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:308)}} > {{ at > com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3905)}} > {{ at > com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:3219)}} > {{ at > org.apache.spark.rdd.RDDOperationScope.toJson(RDDOperationScope.scala:52)}} > {{ at > org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:142)}} > {{ at > org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)}} > {{ at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)}} > {{ at > org.apache.spark.sql.execution.SparkPlan.getByteArrayRdd(SparkPlan.scala:247)}} > {{ at > org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:339)}} > {{ at > org.apache.spark.sql.execution.CollectLimitExec.executeCollect(limit.scala:38)}} > {{ at org.apache.spark.sql.Dataset.collectFromPlan(Dataset.scala:3384)}} > {{ at org.apache.spark.sql.Dataset.$anonfun$head$1(Dataset.scala:2545)}} > {{ at org.apache.spark.sql.Dataset.$anonfun$withAction$2(Dataset.scala:3365)}} > {{ at > org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:78)}} > {{ at > org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)}} > {{ at > org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)}} > {{ at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3365)}} > {{ at org.apache.spark.sql.Dataset.head(Dataset.scala:2545)}} > {{ at org.apache.spark.sql.Dataset.take(Dataset.scala:2759)}} > {{ at > org.apache.spark.sql.execution.datasources.csv.TextInputCSVDataSource$.infer(CSVDataSource.scala:232)}} > {{ at > org.apache.spark.sql.execution.datasources.csv.CSVDataSource.inferSchema(CSVDataSource.scala:68)}} > {{ at > org.apache.spark.sql.execution.datasources.csv.CSVFileFormat.inferSchema(CSVFileFormat.scala:63)}} > {{ at > org.apache.spark.sql.execution.datasources.DataSource.$anonfun$getOrInferFileFormatSchema$12(DataSource.scala:183)}} > {{ at scala.Option.orElse(Option.scala:289)}} > {{ at > org.apache.spark.sql.execution.datasources.DataSource.getOrInferFileFormatSchema(DataSource.scala:180)}} > {{ at > org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:373)}} > {{ at > org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)}} > {{ at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)}} > {{ at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:178)}} > {{ at > test.fr.comptes.france.metier.application.spark.ComptesResultatsIT.datasetComptesResultats(ComptesResultatsIT.java:44)}} > {{ at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)}} > {{ at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}} > {{ at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}} > {{ at java.lang.reflect.Method.invoke(Method.java:498)}} > {{ at > org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:532)}} > {{ at > org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:115)}} > {{ at > org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:171)}} > {{ at > org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:72)}} > {{ at > org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:167)}} > {{ at > org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:114)}} > {{ at > org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:59)}} > {{ at > org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$4(NodeTestTask.java:108)}} > {{ at > org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:72)}} > {{ at > org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:98)}} > {{ at > org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:74)}} > {{ at java.util.ArrayList.forEach(ArrayList.java:1249)}} > {{ at > org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)}} > {{ at > org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$4(NodeTestTask.java:112)}} > {{ at > org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:72)}} > {{ at > org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:98)}} > {{ at > org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:74)}} > {{ at java.util.ArrayList.forEach(ArrayList.java:1249)}} > {{ at > org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)}} > {{ at > org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$4(NodeTestTask.java:112)}} > {{ at > org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:72)}} > {{ at > org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:98)}} > {{ at > org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:74)}} > {{ at > org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:32)}} > {{ at > org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)}} > {{ at > org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:51)}} > {{ at > org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:170)}} > {{ at > org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:154)}} > {{ at > org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:90)}} > {{ at > org.eclipse.jdt.internal.junit5.runner.JUnit5TestReference.run(JUnit5TestReference.java:89)}} > {{ at > org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:41)}} > {{ at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:541)}} > {{ at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:763)}} > {{ at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:463)}} > {{ at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:209)}}{{ > }} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org