paranamer is not a direct Crail dependency but is a recursive dependency of
hadoop-common respectively avro.
You can see all the dependencies and dependencies of dependencies by running
"mvn dependency:tree".
I checked if using a newer Hadoop version helps but even for the latest
3.2.0 it still depends on paranamer 2.3. Can you try just deleting the
paranamer jar I doubt that its content is ever loaded since we do not use
the avro file format. I'm not sure what the right solution for this problem
is. Can you please create a JIRA issue so that we can take a look at it.
Regards,
Jonas
On Mon, 24 Jun 2019 21:47:21 +0000
David Crespi <[email protected]> wrote:
Not sure if you’ve seen this
(https://jira.apache.org/jira/browse/SPARK-22128)
but it appears that paranamer 2.3 has an issue with Scala 2.12.
Which is what I’m
running. How do I update crail to use paranamer 2.8… as I mentioned
before, I’m
not rally that familiar with java.
Regards,
David
C: 714-476-2692
________________________________
From: Jonas Pfefferle <[email protected]>
Sent: Friday, June 21, 2019 12:47:34 AM
To: David Crespi
Subject: Re: Question on wildcard used with spark-default.conf
That probably has to do with the fact that you are using Spark 2.4.2
and
some of the dependencies used in Spark have different versions in
Crail.
Unfortunately, Java does not handle different versions of
dependencies and
picks whatever is first in the classpath. What Hadoop version are
you using
with Spark? Try using the same with the Crail build. This might fix
the
issue. Don't forget to do a "mvn clean" otherwise you might end up
with
multiple versions of a jar.
Cheers,
Jonas
On Thu, 20 Jun 2019 14:08:14 +0000
David Crespi <[email protected]> wrote:
When I remove the environment variable and my spark-defaults.conf
file contains:
spark.driver.extraClassPath /crail/jars/*
spark.executor.extraClassPath /crail/jars/*
I get the error below. I can’t even used $CRAIL_HOME, as it doesn’t
get resolved.
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException:
17418
at
com.thoughtworks.paranamer.BytecodeReadingParanamer$ClassReader.accept(BytecodeReadingParanamer.java:532)
at
com.thoughtworks.paranamer.BytecodeReadingParanamer$ClassReader.access$200(BytecodeReadingParanamer.java:315)
at
com.thoughtworks.paranamer.BytecodeReadingParanamer.lookupParameterNames(BytecodeReadingParanamer.java:102)
at
com.thoughtworks.paranamer.CachingParanamer.lookupParameterNames(CachingParanamer.java:76)
at
com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.getCtorParams(BeanIntrospector.scala:44)
at
com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$1(BeanIntrospector.scala:58)
at
com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$1$adapted(BeanIntrospector.scala:58)
at
scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:244)
at scala.collection.Iterator.foreach(Iterator.scala:941)
at scala.collection.Iterator.foreach$(Iterator.scala:941)
at
scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
at
scala.collection.IterableLike.foreach(IterableLike.scala:74)
at
scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at
scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at
scala.collection.TraversableLike.flatMap(TraversableLike.scala:244)
at
scala.collection.TraversableLike.flatMap$(TraversableLike.scala:241)
at
scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)
at
com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.findConstructorParam$1(BeanIntrospector.scala:58)
at
com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$19(BeanIntrospector.scala:176)
at
scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237)
at
scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
at
scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
at
scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
at
scala.collection.TraversableLike.map(TraversableLike.scala:237)
at
scala.collection.TraversableLike.map$(TraversableLike.scala:230)
at
scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:198)
at
com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$14(BeanIntrospector.scala:170)
at
com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$14$adapted(BeanIntrospector.scala:169)
at
scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:244)
at scala.collection.immutable.List.foreach(List.scala:392)
at
scala.collection.TraversableLike.flatMap(TraversableLike.scala:244)
at
scala.collection.TraversableLike.flatMap$(TraversableLike.scala:241)
at scala.collection.immutable.List.flatMap(List.scala:355)
at
com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.apply(BeanIntrospector.scala:169)
at
com.fasterxml.jackson.module.scala.introspect.ScalaAnnotationIntrospector$._descriptorFor(ScalaAnnotationIntrospectorModule.scala:22)
at
com.fasterxml.jackson.module.scala.introspect.ScalaAnnotationIntrospector$.fieldName(ScalaAnnotationIntrospectorModule.scala:30)
at
com.fasterxml.jackson.module.scala.introspect.ScalaAnnotationIntrospector$.findImplicitPropertyName(ScalaAnnotationIntrospectorModule.scala:78)
at
com.fasterxml.jackson.databind.introspect.AnnotationIntrospectorPair.findImplicitPropertyName(AnnotationIntrospectorPair.java:467)
at
com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector._addFields(POJOPropertiesCollector.java:351)
at
com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.collectAll(POJOPropertiesCollector.java:283)
at
com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.getJsonValueMethod(POJOPropertiesCollector.java:169)
at
com.fasterxml.jackson.databind.introspect.BasicBeanDescription.findJsonValueMethod(BasicBeanDescription.java:223)
at
com.fasterxml.jackson.databind.ser.BasicSerializerFactory.findSerializerByAnnotations(BasicSerializerFactory.java:348)
at
com.fasterxml.jackson.databind.ser.BeanSerializerFactory._createSerializer2(BeanSerializerFactory.java:210)
at
com.fasterxml.jackson.databind.ser.BeanSerializerFactory.createSerializer(BeanSerializerFactory.java:153)
at
com.fasterxml.jackson.databind.SerializerProvider._createUntypedSerializer(SerializerProvider.java:1203)
at
com.fasterxml.jackson.databind.SerializerProvider._createAndCacheUntypedSerializer(SerializerProvider.java:1157)
at
com.fasterxml.jackson.databind.SerializerProvider.findValueSerializer(SerializerProvider.java:481)
at
com.fasterxml.jackson.databind.SerializerProvider.findTypedValueSerializer(SerializerProvider.java:679)
at
com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:107)
at
com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3559)
at
com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:2927)
at
org.apache.spark.rdd.RDDOperationScope.toJson(RDDOperationScope.scala:52)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:145)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at
org.apache.spark.SparkContext.withScope(SparkContext.scala:699)
at
org.apache.spark.SparkContext.parallelize(SparkContext.scala:716)
at
com.github.ehiggs.spark.terasort.TeraGen$.main(TeraGen.scala:66)
at
com.github.ehiggs.spark.terasort.TeraGen.main(TeraGen.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
at
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
at
org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
at
org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
at
org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
at
org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Regards,
David
C: 714-476-2692
________________________________
From: Jonas Pfefferle <[email protected]>
Sent: Thursday, June 20, 2019 6:18:32 AM
To: David Crespi; [email protected]
Subject: Re: Question on wildcard used with spark-default.conf
For Crail I put all jars in the directory including all dependencies.
If you
build it from source these will be in assembly/...
Regards,
Jonas
On Thu, 20 Jun 2019 13:16:12 +0000
David Crespi <[email protected]> wrote:
I started off doing that way, and I think I was hitting another java
error with something like
a “number out of bounds” error, as in too many files perhaps. Do
you only put specific files
into the jar directory? I’ll go back and play around with it again
today and report back.
Regards,
David
________________________________
From: Jonas Pfefferle <[email protected]>
Sent: Thursday, June 20, 2019 12:17:36 AM
To: [email protected]; David Crespi
Subject: Re: Question on wildcard used with spark-default.conf
Hi David,
the wildcard works for me without problem. However I do not use the
environment variable. Can you try putting the actual Crail path
instead of
the env variable.
Jonas
On Wed, 19 Jun 2019 22:33:09 +0000
David Crespi <[email protected]> wrote:
A question on what is documented on the crail website for
configuring to use
the HDFS adapter. The website shows the use of a wildcard (in red):
spark.driver.extraClassPath
$CRAIL_HOME/jars/*:$CRAIL_HOME/jars/crail-spark-1.0.jar:.
spark.executor.extraClassPath
$CRAIL_HOME/jars/*:$CRAIL_HOME/jars/crail-spark-1.0.jar:.
But this didn’t work for me. I had to put all the specific jars
onto this path in order for this to work,
such as the lines below. Is there a trick to using the wildcard?
spark.driver.extraClassPath
/crail/jars/crail-client-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-hdfs-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-namenode-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-darpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-narpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-spark-1.0.jar:/crail/jars/crail-storage-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-narpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-nvmf-1.2-incubating-SNAPSHOT-tests.jar:/crail/jars/crail-storage-nvmf-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-rdma-1.2-incubating-SNAPSHOT.jar:/crail/jars/darpc-1.8.jar:/crail/jars/disni-2.0.jar:/crail/jars/htrace-core-3.1.0-incubating.jar:/crail/jars/narpc-1.4.jar:/jNVMf/target/*:/dinsi/target/*
spark.executor.extraClassPath
/crail/jars/crail-client-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-hdfs-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-namenode-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-darpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-narpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-spark-1.0.jar:/crail/jars/crail-storage-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-narpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-nvmf-1.2-incubating-SNAPSHOT-tests.jar:/crail/jars/crail-storage-nvmf-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-rdma-1.2-incubating-SNAPSHOT.jar:/crail/jars/darpc-1.8.jar:/crail/jars/disni-2.0.jar:/crail/jars/htrace-core-3.1.0-incubating.jar:/crail/jars/narpc-1.4.jar:/jNVMf/target/*:/dinsi/target/*
This is the error I get with using the wildcard:
Exception in thread "main" java.lang.RuntimeException:
java.lang.ClassNotFoundException: Class
org.apache.crail.hdfs.CrailHadoopFileSystem not found
And if I add that one to the path, then I get the next one it
doesn’t find.
Regards,
David
