Hi Todd,
So I am already specifying joda-time-2.7 (have tried 2.2, 2.3, 2.6, 2.7) in
the --jars option. I tried using the joda-time bundle jar (
http://mvnrepository.com/artifact/org.apache.servicemix.bundles/org.apache.servicemix.bundles.joda-time/2.3_1)
which comes with joda-convert.
I know its pointing to the appropriate jar as I included this in my java
program: System.out.println(DateTimeFormat.forPattern("yyyyMMdd")
.getClass()
.getProtectionDomain()
.getCodeSource()
.getLocation());
and it tells me that it's pointing to the jar that i'm giving it in the
--jars option. It would be helpful to know if the error I am getting is
because of spark-submit or the java app. Thank you!
Exception in thread "main" java.lang.NoClassDefFoundError:
org/joda/time/format/DateTimeFormat
at com.amazonaws.auth.AWS4Signer.<clinit>(AWS4Signer.java:44)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at java.lang.Class.newInstance(Class.java:374)
at
com.amazonaws.auth.SignerFactory.createSigner(SignerFactory.java:119)
at
com.amazonaws.auth.SignerFactory.lookupAndCreateSigner(SignerFactory.java:105)
at com.amazonaws.auth.SignerFactory.getSigner(SignerFactory.java:78)
at
com.amazonaws.AmazonWebServiceClient.computeSignerByServiceRegion(AmazonWebServiceClient.java:307)
at
com.amazonaws.AmazonWebServiceClient.computeSignerByURI(AmazonWebServiceClient.java:280)
at
com.amazonaws.AmazonWebServiceClient.setEndpoint(AmazonWebServiceClient.java:160)
at
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.setEndpoint(AmazonDynamoDBClient.java:2946)
at
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.init(AmazonDynamoDBClient.java:351)
at
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.<init>(AmazonDynamoDBClient.java:273)
at
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.<init>(AmazonDynamoDBClient.java:250)
at AmazonDynamoDBSample.init(AmazonDynamoDBSample.java:81)
at AmazonDynamoDBSample.main(AmazonDynamoDBSample.java:87)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException:
org.joda.time.format.DateTimeFormat
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
Best,
Su
On Fri, Feb 27, 2015 at 4:14 PM, Todd Nist <[email protected]> wrote:
> You can specify these jars (joda-time-2.7.jar, joda-convert-1.7.jar)
> either as part of your build and assembly or via the --jars option to
> spark-submit.
>
> HTH.
>
> On Fri, Feb 27, 2015 at 2:48 PM, Su She <[email protected]> wrote:
>
>> Hello Everyone,
>>
>> I'm having some issues launching (non-spark) applications via the
>> spark-submit commands. The common error I am getting is c/p below. I am
>> able to submit a spark streaming/kafka spark application, but can't start a
>> dynamoDB java app. The common error is related to joda-time.
>>
>> 1) I realized spark-submit was pointing to joda-time-1.6 in the
>> hadoop/lib,so I deleted this and my error changed from NoSuchMethodFound to
>> NoClassDefFoundError.
>>
>> Instead of pointing to the other version of joda-time in the hadoop/lib,
>> it now pointed to the jars I set a path to in my spark-submit command (I
>> tried joda-time versions 2.2, 2.3, 2.6, 2.7), but still got the errors
>>
>> 2) My rudimentary theory is that spark-submit uses < joda-time-2.0, but
>> the applications I'm running need >2.0.
>>
>> Thank you for the help!
>>
>> Exception in thread "main" java.lang.NoClassDefFoundError:
>> org/joda/time/format/DateTimeFormat
>> at com.amazonaws.auth.AWS4Signer.<clinit>(AWS4Signer.java:44)
>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>> Method)
>> at
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>> at java.lang.Class.newInstance(Class.java:374)
>> at
>> com.amazonaws.auth.SignerFactory.createSigner(SignerFactory.java:119)
>> at
>> com.amazonaws.auth.SignerFactory.lookupAndCreateSigner(SignerFactory.java:105)
>> at
>> com.amazonaws.auth.SignerFactory.getSigner(SignerFactory.java:78)
>> at
>> com.amazonaws.AmazonWebServiceClient.computeSignerByServiceRegion(AmazonWebServiceClient.java:307)
>> at
>> com.amazonaws.AmazonWebServiceClient.computeSignerByURI(AmazonWebServiceClient.java:280)
>> at
>> com.amazonaws.AmazonWebServiceClient.setEndpoint(AmazonWebServiceClient.java:160)
>> at
>> com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.setEndpoint(AmazonDynamoDBClient.java:2946)
>> at
>> com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.init(AmazonDynamoDBClient.java:351)
>> at
>> com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.<init>(AmazonDynamoDBClient.java:273)
>> at
>> com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.<init>(AmazonDynamoDBClient.java:250)
>> at AmazonDynamoDBSample.init(AmazonDynamoDBSample.java:81)
>> at AmazonDynamoDBSample.main(AmazonDynamoDBSample.java:87)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at
>> org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358)
>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>> Caused by: java.lang.ClassNotFoundException:
>> org.joda.time.format.DateTimeFormat
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>>
>>
>
