Yes I am reading from s3 bucket .. Strangely the error goes off when I remove the properties girl parameter .
On Sep 6, 2016 8:35 PM, "Sonal Goyal" <sonalgoy...@gmail.com> wrote: > Looks like a classpath issue - Caused by: java.lang.ClassNotFoundException: > com.amazonaws.services.s3.AmazonS3 > > Are you using S3 somewhere? Are the required jars in place? > > Best Regards, > Sonal > Founder, Nube Technologies <http://www.nubetech.co> > Reifier at Strata Hadoop World > <https://www.youtube.com/watch?v=eD3LkpPQIgM> > Reifier at Spark Summit 2015 > <https://spark-summit.org/2015/events/real-time-fuzzy-matching-with-spark-and-elastic-search/> > > <http://in.linkedin.com/in/sonalgoyal> > > > > On Tue, Sep 6, 2016 at 4:45 PM, Divya Gehlot <divya.htco...@gmail.com> > wrote: > >> Hi, >> I am getting below error if I try to use properties file paramater in >> spark-submit >> >> Exception in thread "main" java.util.ServiceConfigurationError: >> org.apache.hadoop.fs.FileSystem: Provider >> org.apache.hadoop.fs.s3a.S3AFileSystem >> could not be instantiated >> at java.util.ServiceLoader.fail(ServiceLoader.java:224) >> at java.util.ServiceLoader.access$100(ServiceLoader.java:181) >> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:377) >> at java.util.ServiceLoader$1.next(ServiceLoader.java:445) >> at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2673) >> at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSyste >> m.java:2684) >> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2701) >> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91) >> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem >> .java:2737) >> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2719) >> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:375) >> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:174) >> at org.apache.spark.deploy.yarn.ApplicationMaster.run(Applicati >> onMaster.scala:142) >> at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$main >> $1.apply$mcV$sp(ApplicationMaster.scala:653) >> at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHad >> oopUtil.scala:69) >> at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHad >> oopUtil.scala:68) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAs(Subject.java:415) >> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro >> upInformation.java:1657) >> at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(Spark >> HadoopUtil.scala:68) >> at org.apache.spark.deploy.yarn.ApplicationMaster$.main(Applica >> tionMaster.scala:651) >> at org.apache.spark.deploy.yarn.ApplicationMaster.main(Applicat >> ionMaster.scala) >> Caused by: java.lang.NoClassDefFoundError: com/amazonaws/services/s3/Amaz >> onS3 >> at java.lang.Class.getDeclaredConstructors0(Native Method) >> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2595) >> at java.lang.Class.getConstructor0(Class.java:2895) >> at java.lang.Class.newInstance(Class.java:354) >> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:373) >> ... 19 more >> Caused by: java.lang.ClassNotFoundException: >> com.amazonaws.services.s3.AmazonS3 >> at java.net.URLClassLoader$1.run(URLClassLoader.java:366) >> at java.net.URLClassLoader$1.run(URLClassLoader.java:355) >> at java.security.AccessController.doPrivileged(Native Method) >> at java.net.URLClassLoader.findClass(URLClassLoader.java:354) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:425) >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:358) >> ... 24 more >> End of LogType:stderr >> >> If I remove the --properties-file parameter >> the error is gone >> >> Would really appreciate the help . >> >> >> >> Thanks, >> Divya >> > >
