Hi Siva This is what i have for jars. Did you manage to run with these or different versions ?
<dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_2.10</artifactId> <version>1.6.1</version> </dependency> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-sql_2.10</artifactId> <version>1.6.1</version> </dependency> <dependency> <groupId>com.databricks</groupId> <artifactId>spark-xml_2.10</artifactId> <version>0.2.0</version> </dependency> <dependency> <groupId>org.scala-lang</groupId> <artifactId>scala-library</artifactId> <version>2.10.6</version> </dependency> Thanks VG On Fri, Jun 17, 2016 at 4:16 PM, Siva A <siva9940261...@gmail.com> wrote: > Hi Marco, > > I did run in IDE(Intellij) as well. It works fine. > VG, make sure the right jar is in classpath. > > --Siva > > On Fri, Jun 17, 2016 at 4:11 PM, Marco Mistroni <mmistr...@gmail.com> > wrote: > >> and your eclipse path is correct? >> i suggest, as Siva did before, to build your jar and run it via >> spark-submit by specifying the --packages option >> it's as simple as run this command >> >> spark-submit --packages >> com.databricks:spark-xml_<scalaversion>:<packageversion> --class <Name of >> your class containing main> <path to your jar> >> >> Indeed, if you have only these lines to run, why dont you try them in >> spark-shell ? >> >> hth >> >> On Fri, Jun 17, 2016 at 11:32 AM, VG <vlin...@gmail.com> wrote: >> >>> nopes. eclipse. >>> >>> >>> On Fri, Jun 17, 2016 at 3:58 PM, Siva A <siva9940261...@gmail.com> >>> wrote: >>> >>>> If you are running from IDE, Are you using Intellij? >>>> >>>> On Fri, Jun 17, 2016 at 3:20 PM, Siva A <siva9940261...@gmail.com> >>>> wrote: >>>> >>>>> Can you try to package as a jar and run using spark-submit >>>>> >>>>> Siva >>>>> >>>>> On Fri, Jun 17, 2016 at 3:17 PM, VG <vlin...@gmail.com> wrote: >>>>> >>>>>> I am trying to run from IDE and everything else is working fine. >>>>>> I added spark-xml jar and now I ended up into this dependency >>>>>> >>>>>> 6/06/17 15:15:57 INFO BlockManagerMaster: Registered BlockManager >>>>>> Exception in thread "main" *java.lang.NoClassDefFoundError: >>>>>> scala/collection/GenTraversableOnce$class* >>>>>> at >>>>>> org.apache.spark.sql.execution.datasources.CaseInsensitiveMap.<init>(ddl.scala:150) >>>>>> at >>>>>> org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:154) >>>>>> at >>>>>> org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:119) >>>>>> at >>>>>> org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:109) >>>>>> at org.ariba.spark.PostsProcessing.main(PostsProcessing.java:19) >>>>>> Caused by:* java.lang.ClassNotFoundException: >>>>>> scala.collection.GenTraversableOnce$class* >>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381) >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >>>>>> ... 5 more >>>>>> 16/06/17 15:15:58 INFO SparkContext: Invoking stop() from shutdown >>>>>> hook >>>>>> >>>>>> >>>>>> >>>>>> On Fri, Jun 17, 2016 at 2:59 PM, Marco Mistroni <mmistr...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> So you are using spark-submit or spark-shell? >>>>>>> >>>>>>> you will need to launch either by passing --packages option (like in >>>>>>> the example below for spark-csv). you will need to iknow >>>>>>> >>>>>>> --packages com.databricks:spark-xml_<scala.version>:<package version> >>>>>>> >>>>>>> hth >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Fri, Jun 17, 2016 at 10:20 AM, VG <vlin...@gmail.com> wrote: >>>>>>> >>>>>>>> Apologies for that. >>>>>>>> I am trying to use spark-xml to load data of a xml file. >>>>>>>> >>>>>>>> here is the exception >>>>>>>> >>>>>>>> 16/06/17 14:49:04 INFO BlockManagerMaster: Registered BlockManager >>>>>>>> Exception in thread "main" java.lang.ClassNotFoundException: Failed >>>>>>>> to find data source: org.apache.spark.xml. Please find packages at >>>>>>>> http://spark-packages.org >>>>>>>> at >>>>>>>> org.apache.spark.sql.execution.datasources.ResolvedDataSource$.lookupDataSource(ResolvedDataSource.scala:77) >>>>>>>> at >>>>>>>> org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:102) >>>>>>>> at >>>>>>>> org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:119) >>>>>>>> at >>>>>>>> org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:109) >>>>>>>> at org.ariba.spark.PostsProcessing.main(PostsProcessing.java:19) >>>>>>>> Caused by: java.lang.ClassNotFoundException: >>>>>>>> org.apache.spark.xml.DefaultSource >>>>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381) >>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >>>>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) >>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >>>>>>>> at >>>>>>>> org.apache.spark.sql.execution.datasources.ResolvedDataSource$$anonfun$4$$anonfun$apply$1.apply(ResolvedDataSource.scala:62) >>>>>>>> at >>>>>>>> org.apache.spark.sql.execution.datasources.ResolvedDataSource$$anonfun$4$$anonfun$apply$1.apply(ResolvedDataSource.scala:62) >>>>>>>> at scala.util.Try$.apply(Try.scala:192) >>>>>>>> at >>>>>>>> org.apache.spark.sql.execution.datasources.ResolvedDataSource$$anonfun$4.apply(ResolvedDataSource.scala:62) >>>>>>>> at >>>>>>>> org.apache.spark.sql.execution.datasources.ResolvedDataSource$$anonfun$4.apply(ResolvedDataSource.scala:62) >>>>>>>> at scala.util.Try.orElse(Try.scala:84) >>>>>>>> at >>>>>>>> org.apache.spark.sql.execution.datasources.ResolvedDataSource$.lookupDataSource(ResolvedDataSource.scala:62) >>>>>>>> ... 4 more >>>>>>>> >>>>>>>> Code >>>>>>>> SQLContext sqlContext = new SQLContext(sc); >>>>>>>> DataFrame df = sqlContext.read() >>>>>>>> .format("org.apache.spark.xml") >>>>>>>> .option("rowTag", "row") >>>>>>>> .load("A.xml"); >>>>>>>> >>>>>>>> Any suggestions please .. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Jun 17, 2016 at 2:42 PM, Marco Mistroni < >>>>>>>> mmistr...@gmail.com> wrote: >>>>>>>> >>>>>>>>> too little info >>>>>>>>> it'll help if you can post the exception and show your sbt file >>>>>>>>> (if you are using sbt), and provide minimal details on what you are >>>>>>>>> doing >>>>>>>>> kr >>>>>>>>> >>>>>>>>> On Fri, Jun 17, 2016 at 10:08 AM, VG <vlin...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Failed to find data source: com.databricks.spark.xml >>>>>>>>>> >>>>>>>>>> Any suggestions to resolve this >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
