Sorry buddy, didn't get your question quite right. Just to test, I created a scala class with spark csv and it seemed to work.
Donno if that would help much, but here are the env details: EMR 2.7.3 scalaVersion := "2.11.8" Spark version 2.0.2 On Fri, 7 Apr 2017 at 17:51 nayan sharma <nayansharm...@gmail.com> wrote: > Hi Yash, > I know this will work perfect but here I wanted to read the csv using the > assembly jar file. > > Thanks, > Nayan > > On 07-Apr-2017, at 10:02 AM, Yash Sharma <yash...@gmail.com> wrote: > > Hi Nayan, > I use the --packages with the spark shell and the spark submit. Could you > please try that and let us know: > Command: > > spark-submit --packages com.databricks:spark-csv_2.11:1.4.0 > > > On Fri, 7 Apr 2017 at 00:39 nayan sharma <nayansharm...@gmail.com> wrote: > > spark version 1.6.2 > scala version 2.10.5 > > On 06-Apr-2017, at 8:05 PM, Jörn Franke <jornfra...@gmail.com> wrote: > > And which version does your Spark cluster use? > > On 6. Apr 2017, at 16:11, nayan sharma <nayansharm...@gmail.com> wrote: > > scalaVersion := “2.10.5" > > > > > > On 06-Apr-2017, at 7:35 PM, Jörn Franke <jornfra...@gmail.com> wrote: > > Maybe your Spark is based on scala 2.11, but you compile it for 2.10 or > the other way around? > > On 6. Apr 2017, at 15:54, nayan sharma <nayansharm...@gmail.com> wrote: > > In addition I am using spark version 1.6.2 > Is there any chance of error coming because of Scala version or > dependencies are not matching.?I just guessed. > > Thanks, > Nayan > > > > On 06-Apr-2017, at 7:16 PM, nayan sharma <nayansharm...@gmail.com> wrote: > > Hi Jorn, > Thanks for replying. > > jar -tf catalyst-data-prepration-assembly-1.0.jar | grep csv > > after doing this I have found a lot of classes under > com/databricks/spark/csv/ > > do I need to check for any specific class ?? > > Regards, > Nayan > > On 06-Apr-2017, at 6:42 PM, Jörn Franke <jornfra...@gmail.com> wrote: > > Is the library in your assembly jar? > > On 6. Apr 2017, at 15:06, nayan sharma <nayansharm...@gmail.com> wrote: > > Hi All, > I am getting error while loading CSV file. > > val > datacsv=sqlContext.read.format("com.databricks.spark.csv").option("header", > "true").load("timeline.csv") > java.lang.NoSuchMethodError: > org.apache.commons.csv.CSVFormat.withQuote(Ljava/lang/Character;)Lorg/apache/commons/csv/CSVFormat; > > > I have added the dependencies in sbt file > > // Spark Additional Library - CSV Read as DFlibraryDependencies += > "com.databricks" %% "spark-csv" % “1.5.0" > > *and starting the spark-shell with command* > > spark-shell --master yarn-client --jars > /opt/packages/xxxx-data-prepration/target/scala-2.10/xxxx-data-prepration-assembly-1.0.jar > --name nayan > > > > Thanks for any help!! > > > Thanks, > Nayan > > > > > > >