Hi Nayan,
I use the --packages with the spark shell and the spark submit. Could you
please try that and let us know:
Command:
spark-submit --packages com.databricks:spark-csv_2.11:1.4.0
On Fri, 7 Apr 2017 at 00:39 nayan sharma <nayansharm...@gmail.com> wrote:
> spark version 1.6.2
> scala version 2.10.5
>
> On 06-Apr-2017, at 8:05 PM, Jörn Franke <jornfra...@gmail.com> wrote:
>
> And which version does your Spark cluster use?
>
> On 6. Apr 2017, at 16:11, nayan sharma <nayansharm...@gmail.com> wrote:
>
> scalaVersion := “2.10.5"
>
>
>
>
>
> On 06-Apr-2017, at 7:35 PM, Jörn Franke <jornfra...@gmail.com> wrote:
>
> Maybe your Spark is based on scala 2.11, but you compile it for 2.10 or
> the other way around?
>
> On 6. Apr 2017, at 15:54, nayan sharma <nayansharm...@gmail.com> wrote:
>
> In addition I am using spark version 1.6.2
> Is there any chance of error coming because of Scala version or
> dependencies are not matching.?I just guessed.
>
> Thanks,
> Nayan
>
>
>
> On 06-Apr-2017, at 7:16 PM, nayan sharma <nayansharm...@gmail.com> wrote:
>
> Hi Jorn,
> Thanks for replying.
>
> jar -tf catalyst-data-prepration-assembly-1.0.jar | grep csv
>
> after doing this I have found a lot of classes under
> com/databricks/spark/csv/
>
> do I need to check for any specific class ??
>
> Regards,
> Nayan
>
> On 06-Apr-2017, at 6:42 PM, Jörn Franke <jornfra...@gmail.com> wrote:
>
> Is the library in your assembly jar?
>
> On 6. Apr 2017, at 15:06, nayan sharma <nayansharm...@gmail.com> wrote:
>
> Hi All,
> I am getting error while loading CSV file.
>
> val
> datacsv=sqlContext.read.format("com.databricks.spark.csv").option("header",
> "true").load("timeline.csv")
> java.lang.NoSuchMethodError:
> org.apache.commons.csv.CSVFormat.withQuote(Ljava/lang/Character;)Lorg/apache/commons/csv/CSVFormat;
>
>
> I have added the dependencies in sbt file
>
> // Spark Additional Library - CSV Read as DFlibraryDependencies +=
> "com.databricks" %% "spark-csv" % “1.5.0"
>
> *and starting the spark-shell with command*
>
> spark-shell --master yarn-client --jars
> /opt/packages/xxxx-data-prepration/target/scala-2.10/xxxx-data-prepration-assembly-1.0.jar
> --name nayan
>
>
>
> Thanks for any help!!
>
>
> Thanks,
> Nayan
>
>
>
>
>
>
