Re: Error while reading the CSV

[Praneeth Gayam]

Try the following

spark-shell --master yarn-client  --name nayan  /opt/packages/-data-
prepration/target/scala-2.10/-data-prepration-assembly-1.0.jar


On Thu, Apr 6, 2017 at 6:36 PM, nayan sharma 
wrote:

> Hi All,
> I am getting error while loading CSV file.
>
> val 
> datacsv=sqlContext.read.format("com.databricks.spark.csv").option("header",
> "true").load("timeline.csv")
> java.lang.NoSuchMethodError: org.apache.commons.csv.
> CSVFormat.withQuote(Ljava/lang/Character;)Lorg/apache/
> commons/csv/CSVFormat;
>
>
> I have added the dependencies in sbt file
>
> // Spark Additional Library - CSV Read as DFlibraryDependencies += 
> "com.databricks" %% "spark-csv" % “1.5.0"
>
> *and starting the spark-shell with command*
>
> spark-shell --master yarn-client  --jars /opt/packages/-data-
> prepration/target/scala-2.10/-data-prepration-assembly-1.0.jar --name
> nayan
>
>
>
> Thanks for any help!!
>
>
> Thanks,
> Nayan
>

Re: Error while reading the CSV

[Yash Sharma]

Sorry buddy, didn't get your question quite right.
Just to test, I created a scala class with spark csv and it seemed to work.

Donno if that would help much, but here are the env details:
EMR 2.7.3
scalaVersion := "2.11.8"
Spark version 2.0.2



On Fri, 7 Apr 2017 at 17:51 nayan sharma  wrote:

> Hi Yash,
> I know this will work perfect but here I wanted  to read the csv using the
> assembly jar file.
>
> Thanks,
> Nayan
>
> On 07-Apr-2017, at 10:02 AM, Yash Sharma  wrote:
>
> Hi Nayan,
> I use the --packages with the spark shell and the spark submit. Could you
> please try that and let us know:
> Command:
>
> spark-submit --packages com.databricks:spark-csv_2.11:1.4.0
>
>
> On Fri, 7 Apr 2017 at 00:39 nayan sharma  wrote:
>
> spark version 1.6.2
> scala version 2.10.5
>
> On 06-Apr-2017, at 8:05 PM, Jörn Franke  wrote:
>
> And which version does your Spark cluster use?
>
> On 6. Apr 2017, at 16:11, nayan sharma  wrote:
>
> scalaVersion := “2.10.5"
>
>
>
>
>
> On 06-Apr-2017, at 7:35 PM, Jörn Franke  wrote:
>
> Maybe your Spark is based on scala 2.11, but you compile it for 2.10 or
> the other way around?
>
> On 6. Apr 2017, at 15:54, nayan sharma  wrote:
>
> In addition I am using spark version 1.6.2
> Is there any chance of error coming because of Scala version or
> dependencies are not matching.?I just guessed.
>
> Thanks,
> Nayan
>
>
>
> On 06-Apr-2017, at 7:16 PM, nayan sharma  wrote:
>
> Hi Jorn,
> Thanks for replying.
>
> jar -tf catalyst-data-prepration-assembly-1.0.jar | grep csv
>
> after doing this I have found a lot of classes under
> com/databricks/spark/csv/
>
> do I need to check for any specific class ??
>
> Regards,
> Nayan
>
> On 06-Apr-2017, at 6:42 PM, Jörn Franke  wrote:
>
> Is the library in your assembly jar?
>
> On 6. Apr 2017, at 15:06, nayan sharma  wrote:
>
> Hi All,
> I am getting error while loading CSV file.
>
> val
> datacsv=sqlContext.read.format("com.databricks.spark.csv").option("header",
> "true").load("timeline.csv")
> java.lang.NoSuchMethodError:
> org.apache.commons.csv.CSVFormat.withQuote(Ljava/lang/Character;)Lorg/apache/commons/csv/CSVFormat;
>
>
> I have added the dependencies in sbt file
>
> // Spark Additional Library - CSV Read as DFlibraryDependencies += 
> "com.databricks" %% "spark-csv" % “1.5.0"
>
> *and starting the spark-shell with command*
>
> spark-shell --master yarn-client  --jars
> /opt/packages/-data-prepration/target/scala-2.10/-data-prepration-assembly-1.0.jar
> --name nayan
>
>
>
> Thanks for any help!!
>
>
> Thanks,
> Nayan
>
>
>
>
>
>
>

Re: Error while reading the CSV

[nayan sharma]

Hi Yash,
I know this will work perfect but here I wanted  to read the csv using the 
assembly jar file.

Thanks,
Nayan

> On 07-Apr-2017, at 10:02 AM, Yash Sharma  wrote:
> 
> Hi Nayan,
> I use the --packages with the spark shell and the spark submit. Could you 
> please try that and let us know:
> Command:
> spark-submit --packages com.databricks:spark-csv_2.11:1.4.0
> 
> On Fri, 7 Apr 2017 at 00:39 nayan sharma  > wrote:
> spark version 1.6.2
> scala version 2.10.5
> 
>> On 06-Apr-2017, at 8:05 PM, Jörn Franke > > wrote:
>> 
>> And which version does your Spark cluster use?
>> 
>> On 6. Apr 2017, at 16:11, nayan sharma > > wrote:
>> 
>>> scalaVersion := “2.10.5"
>>> 
>>> 
>>> 
>>> 
 On 06-Apr-2017, at 7:35 PM, Jörn Franke >>> > wrote:
 
 Maybe your Spark is based on scala 2.11, but you compile it for 2.10 or 
 the other way around?
 
 On 6. Apr 2017, at 15:54, nayan sharma >>> > wrote:
 
> In addition I am using spark version 1.6.2
> Is there any chance of error coming because of Scala version or 
> dependencies are not matching.?I just guessed.
> 
> Thanks,
> Nayan
> 
>  
>> On 06-Apr-2017, at 7:16 PM, nayan sharma > > wrote:
>> 
>> Hi Jorn,
>> Thanks for replying.
>> 
>> jar -tf catalyst-data-prepration-assembly-1.0.jar | grep csv
>> 
>> after doing this I have found a lot of classes under 
>> com/databricks/spark/csv/
>> 
>> do I need to check for any specific class ??
>> 
>> Regards,
>> Nayan
>>> On 06-Apr-2017, at 6:42 PM, Jörn Franke >> > wrote:
>>> 
>>> Is the library in your assembly jar?
>>> 
>>> On 6. Apr 2017, at 15:06, nayan sharma >> > wrote:
>>> 
 Hi All,
 I am getting error while loading CSV file.
 
 val 
 datacsv=sqlContext.read.format("com.databricks.spark.csv").option("header",
  "true").load("timeline.csv")
 java.lang.NoSuchMethodError: 
 org.apache.commons.csv.CSVFormat.withQuote(Ljava/lang/Character;)Lorg/apache/commons/csv/CSVFormat;
 
 
 I have added the dependencies in sbt file 
 // Spark Additional Library - CSV Read as DF
 libraryDependencies += "com.databricks" %% "spark-csv" % “1.5.0"
 and starting the spark-shell with command
 
 spark-shell --master yarn-client  --jars 
 /opt/packages/-data-prepration/target/scala-2.10/-data-prepration-assembly-1.0.jar
  --name nayan 
 
 
 
 Thanks for any help!!
 
 
 Thanks,
 Nayan
>> 
> 
>>> 
> 

Re: Error while reading the CSV

[Yash Sharma]

Hi Nayan,
I use the --packages with the spark shell and the spark submit. Could you
please try that and let us know:
Command:

spark-submit --packages com.databricks:spark-csv_2.11:1.4.0


On Fri, 7 Apr 2017 at 00:39 nayan sharma  wrote:

> spark version 1.6.2
> scala version 2.10.5
>
> On 06-Apr-2017, at 8:05 PM, Jörn Franke  wrote:
>
> And which version does your Spark cluster use?
>
> On 6. Apr 2017, at 16:11, nayan sharma  wrote:
>
> scalaVersion := “2.10.5"
>
>
>
>
>
> On 06-Apr-2017, at 7:35 PM, Jörn Franke  wrote:
>
> Maybe your Spark is based on scala 2.11, but you compile it for 2.10 or
> the other way around?
>
> On 6. Apr 2017, at 15:54, nayan sharma  wrote:
>
> In addition I am using spark version 1.6.2
> Is there any chance of error coming because of Scala version or
> dependencies are not matching.?I just guessed.
>
> Thanks,
> Nayan
>
>
>
> On 06-Apr-2017, at 7:16 PM, nayan sharma  wrote:
>
> Hi Jorn,
> Thanks for replying.
>
> jar -tf catalyst-data-prepration-assembly-1.0.jar | grep csv
>
> after doing this I have found a lot of classes under
> com/databricks/spark/csv/
>
> do I need to check for any specific class ??
>
> Regards,
> Nayan
>
> On 06-Apr-2017, at 6:42 PM, Jörn Franke  wrote:
>
> Is the library in your assembly jar?
>
> On 6. Apr 2017, at 15:06, nayan sharma  wrote:
>
> Hi All,
> I am getting error while loading CSV file.
>
> val
> datacsv=sqlContext.read.format("com.databricks.spark.csv").option("header",
> "true").load("timeline.csv")
> java.lang.NoSuchMethodError:
> org.apache.commons.csv.CSVFormat.withQuote(Ljava/lang/Character;)Lorg/apache/commons/csv/CSVFormat;
>
>
> I have added the dependencies in sbt file
>
> // Spark Additional Library - CSV Read as DFlibraryDependencies += 
> "com.databricks" %% "spark-csv" % “1.5.0"
>
> *and starting the spark-shell with command*
>
> spark-shell --master yarn-client  --jars
> /opt/packages/-data-prepration/target/scala-2.10/-data-prepration-assembly-1.0.jar
> --name nayan
>
>
>
> Thanks for any help!!
>
>
> Thanks,
> Nayan
>
>
>
>
>
>

Re: Error while reading the CSV

[nayan sharma]

spark version 1.6.2
scala version 2.10.5

> On 06-Apr-2017, at 8:05 PM, Jörn Franke  wrote:
> 
> And which version does your Spark cluster use?
> 
> On 6. Apr 2017, at 16:11, nayan sharma  > wrote:
> 
>> scalaVersion := “2.10.5"
>> 
>> 
>> 
>> 
>>> On 06-Apr-2017, at 7:35 PM, Jörn Franke >> > wrote:
>>> 
>>> Maybe your Spark is based on scala 2.11, but you compile it for 2.10 or the 
>>> other way around?
>>> 
>>> On 6. Apr 2017, at 15:54, nayan sharma >> > wrote:
>>> 
 In addition I am using spark version 1.6.2
 Is there any chance of error coming because of Scala version or 
 dependencies are not matching.?I just guessed.
 
 Thanks,
 Nayan
 
  
> On 06-Apr-2017, at 7:16 PM, nayan sharma  > wrote:
> 
> Hi Jorn,
> Thanks for replying.
> 
> jar -tf catalyst-data-prepration-assembly-1.0.jar | grep csv
> 
> after doing this I have found a lot of classes under 
> com/databricks/spark/csv/
> 
> do I need to check for any specific class ??
> 
> Regards,
> Nayan
>> On 06-Apr-2017, at 6:42 PM, Jörn Franke > > wrote:
>> 
>> Is the library in your assembly jar?
>> 
>> On 6. Apr 2017, at 15:06, nayan sharma > > wrote:
>> 
>>> Hi All,
>>> I am getting error while loading CSV file.
>>> 
>>> val 
>>> datacsv=sqlContext.read.format("com.databricks.spark.csv").option("header",
>>>  "true").load("timeline.csv")
>>> java.lang.NoSuchMethodError: 
>>> org.apache.commons.csv.CSVFormat.withQuote(Ljava/lang/Character;)Lorg/apache/commons/csv/CSVFormat;
>>> 
>>> 
>>> I have added the dependencies in sbt file 
>>> // Spark Additional Library - CSV Read as DF
>>> libraryDependencies += "com.databricks" %% "spark-csv" % “1.5.0"
>>> and starting the spark-shell with command
>>> 
>>> spark-shell --master yarn-client  --jars 
>>> /opt/packages/-data-prepration/target/scala-2.10/-data-prepration-assembly-1.0.jar
>>>  --name nayan 
>>> 
>>> 
>>> 
>>> Thanks for any help!!
>>> 
>>> 
>>> Thanks,
>>> Nayan
> 
 
>> 

Re: Error while reading the CSV

[Jörn Franke]

And which version does your Spark cluster use?

> On 6. Apr 2017, at 16:11, nayan sharma  wrote:
> 
> scalaVersion := “2.10.5"
> 
> 
> 
> 
>> On 06-Apr-2017, at 7:35 PM, Jörn Franke  wrote:
>> 
>> Maybe your Spark is based on scala 2.11, but you compile it for 2.10 or the 
>> other way around?
>> 
>>> On 6. Apr 2017, at 15:54, nayan sharma  wrote:
>>> 
>>> In addition I am using spark version 1.6.2
>>> Is there any chance of error coming because of Scala version or 
>>> dependencies are not matching.?I just guessed.
>>> 
>>> Thanks,
>>> Nayan
>>> 
>>>  
 On 06-Apr-2017, at 7:16 PM, nayan sharma  wrote:
 
 Hi Jorn,
 Thanks for replying.
 
 jar -tf catalyst-data-prepration-assembly-1.0.jar | grep csv
 
 after doing this I have found a lot of classes under 
 com/databricks/spark/csv/
 
 do I need to check for any specific class ??
 
 Regards,
 Nayan
> On 06-Apr-2017, at 6:42 PM, Jörn Franke  wrote:
> 
> Is the library in your assembly jar?
> 
>> On 6. Apr 2017, at 15:06, nayan sharma  wrote:
>> 
>> Hi All,
>> I am getting error while loading CSV file.
>> 
>> val 
>> datacsv=sqlContext.read.format("com.databricks.spark.csv").option("header",
>>  "true").load("timeline.csv")
>> java.lang.NoSuchMethodError: 
>> org.apache.commons.csv.CSVFormat.withQuote(Ljava/lang/Character;)Lorg/apache/commons/csv/CSVFormat;
>> 
>> 
>> I have added the dependencies in sbt file 
>> // Spark Additional Library - CSV Read as DF
>> libraryDependencies += "com.databricks" %% "spark-csv" % “1.5.0"
>> and starting the spark-shell with command
>> 
>> spark-shell --master yarn-client  --jars 
>> /opt/packages/-data-prepration/target/scala-2.10/-data-prepration-assembly-1.0.jar
>>  --name nayan 
>> 
>> 
>> 
>> Thanks for any help!!
>> 
>> 
>> Thanks,
>> Nayan
 
>>> 
> 

Re: Error while reading the CSV

[nayan sharma]

scalaVersion := “2.10.5"




> On 06-Apr-2017, at 7:35 PM, Jörn Franke  wrote:
> 
> Maybe your Spark is based on scala 2.11, but you compile it for 2.10 or the 
> other way around?
> 
> On 6. Apr 2017, at 15:54, nayan sharma  > wrote:
> 
>> In addition I am using spark version 1.6.2
>> Is there any chance of error coming because of Scala version or dependencies 
>> are not matching.?I just guessed.
>> 
>> Thanks,
>> Nayan
>> 
>>  
>>> On 06-Apr-2017, at 7:16 PM, nayan sharma >> > wrote:
>>> 
>>> Hi Jorn,
>>> Thanks for replying.
>>> 
>>> jar -tf catalyst-data-prepration-assembly-1.0.jar | grep csv
>>> 
>>> after doing this I have found a lot of classes under 
>>> com/databricks/spark/csv/
>>> 
>>> do I need to check for any specific class ??
>>> 
>>> Regards,
>>> Nayan
 On 06-Apr-2017, at 6:42 PM, Jörn Franke >>> > wrote:
 
 Is the library in your assembly jar?
 
 On 6. Apr 2017, at 15:06, nayan sharma >>> > wrote:
 
> Hi All,
> I am getting error while loading CSV file.
> 
> val 
> datacsv=sqlContext.read.format("com.databricks.spark.csv").option("header",
>  "true").load("timeline.csv")
> java.lang.NoSuchMethodError: 
> org.apache.commons.csv.CSVFormat.withQuote(Ljava/lang/Character;)Lorg/apache/commons/csv/CSVFormat;
> 
> 
> I have added the dependencies in sbt file 
> // Spark Additional Library - CSV Read as DF
> libraryDependencies += "com.databricks" %% "spark-csv" % “1.5.0"
> and starting the spark-shell with command
> 
> spark-shell --master yarn-client  --jars 
> /opt/packages/-data-prepration/target/scala-2.10/-data-prepration-assembly-1.0.jar
>  --name nayan 
> 
> 
> 
> Thanks for any help!!
> 
> 
> Thanks,
> Nayan
>>> 
>> 

Re: Error while reading the CSV

[Jörn Franke]

Maybe your Spark is based on scala 2.11, but you compile it for 2.10 or the 
other way around?

> On 6. Apr 2017, at 15:54, nayan sharma  wrote:
> 
> In addition I am using spark version 1.6.2
> Is there any chance of error coming because of Scala version or dependencies 
> are not matching.?I just guessed.
> 
> Thanks,
> Nayan
> 
>  
>> On 06-Apr-2017, at 7:16 PM, nayan sharma  wrote:
>> 
>> Hi Jorn,
>> Thanks for replying.
>> 
>> jar -tf catalyst-data-prepration-assembly-1.0.jar | grep csv
>> 
>> after doing this I have found a lot of classes under 
>> com/databricks/spark/csv/
>> 
>> do I need to check for any specific class ??
>> 
>> Regards,
>> Nayan
>>> On 06-Apr-2017, at 6:42 PM, Jörn Franke  wrote:
>>> 
>>> Is the library in your assembly jar?
>>> 
 On 6. Apr 2017, at 15:06, nayan sharma  wrote:
 
 Hi All,
 I am getting error while loading CSV file.
 
 val 
 datacsv=sqlContext.read.format("com.databricks.spark.csv").option("header",
  "true").load("timeline.csv")
 java.lang.NoSuchMethodError: 
 org.apache.commons.csv.CSVFormat.withQuote(Ljava/lang/Character;)Lorg/apache/commons/csv/CSVFormat;
 
 
 I have added the dependencies in sbt file 
 // Spark Additional Library - CSV Read as DF
 libraryDependencies += "com.databricks" %% "spark-csv" % “1.5.0"
 and starting the spark-shell with command
 
 spark-shell --master yarn-client  --jars 
 /opt/packages/-data-prepration/target/scala-2.10/-data-prepration-assembly-1.0.jar
  --name nayan 
 
 
 
 Thanks for any help!!
 
 
 Thanks,
 Nayan
>> 
> 

Re: Error while reading the CSV

[nayan sharma]

In addition I am using spark version 1.6.2
Is there any chance of error coming because of Scala version or dependencies 
are not matching.?I just guessed.

Thanks,
Nayan

 
> On 06-Apr-2017, at 7:16 PM, nayan sharma  wrote:
> 
> Hi Jorn,
> Thanks for replying.
> 
> jar -tf catalyst-data-prepration-assembly-1.0.jar | grep csv
> 
> after doing this I have found a lot of classes under 
> com/databricks/spark/csv/
> 
> do I need to check for any specific class ??
> 
> Regards,
> Nayan
>> On 06-Apr-2017, at 6:42 PM, Jörn Franke > > wrote:
>> 
>> Is the library in your assembly jar?
>> 
>> On 6. Apr 2017, at 15:06, nayan sharma > > wrote:
>> 
>>> Hi All,
>>> I am getting error while loading CSV file.
>>> 
>>> val 
>>> datacsv=sqlContext.read.format("com.databricks.spark.csv").option("header", 
>>> "true").load("timeline.csv")
>>> java.lang.NoSuchMethodError: 
>>> org.apache.commons.csv.CSVFormat.withQuote(Ljava/lang/Character;)Lorg/apache/commons/csv/CSVFormat;
>>> 
>>> 
>>> I have added the dependencies in sbt file 
>>> // Spark Additional Library - CSV Read as DF
>>> libraryDependencies += "com.databricks" %% "spark-csv" % “1.5.0"
>>> and starting the spark-shell with command
>>> 
>>> spark-shell --master yarn-client  --jars 
>>> /opt/packages/-data-prepration/target/scala-2.10/-data-prepration-assembly-1.0.jar
>>>  --name nayan 
>>> 
>>> 
>>> 
>>> Thanks for any help!!
>>> 
>>> 
>>> Thanks,
>>> Nayan
> 

Re: Error while reading the CSV

[nayan sharma]

Hi Jorn,
Thanks for replying.

jar -tf catalyst-data-prepration-assembly-1.0.jar | grep csv

after doing this I have found a lot of classes under 
com/databricks/spark/csv/

do I need to check for any specific class ??

Regards,
Nayan
> On 06-Apr-2017, at 6:42 PM, Jörn Franke  wrote:
> 
> Is the library in your assembly jar?
> 
> On 6. Apr 2017, at 15:06, nayan sharma  > wrote:
> 
>> Hi All,
>> I am getting error while loading CSV file.
>> 
>> val 
>> datacsv=sqlContext.read.format("com.databricks.spark.csv").option("header", 
>> "true").load("timeline.csv")
>> java.lang.NoSuchMethodError: 
>> org.apache.commons.csv.CSVFormat.withQuote(Ljava/lang/Character;)Lorg/apache/commons/csv/CSVFormat;
>> 
>> 
>> I have added the dependencies in sbt file 
>> // Spark Additional Library - CSV Read as DF
>> libraryDependencies += "com.databricks" %% "spark-csv" % “1.5.0"
>> and starting the spark-shell with command
>> 
>> spark-shell --master yarn-client  --jars 
>> /opt/packages/-data-prepration/target/scala-2.10/-data-prepration-assembly-1.0.jar
>>  --name nayan 
>> 
>> 
>> 
>> Thanks for any help!!
>> 
>> 
>> Thanks,
>> Nayan

Re: Error while reading the CSV

[Jörn Franke]

Is the library in your assembly jar?

> On 6. Apr 2017, at 15:06, nayan sharma  wrote:
> 
> Hi All,
> I am getting error while loading CSV file.
> 
> val 
> datacsv=sqlContext.read.format("com.databricks.spark.csv").option("header", 
> "true").load("timeline.csv")
> java.lang.NoSuchMethodError: 
> org.apache.commons.csv.CSVFormat.withQuote(Ljava/lang/Character;)Lorg/apache/commons/csv/CSVFormat;
> 
> 
> I have added the dependencies in sbt file 
> // Spark Additional Library - CSV Read as DF
> libraryDependencies += "com.databricks" %% "spark-csv" % “1.5.0"
> and starting the spark-shell with command
> 
> spark-shell --master yarn-client  --jars 
> /opt/packages/-data-prepration/target/scala-2.10/-data-prepration-assembly-1.0.jar
>  --name nayan 
> 
> 
> 
> Thanks for any help!!
> 
> 
> Thanks,
> Nayan

Error while reading the CSV

[nayan sharma]

Hi All,
I am getting error while loading CSV file.

val datacsv=sqlContext.read.format("com.databricks.spark.csv").option("header", 
"true").load("timeline.csv")
java.lang.NoSuchMethodError: 
org.apache.commons.csv.CSVFormat.withQuote(Ljava/lang/Character;)Lorg/apache/commons/csv/CSVFormat;


I have added the dependencies in sbt file 
// Spark Additional Library - CSV Read as DF
libraryDependencies += "com.databricks" %% "spark-csv" % “1.5.0"
and starting the spark-shell with command

spark-shell --master yarn-client  --jars 
/opt/packages/-data-prepration/target/scala-2.10/-data-prepration-assembly-1.0.jar
 --name nayan 



Thanks for any help!!


Thanks,
Nayan