Yes, that is expected to move on. If it looks it is waiting for something,
my first instinct would be to check network connectivity such as your
cluster must have access back to your Mac to read the file (it is probably
waiting to time out)
On Mon, Jul 11, 2016 at 12:59 PM, Jean Georges Perrin <j...@jgp.net> wrote:
> Good for the file :)
>
> No it goes on... Like if it was waiting for something
>
> jg
>
>
> On Jul 10, 2016, at 22:55, ayan guha <guha.a...@gmail.com> wrote:
>
> Is this terminating the execution or spark application still runs after
> this error?
>
> One thing for sure, it is looking for local file on driver (ie your mac) @
> location: file:/Users/jgp/Documents/Data/restaurants-data.json
>
> On Mon, Jul 11, 2016 at 12:33 PM, Jean Georges Perrin <j...@jgp.net> wrote:
>
>>
>> I have my dev environment on my Mac. I have a dev Spark server on a
>> freshly installed physical Ubuntu box.
>>
>> I had some connection issues, but it is now all fine.
>>
>> In my code, running on the Mac, I have:
>>
>> 1 SparkConf conf = new SparkConf().setAppName("myapp").setMaster("
>> spark://10.0.100.120:7077");
>> 2 JavaSparkContext javaSparkContext = new JavaSparkContext(conf);
>> 3 javaSparkContext.setLogLevel("WARN");
>> 4 SQLContext sqlContext = new SQLContext(javaSparkContext);
>> 5
>> 6 // Restaurant Data
>> 7 df = sqlContext.read().option("dateFormat", "yyyy-mm-dd").json(source
>> .getLocalStorage());
>>
>>
>> 1) Clarification question: This code runs on my mac, connects to the
>> server, but line #7 assumes the file is on my mac, not on the server, right?
>>
>> 2) On line 7, I get an exception:
>>
>> 16-07-10 22:20:04:143 DEBUG - address: jgp-MacBook-Air.local/
>> 10.0.100.100 isLoopbackAddress: false, with host 10.0.100.100
>> jgp-MacBook-Air.local
>> 16-07-10 22:20:04:240 INFO
>> org.apache.spark.sql.execution.datasources.json.JSONRelation - Listing
>> file:/Users/jgp/Documents/Data/restaurants-data.json on driver
>> 16-07-10 22:20:04:288 DEBUG org.apache.hadoop.util.Shell - Failed to
>> detect a valid hadoop home directory
>> java.io.IOException: HADOOP_HOME or hadoop.home.dir are not set.
>> at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:225)
>> at org.apache.hadoop.util.Shell.<clinit>(Shell.java:250)
>> at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:76)
>> at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(
>> FileInputFormat.java:447)
>> at org.apache.spark.sql.execution.datasources.json.JSONRelation.org
>> $apache$spark$sql$execution$datasources$json$JSONRelation$$createBaseRdd(JSONRelation.scala:98)
>> at
>> org.apache.spark.sql.execution.datasources.json.JSONRelation$$anonfun$4$$anonfun$apply$1.apply(JSONRelation.scala:115)
>> at
>> org.apache.spark.sql.execution.datasources.json.JSONRelation$$anonfun$4$$anonfun$apply$1.apply(JSONRelation.scala:115)
>> at scala.Option.getOrElse(Option.scala:120)
>> at
>> org.apache.spark.sql.execution.datasources.json.JSONRelation$$anonfun$4.apply(JSONRelation.scala:115)
>> at
>> org.apache.spark.sql.execution.datasources.json.JSONRelation$$anonfun$4.apply(JSONRelation.scala:109)
>> at scala.Option.getOrElse(Option.scala:120)
>>
>> Do I have to install HADOOP on the server? - I imagine that from:
>> java.io.IOException: HADOOP_HOME or hadoop.home.dir are not set.
>>
>> TIA,
>>
>> jg
>>
>>
>
>
> --
> Best Regards,
> Ayan Guha
>
>
--
Best Regards,
Ayan Guha
