Re: Issue with SparkR setup on RStudio

2017-01-04 Thread Md. Rezaul Karim 
Cheung,

The problem has been solved after switching from Windows to Linux
environment.

Thanks.



Regards,
_
*Md. Rezaul Karim* BSc, MSc
PhD Researcher, INSIGHT Centre for Data Analytics
National University of Ireland, Galway
IDA Business Park, Dangan, Galway, Ireland
Web: http://www.reza-analytics.eu/index.html
<http://139.59.184.114/index.html>

On 2 January 2017 at 18:59, Felix Cheung <felixcheun...@hotmail.com> wrote:

> Perhaps it is with
>
> spark.sql.warehouse.dir="E:/Exp/"
>
> That you have in the sparkConfig parameter.
>
> Unfortunately the exception stack is fairly far away from the actual
> error, but from the top of my head spark.sql.warehouse.dir and HADOOP_HOME
> are the two different pieces that is not set in the Windows tests.
>
>
> _
> From: Md. Rezaul Karim <rezaul.ka...@insight-centre.org>
> Sent: Monday, January 2, 2017 7:58 AM
> Subject: Re: Issue with SparkR setup on RStudio
> To: Felix Cheung <felixcheun...@hotmail.com>
> Cc: spark users <user@spark.apache.org>
>
>
> Hello Cheung,
>
> Happy New Year!
>
> No, I did not configure Hive on my machine. Even I have tried not setting
> the HADOOP_HOME but getting the same error.
>
>
>
> Regards,
> _
> *Md. Rezaul Karim* BSc, MSc
> PhD Researcher, INSIGHT Centre for Data Analytics
> National University of Ireland, Galway
> IDA Business Park, Dangan, Galway, Ireland
> Web:http://www.reza-analytics.eu/index.html
> <http://139.59.184.114/index.html>
>
> On 29 December 2016 at 19:16, Felix Cheung <felixcheun...@hotmail.com>
> wrote:
>
>> Any reason you are setting HADOOP_HOME?
>>
>> From the error it seems you are running into issue with Hive config
>> likely with trying to load hive-site.xml. Could you try not setting
>> HADOOP_HOME
>>
>>
>> --
>> *From:* Md. Rezaul Karim <rezaul.ka...@insight-centre.org>
>> *Sent:* Thursday, December 29, 2016 10:24:57 AM
>> *To:* spark users
>> *Subject:* Issue with SparkR setup on RStudio
>>
>>
>> Dear Spark users,
>>
>> I am trying to setup SparkR on RStudio to perform some basic data
>> manipulations and MLmodeling.  However, I am a strange error while
>> creating SparkR session or DataFrame that 
>> says:java.lang.IllegalArgumentException
>> Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState.
>>
>> According to Spark documentation athttp://spark.apache.org/
>> docs/latest/sparkr.html#starting-up-sparksession, I don’t need to
>> configure Hive path or related variables.
>>
>> I have the following source code:
>>
>> SPARK_HOME = "C:/spark-2.1.0-bin-hadoop2.7"
>> HADOOP_HOME= "C:/spark-2.1.0-bin-hadoop2.7/bin/"
>>
>> library(SparkR, lib.loc = c(file.path(Sys.getenv("SPARK_HOME"), "R",
>> "lib")))
>> sparkR.session(appName = "SparkR-DataFrame-example", master = "local[*]",
>> sparkConfig = list(spark.sql.warehouse.dir="E:/Exp/",
>> spark.driver.memory = "8g"), enableHiveSupport = TRUE)
>>
>> # Create a simple local data.frame
>> localDF <- data.frame(name=c("John", "Smith", "Sarah"), age=c(19, 23, 18))
>> # Convert local data frame to a SparkDataFrame
>> df <- createDataFrame(localDF)
>> print(df)
>> head(df)
>> sparkR.session.stop()
>>
>> Please note that the HADOOP_HOME contains the ‘*winutils.exe’* file. The
>> details of the eror is as follows:
>>
>> Error in handleErrors(returnStatus, conn) :  
>> java.lang.IllegalArgumentException: Error while instantiating 
>> 'org.apache.spark.sql.hive.HiveSessionState':
>>
>>at 
>> org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$reflect(SparkSession.scala:981)
>>
>>at 
>> org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:110)
>>
>>at 
>> org.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:109)
>>
>>at 
>> org.apache.spark.sql.api.r.SQLUtils$$anonfun$setSparkContextSessionConf$2.apply(SQLUtils.scala:67)
>>
>>at 
>> org.apache.spark.sql.api.r.SQLUtils$$anonfun$setSparkContextSessionConf$2.apply(SQLUtils.scala:66)
>>
>>at 
>> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
>>
>>at scala.colle

Re: Issue with SparkR setup on RStudio

2017-01-02 Thread Felix Cheung 
Perhaps it is with

spark.sql.warehouse.dir="E:/Exp/"

That you have in the sparkConfig parameter.

Unfortunately the exception stack is fairly far away from the actual error, but 
from the top of my head spark.sql.warehouse.dir and HADOOP_HOME are the two 
different pieces that is not set in the Windows tests.


_
From: Md. Rezaul Karim 
<rezaul.ka...@insight-centre.org<mailto:rezaul.ka...@insight-centre.org>>
Sent: Monday, January 2, 2017 7:58 AM
Subject: Re: Issue with SparkR setup on RStudio
To: Felix Cheung <felixcheun...@hotmail.com<mailto:felixcheun...@hotmail.com>>
Cc: spark users <user@spark.apache.org<mailto:user@spark.apache.org>>


Hello Cheung,

Happy New Year!

No, I did not configure Hive on my machine. Even I have tried not setting the 
HADOOP_HOME but getting the same error.



Regards,
_
Md. Rezaul Karim BSc, MSc
PhD Researcher, INSIGHT Centre for Data Analytics
National University of Ireland, Galway
IDA Business Park, Dangan, Galway, Ireland
Web:http://www.reza-analytics.eu/index.html<http://139.59.184.114/index.html>

On 29 December 2016 at 19:16, Felix Cheung 
<felixcheun...@hotmail.com<mailto:felixcheun...@hotmail.com>> wrote:
Any reason you are setting HADOOP_HOME?

>From the error it seems you are running into issue with Hive config likely 
>with trying to load hive-site.xml. Could you try not setting HADOOP_HOME



From: Md. Rezaul Karim 
<rezaul.ka...@insight-centre.org<mailto:rezaul.ka...@insight-centre.org>>
Sent: Thursday, December 29, 2016 10:24:57 AM
To: spark users
Subject: Issue with SparkR setup on RStudio

Dear Spark users,
I am trying to setup SparkR on RStudio to perform some basic data manipulations 
and MLmodeling.  However, I am a strange error while creating SparkR session or 
DataFrame that says:java.lang.IllegalArgumentException Error while 
instantiating 'org.apache.spark.sql.hive.HiveSessionState.
According to Spark documentation 
athttp://spark.apache.org/docs/latest/sparkr.html#starting-up-sparksession, I 
don’t need to configure Hive path or related variables.
I have the following source code:

SPARK_HOME = "C:/spark-2.1.0-bin-hadoop2.7"
HADOOP_HOME= "C:/spark-2.1.0-bin-hadoop2.7/bin/"

library(SparkR, lib.loc = c(file.path(Sys.getenv("SPARK_HOME"), "R", "lib")))
sparkR.session(appName = "SparkR-DataFrame-example", master = "local[*]", 
sparkConfig = list(spark.sql.warehouse.dir="E:/Exp/", spark.driver.memory = 
"8g"), enableHiveSupport = TRUE)

# Create a simple local data.frame
localDF <- data.frame(name=c("John", "Smith", "Sarah"), age=c(19, 23, 18))
# Convert local data frame to a SparkDataFrame
df <- createDataFrame(localDF)
print(df)
head(df)
sparkR.session.stop()
Please note that the HADOOP_HOME contains the ‘winutils.exe’ file. The details 
of the eror is as follows:

Error in handleErrors(returnStatus, conn) :  
java.lang.IllegalArgumentException: Error while instantiating 
'org.apache.spark.sql.hive.HiveSessionState':

   at 
org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$reflect(SparkSession.scala:981)

   at 
org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:110)

   at 
org.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:109)

   at 
org.apache.spark.sql.api.r.SQLUtils$$anonfun$setSparkContextSessionConf$2.apply(SQLUtils.scala:67)

   at 
org.apache.spark.sql.api.r.SQLUtils$$anonfun$setSparkContextSessionConf$2.apply(SQLUtils.scala:66)

   at 
scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)

   at scala.collection.Iterator$class.foreach(Iterator.scala:893)

   at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)

   at 
scala.collection.IterableLike$class.foreach(IterableLike.scala:72)

   at scala.collection.AbstractIterable.foreach(Iterable.scala:54)

   at scala.collection.Traversabl



 Any kind of help would be appreciated.


Regards,
_
Md. Rezaul Karim BSc, MSc
PhD Researcher, INSIGHT Centre for Data Analytics
National University of Ireland, Galway
IDA Business Park, Dangan, Galway, Ireland
Web:http://www.reza-analytics.eu/index.html<http://139.59.184.114/index.html>

Re: Issue with SparkR setup on RStudio

2017-01-02 Thread Md. Rezaul Karim 
Hello Cheung,

Happy New Year!

No, I did not configure Hive on my machine. Even I have tried not setting
the HADOOP_HOME but getting the same error.



Regards,
_
*Md. Rezaul Karim* BSc, MSc
PhD Researcher, INSIGHT Centre for Data Analytics
National University of Ireland, Galway
IDA Business Park, Dangan, Galway, Ireland
Web: http://www.reza-analytics.eu/index.html


On 29 December 2016 at 19:16, Felix Cheung 
wrote:

> Any reason you are setting HADOOP_HOME?
>
> From the error it seems you are running into issue with Hive config likely
> with trying to load hive-site.xml. Could you try not setting HADOOP_HOME
>
>
> --
> *From:* Md. Rezaul Karim 
> *Sent:* Thursday, December 29, 2016 10:24:57 AM
> *To:* spark users
> *Subject:* Issue with SparkR setup on RStudio
>
>
> Dear Spark users,
>
> I am trying to setup SparkR on RStudio to perform some basic data
> manipulations and ML modeling.  However, I am a strange error while
> creating SparkR session or DataFrame that says: 
> java.lang.IllegalArgumentException
> Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState.
>
> According to Spark documentation at http://spark.apache.org/docs/
> latest/sparkr.html#starting-up-sparksession, I don’t need to configure
> Hive path or related variables.
>
> I have the following source code:
>
> SPARK_HOME = "C:/spark-2.1.0-bin-hadoop2.7"
> HADOOP_HOME= "C:/spark-2.1.0-bin-hadoop2.7/bin/"
>
> library(SparkR, lib.loc = c(file.path(Sys.getenv("SPARK_HOME"), "R",
> "lib")))
> sparkR.session(appName = "SparkR-DataFrame-example", master = "local[*]",
> sparkConfig = list(spark.sql.warehouse.dir="E:/Exp/", spark.driver.memory
> = "8g"), enableHiveSupport = TRUE)
>
> # Create a simple local data.frame
> localDF <- data.frame(name=c("John", "Smith", "Sarah"), age=c(19, 23, 18))
> # Convert local data frame to a SparkDataFrame
> df <- createDataFrame(localDF)
> print(df)
> head(df)
> sparkR.session.stop()
>
> Please note that the HADOOP_HOME  contains the ‘*winutils.exe’* file. The
> details of the eror is as follows:
>
> Error in handleErrors(returnStatus, conn) :  
> java.lang.IllegalArgumentException: Error while instantiating 
> 'org.apache.spark.sql.hive.HiveSessionState':
>
>at 
> org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$reflect(SparkSession.scala:981)
>
>at 
> org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:110)
>
>at 
> org.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:109)
>
>at 
> org.apache.spark.sql.api.r.SQLUtils$$anonfun$setSparkContextSessionConf$2.apply(SQLUtils.scala:67)
>
>at 
> org.apache.spark.sql.api.r.SQLUtils$$anonfun$setSparkContextSessionConf$2.apply(SQLUtils.scala:66)
>
>at 
> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
>
>at scala.collection.Iterator$class.foreach(Iterator.scala:893)
>
>at 
> scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
>
>at 
> scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
>
>at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
>
>at scala.collection.Traversabl
>
>
>  Any kind of help would be appreciated.
>
>
>
>
> Regards,
> _
> *Md. Rezaul Karim* BSc, MSc
> PhD Researcher, INSIGHT Centre for Data Analytics
> National University of Ireland, Galway
> IDA Business Park, Dangan, Galway, Ireland
> Web: http://www.reza-analytics.eu/index.html
> 
>

Re: Issue with SparkR setup on RStudio

2016-12-29 Thread Felix Cheung 
Any reason you are setting HADOOP_HOME?

>From the error it seems you are running into issue with Hive config likely 
>with trying to load hive-site.xml. Could you try not setting HADOOP_HOME



From: Md. Rezaul Karim 
Sent: Thursday, December 29, 2016 10:24:57 AM
To: spark users
Subject: Issue with SparkR setup on RStudio

Dear Spark users,
I am trying to setup SparkR on RStudio to perform some basic data manipulations 
and ML modeling.  However, I am a strange error while creating SparkR session 
or DataFrame that says: java.lang.IllegalArgumentException Error while 
instantiating 'org.apache.spark.sql.hive.HiveSessionState.
According to Spark documentation at 
http://spark.apache.org/docs/latest/sparkr.html#starting-up-sparksession, I 
don't need to configure Hive path or related variables.
I have the following source code:

SPARK_HOME = "C:/spark-2.1.0-bin-hadoop2.7"
HADOOP_HOME= "C:/spark-2.1.0-bin-hadoop2.7/bin/"

library(SparkR, lib.loc = c(file.path(Sys.getenv("SPARK_HOME"), "R", "lib")))
sparkR.session(appName = "SparkR-DataFrame-example", master = "local[*]", 
sparkConfig = list(spark.sql.warehouse.dir="E:/Exp/", spark.driver.memory = 
"8g"), enableHiveSupport = TRUE)

# Create a simple local data.frame
localDF <- data.frame(name=c("John", "Smith", "Sarah"), age=c(19, 23, 18))
# Convert local data frame to a SparkDataFrame
df <- createDataFrame(localDF)
print(df)
head(df)
sparkR.session.stop()
Please note that the HADOOP_HOME  contains the 'winutils.exe' file. The details 
of the eror is as follows:

Error in handleErrors(returnStatus, conn) :  
java.lang.IllegalArgumentException: Error while instantiating 
'org.apache.spark.sql.hive.HiveSessionState':

   at 
org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$reflect(SparkSession.scala:981)

   at 
org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:110)

   at 
org.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:109)

   at 
org.apache.spark.sql.api.r.SQLUtils$$anonfun$setSparkContextSessionConf$2.apply(SQLUtils.scala:67)

   at 
org.apache.spark.sql.api.r.SQLUtils$$anonfun$setSparkContextSessionConf$2.apply(SQLUtils.scala:66)

   at 
scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)

   at scala.collection.Iterator$class.foreach(Iterator.scala:893)

   at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)

   at 
scala.collection.IterableLike$class.foreach(IterableLike.scala:72)

   at scala.collection.AbstractIterable.foreach(Iterable.scala:54)

   at scala.collection.Traversabl



 Any kind of help would be appreciated.


Regards,
_
Md. Rezaul Karim BSc, MSc
PhD Researcher, INSIGHT Centre for Data Analytics
National University of Ireland, Galway
IDA Business Park, Dangan, Galway, Ireland
Web: http://www.reza-analytics.eu/index.html