Re: sparkR ORC support.

at > spark_api.R#108 > > On Wed, Jan 6, 2016 at 10:30 AM, Felix Cheung <felixcheun...@hotmail.com> > wrote: > > Firstly I don't have ORC data to verify but this should work: > > df <- loadDF(sqlContext, "data/path", "orc") > > Secondly, could you check i

Re: sparkR ORC support.

t;- sparkR.init() >> hivecontext <- sparkRHive.init(sc) >> df <- loadDF(hivecontext, "/data/ingest/sparktest1/", "orc") >> >> >> >> -- >> Date: Tue, 12 Jan 2016 14:28:58 +0530 >> Subject: Re: s

RE: sparkR ORC support.

Hive.init(sc)df <- loadDF(hivecontext, "/data/ingest/sparktest1/", "orc") Date: Tue, 12 Jan 2016 14:28:58 +0530 Subject: Re: sparkR ORC support. From: sand...@infoworks.io To: felixcheun...@hotmail.com CC: yblia...@gmail.com; user@spark.apache.org; premsure...@gmail.com;

Re: sparkR ORC support.

t; hivecontext <- sparkRHive.init(sc) > df <- loadDF(hivecontext, "/data/ingest/sparktest1/", "orc") > > > > -- > Date: Tue, 12 Jan 2016 14:28:58 +0530 > Subject: Re: sparkR ORC support. > From: sand...@infoworks.i

Re: sparkR ORC support.

;>> >>> >>> Sys.setenv(SPARK_HOME="/usr/hdp/current/spark-client") >>> >>> .libPaths(c(file.path(Sys.getenv("SPARK_HOME"), "R", "lib"), >>> .libPaths())) >>> library(SparkR) >>> >>> sc <

Re: sparkR ORC support.

1/", "orc") > > > Is there a reason you want to call stop? If you do, you would need to call > the line hivecontext <- sparkRHive.init(sc) again. > > > _ > From: Sandeep Khurana <sand...@infoworks.io> > Sent: Tuesday, Jan

Re: sparkR ORC support.

would need to call the line hivecontext <- sparkRHive.init(sc) again. _ From: Sandeep Khurana <sand...@infoworks.io> Sent: Tuesday, January 12, 2016 5:20 AM Subject: Re: sparkR ORC support. To: Felix Cheung <felixcheun...@hotmail.com> Cc: spark users

Re: sparkR ORC support.

there is any error > message there. > > _ > From: Prem Sure <premsure...@gmail.com> > Sent: Tuesday, January 5, 2016 8:12 AM > Subject: Re: sparkR ORC support. > To: Sandeep Khurana <sand...@infoworks.io> > Cc: spark users <user@spark.

Re: sparkR ORC support.

after sparkR.init() - please check if there is any error >> message there. >> >> _ >> From: Prem Sure <premsure...@gmail.com> >> Sent: Tuesday, January 5, 2016 8:12 AM >> Subject: Re: sparkR ORC support. >> To: Sandeep Khurana &

Re: sparkR ORC support.

o verify but this should work: >> >> df <- loadDF(sqlContext, "data/path", "orc") >> >> Secondly, could you check if sparkR.stop() was called? sparkRHive.init() >> should be called after sparkR.init() - please check if there is any error >> messag

Re: sparkR ORC support.

Yes Sandeep, also copy hive-site.xml too to spark conf directory. On Tue, Jan 5, 2016 at 10:07 AM, Sandeep Khurana wrote: > Also, do I need to setup hive in spark as per the link > http://stackoverflow.com/questions/26360725/accesing-hive-tables-in-spark > ? > > We might

Re: sparkR ORC support.

Also, do I need to setup hive in spark as per the link http://stackoverflow.com/questions/26360725/accesing-hive-tables-in-spark ? We might need to copy hdfs-site.xml file to spark conf directory ? On Tue, Jan 5, 2016 at 8:28 PM, Sandeep Khurana wrote: > Deepak > > Tried

Re: sparkR ORC support.

Deepak Tried this. Getting this error now rror in sql(hivecontext, "FROM CATEGORIES SELECT category_id", "") : unused argument ("") On Tue, Jan 5, 2016 at 6:48 PM, Deepak Sharma wrote: > Hi Sandeep > can you try this ? > > results <- sql(hivecontext, "FROM test

Re: sparkR ORC support.

re is any error message there. _ From: Prem Sure <premsure...@gmail.com> Sent: Tuesday, January 5, 2016 8:12 AM Subject: Re: sparkR ORC support. To: Sandeep Khurana <sand...@infoworks.io> Cc: spark users <user@spark.apache.org>, Deepak Sharma <deepakmc...@gmail.com>

sparkR ORC support.

Hello I need to read an ORC files in hdfs in R using spark. I am not able to find a package to do that. Can anyone help with documentation or example for this purpose? -- Architect Infoworks.io http://Infoworks.io

Re: sparkR ORC support.

Hi Sandeep can you try this ? results <- sql(hivecontext, "FROM test SELECT id","") Thanks Deepak On Tue, Jan 5, 2016 at 5:49 PM, Sandeep Khurana wrote: > Thanks Deepak. > > I tried this as well. I created a hivecontext with "hivecontext <<- > sparkRHive.init(sc) "

Re: sparkR ORC support.

Hi Sandeep I am not sure if ORC can be read directly in R. But there can be a workaround .First create hive table on top of ORC files and then access hive table in R. Thanks Deepak On Tue, Jan 5, 2016 at 4:57 PM, Sandeep Khurana wrote: > Hello > > I need to read an ORC

Re: sparkR ORC support.

Thanks Deepak. I tried this as well. I created a hivecontext with "hivecontext <<- sparkRHive.init(sc) " . When I tried to read hive table from this , results <- sql(hivecontext, "FROM test SELECT id") I get below error, Error in callJMethod(sqlContext, "sql", sqlQuery) : Invalid jobj