of warehouse. All the users get translated to hive
when interacting with hiveserver2. But i think HiveContext is bypassing
hiveserver2.
On Wednesday, June 17, 2015, ayan guha guha.a...@gmail.com wrote:
Try to grant read execute access through sentry.
On 18 Jun 2015 05:47, Nitin kak nitinkak
Hey, I have exactly this question. Did you get an answer to it?
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Does-HiveContext-connect-to-HiveServer2-tp22200p23431.html
Sent from the Apache Spark User List mailing list archive at Nabble.com
: Mon, 20 Apr 2015 14:03:55 -0700
Subject: Re: GSSException when submitting Spark job in yarn-cluster mode
with HiveContext APIs on Kerberos cluster
From: van...@cloudera.com
To: alee...@hotmail.com
CC: user@spark.apache.org
I think you want to take a look at:
https
HiveContext is bypassing
hiveserver2.
On Wednesday, June 17, 2015, ayan guha guha.a...@gmail.com wrote:
Try to grant read execute access through sentry.
On 18 Jun 2015 05:47, Nitin kak nitinkak...@gmail.com wrote:
I am trying to run a hive query from Spark code using HiveContext
object
Hi All.
I have a partitioned table in Hive. The use case is to drop one of the
partitions before inserting new data every time the Spark process runs. I
am using the Hivecontext to read and write (dynamic partitions) and also to
alter the table to drop the partition before insert. Everything runs
and in Row to
the end of the sequence, then it works correctly...
On 16. juni 2015 11:14, patcharee wrote:
Hi,
I am using spark 1.4 and HiveContext to append data into a partitioned
hive table. I found that the data insert into the table is correct, but the
partition(folder) created
using spark 1.4 and HiveContext to append data into a partitioned
hive table. I found that the data insert into the table is correct, but the
partition(folder) created is totally wrong.
Below is my code snippet
I am trying to run a hive query from Spark code using HiveContext object.
It was running fine earlier but since the Apache Sentry has been set
installed the process is failing with this exception :
*org.apache.hadoop.security.AccessControlException: Permission denied:
user=kakn, access
Try to grant read execute access through sentry.
On 18 Jun 2015 05:47, Nitin kak nitinkak...@gmail.com wrote:
I am trying to run a hive query from Spark code using HiveContext object.
It was running fine earlier but since the Apache Sentry has been set
installed the process is failing
With Sentry, only hive user has the permission for read/write/execute on
the subdirectories of warehouse. All the users get translated to hive
when interacting with hiveserver2. But i think HiveContext is bypassing
hiveserver2.
On Wednesday, June 17, 2015, ayan guha guha.a...@gmail.com wrote
With Sentry, only hive user has the permission for read/write/execute on
the subdirectories of warehouse. All the users get translated to hive
when interacting with hiveserver2. But i think HiveContext is bypassing
hiveserver2.
On Wednesday, June 17, 2015, ayan guha guha.a...@gmail.com wrote
using spark 1.4 and HiveContext to append data into a
partitioned hive table. I found that the data insert into the table
is correct, but the partition(folder) created is totally wrong.
Below is my code snippet
I am trying to run a hive query from Spark code using HiveContext object. It
was running fine earlier but since the Apache Sentry has been set installed
the process is failing with this exception :
/org.apache.hadoop.security.AccessControlException: Permission denied:
user=kakn, access
Hi there!
It seems like you have Read/Execute access permission (and no
update/insert/delete access). What operation are you performing?
Ajay
On Jun 17, 2015, at 5:24 PM, nitinkak001 nitinkak...@gmail.com wrote:
I am trying to run a hive query from Spark code using HiveContext object
Yes, it is thread safe. That’s how Spark SQL JDBC Server works.
Cheng Hao
From: V Dineshkumar [mailto:developer.dines...@gmail.com]
Sent: Wednesday, June 17, 2015 9:44 PM
To: user@spark.apache.org
Subject: Is HiveContext Thread Safe?
Hi,
I have a HiveContext which I am using in multiple
Hi,
I have a HiveContext which I am using in multiple threads to submit a
Spark SQL query using *sql* method.
I just wanted to know whether this method is thread-safe or not?Will all my
queries be submitted at the same time independent of each other or will be
submitted sequential one after
I found if I move the partitioned columns in schemaString and in Row to
the end of the sequence, then it works correctly...
On 16. juni 2015 11:14, patcharee wrote:
Hi,
I am using spark 1.4 and HiveContext to append data into a partitioned
hive table. I found that the data insert
Hi,
I am using spark 1.4 and HiveContext to append data into a partitioned
hive table. I found that the data insert into the table is correct, but
the partition(folder) created is totally wrong.
Below is my code snippet
I got it. Thanks!
Patcharee
On 13/06/15 23:00, Will Briggs wrote:
The context that is created by spark-shell is actually an instance of
HiveContext. If you want to use it programmatically in your driver, you need to
make sure that your context is a HiveContext, and not a SQLContext.
https
As the error message says, were you using a |SQLContext| instead of a
|HiveContext| to create the DataFrame?
In Spark shell, although the variable name is |sqlContext|, the type of
that variable is actually |org.apache.spark.sql.hive.HiveContext|, which
has the ability to communicate
by spark-submit I get
Exception in thread main java.lang.RuntimeException: Tables created
with SQLContext must be TEMPORARY. Use a HiveContext instead
But the job works fine on spark-shell. What can be wrong?
BR,
Patcharee
Are you using RC4?
On Wed, Jun 3, 2015 at 10:58 PM, Night Wolf nightwolf...@gmail.com wrote:
Thanks Yin, that seems to work with the Shell. But on a compiled
application with Spark-submit it still fails with the same exception.
On Thu, Jun 4, 2015 at 2:46 PM, Yin Huai yh...@databricks.com
Thanks Yin, that seems to work with the Shell. But on a compiled
application with Spark-submit it still fails with the same exception.
On Thu, Jun 4, 2015 at 2:46 PM, Yin Huai yh...@databricks.com wrote:
Can you put the following setting in spark-defaults.conf and try again?
Hi all,
Trying out Spark 1.4 RC4 on MapR4/Hadoop 2.5.1 running in yarn-client mode with
Hive support.
*Build command;*
./make-distribution.sh --name mapr4.0.2_yarn_j6_2.10 --tgz -Pyarn -Pmapr4
-Phadoop-2.4 -Pmapr4 -Phive -Phadoop-provided
-Dhadoop.version=2.5.1-mapr-1501
...@gmail.com
wrote:
I am trying to run a Hive query from Spark using HiveContext. Here is the
code
/ val conf = new SparkConf().setAppName(HiveSparkIntegrationTest)
conf.set(spark.executor.extraClassPath,
/opt/cloudera/parcels/CDH-5.2.0-1.cdh5.2.0.p0.36/lib/hive/lib);
conf.set
:
I am trying to run a Hive query from Spark using HiveContext. Here is the
code
/ val conf = new SparkConf().setAppName(HiveSparkIntegrationTest)
conf.set(spark.executor.extraClassPath,
/opt/cloudera/parcels/CDH-5.2.0-1.cdh5.2.0.p0.36/lib/hive/lib);
conf.set
Viswanathan
Subject: HiveContext fails when querying large external Parquet tables
Hi all,
(This email was easier to write in markdown, so I’ve created a gist with its
contents here: https://gist.github.com/ottomata/f91ea76cece97444e269
https://gist.github.com/ottomata/f91ea76cece97444e269
...@wikimedia.org /divdivDate:05/22/2015 3:51 PM (GMT-05:00)
/divdivTo: user@spark.apache.org /divdivCc: Joseph Allemandou
jalleman...@wikimedia.org,Madhumitha Viswanathan mviswanat...@wikimedia.org
/divdivSubject: HiveContext fails when querying large external Parquet
tables /divdiv
/divHi all
upgraded to CDH 5.4.0 which comes with Spark 1.3.0 and Hive
1.1.0. Previously we were on CDH 5.3.x, running Spark 1.2.0 and Hive 0.13.0.
Since upgrading, we can no longer query our large webrequest dataset using
HiveContext. HiveContext + Parquet and other file types work fine with
external
give a partition column without spaces, all works fine.
Please provide your inputs.
Regards,
Sam
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Hive-partition-table-read-using-hiveContext-spark-1-3-1-tp22894.html
Sent from the Apache Spark User List mailing
I don't think that works:
https://cwiki.apache.org/confluence/display/Hive/AdminManual+Configuration
On Tue, May 5, 2015 at 6:25 PM, nitinkak001 nitinkak...@gmail.com wrote:
I am running hive queries from HiveContext, for which we need a
hive-site.xml.
Is it possible to replace it with hive
I am running hive queries from HiveContext, for which we need a
hive-site.xml.
Is it possible to replace it with hive-config.xml? I tried but does not
work. Just want a conformation.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Possible-to-use-hive
(sSET $key=$value)
}
From: madhu phatak
Date: 2015-04-23 02:17
To: Michael Armbrust
CC: Ophir Cohen; Hao Ren; user
Subject: Re: HiveContext setConf seems not stable
Hi,
calling getConf don't solve the issue. Even many hive specific queries are
broken. Seems like no hive configurations are getting
Lately we upgraded our Spark to 1.3.
Not surprisingly, over the way I find few incomputability between the
versions and quite expected.
I found change that I'm interesting to understand it origin.
env: Amazon EMR, Spark 1.3, Hive 0.13, Hadoop 2.4
In Spark 1.2.1 I ran from the code query such:
BTW
This:
hc.sql(show tables).collect
Works great!
On Tue, Apr 21, 2015 at 10:49 AM, Ophir Cohen oph...@gmail.com wrote:
Lately we upgraded our Spark to 1.3.
Not surprisingly, over the way I find few incomputability between the
versions and quite expected.
I found change that I'm
): HiveContext = {
val hc: HiveContext = new HiveContext(sc)
hc.setConf(hive.exec.compress.output, true)
hc.setConf(mapreduce.output.fileoutputformat.compress.codec,
org.apache.hadoop.io.compress.SnappyCodec)
hc.setConf(mapreduce.output.fileoutputformat.compress.type, BLOCK
Hi Marcelo,
Exactly what I need to track, thanks for the JIRA pointer.
Date: Mon, 20 Apr 2015 14:03:55 -0700
Subject: Re: GSSException when submitting Spark job in yarn-cluster mode with
HiveContext APIs on Kerberos cluster
From: van...@cloudera.com
To: alee...@hotmail.com
CC: user
Is HiveContext still preferred over SQLContext?
What are the current (1.3.1) diferences between them?
thanks
Daniel
(hive.metastore.warehouse.dir, warehousePath.toString)
}
Cheers
On Wed, Apr 8, 2015 at 1:07 PM, Daniel Siegmann
daniel.siegm...@teamaol.com wrote:
I am trying to unit test some code which takes an existing HiveContext
and uses it to execute a CREATE TABLE query (among other things).
Unfortunately I've
I am trying to unit test some code which takes an existing HiveContext and
uses it to execute a CREATE TABLE query (among other things). Unfortunately
I've run into some hurdles trying to unit test this, and I'm wondering if
anyone has a good approach.
The metastore DB is automatically created
)
setConf(hive.metastore.warehouse.dir, warehousePath.toString)
}
Cheers
On Wed, Apr 8, 2015 at 1:07 PM, Daniel Siegmann daniel.siegm...@teamaol.com
wrote:
I am trying to unit test some code which takes an existing HiveContext and
uses it to execute a CREATE TABLE query (among other things
}
object Main extends App {
val conf = new SparkConf()
.setAppName(context-test)
.setMaster(local[8])
val sc = new SparkContext(conf)
val hc = new HiveContext(sc)
*hc.setConf(spark.sql.shuffle.partitions, 10)*
* hc.setConf(hive.metastore.warehouse.dir,
/home/spark/hive
HiveContext(sc)
*hc.setConf(spark.sql.shuffle.partitions, 10)*
* hc.setConf(hive.metastore.warehouse.dir,
/home/spark/hive/warehouse_test)*
hc.getAllConfs filter(_._1.contains(warehouse.dir)) foreach println
hc.getAllConfs filter(_._1.contains(shuffle.partitions)) foreach
println
org.apache.spark.{SparkConf, SparkContext}
object Main extends App {
val conf = new SparkConf()
.setAppName(context-test)
.setMaster(local[8])
val sc = new SparkContext(conf)
val hc = new HiveContext(sc)
*hc.setConf(spark.sql.shuffle.partitions, 10)*
* hc.setConf(hive.metastore.warehouse.dir
Try this in Spark shell:
|import org.apache.spark.api.java.JavaSparkContext
import org.apache.spark.sql.hive.HiveContext
val jsc = new JavaSparkContext(sc)
val hc = new HiveContext(jsc.sc)
|
(I never mentioned that JavaSparkContext extends SparkContext…)
Cheng
On 3/30/15 8:28 PM
thanks. That is what I have tried. JavaSparkContext does not extend
SparkContext, it can not be used here.
Anyone else know whether we can use HiveContext with JavaSparkContext, from
API documents, seems this is not supported. thanks.
On Sun, Mar 29, 2015 at 9:24 AM, Cheng Lian lian.cs
You may simply pass in JavaSparkContext.sc
On 3/29/15 9:25 PM, Vincent He wrote:
All,
I try Spark SQL with Java, I find HiveContext does not accept
JavaSparkContext, is this true? Or any special build of Spark I need
to do (I build with Hive and thrift server)? Can we use HiveContext
thanks .
It does not work, and can not pass compile as HiveContext constructor does
not accept JaveSparkContext and JaveSparkContext is not subclass of
SparkContext.
Anyone else have any idea? I suspect this is supported now.
On Sun, Mar 29, 2015 at 8:54 AM, Cheng Lian lian.cs@gmail.com wrote
All,
I try Spark SQL with Java, I find HiveContext does not accept
JavaSparkContext, is this true? Or any special build of Spark I need to do
(I build with Hive and thrift server)? Can we use HiveContext in Java?
thanks in advance.
I mean JavaSparkContext has a field name sc, whose type is
SparkContext. You may pass this sc to HiveContext.
On 3/29/15 9:59 PM, Vincent He wrote:
thanks .
It does not work, and can not pass compile as HiveContext constructor
does not accept JaveSparkContext and JaveSparkContext
I am wondering if HiveContext connects to HiveServer2 or does it work though
Hive CLI. The reason I am asking is because Cloudera has deprecated Hive
CLI.
If the connection is through HiverServer2, is there a way to specify user
credentials?
--
View this message in context:
http://apache
van...@cloudera.com
wrote:
It does neither. If you provide a Hive configuration to Spark,
HiveContext will connect to your metastore server, otherwise it will
create its own metastore in the working directory (IIRC).
On Tue, Mar 24, 2015 at 8:58 AM, nitinkak001 nitinkak...@gmail.com
wrote:
I
It does neither. If you provide a Hive configuration to Spark,
HiveContext will connect to your metastore server, otherwise it will
create its own metastore in the working directory (IIRC).
On Tue, Mar 24, 2015 at 8:58 AM, nitinkak001 nitinkak...@gmail.com wrote:
I am wondering if HiveContext
Another question related to this, how can we propagate the hive-site.xml to
all workers when running in the yarn cluster mode?
On Tue, Mar 24, 2015 at 10:09 AM, Marcelo Vanzin van...@cloudera.com
wrote:
It does neither. If you provide a Hive configuration to Spark,
HiveContext will connect
Initially, an attribute reference (column reference), like selecting a
column from a table, is not resolved since we do not know if the reference
is valid or not (if this column exists in the underlying table). In the
query compilation process, we will first analyze this query and resolved
those
Interesting, I thought the problem is with the method itself.
I will check it soon and update.
Can you elaborate what does it mean the # and the number? Is that a
reference to the field in the rdd?
Thank you,
Ophir
On Mar 17, 2015 7:06 PM, Yin Huai yh...@databricks.com wrote:
Seems
The number is an id we used internally to identify an resolved Attribute.
Looks like basic_null_diluted_d was not resolved since there is no id
associated with it.
On Tue, Mar 17, 2015 at 2:08 PM, Ophir Cohen oph...@gmail.com wrote:
Interesting, I thought the problem is with the method itself.
Thanks you for the answer and one more question: what does it mean
'resolved attribute'?
On Mar 17, 2015 8:14 PM, Yin Huai yh...@databricks.com wrote:
The number is an id we used internally to identify an resolved Attribute.
Looks like basic_null_diluted_d was not resolved since there is no id
Hi Guys,
I'm registering a function using:
sqlc.registerFunction(makeEstEntry,ReutersDataFunctions.makeEstEntry _)
Then I register the table and try to query the table using that function
and I get:
org.apache.spark.sql.catalyst.errors.package$TreeNodeException: Unresolved
attributes:
Seems basic_null_diluted_d was not resolved? Can you check if
basic_null_diluted_d is in you table?
On Tue, Mar 17, 2015 at 9:34 AM, Ophir Cohen oph...@gmail.com wrote:
Hi Guys,
I'm registering a function using:
sqlc.registerFunction(makeEstEntry,ReutersDataFunctions.makeEstEntry _)
Then I
for the response.
For Q1, in my case, I have a tool on SparkShell which serves multiple
users where they can use different Hive installation. I take a look at the
code of HiveContext. It looks like I cannot do that today because catalog
field cannot be changed after initialize.
/* A catalyst
Hao, thanks for the response.
For Q1, in my case, I have a tool on SparkShell which serves multiple
users where they can use different Hive installation. I take a look at
the code of HiveContext. It looks like I cannot do that today because
catalog field cannot be changed after initialize
I'm using Spark 1.3.0 RC3 build with Hive support.
In Spark Shell, I want to reuse the HiveContext instance to different
warehouse locations. Below are the steps for my test (Assume I have
loaded a file into table src).
==
15/03/10 18:22:59 INFO SparkILoop: Created sql context
I am not so sure if Hive supports change the metastore after initialized, I
guess not. Spark SQL totally rely on Hive Metastore in HiveContext, probably
that's why it doesn't work as expected for Q1.
BTW, in most of cases, people configure the metastore settings in
hive-site.xml
I am trying to run a Hive query from Spark using HiveContext. Here is the
code
/ val conf = new SparkConf().setAppName(HiveSparkIntegrationTest)
conf.set(spark.executor.extraClassPath,
/opt/cloudera/parcels/CDH-5.2.0-1.cdh5.2.0.p0.36/lib/hive/lib);
conf.set
On Fri, Mar 6, 2015 at 2:47 PM, nitinkak001 nitinkak...@gmail.com wrote:
I am trying to run a Hive query from Spark using HiveContext. Here is the
code
/ val conf = new SparkConf().setAppName(HiveSparkIntegrationTest)
conf.set(spark.executor.extraClassPath,
/opt/cloudera/parcels/CDH
from Calliope) , then I registered the rdd as a
temp table , next I tried to query it using HiveContext, but it seems that
hive context can not see the registered table suing SQL context. Is this a
normal case?
Stack trace:
ERROR hive.ql.metadata.Hive -
NoSuchObjectException
wrote:
Hi,
I did an experiment with Hive and SQL context , I queried Cassandra using
CassandraAwareSQLContext
(a custom SQL context from Calliope) , then I registered the rdd as a
temp table , next I tried to query it using HiveContext, but it seems that
hive context can not see the registered
Hi,
I did an experiment with Hive and SQL context , I queried Cassandra
using CassandraAwareSQLContext
(a custom SQL context from Calliope) , then I registered the rdd as a
temp table , next I tried to query it using HiveContext, but it seems that
hive context can not see the registered table
Hi Sreeharsha,
My data is in HDFS. I am trying to use Spark HiveContext (instead of
SQLContext) to fire queries on my data just because HiveContext supports
more operations.
Sreeharsha wrote
Change derby to mysql and check once me to faced the same issue
I am pretty new to Spark
Correction,
should be HADOOP_CONF_DIR=/etc/hive/conf spark-shell --driver-class-path
'/data/opt/cloudera/parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/lib/hive/lib/*'
--driver-java-options '-Dspark.executor.extraClassPath=/opt/cloudera/
parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/lib/hive/lib/*'
On Fri, Feb 20, 2015
Correction,
should be HADOOP_CONF_DIR=/etc/hive/conf --driver-class-path
'/data/opt/cloudera/parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/lib/hive/lib/*'
--driver-java-options '-Dspark.executor.extraClassPath=/opt/cloudera/
parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/lib/hive/lib/*'
On Fri, Feb 20, 2015 at 3:43 PM,
That worked perfectly...thanks so much!
On Fri, Feb 20, 2015 at 3:49 PM, Sourigna Phetsarath
gna.phetsar...@teamaol.com wrote:
Correction,
should be HADOOP_CONF_DIR=/etc/hive/conf spark-shell --driver-class-path
'/data/opt/cloudera/parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/lib/hive/lib/*'
Try it without
--master yarn-cluster
if you are trying to run a spark-shell. :)
On Fri, Feb 20, 2015 at 3:18 PM, chirag lakhani chirag.lakh...@gmail.com
wrote:
I tried
spark-shell --master yarn-cluster --driver-class-path
Thanks! I am able to login to Spark now but I am still getting the same
error
scala sqlContext.sql(FROM analytics.trainingdatafinal SELECT
*).collect().foreach(println)
15/02/20 14:40:22 INFO ParseDriver: Parsing command: FROM
analytics.trainingdatafinal SELECT *
15/02/20 14:40:22 INFO
Also, you might want to add the hadoop configs:
HADOOP_CONF_DIR=/etc/hadoop/conf:/etc/hive/conf --driver-class-path
'/data/opt/cloudera/parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/lib/hive/lib/*'
--driver-java-options '-Dspark.executor.extraClassPath=/opt/cloudera/
I am trying to access a hive table using spark sql but I am having
trouble. I followed the instructions in a cloudera community board which
stated
1) Import hive jars into the class path
export SPARK_CLASSPATH=$(find
/data/opt/cloudera/parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/lib/hive/lib/ -name
Chirag,
This worked for us:
spark-submit --master yarn-cluster --driver-class-path
'/opt/cloudera/parcels/CDH/lib/hive/lib/*' --driver-java-options
'-Dspark.executor.extraClassPath=/opt/cloudera/parcels/CDH/lib/hive/lib/*'
...
Let me know, if you have any issues.
On Fri, Feb 20, 2015 at 2:43
I tried
spark-shell --master yarn-cluster --driver-class-path
'/data/opt/cloudera/parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/lib/hive/lib/*'
--driver-java-options
'-Dspark.executor.extraClassPath=/opt/cloudera/parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/lib/hive/lib/*'
and I get the following error
Error: Cluster
It works now using 1.2.1. Thanks for all the help. Spark rocks !!
-
Thanks,
Roy
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Re-HiveContext-created-SchemaRDD-s-saveAsTable-is-not-working-on-1-2-0-tp21442p21664.html
Sent from the Apache Spark User
Hi,
I've been reading about Spark SQL and people suggest that using HiveContext
is better. So can anyone please suggest a solution to the above problem.
This is stopping me from moving forward with HiveContext.
Thanks
Harika
--
View this message in context:
http://apache-spark-user-list
: HiveContext in SparkSQL - concurrency issues
Hi,
I've been reading about Spark SQL and people suggest that using HiveContext
is better. So can anyone please suggest a solution to the above problem.
This is stopping me from moving forward with HiveContext.
Thanks
Harika
--
View this message in context
();
}}
With the create table and load data commands commented out, the query
command can be executed successfully, but I come to ClassNotFoundExceptions
if these two commands are executed inside HiveContext, even with different
error messages,
The create table command will cause the following
, but I come to ClassNotFoundExceptions if these
two commands are executed inside HiveContext, even with different error
messages,
The create table command will cause the following:
Exception in thread main
org.apache.spark.sql.execution.QueryExecutionException: FAILED: Hive
://apache-spark-user-list.1001560.n3.nabble.com/Is-there-a-way-to-access-Hive-UDFs-in-a-HiveContext-tp21510p21527.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr
Hi
Are you trying to run a spark job from inside eclipse? and want the job to
access hive configuration options.? To access hive tables?
Thanks
Arush
On Tue, Feb 3, 2015 at 7:24 AM, guxiaobo1982 guxiaobo1...@qq.com wrote:
Hi,
I know two options, one for spark_submit, the other one for
Version: Spark 1.2 on CDH 5.3.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Is-there-a-way-to-access-Hive-UDFs-in-a-HiveContext-tp21510.html
Sent from the Apache Spark User List mailing list archive at Nabble.com
Hi,
I've been trying to use HiveContext(instead of SQLContext) in my SparkSQL
application and when I run the application simultaneously, it only works on
the first call and every other call throws the following error-
ERROR Datastore.Schema: Failed initialising database.
Failed to start
Hi,
I know two options, one for spark_submit, the other one for spark-shell, but
how to set for programs running inside eclipse?
Regards,
that are created from a
HiveContext. If I understand correctly, here the SchemaRDD means those
generated by HiveContext.sql, instead of applySchema.
Thanks.
Zhan Zhang
On Jan 29, 2015, at 9:38 PM, matroyd debajyoti@healthagen.com
wrote:
Hi, I am trying saveAsTable on SchemaRDD created
with
SchemaRDDs that are created from a HiveContext”. If I
understand correctly, here the SchemaRDD means those
generated by HiveContext.sql, instead of applySchema.
Thanks.
Zhan Zhang
On Jan 29, 2015, at 9:38 PM, matroyd [hidden email
Hi,I am trying saveAsTable on SchemaRDD created from HiveContext and it
fails. This is on Spark 1.2.0.Following are details of the code, command and
exceptions:
http://stackoverflow.com/questions/28222496/how-to-enable-sql-on-schemardd-via-the-jdbc-interface-is-it-even-possible
http
I think it is expected. Refer to the comments in saveAsTable Note that this
currently only works with SchemaRDDs that are created from a HiveContext”. If I
understand correctly, here the SchemaRDD means those generated by
HiveContext.sql, instead of applySchema.
Thanks.
Zhan Zhang
On Jan 29
in saveAsTable Note that
this currently only works with SchemaRDDs that are created from a
HiveContext. If I understand correctly, here the SchemaRDD means those
generated by HiveContext.sql, instead of applySchema.
Thanks.
Zhan Zhang
On Jan 29, 2015, at 9:38 PM, matroyd debajyoti
: select key, value.percent from
mytable_data lateral view explode(audiences) f as key, value limit 3;
But there is a bug in HiveContext:
https://issues.apache.org/jira/browse/SPARK-5237
I am working on it now, hopefully make a patch soon.
Cheng Hao
--
View this message in context
Wow, glad to know that it works well, and sorry, the Jira is another issue,
which is not the same case here.
From: Bagmeet Behera [mailto:bagme...@gmail.com]
Sent: Saturday, January 17, 2015 12:47 AM
To: Cheng, Hao
Subject: Re: using hiveContext to select a nested Map-data-type from
Hi all,
Any help on the following is very much appreciated.
=
Problem:
On a schemaRDD read from a parquet file (data within file uses AVRO model)
using the HiveContext:
I can't figure out how to 'select' or use 'where' clause, to filter
rows on a field
Hi, BB
Ideally you can do the query like: select key, value.percent from
mytable_data lateral view explode(audiences) f as key, value limit 3;
But there is a bug in HiveContext:
https://issues.apache.org/jira/browse/SPARK-5237
I am working on it now, hopefully make a patch soon.
Cheng
that acts as a receiver
for a streaming application? If yes, what is the recommended way of passing
the SparkContext to the actor?
Thanks for your help.
- Ranga
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-HiveContext-within-Custom
.
- Ranga
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-HiveContext-within-Custom-Actor-tp20892.html
Sent from the Apache Spark User List mailing list archive at Nabble.com
201 - 300 of 390 matches
Mail list logo
