problem first?
*From:* Todd Nist [mailto:tsind...@gmail.com]
*Sent:* Thursday, March 19, 2015 7:49 AM
*To:* user@spark.apache.org
*Subject:* [SQL] Elasticsearch-hadoop, exception creating temporary table
I am attempting to access ElasticSearch and expose it’s data through
SparkSQL using
I am attempting to access ElasticSearch and expose it’s data through
SparkSQL using the elasticsearch-hadoop project. I am encountering the
following exception when trying to create a Temporary table from a resource
in ElasticSearch.:
15/03/18 07:54:46 INFO DAGScheduler: Job 2 finished: runJob
I am attempting to access ElasticSearch and expose it’s data through
SparkSQL using the elasticsearch-hadoop project. I am encountering the
following exception when trying to create a Temporary table from a resource
in ElasticSearch.:
15/03/18 07:54:46 INFO DAGScheduler: Job 2 finished: runJob
a deployment of the
spark distribution or any other config change to support a spark job.
Isn't that correct?
On Tue, Mar 17, 2015 at 6:19 PM, Todd Nist tsind...@gmail.com wrote:
Hi Bharath,
Do you have these entries in your $SPARK_HOME/conf/spark-defaults.conf
file?
spark.driver.extraJavaOptions
in the yarn cluster? I'd assume that the latter shouldn't be
necessary.
On Mon, Mar 16, 2015 at 8:38 PM, Todd Nist tsind...@gmail.com wrote:
Hi Bharath,
I ran into the same issue a few days ago, here is a link to a post on
Horton's fourm.
http://hortonworks.com/community/forums/search
Hi Bharath,
I ran into the same issue a few days ago, here is a link to a post on
Horton's fourm. http://hortonworks.com/community/forums/search/spark+1.2.1/
Incase anyone else needs to perform this these are the steps I took to get
it to work with Spark 1.2.1 as well as Spark 1.3.0-RC3:
1.
There is the PR https://github.com/apache/spark/pull/2077 for doing this.
On Fri, Mar 13, 2015 at 6:42 AM, t1ny wbr...@gmail.com wrote:
Hi all,
We are looking for a tool that would let us visualize the DAG generated by
a
Spark application as a simple graph.
This graph would represent the
is also based on scala, I was looking for some help with
java Apis.
*Thanks,*
*Udbhav Agarwal*
*From:* Todd Nist [mailto:tsind...@gmail.com]
*Sent:* 12 March, 2015 5:28 PM
*To:* Udbhav Agarwal
*Cc:* Akhil Das; user@spark.apache.org
*Subject:* Re: hbase sql query
Have you considered
Have you considered using the spark-hbase-connector for this:
https://github.com/nerdammer/spark-hbase-connector
On Thu, Mar 12, 2015 at 5:19 AM, Udbhav Agarwal udbhav.agar...@syncoms.com
wrote:
Thanks Akhil.
Additionaly if we want to do sql query we need to create JavaPairRdd, then
On Thu, Mar 5, 2015 at 10:04 AM, Todd Nist tsind...@gmail.com wrote:
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:166)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163
failed in the first place.
Thanks.
Zhan Zhang
On Mar 6, 2015, at 9:59 AM, Todd Nist tsind...@gmail.com wrote:
First, thanks to everyone for their assistance and recommendations.
@Marcelo
I applied the patch that you recommended and am now able to get into the
shell, thank you worked
, at 11:40 AM, Zhan Zhang zzh...@hortonworks.com wrote:
You are using 1.2.1 right? If so, please add java-opts in conf
directory and give it a try.
[root@c6401 conf]# more java-opts
-Dhdp.version=2.2.2.0-2041
Thanks.
Zhan Zhang
On Mar 6, 2015, at 11:35 AM, Todd Nist tsind
I am running Spark on a HortonWorks HDP Cluster. I have deployed there
prebuilt version but it is only for Spark 1.2.0 not 1.2.1 and there are a
few fixes and features in there that I would like to leverage.
I just downloaded the spark-1.2.1 source and built it to support Hadoop 2.6
by doing the
:
-Djackson.version=1.9.3
Cheers
On Thu, Mar 5, 2015 at 10:04 AM, Todd Nist tsind...@gmail.com wrote:
I am running Spark on a HortonWorks HDP Cluster. I have deployed there
prebuilt version but it is only for Spark 1.2.0 not 1.2.1 and there are
a
few fixes and features in there that I would like
Hi Srini,
If you start the $SPARK_HOME/sbin/start-history-server, you should be able
to see the basic spark ui. You will not see the master, but you will be
able to see the rest as I recall. You also need to add an entry into the
spark-defaults.conf, something like this:
*## Make sure the host
You can specify these jars (joda-time-2.7.jar, joda-convert-1.7.jar) either
as part of your build and assembly or via the --jars option to spark-submit.
HTH.
On Fri, Feb 27, 2015 at 2:48 PM, Su She suhsheka...@gmail.com wrote:
Hello Everyone,
I'm having some issues launching (non-spark)
Hi Kannan,
I believe you should be able to use the --jars for this when invoke the
spark-shell or perform a spark-submit. Per docs:
--jars JARSComma-separated list of local jars to include on the
driver
and executor classpaths.
HTH.
-Todd
On Thu, Feb
Hi Kannan,
Issues with using --jars make sense. I believe you can set the classpath
via the use the --conf spark.executor.extraClassPath= or in your driver
with .set(spark.executor.extraClassPath, .)
I believe you are correct with the localize as well as long as your
guaranteed that all
Hi Emre,
Have you tried adjusting these:
.set(spark.akka.frameSize, 500).set(spark.akka.askTimeout,
30).set(spark.core.connection.ack.wait.timeout, 600)
-Todd
On Fri, Feb 20, 2015 at 8:14 AM, Emre Sevinc emre.sev...@gmail.com wrote:
Hello,
We are building a Spark Streaming application that
Hi Dhimant,
I believe if you change your spark-shell to pass -driver-class-path
/usr/local/spark/lib/mysql-connector-java-5.1.34-bin.jar vs putting it in
--jars.
-Todd
On Wed, Feb 18, 2015 at 10:41 PM, Dhimant dhimant84.jays...@gmail.com
wrote:
Found solution from one of the post found on
I am able to connect by doing the following using the Tableau Initial SQL
and a custom query:
1.
First ingest csv file or json and save out to file system:
import org.apache.spark.sql.SQLContext
import com.databricks.spark.csv._
val sqlContext = new SQLContext(sc)
val demo =
in the schema. In that case you will either have to generate the Hive
tables externally from Spark or use Spark to process the data and save them
using a HiveContext.
From: Todd Nist
Date: Wednesday, February 11, 2015 at 7:53 PM
To: Andrew Lee
Cc: Arush Kharbanda, user@spark.apache.org
What does your hive-site.xml look like? Do you actually have a directory
at the location shown in the error? i.e does /user/hive/warehouse/src
exist? You should be able to override this by specifying the following:
--hiveconf
hive.metastore.warehouse.dir=/location/where/your/warehouse/exists
.html
On Thu, Feb 12, 2015 at 7:24 AM, Todd Nist tsind...@gmail.com wrote:
I have a question with regards to accessing SchemaRDD’s and Spark SQL
temp tables via the thrift server. It appears that a SchemaRDD when
created is only available in the local namespace / context and are
unavailable
I have a question with regards to accessing SchemaRDD’s and Spark SQL temp
tables via the thrift server. It appears that a SchemaRDD when created is
only available in the local namespace / context and are unavailable to
external services accessing Spark through thrift server via ODBC; is this
11, 2015 at 3:59 PM, Todd Nist tsind...@gmail.com wrote:
Hi Arush,
So yes I want to create the tables through Spark SQL. I have placed the
hive-site.xml file inside of the $SPARK_HOME/conf directory I thought that
was all I should need to do to have the thriftserver use it. Perhaps my
hive
using --files hive-site.xml.
similarly you can specify the same metastore to your spark-submit or
sharp-shell using the same option.
On Wed, Feb 11, 2015 at 5:23 AM, Todd Nist tsind...@gmail.com wrote:
Arush,
As for #2 do you mean something like this from the docs:
// sc is an existing
;
NULLMichael
30 Andy
19 Justin
NULLMichael
30 Andy
19 Justin
Time taken: 0.576 seconds
From: Todd Nist
Date: Tuesday, February 10, 2015 at 6:49 PM
To: Silvio Fiorito
Cc: user@spark.apache.org
Subject: Re: SparkSQL + Tableau Connector
Hi Silvio,
Ah, I like
Hi,
I'm trying to understand how and what the Tableau connector to SparkSQL is
able to access. My understanding is it needs to connect to the
thriftserver and I am not sure how or if it exposes parquet, json,
schemaRDDs, or does it only expose schemas defined in the metastore / hive.
For
/resources/kv1.txt' INTO TABLE src)
// Queries are expressed in HiveQLsqlContext.sql(FROM src SELECT key,
value).collect().foreach(println)
Or did you have something else in mind?
-Todd
On Tue, Feb 10, 2015 at 6:35 PM, Todd Nist tsind...@gmail.com wrote:
Arush,
Thank you will take a look
fashion, sort of related to question 2 you would need to configure thrift
to read from the metastore you expect it read from - by default it reads
from metastore_db directory present in the directory used to launch the
thrift server.
On 11 Feb 2015 01:35, Todd Nist tsind...@gmail.com wrote:
Hi
users using org.apache.spark.sql.parquet options
(path 'examples/src/main/resources/users.parquet’)
cache table users
From: Todd Nist
Date: Tuesday, February 10, 2015 at 3:03 PM
To: user@spark.apache.org
Subject: SparkSQL + Tableau Connector
Hi,
I'm trying to understand how and what
Hi Ashu,
Per the documents:
Configuration of Hive is done by placing your hive-site.xml file in conf/.
For example, you can place a something like this in your
$SPARK_HOME/conf/hive-site.xml file:
configuration
property
namehive.metastore.uris/name
*!-- Ensure that the following statement
*@Sasi*
You should be able to create a job something like this:
package io.radtech.spark.jobserver
import java.util.UUID
import org.apache.spark.{ SparkConf, SparkContext }
import org.apache.spark.rdd.RDD
import org.joda.time.DateTime
import com.datastax.spark.connector.types.TypeConverter
101 - 134 of 134 matches
Mail list logo
