Cannot create parquet with snappy output for hive external table

Hi Group, I am not able to load data into external hive table which is partitioned. Trace :- 1. create external table test(id int, name string) stored as parquet location 'hdfs://testcluster/user/abc/test' tblproperties ('PARQUET.COMPRESS'='SNAPPY'); 2.Spark code val spark =

Re: problems with checkpoint and spark sql

Hi David, You got any solution for this ? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/problems-with-checkpoint-and-spark-sql-tp26080p27773.html Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: Bad Digest error while doing aws s3 put

Hi , I am getting the following error while reading the huge data from S3 and after processing ,writing data to S3 again. Did you find any solution for this ? 16/02/07 07:41:59 WARN scheduler.TaskSetManager: Lost task 144.2 in stage 3.0 (TID 169, ip-172-31-7-26.us-west-2.compute.internal):

Explanation streaming-cep-engine with example

Hi, Can someone explain how spark streaming cep engine works ? How to use it with sample example? http://spark-packages.org/package/Stratio/streaming-cep-engine -- View this message in context:

Error while Insert data into hive table via spark

Hi, I have configured apache spark 1.3.0 with hive 1.0.0 and hadoop 2.6.0. I am able to create table and retrive data from hive tables via following commands ,but not able insert data into table. scala sqlContext.sql(CREATE TABLE IF NOT EXISTS newtable (key INT)); scala sqlContext.sql(select *

No suitable driver found error, Create table in hive from spark sql

No suitable driver found error, Create table in hive from spark sql. I am trying to execute following example. SPARKGIT: spark/examples/src/main/scala/org/apache/spark/examples/sql/hive/HiveFromSpark.scala My setup :- hadoop 1.6,spark 1.2, hive 1.0, mysql server (installed via yum install

Re: No suitable driver found error, Create table in hive from spark sql

Found solution from one of the post found on internet. I updated spark/bin/compute-classpath.sh and added database connector jar into classpath. CLASSPATH=$CLASSPATH:/data/mysql-connector-java-5.1.14-bin.jar -- View this message in context:

Change number of workers and memory

I am having a spark cluster having some high performance nodes and others are having commodity specs (lower configuration). When I configure worker memory and instances in spark-env.sh, it reflects to all the nodes. Can I change SPARK_WORKER_MEMORY and SPARK_WORKER_INSTANCES properties per

Re: error: type mismatch while Union

Thank you Aaron for pointing out problem. This only happens when I run this code in spark-shell but not when i submit the job. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/error-type-mismatch-while-Union-tp13547p13677.html Sent from the Apache Spark User

Re: error: type mismatch while Union

I am using Spark version 1.0.2 -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/error-type-mismatch-while-Union-tp13547p13618.html Sent from the Apache Spark User List mailing list archive at Nabble.com.

error: type mismatch while assigning RDD to RDD val object

I am receiving following error in Spark-Shell while executing following code. /class LogRecrod(logLine: String) extends Serializable { val splitvals = logLine.split(,); val strIp: String = splitvals(0) val hostname: String = splitvals(1) val server_name: String = splitvals(2)

Multiple spark shell sessions

Hi, I am receiving following error while connecting the spark server via shell if one shell is already open. How can I open multiple sessions ? Does anyone know abt Workflow Engine/Job Server like apache oozie for spark ? / Welcome to __ / __/__ ___ _/ /__

Re: Multiple spark shell sessions

Thanks Yana, I am able to execute application and command via another session, i also received another port for UI application. Thanks, Dhimant -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Multiple-spark-shell-sessions-tp13441p13459.html Sent from

java.io.NotSerializableException exception - custom Accumulator

Hi , I am getting java.io.NotSerializableException exception while executing following program. import org.apache.spark.SparkContext._ import org.apache.spark.SparkContext import org.apache.spark.AccumulatorParam object App { class Vector (val data: Array[Double]) {} implicit object VectorAP