RE: SparkSQL 1.1 hang when DROP or LOAD

Seems like the thriftServer cannot connect to Zookeeper, so it cannot get lock. This is how it the log looks when I run SparkSQL: load data inpath kv1.txt into table src; log: 14/09/16 14:40:47 INFO Driver: PERFLOG method=acquireReadWriteLocks 14/09/16 14:40:47 INFO ClientCnxn: Opening socket

RE: SparkSQL 1.1 hang when DROP or LOAD

Hi, Hao Cheng. I have done other tests. And the result shows the thriftServer can connect to Zookeeper. However, I found some more interesting things. And I think I have found a bug! Test procedure: Test1: (0) Use beeline to connect to thriftServer. (1) Switch database use dw_op1; (OK) The logs

SparkSQL 1.1 hang when DROP or LOAD

I started sparkSQL thrift server: sbin/start-thriftserver.sh Then I use beeline to connect to it: bin/beeline !connect jdbc:hive2://localhost:1 op1 op1 I have created a database for user op1. create database dw_op1; And grant all privileges to user op1; grant all on database dw_op1 to user

RE: SparkSQL 1.1 hang when DROP or LOAD

Hi， Hao Cheng， Here is the Spark\Hadoop version: Spark version = 1.1.0 Hadoop version = 2.0.0-cdh4.6.0 And hive-site.xml: configuration property namefs.default.name/name valuehdfs://ns/value /property property namedfs.nameservices/name valuens/value /property

RE: SparkSQL 1.1 hang when DROP or LOAD

Hi， Hao Cheng， Here is the Spark\Hadoop version: Spark version = 1.1.0 Hadoop version = 2.0.0-cdh4.6.0 And hive-site.xml: configuration property namefs.default.name/name valuehdfs://ns/value /property property namedfs.nameservices/name valuens/value /property

RE: SparkSQL 1.1 hang when DROP or LOAD

Hi， Hao Cheng， This is my spark assembly jar name： spark-assembly-1.1.0-hadoop2.0.0-cdh4.6.0.jar I compiled spark 1.1.0 with following cmd: export MAVEN_OPTS=-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m mvn -Dhadoop.version=2.0.0-cdh4.6.0 -Phive -Pspark-ganglia-lgpl -DskipTests

SparkSQL hang due to PERFLOG method=acquireReadWriteLocks

I am running Spark Standalone mode with Spark 1.1 I started SparkSQL thrift server as follows: ./sbin/start-thriftserver.sh Then I use beeline to connect to it. Now, I can CREATE, SELECT, SHOW the databases or the tables; But when I DROP or Load data inpath 'kv1.txt' into table src, the Beeline

RE: problem connection to hdfs on localhost from spark-shell

Change your conf/spark-env.sh: export HADOOP_CONF_DIR=/etc/hadoop/confexport YARN_CONF_DIR=/etc/hadoop/conf Date: Thu, 28 Aug 2014 16:19:05 -0700 From: ml-node+s1001560n13074...@n3.nabble.com To: linkpatrick...@live.com Subject: problem connection to hdfs on localhost from spark-shell

RE: The concurrent model of spark job/stage/task

Hi, I think an example will help illustrate the model better. /*** SimpleApp.scala ***/import org.apache.spark.SparkContextimport org.apache.spark.SparkContext._ object SimpleApp { def main(args: Array[String]) {val logFile = $YOUR_SPARK_HOME/README.md val sc = new SparkContext(local,

RE: org.apache.hadoop.io.compress.SnappyCodec not found

Hi, You can set the settings in conf/spark-env.sh like this:export SPARK_LIBRARY_PATH=/usr/lib/hadoop/lib/native/ SPARK_JAVA_OPTS+=-Djava.library.path=$SPARK_LIBRARY_PATH SPARK_JAVA_OPTS+=-Dspark.io.compression.codec=org.apache.spark.io.SnappyCompressionCodec

[Compile error] Spark 1.0.2 against cloudera 2.0.0-cdh4.6.0 error

Hi, Following the document: # Cloudera CDH 4.2.0 mvn -Pyarn-alpha -Dhadoop.version=2.0.0-cdh4.2.0 -DskipTests clean package I compile Spark 1.0.2 with this cmd: mvn -Pyarn-alpha -Dhadoop.version=2.0.0-cdh4.6.0 -DskipTests clean package However, I got two errors: [INFO] Compiling 14 Scala

Cannot connect to hive metastore

Seems like the mysql connector jar is not included in the classpath. Where can I set the jar to the classpath? hive-site.xml: property namejavax.jdo.option.ConnectionURL/name valuejdbc:mysql://localhost:3306/metastore?createDatabaseIfNotExist=trueamp;characterEncoding=UTF-8/value