Re: Strange behavior of spark-shell while accessing hdfs

Thanks guys for the info. I have to use yarn to access a kerberos cluster. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Strange-behavior-of-spark-shell-while-accessing-hdfs-tp18549p18677.html Sent from the Apache Spark User List mailing list archive at

Re: spark-shell exception while running in YARN mode

The Pi example gives same error in yarn mode HADOOP_CONF_DIR=/home/gs/conf/current ./spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client ../examples/target/spark-examples_2.10-1.2.0-SNAPSHOT.jar What could be wrong here? -- View this message in context:

Re: disable log4j for spark-shell

Tried --driver-java-options and SPARK_JAVA_OPTS, none of them worked Had to change the default one and rebuilt. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/disable-log4j-for-spark-shell-tp11278p18513.html Sent from the Apache Spark User List mailing

Re: disable log4j for spark-shell

Even after changing core/src/main/resources/org/apache/spark/log4j-defaults.properties to WARN followed by a rebuild, the log level is still INFO. Any other suggestions? -- View this message in context:

Re: disable log4j for spark-shell

Some console messages: 14/11/10 20:04:33 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:46713 14/11/10 20:04:33 INFO util.Utils: Successfully started service 'HTTP file server' on port 46713. 14/11/10 20:04:34 INFO server.Server: jetty-8.y.z-SNAPSHOT 14/11/10 20:04:34 INFO

Strange behavior of spark-shell while accessing hdfs

I am trying spark-shell on a single host and got some strange behavior of spark-shell. If I run bin/spark-shell without connecting a master, it can access a hdfs file on a remote cluster with kerberos authentication. scala val textFile =

Spark 1.1.0 with Hadoop 2.5.0

Does Spark 1.1.0 work with Hadoop 2.5.0? The maven build instruction only has command options up to hadoop 2.4. Anybody ever made it work? I am trying to run spark-sql with hive 0.12 on top of hadoop 2.5.0 but can't make it work. -- View this message in context:

Re: How to make ./bin/spark-sql work with hive?

No, it is hive 0.12.4. Let me try your suggestion. It is an existing hive db. I am using the original hive-site.xml as is. Sent from my iPhone On Oct 3, 2014, at 5:02 PM, Edwin Chiu edwin.c...@manage.com wrote: Are you using hive 0.13? Switching back to HadoopDefaultAuthenticator in