Re: Spark job is failing with kerberos error while creating hive context in yarn-cluster mode (through spark-submit)

I have a custom hive-site.xml for spark in sparks conf directory. These properties are the minimal ones that you need for spark, I believe. hive.metastore.kerberos.principal = copy from your hive-site.xml, i.e. "hive/_h...@foo.com" hive.metastore.uris = copy from your hive-site.xml, i.e.

Error trying to get DF for Hive table stored HBase

I’m trying to create a DF for an external Hive table that is in HBase. I get the a NoSuchMethodError

Re: Accessing external Kerberised resources from Spark executors in Yarn client/cluster mode

Another thing to check is to make sure each one of you executor nodes has the JCE jars installed. try{ javax.crypto.Cipher.getMaxAllowedKeyLength("AES") > 128 } catch { case e:java.security.NoSuchAlgorithmException => false } Setting "-Dsun.security.krb5.debug=true” and

Re: Spark_1.5.1_on_HortonWorks

I have been running 1.5.1 with Hive in secure mode on HDP 2.2.4 without any problems. Doug > On Oct 21, 2015, at 12:05 AM, Ajay Chander wrote: > > Hi Everyone, > > Any one has any idea if spark-1.5.1 is available as a service on HortonWorks > ? I have spark-1.3.1

Re: Unable to start spark-shell on YARN

The error is because the shell is trying to resolve hdp.version and can’t. To fix this, you need to put a file called java-opts in your conf directory that has something like this. -Dhdp.version=2.x.x.x Where 2.x.x.x is there version of hdp that you are using. Cheers, Doug > On Sep 24, 2015,

Re: Local Repartition

Hi Daniel, Take a look at .coalesce() I’ve seen good results by coalescing to num executors * 10, but I’m still trying to figure out the optimal number of partitions per executor. To get the number of executors, sc.getConf.getInt(“spark.executor.instances”,-1) Cheers, Doug On Jul 20, 2015,

Re: Spark 1.4 on HortonWork HDP 2.2

If you run Hadoop in secure mode and want to talk to Hive 0.14, it won’t work, see SPARK-5111 I have a patched version of 1.3.1 that I’ve been using. I haven’t had the time to get 1.4.0 working. Cheers, Doug On Jun 19, 2015, at 8:39 AM, ayan guha guha.a...@gmail.com wrote: I think you

Re: Spark 1.4.0-rc4 HiveContext.table(db.tbl) NoSuchTableException

, sqlContext.table does not officially support database name. It only supports table name as the parameter. We will add a method to support database name in future. Thanks, Yin On Thu, Jun 4, 2015 at 8:10 AM, Doug Balog doug.sparku...@dugos.com wrote: Hi Yin, I’m very surprised

Re: Spark 1.4.0-rc4 HiveContext.table(db.tbl) NoSuchTableException

, 2015 at 10:45 AM, Doug Balog doug.sparku...@dugos.com wrote: Hi, sqlContext.table(“db.tbl”) isn’t working for me, I get a NoSuchTableException. But I can access the table via sqlContext.sql(“select * from db.tbl”) So I know it has the table info from the metastore. Anyone else see

Spark 1.4.0-rc4 HiveContext.table(db.tbl) NoSuchTableException

Hi, sqlContext.table(“db.tbl”) isn’t working for me, I get a NoSuchTableException. But I can access the table via sqlContext.sql(“select * from db.tbl”) So I know it has the table info from the metastore. Anyone else see this ? I’ll keep digging. I compiled via make-distribution -Pyarn

Re: Spark Job triggers second attempt

I bet you are running on YARN in cluster mode. If you are running on yarn in client mode, .set(“spark.yarn.maxAppAttempts”,”1”) works as you expect, because YARN doesn’t start your app on the cluster until you call SparkContext(). But If you are running on yarn in cluster mode, the driver

Re: HDP 2.2 AM abort : Unable to find ExecutorLauncher class

The “best” solution to spark-shell’s problem is creating a file $SPARK_HOME/conf/java-opts with “-Dhdp.version=2.2.0.0-2014” Cheers, Doug On Mar 28, 2015, at 1:25 PM, Michael Stone mst...@mathom.us wrote: I've also been having trouble running 1.3.0 on HDP. The

Re: HDP 2.2 AM abort : Unable to find ExecutorLauncher class

an opinion ? Doug On Mar 19, 2015, at 5:51 PM, Doug Balog doug.sparku...@dugos.com wrote: I’m seeing the same problem. I’ve set logging to DEBUG, and I think some hints are in the “Yarn AM launch context” that is printed out before Yarn runs java. My next step is to talk to the admins

Re: HDP 2.2 AM abort : Unable to find ExecutorLauncher class

I’m seeing the same problem. I’ve set logging to DEBUG, and I think some hints are in the “Yarn AM launch context” that is printed out before Yarn runs java. My next step is to talk to the admins and get them to set yarn.nodemanager.delete.debug-delay-sec in the config, as recommended in