That is a much better solution than how I resolved it. I got around it by
placing comma separated jar paths for all the hive related jars in --jars
clause.
I will try your solution. Thanks for sharing it.
On Tue, May 26, 2015 at 4:14 AM, Mohammad Islam <misla...@yahoo.com> wrote:
> I got a similar problem.
> I'm not sure if your problem is already resolved.
>
> For the record, I solved this type of error by calling sc..setMaster("
> yarn-cluster");
>
> If you find the solution, please let us know.
>
> Regards,
> Mohammad
>
>
>
>
>
> On Friday, March 6, 2015 2:47 PM, nitinkak001 <nitinkak...@gmail.com>
> wrote:
>
>
> I am trying to run a Hive query from Spark using HiveContext. Here is the
> code
>
> / val conf = new SparkConf().setAppName("HiveSparkIntegrationTest")
>
>
> conf.set("spark.executor.extraClassPath",
> "/opt/cloudera/parcels/CDH-5.2.0-1.cdh5.2.0.p0.36/lib/hive/lib");
> conf.set("spark.driver.extraClassPath",
> "/opt/cloudera/parcels/CDH-5.2.0-1.cdh5.2.0.p0.36/lib/hive/lib");
> conf.set("spark.yarn.am.waitTime", "300000L")
>
> val sc = new SparkContext(conf)
>
> val sqlContext = new HiveContext(sc)
>
> def inputRDD = sqlContext.sql("describe
> spark_poc.src_digital_profile_user");
>
> inputRDD.collect().foreach { println }
>
> println(inputRDD.schema.getClass.getName)
> /
>
> Getting this exception. Any clues? The weird part is if I try to do the
> same
> thing but in Java instead of Scala, it runs fine.
>
> /Exception in thread "Driver" java.lang.NullPointerException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
>
> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:162)
> 15/03/06 17:39:32 ERROR yarn.ApplicationMaster: SparkContext did not
> initialize after waiting for 10000 ms. Please check earlier log output for
> errors. Failing the application.
> Exception in thread "main" java.lang.NullPointerException
> at
>
> org.apache.spark.deploy.yarn.ApplicationMaster.waitForSparkContextInitialized(ApplicationMaster.scala:218)
> at
>
> org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:110)
> at
>
> org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$main$1.apply$mcV$sp(ApplicationMaster.scala:434)
> at
>
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:53)
> at
>
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:52)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at
>
> org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:52)
> at
>
> org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:433)
> at
>
> org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
> 15/03/06 17:39:32 INFO yarn.ApplicationMaster: AppMaster received a
> signal./
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/HiveContext-test-Spark-Context-did-not-initialize-after-waiting-10000ms-tp21953.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>
>
>
