[ https://issues.apache.org/jira/browse/SPARK-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved SPARK-6524. ------------------------------ Resolution: Not a Problem Going to go out on a limb and say this looks like an app problem, or at least, the next step is on user@ > Problem connecting JAVA API to Spark Yarn Cluster or yarn Client > ---------------------------------------------------------------- > > Key: SPARK-6524 > URL: https://issues.apache.org/jira/browse/SPARK-6524 > Project: Spark > Issue Type: Question > Components: Java API, YARN > Affects Versions: 1.2.0 > Environment: Ubuntu 14.10 > java JDK 1.8 > Reporter: milan.b > > Hi Team, > I am trying to submit a spark job to yarn-cluster or yarn client > using Java API but I was unable to do so. Following is the configuration code > > System.setProperty("SPARK_YARN_MODE", "true"); > SparkConf sparkYarnConf = new SparkConf().setAppName("Spark yarn") > .setMaster("yarn-cluster") > .set("spark.executor.memory", "258m") > .set("spark.driver.memory", "2588m") > .set("spark.yarn.app.id", "append"); > ClientArguments clientArgs =new ClientArguments(args, > sparkYarnConf); > Error : > org.apache.spark.util.Utils] (MSC service thread 1-2) Service 'SparkUI' could > not bind on port 4043. Attempting port 4044. > 20:57:41,808 ERROR [stderr] (MSC service thread 1-2) 15/03/24 20:57:41 WARN > Utils: Service 'SparkUI' could not bind on port 4043. Attempting port 4044. > 20:57:41,842 INFO [org.apache.spark.util.Utils] (MSC service thread 1-2) > Successfully started service 'SparkUI' on port 4044. > 20:57:41,846 ERROR [stderr] (MSC service thread 1-2) 15/03/24 20:57:41 INFO > Utils: Successfully started service 'SparkUI' on port 4044. > 20:57:41,850 INFO [org.apache.spark.ui.SparkUI] (MSC service thread 1-2) > Started SparkUI at http://D7271:4044 > 20:57:41,850 ERROR [stderr] (MSC service thread 1-2) 15/03/24 20:57:41 INFO > SparkUI: Started SparkUI at http://D7271:4044 > 20:57:41,968 INFO [org.apache.spark.scheduler.cluster.YarnClusterScheduler] > (MSC service thread 1-2) Created YarnClusterScheduler > 20:57:41,968 ERROR [stderr] (MSC service thread 1-2) 15/03/24 20:57:41 INFO > YarnClusterScheduler: Created YarnClusterScheduler > 20:57:42,243 INFO [org.apache.spark.network.netty.NettyBlockTransferService] > (MSC service thread 1-2) Server created on 45902 > 20:57:42,243 ERROR [stderr] (MSC service thread 1-2) 15/03/24 20:57:42 INFO > NettyBlockTransferService: Server created on 45902 > 20:57:42,247 INFO [org.apache.spark.storage.BlockManagerMaster] (MSC service > thread 1-2) Trying to register BlockManager > 20:57:42,247 ERROR [stderr] (MSC service thread 1-2) 15/03/24 20:57:42 INFO > BlockManagerMaster: Trying to register BlockManager > 20:57:42,264 INFO [org.apache.spark.storage.BlockManagerMasterActor] > (sparkDriver-akka.actor.default-dispatcher-2) Registering block manager > D7271:45902 with 246.0 MB RAM, BlockManagerId(<driver>, D7271, 45902) > 20:57:42,265 ERROR [stderr] (sparkDriver-akka.actor.default-dispatcher-2) > 15/03/24 20:57:42 INFO BlockManagerMasterActor: Registering block manager > D7271:45902 with 246.0 MB RAM, BlockManagerId(<driver>, D7271, 45902) > 20:57:42,272 INFO [org.apache.spark.storage.BlockManagerMaster] (MSC service > thread 1-2) Registered BlockManager > 20:57:42,276 ERROR [stderr] (MSC service thread 1-2) 15/03/24 20:57:42 INFO > BlockManagerMaster: Registered BlockManager > 20:57:42,330 ERROR [org.springframework.web.context.ContextLoader] (MSC > service thread 1-2) Context initialization failed: > org.springframework.beans.factory.BeanCreationException: Error creating bean > with name 'exampleInitBean' defined in ServletContext resource > [/WEB-INF/mvc-dispatcher-servlet.xml]: Invocation of init method failed; > nested exception is java.lang.NullPointerException > at > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1553) > [spring-beans-4.0.6.RELEASE.jar:4.0.6.RELEASE] > at > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:539) > [spring-beans-4.0.6.RELEASE.jar:4.0.6.RELEASE] > at > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:475) > [spring-beans-4.0.6.RELEASE.jar:4.0.6.RELEASE] > at > org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:302) > [spring-beans-4.0.6.RELEASE.jar:4.0.6.RELEASE] > at > org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228) > [spring-beans-4.0.6.RELEASE.jar:4.0.6.RELEASE] > at > org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:298) > [spring-beans-4.0.6.RELEASE.jar:4.0.6.RELEASE] > at > org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193) > [spring-beans-4.0.6.RELEASE.jar:4.0.6.RELEASE] > at > org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:703) > [spring-beans-4.0.6.RELEASE.jar:4.0.6.RELEASE] > at > org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:760) > [spring-context-4.0.6.RELEASE.jar:4.0.6.RELEASE] > at > org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:482) > [spring-context-4.0.6.RELEASE.jar:4.0.6.RELEASE] > at > org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:403) > [spring-web-4.0.6.RELEASE.jar:4.0.6.RELEASE] > at > org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:306) > [spring-web-4.0.6.RELEASE.jar:4.0.6.RELEASE] > at > org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:106) > [spring-web-4.0.6.RELEASE.jar:4.0.6.RELEASE] > at > io.undertow.servlet.core.ApplicationListeners.contextInitialized(ApplicationListeners.java:173) > [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] > at > io.undertow.servlet.core.DeploymentManagerImpl.deploy(DeploymentManagerImpl.java:193) > [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] > at > org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:87) > at > org.wildfly.extension.undertow.deployment.UndertowDeploymentService.start(UndertowDeploymentService.java:72) > at > org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948) > at > org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [rt.jar:1.8.0_31] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [rt.jar:1.8.0_31] > at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_31] > Caused by: java.lang.NullPointerException > at > org.apache.spark.deploy.yarn.ApplicationMaster$.sparkContextInitialized(ApplicationMaster.scala:534) > [spark-yarn_2.10-1.2.1.jar:1.2.1] > at > org.apache.spark.scheduler.cluster.YarnClusterScheduler.postStartHook(YarnClusterScheduler.scala:46) > [spark-yarn_2.10-1.2.1.jar:1.2.1] > at org.apache.spark.SparkContext.<init>(SparkContext.scala:509) > [spark-core_2.10-1.2.1.jar:1.2.1] > at > org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:61) > [spark-core_2.10-1.2.1.jar:1.2.1] > at > org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:75) > [spark-core_2.10-1.2.1.jar:1.2.1] > at com.prodapt.spark.util.SparkUtil.initSparkContext(SparkUtil.java:52) > [spark-pro-utils-1.0.jar:] > at > com.prodapt.billanalysis.spark.SparkInitializer.initializeSpark(SparkInitializer.java:19) > [classes:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > [rt.jar:1.8.0_31] > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > [rt.jar:1.8.0_31] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > [rt.jar:1.8.0_31] > at java.lang.reflect.Method.invoke(Method.java:483) [rt.jar:1.8.0_31] > at > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1681) > [spring-beans-4.0.6.RELEASE.jar:4.0.6.RELEASE] > at > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1620) > [spring-beans-4.0.6.RELEASE.jar:4.0.6.RELEASE] > at > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1549) > [spring-beans-4.0.6.RELEASE.jar:4.0.6.RELEASE] > ... 21 more > Any help is appreciated > Thanks and Regards > Milan.B -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org