Hi All, I am trying to execute batch processing in yarn-cluster mode i.e. I have many sql insert queries,based on argument provided it will it will fetch the queries ,create context , schema RDD and insert in hive tables,
Please Note- in standalone mode its working and in cluster mode working is I configured one query,also I have configured yarn.nodemanager.delete.debug-sec = 600 I am using below command- spark-submit --jars ./analiticlibs/utils-common-1.0.0.jar,./analiticlibs/mysql-connector-java-5.1.17.jar,./analiticlibs/log4j-1.2.17.jar --files datasource.properties,log4j.properties,hive-site.xml --deploy-mode cluster --master yarn --num-executors 1 --driver-memory 2g --driver-java-options "-XX:MaxPermSize=1G" --executor-memory 1g --executor-cores 1 --class com.java.analitics.jobs.StandaloneAggregationJob sparkanalitics-1.0.0.jar daily_agg 2015-04-21 Exception from Container log- Exception in thread "Driver" java.lang.ArrayIndexOutOfBoundsException: 2 at com.java.analitics.jobs.StandaloneAggregationJob.main(StandaloneAggregationJob.java:62) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:427) exception in our exception log file- diagnostics: Application application_1429800386537_0001 failed 2 times due to AM Container for appattempt_1429800386537_0001_000002 exited with exitCode: 15 due to: Exception from container-launch. Container id: container_1429800386537_0001_02_000001 Exit code: 15 Stack trace: ExitCodeException exitCode=15: at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) at org.apache.hadoop.util.Shell.run(Shell.java:455) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Container exited with a non-zero exit code 15 .Failing this attempt.. Failing the application. ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: root.hdfs start time: 1429800525569 final status: FAILED tracking URL: http://tejas.alcatel.com:8088/cluster/app/application_1429800386537_0001 user: hdfs 2015-04-23 20:19:27 DEBUG Client - stopping client from cache: org.apache.hadoop.ipc.Client@12f5f40b 2015-04-23 20:19:27 DEBUG Utils - Shutdown hook called need urgent support, -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-yarn-cluster-job-failing-in-batch-processing-tp22626.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org