Hi, Thanks for pointing out the Phoenix discrepancy! I was using a phoenix 4.4. jar for hbase 1.1 release however I was using Hbase 0.98 . Have fixed the above issue. I am still unable to go ahead with the Streaming job in cluster mode with the following trace:
Application application_1452763526769_0021 failed 2 times due to AM Container for appattempt_1452763526769_0021_000002 exited with exitCode: -1000 For more detailed output, check application tracking page:http://slave1:8088/proxy/application_1452763526769_0021/Then, click on links to logs of each attempt. Diagnostics: File does not exist: hdfs://slave1:9000/user/hduser/.sparkStaging/application_1452763526769_0021/__hadoop_conf__7080838197861423764.zip java.io.FileNotFoundException: File does not exist: hdfs://slave1:9000/user/hduser/.sparkStaging/application_1452763526769_0021/__hadoop_conf__7080838197861423764.zip at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1122) at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114) at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:251) at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:61) at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359) at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:357) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:356) at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Failing this attempt. Failing the application. Not sure what’s this _Hadoop_conf-XXX.zip. Can some one pls guide me . I am submitting a Spark Streaming job whjic is reading from a Kafka topic and dumoing data in to hbase tables via Phoenix API. The job is behaving as expected in local mode. Thanks Siddharth Ubale From: Ted Yu [mailto:yuzhih...@gmail.com] Sent: Friday, January 15, 2016 8:08 PM To: Siddharth Ubale <siddharth.ub...@syncoms.com> Cc: user@spark.apache.org Subject: Re: Spark App -Yarn-Cluster-Mode ===> Hadoop_conf_**.zip file. Interesting. Which hbase / Phoenix releases are you using ? The following method has been removed from Put: public Put setWriteToWAL(boolean write) { Please make sure the Phoenix release is compatible with your hbase version. Cheers On Fri, Jan 15, 2016 at 6:20 AM, Siddharth Ubale <siddharth.ub...@syncoms.com<mailto:siddharth.ub...@syncoms.com>> wrote: Hi, This is the log from the application : 16/01/15 19:23:19 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with SUCCEEDED (diag message: Shutdown hook called before final status was reported.) 16/01/15 19:23:19 INFO yarn.ApplicationMaster: Deleting staging directory .sparkStaging/application_1452763526769_0011 16/01/15 19:23:19 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 16/01/15 19:23:19 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. 16/01/15 19:23:19 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remoting shut down. 16/01/15 19:23:19 INFO util.Utils: Shutdown hook called 16/01/15 19:23:19 INFO client.HConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x1523f753f6f0061 16/01/15 19:23:19 INFO zookeeper.ClientCnxn: EventThread shut down 16/01/15 19:23:19 INFO zookeeper.ZooKeeper: Session: 0x1523f753f6f0061 closed 16/01/15 19:23:19 ERROR yarn.ApplicationMaster: User class threw exception: java.lang.NoSuchMethodError: org.apache.hadoop.hbase.client.Put.setWriteToWAL(Z)Lorg/apache/hadoop/hbase/client/Put; java.lang.NoSuchMethodError: org.apache.hadoop.hbase.client.Put.setWriteToWAL(Z)Lorg/apache/hadoop/hbase/client/Put; at org.apache.phoenix.schema.PTableImpl$PRowImpl.newMutations(PTableImpl.java:639) at org.apache.phoenix.schema.PTableImpl$PRowImpl.<init>(PTableImpl.java:632) at org.apache.phoenix.schema.PTableImpl.newRow(PTableImpl.java:557) at org.apache.phoenix.schema.PTableImpl.newRow(PTableImpl.java:573) at org.apache.phoenix.execute.MutationState.addRowMutations(MutationState.java:185) at org.apache.phoenix.execute.MutationState.access$200(MutationState.java:79) at org.apache.phoenix.execute.MutationState$2.init(MutationState.java:258) at org.apache.phoenix.execute.MutationState$2.<init>(MutationState.java:255) at org.apache.phoenix.execute.MutationState.toMutations(MutationState.java:253) at org.apache.phoenix.execute.MutationState.toMutations(MutationState.java:243) at org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:1840) at org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:744) at org.apache.phoenix.compile.CreateTableCompiler$2.execute(CreateTableCompiler.java:186) at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:303) at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:295) at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53) at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:293) at org.apache.phoenix.jdbc.PhoenixStatement.executeUpdate(PhoenixStatement.java:1236) at org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(ConnectionQueryServicesImpl.java:1891) at org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(ConnectionQueryServicesImpl.java:1860) at org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:77) at org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:1860) at org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:162) at org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.connect(PhoenixEmbeddedDriver.java:131) at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:133) at java.sql.DriverManager.getConnection(DriverManager.java:664) at java.sql.DriverManager.getConnection(DriverManager.java:270) at spark.phoenix.PhoenixConnect.getConnection(PhoenixConnect.java:26) at spark.stream.eventStream.startStream(eventStream.java:105) at time.series.wo.agg.InputStreamSpark.main(InputStreamSpark.java:38) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:483) Thanks, Siddharth From: Ted Yu [mailto:yuzhih...@gmail.com<mailto:yuzhih...@gmail.com>] Sent: Friday, January 15, 2016 7:43 PM To: Siddharth Ubale <siddharth.ub...@syncoms.com<mailto:siddharth.ub...@syncoms.com>> Cc: user@spark.apache.org<mailto:user@spark.apache.org> Subject: Re: Spark App -Yarn-Cluster-Mode ===> Hadoop_conf_**.zip file. bq. check application tracking page:http://slave1:8088/proxy/application_1452763526769_0011/ Then<http://slave1:8088/proxy/application_1452763526769_0011/Then>, ... Have you done the above to see what error was in each attempt ? Which Spark / hadoop release are you using ? Thanks On Fri, Jan 15, 2016 at 5:58 AM, Siddharth Ubale <siddharth.ub...@syncoms.com<mailto:siddharth.ub...@syncoms.com>> wrote: Hi, I am trying to run a Spark streaming application in yarn-cluster mode. However I am facing an issue where the job ends asking for a particular Hadoop_conf_**.zip file in hdfs location. Can any one guide with this? The application works fine in local mode only it stops abruptly for want of memory. Below is the error stack trace: diagnostics: Application application_1452763526769_0011 failed 2 times due to AM Container for appattempt_1452763526769_0011_000002 exited with exitCode: -1000 For more detailed output, check application tracking page:http://slave1:8088/proxy/application_1452763526769_0011/Then, click on links to logs of each attempt. Diagnostics: File does not exist: hdfs://slave1:9000/user/hduser/.sparkStaging/application_1452763526769_0011/__hadoop_conf__1057113228186399290.zip java.io.FileNotFoundException: File does not exist: hdfs://slave1:9000/user/hduser/.sparkStaging/application_1452763526769_0011/__hadoop_conf__1057113228186399290.zip at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1122) at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114) at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:251) at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:61) at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359) at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:357) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:356) at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Failing this attempt. Failing the application. ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: default start time: 1452866026622 final status: FAILED tracking URL: http://slave1:8088/cluster/app/application_1452763526769_0011 user: hduser Exception in thread "main" org.apache.spark.SparkException: Application application_1452763526769_0011 finished with failed status at org.apache.spark.deploy.yarn.Client.run(Client.scala:841) at org.apache.spark.deploy.yarn.Client$.main(Client.scala:867) at org.apache.spark.deploy.yarn.Client.main(Client.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 16/01/15 19:23:53 INFO Utils: Shutdown hook called 16/01/15 19:23:53 INFO Utils: Deleting directory /tmp/spark-b6ebcb83-efff-432a-9a7a-b4764f482d81 java.lang.UNIXProcess$ProcessPipeOutputStream@7a0a6f73<mailto:java.lang.UNIXProcess$ProcessPipeOutputStream@7a0a6f73> 1 Siddharth Ubale, Synchronized Communications #43, Velankani Tech Park, Block No. II, 3rd Floor, Electronic City Phase I, Bangalore – 560 100 Tel : +91 80 3202 4060<tel:%2B91%2080%203202%204060> Web: www.syncoms.com<http://www.syncoms.com/> [LogoNEWmohLARGE] London|Bangalore|Orlando we innovate, plan, execute, and transform the business
