FSDataInputStream
Hi We're seeing exceptions when closing a FSDataInputStream. I'm not sure how to interpret the exception. Is there anything that can be done to avoid it? Cheers, -Kristoffer [2016-07-29 09:28:20,162] ERROR Error closing hdfs://hdpcluster/tmp/kafka-connect/logs/sting_actions_inscreen/83/log. (io.confluent.connect.hdfs.TopicPartitionWriter:328) org.apache.kafka.connect.errors.ConnectException: Error closing hdfs://hdpcluster/tmp/kafka-connect/logs/sting_actions_inscreen/83/log at io.confluent.connect.hdfs.wal.FSWAL.close(FSWAL.java:156) at io.confluent.connect.hdfs.TopicPartitionWriter.close(TopicPartitionWriter.java:326) at io.confluent.connect.hdfs.DataWriter.close(DataWriter.java:296) at io.confluent.connect.hdfs.HdfsSinkTask.close(HdfsSinkTask.java:109) at org.apache.kafka.connect.runtime.WorkerSinkTask.commitOffsets(WorkerSinkTask.java:290) at org.apache.kafka.connect.runtime.WorkerSinkTask.closePartitions(WorkerSinkTask.java:421) at org.apache.kafka.connect.runtime.WorkerSinkTask.access$1100(WorkerSinkTask.java:54) at org.apache.kafka.connect.runtime.WorkerSinkTask$HandleRebalance.onPartitionsRevoked(WorkerSinkTask.java:465) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinPrepare(ConsumerCoordinator.java:283) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:212) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.ensurePartitionAssignment(ConsumerCoordinator.java:345) at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:977) at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:937) at org.apache.kafka.connect.runtime.WorkerSinkTask.pollConsumer(WorkerSinkTask.java:305) at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:222) at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:170) at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:142) at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:140) at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:175) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException): BP-141202528-10.3.138.26-1448020478061:blk_1098384937_24779008 does not exist or is not under Constructionnull at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkUCBlock(FSNamesystem.java:6344) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updateBlockForPipeline(FSNamesystem.java:6411) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.updateBlockForPipeline(NameNodeRpcServer.java:870) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolServerSideTranslatorPB.java:955) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2147) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2145) at org.apache.hadoop.ipc.Client.call(Client.java:1468) at org.apache.hadoop.ipc.Client.call(Client.java:1399) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) at com.sun.proxy.$Proxy48.updateBlockForPipeline(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolTranslatorPB.java:877) at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy49.updateBlockForPipeline(Unknown Source) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1266) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:594)
Re: Yarn app: Cannot run "java -jar" container
I got the application work now, thanks Hitesh! But why isn't the container stdout/stderr visible from the ApplicationMaster UI. Is manual ssh the only way to look at them? On Fri, Jan 22, 2016 at 9:19 PM, Kristoffer Sjögren <sto...@gmail.com> wrote: > Thanks for the tip Hitesh - that's really helpful. > > On Fri, Jan 22, 2016 at 7:47 PM, Hitesh Shah <hit...@apache.org> wrote: >> Ideally, the “yarn logs -application” command should give you the logs for >> the container in question and the stdout/stderr there usually gives you a >> good indication on what is going wrong. >> >> Second more complex option: >>- Set yarn.nodemanager.delete.debug-delay-sec to say 1200 or a large >> enough value. Restart all NMs. >>- Run your application. >>- Find the node on which your container failed. >>- Search through the yarn nodemanager local-dirs to find the >> launch_container.sh for your container. >>- Look at its contents to see if things are being setup correctly. Run it >> manually to debug. >> >> — Hitesh >> >> On Jan 22, 2016, at 5:52 AM, Kristoffer Sjögren <sto...@gmail.com> wrote: >> >>> Hi >>> >>> I'm trying to run a yarn 2.7.1 application using a basic boilerplate >>> [1]. But I have trouble running the container with an executable jar >>> file using the following args list. >>> >>>List arg = Collections.singletonList( >>> "/usr/jdk64/jdk1.8.0_40/bin/java -jar app-1.0.0-SNAPSHOT.jar" + >>>" 1>" + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/stdout" + >>>" 2>" + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/stderr" >>>); >>> >>> I'm not really sure why it doesn't work because the diagnostics >>> message on tells me exit code 1 (see below). Neither the stdout or >>> stderr files appear, and I have tried to pipe them to /tmp/stdout etc. >>> >>> How do I debug this error? Is the diagnostics message the only way? I >>> have tried a gazillion different combinations of running the >>> container, and the process is very time consuming and frustrating when >>> there isn't any information to debug. >>> >>> Any tips or pointers on how to trace this error down? >>> >>> Cheers, >>> -Kristoffer >>> >>> [1] >>> https://github.com/hortonworks/simple-yarn-app/tree/master/src/main/java/com/hortonworks/simpleyarnapp >>> >>> >>> Stack trace: ExitCodeException exitCode=1: >>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:576) >>> at org.apache.hadoop.util.Shell.run(Shell.java:487) >>> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753) >>> at >>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212) >>> at >>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) >>> at >>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>> at java.lang.Thread.run(Thread.java:745) >>> >>> >>> Container exited with a non-zero exit code 1 >>> >>> - >>> To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org >>> For additional commands, e-mail: user-h...@hadoop.apache.org >>> >> - To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org For additional commands, e-mail: user-h...@hadoop.apache.org
Re: Yarn app: Cannot run "java -jar" container
Thanks for the tip Hitesh - that's really helpful. On Fri, Jan 22, 2016 at 7:47 PM, Hitesh Shah <hit...@apache.org> wrote: > Ideally, the “yarn logs -application” command should give you the logs for > the container in question and the stdout/stderr there usually gives you a > good indication on what is going wrong. > > Second more complex option: >- Set yarn.nodemanager.delete.debug-delay-sec to say 1200 or a large > enough value. Restart all NMs. >- Run your application. >- Find the node on which your container failed. >- Search through the yarn nodemanager local-dirs to find the > launch_container.sh for your container. >- Look at its contents to see if things are being setup correctly. Run it > manually to debug. > > — Hitesh > > On Jan 22, 2016, at 5:52 AM, Kristoffer Sjögren <sto...@gmail.com> wrote: > >> Hi >> >> I'm trying to run a yarn 2.7.1 application using a basic boilerplate >> [1]. But I have trouble running the container with an executable jar >> file using the following args list. >> >>List arg = Collections.singletonList( >> "/usr/jdk64/jdk1.8.0_40/bin/java -jar app-1.0.0-SNAPSHOT.jar" + >>" 1>" + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/stdout" + >>" 2>" + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/stderr" >>); >> >> I'm not really sure why it doesn't work because the diagnostics >> message on tells me exit code 1 (see below). Neither the stdout or >> stderr files appear, and I have tried to pipe them to /tmp/stdout etc. >> >> How do I debug this error? Is the diagnostics message the only way? I >> have tried a gazillion different combinations of running the >> container, and the process is very time consuming and frustrating when >> there isn't any information to debug. >> >> Any tips or pointers on how to trace this error down? >> >> Cheers, >> -Kristoffer >> >> [1] >> https://github.com/hortonworks/simple-yarn-app/tree/master/src/main/java/com/hortonworks/simpleyarnapp >> >> >> Stack trace: ExitCodeException exitCode=1: >> at org.apache.hadoop.util.Shell.runCommand(Shell.java:576) >> at org.apache.hadoop.util.Shell.run(Shell.java:487) >> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753) >> at >> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212) >> at >> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) >> at >> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> at java.lang.Thread.run(Thread.java:745) >> >> >> Container exited with a non-zero exit code 1 >> >> - >> To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org >> For additional commands, e-mail: user-h...@hadoop.apache.org >> > - To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org For additional commands, e-mail: user-h...@hadoop.apache.org
Yarn app: Cannot run "java -jar" container
Hi I'm trying to run a yarn 2.7.1 application using a basic boilerplate [1]. But I have trouble running the container with an executable jar file using the following args list. List arg = Collections.singletonList( "/usr/jdk64/jdk1.8.0_40/bin/java -jar app-1.0.0-SNAPSHOT.jar" + " 1>" + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/stdout" + " 2>" + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/stderr" ); I'm not really sure why it doesn't work because the diagnostics message on tells me exit code 1 (see below). Neither the stdout or stderr files appear, and I have tried to pipe them to /tmp/stdout etc. How do I debug this error? Is the diagnostics message the only way? I have tried a gazillion different combinations of running the container, and the process is very time consuming and frustrating when there isn't any information to debug. Any tips or pointers on how to trace this error down? Cheers, -Kristoffer [1] https://github.com/hortonworks/simple-yarn-app/tree/master/src/main/java/com/hortonworks/simpleyarnapp Stack trace: ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:576) at org.apache.hadoop.util.Shell.run(Shell.java:487) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Container exited with a non-zero exit code 1 - To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org For additional commands, e-mail: user-h...@hadoop.apache.org
