I pasted both stdout and stderr in my previous post. Den 23 jan 2016 22:50 skrev "Poorna Chandra" <[email protected]>:
> Hi Kristoffer, > > Looks like container_e29_1453498444043_0012_01_000002 could not be started > due to some issue. Can you attach the stdout and stderr logs for > container_e29_1453498444043_0012_01_000002? > > Poorna. > > > On Sat, Jan 23, 2016 at 3:53 AM, Kristoffer Sjögren <[email protected]> > wrote: > > > Yes, that almost worked. Now the application starts on Yarn and after > > a while an exception is thrown and the application exits with code 10. > > > > > > Application > > > > About > > Jobs > > > > Tools > > > > Log Type: stdout > > > > Log Upload Time: Sat Jan 23 12:49:41 +0100 2016 > > > > Log Length: 21097 > > > > UnJar appMaster.jar to tmp/twill.launcher-1453549768670-0 > > Launch class (org.apache.twill.internal.appmaster.ApplicationMasterMain) > > with classpath: > > > > > [file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/classes, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/resources, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/commons-cli-1.2.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/scala-library-2.10.1.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/logback-classic-1.0.9.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/commons-math3-3.1.1.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/logback-core-1.0.9.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/xmlenc-0.52.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jsch-0.1.42.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/httpclient-4.1.2.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/commons-configuration-1.6.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/metrics-core-2.2.0.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jetty-6.1.26.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/hadoop-yarn-api-2.7.1.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/hadoop-annotations-2.7.1.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/guice-3.0.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/commons-net-3.1.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jetty-util-6.1.26.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/kafka_2.10-0.8.0.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/twill-api-0.6.0-incubating.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/slf4j-api-1.7.10.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/paranamer-2.3.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/protobuf-java-2.5.0.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/apacheds-kerberos-codec-2.0.0-M15.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/avro-1.7.4.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/commons-compress-1.4.1.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/hadoop-auth-2.7.1.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/zookeeper-3.4.6.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jersey-core-1.9.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/curator-client-2.7.1.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/twill-zookeeper-0.6.0-incubating.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jersey-client-1.9.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/gson-2.2.4.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/hadoop-yarn-common-2.7.1.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/hadoop-hdfs-2.7.1.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/api-asn1-api-1.0.0-M20.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/twill-discovery-core-0.6.0-incubating.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/commons-collections-3.2.1.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/netty-3.7.0.Final.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/slf4j-log4j12-1.7.10.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/hadoop-common-2.7.1.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jackson-mapper-asl-1.9.13.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/zkclient-0.3.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jackson-jaxrs-1.9.13.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jackson-xc-1.9.13.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jsr305-3.0.0.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/snappy-java-1.0.4.1.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/log4j-1.2.17.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/commons-codec-1.4.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/asm-all-5.0.2.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/netty-all-4.0.23.Final.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/servlet-api-2.5.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/guava-13.0.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jopt-simple-3.2.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/curator-framework-2.7.1.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/hadoop-yarn-client-2.7.1.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/commons-httpclient-3.1.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/twill-discovery-api-0.6.0-incubating.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/commons-lang-2.6.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/httpcore-4.1.2.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/twill-yarn-0.6.0-incubating.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/api-util-1.0.0-M20.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/htrace-core-3.1.0-incubating.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/twill-common-0.6.0-incubating.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/commons-io-2.4.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jersey-server-1.9.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/apacheds-i18n-2.0.0-M15.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/commons-logging-1.1.3.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/twill-core-0.6.0-incubating.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jackson-core-asl-1.9.13.jar, > > > > > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/javax.inject-1.jar] > > Launching main: public static void > > > > > org.apache.twill.internal.appmaster.ApplicationMasterMain.main(java.lang.String[]) > > throws java.lang.Exception [] > > 12:49:29.586 [main] DEBUG o.a.h.s.a.util.KerberosName - Kerberos krb5 > > configuration not found, setting default realm to empty > > 12:49:30.083 [main] DEBUG o.a.h.h.p.d.s.DataTransferSaslUtil - > > DataTransferProtocol not using SaslPropertiesResolver, no QOP found in > > configuration for dfs.data.transfer.protection > > 12:49:30.552 [main] INFO o.apache.twill.internal.ServiceMain - > > Starting service ApplicationMasterService [NEW]. > > 12:49:30.600 [kafka-publisher] WARN o.a.t.i.k.c.SimpleKafkaPublisher > > - Broker list is empty. No Kafka producer is created. > > 12:49:30.704 [TrackerService STARTING] INFO > > o.a.t.i.appmaster.TrackerService - Tracker service started at > > http://hdfs-ix03.se-ix.delta.prod:51793 > > 12:49:30.922 [TwillZKPathService STARTING] INFO > > o.a.t.i.ServiceMain$TwillZKPathService - Creating container ZK path: > > zookeeper01.se-ix.delta.prod/JarApp/be4bbf01-5e72-4147-b2eb-b84e19214b5b > > 12:49:31.102 [kafka-publisher] INFO o.a.t.i.k.c.SimpleKafkaPublisher > > - Update Kafka producer broker list: hdfs-ix03.se-ix.delta.prod:58668 > > 12:49:31.288 [ApplicationMasterService] INFO > > o.a.t.internal.AbstractTwillService - Create live node > > > > > zookeeper01.se-ix.delta.prod/JarApp/instances/be4bbf01-5e72-4147-b2eb-b84e19214b5b > > 12:49:31.308 [ApplicationMasterService] INFO > > o.a.t.i.a.ApplicationMasterService - Start application master with > > spec: > > > {"name":"JarApp","runnables":{"JarRunnable":{"name":"JarRunnable","runnable":{"classname":"org.apache.twill.ext.BundledJarRunnable","name":"JarRunnable","arguments":{}},"resources":{"cores":1,"memorySize":512,"instances":1,"uplink":-1,"downlink":-1},"files":[{"name":"twill-app-1.0.0-SNAPSHOT.jar","uri":"hdfs://hdpcluster/user/stoffe/JarApp/be4bbf01-5e72-4147-b2eb-b84e19214b5b/twill-app-1.0.0-SNAPSHOT.jar.e09cf92c-56f7-42a8-84ac-53f2665afa1d.jar","lastModified":1453549766870,"size":7090,"archive":false,"pattern":null}]}},"orders":[{"names":["JarRunnable"],"type":"STARTED"}],"placementPolicies":[],"handler":{"classname":"org.apache.twill.internal.LogOnlyEventHandler","configs":{}}} > > 12:49:31.318 [main] INFO o.apache.twill.internal.ServiceMain - > > Service ApplicationMasterService [RUNNING] started. > > 12:49:31.344 [ApplicationMasterService] INFO > > o.a.t.i.a.ApplicationMasterService - Request 1 container with > > capability <memory:512, vCores:1> for runnable JarRunnable > > 12:49:33.368 [ApplicationMasterService] INFO > > o.a.t.i.a.ApplicationMasterService - Got container > > container_e29_1453498444043_0012_01_000002 > > 12:49:33.369 [ApplicationMasterService] INFO > > o.a.t.i.a.ApplicationMasterService - Starting runnable JarRunnable > > with > > > RunnableProcessLauncher{container=org.apache.twill.internal.yarn.Hadoop21YarnContainerInfo@5e82cebd > > } > > 12:49:33.417 [ApplicationMasterService] INFO > > o.a.t.i.a.RunnableProcessLauncher - Launching in container > > container_e29_1453498444043_0012_01_000002 at > > hdfs-ix03.se-ix.delta.prod:45454, [$JAVA_HOME/bin/java > > -Djava.io.tmpdir=tmp -Dyarn.container=$YARN_CONTAINER_ID > > -Dtwill.runnable=$TWILL_APP_NAME.$TWILL_RUNNABLE_NAME -cp > > launcher.jar:$HADOOP_CONF_DIR -Xmx359m > > org.apache.twill.launcher.TwillLauncher container.jar > > org.apache.twill.internal.container.TwillContainerMain true > > 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr] > > 12:49:33.473 [ApplicationMasterService] INFO > > o.a.t.i.a.ApplicationMasterService - Runnable JarRunnable fully > > provisioned with 1 instances. > > 12:49:35.302 [zk-client-EventThread] INFO > > o.a.t.i.TwillContainerLauncher - Container LiveNodeData updated: > > > > > {"data":{"containerId":"container_e29_1453498444043_0012_01_000002","host":"hdfs-ix03.se-ix.delta.prod"}} > > 12:49:37.484 [ApplicationMasterService] INFO > > o.a.t.i.a.ApplicationMasterService - Container > > container_e29_1453498444043_0012_01_000002 completed with > > COMPLETE:Exception from container-launch. > > Container id: container_e29_1453498444043_0012_01_000002 > > Exit code: 10 > > Stack trace: ExitCodeException exitCode=10: > > at org.apache.hadoop.util.Shell.runCommand(Shell.java:576) > > at org.apache.hadoop.util.Shell.run(Shell.java:487) > > at > > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753) > > at > > > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212) > > at > > > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) > > at > > > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > at > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > > at java.lang.Thread.run(Thread.java:745) > > > > > > Container exited with a non-zero exit code 10 > > . > > 12:49:37.488 [ApplicationMasterService] WARN > > o.a.t.i.appmaster.RunningContainers - Container > > container_e29_1453498444043_0012_01_000002 exited abnormally with > > state COMPLETE, exit code 10. > > 12:49:37.496 [ApplicationMasterService] INFO > > o.a.t.i.a.ApplicationMasterService - All containers completed. > > Shutting down application master. > > 12:49:37.498 [ApplicationMasterService] INFO > > o.a.t.i.a.ApplicationMasterService - Stop application master with > > spec: > > > {"name":"JarApp","runnables":{"JarRunnable":{"name":"JarRunnable","runnable":{"classname":"org.apache.twill.ext.BundledJarRunnable","name":"JarRunnable","arguments":{}},"resources":{"cores":1,"memorySize":512,"instances":1,"uplink":-1,"downlink":-1},"files":[{"name":"twill-app-1.0.0-SNAPSHOT.jar","uri":"hdfs://hdpcluster/user/stoffe/JarApp/be4bbf01-5e72-4147-b2eb-b84e19214b5b/twill-app-1.0.0-SNAPSHOT.jar.e09cf92c-56f7-42a8-84ac-53f2665afa1d.jar","lastModified":1453549766870,"size":7090,"archive":false,"pattern":null}]}},"orders":[{"names":["JarRunnable"],"type":"STARTED"}],"placementPolicies":[],"handler":{"classname":"org.apache.twill.internal.LogOnlyEventHandler","configs":{}}} > > 12:49:37.500 [ApplicationMasterService] INFO > > o.a.t.i.appmaster.RunningContainers - Stopping all instances of > > JarRunnable > > 12:49:37.500 [ApplicationMasterService] INFO > > o.a.t.i.appmaster.RunningContainers - Terminated all instances of > > JarRunnable > > 12:49:37.512 [ApplicationMasterService] INFO > > o.a.t.i.a.ApplicationMasterService - Application directory deleted: > > hdfs://hdpcluster/user/stoffe/JarApp/be4bbf01-5e72-4147-b2eb-b84e19214b5b > > 12:49:37.512 [ApplicationMasterService] INFO > > o.a.t.internal.AbstractTwillService - Remove live node > > > > > zookeeper01.se-ix.delta.prod/JarApp/instances/be4bbf01-5e72-4147-b2eb-b84e19214b5b > > 12:49:37.516 [ApplicationMasterService] INFO > > o.a.t.internal.AbstractTwillService - Service ApplicationMasterService > > with runId be4bbf01-5e72-4147-b2eb-b84e19214b5b shutdown completed > > 12:49:37.516 [main] INFO o.apache.twill.internal.ServiceMain - > > Service ApplicationMasterService [TERMINATED] completed. > > 12:49:39.676 [kafka-publisher] WARN o.a.t.i.k.c.SimpleKafkaPublisher > > - Broker list is empty. No Kafka producer is created. > > 12:49:40.037 [TwillZKPathService STOPPING] INFO > > o.a.t.i.ServiceMain$TwillZKPathService - Removing container ZK path: > > zookeeper01.se-ix.delta.prod/JarApp/be4bbf01-5e72-4147-b2eb-b84e19214b5b > > 12:49:40.248 [TrackerService STOPPING] INFO > > o.a.t.i.appmaster.TrackerService - Tracker service stopped at > > http://hdfs-ix03.se-ix.delta.prod:51793 > > Main class completed. > > Launcher completed > > Cleanup directory tmp/twill.launcher-1453549768670-0 > > > > > > > > SLF4J: Class path contains multiple SLF4J bindings. > > SLF4J: Found binding in > > > > > [jar:file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/logback-classic-1.0.9.jar!/org/slf4j/impl/StaticLoggerBinder.class] > > SLF4J: Found binding in > > > > > [jar:file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] > > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > > explanation. > > SLF4J: Actual binding is of type > > [ch.qos.logback.classic.util.ContextSelectorStaticBinder] > > 16/01/23 12:49:29 INFO impl.ContainerManagementProtocolProxy: > > yarn.client.max-cached-nodemanagers-proxies : 0 > > 16/01/23 12:49:30 INFO utils.VerifiableProperties: Verifying properties > > 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property log.dir is > > overridden to > > > /hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/kafka-logs > > 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property > > default.replication.factor is overridden to 1 > > 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property port is > > overridden to 58668 > > 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property > > socket.request.max.bytes is overridden to 104857600 > > 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property > > socket.send.buffer.bytes is overridden to 1048576 > > 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property > > log.flush.interval.ms is overridden to 1000 > > 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property > > zookeeper.connect is overridden to > > > > > zookeeper01.se-ix.delta.prod/JarApp/be4bbf01-5e72-4147-b2eb-b84e19214b5b/kafka > > 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property broker.id > > is overridden to 1 > > 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property > > log.retention.hours is overridden to 24 > > 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property > > socket.receive.buffer.bytes is overridden to 1048576 > > 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property > > zookeeper.connection.timeout.ms is overridden to 3000 > > 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property > > num.partitions is overridden to 1 > > 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property > > log.flush.interval.messages is overridden to 10000 > > 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property > > log.segment.bytes is overridden to 536870912 > > 16/01/23 12:49:30 INFO client.ConfiguredRMFailoverProxyProvider: > > Failing over to rm2 > > 16/01/23 12:49:30 INFO server.KafkaServer: [Kafka Server 1], Starting > > 16/01/23 12:49:30 INFO log.LogManager: [Log Manager on Broker 1] Log > > directory > > > '/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/kafka-logs' > > not found, creating it. > > 16/01/23 12:49:30 INFO log.LogManager: [Log Manager on Broker 1] > > Starting log cleaner every 600000 ms > > 16/01/23 12:49:30 INFO log.LogManager: [Log Manager on Broker 1] > > Starting log flusher every 3000 ms with the following overrides Map() > > 16/01/23 12:49:30 INFO network.Acceptor: Awaiting socket connections > > on 0.0.0.0:58668. > > 16/01/23 12:49:30 INFO network.SocketServer: [Socket Server on Broker > > 1], Started > > 16/01/23 12:49:30 INFO server.KafkaZooKeeper: connecting to ZK: > > > > > zookeeper01.se-ix.delta.prod/JarApp/be4bbf01-5e72-4147-b2eb-b84e19214b5b/kafka > > 16/01/23 12:49:30 INFO zkclient.ZkEventThread: Starting ZkClient event > > thread. > > 16/01/23 12:49:31 INFO zkclient.ZkClient: zookeeper state changed > > (SyncConnected) > > 16/01/23 12:49:31 INFO utils.ZkUtils$: Registered broker 1 at path > > /brokers/ids/1 with address hdfs-ix03.se-ix.delta.prod:58668. > > 16/01/23 12:49:31 INFO server.KafkaServer: [Kafka Server 1], > > Connecting to ZK: > > > > > zookeeper01.se-ix.delta.prod/JarApp/be4bbf01-5e72-4147-b2eb-b84e19214b5b/kafka > > 16/01/23 12:49:31 INFO utils.VerifiableProperties: Verifying properties > > 16/01/23 12:49:31 INFO utils.VerifiableProperties: Property > > metadata.broker.list is overridden to hdfs-ix03.se-ix.delta.prod:58668 > > 16/01/23 12:49:31 INFO utils.VerifiableProperties: Property > > request.required.acks is overridden to 1 > > 16/01/23 12:49:31 INFO utils.VerifiableProperties: Property > > partitioner.class is overridden to > > org.apache.twill.internal.kafka.client.IntegerPartitioner > > 16/01/23 12:49:31 INFO utils.VerifiableProperties: Property > > compression.codec is overridden to snappy > > 16/01/23 12:49:31 INFO utils.VerifiableProperties: Property > > key.serializer.class is overridden to > > org.apache.twill.internal.kafka.client.IntegerEncoder > > 16/01/23 12:49:31 INFO utils.VerifiableProperties: Property > > serializer.class is overridden to > > org.apache.twill.internal.kafka.client.ByteBufferEncoder > > 16/01/23 12:49:31 INFO utils.Mx4jLoader$: Will not load MX4J, > > mx4j-tools.jar is not in the classpath > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: > > Controller starting up > > 16/01/23 12:49:31 INFO server.ZookeeperLeaderElector: 1 successfully > > elected as leader > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: > > Broker 1 starting become controller state transition > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: > > Controller 1 incremented epoch to 1 > > 16/01/23 12:49:31 INFO client.ClientUtils$: Fetching metadata from > > broker id:0,host:hdfs-ix03.se-ix.delta.prod,port:58668 with > > correlation id 0 for 1 topic(s) Set(log) > > 16/01/23 12:49:31 INFO producer.SyncProducer: Connected to > > hdfs-ix03.se-ix.delta.prod:58668 for producing > > 16/01/23 12:49:31 INFO controller.RequestSendThread: > > [Controller-1-to-broker-1-send-thread], Starting > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: > > Currently active brokers in the cluster: Set(1) > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: > > Currently shutting brokers in the cluster: Set() > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: > > Current list of topics in the cluster: Set() > > 16/01/23 12:49:31 INFO controller.ReplicaStateMachine: [Replica state > > machine on controller 1]: No state transitions triggered since no > > partitions are assigned to brokers 1 > > 16/01/23 12:49:31 INFO controller.ReplicaStateMachine: [Replica state > > machine on controller 1]: Invoking state change to OnlineReplica for > > replicas > > 16/01/23 12:49:31 INFO controller.ReplicaStateMachine: [Replica state > > machine on controller 1]: Started replica state machine with initial > > state -> Map() > > 16/01/23 12:49:31 INFO controller.PartitionStateMachine: [Partition > > state machine on Controller 1]: Started partition state machine with > > initial state -> Map() > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: > > Broker 1 is ready to serve as the new controller with epoch 1 > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: > > Partitions being reassigned: Map() > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: > > Partitions already reassigned: List() > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: > > Resuming reassignment of partitions: Map() > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: > > Partitions undergoing preferred replica election: > > 16/01/23 12:49:31 INFO admin.AdminUtils$: Topic creation { > > "partitions":{ "0":[ 1 ] }, "version":1 } > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: > > Partitions that completed preferred replica election: > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: > > Resuming preferred replica election for partitions: > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: > > Starting preferred replica leader election for partitions > > 16/01/23 12:49:31 INFO controller.PartitionStateMachine: [Partition > > state machine on Controller 1]: Invoking state change to > > OnlinePartition for partitions > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: > > Controller startup complete > > 16/01/23 12:49:31 INFO server.KafkaApis: [KafkaApi-1] Auto creation of > > topic log with 1 partitions and replication factor 1 is successful! > > 16/01/23 12:49:31 INFO server.KafkaServer: [Kafka Server 1], Started > > 16/01/23 12:49:31 INFO > > server.ZookeeperLeaderElector$LeaderChangeListener: New leader is 1 > > 16/01/23 12:49:31 INFO controller.ControllerEpochListener: > > [ControllerEpochListener on 1]: Initialized controller epoch to 1 and > > zk version 0 > > 16/01/23 12:49:31 INFO producer.SyncProducer: Disconnecting from > > hdfs-ix03.se-ix.delta.prod:58668 > > 16/01/23 12:49:31 WARN producer.BrokerPartitionInfo: Error while > > fetching metadata [{TopicMetadata for topic log -> > > No partition metadata for topic log due to > > kafka.common.LeaderNotAvailableException}] for topic [log]: class > > kafka.common.LeaderNotAvailableException > > 16/01/23 12:49:31 INFO client.ClientUtils$: Fetching metadata from > > broker id:0,host:hdfs-ix03.se-ix.delta.prod,port:58668 with > > correlation id 1 for 1 topic(s) Set(log) > > 16/01/23 12:49:31 INFO producer.SyncProducer: Connected to > > hdfs-ix03.se-ix.delta.prod:58668 for producing > > 16/01/23 12:49:31 INFO > > controller.PartitionStateMachine$TopicChangeListener: > > [TopicChangeListener on Controller 1]: New topics: [Set(log)], deleted > > topics: [Set()], new partition replica assignment [Map([log,0] -> > > List(1))] > > 16/01/23 12:49:31 INFO admin.AdminUtils$: Topic creation { > > "partitions":{ "0":[ 1 ] }, "version":1 } > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: New > > topic creation callback for [log,0] > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: New > > partition creation callback for [log,0] > > 16/01/23 12:49:31 INFO producer.SyncProducer: Disconnecting from > > hdfs-ix03.se-ix.delta.prod:58668 > > 16/01/23 12:49:31 INFO controller.PartitionStateMachine: [Partition > > state machine on Controller 1]: Invoking state change to NewPartition > > for partitions [log,0] > > 16/01/23 12:49:31 WARN producer.BrokerPartitionInfo: Error while > > fetching metadata [{TopicMetadata for topic log -> > > No partition metadata for topic log due to > > kafka.common.LeaderNotAvailableException}] for topic [log]: class > > kafka.common.LeaderNotAvailableException > > 16/01/23 12:49:31 INFO network.Processor: Closing socket connection to > > /10.3.24.22. > > 16/01/23 12:49:31 INFO network.Processor: Closing socket connection to > > /10.3.24.22. > > 16/01/23 12:49:31 ERROR async.DefaultEventHandler: Failed to collate > > messages by topic, partition due to: Failed to fetch topic metadata > > for topic: log > > 16/01/23 12:49:31 INFO async.DefaultEventHandler: Back off for 100 ms > > before retrying send. Remaining retries = 3 > > 16/01/23 12:49:31 INFO controller.ReplicaStateMachine: [Replica state > > machine on controller 1]: Invoking state change to NewReplica for > > replicas PartitionAndReplica(log,0,1) > > 16/01/23 12:49:31 INFO controller.PartitionStateMachine: [Partition > > state machine on Controller 1]: Invoking state change to > > OnlinePartition for partitions [log,0] > > 16/01/23 12:49:31 INFO controller.ReplicaStateMachine: [Replica state > > machine on controller 1]: Invoking state change to OnlineReplica for > > replicas PartitionAndReplica(log,0,1) > > 16/01/23 12:49:31 INFO server.ReplicaManager: [Replica Manager on > > Broker 1]: Handling LeaderAndIsr request > > > > > Name:LeaderAndIsrRequest;Version:0;Controller:1;ControllerEpoch:1;CorrelationId:6;ClientId:id_1-host_null-port_58668;PartitionState:(log,0) > > -> > > > (LeaderAndIsrInfo:(Leader:1,ISR:1,LeaderEpoch:0,ControllerEpoch:1),ReplicationFactor:1),AllReplicas:1);Leaders:id:1,host:hdfs-ix03.se-ix.delta.prod,port:58668 > > 16/01/23 12:49:31 INFO server.ReplicaFetcherManager: > > [ReplicaFetcherManager on broker 1] Removing fetcher for partition > > [log,0] > > 16/01/23 12:49:31 INFO log.Log: [Kafka Log on Broker 1], Completed > > load of log log-0 with log end offset 0 > > 16/01/23 12:49:31 INFO log.LogManager: [Log Manager on Broker 1] > > Created log for partition [log,0] in > > > > > /hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/kafka-logs. > > 16/01/23 12:49:31 WARN server.HighwaterMarkCheckpoint: No > > highwatermark file is found. Returning 0 as the highwatermark for > > partition [log,0] > > 16/01/23 12:49:31 INFO server.ReplicaManager: [Replica Manager on > > Broker 1]: Handled leader and isr request > > > > > Name:LeaderAndIsrRequest;Version:0;Controller:1;ControllerEpoch:1;CorrelationId:6;ClientId:id_1-host_null-port_58668;PartitionState:(log,0) > > -> > > > (LeaderAndIsrInfo:(Leader:1,ISR:1,LeaderEpoch:0,ControllerEpoch:1),ReplicationFactor:1),AllReplicas:1);Leaders:id:1,host:hdfs-ix03.se-ix.delta.prod,port:58668 > > 16/01/23 12:49:31 INFO client.ClientUtils$: Fetching metadata from > > broker id:0,host:hdfs-ix03.se-ix.delta.prod,port:58668 with > > correlation id 2 for 1 topic(s) Set(log) > > 16/01/23 12:49:31 INFO producer.SyncProducer: Connected to > > hdfs-ix03.se-ix.delta.prod:58668 for producing > > 16/01/23 12:49:31 INFO admin.AdminUtils$: Topic creation { > > "partitions":{ "0":[ 1 ] }, "version":1 } > > 16/01/23 12:49:31 INFO producer.SyncProducer: Disconnecting from > > hdfs-ix03.se-ix.delta.prod:58668 > > 16/01/23 12:49:31 INFO network.Processor: Closing socket connection to > > /10.3.24.22. > > 16/01/23 12:49:31 WARN producer.BrokerPartitionInfo: Error while > > fetching metadata [{TopicMetadata for topic log -> > > No partition metadata for topic log due to > > kafka.common.LeaderNotAvailableException}] for topic [log]: class > > kafka.common.LeaderNotAvailableException > > 16/01/23 12:49:31 INFO client.ClientUtils$: Fetching metadata from > > broker id:0,host:hdfs-ix03.se-ix.delta.prod,port:58668 with > > correlation id 3 for 1 topic(s) Set(log) > > 16/01/23 12:49:31 INFO producer.SyncProducer: Connected to > > hdfs-ix03.se-ix.delta.prod:58668 for producing > > 16/01/23 12:49:31 INFO producer.SyncProducer: Disconnecting from > > hdfs-ix03.se-ix.delta.prod:58668 > > 16/01/23 12:49:31 INFO network.Processor: Closing socket connection to > > /10.3.24.22. > > 16/01/23 12:49:31 INFO producer.SyncProducer: Connected to > > hdfs-ix03.se-ix.delta.prod:58668 for producing > > 16/01/23 12:49:33 INFO impl.AMRMClientImpl: Received new token for : > > hdfs-ix03.se-ix.delta.prod:45454 > > 16/01/23 12:49:33 INFO impl.ContainerManagementProtocolProxy: Opening > > proxy : hdfs-ix03.se-ix.delta.prod:45454 > > 16/01/23 12:49:35 INFO network.Processor: Closing socket connection to > > /10.3.24.22. > > 16/01/23 12:49:35 INFO network.Processor: Closing socket connection to > > /10.3.24.22. > > 16/01/23 12:49:39 INFO server.KafkaServer: [Kafka Server 1], Shutting > down > > 16/01/23 12:49:39 INFO server.KafkaZooKeeper: Closing zookeeper client... > > 16/01/23 12:49:39 INFO zkclient.ZkEventThread: Terminate ZkClient event > > thread. > > 16/01/23 12:49:39 INFO network.SocketServer: [Socket Server on Broker > > 1], Shutting down > > 16/01/23 12:49:39 INFO network.SocketServer: [Socket Server on Broker > > 1], Shutdown completed > > 16/01/23 12:49:39 INFO server.KafkaRequestHandlerPool: [Kafka Request > > Handler on Broker 1], shutting down > > 16/01/23 12:49:39 INFO server.KafkaRequestHandlerPool: [Kafka Request > > Handler on Broker 1], shutted down completely > > 16/01/23 12:49:39 INFO utils.KafkaScheduler: Shutdown Kafka scheduler > > 16/01/23 12:49:39 INFO server.ReplicaManager: [Replica Manager on > > Broker 1]: Shut down > > 16/01/23 12:49:39 INFO server.ReplicaFetcherManager: > > [ReplicaFetcherManager on broker 1] shutting down > > 16/01/23 12:49:39 INFO server.ReplicaFetcherManager: > > [ReplicaFetcherManager on broker 1] shutdown completed > > 16/01/23 12:49:39 INFO server.ReplicaManager: [Replica Manager on > > Broker 1]: Shutted down completely > > 16/01/23 12:49:40 INFO controller.RequestSendThread: > > [Controller-1-to-broker-1-send-thread], Shutting down > > 16/01/23 12:49:40 INFO controller.RequestSendThread: > > [Controller-1-to-broker-1-send-thread], Stopped > > 16/01/23 12:49:40 INFO controller.RequestSendThread: > > [Controller-1-to-broker-1-send-thread], Shutdown completed > > 16/01/23 12:49:40 INFO controller.KafkaController: [Controller 1]: > > Controller shutdown complete > > 16/01/23 12:49:40 INFO server.KafkaServer: [Kafka Server 1], Shut down > > completed > > 16/01/23 12:49:40 INFO impl.ContainerManagementProtocolProxy: Opening > > proxy : hdfs-ix03.se-ix.delta.prod:45454 > > 16/01/23 12:49:40 INFO impl.AMRMClientImpl: Waiting for application to > > be successfully unregistered. > > 16/01/23 12:49:40 INFO producer.SyncProducer: Disconnecting from > > hdfs-ix03.se-ix.delta.prod:58668 > > 16/01/23 12:49:40 WARN async.DefaultEventHandler: Failed to send > > producer request with correlation id 35 to broker 1 with data for > > partitions [log,0] > > java.nio.channels.ClosedByInterruptException > > at > > > java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202) > > at sun.nio.ch.SocketChannelImpl.poll(SocketChannelImpl.java:957) > > at > sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:204) > > at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) > > at > > > java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385) > > at kafka.utils.Utils$.read(Unknown Source) > > at kafka.network.BoundedByteBufferReceive.readFrom(Unknown Source) > > at kafka.network.Receive$class.readCompletely(Unknown Source) > > at kafka.network.BoundedByteBufferReceive.readCompletely(Unknown Source) > > at kafka.network.BlockingChannel.receive(Unknown Source) > > at kafka.producer.SyncProducer.liftedTree1$1(Unknown Source) > > at > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(Unknown > > Source) > > at > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(Unknown > > Source) > > at > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(Unknown > > Source) > > at > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(Unknown > > Source) > > at kafka.metrics.KafkaTimer.time(Unknown Source) > > at kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(Unknown > Source) > > at kafka.producer.SyncProducer$$anonfun$send$1.apply(Unknown Source) > > at kafka.producer.SyncProducer$$anonfun$send$1.apply(Unknown Source) > > at kafka.metrics.KafkaTimer.time(Unknown Source) > > at kafka.producer.SyncProducer.send(Unknown Source) > > at > > > kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$send(Unknown > > Source) > > at > > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$2.apply(Unknown > > Source) > > at > > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$2.apply(Unknown > > Source) > > at > > > scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772) > > at > > > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) > > at > > > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) > > at > > > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226) > > at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) > > at scala.collection.mutable.HashMap.foreach(HashMap.scala:98) > > at > > > scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771) > > at > kafka.producer.async.DefaultEventHandler.dispatchSerializedData(Unknown > > Source) > > at kafka.producer.async.DefaultEventHandler.handle(Unknown Source) > > at kafka.producer.Producer.send(Unknown Source) > > at kafka.javaapi.producer.Producer.send(Unknown Source) > > at > > > org.apache.twill.internal.kafka.client.SimpleKafkaPublisher$SimplePreparer.send(SimpleKafkaPublisher.java:122) > > at > > > org.apache.twill.internal.logging.KafkaAppender.doPublishLogs(KafkaAppender.java:268) > > at > > > org.apache.twill.internal.logging.KafkaAppender.publishLogs(KafkaAppender.java:228) > > at > > > org.apache.twill.internal.logging.KafkaAppender.access$700(KafkaAppender.java:66) > > at > > > org.apache.twill.internal.logging.KafkaAppender$2.run(KafkaAppender.java:280) > > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > > at > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > > at > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > > at > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > > at java.lang.Thread.run(Thread.java:745) > > 16/01/23 12:49:40 INFO async.DefaultEventHandler: Back off for 100 ms > > before retrying send. Remaining retries = 3 > > 16/01/23 12:49:40 INFO producer.Producer: Shutting down producer > > 16/01/23 12:49:40 INFO producer.ProducerPool: Closing all sync producers > > > > > > On Sat, Jan 23, 2016 at 1:22 AM, Terence Yim <[email protected]> wrote: > > > Hi, > > > > > > It's due to a very old version of ASM library that bring it by > > hadoop/yarn. > > > Please add exclusion of asm library to all hadoop dependencies. > > > > > > <exclusion> > > > <groupId>asm</groupId> > > > <artifactId>asm</artifactId> > > > </exclusion> > > > > > > Terence > > > > > > > > > On Fri, Jan 22, 2016 at 2:34 PM, Kristoffer Sjögren <[email protected]> > > > wrote: > > > > > >> Further adding the following dependencies cause another exception. > > >> > > >> <dependency> > > >> <groupId>com.google.guava</groupId> > > >> <artifactId>guava</artifactId> > > >> <version>13.0</version> > > >> </dependency> > > >> <dependency> > > >> <groupId>org.apache.hadoop</groupId> > > >> <artifactId>hadoop-hdfs</artifactId> > > >> <version>2.7.1</version> > > >> </dependency> > > >> > > >> Exception in thread " STARTING" > > >> java.lang.IncompatibleClassChangeError: class > > >> org.apache.twill.internal.utils.Dependencies$DependencyClassVisitor > > >> has interface org.objectweb.asm.ClassVisitor as super class > > >> at java.lang.ClassLoader.defineClass1(Native Method) > > >> at java.lang.ClassLoader.defineClass(ClassLoader.java:760) > > >> at > > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > > >> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) > > >> at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > > >> at java.net.URLClassLoader$1.run(URLClassLoader.java:368) > > >> at java.net.URLClassLoader$1.run(URLClassLoader.java:362) > > >> at java.security.AccessController.doPrivileged(Native Method) > > >> at java.net.URLClassLoader.findClass(URLClassLoader.java:361) > > >> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > > >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > > >> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > > >> at > > >> > > > org.apache.twill.internal.utils.Dependencies.findClassDependencies(Dependencies.java:86) > > >> at > > >> > > > org.apache.twill.internal.ApplicationBundler.findDependencies(ApplicationBundler.java:198) > > >> at > > >> > > > org.apache.twill.internal.ApplicationBundler.createBundle(ApplicationBundler.java:155) > > >> at > > >> > > > org.apache.twill.internal.ApplicationBundler.createBundle(ApplicationBundler.java:126) > > >> at > > >> > > > org.apache.twill.yarn.YarnTwillPreparer.createAppMasterJar(YarnTwillPreparer.java:402) > > >> at > > >> > > > org.apache.twill.yarn.YarnTwillPreparer.access$200(YarnTwillPreparer.java:108) > > >> at > > >> > > > org.apache.twill.yarn.YarnTwillPreparer$1.call(YarnTwillPreparer.java:299) > > >> at > > >> > > > org.apache.twill.yarn.YarnTwillPreparer$1.call(YarnTwillPreparer.java:289) > > >> at > > >> > > > org.apache.twill.yarn.YarnTwillController.doStartUp(YarnTwillController.java:97) > > >> at > > >> > > > org.apache.twill.internal.AbstractZKServiceController.startUp(AbstractZKServiceController.java:76) > > >> at > > >> > > > org.apache.twill.internal.AbstractExecutionServiceController$ServiceDelegate.startUp(AbstractExecutionServiceController.java:175) > > >> at > > >> > > > com.google.common.util.concurrent.AbstractIdleService$1$1.run(AbstractIdleService.java:43) > > >> at java.lang.Thread.run(Thread.java:745) > > >> > > >> On Fri, Jan 22, 2016 at 11:32 PM, Kristoffer Sjögren < > [email protected]> > > >> wrote: > > >> > Add those dependencies fail with the following exception. > > >> > > > >> > Exception in thread "main" java.lang.AbstractMethodError: > > >> > > > >> > > > org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.getProxy()Lorg/apache/hadoop/io/retry/FailoverProxyProvider$ProxyInfo; > > >> > at > > >> > > > org.apache.hadoop.io.retry.RetryInvocationHandler.<init>(RetryInvocationHandler.java:73) > > >> > at > > >> > > > org.apache.hadoop.io.retry.RetryInvocationHandler.<init>(RetryInvocationHandler.java:64) > > >> > at org.apache.hadoop.io.retry.RetryProxy.create(RetryProxy.java:59) > > >> > at > > >> > > > org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:149) > > >> > at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:569) > > >> > at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:512) > > >> > at > > >> > > > org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:142) > > >> > at > > org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653) > > >> > at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92) > > >> > at > > >> > org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687) > > >> > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669) > > >> > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371) > > >> > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:170) > > >> > at > > >> > > > org.apache.twill.yarn.YarnTwillRunnerService.createDefaultLocationFactory(YarnTwillRunnerService.java:615) > > >> > at > > >> > > > org.apache.twill.yarn.YarnTwillRunnerService.<init>(YarnTwillRunnerService.java:149) > > >> > at deephacks.BundledJarExample.main(BundledJarExample.java:70) > > >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > >> > at > > >> > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > >> > at > > >> > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > >> > at java.lang.reflect.Method.invoke(Method.java:497) > > >> > at > > com.intellij.rt.execution.application.AppMain.main(AppMain.java:144) > > >> > > > >> > On Fri, Jan 22, 2016 at 10:58 PM, Terence Yim <[email protected]> > > wrote: > > >> >> Hi, > > >> >> > > >> >> If you run it from IDE, you and simply add a dependency on hadoop > > with > > >> >> version 2.7.1. E.g. if you are using Maven, you can add the > > following to > > >> >> your pom.xml dependencies section. > > >> >> > > >> >> <dependency> > > >> >> <groupId>org.apache.hadoop</groupId> > > >> >> <artifactId>hadoop-yarn-api</artifactId> > > >> >> <version>2.7.1</version> > > >> >> </dependency> > > >> >> <dependency> > > >> >> <groupId>org.apache.hadoop</groupId> > > >> >> <artifactId>hadoop-yarn-common</artifactId> > > >> >> <version>2.7.1</version> > > >> >> </dependency> > > >> >> <dependency> > > >> >> <groupId>org.apache.hadoop</groupId> > > >> >> <artifactId>hadoop-yarn-client</artifactId> > > >> >> <version>2.7.1</version> > > >> >> </dependency> > > >> >> <dependency> > > >> >> <groupId>org.apache.hadoop</groupId> > > >> >> <artifactId>hadoop-common</artifactId> > > >> >> <version>2.7.1</version> > > >> >> </dependency> > > >> >> > > >> >> Terence > > >> >> > > >> >> On Fri, Jan 22, 2016 at 12:47 PM, Kristoffer Sjögren < > > [email protected]> > > >> >> wrote: > > >> >> > > >> >>> I run it from IDE right now, but would like to create a command > line > > >> >>> app eventually. > > >> >>> > > >> >>> I should clarify that the exception above is thrown on the YARN > > node, > > >> >>> not in the IDE. > > >> >>> > > >> >>> On Fri, Jan 22, 2016 at 9:32 PM, Terence Yim <[email protected]> > > wrote: > > >> >>> > Hi Kristoffer, > > >> >>> > > > >> >>> > The example itself shouldn't need any modification. However, how > > do > > >> >>> > you run that class? Do you run it from IDE or from command line > > using > > >> >>> > "java" command? > > >> >>> > > > >> >>> > Terence > > >> >>> > > > >> >>> > On Fri, Jan 22, 2016 at 12:16 PM, Kristoffer Sjögren < > > >> [email protected]> > > >> >>> wrote: > > >> >>> >> Hi Terence, > > >> >>> >> > > >> >>> >> I'm quite new to Twill and not sure how to do that exactly. > Could > > >> you > > >> >>> >> show me how to modify the following example to do the same? > > >> >>> >> > > >> >>> >> > > >> >>> > > >> > > > https://github.com/apache/incubator-twill/blob/master/twill-examples/yarn/src/main/java/org/apache/twill/example/yarn/BundledJarExample.java > > >> >>> >> > > >> >>> >> On Fri, Jan 22, 2016 at 7:24 PM, Terence Yim <[email protected] > > > > >> wrote: > > >> >>> >>> Hi Kristoffer, > > >> >>> >>> > > >> >>> >>> Seems like the exception comes from the YARN class > > >> "ConverterUtils". I > > >> >>> >>> believe need to start the application with the version 2.7.1 > > Hadoop > > >> >>> >>> Jars. How to do start the twill application? Usually on a > > cluster > > >> with > > >> >>> >>> hadoop installed, you can get all the hadoop jars in the > > classpath > > >> by > > >> >>> >>> running this: > > >> >>> >>> > > >> >>> >>> export CP=`hadoop classpath` > > >> >>> >>> java -cp .:$CP YourApp ... > > >> >>> >>> > > >> >>> >>> Assuming your app classes and Twill jars are in the current > > >> directory. > > >> >>> >>> > > >> >>> >>> Terence > > >> >>> >>> > > >> >>> >>> On Fri, Jan 22, 2016 at 4:54 AM, Kristoffer Sjögren < > > >> [email protected]> > > >> >>> wrote: > > >> >>> >>>> Here's the full stacktrace. > > >> >>> >>>> > > >> >>> >>>> Exception in thread "main" > > >> java.lang.reflect.InvocationTargetException > > >> >>> >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > > >> >>> >>>> at > > >> >>> > > >> > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > >> >>> >>>> at > > >> >>> > > >> > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > >> >>> >>>> at java.lang.reflect.Method.invoke(Method.java:497) > > >> >>> >>>> at > > >> org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:89) > > >> >>> >>>> Caused by: java.lang.RuntimeException: > > >> >>> >>>> java.lang.reflect.InvocationTargetException > > >> >>> >>>> at > > >> com.google.common.base.Throwables.propagate(Throwables.java:160) > > >> >>> >>>> at > > >> >>> > > >> > > > org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61) > > >> >>> >>>> at > > >> >>> > > >> > > > org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:77) > > >> >>> >>>> ... 5 more > > >> >>> >>>> Caused by: java.lang.reflect.InvocationTargetException > > >> >>> >>>> at > > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > > >> >>> Method) > > >> >>> >>>> at > > >> >>> > > >> > > > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > > >> >>> >>>> at > > >> >>> > > >> > > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > > >> >>> >>>> at > > java.lang.reflect.Constructor.newInstance(Constructor.java:422) > > >> >>> >>>> at > > >> >>> > > >> > > > org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58) > > >> >>> >>>> ... 6 more > > >> >>> >>>> Caused by: java.lang.IllegalArgumentException: Invalid > > >> ContainerId: > > >> >>> >>>> container_e25_1453466340022_0004_01_000001 > > >> >>> >>>> at > > >> >>> > > >> > > > org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182) > > >> >>> >>>> at > > >> >>> > > >> > > > org.apache.twill.internal.yarn.AbstractYarnAMClient.<init>(AbstractYarnAMClient.java:83) > > >> >>> >>>> at > > >> >>> > > >> > > > org.apache.twill.internal.yarn.Hadoop21YarnAMClient.<init>(Hadoop21YarnAMClient.java:65) > > >> >>> >>>> at > > >> >>> > > >> > > > org.apache.twill.internal.yarn.Hadoop22YarnAMClient.<init>(Hadoop22YarnAMClient.java:34) > > >> >>> >>>> ... 11 more > > >> >>> >>>> Caused by: java.lang.NumberFormatException: For input string: > > >> "e25" > > >> >>> >>>> at > > >> >>> > > >> > > > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > > >> >>> >>>> at java.lang.Long.parseLong(Long.java:589) > > >> >>> >>>> at java.lang.Long.parseLong(Long.java:631) > > >> >>> >>>> at > > >> >>> > > >> > > > org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137) > > >> >>> >>>> at > > >> >>> > > >> > > > org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177) > > >> >>> >>>> ... 14 more > > >> >>> >>>> > > >> >>> >>>> On Thu, Jan 21, 2016 at 10:59 PM, Kristoffer Sjögren < > > >> >>> [email protected]> wrote: > > >> >>> >>>>> Hi > > >> >>> >>>>> > > >> >>> >>>>> I'm trying the basic example [1] on yarn 2.7.1 but get an > > >> exception > > >> >>> as > > >> >>> >>>>> soon as the application starts on the resource manager that > > >> tells me > > >> >>> >>>>> the container id cannot be parsed. > > >> >>> >>>>> > > >> >>> >>>>> java.lang.IllegalArgumentException: Invalid containerId: > > >> >>> >>>>> container_e04_1427159778706_0002_01_000001 > > >> >>> >>>>> > > >> >>> >>>>> I don't have the exact stacktrace but I recall it failing in > > >> >>> >>>>> ConverterUtils.toContainerId because it assumes that that > the > > >> first > > >> >>> >>>>> token is an application attempt to be parsed as an integer. > > This > > >> >>> class > > >> >>> >>>>> resides in hadoop-yarn-common 2.3.0. > > >> >>> >>>>> > > >> >>> >>>>> Is there any way to either tweak the container id or make > > twill > > >> use > > >> >>> >>>>> the 2.7.1 jar instead? > > >> >>> >>>>> > > >> >>> >>>>> Cheers, > > >> >>> >>>>> -Kristoffer > > >> >>> >>>>> > > >> >>> >>>>> > > >> >>> >>>>> [1] > > >> >>> > > >> > > > https://github.com/apache/incubator-twill/blob/master/twill-examples/yarn/src/main/java/org/apache/twill/example/yarn/BundledJarExample.java > > >> >>> > > >> > > >
