Hi! It looks like multiple Hadoop versions are in the classpath. Flink's hadoop jar and the EMR Hadoop jars. I would simply drop Flink's own Hadoop dependency and only use the EMR Hadoop jars.
Delete the 'flink-shaded-hadoop2-uber' jar from Flink, and make sure the setup is such that the Hadoop lib environment variable is set. Then it should not have conflicts any more. On Sun, Oct 8, 2017 at 12:08 AM, Chen Qin <qinnc...@gmail.com> wrote: > Attached my side project verified working to deploy jobmanager and > taskmanager as stateless service(non yarn/mesos), configuration here > > https://github.com/chenqin/flink-jar/tree/master/config/hadoop > > more detail here > https://github.com/chenqin/flink-jar/blob/master/src/ > main/java/FlinkBootstrap.java#L49 > > On Fri, Oct 6, 2017 at 10:26 PM, Bowen Li <bowen...@offerupnow.com> wrote: > > > Hi Andy, > > > > I believe it's because you didn't set your s3 impl correctly. Try to set > > your core-site.xml by following https://ci.apache.org/ > > projects/flink/flink-docs-release-1.4/ops/deployment/ > > aws.html#s3afilesystem- > > recommended > > > > Bowen > > > > On Fri, Oct 6, 2017 at 7:59 AM, Andy M. <ajm2...@gmail.com> wrote: > > > > > Hi Till, > > > > > > Seems like everything is in line there. hadoop-common.jar -> > > > hadoop-common-2.7.3-amzn-3.jar > > > > > > And when i decompiled that jar I see public void > > addResource(Configuration > > > conf) in org/apache/hadoop/conf/Configuration.java > > > > > > I agree that an incorrect version of the jar is probably being run, is > > > there a way to limit the classpath for the TaskManager when starting > the > > > job? > > > > > > Thank you > > > > > > On Fri, Oct 6, 2017 at 6:49 AM, Till Rohrmann <trohrm...@apache.org> > > > wrote: > > > > > > > Hi Andy, > > > > > > > > could you check which Hadoop version this jar > > > > /usr/lib/hadoop/hadoop-common.jar is? Maybe also checking whether > the > > > > contained hadoop Configuration class has the method > > > > Configuration.addResource(Lorg/apache/hadoop/conf/Configuration;)V. > > > Maybe > > > > this jar is the culprit because it comes from a different Hadoop > > version. > > > > > > > > Cheers, > > > > Till > > > > > > > > > > > > On Thu, Oct 5, 2017 at 4:22 PM, Andy M. <ajm2...@gmail.com> wrote: > > > > > > > > > Hi Till, > > > > > > > > > > I believe this is what you are looking for, classpath is much > bigger > > > for > > > > > the task manager. I can also post the whole log file if needed: > > > > > > > > > > 2017-10-05 14:17:53,038 INFO org.apache.flink.yarn. > > > > YarnTaskManagerRunner > > > > > - Classpath: > > > > > flink-consumer.jar:lib/flink-dist_2.11-1.3.2.jar:lib/flink- > > > > > python_2.11-1.3.2.jar:lib/flink-shaded-hadoop2-uber-1.3. > > > > > 2.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar: > > > > > log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/ > > > > > etc/hadoop/conf:/usr/lib/hadoop/hadoop-common-2.7.3- > > > > > amzn-3-tests.jar:/usr/lib/hadoop/hadoop-annotations-2.7. > > > > > 3-amzn-3.jar:/usr/lib/hadoop/hadoop-distcp.jar:/usr/lib/ > > > > > hadoop/hadoop-auth-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop- > > > > > nfs-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-streaming-2.7.3- > > > > > amzn-3.jar:/usr/lib/hadoop/hadoop-ant-2.7.3-amzn-3.jar:/ > > > > > usr/lib/hadoop/hadoop-distcp-2.7.3-amzn-3.jar:/usr/lib/ > > > > > hadoop/hadoop-datajoin.jar:/usr/lib/hadoop/hadoop- > > > > > streaming.jar:/usr/lib/hadoop/hadoop-common.jar:/usr/lib/ > > > > > hadoop/hadoop-ant.jar:/usr/lib/hadoop/hadoop-sls.jar:/ > > > > > usr/lib/hadoop/hadoop-azure-2.7.3-amzn-3.jar:/usr/lib/ > > > > > hadoop/hadoop-nfs.jar:/usr/lib/hadoop/hadoop-extras-2.7. > > > > > 3-amzn-3.jar:/usr/lib/hadoop/hadoop-gridmix.jar:/usr/lib/ > > > > > hadoop/hadoop-common-2.7.3-amzn-3.jar:/usr/lib/hadoop/ > > > > > hadoop-annotations.jar:/usr/lib/hadoop/hadoop-openstack-2. > > > > > 7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-archives-2.7.3- > > > > > amzn-3.jar:/usr/lib/hadoop/hadoop-azure.jar:/usr/lib/ > > > > > hadoop/hadoop-extras.jar:/usr/lib/hadoop/hadoop-openstack. > > > > > jar:/usr/lib/hadoop/hadoop-rumen.jar:/usr/lib/hadoop/ > > > > > hadoop-aws-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop- > > > > > datajoin-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop- > > > > > archives.jar:/usr/lib/hadoop/hadoop-aws.jar:/usr/lib/ > > > > > hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-rumen-2.7.3- > > > > > amzn-3.jar:/usr/lib/hadoop/hadoop-sls-2.7.3-amzn-3.jar:/ > > > > > usr/lib/hadoop/hadoop-gridmix-2.7.3-amzn-3.jar:/usr/lib/ > > > > > hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/jaxb-api-2. > > > > > 2.2.jar:/usr/lib/hadoop/lib/htrace-core-3.1.0-incubating. > > > > > jar:/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar:/usr/ > > > > > lib/hadoop/lib/httpclient-4.5.3.jar:/usr/lib/hadoop/lib/ > > > > > httpcore-4.4.4.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4. > > > > > 1.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0. > > > > > jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/ > > > > > lib/activation-1.1.jar:/usr/lib/hadoop/lib/jersey-server- > > > > > 1.9.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/ > > > > > usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/ > > > > > gson-2.2.4.jar:/usr/lib/hadoop/lib/commons-digester-1. > > > > > 8.jar:/usr/lib/hadoop/lib/jackson-xc-1.9.13.jar:/usr/ > > > > > lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/ > > > > > apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop/lib/commons- > > > > > httpclient-3.1.jar:/usr/lib/hadoop/lib/curator-client-2.7. > > > > > 1.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/ > > > > > lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/ > > > > > commons-net-3.1.jar:/usr/lib/hadoop/lib/jets3t-0.9.0.jar:/ > > > > > usr/lib/hadoop/lib/commons-io-2.4.jar:/usr/lib/hadoop/lib/ > > > > > xmlenc-0.52.jar:/usr/lib/hadoop/lib/jersey-json-1.9. > > > > > jar:/usr/lib/hadoop/lib/avro-1.7.4.jar:/usr/lib/hadoop/lib/ > > > > > commons-math3-3.1.1.jar:/usr/lib/hadoop/lib/log4j-1.2.17. > > > > > jar:/usr/lib/hadoop/lib/commons-compress-1.4.1.jar:/ > > > > > usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/ > > > > > jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/netty-3.6.2. > > > > > Final.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/ > > > > > lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/api-asn1- > > > > > api-1.0.0-M20.jar:/usr/lib/hadoop/lib/jackson-mapper-asl- > > > > > 1.9.13.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26-emr.jar: > > > > > /usr/lib/hadoop/lib/xz-1.0.jar:/usr/lib/hadoop/lib/ > > > > > jackson-core-asl-1.9.13.jar:/usr/lib/hadoop/lib/commons- > > > > > cli-1.2.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/ > > > > > lib/hadoop/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop/lib/ > > > > > apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop/lib/ > > > > > commons-collections-3.2.2.jar:/usr/lib/hadoop/lib/junit-4. > > > > > 11.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar: > > > > > /usr/lib/hadoop/lib/api-util-1.0.0-M20.jar:/usr/lib/hadoop/ > > > > > lib/stax-api-1.0-2.jar:/usr/lib/hadoop/lib/slf4j-api-1.7. > > > > > 10.jar:/usr/lib/hadoop/lib/curator-recipes-2.7.1.jar:/ > > > > > usr/lib/hadoop/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop/ > > > > > lib/jsr305-3.0.0.jar:/usr/lib/hadoop/lib/zookeeper-3.4.10. > > > > > jar:/usr/lib/hadoop/lib/commons-lang-2.6.jar:/usr/lib/ > > > > > hadoop/lib/jersey-core-1.9.jar:/usr/lib/hadoop/lib/ > > > > > curator-framework-2.7.1.jar:/usr/lib/hadoop/lib/jackson- > > > > > jaxrs-1.9.13.jar:/usr/lib/hadoop/lib/java-xmlbuilder-0. > > > > > 4.jar:/usr/lib/hadoop/lib/commons-logging-1.1.3.jar:/ > > > > > usr/lib/hadoop-hdfs/hadoop-hdfs-nfs-2.7.3-amzn-3.jar:/ > > > > > usr/lib/hadoop-hdfs/hadoop-hdfs-2.7.3-amzn-3-tests.jar:/ > > > > > usr/lib/hadoop-hdfs/hadoop-hdfs.jar:/usr/lib/hadoop-hdfs/ > > > > > hadoop-hdfs-nfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.7.3- > > > > > amzn-3.jar:/usr/lib/hadoop-hdfs/lib/htrace-core-3.1.0- > > > > > incubating.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java- > > > > > 2.5.0.jar:/usr/lib/hadoop-hdfs/lib/leveldbjni-all-1.8. > > > > > jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1.9.jar:/usr/ > > > > > lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-hdfs/ > > > > > lib/commons-codec-1.4.jar:/usr/lib/hadoop-hdfs/lib/ > > > > > commons-io-2.4.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.52. > > > > > jar:/usr/lib/hadoop-hdfs/lib/netty-all-4.0.23.Final.jar:/ > > > > > usr/lib/hadoop-hdfs/lib/xercesImpl-2.9.1.jar:/usr/lib/ > > > > > hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/ > > > > > netty-3.6.2.Final.jar:/usr/lib/hadoop-hdfs/lib/xml-apis- > > > > > 1.3.04.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/ > > > > > hadoop-hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/ > > > > > lib/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/usr/lib/ > > > > > hadoop-hdfs/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop- > > > > > hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/ > > > > > lib/commons-cli-1.2.jar:/usr/lib/hadoop-hdfs/lib/servlet- > > > > > api-2.5.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26-emr.jar: > > > > > /usr/lib/hadoop-hdfs/lib/jsr305-3.0.0.jar:/usr/lib/ > > > > > hadoop-hdfs/lib/commons-lang-2.6.jar:/usr/lib/hadoop-hdfs/ > > > > > lib/jersey-core-1.9.jar:/usr/lib/hadoop-hdfs/lib/commons- > > > > > logging-1.1.3.jar:/usr/lib/hadoop-mapreduce/jettison-1.1. > > > > > jar:/usr/lib/hadoop-mapreduce/jaxb-api-2.2.2.jar:/usr/lib/ > > > > > hadoop-mapreduce/hadoop-distcp.jar:/usr/lib/hadoop- > > > > > mapreduce/htrace-core-3.1.0-incubating.jar:/usr/lib/ > > > > > hadoop-mapreduce/hadoop-auth-2.7.3-amzn-3.jar:/usr/lib/ > > > > > hadoop-mapreduce/protobuf-java-2.5.0.jar:/usr/lib/ > > > > hadoop-mapreduce/hadoop- > > > > > mapreduce-client-core.jar:/usr/lib/hadoop-mapreduce/ > > > > > httpclient-4.5.3.jar:/usr/lib/hadoop-mapreduce/hadoop- > > > > > mapreduce-client-shuffle.jar:/usr/lib/hadoop-mapreduce/ > > > > > hadoop-mapreduce-client-common-2.7.3-amzn-3.jar:/usr/ > > > > > lib/hadoop-mapreduce/httpcore-4.4.4.jar:/usr/lib/hadoop- > > > > > mapreduce/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/ > > > > > commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-mapreduce/ > > > > > hadoop-mapreduce-client-core-2.7.3-amzn-3.jar:/usr/lib/ > > > > > hadoop-mapreduce/jsp-api-2.1.jar:/usr/lib/hadoop-mapreduce/ > > > > > jmespath-java-1.11.160.jar:/usr/lib/hadoop-mapreduce/ > > > > > hadoop-mapreduce-client-hs-plugins.jar:/usr/lib/hadoop- > > > > > mapreduce/activation-1.1.jar:/usr/lib/hadoop-mapreduce/ > > > > > hadoop-mapreduce-client-jobclient-2.7.3-amzn-3-tests. > > > > > jar:/usr/lib/hadoop-mapreduce/jersey-server-1.9.jar:/usr/ > > > > > lib/hadoop-mapreduce/hadoop-streaming-2.7.3-amzn-3.jar:/ > > > > > usr/lib/hadoop-mapreduce/commons-beanutils-1.7.0.jar:/ > > > > > usr/lib/hadoop-mapreduce/hadoop-ant-2.7.3-amzn-3.jar:/ > > > > > usr/lib/hadoop-mapreduce/guava-11.0.2.jar:/usr/lib/ > > > > > hadoop-mapreduce/gson-2.2.4.jar:/usr/lib/hadoop-mapreduce/ > > > > > hadoop-distcp-2.7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/ > > > > > hadoop-mapreduce-client-app-2.7.3-amzn-3.jar:/usr/lib/ > > > > > hadoop-mapreduce/hadoop-mapreduce-examples-2.7.3-amzn- > > > > > 3.jar:/usr/lib/hadoop-mapreduce/commons-digester-1. > > > > 8.jar:/usr/lib/hadoop- > > > > > mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.3-amzn- > > > > > 3.jar:/usr/lib/hadoop-mapreduce/jackson-xc-1.9.13. > > > > > jar:/usr/lib/hadoop-mapreduce/paranamer-2.3.jar:/usr/lib/ > > > > > hadoop-mapreduce/apacheds-i18n-2.0.0-M15.jar:/usr/lib/ > > > > > hadoop-mapreduce/commons-httpclient-3.1.jar:/usr/lib/ > > > > > hadoop-mapreduce/curator-client-2.7.1.jar:/usr/lib/ > > > > > hadoop-mapreduce/hadoop-datajoin.jar:/usr/lib/hadoop- > > > > > mapreduce/hadoop-streaming.jar:/usr/lib/hadoop-mapreduce/ > > > > > commons-codec-1.4.jar:/usr/lib/hadoop-mapreduce/commons- > > > > > net-3.1.jar:/usr/lib/hadoop-mapreduce/jets3t-0.9.0.jar:/ > > > > > usr/lib/hadoop-mapreduce/commons-io-2.4.jar:/usr/lib/ > > > > > hadoop-mapreduce/hadoop-ant.jar:/usr/lib/hadoop-mapreduce/ > > > > > xmlenc-0.52.jar:/usr/lib/hadoop-mapreduce/jersey-json- > > > > > 1.9.jar:/usr/lib/hadoop-mapreduce/hadoop-sls.jar:/usr/ > > > > > lib/hadoop-mapreduce/avro-1.7.4.jar:/usr/lib/hadoop- > > > > > mapreduce/aws-java-sdk-core-1.11.160.jar:/usr/lib/hadoop- > > > > > mapreduce/aws-java-sdk-s3-1.11.160.jar:/usr/lib/hadoop- > > > > > mapreduce/jackson-dataformat-cbor-2.6.6.jar:/usr/lib/ > > > > > hadoop-mapreduce/hadoop-azure-2.7.3-amzn-3.jar:/usr/lib/ > > > > > hadoop-mapreduce/commons-math3-3.1.1.jar:/usr/lib/ > > > > > hadoop-mapreduce/ion-java-1.0.2.jar:/usr/lib/hadoop- > > > > > mapreduce/azure-storage-2.0.0.jar:/usr/lib/hadoop-mapreduce/ > > > > > hadoop-mapreduce-client-shuffle-2.7.3-amzn-3.jar:/usr/ > > > > > lib/hadoop-mapreduce/log4j-1.2.17.jar:/usr/lib/hadoop- > > > > > mapreduce/commons-compress-1.4.1.jar:/usr/lib/hadoop- > > > > > mapreduce/hadoop-extras-2.7.3-amzn-3.jar:/usr/lib/hadoop- > > > > > mapreduce/hadoop-gridmix.jar:/usr/lib/hadoop-mapreduce/jsch- > > > > > 0.1.42.jar:/usr/lib/hadoop-mapreduce/joda-time-2.8.1.jar: > > > > > /usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-2. > > > > > 7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/jaxb-impl-2. > > > > > 2.3-1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce- > > > > > client-hs.jar:/usr/lib/hadoop-mapreduce/netty-3.6.2.Final. > > > > > jar:/usr/lib/hadoop-mapreduce/mockito-all-1.8.5.jar:/usr/ > > > > > lib/hadoop-mapreduce/hadoop-openstack-2.7.3-amzn-3.jar:/ > > > > > usr/lib/hadoop-mapreduce/hadoop-mapreduce-client- > > > > > common.jar:/usr/lib/hadoop-mapreduce/hadoop-archives-2.7. > > > > > 3-amzn-3.jar:/usr/lib/hadoop-mapreduce/hadoop-azure.jar:/ > > > > > usr/lib/hadoop-mapreduce/hadoop-mapreduce-client- > > > > > jobclient.jar:/usr/lib/hadoop-mapreduce/hadoop-extras.jar:/ > > > > > usr/lib/hadoop-mapreduce/asm-3.2.jar:/usr/lib/hadoop- > > > > > mapreduce/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop- > > > > > mapreduce/hadoop-openstack.jar:/usr/lib/hadoop-mapreduce/ > > > > > hadoop-mapreduce-client-jobclient-2.7.3-amzn-3.jar:/ > > > > > usr/lib/hadoop-mapreduce/hadoop-rumen.jar:/usr/lib/ > > > > > hadoop-mapreduce/aws-java-sdk-kms-1.11.160.jar:/usr/lib/ > > > > > hadoop-mapreduce/jackson-mapper-asl-1.9.13.jar:/usr/ > > > > > lib/hadoop-mapreduce/hadoop-aws-2.7.3-amzn-3.jar:/usr/lib/ > > > > > hadoop-mapreduce/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop- > > > > > mapreduce/hadoop-datajoin-2.7.3-amzn-3.jar:/usr/lib/hadoop- > > > > > mapreduce/xz-1.0.jar:/usr/lib/hadoop-mapreduce/hadoop- > > > > > archives.jar:/usr/lib/hadoop-mapreduce/jackson-core-asl-1. > > > > > 9.13.jar:/usr/lib/hadoop-mapreduce/commons-lang3-3.3.2. > > > > > jar:/usr/lib/hadoop-mapreduce/jackson-core-2.6.6.jar:/usr/ > > > > > lib/hadoop-mapreduce/commons-cli-1.2.jar:/usr/lib/hadoop- > > > > > mapreduce/servlet-api-2.5.jar:/usr/lib/hadoop-mapreduce/ > > > > > jetty-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/apacheds- > > > > > kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/ > > > > > hadoop-aws.jar:/usr/lib/hadoop-mapreduce/hadoop-auth. > > > > > jar:/usr/lib/hadoop-mapreduce/hadoop-rumen-2.7.3-amzn-3.jar: > > > > > /usr/lib/hadoop-mapreduce/commons-collections-3.2.2.jar: > > > > > /usr/lib/hadoop-mapreduce/hadoop-sls-2.7.3-amzn-3.jar:/ > > > > > usr/lib/hadoop-mapreduce/metrics-core-3.0.1.jar:/usr/ > > > > > lib/hadoop-mapreduce/jackson-annotations-2.6.6.jar:/usr/ > > > > > lib/hadoop-mapreduce/junit-4.11.jar:/usr/lib/hadoop- > > mapreduce/commons- > > > > > configuration-1.6.jar:/usr/lib/hadoop-mapreduce/api-util- > > > > > 1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/stax-api-1.0-2.jar:/ > > > > > usr/lib/hadoop-mapreduce/curator-recipes-2.7.1.jar:/ > > > > > usr/lib/hadoop-mapreduce/hamcrest-core-1.3.jar:/usr/ > > > > > lib/hadoop-mapreduce/jackson-databind-2.6.6.jar:/usr/lib/ > > > > > hadoop-mapreduce/jsr305-3.0.0.jar:/usr/lib/hadoop-mapreduce/ > > > > > hadoop-mapreduce-client-app.jar:/usr/lib/hadoop-mapreduce/ > > > > > zookeeper-3.4.10.jar:/usr/lib/hadoop-mapreduce/commons-lang- > > > > > 2.6.jar:/usr/lib/hadoop-mapreduce/jersey-core-1.9.jar: > > > > > /usr/lib/hadoop-mapreduce/curator-framework-2.7.1.jar:/ > > > > > usr/lib/hadoop-mapreduce/jackson-jaxrs-1.9.13.jar:/usr/ > > > > > lib/hadoop-mapreduce/hadoop-gridmix-2.7.3-amzn-3.jar:/usr/ > > > > > lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar:/usr/ > > > > > lib/hadoop-mapreduce/java-xmlbuilder-0.4.jar:/usr/lib/ > > > > > hadoop-mapreduce/commons-logging-1.1.3.jar:/usr/lib/ > > > > > hadoop-mapreduce/lib/protobuf-java-2.5.0.jar:/usr/lib/ > > > > > hadoop-mapreduce/lib/leveldbjni-all-1.8.jar:/usr/ > > > > lib/hadoop-mapreduce/lib/ > > > > > snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/lib/ > > > > > jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/lib/ > > > > > paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/lib/jersey- > > > > > guice-1.9.jar:/usr/lib/hadoop-mapreduce/lib/commons-io-2.4. > > > > > jar:/usr/lib/hadoop-mapreduce/lib/avro-1.7.4.jar:/usr/lib/ > > > > > hadoop-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop- > > > > > mapreduce/lib/commons-compress-1.4.1.jar:/usr/lib/ > > > > > hadoop-mapreduce/lib/javax.inject-1.jar:/usr/lib/hadoop- > > > > > mapreduce/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop- > > > > > mapreduce/lib/aopalliance-1.0.jar:/usr/lib/hadoop-mapreduce/ > > > > > lib/asm-3.2.jar:/usr/lib/hadoop-mapreduce/lib/jackson- > > > > > mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/ > > > > > guice-servlet-3.0.jar:/usr/lib/hadoop-mapreduce/lib/xz-1. > > > > > 0.jar:/usr/lib/hadoop-mapreduce/lib/guice-3.0.jar:/ > > > > > usr/lib/hadoop-mapreduce/lib/jackson-core-asl-1.9.13.jar:/ > > > > > usr/lib/hadoop-mapreduce/lib/junit-4.11.jar:/usr/lib/ > > > > > hadoop-mapreduce/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop- > > > > > mapreduce/lib/jersey-core-1.9.jar:/usr/lib/hadoop-yarn/ > > > > > hadoop-yarn-server-common.jar:/usr/lib/hadoop-yarn/hadoop- > > yarn-server- > > > > > sharedcachemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn- > > > > > server-web-proxy-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/ > > > > > hadoop-yarn-common-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/ > > > > > hadoop-yarn-server-tests-2.7.3-amzn-3.jar:/usr/lib/hadoop- > > > > > yarn/hadoop-yarn-applications-distributedshell.jar:/usr/lib/ > > > > > hadoop-yarn/hadoop-yarn-applications-distributedshell- > > > > > 2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/hadoop-yarn- > > > > > server-resourcemanager.jar:/usr/lib/hadoop-yarn/hadoop- > yarn-server- > > > > > sharedcachemanager-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/ > > > > > hadoop-yarn-server-applicationhistoryservice.jar: > > > > > /usr/lib/hadoop-yarn/hadoop-yarn-server-tests.jar:/usr/ > > > > > lib/hadoop-yarn/hadoop-yarn-server-nodemanager-2.7.3-amzn- > > > > > 3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-client-2.7.3-amzn- > > > > > 3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server- > > > > > applicationhistoryservice-2.7.3-amzn-3.jar:/usr/lib/hadoop- > > > > > yarn/hadoop-yarn-server-common-2.7.3-amzn-3.jar:/usr/ > > > > > lib/hadoop-yarn/hadoop-yarn-registry.jar:/usr/lib/hadoop- > > > > > yarn/hadoop-yarn-api-2.7.3-amzn-3.jar:/usr/lib/hadoop- > > > > > yarn/hadoop-yarn-server-resourcemanager-2.7.3-amzn-3. > > > > > jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications- > > > > > unmanaged-am-launcher.jar:/usr/lib/hadoop-yarn/hadoop- > > > > > yarn-registry-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/ > > > > > hadoop-yarn-client.jar:/usr/lib/hadoop-yarn/hadoop-yarn- > > > > > common.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-web- > > > > > proxy.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server- > > > > > nodemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-api. > > > > > jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications- > > > > > unmanaged-am-launcher-2.7.3-amzn-3.jar:/usr/lib/hadoop- > > > > > yarn/lib/jettison-1.1.jar:/usr/lib/hadoop-yarn/lib/jaxb- > > > > > api-2.2.2.jar:/usr/lib/hadoop-yarn/lib/protobuf-java-2.5.0. > > > > > jar:/usr/lib/hadoop-yarn/lib/leveldbjni-all-1.8.jar:/usr/ > > > > > lib/hadoop-yarn/lib/activation-1.1.jar:/usr/lib/ > > > > > hadoop-yarn/lib/jersey-server-1.9.jar:/usr/lib/hadoop-yarn/ > > > > > lib/guava-11.0.2.jar:/usr/lib/hadoop-yarn/lib/jackson-xc-1. > > > > > 9.13.jar:/usr/lib/hadoop-yarn/lib/commons-codec-1.4.jar:/ > > > > > usr/lib/hadoop-yarn/lib/jersey-guice-1.9.jar:/usr/lib/ > > > > > hadoop-yarn/lib/commons-io-2.4.jar:/usr/lib/hadoop-yarn/ > > > > > lib/jersey-json-1.9.jar:/usr/lib/hadoop-yarn/lib/jersey- > > > > > client-1.9.jar:/usr/lib/hadoop-yarn/lib/log4j-1.2.17. > > > > > jar:/usr/lib/hadoop-yarn/lib/commons-compress-1.4.1.jar:/ > > > > > usr/lib/hadoop-yarn/lib/javax.inject-1.jar:/usr/lib/hadoop- > > > > > yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-yarn/lib/ > > > > > netty-3.6.2.Final.jar:/usr/lib/hadoop-yarn/lib/ > > > > > aopalliance-1.0.jar:/usr/lib/hadoop-yarn/lib/asm-3.2.jar:/ > > > > > usr/lib/hadoop-yarn/lib/zookeeper-3.4.10-tests.jar:/ > > > > > usr/lib/hadoop-yarn/lib/jackson-mapper-asl-1.9.13.jar: > > > > > /usr/lib/hadoop-yarn/lib/guice-servlet-3.0.jar:/usr/ > > > > > lib/hadoop-yarn/lib/jetty-util-6.1.26-emr.jar:/usr/lib/ > > > > > hadoop-yarn/lib/xz-1.0.jar:/usr/lib/hadoop-yarn/lib/guice- > > > > > 3.0.jar:/usr/lib/hadoop-yarn/lib/jackson-core-asl-1.9.13. > > > > > jar:/usr/lib/hadoop-yarn/lib/commons-cli-1.2.jar:/usr/lib/ > > > > > hadoop-yarn/lib/servlet-api-2.5.jar:/usr/lib/hadoop-yarn/ > > > > > lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop-yarn/lib/commons- > > > > > collections-3.2.2.jar:/usr/lib/hadoop-yarn/lib/stax-api- > > > > > 1.0-2.jar:/usr/lib/hadoop-yarn/lib/jsr305-3.0.0.jar:/ > > > > > usr/lib/hadoop-yarn/lib/zookeeper-3.4.10.jar:/usr/lib/ > > > > > hadoop-yarn/lib/commons-lang-2.6.jar:/usr/lib/hadoop-yarn/ > > > > > lib/jersey-core-1.9.jar:/usr/lib/hadoop-yarn/lib/jackson- > > > > > jaxrs-1.9.13.jar:/usr/lib/hadoop-yarn/lib/commons- > > > > > logging-1.1.3.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo.jar: > > > > > /usr/lib/hadoop-lzo/lib/hadoop-lzo-0.4.19.jar:/usr/ > > > > > share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/ > > > > > jmespath-java-1.11.129.jar:/usr/share/aws/emr/emrfs/lib/ > > > > > bcpkix-jdk15on-1.51.jar:/usr/share/aws/emr/emrfs/lib/jcl- > > > > > over-slf4j-1.7.21.jar:/usr/share/aws/emr/emrfs/lib/ion- > > > > > java-1.0.2.jar:/usr/share/aws/emr/emrfs/lib/slf4j-api-1.7. > > > > > 21.jar:/usr/share/aws/emr/emrfs/lib/javax.inject-1.jar:/ > > > > > usr/share/aws/emr/emrfs/lib/aopalliance-1.0.jar:/usr/ > > > > > share/aws/emr/emrfs/lib/bcprov-jdk15on-1.51.jar:/usr/ > > > > > share/aws/emr/emrfs/lib/emrfs-hadoop-assembly-2.18.0.jar:/ > > > > > usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/ > > > > > lib/*:/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar:/usr/ > > > > > share/aws/emr/goodies/lib/emr-hadoop-goodies.jar:/usr/share/ > > > > > aws/emr/kinesis/lib/emr-kinesis-hadoop.jar:/usr/share/ > > > > > aws/emr/cloudwatch-sink/lib/cloudwatch-sink-1.0.0.jar:/ > > > > > usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink.jar:/ > > > > > usr/share/aws/aws-java-sdk/aws-java-sdk-inspector-1.11. > > > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-api- > > > > > gateway-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > > > > > java-sdk-kinesis-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > aws-java-sdk- > > > > > elasticloadbalancingv2-1.11.160.jar:/usr/share/aws/aws- > > > > > java-sdk/aws-java-sdk-rekognition-1.11.160.jar:/usr/ > > > > > share/aws/aws-java-sdk/aws-java-sdk-test-utils-1.11.160. > > > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > elasticbeanstalk-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > > > aws-java-sdk-cloudhsm-1.11.160.jar:/usr/share/aws/aws- > > > > > java-sdk/aws-java-sdk-iam-1.11.160.jar:/usr/share/aws/aws- > > > > > java-sdk/aws-java-sdk-cognitoidp-1.11.160.jar:/usr/ > > > > > share/aws/aws-java-sdk/aws-java-sdk-mechanicalturkrequester-1.11. > > > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > marketplacecommerceanalytics-1.11.160.jar:/usr/share/aws/ > > > > > aws-java-sdk/aws-java-sdk-snowball-1.11.160.jar:/usr/ > > > > > share/aws/aws-java-sdk/aws-java-sdk-polly-1.11.160.jar:/ > > > > > usr/share/aws/aws-java-sdk/jmespath-java-1.11.160.jar:/ > > > > > usr/share/aws/aws-java-sdk/aws-java-sdk-codebuild-1.11. > > > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-emr-1. > > > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > simpleworkflow-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > > > aws-java-sdk-machinelearning-1.11.160.jar:/usr/share/aws/ > > > > > aws-java-sdk/aws-java-sdk-pinpoint-1.11.160.jar:/usr/ > > > > > share/aws/aws-java-sdk/aws-java-sdk-appstream-1.11.160. > > > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sqs-1.11.160. > > > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opensdk-1.11. > > > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > greengrass-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > > > > > java-sdk-ecs-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > > > aws-java-sdk-xray-1.11.160.jar:/usr/share/aws/aws-java- > > > > > sdk/aws-java-sdk-cloudtrail-1.11.160.jar:/usr/share/aws/aws- > > > > > java-sdk/aws-java-sdk-cloudwatchmetrics-1.11.160. > > > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > elasticloadbalancing-1.11.160.jar:/usr/share/aws/aws-java- > > > > > sdk/aws-java-sdk-codepipeline-1.11.160.jar:/usr/share/aws/ > > > > > aws-java-sdk/aws-java-sdk-cloudwatch-1.11.160.jar:/usr/ > > > > > share/aws/aws-java-sdk/aws-java-sdk-shield-1.11.160.jar:/ > > > > > usr/share/aws/aws-java-sdk/aws-java-sdk-config-1.11.160. > > > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > stepfunctions-1.11.160.jar:/ > > > > > usr/share/aws/aws-java-sdk/aws-java-sdk-gamelift-1.11. > > > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-rds-1. > > > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-core-1. > > > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > datapipeline-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > > > aws-java-sdk-s3-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > > > aws-java-sdk-dax-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > > > aws-java-sdk-opsworks-1.11.160.jar:/usr/share/aws/aws- > > > > > java-sdk/aws-java-sdk-servicecatalog-1.11.160.jar:/ > > > > > usr/share/aws/aws-java-sdk/aws-java-sdk-cloudsearch-1.11. > > > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dms-1. > > > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > directory-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > > > > > java-sdk-opsworkscm-1.11.160.jar:/usr/share/aws/aws-java- > > > > sdk/aws-java-sdk- > > > > > cloudformation-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > > > aws-java-sdk-cloudfront-1.11.160.jar:/usr/share/aws/aws- > > > > > java-sdk/aws-java-sdk-budgets-1.11.160.jar:/usr/share/aws/ > > > > > aws-java-sdk/aws-java-sdk-clouddirectory-1.11.160.jar:/ > > > > > usr/share/aws/aws-java-sdk/aws-java-sdk-importexport-1. > > > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lex-1. > > > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > marketplaceentitlement-1.11.160.jar:/usr/share/aws/aws- > > > > > java-sdk/aws-java-sdk-dynamodb-1.11.160.jar:/usr/ > > > > > share/aws/aws-java-sdk/aws-java-sdk-autoscaling-1.11.160. > > > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > elastictranscoder-1.11.160. > > > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > organizations-1.11.160.jar:/ > > > > > usr/share/aws/aws-java-sdk/aws-java-sdk-workspaces-1.11. > > > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ssm-1. > > > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > servermigration-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > > > aws-java-sdk-events-1.11.160.jar:/usr/share/aws/aws-java- > > > > sdk/aws-java-sdk- > > > > > applicationautoscaling-1.11.160.jar:/usr/share/aws/aws- > > > > > java-sdk/aws-java-sdk-health-1.11.160.jar:/usr/share/aws/ > > > > > aws-java-sdk/aws-java-sdk-kms-1.11.160.jar:/usr/share/aws/ > > > > > aws-java-sdk/aws-java-sdk-logs-1.11.160.jar:/usr/share/ > > > > > aws/aws-java-sdk/aws-java-sdk-codestar-1.11.160.jar:/usr/ > > > > > share/aws/aws-java-sdk/aws-java-sdk-route53-1.11.160.jar: > > > > > /usr/share/aws/aws-java-sdk/aws-java-sdk-redshift-1.11. > > > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > marketplacemeteringservice-1.11.160.jar:/usr/share/aws/aws- > > > > > java-sdk/aws-java-sdk-sns-1.11.160.jar:/usr/share/aws/aws- > > > > > java-sdk/aws-java-sdk-batch-1.11.160.jar:/usr/share/aws/aws- > > > > > java-sdk/aws-java-sdk-waf-1.11.160.jar:/usr/share/aws/aws- > > > > > java-sdk/aws-java-sdk-simpledb-1.11.160.jar:/usr/ > > > > > share/aws/aws-java-sdk/aws-java-sdk-codedeploy-1.11.160. > > > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ec2-1.11.160. > > > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-models-1.11. > > > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > devicefarm-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > > > > > java-sdk-cognitoidentity-1.11.160.jar:/usr/share/aws/aws- > > > > > java-sdk/aws-java-sdk-lexmodelbuilding-1.11.160.jar: > > > > > /usr/share/aws/aws-java-sdk/aws-java-sdk-directconnect-1. > > > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > elasticache-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > > > > > java-sdk-costandusagereport-1.11.160.jar:/usr/share/aws/aws- > > > > > java-sdk/aws-java-sdk-discovery-1.11.160.jar:/usr/ > > > > > share/aws/aws-java-sdk/aws-java-sdk-resourcegroupstaggingapi-1.11. > > > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ses-1. > > > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lambda- > > > > > 1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > workdocs-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > > > > > java-sdk-code-generator-1.11.160.jar:/usr/share/aws/aws- > > > > > java-sdk/aws-java-sdk-cognitosync-1.11.160.jar:/usr/ > > > > > share/aws/aws-java-sdk/aws-java-sdk-efs-1.11.160.jar:/ > > > > > usr/share/aws/aws-java-sdk/aws-java-sdk-sts-1.11.160.jar: > > > > > /usr/share/aws/aws-java-sdk/aws-java-sdk-athena-1.11.160. > > > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codecommit-1. > > > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > storagegateway-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > > > aws-java-sdk-lightsail-1.11.160.jar:/usr/share/aws/aws- > > > > > java-sdk/aws-java-sdk-acm-1.11.160.jar:/usr/share/aws/aws- > > > > > java-sdk/aws-java-sdk-1.11.160.jar:/usr/share/aws/aws- > > > > > java-sdk/aws-java-sdk-glacier-1.11.160.jar:/usr/share/aws/ > > > > > aws-java-sdk/aws-java-sdk-ecr-1.11.160.jar:/usr/share/aws/ > > > > > aws-java-sdk/aws-java-sdk-support-1.11.160.jar:/usr/ > > > > > share/aws/aws-java-sdk/aws-java-sdk-codegen-maven-plugin- > > > > > 1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > elasticsearch-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > > > aws-java-sdk-iot-1.11.160.jar > > > > > > > > > > Thank you > > > > > > > > > > On Thu, Oct 5, 2017 at 5:25 AM, Till Rohrmann < > trohrm...@apache.org> > > > > > wrote: > > > > > > > > > > > Hi Andy, > > > > > > > > > > > > the CliFrontend is not executed via Yarn, thus, it is not > affected > > by > > > > > > dependencies which are added due to the underlying Yarn cluster. > > > > > Therefore, > > > > > > it would be helpful to look at the TaskManager logs. Either you > > have > > > > > > enabled log aggregation on your Yarn cluster, then you can obtain > > the > > > > > logs > > > > > > via `yarn logs -applicationId <application ID>` or you have to > > > retrieve > > > > > > them from the machines where they were running (either by going > > > > directly > > > > > > there or via the Yarn web interface). > > > > > > > > > > > > Cheers, > > > > > > Till > > > > > > > > > > > > On Wed, Oct 4, 2017 at 4:27 PM, Andy M. <ajm2...@gmail.com> > wrote: > > > > > > > > > > > > > Hi Till, > > > > > > > > > > > > > > That is actually the classpath used by the flink bash > script(that > > > > > > launches > > > > > > > the jar using the java command). I changed the execute to an > > echo, > > > > and > > > > > > > grabbed that for the CLI arguments. > > > > > > > > > > > > > > I believe this is the class path from the log file(although it > > > might > > > > > not > > > > > > be > > > > > > > the taskmanager log, is that any different from what would be > in > > my > > > > > > > flink-1.3.2/log folder?): > > > > > > > > > > > > > > 2017-10-02 20:03:26,450 INFO org.apache.flink.client. > > CliFrontend > > > > > > > - Classpath: > > > > > > > /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/ > > > > > > > home/hadoop/flink-1.3.2/lib/flink-shaded-hadoop2-uber-1.3. > > > > > > > 2.jar:/home/hadoop/flink-1.3.2/lib/log4j-1.2.17.jar:/home/ > > > > > > > hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7.jar:/home/ > > > > > > > hadoop/flink-1.3.2/lib/flink-dist_2.11-1.3.2.jar::/etc/ > > > hadoop/conf: > > > > > > > > > > > > > > If that doesn't seem right, and you can point me in the right > > > > direction > > > > > > as > > > > > > > to where the TaskManager logs would be, I would be happy to > grab > > > the > > > > > > > information your looking for. > > > > > > > > > > > > > > Thank you > > > > > > > > > > > > > > On Wed, Oct 4, 2017 at 3:27 AM, Till Rohrmann < > > > trohrm...@apache.org> > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Andy, > > > > > > > > > > > > > > > > this looks to me indeed like a dependency problem. I assume > > that > > > > EMR > > > > > or > > > > > > > > something else is pulling in an incompatible version of > Hadoop. > > > > > > > > > > > > > > > > The classpath you've posted, is this the one logged in the > log > > > > files > > > > > > > > (TaskManager log) or did you compile it yourself? In the > latter > > > > case, > > > > > > it > > > > > > > > would also be helpful to get access to the TaskManager logs. > > > > > > > > > > > > > > > > Cheers, > > > > > > > > Till > > > > > > > > > > > > > > > > On Mon, Oct 2, 2017 at 10:20 PM, Andy M. <ajm2...@gmail.com> > > > > wrote: > > > > > > > > > > > > > > > > > Hi Fabian, > > > > > > > > > > > > > > > > > > 1) I have looked at the linked docs, and from what I can > tell > > > no > > > > > > setup > > > > > > > > > should really need to be done to get Flink working(Other > than > > > > > > > downloading > > > > > > > > > the correct binaries, which I believe I did) > > > > > > > > > 2) I have downloaded the Flink 1.3.2 > > binaries(flink-1.3.2-bin- > > > > > > > > > hadoop27-scala_2.11.tgz > > > > > > > > > <http://apache.claz.org/flink/flink-1.3.2/flink-1.3.2-bin- > > > > > > > > > hadoop27-scala_2.11.tgz>) > > > > > > > > > This is for hadoop 2.7.X, which matches EMR 5.8.0. > > > > > > > > > > > > > > > > > > I appreciate any help or guidance you can provide me in > > fixing > > > my > > > > > > > > problems, > > > > > > > > > please let me know if there is anything else I can provide > > you. > > > > > > > > > > > > > > > > > > Thank you > > > > > > > > > > > > > > > > > > On Mon, Oct 2, 2017 at 4:12 PM, Fabian Hueske < > > > fhue...@gmail.com > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi Andy, > > > > > > > > > > > > > > > > > > > > I'm not an AWS expert, so I'll just check on some common > > > > issues. > > > > > > > > > > > > > > > > > > > > I guess you already had a look at the Flink docs for > > AWS/EMR > > > > but > > > > > > I'll > > > > > > > > > post > > > > > > > > > > the link just be to sure [1]. > > > > > > > > > > > > > > > > > > > > Since you are using Flink 1.3.2 (EMR 5.8.0 comes with > Flink > > > > > 1.3.1) > > > > > > > did > > > > > > > > > you > > > > > > > > > > built Flink yourself or did you download the binaries? > > > > > > > > > > Does the Hadoop version of the Flink build match the > Hadoop > > > > > version > > > > > > > of > > > > > > > > > EMR > > > > > > > > > > 5.8.0, i.e., Hadoop 2.7.x? > > > > > > > > > > > > > > > > > > > > Best, Fabian > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > https://ci.apache.org/projects/flink/flink-docs- > > > > > > > > > release-1.3/setup/aws.html > > > > > > > > > > > > > > > > > > > > 2017-10-02 21:51 GMT+02:00 Andy M. <ajm2...@gmail.com>: > > > > > > > > > > > > > > > > > > > > > Hi Fabian, > > > > > > > > > > > > > > > > > > > > > > Sorry, I just realized I forgot to include that part. > > The > > > > > error > > > > > > > > > returned > > > > > > > > > > > is: > > > > > > > > > > > > > > > > > > > > > > java.lang.NoSuchMethodError: > > > > > > > > > > > org.apache.hadoop.conf.Configuration.addResource( > > > > > > > > > > Lorg/apache/hadoop/conf/ > > > > > > > > > > > Configuration;)V > > > > > > > > > > > at com.amazon.ws.emr.hadoop.fs. > > > EmrFileSystem.initialize( > > > > > > > > > > > EmrFileSystem.java:93) > > > > > > > > > > > at org.apache.flink.runtime.fs. > > hdfs.HadoopFileSystem. > > > > > > > > > > > initialize(HadoopFileSystem.java:328) > > > > > > > > > > > at org.apache.flink.core.fs.FileSystem. > > > > > > getUnguardedFileSystem( > > > > > > > > > > > FileSystem.java:350) > > > > > > > > > > > at org.apache.flink.core.fs. > > FileSystem.get(FileSystem. > > > > > > > java:389) > > > > > > > > > > > at org.apache.flink.core.fs.Path. > > > > > > getFileSystem(Path.java:293) > > > > > > > > > > > at org.apache.flink.runtime.state.filesystem. > > > > > > > > > > > FsCheckpointStreamFactory.<init>( > > > FsCheckpointStreamFactory. > > > > > > > java:99) > > > > > > > > > > > at org.apache.flink.runtime.state.filesystem. > > > > > FsStateBackend. > > > > > > > > > > > createStreamFactory(FsStateBackend.java:282) > > > > > > > > > > > at org.apache.flink.contrib.streaming.state. > > > > > > > RocksDBStateBackend. > > > > > > > > > > > createStreamFactory(RocksDBStateBackend.java:273 > > > > > > > > > > > > > > > > > > > > > > I believe it has something to do with the classpath, > but > > I > > > am > > > > > > > unsure > > > > > > > > > why > > > > > > > > > > or > > > > > > > > > > > how to fix it. The classpath being used during the > > > execution > > > > > is: > > > > > > > > > > > /home/hadoop/flink-1.3.2/lib/ > > flink-python_2.11-1.3.2.jar:/ > > > > > > > > > > > home/hadoop/flink-1.3.2/ > lib/flink-shaded-hadoop2- > > > > > > > > > > > uber-1.3.2.jar:/home/hadoop/ > > > flink-1.3.2/lib/log4j-1.2. > > > > > > > > > > > 17.jar:/home/hadoop/flink-1. > > > 3.2/lib/slf4j-log4j12-1.7.7. > > > > > > > > > > > jar:/home/hadoop/flink-1. > > > 3.2/lib/flink-dist_2.11-1.3. > > > > > > > > > > > 2.jar::/etc/hadoop/conf: > > > > > > > > > > > > > > > > > > > > > > I decompiled flink-shaded-hadoop2-uber-1.3.2.jar > and > > > it > > > > > > seems > > > > > > > > the > > > > > > > > > > > addResource function does seem to be there. > > > > > > > > > > > > > > > > > > > > > > Thank you > > > > > > > > > > > > > > > > > > > > > > On Mon, Oct 2, 2017 at 3:43 PM, Fabian Hueske < > > > > > fhue...@gmail.com > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Hi Andy, > > > > > > > > > > > > > > > > > > > > > > > > can you describe in more detail what exactly isn't > > > working? > > > > > > > > > > > > Do you see error messages in the log files or on the > > > > console? > > > > > > > > > > > > > > > > > > > > > > > > Thanks, Fabian > > > > > > > > > > > > > > > > > > > > > > > > 2017-10-02 15:52 GMT+02:00 Andy M. < > ajm2...@gmail.com > > >: > > > > > > > > > > > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > > > > > > > > > > > > > I am about to deploy my first Flink projects to > > > > > production, > > > > > > > but > > > > > > > > I > > > > > > > > > am > > > > > > > > > > > > > running into a very big hurdle. I am unable to > > launch > > > my > > > > > > > project > > > > > > > > > so > > > > > > > > > > it > > > > > > > > > > > > can > > > > > > > > > > > > > write to an S3 bucket. My project is running on an > > EMR > > > > > > > cluster, > > > > > > > > > > where > > > > > > > > > > > I > > > > > > > > > > > > > have installed Flink 1.3.2. I am using Yarn to > > launch > > > > the > > > > > > > > > > application, > > > > > > > > > > > > and > > > > > > > > > > > > > it seems to run fine unless I am trying to enable > > check > > > > > > > > > > pointing(with a > > > > > > > > > > > > S3 > > > > > > > > > > > > > target). I am looking to use RocksDB as my > > > > check-pointing > > > > > > > > backend. > > > > > > > > > > I > > > > > > > > > > > > have > > > > > > > > > > > > > asked a few places, and I am still unable to find a > > > > > solution > > > > > > to > > > > > > > > > this > > > > > > > > > > > > > problem. Here are my steps for creating a cluster, > > and > > > > > > > launching > > > > > > > > > my > > > > > > > > > > > > > application, perhaps I am missing a step. I'd be > > happy > > > > to > > > > > > > > provide > > > > > > > > > > any > > > > > > > > > > > > > additional information if needed. > > > > > > > > > > > > > > > > > > > > > > > > > > AWS Portal: > > > > > > > > > > > > > > > > > > > > > > > > > > 1) EMR -> Create Cluster > > > > > > > > > > > > > 2) Advanced Options > > > > > > > > > > > > > 3) Release = emr-5.8.0 > > > > > > > > > > > > > 4) Only select Hadoop 2.7.3 > > > > > > > > > > > > > 5) Next -> Next -> Next -> Create Cluster ( I > do > > > fill > > > > > out > > > > > > > > > > > > > names/keys/etc) > > > > > > > > > > > > > > > > > > > > > > > > > > Once the cluster is up I ssh into the Master and do > > the > > > > > > > > following: > > > > > > > > > > > > > > > > > > > > > > > > > > 1 wget > > > > > > > > > > > > > http://apache.claz.org/flink/ > > > > flink-1.3.2/flink-1.3.2-bin- > > > > > > > > > > > > > hadoop27-scala_2.11.tgz > > > > > > > > > > > > > 2 tar -xzf flink-1.3.2-bin-hadoop27- > > > scala_2.11.tgz > > > > > > > > > > > > > 3 cd flink-1.3.2 > > > > > > > > > > > > > 4 ./bin/yarn-session.sh -n 2 -tm 5120 -s 4 -d > > > > > > > > > > > > > 5 Change conf/flink-conf.yaml > > > > > > > > > > > > > 6 ./bin/flink run -m yarn-cluster -yn 1 > > > > > > > ~/flink-consumer.jar > > > > > > > > > > > > > > > > > > > > > > > > > > My conf/flink-conf.yaml I add the following fields: > > > > > > > > > > > > > > > > > > > > > > > > > > state.backend: rocksdb > > > > > > > > > > > > > state.backend.fs.checkpointdir: > > > s3:/bucket/location > > > > > > > > > > > > > state.checkpoints.dir: s3:/bucket/location > > > > > > > > > > > > > > > > > > > > > > > > > > My program's checkpointing setup: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > env.enableCheckpointing(getCheckpointRate, > > > > > > > > > CheckpointingMode.EXACTLY_ > > > > > > > > > > > > ONCE) > > > > > > > > > > > > > > > > > > > > > > > > > > env.getCheckpointConfig. > > enableExternalizedCheckpoints( > > > > > > > > > > > > > ExternalizedCheckpointCleanup. > > RETAIN_ON_CANCELLATION) > > > > > > > > > > > > > > > > > > > > > > > > > > env.getCheckpointConfig. > > setMinPauseBetweenCheckpoints( > > > > > > > > > > > > > getCheckpointMinPause) > > > > > > > > > > > > > env.getCheckpointConfig.setCheckpointTimeout( > > > > > > > > > > getCheckpointTimeout) > > > > > > > > > > > > > env.getCheckpointConfig. > > > > setMaxConcurrentCheckpoints(1) > > > > > > > > > > > > > env.setStateBackend(new > > RocksDBStateBackend("s3:// > > > > > > > > > > > bucket/location", > > > > > > > > > > > > > true)) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
