Hi Ajoscha,
That didn't seem to do the trick either. Do the following look correct?
I see 5.9.0 is released with Flink 1.3.2, so I tried that, and got the same
problem, all I did was upload my Scala .jar to the master, updated my
flink-conf.yaml, set my env variables, and ran it with the following
command: HADOOP_CONF_DIR=/etc/hadoop/conf flink run -m yarn-cluster -yn 5
-ys 2 ~/flink-consumer.jar
I am beginning to think something else may be wrong with my configuration.
Does the following look correct?
In flink-conf.yaml:
state.backend: rocksdb
state.backend.fs.checkpointdir: s3://org/flink-project/state
state.checkpoints.dir: s3://org/flink-project/state
In the code:
env.enableCheckpointing(getCheckpointRate,CheckpointingMode.EXACTLY_ONCE)
env.getCheckpointConfig.enableExternalizedCheckpoints(ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION)
env.getCheckpointConfig.setMinPauseBetweenCheckpoints(1000)
env.getCheckpointConfig.setCheckpointTimeout(5000)
env.getCheckpointConfig.setMaxConcurrentCheckpoints(1)
env.setStateBackend(new RocksDBStateBackend("s3://org/flink-project/state",
true))
Thank you
On Wed, Oct 11, 2017 at 5:19 AM, Aljoscha Krettek <aljos...@apache.org>
wrote:
> Hi Andy,
>
> I remember that I was testing a job with almost exactly the same setup as
> part of the Flink 1.3.2 release testing. The command I used to start my job
> is roughly this:
>
> HADOOP_CONF_DIR=/etc/hadoop/conf bin/flink run -c my.main.Class -m
> yarn-cluster -yn 5 -ys 2 ...
>
> i.e. I export the proper hadoop config dir and I run a per-job YARN
> cluster. I think I also exported the result of "hadoop classpath" as
> HADOOP_CLASSPATH.
>
> Best,
> Aljoscha
>
> > On 10. Oct 2017, at 16:43, Andy M. <ajm2...@gmail.com> wrote:
> >
> > Hello,
> >
> > Bowen: Unless I am missing something, it says there needs to be no setup
> > on EMR, Each topic says: "You don’t have to configure this manually if
> you
> > are running Flink on EMR." S3 access from CLI works fine on my clusters.
> >
> > Chen: Thank you for this, I will look into this if I am unable to get
> this
> > running on YARN successfully.
> >
> > Stephan: Removing the said library causes the flink
> > (flink-1.3.2/bin/flink) bash script to fail. The underlying Java needs
> > this to work. I tried explicitly setting the classpath for the java call
> > as well to point to the hadoop library jars. This is the original java
> > command that I was trying to run:
> >
> > java
> > -Dlog.file=/home/hadoop/flink-1.3.2/log/flink-hadoop-client-
> ip-172-31-19-27.log
> > -Dlog4j.configuration=file:/home/hadoop/flink-1.3.2/conf/
> log4j-cli.properties
> > -Dlogback.configurationFile=file:/home/hadoop/flink-1.3.2/
> conf/logback.xml
> > -classpath
> > /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/
> home/hadoop/flink-1.3.2/lib/flink-shaded-hadoop2-uber-1.3.
> 2.jar:/home/hadoop/flink-1.3.2/lib/log4j-1.2.17.jar:/home/
> hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7.jar:/home/
> hadoop/flink-1.3.2/lib/flink-dist_2.11-1.3.2.jar::/etc/hadoop/conf:
> > org.apache.flink.client.CliFrontend run -m yarn-cluster -yn 1
> > /home/hadoop/flink-consumer.jar
> >
> >
> > This is what I changed it too(removing the shadded-hadoop2-uber jar and
> > adding in the hadoop folder):
> >
> > java
> > -Dlog.file=/home/hadoop/flink-1.3.2/log/flink-hadoop-client-
> ip-172-31-19-27.log
> > -Dlog4j.configuration=file:/home/hadoop/flink-1.3.2/conf/
> log4j-cli.properties
> > -Dlogback.configurationFile=file:/home/hadoop/flink-1.3.2/
> conf/logback.xml
> > -classpath
> > /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/
> home/hadoop/flink-1.3.2/lib/log4j-1.2.17.jar:/home/hadoop/
> flink-1.3.2/lib/slf4j-log4j12-1.7.7.jar:/home/hadoop/flink-
> 1.3.2/lib/flink-dist_2.11-1.3.2.jar:/usr/lib/hadoop/lib/
> activation-1.1.jar:/usr/lib/hadoop/lib/commons-io-2.4.jar:
> /usr/lib/hadoop/lib/jackson-mapper-asl-1.9.13.jar:/usr/
> lib/hadoop/lib/log4j-1.2.17.jar:/usr/lib/hadoop/lib/
> apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop/lib/commons-
> lang-2.6.jar:/usr/lib/hadoop/lib/jackson-xc-1.9.13.jar:/
> usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/
> lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop/
> lib/commons-logging-1.1.3.jar:/usr/lib/hadoop/lib/java-
> xmlbuilder-0.4.jar:/usr/lib/hadoop/lib/netty-3.6.2.Final.
> jar:/usr/lib/hadoop/lib/api-asn1-api-1.0.0-M20.jar:/usr/
> lib/hadoop/lib/commons-math3-3.1.1.jar:/usr/lib/hadoop/lib/
> jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/
> usr/lib/hadoop/lib/api-util-1.0.0-M20.jar:/usr/lib/hadoop/
> lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.
> 3-1.jar:/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar:/usr/
> lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/curator-
> client-2.7.1.jar:/usr/lib/hadoop/lib/jersey-core-1.9.
> jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/lib/
> hadoop/lib/avro-1.7.4.jar:/usr/lib/hadoop/lib/curator-
> framework-2.7.1.jar:/usr/lib/hadoop/lib/jersey-json-1.9.
> jar:/usr/lib/hadoop/lib/slf4j-api-1.7.10.jar:/usr/lib/
> hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/
> curator-recipes-2.7.1.jar:/usr/lib/hadoop/lib/jersey-
> server-1.9.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.
> 10.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.
> jar:/usr/lib/hadoop/lib/gson-2.2.4.jar:/usr/lib/hadoop/lib/
> jets3t-0.9.0.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4.
> 1.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/
> hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/jettison-1.
> 1.jar:/usr/lib/hadoop/lib/stax-api-1.0-2.jar:/usr/lib/
> hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/
> hamcrest-core-1.3.jar:/usr/lib/hadoop/lib/jetty-6.1.26-
> emr.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/commons-
> collections-3.2.2.jar:/usr/lib/hadoop/lib/htrace-core-3.
> 1.0-incubating.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26-
> emr.jar:/usr/lib/hadoop/lib/xz-1.0.jar:/usr/lib/hadoop/
> lib/commons-compress-1.4.1.jar:/usr/lib/hadoop/lib/
> httpclient-4.5.3.jar:/usr/lib/hadoop/lib/jsch-0.1.42.jar:/
> usr/lib/hadoop/lib/zookeeper-3.4.10.jar:/usr/lib/hadoop/
> lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/
> httpcore-4.4.4.jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/
> usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/
> hadoop/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop/lib/
> jsr305-3.0.0.jar:/usr/lib/hadoop/lib/commons-httpclient-
> 3.1.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/
> lib/hadoop/lib/junit-4.11.jar:/etc/hadoop/conf
> > org.apache.flink.client.CliFrontend run -m yarn-cluster -yn 1
> > /home/hadoop/flink-consumer.jar
> >
> > The later throws the following error:
> >
> > SLF4J: Class path contains multiple SLF4J bindings.
> > SLF4J: Found binding in
> > [jar:file:/home/hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7.
> jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > SLF4J: Found binding in
> > [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/
> slf4j/impl/StaticLoggerBinder.class]
> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> > explanation.
> > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> > Exception in thread "main" java.lang.NoClassDefFoundError:
> > org/apache/hadoop/util/VersionInfo
> > at
> > org.apache.flink.runtime.util.EnvironmentInformation.logEnvironmentInfo(
> EnvironmentInformation.java:283)
> > at org.apache.flink.client.CliFrontend.main(CliFrontend.
> java:1124)
> > Caused by: java.lang.ClassNotFoundException:
> > org.apache.hadoop.util.VersionInfo
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > ... 2 more
> >
> > Simply removing the .jar from the folder causes the same error.
> >
> > Thank you
> >
> >
> >
> >
> > On Mon, Oct 9, 2017 at 5:46 AM, Stephan Ewen <se...@apache.org> wrote:
> >
> >> Hi!
> >>
> >> It looks like multiple Hadoop versions are in the classpath. Flink's
> hadoop
> >> jar and the EMR Hadoop jars.
> >> I would simply drop Flink's own Hadoop dependency and only use the EMR
> >> Hadoop jars.
> >>
> >> Delete the 'flink-shaded-hadoop2-uber' jar from Flink, and make sure
> the
> >> setup is such that the Hadoop lib environment variable is set. Then it
> >> should not have conflicts any more.
> >>
> >>
> >>
> >> On Sun, Oct 8, 2017 at 12:08 AM, Chen Qin <qinnc...@gmail.com> wrote:
> >>
> >>> Attached my side project verified working to deploy jobmanager and
> >>> taskmanager as stateless service(non yarn/mesos), configuration here
> >>>
> >>> https://github.com/chenqin/flink-jar/tree/master/config/hadoop
> >>>
> >>> more detail here
> >>> https://github.com/chenqin/flink-jar/blob/master/src/
> >>> main/java/FlinkBootstrap.java#L49
> >>>
> >>> On Fri, Oct 6, 2017 at 10:26 PM, Bowen Li <bowen...@offerupnow.com>
> >> wrote:
> >>>
> >>>> Hi Andy,
> >>>>
> >>>> I believe it's because you didn't set your s3 impl correctly. Try to
> >> set
> >>>> your core-site.xml by following https://ci.apache.org/
> >>>> projects/flink/flink-docs-release-1.4/ops/deployment/
> >>>> aws.html#s3afilesystem-
> >>>> recommended
> >>>>
> >>>> Bowen
> >>>>
> >>>> On Fri, Oct 6, 2017 at 7:59 AM, Andy M. <ajm2...@gmail.com> wrote:
> >>>>
> >>>>> Hi Till,
> >>>>>
> >>>>> Seems like everything is in line there. hadoop-common.jar ->
> >>>>> hadoop-common-2.7.3-amzn-3.jar
> >>>>>
> >>>>> And when i decompiled that jar I see public void
> >>>> addResource(Configuration
> >>>>> conf) in org/apache/hadoop/conf/Configuration.java
> >>>>>
> >>>>> I agree that an incorrect version of the jar is probably being run,
> >> is
> >>>>> there a way to limit the classpath for the TaskManager when starting
> >>> the
> >>>>> job?
> >>>>>
> >>>>> Thank you
> >>>>>
> >>>>> On Fri, Oct 6, 2017 at 6:49 AM, Till Rohrmann <trohrm...@apache.org>
> >>>>> wrote:
> >>>>>
> >>>>>> Hi Andy,
> >>>>>>
> >>>>>> could you check which Hadoop version this jar
> >>>>>> /usr/lib/hadoop/hadoop-common.jar is? Maybe also checking whether
> >>> the
> >>>>>> contained hadoop Configuration class has the method
> >>>>>> Configuration.addResource(Lorg/apache/hadoop/conf/
> >> Configuration;)V.
> >>>>> Maybe
> >>>>>> this jar is the culprit because it comes from a different Hadoop
> >>>> version.
> >>>>>>
> >>>>>> Cheers,
> >>>>>> Till
> >>>>>>
> >>>>>>
> >>>>>> On Thu, Oct 5, 2017 at 4:22 PM, Andy M. <ajm2...@gmail.com> wrote:
> >>>>>>
> >>>>>>> Hi Till,
> >>>>>>>
> >>>>>>> I believe this is what you are looking for, classpath is much
> >>> bigger
> >>>>> for
> >>>>>>> the task manager. I can also post the whole log file if needed:
> >>>>>>>
> >>>>>>> 2017-10-05 14:17:53,038 INFO org.apache.flink.yarn.
> >>>>>> YarnTaskManagerRunner
> >>>>>>> - Classpath:
> >>>>>>> flink-consumer.jar:lib/flink-dist_2.11-1.3.2.jar:lib/flink-
> >>>>>>> python_2.11-1.3.2.jar:lib/flink-shaded-hadoop2-uber-1.3.
> >>>>>>> 2.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar:
> >>>>>>> log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/
> >>>>>>> etc/hadoop/conf:/usr/lib/hadoop/hadoop-common-2.7.3-
> >>>>>>> amzn-3-tests.jar:/usr/lib/hadoop/hadoop-annotations-2.7.
> >>>>>>> 3-amzn-3.jar:/usr/lib/hadoop/hadoop-distcp.jar:/usr/lib/
> >>>>>>> hadoop/hadoop-auth-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-
> >>>>>>> nfs-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-streaming-2.7.3-
> >>>>>>> amzn-3.jar:/usr/lib/hadoop/hadoop-ant-2.7.3-amzn-3.jar:/
> >>>>>>> usr/lib/hadoop/hadoop-distcp-2.7.3-amzn-3.jar:/usr/lib/
> >>>>>>> hadoop/hadoop-datajoin.jar:/usr/lib/hadoop/hadoop-
> >>>>>>> streaming.jar:/usr/lib/hadoop/hadoop-common.jar:/usr/lib/
> >>>>>>> hadoop/hadoop-ant.jar:/usr/lib/hadoop/hadoop-sls.jar:/
> >>>>>>> usr/lib/hadoop/hadoop-azure-2.7.3-amzn-3.jar:/usr/lib/
> >>>>>>> hadoop/hadoop-nfs.jar:/usr/lib/hadoop/hadoop-extras-2.7.
> >>>>>>> 3-amzn-3.jar:/usr/lib/hadoop/hadoop-gridmix.jar:/usr/lib/
> >>>>>>> hadoop/hadoop-common-2.7.3-amzn-3.jar:/usr/lib/hadoop/
> >>>>>>> hadoop-annotations.jar:/usr/lib/hadoop/hadoop-openstack-2.
> >>>>>>> 7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-archives-2.7.3-
> >>>>>>> amzn-3.jar:/usr/lib/hadoop/hadoop-azure.jar:/usr/lib/
> >>>>>>> hadoop/hadoop-extras.jar:/usr/lib/hadoop/hadoop-openstack.
> >>>>>>> jar:/usr/lib/hadoop/hadoop-rumen.jar:/usr/lib/hadoop/
> >>>>>>> hadoop-aws-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-
> >>>>>>> datajoin-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-
> >>>>>>> archives.jar:/usr/lib/hadoop/hadoop-aws.jar:/usr/lib/
> >>>>>>> hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-rumen-2.7.3-
> >>>>>>> amzn-3.jar:/usr/lib/hadoop/hadoop-sls-2.7.3-amzn-3.jar:/
> >>>>>>> usr/lib/hadoop/hadoop-gridmix-2.7.3-amzn-3.jar:/usr/lib/
> >>>>>>> hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/jaxb-api-2.
> >>>>>>> 2.2.jar:/usr/lib/hadoop/lib/htrace-core-3.1.0-incubating.
> >>>>>>> jar:/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar:/usr/
> >>>>>>> lib/hadoop/lib/httpclient-4.5.3.jar:/usr/lib/hadoop/lib/
> >>>>>>> httpcore-4.4.4.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4.
> >>>>>>> 1.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.
> >>>>>>> jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/
> >>>>>>> lib/activation-1.1.jar:/usr/lib/hadoop/lib/jersey-server-
> >>>>>>> 1.9.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/
> >>>>>>> usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/
> >>>>>>> gson-2.2.4.jar:/usr/lib/hadoop/lib/commons-digester-1.
> >>>>>>> 8.jar:/usr/lib/hadoop/lib/jackson-xc-1.9.13.jar:/usr/
> >>>>>>> lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/
> >>>>>>> apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop/lib/commons-
> >>>>>>> httpclient-3.1.jar:/usr/lib/hadoop/lib/curator-client-2.7.
> >>>>>>> 1.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/
> >>>>>>> lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/
> >>>>>>> commons-net-3.1.jar:/usr/lib/hadoop/lib/jets3t-0.9.0.jar:/
> >>>>>>> usr/lib/hadoop/lib/commons-io-2.4.jar:/usr/lib/hadoop/lib/
> >>>>>>> xmlenc-0.52.jar:/usr/lib/hadoop/lib/jersey-json-1.9.
> >>>>>>> jar:/usr/lib/hadoop/lib/avro-1.7.4.jar:/usr/lib/hadoop/lib/
> >>>>>>> commons-math3-3.1.1.jar:/usr/lib/hadoop/lib/log4j-1.2.17.
> >>>>>>> jar:/usr/lib/hadoop/lib/commons-compress-1.4.1.jar:/
> >>>>>>> usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/
> >>>>>>> jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/netty-3.6.2.
> >>>>>>> Final.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/
> >>>>>>> lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/api-asn1-
> >>>>>>> api-1.0.0-M20.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-
> >>>>>>> 1.9.13.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26-emr.jar:
> >>>>>>> /usr/lib/hadoop/lib/xz-1.0.jar:/usr/lib/hadoop/lib/
> >>>>>>> jackson-core-asl-1.9.13.jar:/usr/lib/hadoop/lib/commons-
> >>>>>>> cli-1.2.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/
> >>>>>>> lib/hadoop/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop/lib/
> >>>>>>> apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop/lib/
> >>>>>>> commons-collections-3.2.2.jar:/usr/lib/hadoop/lib/junit-4.
> >>>>>>> 11.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:
> >>>>>>> /usr/lib/hadoop/lib/api-util-1.0.0-M20.jar:/usr/lib/hadoop/
> >>>>>>> lib/stax-api-1.0-2.jar:/usr/lib/hadoop/lib/slf4j-api-1.7.
> >>>>>>> 10.jar:/usr/lib/hadoop/lib/curator-recipes-2.7.1.jar:/
> >>>>>>> usr/lib/hadoop/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop/
> >>>>>>> lib/jsr305-3.0.0.jar:/usr/lib/hadoop/lib/zookeeper-3.4.10.
> >>>>>>> jar:/usr/lib/hadoop/lib/commons-lang-2.6.jar:/usr/lib/
> >>>>>>> hadoop/lib/jersey-core-1.9.jar:/usr/lib/hadoop/lib/
> >>>>>>> curator-framework-2.7.1.jar:/usr/lib/hadoop/lib/jackson-
> >>>>>>> jaxrs-1.9.13.jar:/usr/lib/hadoop/lib/java-xmlbuilder-0.
> >>>>>>> 4.jar:/usr/lib/hadoop/lib/commons-logging-1.1.3.jar:/
> >>>>>>> usr/lib/hadoop-hdfs/hadoop-hdfs-nfs-2.7.3-amzn-3.jar:/
> >>>>>>> usr/lib/hadoop-hdfs/hadoop-hdfs-2.7.3-amzn-3-tests.jar:/
> >>>>>>> usr/lib/hadoop-hdfs/hadoop-hdfs.jar:/usr/lib/hadoop-hdfs/
> >>>>>>> hadoop-hdfs-nfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.7.3-
> >>>>>>> amzn-3.jar:/usr/lib/hadoop-hdfs/lib/htrace-core-3.1.0-
> >>>>>>> incubating.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java-
> >>>>>>> 2.5.0.jar:/usr/lib/hadoop-hdfs/lib/leveldbjni-all-1.8.
> >>>>>>> jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1.9.jar:/usr/
> >>>>>>> lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-hdfs/
> >>>>>>> lib/commons-codec-1.4.jar:/usr/lib/hadoop-hdfs/lib/
> >>>>>>> commons-io-2.4.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.52.
> >>>>>>> jar:/usr/lib/hadoop-hdfs/lib/netty-all-4.0.23.Final.jar:/
> >>>>>>> usr/lib/hadoop-hdfs/lib/xercesImpl-2.9.1.jar:/usr/lib/
> >>>>>>> hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/
> >>>>>>> netty-3.6.2.Final.jar:/usr/lib/hadoop-hdfs/lib/xml-apis-
> >>>>>>> 1.3.04.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/
> >>>>>>> hadoop-hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/
> >>>>>>> lib/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/usr/lib/
> >>>>>>> hadoop-hdfs/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop-
> >>>>>>> hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/
> >>>>>>> lib/commons-cli-1.2.jar:/usr/lib/hadoop-hdfs/lib/servlet-
> >>>>>>> api-2.5.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26-emr.jar:
> >>>>>>> /usr/lib/hadoop-hdfs/lib/jsr305-3.0.0.jar:/usr/lib/
> >>>>>>> hadoop-hdfs/lib/commons-lang-2.6.jar:/usr/lib/hadoop-hdfs/
> >>>>>>> lib/jersey-core-1.9.jar:/usr/lib/hadoop-hdfs/lib/commons-
> >>>>>>> logging-1.1.3.jar:/usr/lib/hadoop-mapreduce/jettison-1.1.
> >>>>>>> jar:/usr/lib/hadoop-mapreduce/jaxb-api-2.2.2.jar:/usr/lib/
> >>>>>>> hadoop-mapreduce/hadoop-distcp.jar:/usr/lib/hadoop-
> >>>>>>> mapreduce/htrace-core-3.1.0-incubating.jar:/usr/lib/
> >>>>>>> hadoop-mapreduce/hadoop-auth-2.7.3-amzn-3.jar:/usr/lib/
> >>>>>>> hadoop-mapreduce/protobuf-java-2.5.0.jar:/usr/lib/
> >>>>>> hadoop-mapreduce/hadoop-
> >>>>>>> mapreduce-client-core.jar:/usr/lib/hadoop-mapreduce/
> >>>>>>> httpclient-4.5.3.jar:/usr/lib/hadoop-mapreduce/hadoop-
> >>>>>>> mapreduce-client-shuffle.jar:/usr/lib/hadoop-mapreduce/
> >>>>>>> hadoop-mapreduce-client-common-2.7.3-amzn-3.jar:/usr/
> >>>>>>> lib/hadoop-mapreduce/httpcore-4.4.4.jar:/usr/lib/hadoop-
> >>>>>>> mapreduce/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/
> >>>>>>> commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-mapreduce/
> >>>>>>> hadoop-mapreduce-client-core-2.7.3-amzn-3.jar:/usr/lib/
> >>>>>>> hadoop-mapreduce/jsp-api-2.1.jar:/usr/lib/hadoop-mapreduce/
> >>>>>>> jmespath-java-1.11.160.jar:/usr/lib/hadoop-mapreduce/
> >>>>>>> hadoop-mapreduce-client-hs-plugins.jar:/usr/lib/hadoop-
> >>>>>>> mapreduce/activation-1.1.jar:/usr/lib/hadoop-mapreduce/
> >>>>>>> hadoop-mapreduce-client-jobclient-2.7.3-amzn-3-tests.
> >>>>>>> jar:/usr/lib/hadoop-mapreduce/jersey-server-1.9.jar:/usr/
> >>>>>>> lib/hadoop-mapreduce/hadoop-streaming-2.7.3-amzn-3.jar:/
> >>>>>>> usr/lib/hadoop-mapreduce/commons-beanutils-1.7.0.jar:/
> >>>>>>> usr/lib/hadoop-mapreduce/hadoop-ant-2.7.3-amzn-3.jar:/
> >>>>>>> usr/lib/hadoop-mapreduce/guava-11.0.2.jar:/usr/lib/
> >>>>>>> hadoop-mapreduce/gson-2.2.4.jar:/usr/lib/hadoop-mapreduce/
> >>>>>>> hadoop-distcp-2.7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/
> >>>>>>> hadoop-mapreduce-client-app-2.7.3-amzn-3.jar:/usr/lib/
> >>>>>>> hadoop-mapreduce/hadoop-mapreduce-examples-2.7.3-amzn-
> >>>>>>> 3.jar:/usr/lib/hadoop-mapreduce/commons-digester-1.
> >>>>>> 8.jar:/usr/lib/hadoop-
> >>>>>>> mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.3-amzn-
> >>>>>>> 3.jar:/usr/lib/hadoop-mapreduce/jackson-xc-1.9.13.
> >>>>>>> jar:/usr/lib/hadoop-mapreduce/paranamer-2.3.jar:/usr/lib/
> >>>>>>> hadoop-mapreduce/apacheds-i18n-2.0.0-M15.jar:/usr/lib/
> >>>>>>> hadoop-mapreduce/commons-httpclient-3.1.jar:/usr/lib/
> >>>>>>> hadoop-mapreduce/curator-client-2.7.1.jar:/usr/lib/
> >>>>>>> hadoop-mapreduce/hadoop-datajoin.jar:/usr/lib/hadoop-
> >>>>>>> mapreduce/hadoop-streaming.jar:/usr/lib/hadoop-mapreduce/
> >>>>>>> commons-codec-1.4.jar:/usr/lib/hadoop-mapreduce/commons-
> >>>>>>> net-3.1.jar:/usr/lib/hadoop-mapreduce/jets3t-0.9.0.jar:/
> >>>>>>> usr/lib/hadoop-mapreduce/commons-io-2.4.jar:/usr/lib/
> >>>>>>> hadoop-mapreduce/hadoop-ant.jar:/usr/lib/hadoop-mapreduce/
> >>>>>>> xmlenc-0.52.jar:/usr/lib/hadoop-mapreduce/jersey-json-
> >>>>>>> 1.9.jar:/usr/lib/hadoop-mapreduce/hadoop-sls.jar:/usr/
> >>>>>>> lib/hadoop-mapreduce/avro-1.7.4.jar:/usr/lib/hadoop-
> >>>>>>> mapreduce/aws-java-sdk-core-1.11.160.jar:/usr/lib/hadoop-
> >>>>>>> mapreduce/aws-java-sdk-s3-1.11.160.jar:/usr/lib/hadoop-
> >>>>>>> mapreduce/jackson-dataformat-cbor-2.6.6.jar:/usr/lib/
> >>>>>>> hadoop-mapreduce/hadoop-azure-2.7.3-amzn-3.jar:/usr/lib/
> >>>>>>> hadoop-mapreduce/commons-math3-3.1.1.jar:/usr/lib/
> >>>>>>> hadoop-mapreduce/ion-java-1.0.2.jar:/usr/lib/hadoop-
> >>>>>>> mapreduce/azure-storage-2.0.0.jar:/usr/lib/hadoop-mapreduce/
> >>>>>>> hadoop-mapreduce-client-shuffle-2.7.3-amzn-3.jar:/usr/
> >>>>>>> lib/hadoop-mapreduce/log4j-1.2.17.jar:/usr/lib/hadoop-
> >>>>>>> mapreduce/commons-compress-1.4.1.jar:/usr/lib/hadoop-
> >>>>>>> mapreduce/hadoop-extras-2.7.3-amzn-3.jar:/usr/lib/hadoop-
> >>>>>>> mapreduce/hadoop-gridmix.jar:/usr/lib/hadoop-mapreduce/jsch-
> >>>>>>> 0.1.42.jar:/usr/lib/hadoop-mapreduce/joda-time-2.8.1.jar:
> >>>>>>> /usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-2.
> >>>>>>> 7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/jaxb-impl-2.
> >>>>>>> 2.3-1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-
> >>>>>>> client-hs.jar:/usr/lib/hadoop-mapreduce/netty-3.6.2.Final.
> >>>>>>> jar:/usr/lib/hadoop-mapreduce/mockito-all-1.8.5.jar:/usr/
> >>>>>>> lib/hadoop-mapreduce/hadoop-openstack-2.7.3-amzn-3.jar:/
> >>>>>>> usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-
> >>>>>>> common.jar:/usr/lib/hadoop-mapreduce/hadoop-archives-2.7.
> >>>>>>> 3-amzn-3.jar:/usr/lib/hadoop-mapreduce/hadoop-azure.jar:/
> >>>>>>> usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-
> >>>>>>> jobclient.jar:/usr/lib/hadoop-mapreduce/hadoop-extras.jar:/
> >>>>>>> usr/lib/hadoop-mapreduce/asm-3.2.jar:/usr/lib/hadoop-
> >>>>>>> mapreduce/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop-
> >>>>>>> mapreduce/hadoop-openstack.jar:/usr/lib/hadoop-mapreduce/
> >>>>>>> hadoop-mapreduce-client-jobclient-2.7.3-amzn-3.jar:/
> >>>>>>> usr/lib/hadoop-mapreduce/hadoop-rumen.jar:/usr/lib/
> >>>>>>> hadoop-mapreduce/aws-java-sdk-kms-1.11.160.jar:/usr/lib/
> >>>>>>> hadoop-mapreduce/jackson-mapper-asl-1.9.13.jar:/usr/
> >>>>>>> lib/hadoop-mapreduce/hadoop-aws-2.7.3-amzn-3.jar:/usr/lib/
> >>>>>>> hadoop-mapreduce/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop-
> >>>>>>> mapreduce/hadoop-datajoin-2.7.3-amzn-3.jar:/usr/lib/hadoop-
> >>>>>>> mapreduce/xz-1.0.jar:/usr/lib/hadoop-mapreduce/hadoop-
> >>>>>>> archives.jar:/usr/lib/hadoop-mapreduce/jackson-core-asl-1.
> >>>>>>> 9.13.jar:/usr/lib/hadoop-mapreduce/commons-lang3-3.3.2.
> >>>>>>> jar:/usr/lib/hadoop-mapreduce/jackson-core-2.6.6.jar:/usr/
> >>>>>>> lib/hadoop-mapreduce/commons-cli-1.2.jar:/usr/lib/hadoop-
> >>>>>>> mapreduce/servlet-api-2.5.jar:/usr/lib/hadoop-mapreduce/
> >>>>>>> jetty-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/apacheds-
> >>>>>>> kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/
> >>>>>>> hadoop-aws.jar:/usr/lib/hadoop-mapreduce/hadoop-auth.
> >>>>>>> jar:/usr/lib/hadoop-mapreduce/hadoop-rumen-2.7.3-amzn-3.jar:
> >>>>>>> /usr/lib/hadoop-mapreduce/commons-collections-3.2.2.jar:
> >>>>>>> /usr/lib/hadoop-mapreduce/hadoop-sls-2.7.3-amzn-3.jar:/
> >>>>>>> usr/lib/hadoop-mapreduce/metrics-core-3.0.1.jar:/usr/
> >>>>>>> lib/hadoop-mapreduce/jackson-annotations-2.6.6.jar:/usr/
> >>>>>>> lib/hadoop-mapreduce/junit-4.11.jar:/usr/lib/hadoop-
> >>>> mapreduce/commons-
> >>>>>>> configuration-1.6.jar:/usr/lib/hadoop-mapreduce/api-util-
> >>>>>>> 1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/stax-api-1.0-2.jar:/
> >>>>>>> usr/lib/hadoop-mapreduce/curator-recipes-2.7.1.jar:/
> >>>>>>> usr/lib/hadoop-mapreduce/hamcrest-core-1.3.jar:/usr/
> >>>>>>> lib/hadoop-mapreduce/jackson-databind-2.6.6.jar:/usr/lib/
> >>>>>>> hadoop-mapreduce/jsr305-3.0.0.jar:/usr/lib/hadoop-mapreduce/
> >>>>>>> hadoop-mapreduce-client-app.jar:/usr/lib/hadoop-mapreduce/
> >>>>>>> zookeeper-3.4.10.jar:/usr/lib/hadoop-mapreduce/commons-lang-
> >>>>>>> 2.6.jar:/usr/lib/hadoop-mapreduce/jersey-core-1.9.jar:
> >>>>>>> /usr/lib/hadoop-mapreduce/curator-framework-2.7.1.jar:/
> >>>>>>> usr/lib/hadoop-mapreduce/jackson-jaxrs-1.9.13.jar:/usr/
> >>>>>>> lib/hadoop-mapreduce/hadoop-gridmix-2.7.3-amzn-3.jar:/usr/
> >>>>>>> lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar:/usr/
> >>>>>>> lib/hadoop-mapreduce/java-xmlbuilder-0.4.jar:/usr/lib/
> >>>>>>> hadoop-mapreduce/commons-logging-1.1.3.jar:/usr/lib/
> >>>>>>> hadoop-mapreduce/lib/protobuf-java-2.5.0.jar:/usr/lib/
> >>>>>>> hadoop-mapreduce/lib/leveldbjni-all-1.8.jar:/usr/
> >>>>>> lib/hadoop-mapreduce/lib/
> >>>>>>> snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/lib/
> >>>>>>> jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/lib/
> >>>>>>> paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/lib/jersey-
> >>>>>>> guice-1.9.jar:/usr/lib/hadoop-mapreduce/lib/commons-io-2.4.
> >>>>>>> jar:/usr/lib/hadoop-mapreduce/lib/avro-1.7.4.jar:/usr/lib/
> >>>>>>> hadoop-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop-
> >>>>>>> mapreduce/lib/commons-compress-1.4.1.jar:/usr/lib/
> >>>>>>> hadoop-mapreduce/lib/javax.inject-1.jar:/usr/lib/hadoop-
> >>>>>>> mapreduce/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-
> >>>>>>> mapreduce/lib/aopalliance-1.0.jar:/usr/lib/hadoop-mapreduce/
> >>>>>>> lib/asm-3.2.jar:/usr/lib/hadoop-mapreduce/lib/jackson-
> >>>>>>> mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/
> >>>>>>> guice-servlet-3.0.jar:/usr/lib/hadoop-mapreduce/lib/xz-1.
> >>>>>>> 0.jar:/usr/lib/hadoop-mapreduce/lib/guice-3.0.jar:/
> >>>>>>> usr/lib/hadoop-mapreduce/lib/jackson-core-asl-1.9.13.jar:/
> >>>>>>> usr/lib/hadoop-mapreduce/lib/junit-4.11.jar:/usr/lib/
> >>>>>>> hadoop-mapreduce/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop-
> >>>>>>> mapreduce/lib/jersey-core-1.9.jar:/usr/lib/hadoop-yarn/
> >>>>>>> hadoop-yarn-server-common.jar:/usr/lib/hadoop-yarn/hadoop-
> >>>> yarn-server-
> >>>>>>> sharedcachemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-
> >>>>>>> server-web-proxy-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/
> >>>>>>> hadoop-yarn-common-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/
> >>>>>>> hadoop-yarn-server-tests-2.7.3-amzn-3.jar:/usr/lib/hadoop-
> >>>>>>> yarn/hadoop-yarn-applications-distributedshell.jar:/usr/lib/
> >>>>>>> hadoop-yarn/hadoop-yarn-applications-distributedshell-
> >>>>>>> 2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-
> >>>>>>> server-resourcemanager.jar:/usr/lib/hadoop-yarn/hadoop-
> >>> yarn-server-
> >>>>>>> sharedcachemanager-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/
> >>>>>>> hadoop-yarn-server-applicationhistoryservice.jar:
> >>>>>>> /usr/lib/hadoop-yarn/hadoop-yarn-server-tests.jar:/usr/
> >>>>>>> lib/hadoop-yarn/hadoop-yarn-server-nodemanager-2.7.3-amzn-
> >>>>>>> 3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-client-2.7.3-amzn-
> >>>>>>> 3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-
> >>>>>>> applicationhistoryservice-2.7.3-amzn-3.jar:/usr/lib/hadoop-
> >>>>>>> yarn/hadoop-yarn-server-common-2.7.3-amzn-3.jar:/usr/
> >>>>>>> lib/hadoop-yarn/hadoop-yarn-registry.jar:/usr/lib/hadoop-
> >>>>>>> yarn/hadoop-yarn-api-2.7.3-amzn-3.jar:/usr/lib/hadoop-
> >>>>>>> yarn/hadoop-yarn-server-resourcemanager-2.7.3-amzn-3.
> >>>>>>> jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-
> >>>>>>> unmanaged-am-launcher.jar:/usr/lib/hadoop-yarn/hadoop-
> >>>>>>> yarn-registry-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/
> >>>>>>> hadoop-yarn-client.jar:/usr/lib/hadoop-yarn/hadoop-yarn-
> >>>>>>> common.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-web-
> >>>>>>> proxy.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-
> >>>>>>> nodemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-api.
> >>>>>>> jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-
> >>>>>>> unmanaged-am-launcher-2.7.3-amzn-3.jar:/usr/lib/hadoop-
> >>>>>>> yarn/lib/jettison-1.1.jar:/usr/lib/hadoop-yarn/lib/jaxb-
> >>>>>>> api-2.2.2.jar:/usr/lib/hadoop-yarn/lib/protobuf-java-2.5.0.
> >>>>>>> jar:/usr/lib/hadoop-yarn/lib/leveldbjni-all-1.8.jar:/usr/
> >>>>>>> lib/hadoop-yarn/lib/activation-1.1.jar:/usr/lib/
> >>>>>>> hadoop-yarn/lib/jersey-server-1.9.jar:/usr/lib/hadoop-yarn/
> >>>>>>> lib/guava-11.0.2.jar:/usr/lib/hadoop-yarn/lib/jackson-xc-1.
> >>>>>>> 9.13.jar:/usr/lib/hadoop-yarn/lib/commons-codec-1.4.jar:/
> >>>>>>> usr/lib/hadoop-yarn/lib/jersey-guice-1.9.jar:/usr/lib/
> >>>>>>> hadoop-yarn/lib/commons-io-2.4.jar:/usr/lib/hadoop-yarn/
> >>>>>>> lib/jersey-json-1.9.jar:/usr/lib/hadoop-yarn/lib/jersey-
> >>>>>>> client-1.9.jar:/usr/lib/hadoop-yarn/lib/log4j-1.2.17.
> >>>>>>> jar:/usr/lib/hadoop-yarn/lib/commons-compress-1.4.1.jar:/
> >>>>>>> usr/lib/hadoop-yarn/lib/javax.inject-1.jar:/usr/lib/hadoop-
> >>>>>>> yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-yarn/lib/
> >>>>>>> netty-3.6.2.Final.jar:/usr/lib/hadoop-yarn/lib/
> >>>>>>> aopalliance-1.0.jar:/usr/lib/hadoop-yarn/lib/asm-3.2.jar:/
> >>>>>>> usr/lib/hadoop-yarn/lib/zookeeper-3.4.10-tests.jar:/
> >>>>>>> usr/lib/hadoop-yarn/lib/jackson-mapper-asl-1.9.13.jar:
> >>>>>>> /usr/lib/hadoop-yarn/lib/guice-servlet-3.0.jar:/usr/
> >>>>>>> lib/hadoop-yarn/lib/jetty-util-6.1.26-emr.jar:/usr/lib/
> >>>>>>> hadoop-yarn/lib/xz-1.0.jar:/usr/lib/hadoop-yarn/lib/guice-
> >>>>>>> 3.0.jar:/usr/lib/hadoop-yarn/lib/jackson-core-asl-1.9.13.
> >>>>>>> jar:/usr/lib/hadoop-yarn/lib/commons-cli-1.2.jar:/usr/lib/
> >>>>>>> hadoop-yarn/lib/servlet-api-2.5.jar:/usr/lib/hadoop-yarn/
> >>>>>>> lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop-yarn/lib/commons-
> >>>>>>> collections-3.2.2.jar:/usr/lib/hadoop-yarn/lib/stax-api-
> >>>>>>> 1.0-2.jar:/usr/lib/hadoop-yarn/lib/jsr305-3.0.0.jar:/
> >>>>>>> usr/lib/hadoop-yarn/lib/zookeeper-3.4.10.jar:/usr/lib/
> >>>>>>> hadoop-yarn/lib/commons-lang-2.6.jar:/usr/lib/hadoop-yarn/
> >>>>>>> lib/jersey-core-1.9.jar:/usr/lib/hadoop-yarn/lib/jackson-
> >>>>>>> jaxrs-1.9.13.jar:/usr/lib/hadoop-yarn/lib/commons-
> >>>>>>> logging-1.1.3.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo.jar:
> >>>>>>> /usr/lib/hadoop-lzo/lib/hadoop-lzo-0.4.19.jar:/usr/
> >>>>>>> share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/
> >>>>>>> jmespath-java-1.11.129.jar:/usr/share/aws/emr/emrfs/lib/
> >>>>>>> bcpkix-jdk15on-1.51.jar:/usr/share/aws/emr/emrfs/lib/jcl-
> >>>>>>> over-slf4j-1.7.21.jar:/usr/share/aws/emr/emrfs/lib/ion-
> >>>>>>> java-1.0.2.jar:/usr/share/aws/emr/emrfs/lib/slf4j-api-1.7.
> >>>>>>> 21.jar:/usr/share/aws/emr/emrfs/lib/javax.inject-1.jar:/
> >>>>>>> usr/share/aws/emr/emrfs/lib/aopalliance-1.0.jar:/usr/
> >>>>>>> share/aws/emr/emrfs/lib/bcprov-jdk15on-1.51.jar:/usr/
> >>>>>>> share/aws/emr/emrfs/lib/emrfs-hadoop-assembly-2.18.0.jar:/
> >>>>>>> usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/
> >>>>>>> lib/*:/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar:/usr/
> >>>>>>> share/aws/emr/goodies/lib/emr-hadoop-goodies.jar:/usr/share/
> >>>>>>> aws/emr/kinesis/lib/emr-kinesis-hadoop.jar:/usr/share/
> >>>>>>> aws/emr/cloudwatch-sink/lib/cloudwatch-sink-1.0.0.jar:/
> >>>>>>> usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink.jar:/
> >>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-inspector-1.11.
> >>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-api-
> >>>>>>> gateway-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-
> >>>>>>> java-sdk-kinesis-1.11.160.jar:/usr/share/aws/aws-java-sdk/
> >>>>> aws-java-sdk-
> >>>>>>> elasticloadbalancingv2-1.11.160.jar:/usr/share/aws/aws-
> >>>>>>> java-sdk/aws-java-sdk-rekognition-1.11.160.jar:/usr/
> >>>>>>> share/aws/aws-java-sdk/aws-java-sdk-test-utils-1.11.160.
> >>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-
> >>>>>>> elasticbeanstalk-1.11.160.jar:/usr/share/aws/aws-java-sdk/
> >>>>>>> aws-java-sdk-cloudhsm-1.11.160.jar:/usr/share/aws/aws-
> >>>>>>> java-sdk/aws-java-sdk-iam-1.11.160.jar:/usr/share/aws/aws-
> >>>>>>> java-sdk/aws-java-sdk-cognitoidp-1.11.160.jar:/usr/
> >>>>>>> share/aws/aws-java-sdk/aws-java-sdk-
> >> mechanicalturkrequester-1.11.
> >>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-
> >>>>>>> marketplacecommerceanalytics-1.11.160.jar:/usr/share/aws/
> >>>>>>> aws-java-sdk/aws-java-sdk-snowball-1.11.160.jar:/usr/
> >>>>>>> share/aws/aws-java-sdk/aws-java-sdk-polly-1.11.160.jar:/
> >>>>>>> usr/share/aws/aws-java-sdk/jmespath-java-1.11.160.jar:/
> >>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-codebuild-1.11.
> >>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-emr-1.
> >>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-
> >>>>>>> simpleworkflow-1.11.160.jar:/usr/share/aws/aws-java-sdk/
> >>>>>>> aws-java-sdk-machinelearning-1.11.160.jar:/usr/share/aws/
> >>>>>>> aws-java-sdk/aws-java-sdk-pinpoint-1.11.160.jar:/usr/
> >>>>>>> share/aws/aws-java-sdk/aws-java-sdk-appstream-1.11.160.
> >>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sqs-1.11.160.
> >>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opensdk-1.11.
> >>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-
> >>>>>>> greengrass-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-
> >>>>>>> java-sdk-ecs-1.11.160.jar:/usr/share/aws/aws-java-sdk/
> >>>>>>> aws-java-sdk-xray-1.11.160.jar:/usr/share/aws/aws-java-
> >>>>>>> sdk/aws-java-sdk-cloudtrail-1.11.160.jar:/usr/share/aws/aws-
> >>>>>>> java-sdk/aws-java-sdk-cloudwatchmetrics-1.11.160.
> >>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-
> >>>>>>> elasticloadbalancing-1.11.160.jar:/usr/share/aws/aws-java-
> >>>>>>> sdk/aws-java-sdk-codepipeline-1.11.160.jar:/usr/share/aws/
> >>>>>>> aws-java-sdk/aws-java-sdk-cloudwatch-1.11.160.jar:/usr/
> >>>>>>> share/aws/aws-java-sdk/aws-java-sdk-shield-1.11.160.jar:/
> >>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-config-1.11.160.
> >>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-
> >>>>>> stepfunctions-1.11.160.jar:/
> >>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-gamelift-1.11.
> >>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-rds-1.
> >>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-core-1.
> >>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-
> >>>>>>> datapipeline-1.11.160.jar:/usr/share/aws/aws-java-sdk/
> >>>>>>> aws-java-sdk-s3-1.11.160.jar:/usr/share/aws/aws-java-sdk/
> >>>>>>> aws-java-sdk-dax-1.11.160.jar:/usr/share/aws/aws-java-sdk/
> >>>>>>> aws-java-sdk-opsworks-1.11.160.jar:/usr/share/aws/aws-
> >>>>>>> java-sdk/aws-java-sdk-servicecatalog-1.11.160.jar:/
> >>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-cloudsearch-1.11.
> >>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dms-1.
> >>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-
> >>>>>>> directory-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-
> >>>>>>> java-sdk-opsworkscm-1.11.160.jar:/usr/share/aws/aws-java-
> >>>>>> sdk/aws-java-sdk-
> >>>>>>> cloudformation-1.11.160.jar:/usr/share/aws/aws-java-sdk/
> >>>>>>> aws-java-sdk-cloudfront-1.11.160.jar:/usr/share/aws/aws-
> >>>>>>> java-sdk/aws-java-sdk-budgets-1.11.160.jar:/usr/share/aws/
> >>>>>>> aws-java-sdk/aws-java-sdk-clouddirectory-1.11.160.jar:/
> >>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-importexport-1.
> >>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lex-1.
> >>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-
> >>>>>>> marketplaceentitlement-1.11.160.jar:/usr/share/aws/aws-
> >>>>>>> java-sdk/aws-java-sdk-dynamodb-1.11.160.jar:/usr/
> >>>>>>> share/aws/aws-java-sdk/aws-java-sdk-autoscaling-1.11.160.
> >>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-
> >>>>> elastictranscoder-1.11.160.
> >>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-
> >>>>>> organizations-1.11.160.jar:/
> >>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-workspaces-1.11.
> >>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ssm-1.
> >>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-
> >>>>>>> servermigration-1.11.160.jar:/usr/share/aws/aws-java-sdk/
> >>>>>>> aws-java-sdk-events-1.11.160.jar:/usr/share/aws/aws-java-
> >>>>>> sdk/aws-java-sdk-
> >>>>>>> applicationautoscaling-1.11.160.jar:/usr/share/aws/aws-
> >>>>>>> java-sdk/aws-java-sdk-health-1.11.160.jar:/usr/share/aws/
> >>>>>>> aws-java-sdk/aws-java-sdk-kms-1.11.160.jar:/usr/share/aws/
> >>>>>>> aws-java-sdk/aws-java-sdk-logs-1.11.160.jar:/usr/share/
> >>>>>>> aws/aws-java-sdk/aws-java-sdk-codestar-1.11.160.jar:/usr/
> >>>>>>> share/aws/aws-java-sdk/aws-java-sdk-route53-1.11.160.jar:
> >>>>>>> /usr/share/aws/aws-java-sdk/aws-java-sdk-redshift-1.11.
> >>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-
> >>>>>>> marketplacemeteringservice-1.11.160.jar:/usr/share/aws/aws-
> >>>>>>> java-sdk/aws-java-sdk-sns-1.11.160.jar:/usr/share/aws/aws-
> >>>>>>> java-sdk/aws-java-sdk-batch-1.11.160.jar:/usr/share/aws/aws-
> >>>>>>> java-sdk/aws-java-sdk-waf-1.11.160.jar:/usr/share/aws/aws-
> >>>>>>> java-sdk/aws-java-sdk-simpledb-1.11.160.jar:/usr/
> >>>>>>> share/aws/aws-java-sdk/aws-java-sdk-codedeploy-1.11.160.
> >>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ec2-1.11.160.
> >>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-models-1.11.
> >>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-
> >>>>>>> devicefarm-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-
> >>>>>>> java-sdk-cognitoidentity-1.11.160.jar:/usr/share/aws/aws-
> >>>>>>> java-sdk/aws-java-sdk-lexmodelbuilding-1.11.160.jar:
> >>>>>>> /usr/share/aws/aws-java-sdk/aws-java-sdk-directconnect-1.
> >>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-
> >>>>>>> elasticache-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-
> >>>>>>> java-sdk-costandusagereport-1.11.160.jar:/usr/share/aws/aws-
> >>>>>>> java-sdk/aws-java-sdk-discovery-1.11.160.jar:/usr/
> >>>>>>> share/aws/aws-java-sdk/aws-java-sdk-
> >> resourcegroupstaggingapi-1.11.
> >>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ses-1.
> >>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lambda-
> >>>>>>> 1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-
> >>>>>>> workdocs-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-
> >>>>>>> java-sdk-code-generator-1.11.160.jar:/usr/share/aws/aws-
> >>>>>>> java-sdk/aws-java-sdk-cognitosync-1.11.160.jar:/usr/
> >>>>>>> share/aws/aws-java-sdk/aws-java-sdk-efs-1.11.160.jar:/
> >>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-sts-1.11.160.jar:
> >>>>>>> /usr/share/aws/aws-java-sdk/aws-java-sdk-athena-1.11.160.
> >>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codecommit-1.
> >>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-
> >>>>>>> storagegateway-1.11.160.jar:/usr/share/aws/aws-java-sdk/
> >>>>>>> aws-java-sdk-lightsail-1.11.160.jar:/usr/share/aws/aws-
> >>>>>>> java-sdk/aws-java-sdk-acm-1.11.160.jar:/usr/share/aws/aws-
> >>>>>>> java-sdk/aws-java-sdk-1.11.160.jar:/usr/share/aws/aws-
> >>>>>>> java-sdk/aws-java-sdk-glacier-1.11.160.jar:/usr/share/aws/
> >>>>>>> aws-java-sdk/aws-java-sdk-ecr-1.11.160.jar:/usr/share/aws/
> >>>>>>> aws-java-sdk/aws-java-sdk-support-1.11.160.jar:/usr/
> >>>>>>> share/aws/aws-java-sdk/aws-java-sdk-codegen-maven-plugin-
> >>>>>>> 1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-
> >>>>>>> elasticsearch-1.11.160.jar:/usr/share/aws/aws-java-sdk/
> >>>>>>> aws-java-sdk-iot-1.11.160.jar
> >>>>>>>
> >>>>>>> Thank you
> >>>>>>>
> >>>>>>> On Thu, Oct 5, 2017 at 5:25 AM, Till Rohrmann <
> >>> trohrm...@apache.org>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Hi Andy,
> >>>>>>>>
> >>>>>>>> the CliFrontend is not executed via Yarn, thus, it is not
> >>> affected
> >>>> by
> >>>>>>>> dependencies which are added due to the underlying Yarn
> >> cluster.
> >>>>>>> Therefore,
> >>>>>>>> it would be helpful to look at the TaskManager logs. Either you
> >>>> have
> >>>>>>>> enabled log aggregation on your Yarn cluster, then you can
> >> obtain
> >>>> the
> >>>>>>> logs
> >>>>>>>> via `yarn logs -applicationId <application ID>` or you have to
> >>>>> retrieve
> >>>>>>>> them from the machines where they were running (either by going
> >>>>>> directly
> >>>>>>>> there or via the Yarn web interface).
> >>>>>>>>
> >>>>>>>> Cheers,
> >>>>>>>> Till
> >>>>>>>>
> >>>>>>>> On Wed, Oct 4, 2017 at 4:27 PM, Andy M. <ajm2...@gmail.com>
> >>> wrote:
> >>>>>>>>
> >>>>>>>>> Hi Till,
> >>>>>>>>>
> >>>>>>>>> That is actually the classpath used by the flink bash
> >>> script(that
> >>>>>>>> launches
> >>>>>>>>> the jar using the java command). I changed the execute to an
> >>>> echo,
> >>>>>> and
> >>>>>>>>> grabbed that for the CLI arguments.
> >>>>>>>>>
> >>>>>>>>> I believe this is the class path from the log file(although
> >> it
> >>>>> might
> >>>>>>> not
> >>>>>>>> be
> >>>>>>>>> the taskmanager log, is that any different from what would be
> >>> in
> >>>> my
> >>>>>>>>> flink-1.3.2/log folder?):
> >>>>>>>>>
> >>>>>>>>> 2017-10-02 20:03:26,450 INFO org.apache.flink.client.
> >>>> CliFrontend
> >>>>>>>>> - Classpath:
> >>>>>>>>> /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/
> >>>>>>>>> home/hadoop/flink-1.3.2/lib/flink-shaded-hadoop2-uber-1.3.
> >>>>>>>>> 2.jar:/home/hadoop/flink-1.3.2/lib/log4j-1.2.17.jar:/home/
> >>>>>>>>> hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7.jar:/home/
> >>>>>>>>> hadoop/flink-1.3.2/lib/flink-dist_2.11-1.3.2.jar::/etc/
> >>>>> hadoop/conf:
> >>>>>>>>>
> >>>>>>>>> If that doesn't seem right, and you can point me in the right
> >>>>>> direction
> >>>>>>>> as
> >>>>>>>>> to where the TaskManager logs would be, I would be happy to
> >>> grab
> >>>>> the
> >>>>>>>>> information your looking for.
> >>>>>>>>>
> >>>>>>>>> Thank you
> >>>>>>>>>
> >>>>>>>>> On Wed, Oct 4, 2017 at 3:27 AM, Till Rohrmann <
> >>>>> trohrm...@apache.org>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hi Andy,
> >>>>>>>>>>
> >>>>>>>>>> this looks to me indeed like a dependency problem. I assume
> >>>> that
> >>>>>> EMR
> >>>>>>> or
> >>>>>>>>>> something else is pulling in an incompatible version of
> >>> Hadoop.
> >>>>>>>>>>
> >>>>>>>>>> The classpath you've posted, is this the one logged in the
> >>> log
> >>>>>> files
> >>>>>>>>>> (TaskManager log) or did you compile it yourself? In the
> >>> latter
> >>>>>> case,
> >>>>>>>> it
> >>>>>>>>>> would also be helpful to get access to the TaskManager
> >> logs.
> >>>>>>>>>>
> >>>>>>>>>> Cheers,
> >>>>>>>>>> Till
> >>>>>>>>>>
> >>>>>>>>>> On Mon, Oct 2, 2017 at 10:20 PM, Andy M. <
> >> ajm2...@gmail.com>
> >>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hi Fabian,
> >>>>>>>>>>>
> >>>>>>>>>>> 1) I have looked at the linked docs, and from what I can
> >>> tell
> >>>>> no
> >>>>>>>> setup
> >>>>>>>>>>> should really need to be done to get Flink working(Other
> >>> than
> >>>>>>>>> downloading
> >>>>>>>>>>> the correct binaries, which I believe I did)
> >>>>>>>>>>> 2) I have downloaded the Flink 1.3.2
> >>>> binaries(flink-1.3.2-bin-
> >>>>>>>>>>> hadoop27-scala_2.11.tgz
> >>>>>>>>>>> <http://apache.claz.org/flink/
> >> flink-1.3.2/flink-1.3.2-bin-
> >>>>>>>>>>> hadoop27-scala_2.11.tgz>)
> >>>>>>>>>>> This is for hadoop 2.7.X, which matches EMR 5.8.0.
> >>>>>>>>>>>
> >>>>>>>>>>> I appreciate any help or guidance you can provide me in
> >>>> fixing
> >>>>> my
> >>>>>>>>>> problems,
> >>>>>>>>>>> please let me know if there is anything else I can
> >> provide
> >>>> you.
> >>>>>>>>>>>
> >>>>>>>>>>> Thank you
> >>>>>>>>>>>
> >>>>>>>>>>> On Mon, Oct 2, 2017 at 4:12 PM, Fabian Hueske <
> >>>>> fhue...@gmail.com
> >>>>>>>
> >>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi Andy,
> >>>>>>>>>>>>
> >>>>>>>>>>>> I'm not an AWS expert, so I'll just check on some
> >> common
> >>>>>> issues.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I guess you already had a look at the Flink docs for
> >>>> AWS/EMR
> >>>>>> but
> >>>>>>>> I'll
> >>>>>>>>>>> post
> >>>>>>>>>>>> the link just be to sure [1].
> >>>>>>>>>>>>
> >>>>>>>>>>>> Since you are using Flink 1.3.2 (EMR 5.8.0 comes with
> >>> Flink
> >>>>>>> 1.3.1)
> >>>>>>>>> did
> >>>>>>>>>>> you
> >>>>>>>>>>>> built Flink yourself or did you download the binaries?
> >>>>>>>>>>>> Does the Hadoop version of the Flink build match the
> >>> Hadoop
> >>>>>>> version
> >>>>>>>>> of
> >>>>>>>>>>> EMR
> >>>>>>>>>>>> 5.8.0, i.e., Hadoop 2.7.x?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Best, Fabian
> >>>>>>>>>>>>
> >>>>>>>>>>>> [1]
> >>>>>>>>>>>> https://ci.apache.org/projects/flink/flink-docs-
> >>>>>>>>>>> release-1.3/setup/aws.html
> >>>>>>>>>>>>
> >>>>>>>>>>>> 2017-10-02 21:51 GMT+02:00 Andy M. <ajm2...@gmail.com
> >>> :
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hi Fabian,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Sorry, I just realized I forgot to include that part.
> >>>> The
> >>>>>>> error
> >>>>>>>>>>> returned
> >>>>>>>>>>>>> is:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> java.lang.NoSuchMethodError:
> >>>>>>>>>>>>> org.apache.hadoop.conf.Configuration.addResource(
> >>>>>>>>>>>> Lorg/apache/hadoop/conf/
> >>>>>>>>>>>>> Configuration;)V
> >>>>>>>>>>>>> at com.amazon.ws.emr.hadoop.fs.
> >>>>> EmrFileSystem.initialize(
> >>>>>>>>>>>>> EmrFileSystem.java:93)
> >>>>>>>>>>>>> at org.apache.flink.runtime.fs.
> >>>> hdfs.HadoopFileSystem.
> >>>>>>>>>>>>> initialize(HadoopFileSystem.java:328)
> >>>>>>>>>>>>> at org.apache.flink.core.fs.FileSystem.
> >>>>>>>> getUnguardedFileSystem(
> >>>>>>>>>>>>> FileSystem.java:350)
> >>>>>>>>>>>>> at org.apache.flink.core.fs.
> >>>> FileSystem.get(FileSystem.
> >>>>>>>>> java:389)
> >>>>>>>>>>>>> at org.apache.flink.core.fs.Path.
> >>>>>>>> getFileSystem(Path.java:293)
> >>>>>>>>>>>>> at org.apache.flink.runtime.state.filesystem.
> >>>>>>>>>>>>> FsCheckpointStreamFactory.<init>(
> >>>>> FsCheckpointStreamFactory.
> >>>>>>>>> java:99)
> >>>>>>>>>>>>> at org.apache.flink.runtime.state.filesystem.
> >>>>>>> FsStateBackend.
> >>>>>>>>>>>>> createStreamFactory(FsStateBackend.java:282)
> >>>>>>>>>>>>> at org.apache.flink.contrib.streaming.state.
> >>>>>>>>> RocksDBStateBackend.
> >>>>>>>>>>>>> createStreamFactory(RocksDBStateBackend.java:273
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I believe it has something to do with the classpath,
> >>> but
> >>>> I
> >>>>> am
> >>>>>>>>> unsure
> >>>>>>>>>>> why
> >>>>>>>>>>>> or
> >>>>>>>>>>>>> how to fix it. The classpath being used during the
> >>>>> execution
> >>>>>>> is:
> >>>>>>>>>>>>> /home/hadoop/flink-1.3.2/lib/
> >>>> flink-python_2.11-1.3.2.jar:/
> >>>>>>>>>>>>> home/hadoop/flink-1.3.2/
> >>> lib/flink-shaded-hadoop2-
> >>>>>>>>>>>>> uber-1.3.2.jar:/home/hadoop/
> >>>>> flink-1.3.2/lib/log4j-1.2.
> >>>>>>>>>>>>> 17.jar:/home/hadoop/flink-1.
> >>>>> 3.2/lib/slf4j-log4j12-1.7.7.
> >>>>>>>>>>>>> jar:/home/hadoop/flink-1.
> >>>>> 3.2/lib/flink-dist_2.11-1.3.
> >>>>>>>>>>>>> 2.jar::/etc/hadoop/conf:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I decompiled flink-shaded-hadoop2-uber-1.3.2.jar
> >>> and
> >>>>> it
> >>>>>>>> seems
> >>>>>>>>>> the
> >>>>>>>>>>>>> addResource function does seem to be there.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thank you
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Mon, Oct 2, 2017 at 3:43 PM, Fabian Hueske <
> >>>>>>> fhue...@gmail.com
> >>>>>>>>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi Andy,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> can you describe in more detail what exactly isn't
> >>>>> working?
> >>>>>>>>>>>>>> Do you see error messages in the log files or on
> >> the
> >>>>>> console?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks, Fabian
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> 2017-10-02 15:52 GMT+02:00 Andy M. <
> >>> ajm2...@gmail.com
> >>>>> :
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hello,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I am about to deploy my first Flink projects to
> >>>>>>> production,
> >>>>>>>>> but
> >>>>>>>>>> I
> >>>>>>>>>>> am
> >>>>>>>>>>>>>>> running into a very big hurdle. I am unable to
> >>>> launch
> >>>>> my
> >>>>>>>>> project
> >>>>>>>>>>> so
> >>>>>>>>>>>> it
> >>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>> write to an S3 bucket. My project is running on
> >> an
> >>>> EMR
> >>>>>>>>> cluster,
> >>>>>>>>>>>> where
> >>>>>>>>>>>>> I
> >>>>>>>>>>>>>>> have installed Flink 1.3.2. I am using Yarn to
> >>>> launch
> >>>>>> the
> >>>>>>>>>>>> application,
> >>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>> it seems to run fine unless I am trying to enable
> >>>> check
> >>>>>>>>>>>> pointing(with a
> >>>>>>>>>>>>>> S3
> >>>>>>>>>>>>>>> target). I am looking to use RocksDB as my
> >>>>>> check-pointing
> >>>>>>>>>> backend.
> >>>>>>>>>>>> I
> >>>>>>>>>>>>>> have
> >>>>>>>>>>>>>>> asked a few places, and I am still unable to
> >> find a
> >>>>>>> solution
> >>>>>>>> to
> >>>>>>>>>>> this
> >>>>>>>>>>>>>>> problem. Here are my steps for creating a
> >> cluster,
> >>>> and
> >>>>>>>>> launching
> >>>>>>>>>>> my
> >>>>>>>>>>>>>>> application, perhaps I am missing a step. I'd be
> >>>> happy
> >>>>>> to
> >>>>>>>>>> provide
> >>>>>>>>>>>> any
> >>>>>>>>>>>>>>> additional information if needed.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> AWS Portal:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> 1) EMR -> Create Cluster
> >>>>>>>>>>>>>>> 2) Advanced Options
> >>>>>>>>>>>>>>> 3) Release = emr-5.8.0
> >>>>>>>>>>>>>>> 4) Only select Hadoop 2.7.3
> >>>>>>>>>>>>>>> 5) Next -> Next -> Next -> Create Cluster ( I
> >>> do
> >>>>> fill
> >>>>>>> out
> >>>>>>>>>>>>>>> names/keys/etc)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Once the cluster is up I ssh into the Master and
> >> do
> >>>> the
> >>>>>>>>>> following:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> 1 wget
> >>>>>>>>>>>>>>> http://apache.claz.org/flink/
> >>>>>> flink-1.3.2/flink-1.3.2-bin-
> >>>>>>>>>>>>>>> hadoop27-scala_2.11.tgz
> >>>>>>>>>>>>>>> 2 tar -xzf flink-1.3.2-bin-hadoop27-
> >>>>> scala_2.11.tgz
> >>>>>>>>>>>>>>> 3 cd flink-1.3.2
> >>>>>>>>>>>>>>> 4 ./bin/yarn-session.sh -n 2 -tm 5120 -s 4
> >> -d
> >>>>>>>>>>>>>>> 5 Change conf/flink-conf.yaml
> >>>>>>>>>>>>>>> 6 ./bin/flink run -m yarn-cluster -yn 1
> >>>>>>>>> ~/flink-consumer.jar
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> My conf/flink-conf.yaml I add the following
> >> fields:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> state.backend: rocksdb
> >>>>>>>>>>>>>>> state.backend.fs.checkpointdir:
> >>>>> s3:/bucket/location
> >>>>>>>>>>>>>>> state.checkpoints.dir: s3:/bucket/location
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> My program's checkpointing setup:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> env.enableCheckpointing(getCheckpointRate,
> >>>>>>>>>>> CheckpointingMode.EXACTLY_
> >>>>>>>>>>>>>> ONCE)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> env.getCheckpointConfig.
> >>>> enableExternalizedCheckpoints(
> >>>>>>>>>>>>>>> ExternalizedCheckpointCleanup.
> >>>> RETAIN_ON_CANCELLATION)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> env.getCheckpointConfig.
> >>>> setMinPauseBetweenCheckpoints(
> >>>>>>>>>>>>>>> getCheckpointMinPause)
> >>>>>>>>>>>>>>> env.getCheckpointConfig.
> >> setCheckpointTimeout(
> >>>>>>>>>>>> getCheckpointTimeout)
> >>>>>>>>>>>>>>> env.getCheckpointConfig.
> >>>>>> setMaxConcurrentCheckpoints(1)
> >>>>>>>>>>>>>>> env.setStateBackend(new
> >>>> RocksDBStateBackend("s3://
> >>>>>>>>>>>>> bucket/location",
> >>>>>>>>>>>>>>> true))
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
>
>
