Hi Andy, the CliFrontend is not executed via Yarn, thus, it is not affected by dependencies which are added due to the underlying Yarn cluster. Therefore, it would be helpful to look at the TaskManager logs. Either you have enabled log aggregation on your Yarn cluster, then you can obtain the logs via `yarn logs -applicationId <application ID>` or you have to retrieve them from the machines where they were running (either by going directly there or via the Yarn web interface).
Cheers, Till On Wed, Oct 4, 2017 at 4:27 PM, Andy M. <ajm2...@gmail.com> wrote: > Hi Till, > > That is actually the classpath used by the flink bash script(that launches > the jar using the java command). I changed the execute to an echo, and > grabbed that for the CLI arguments. > > I believe this is the class path from the log file(although it might not be > the taskmanager log, is that any different from what would be in my > flink-1.3.2/log folder?): > > 2017-10-02 20:03:26,450 INFO org.apache.flink.client.CliFrontend > - Classpath: > /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/ > home/hadoop/flink-1.3.2/lib/flink-shaded-hadoop2-uber-1.3. > 2.jar:/home/hadoop/flink-1.3.2/lib/log4j-1.2.17.jar:/home/ > hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7.jar:/home/ > hadoop/flink-1.3.2/lib/flink-dist_2.11-1.3.2.jar::/etc/hadoop/conf: > > If that doesn't seem right, and you can point me in the right direction as > to where the TaskManager logs would be, I would be happy to grab the > information your looking for. > > Thank you > > On Wed, Oct 4, 2017 at 3:27 AM, Till Rohrmann <trohrm...@apache.org> > wrote: > > > Hi Andy, > > > > this looks to me indeed like a dependency problem. I assume that EMR or > > something else is pulling in an incompatible version of Hadoop. > > > > The classpath you've posted, is this the one logged in the log files > > (TaskManager log) or did you compile it yourself? In the latter case, it > > would also be helpful to get access to the TaskManager logs. > > > > Cheers, > > Till > > > > On Mon, Oct 2, 2017 at 10:20 PM, Andy M. <ajm2...@gmail.com> wrote: > > > > > Hi Fabian, > > > > > > 1) I have looked at the linked docs, and from what I can tell no setup > > > should really need to be done to get Flink working(Other than > downloading > > > the correct binaries, which I believe I did) > > > 2) I have downloaded the Flink 1.3.2 binaries(flink-1.3.2-bin- > > > hadoop27-scala_2.11.tgz > > > <http://apache.claz.org/flink/flink-1.3.2/flink-1.3.2-bin- > > > hadoop27-scala_2.11.tgz>) > > > This is for hadoop 2.7.X, which matches EMR 5.8.0. > > > > > > I appreciate any help or guidance you can provide me in fixing my > > problems, > > > please let me know if there is anything else I can provide you. > > > > > > Thank you > > > > > > On Mon, Oct 2, 2017 at 4:12 PM, Fabian Hueske <fhue...@gmail.com> > wrote: > > > > > > > Hi Andy, > > > > > > > > I'm not an AWS expert, so I'll just check on some common issues. > > > > > > > > I guess you already had a look at the Flink docs for AWS/EMR but I'll > > > post > > > > the link just be to sure [1]. > > > > > > > > Since you are using Flink 1.3.2 (EMR 5.8.0 comes with Flink 1.3.1) > did > > > you > > > > built Flink yourself or did you download the binaries? > > > > Does the Hadoop version of the Flink build match the Hadoop version > of > > > EMR > > > > 5.8.0, i.e., Hadoop 2.7.x? > > > > > > > > Best, Fabian > > > > > > > > [1] > > > > https://ci.apache.org/projects/flink/flink-docs- > > > release-1.3/setup/aws.html > > > > > > > > 2017-10-02 21:51 GMT+02:00 Andy M. <ajm2...@gmail.com>: > > > > > > > > > Hi Fabian, > > > > > > > > > > Sorry, I just realized I forgot to include that part. The error > > > returned > > > > > is: > > > > > > > > > > java.lang.NoSuchMethodError: > > > > > org.apache.hadoop.conf.Configuration.addResource( > > > > Lorg/apache/hadoop/conf/ > > > > > Configuration;)V > > > > > at com.amazon.ws.emr.hadoop.fs.EmrFileSystem.initialize( > > > > > EmrFileSystem.java:93) > > > > > at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem. > > > > > initialize(HadoopFileSystem.java:328) > > > > > at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem( > > > > > FileSystem.java:350) > > > > > at org.apache.flink.core.fs.FileSystem.get(FileSystem. > java:389) > > > > > at org.apache.flink.core.fs.Path.getFileSystem(Path.java:293) > > > > > at org.apache.flink.runtime.state.filesystem. > > > > > FsCheckpointStreamFactory.<init>(FsCheckpointStreamFactory. > java:99) > > > > > at org.apache.flink.runtime.state.filesystem.FsStateBackend. > > > > > createStreamFactory(FsStateBackend.java:282) > > > > > at org.apache.flink.contrib.streaming.state. > RocksDBStateBackend. > > > > > createStreamFactory(RocksDBStateBackend.java:273 > > > > > > > > > > I believe it has something to do with the classpath, but I am > unsure > > > why > > > > or > > > > > how to fix it. The classpath being used during the execution is: > > > > > /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/ > > > > > home/hadoop/flink-1.3.2/lib/flink-shaded-hadoop2- > > > > > uber-1.3.2.jar:/home/hadoop/flink-1.3.2/lib/log4j-1.2. > > > > > 17.jar:/home/hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7. > > > > > jar:/home/hadoop/flink-1.3.2/lib/flink-dist_2.11-1.3. > > > > > 2.jar::/etc/hadoop/conf: > > > > > > > > > > I decompiled flink-shaded-hadoop2-uber-1.3.2.jar and it seems > > the > > > > > addResource function does seem to be there. > > > > > > > > > > Thank you > > > > > > > > > > On Mon, Oct 2, 2017 at 3:43 PM, Fabian Hueske <fhue...@gmail.com> > > > wrote: > > > > > > > > > > > Hi Andy, > > > > > > > > > > > > can you describe in more detail what exactly isn't working? > > > > > > Do you see error messages in the log files or on the console? > > > > > > > > > > > > Thanks, Fabian > > > > > > > > > > > > 2017-10-02 15:52 GMT+02:00 Andy M. <ajm2...@gmail.com>: > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > I am about to deploy my first Flink projects to production, > but > > I > > > am > > > > > > > running into a very big hurdle. I am unable to launch my > project > > > so > > > > it > > > > > > can > > > > > > > write to an S3 bucket. My project is running on an EMR > cluster, > > > > where > > > > > I > > > > > > > have installed Flink 1.3.2. I am using Yarn to launch the > > > > application, > > > > > > and > > > > > > > it seems to run fine unless I am trying to enable check > > > > pointing(with a > > > > > > S3 > > > > > > > target). I am looking to use RocksDB as my check-pointing > > backend. > > > > I > > > > > > have > > > > > > > asked a few places, and I am still unable to find a solution to > > > this > > > > > > > problem. Here are my steps for creating a cluster, and > launching > > > my > > > > > > > application, perhaps I am missing a step. I'd be happy to > > provide > > > > any > > > > > > > additional information if needed. > > > > > > > > > > > > > > AWS Portal: > > > > > > > > > > > > > > 1) EMR -> Create Cluster > > > > > > > 2) Advanced Options > > > > > > > 3) Release = emr-5.8.0 > > > > > > > 4) Only select Hadoop 2.7.3 > > > > > > > 5) Next -> Next -> Next -> Create Cluster ( I do fill out > > > > > > > names/keys/etc) > > > > > > > > > > > > > > Once the cluster is up I ssh into the Master and do the > > following: > > > > > > > > > > > > > > 1 wget > > > > > > > http://apache.claz.org/flink/flink-1.3.2/flink-1.3.2-bin- > > > > > > > hadoop27-scala_2.11.tgz > > > > > > > 2 tar -xzf flink-1.3.2-bin-hadoop27-scala_2.11.tgz > > > > > > > 3 cd flink-1.3.2 > > > > > > > 4 ./bin/yarn-session.sh -n 2 -tm 5120 -s 4 -d > > > > > > > 5 Change conf/flink-conf.yaml > > > > > > > 6 ./bin/flink run -m yarn-cluster -yn 1 > ~/flink-consumer.jar > > > > > > > > > > > > > > My conf/flink-conf.yaml I add the following fields: > > > > > > > > > > > > > > state.backend: rocksdb > > > > > > > state.backend.fs.checkpointdir: s3:/bucket/location > > > > > > > state.checkpoints.dir: s3:/bucket/location > > > > > > > > > > > > > > My program's checkpointing setup: > > > > > > > > > > > > > > > > > > > > > env.enableCheckpointing(getCheckpointRate, > > > CheckpointingMode.EXACTLY_ > > > > > > ONCE) > > > > > > > > > > > > > > env.getCheckpointConfig.enableExternalizedCheckpoints( > > > > > > > ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION) > > > > > > > > > > > > > > env.getCheckpointConfig.setMinPauseBetweenCheckpoints( > > > > > > > getCheckpointMinPause) > > > > > > > env.getCheckpointConfig.setCheckpointTimeout( > > > > getCheckpointTimeout) > > > > > > > env.getCheckpointConfig.setMaxConcurrentCheckpoints(1) > > > > > > > env.setStateBackend(new RocksDBStateBackend("s3:// > > > > > bucket/location", > > > > > > > true)) > > > > > > > > > > > > > > > > > > > > > > > > > > > >
