[jira] [Created] (FLINK-10268) Document update deployment/aws HADOOP_CLASSPATH
Andy M created FLINK-10268: -- Summary: Document update deployment/aws HADOOP_CLASSPATH Key: FLINK-10268 URL: https://issues.apache.org/jira/browse/FLINK-10268 Project: Flink Issue Type: Improvement Components: Documentation Reporter: Andy M The Deployment/AWS/Custom EMR Installation documents need to be updated. Currently the steps will result in a ClassNotFoundException. A step needs to be added to include setting HADOOP_CLASSPATH=`hadoop classpath` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Cannot run example on EMR
I am trying to launch a Flink cluster on EMR. I am following directions here: https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/deployment/aws.html#custom-emr-installation - I created a cluster with emr-5.16.0 - Hadoop 2.8.4 - I SSH'd into the master node, downloaded flink-1.6.0-bin-hadoop28-scala_2.11.tgz - I tried to run HADOOP_CONF_DIR=/etc/hadoop/conf ./bin/flink run -m yarn-cluster -yn 1 examples/streaming/WordCount.jar and got the an error: Caused by: java.lang.ClassNotFoundException: javax.ws.rs.ext.MessageBodyReader I also get the same error when trying to launch yarn-session.sh Is anyone able to run 1.6 on EMR and provide me with what I am doing wrong? Thank you
Re: Unable to write snapshots to S3 on EMR
Hi Ajoscha,
That didn't seem to do the trick either. Do the following look correct?
I see 5.9.0 is released with Flink 1.3.2, so I tried that, and got the same
problem, all I did was upload my Scala .jar to the master, updated my
flink-conf.yaml, set my env variables, and ran it with the following
command: HADOOP_CONF_DIR=/etc/hadoop/conf flink run -m yarn-cluster -yn 5
-ys 2 ~/flink-consumer.jar
I am beginning to think something else may be wrong with my configuration.
Does the following look correct?
In flink-conf.yaml:
state.backend: rocksdb
state.backend.fs.checkpointdir: s3://org/flink-project/state
state.checkpoints.dir: s3://org/flink-project/state
In the code:
env.enableCheckpointing(getCheckpointRate,CheckpointingMode.EXACTLY_ONCE)
env.getCheckpointConfig.enableExternalizedCheckpoints(ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION)
env.getCheckpointConfig.setMinPauseBetweenCheckpoints(1000)
env.getCheckpointConfig.setCheckpointTimeout(5000)
env.getCheckpointConfig.setMaxConcurrentCheckpoints(1)
env.setStateBackend(new RocksDBStateBackend("s3://org/flink-project/state",
true))
Thank you
On Wed, Oct 11, 2017 at 5:19 AM, Aljoscha Krettek <aljos...@apache.org>
wrote:
> Hi Andy,
>
> I remember that I was testing a job with almost exactly the same setup as
> part of the Flink 1.3.2 release testing. The command I used to start my job
> is roughly this:
>
> HADOOP_CONF_DIR=/etc/hadoop/conf bin/flink run -c my.main.Class -m
> yarn-cluster -yn 5 -ys 2 ...
>
> i.e. I export the proper hadoop config dir and I run a per-job YARN
> cluster. I think I also exported the result of "hadoop classpath" as
> HADOOP_CLASSPATH.
>
> Best,
> Aljoscha
>
> > On 10. Oct 2017, at 16:43, Andy M. <ajm2...@gmail.com> wrote:
> >
> > Hello,
> >
> > Bowen: Unless I am missing something, it says there needs to be no setup
> > on EMR, Each topic says: "You don’t have to configure this manually if
> you
> > are running Flink on EMR." S3 access from CLI works fine on my clusters.
> >
> > Chen: Thank you for this, I will look into this if I am unable to get
> this
> > running on YARN successfully.
> >
> > Stephan: Removing the said library causes the flink
> > (flink-1.3.2/bin/flink) bash script to fail. The underlying Java needs
> > this to work. I tried explicitly setting the classpath for the java call
> > as well to point to the hadoop library jars. This is the original java
> > command that I was trying to run:
> >
> > java
> > -Dlog.file=/home/hadoop/flink-1.3.2/log/flink-hadoop-client-
> ip-172-31-19-27.log
> > -Dlog4j.configuration=file:/home/hadoop/flink-1.3.2/conf/
> log4j-cli.properties
> > -Dlogback.configurationFile=file:/home/hadoop/flink-1.3.2/
> conf/logback.xml
> > -classpath
> > /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/
> home/hadoop/flink-1.3.2/lib/flink-shaded-hadoop2-uber-1.3.
> 2.jar:/home/hadoop/flink-1.3.2/lib/log4j-1.2.17.jar:/home/
> hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7.jar:/home/
> hadoop/flink-1.3.2/lib/flink-dist_2.11-1.3.2.jar::/etc/hadoop/conf:
> > org.apache.flink.client.CliFrontend run -m yarn-cluster -yn 1
> > /home/hadoop/flink-consumer.jar
> >
> >
> > This is what I changed it too(removing the shadded-hadoop2-uber jar and
> > adding in the hadoop folder):
> >
> > java
> > -Dlog.file=/home/hadoop/flink-1.3.2/log/flink-hadoop-client-
> ip-172-31-19-27.log
> > -Dlog4j.configuration=file:/home/hadoop/flink-1.3.2/conf/
> log4j-cli.properties
> > -Dlogback.configurationFile=file:/home/hadoop/flink-1.3.2/
> conf/logback.xml
> > -classpath
> > /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/
> home/hadoop/flink-1.3.2/lib/log4j-1.2.17.jar:/home/hadoop/
> flink-1.3.2/lib/slf4j-log4j12-1.7.7.jar:/home/hadoop/flink-
> 1.3.2/lib/flink-dist_2.11-1.3.2.jar:/usr/lib/hadoop/lib/
> activation-1.1.jar:/usr/lib/hadoop/lib/commons-io-2.4.jar:
> /usr/lib/hadoop/lib/jackson-mapper-asl-1.9.13.jar:/usr/
> lib/hadoop/lib/log4j-1.2.17.jar:/usr/lib/hadoop/lib/
> apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop/lib/commons-
> lang-2.6.jar:/usr/lib/hadoop/lib/jackson-xc-1.9.13.jar:/
> usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/
> lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop/
> lib/commons-logging-1.1.3.jar:/usr/lib/hadoop/lib/java-
> xmlbuilder-0.4.jar:/usr/lib/hadoop/lib/netty-3.6.2.Final.
> jar:/usr/lib/hadoop/lib/api-asn1-api-1.0.0-M20.jar:/usr/
> lib/hadoop/lib/commons-math3-3.1.1.jar:/usr/lib/hadoop/lib/
> jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/
> usr/lib/hadoop/lib/api-util-1.0.0-M20.jar:/usr/lib/hadoop/
> lib/commons-net-3.1.jar:/usr/lib/
Re: Unable to write snapshots to S3 on EMR
is of type [org.slf4j.impl.Log4jLoggerFactory] Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/util/VersionInfo at org.apache.flink.runtime.util.EnvironmentInformation.logEnvironmentInfo(EnvironmentInformation.java:283) at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1124) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.VersionInfo at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 2 more Simply removing the .jar from the folder causes the same error. Thank you On Mon, Oct 9, 2017 at 5:46 AM, Stephan Ewen <se...@apache.org> wrote: > Hi! > > It looks like multiple Hadoop versions are in the classpath. Flink's hadoop > jar and the EMR Hadoop jars. > I would simply drop Flink's own Hadoop dependency and only use the EMR > Hadoop jars. > > Delete the 'flink-shaded-hadoop2-uber' jar from Flink, and make sure the > setup is such that the Hadoop lib environment variable is set. Then it > should not have conflicts any more. > > > > On Sun, Oct 8, 2017 at 12:08 AM, Chen Qin <qinnc...@gmail.com> wrote: > > > Attached my side project verified working to deploy jobmanager and > > taskmanager as stateless service(non yarn/mesos), configuration here > > > > https://github.com/chenqin/flink-jar/tree/master/config/hadoop > > > > more detail here > > https://github.com/chenqin/flink-jar/blob/master/src/ > > main/java/FlinkBootstrap.java#L49 > > > > On Fri, Oct 6, 2017 at 10:26 PM, Bowen Li <bowen...@offerupnow.com> > wrote: > > > > > Hi Andy, > > > > > > I believe it's because you didn't set your s3 impl correctly. Try to > set > > > your core-site.xml by following https://ci.apache.org/ > > > projects/flink/flink-docs-release-1.4/ops/deployment/ > > > aws.html#s3afilesystem- > > > recommended > > > > > > Bowen > > > > > > On Fri, Oct 6, 2017 at 7:59 AM, Andy M. <ajm2...@gmail.com> wrote: > > > > > > > Hi Till, > > > > > > > > Seems like everything is in line there. hadoop-common.jar -> > > > > hadoop-common-2.7.3-amzn-3.jar > > > > > > > > And when i decompiled that jar I see public void > > > addResource(Configuration > > > > conf) in org/apache/hadoop/conf/Configuration.java > > > > > > > > I agree that an incorrect version of the jar is probably being run, > is > > > > there a way to limit the classpath for the TaskManager when starting > > the > > > > job? > > > > > > > > Thank you > > > > > > > > On Fri, Oct 6, 2017 at 6:49 AM, Till Rohrmann <trohrm...@apache.org> > > > > wrote: > > > > > > > > > Hi Andy, > > > > > > > > > > could you check which Hadoop version this jar > > > > > /usr/lib/hadoop/hadoop-common.jar is? Maybe also checking whether > > the > > > > > contained hadoop Configuration class has the method > > > > > Configuration.addResource(Lorg/apache/hadoop/conf/ > Configuration;)V. > > > > Maybe > > > > > this jar is the culprit because it comes from a different Hadoop > > > version. > > > > > > > > > > Cheers, > > > > > Till > > > > > > > > > > > > > > > On Thu, Oct 5, 2017 at 4:22 PM, Andy M. <ajm2...@gmail.com> wrote: > > > > > > > > > > > Hi Till, > > > > > > > > > > > > I believe this is what you are looking for, classpath is much > > bigger > > > > for > > > > > > the task manager. I can also post the whole log file if needed: > > > > > > > > > > > > 2017-10-05 14:17:53,038 INFO org.apache.flink.yarn. > > > > > YarnTaskManagerRunner > > > > > > - Classpath: > > > > > > flink-consumer.jar:lib/flink-dist_2.11-1.3.2.jar:lib/flink- > > > > > > python_2.11-1.3.2.jar:lib/flink-shaded-hadoop2-uber-1.3. > > > > > > 2.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar: > > > > > > log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/ > > > > > > etc/hadoop/conf:/usr/lib/hadoop/hadoop-common-2.7.3- >
Re: Unable to write snapshots to S3 on EMR
Hi Till, Seems like everything is in line there. hadoop-common.jar -> hadoop-common-2.7.3-amzn-3.jar And when i decompiled that jar I see public void addResource(Configuration conf) in org/apache/hadoop/conf/Configuration.java I agree that an incorrect version of the jar is probably being run, is there a way to limit the classpath for the TaskManager when starting the job? Thank you On Fri, Oct 6, 2017 at 6:49 AM, Till Rohrmann <trohrm...@apache.org> wrote: > Hi Andy, > > could you check which Hadoop version this jar > /usr/lib/hadoop/hadoop-common.jar is? Maybe also checking whether the > contained hadoop Configuration class has the method > Configuration.addResource(Lorg/apache/hadoop/conf/Configuration;)V. Maybe > this jar is the culprit because it comes from a different Hadoop version. > > Cheers, > Till > > > On Thu, Oct 5, 2017 at 4:22 PM, Andy M. <ajm2...@gmail.com> wrote: > > > Hi Till, > > > > I believe this is what you are looking for, classpath is much bigger for > > the task manager. I can also post the whole log file if needed: > > > > 2017-10-05 14:17:53,038 INFO org.apache.flink.yarn. > YarnTaskManagerRunner > > - Classpath: > > flink-consumer.jar:lib/flink-dist_2.11-1.3.2.jar:lib/flink- > > python_2.11-1.3.2.jar:lib/flink-shaded-hadoop2-uber-1.3. > > 2.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar: > > log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/ > > etc/hadoop/conf:/usr/lib/hadoop/hadoop-common-2.7.3- > > amzn-3-tests.jar:/usr/lib/hadoop/hadoop-annotations-2.7. > > 3-amzn-3.jar:/usr/lib/hadoop/hadoop-distcp.jar:/usr/lib/ > > hadoop/hadoop-auth-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop- > > nfs-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-streaming-2.7.3- > > amzn-3.jar:/usr/lib/hadoop/hadoop-ant-2.7.3-amzn-3.jar:/ > > usr/lib/hadoop/hadoop-distcp-2.7.3-amzn-3.jar:/usr/lib/ > > hadoop/hadoop-datajoin.jar:/usr/lib/hadoop/hadoop- > > streaming.jar:/usr/lib/hadoop/hadoop-common.jar:/usr/lib/ > > hadoop/hadoop-ant.jar:/usr/lib/hadoop/hadoop-sls.jar:/ > > usr/lib/hadoop/hadoop-azure-2.7.3-amzn-3.jar:/usr/lib/ > > hadoop/hadoop-nfs.jar:/usr/lib/hadoop/hadoop-extras-2.7. > > 3-amzn-3.jar:/usr/lib/hadoop/hadoop-gridmix.jar:/usr/lib/ > > hadoop/hadoop-common-2.7.3-amzn-3.jar:/usr/lib/hadoop/ > > hadoop-annotations.jar:/usr/lib/hadoop/hadoop-openstack-2. > > 7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-archives-2.7.3- > > amzn-3.jar:/usr/lib/hadoop/hadoop-azure.jar:/usr/lib/ > > hadoop/hadoop-extras.jar:/usr/lib/hadoop/hadoop-openstack. > > jar:/usr/lib/hadoop/hadoop-rumen.jar:/usr/lib/hadoop/ > > hadoop-aws-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop- > > datajoin-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop- > > archives.jar:/usr/lib/hadoop/hadoop-aws.jar:/usr/lib/ > > hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-rumen-2.7.3- > > amzn-3.jar:/usr/lib/hadoop/hadoop-sls-2.7.3-amzn-3.jar:/ > > usr/lib/hadoop/hadoop-gridmix-2.7.3-amzn-3.jar:/usr/lib/ > > hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/jaxb-api-2. > > 2.2.jar:/usr/lib/hadoop/lib/htrace-core-3.1.0-incubating. > > jar:/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar:/usr/ > > lib/hadoop/lib/httpclient-4.5.3.jar:/usr/lib/hadoop/lib/ > > httpcore-4.4.4.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4. > > 1.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0. > > jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/ > > lib/activation-1.1.jar:/usr/lib/hadoop/lib/jersey-server- > > 1.9.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/ > > usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/ > > gson-2.2.4.jar:/usr/lib/hadoop/lib/commons-digester-1. > > 8.jar:/usr/lib/hadoop/lib/jackson-xc-1.9.13.jar:/usr/ > > lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/ > > apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop/lib/commons- > > httpclient-3.1.jar:/usr/lib/hadoop/lib/curator-client-2.7. > > 1.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/ > > lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/ > > commons-net-3.1.jar:/usr/lib/hadoop/lib/jets3t-0.9.0.jar:/ > > usr/lib/hadoop/lib/commons-io-2.4.jar:/usr/lib/hadoop/lib/ > > xmlenc-0.52.jar:/usr/lib/hadoop/lib/jersey-json-1.9. > > jar:/usr/lib/hadoop/lib/avro-1.7.4.jar:/usr/lib/hadoop/lib/ > > commons-math3-3.1.1.jar:/usr/lib/hadoop/lib/log4j-1.2.17. > > jar:/usr/lib/hadoop/lib/commons-compress-1.4.1.jar:/ > > usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/ > > jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/netty-3.6.2. > > Final.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/ > > lib/hadoop/lib/asm-3.2.jar:/usr/lib/hado
Re: Unable to write snapshots to S3 on EMR
/share/aws/aws-java-sdk/aws-java-sdk-dms-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-directory-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opsworkscm-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudformation-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudfront-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-budgets-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-clouddirectory-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-importexport-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lex-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-marketplaceentitlement-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dynamodb-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-autoscaling-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elastictranscoder-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-organizations-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-workspaces-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ssm-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-servermigration-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-events-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-applicationautoscaling-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-health-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-kms-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-logs-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codestar-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-route53-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-redshift-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-marketplacemeteringservice-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sns-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-batch-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-waf-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-simpledb-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codedeploy-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ec2-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-models-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-devicefarm-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cognitoidentity-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lexmodelbuilding-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-directconnect-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticache-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-costandusagereport-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-discovery-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-resourcegroupstaggingapi-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ses-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lambda-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-workdocs-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-code-generator-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cognitosync-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-efs-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sts-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-athena-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codecommit-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-storagegateway-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lightsail-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-acm-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-glacier-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ecr-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-support-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codegen-maven-plugin-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticsearch-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iot-1.11.160.jar Thank you On Thu, Oct 5, 2017 at 5:25 AM, Till Rohrmann <trohrm...@apache.org> wrote: > Hi Andy, > > the CliFrontend is not executed via Yarn, thus, it is not affected by > dependencies which are added due to the underlying Yarn cluster. Therefore, > it would be helpful to look at the TaskManager logs. Either you have > enabled log aggregation on your Yarn cluster, then you can obtain the logs > via `yarn logs -applicationId ` or you have to retrieve > them from the machines where they were running (either by going directly > there or via the Yarn web interface). > > Cheers, > Till > > On Wed, Oct 4, 2017 at 4:27 PM, Andy M. <ajm2...@gmail.com> wrote: > > > Hi Till, > > > > That is actually the classpath used by the flink bash script(that > launches > > the jar using the java command). I changed the execute to an echo, and > > grabbed that for the CLI arguments. > > > > I believe this is the class path from the log file(although it might not > be > > the taskmanager log, is that any different from what would b
Re: Unable to write snapshots to S3 on EMR
Hi Fabian, 1) I have looked at the linked docs, and from what I can tell no setup should really need to be done to get Flink working(Other than downloading the correct binaries, which I believe I did) 2) I have downloaded the Flink 1.3.2 binaries(flink-1.3.2-bin- hadoop27-scala_2.11.tgz <http://apache.claz.org/flink/flink-1.3.2/flink-1.3.2-bin-hadoop27-scala_2.11.tgz>) This is for hadoop 2.7.X, which matches EMR 5.8.0. I appreciate any help or guidance you can provide me in fixing my problems, please let me know if there is anything else I can provide you. Thank you On Mon, Oct 2, 2017 at 4:12 PM, Fabian Hueske <fhue...@gmail.com> wrote: > Hi Andy, > > I'm not an AWS expert, so I'll just check on some common issues. > > I guess you already had a look at the Flink docs for AWS/EMR but I'll post > the link just be to sure [1]. > > Since you are using Flink 1.3.2 (EMR 5.8.0 comes with Flink 1.3.1) did you > built Flink yourself or did you download the binaries? > Does the Hadoop version of the Flink build match the Hadoop version of EMR > 5.8.0, i.e., Hadoop 2.7.x? > > Best, Fabian > > [1] > https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/aws.html > > 2017-10-02 21:51 GMT+02:00 Andy M. <ajm2...@gmail.com>: > > > Hi Fabian, > > > > Sorry, I just realized I forgot to include that part. The error returned > > is: > > > > java.lang.NoSuchMethodError: > > org.apache.hadoop.conf.Configuration.addResource( > Lorg/apache/hadoop/conf/ > > Configuration;)V > > at com.amazon.ws.emr.hadoop.fs.EmrFileSystem.initialize( > > EmrFileSystem.java:93) > > at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem. > > initialize(HadoopFileSystem.java:328) > > at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem( > > FileSystem.java:350) > > at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:389) > > at org.apache.flink.core.fs.Path.getFileSystem(Path.java:293) > > at org.apache.flink.runtime.state.filesystem. > > FsCheckpointStreamFactory.(FsCheckpointStreamFactory.java:99) > > at org.apache.flink.runtime.state.filesystem.FsStateBackend. > > createStreamFactory(FsStateBackend.java:282) > > at org.apache.flink.contrib.streaming.state.RocksDBStateBackend. > > createStreamFactory(RocksDBStateBackend.java:273 > > > > I believe it has something to do with the classpath, but I am unsure why > or > > how to fix it. The classpath being used during the execution is: > > /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/ > > home/hadoop/flink-1.3.2/lib/flink-shaded-hadoop2- > > uber-1.3.2.jar:/home/hadoop/flink-1.3.2/lib/log4j-1.2. > > 17.jar:/home/hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7. > > jar:/home/hadoop/flink-1.3.2/lib/flink-dist_2.11-1.3. > > 2.jar::/etc/hadoop/conf: > > > > I decompiled flink-shaded-hadoop2-uber-1.3.2.jar and it seems the > > addResource function does seem to be there. > > > > Thank you > > > > On Mon, Oct 2, 2017 at 3:43 PM, Fabian Hueske <fhue...@gmail.com> wrote: > > > > > Hi Andy, > > > > > > can you describe in more detail what exactly isn't working? > > > Do you see error messages in the log files or on the console? > > > > > > Thanks, Fabian > > > > > > 2017-10-02 15:52 GMT+02:00 Andy M. <ajm2...@gmail.com>: > > > > > > > Hello, > > > > > > > > I am about to deploy my first Flink projects to production, but I am > > > > running into a very big hurdle. I am unable to launch my project so > it > > > can > > > > write to an S3 bucket. My project is running on an EMR cluster, > where > > I > > > > have installed Flink 1.3.2. I am using Yarn to launch the > application, > > > and > > > > it seems to run fine unless I am trying to enable check > pointing(with a > > > S3 > > > > target). I am looking to use RocksDB as my check-pointing backend. > I > > > have > > > > asked a few places, and I am still unable to find a solution to this > > > > problem. Here are my steps for creating a cluster, and launching my > > > > application, perhaps I am missing a step. I'd be happy to provide > any > > > > additional information if needed. > > > > > > > > AWS Portal: > > > > > > > > 1) EMR -> Create Cluster > > > > 2) Advanced Options > > > > 3) Release = emr-5.8.0 > > > > 4) Only sele
Re: Unable to write snapshots to S3 on EMR
Hi Fabian,
Sorry, I just realized I forgot to include that part. The error returned
is:
java.lang.NoSuchMethodError:
org.apache.hadoop.conf.Configuration.addResource(Lorg/apache/hadoop/conf/Configuration;)V
at
com.amazon.ws.emr.hadoop.fs.EmrFileSystem.initialize(EmrFileSystem.java:93)
at
org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.initialize(HadoopFileSystem.java:328)
at
org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:350)
at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:389)
at org.apache.flink.core.fs.Path.getFileSystem(Path.java:293)
at
org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory.(FsCheckpointStreamFactory.java:99)
at
org.apache.flink.runtime.state.filesystem.FsStateBackend.createStreamFactory(FsStateBackend.java:282)
at
org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createStreamFactory(RocksDBStateBackend.java:273
I believe it has something to do with the classpath, but I am unsure why or
how to fix it. The classpath being used during the execution is:
/home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/home/hadoop/flink-1.3.2/lib/flink-shaded-hadoop2-uber-1.3.2.jar:/home/hadoop/flink-1.3.2/lib/log4j-1.2.17.jar:/home/hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7.jar:/home/hadoop/flink-1.3.2/lib/flink-dist_2.11-1.3.2.jar::/etc/hadoop/conf:
I decompiled flink-shaded-hadoop2-uber-1.3.2.jar and it seems the
addResource function does seem to be there.
Thank you
On Mon, Oct 2, 2017 at 3:43 PM, Fabian Hueske <fhue...@gmail.com> wrote:
> Hi Andy,
>
> can you describe in more detail what exactly isn't working?
> Do you see error messages in the log files or on the console?
>
> Thanks, Fabian
>
> 2017-10-02 15:52 GMT+02:00 Andy M. <ajm2...@gmail.com>:
>
> > Hello,
> >
> > I am about to deploy my first Flink projects to production, but I am
> > running into a very big hurdle. I am unable to launch my project so it
> can
> > write to an S3 bucket. My project is running on an EMR cluster, where I
> > have installed Flink 1.3.2. I am using Yarn to launch the application,
> and
> > it seems to run fine unless I am trying to enable check pointing(with a
> S3
> > target). I am looking to use RocksDB as my check-pointing backend. I
> have
> > asked a few places, and I am still unable to find a solution to this
> > problem. Here are my steps for creating a cluster, and launching my
> > application, perhaps I am missing a step. I'd be happy to provide any
> > additional information if needed.
> >
> > AWS Portal:
> >
> > 1) EMR -> Create Cluster
> > 2) Advanced Options
> > 3) Release = emr-5.8.0
> > 4) Only select Hadoop 2.7.3
> > 5) Next -> Next -> Next -> Create Cluster ( I do fill out
> > names/keys/etc)
> >
> > Once the cluster is up I ssh into the Master and do the following:
> >
> > 1 wget
> > http://apache.claz.org/flink/flink-1.3.2/flink-1.3.2-bin-
> > hadoop27-scala_2.11.tgz
> > 2 tar -xzf flink-1.3.2-bin-hadoop27-scala_2.11.tgz
> > 3 cd flink-1.3.2
> > 4 ./bin/yarn-session.sh -n 2 -tm 5120 -s 4 -d
> > 5 Change conf/flink-conf.yaml
> > 6 ./bin/flink run -m yarn-cluster -yn 1 ~/flink-consumer.jar
> >
> > My conf/flink-conf.yaml I add the following fields:
> >
> > state.backend: rocksdb
> > state.backend.fs.checkpointdir: s3:/bucket/location
> > state.checkpoints.dir: s3:/bucket/location
> >
> > My program's checkpointing setup:
> >
> >
> > env.enableCheckpointing(getCheckpointRate,CheckpointingMode.EXACTLY_
> ONCE)
> >
> > env.getCheckpointConfig.enableExternalizedCheckpoints(
> > ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION)
> >
> > env.getCheckpointConfig.setMinPauseBetweenCheckpoints(
> > getCheckpointMinPause)
> > env.getCheckpointConfig.setCheckpointTimeout(getCheckpointTimeout)
> > env.getCheckpointConfig.setMaxConcurrentCheckpoints(1)
> > env.setStateBackend(new RocksDBStateBackend("s3://bucket/location",
> > true))
> >
>
Unable to write snapshots to S3 on EMR
Hello,
I am about to deploy my first Flink projects to production, but I am
running into a very big hurdle. I am unable to launch my project so it can
write to an S3 bucket. My project is running on an EMR cluster, where I
have installed Flink 1.3.2. I am using Yarn to launch the application, and
it seems to run fine unless I am trying to enable check pointing(with a S3
target). I am looking to use RocksDB as my check-pointing backend. I have
asked a few places, and I am still unable to find a solution to this
problem. Here are my steps for creating a cluster, and launching my
application, perhaps I am missing a step. I'd be happy to provide any
additional information if needed.
AWS Portal:
1) EMR -> Create Cluster
2) Advanced Options
3) Release = emr-5.8.0
4) Only select Hadoop 2.7.3
5) Next -> Next -> Next -> Create Cluster ( I do fill out
names/keys/etc)
Once the cluster is up I ssh into the Master and do the following:
1 wget
http://apache.claz.org/flink/flink-1.3.2/flink-1.3.2-bin-hadoop27-scala_2.11.tgz
2 tar -xzf flink-1.3.2-bin-hadoop27-scala_2.11.tgz
3 cd flink-1.3.2
4 ./bin/yarn-session.sh -n 2 -tm 5120 -s 4 -d
5 Change conf/flink-conf.yaml
6 ./bin/flink run -m yarn-cluster -yn 1 ~/flink-consumer.jar
My conf/flink-conf.yaml I add the following fields:
state.backend: rocksdb
state.backend.fs.checkpointdir: s3:/bucket/location
state.checkpoints.dir: s3:/bucket/location
My program's checkpointing setup:
env.enableCheckpointing(getCheckpointRate,CheckpointingMode.EXACTLY_ONCE)
env.getCheckpointConfig.enableExternalizedCheckpoints(ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION)
env.getCheckpointConfig.setMinPauseBetweenCheckpoints(getCheckpointMinPause)
env.getCheckpointConfig.setCheckpointTimeout(getCheckpointTimeout)
env.getCheckpointConfig.setMaxConcurrentCheckpoints(1)
env.setStateBackend(new RocksDBStateBackend("s3://bucket/location",
true))
