Andy M created FLINK-10268:
--
Summary: Document update deployment/aws HADOOP_CLASSPATH
Key: FLINK-10268
URL: https://issues.apache.org/jira/browse/FLINK-10268
Project: Flink
Issue Type: Improvement
I am trying to launch a Flink cluster on EMR. I am following directions
here:
https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/deployment/aws.html#custom-emr-installation
- I created a cluster with emr-5.16.0 - Hadoop 2.8.4
- I SSH'd into the master node,
downloaded flink-1.6.0-bin
r -yn 5 -ys 2 ...
>
> i.e. I export the proper hadoop config dir and I run a per-job YARN
> cluster. I think I also exported the result of "hadoop classpath" as
> HADOOP_CLASSPATH.
>
> Best,
> Aljoscha
>
> > On 10. Oct 2017, at 16:43, Andy M. wrote:
> >
#x27;s because you didn't set your s3 impl correctly. Try to
> set
> > > your core-site.xml by following https://ci.apache.org/
> > > projects/flink/flink-docs-release-1.4/ops/deployment/
> > > aws.html#s3afilesystem-
> > > recommended
> > >
the
> contained hadoop Configuration class has the method
> Configuration.addResource(Lorg/apache/hadoop/conf/Configuration;)V. Maybe
> this jar is the culprit because it comes from a different Hadoop version.
>
> Cheers,
> Till
>
>
> On Thu, Oct 5, 2017 at 4:22 PM, Andy M
you have to retrieve
> them from the machines where they were running (either by going directly
> there or via the Yarn web interface).
>
> Cheers,
> Till
>
> On Wed, Oct 4, 2017 at 4:27 PM, Andy M. wrote:
>
> > Hi Till,
> >
> > That is actually the classpath
've posted, is this the one logged in the log files
> (TaskManager log) or did you compile it yourself? In the latter case, it
> would also be helpful to get access to the TaskManager logs.
>
> Cheers,
> Till
>
> On Mon, Oct 2, 2017 at 10:20 PM, Andy M. wrote:
>
> &
/flink-docs-release-1.3/setup/aws.html
>
> 2017-10-02 21:51 GMT+02:00 Andy M. :
>
> > Hi Fabian,
> >
> > Sorry, I just realized I forgot to include that part. The error returned
> > is:
> >
> > java.lang.NoSuchMethodError
function does seem to be there.
Thank you
On Mon, Oct 2, 2017 at 3:43 PM, Fabian Hueske wrote:
> Hi Andy,
>
> can you describe in more detail what exactly isn't working?
> Do you see error messages in the log files or on the console?
>
> Thanks, Fabian
>
> 2017
Hello,
I am about to deploy my first Flink projects to production, but I am
running into a very big hurdle. I am unable to launch my project so it can
write to an S3 bucket. My project is running on an EMR cluster, where I
have installed Flink 1.3.2. I am using Yarn to launch the application,
10 matches
Mail list logo
