Hi, sorry for resurrecting the old thread, but I have precisely the same issue
with a different filesystem.
I've tried using plugins dir, setting FLINK_PLUGINS_DIR, etc. - nothing works
locally. I added a breakpoint to the PluginConfig.getPluginsDir method and
confirmed it's not even called.
Could this be a LocalStreamEnvironment limitation? Is there any way to enable
plugin loading locally?
Thanks!
On 2021/06/21 11:13:29, Yuval Itzchakov <yuva...@gmail.com> wrote:
> Currently I have the s3-hadoop dependency in my build.sbt.
>
> I guess I need to move it to the PLUGIN directory locally as well, didn't
> think of that.
>
> On Mon, Jun 21, 2021, 13:35 Dawid Wysakowicz <dwysakow...@apache.org> wrote:
>
> > Hi,
> >
> > Where do you use the AWS SDK v2? Filesystems should use the plugins
> > subsystem[1]. Therefore when running from within the IDE make sure you have
> > the s3 plugin in ${WORKING_DIR}/plugins/s3 and put the flink-s3-fs-hadoop
> > dependency there. Or specify a different directory for plugins via the ENV
> > variable: FLINK_PLUGINS_DIR.
> >
> > If you want to use the AWS SDK v2 for the s3 filesystem I guess you need
> > to build the jar yourself.
> >
> > Best,
> >
> > Dawid
> >
> > [1]
> > https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/filesystems/plugins/
> >
> >
> > On 20/06/2021 15:00, Yuval Itzchakov wrote:
> >
> > Hi,
> >
> > I'm trying to run Flink with AWS SDK v2 and restore a Flink savepoint
> > that's stored in S3.
> >
> > When adding the dependency on flink-s3-fs-hadoop locally, I fail to run
> > Flink at runtime with the following error:
> >
> > Caused by: java.lang.invoke.LambdaConversionException: Invalid receiver
> > type interface org.apache.http.Header; not a subtype of implementation type
> > interface org.apache.http.NameValuePair
> >
> > [image: image.png]
> >
> > This happens because the flink-s3 JAR is a fat JAR containing an old
> > version of org.apache.http.HttpMessage which does not implement the
> > NameValuePair interface (there was a non backwards compatible change in the
> > library, see https://github.com/aws/aws-sdk-java-v2/issues/652)
> >
> > [image: image.png]
> >
> > Since the library itself isn't shaded, it gets loaded first overriding the
> > HttpClient newer version I have on the classpath and blows up when
> > bootstrapping Flink.
> >
> > Any advice on how to work around this?
> > --
> > Best Regards,
> > Yuval Itzchakov.
> >
> >
>
