There is no good way in 1.15 IIRC.
Adding a dependency on flink-s3-fs-hadoop _can_ work, if you dont run
into dependency conflicts.
Otherwise you have to create a plugin manager yourself, point it to some
local directory via a system property (I think?), and then eagerly call
FileSystem#initialize with the plugin manager.
On 04/01/2023 02:11, Yaroslav Tkachenko wrote:
Hey Ken,
I have flink-s3-fs-hadoop as a provided dependency in my project, and
I've configured my IDE to include provided dependencies when starting
applications. Works just fine.
On Tue, Jan 3, 2023 at 5:06 PM Ken Krugler
<kkrugler_li...@transpac.com> wrote:
Hi all,
With Flink 1.15.x, is there a way to use the S3 Presto plugin when
running code on the MiniCluster?
I can’t just add that jar as a dependency when testing, as I get:
java.lang.NoClassDefFoundError: Could not initialize class
com.facebook.presto.hive.s3.PrestoS3FileSystem
at
org.apache.flink.fs.s3presto.S3FileSystemFactory.createHadoopFileSystem(S3FileSystemFactory.java:88)
~[flink-s3-fs-presto-1.15.1.jar:1.15.1]
at
org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory.create(AbstractS3FileSystemFactory.java:126)
~[flink-s3-fs-presto-1.15.1.jar:1.15.1]
I assume that’s because of this warning in the Flink docs:
The s3 file systems (flink-s3-fs-presto and flink-s3-fs-hadoop)
can only be used as plugins as we already removed the
relocations. Placing them in libs/ will result in system failures.
In the latest Flink JavaDocs, there’s a way to specify the
PluginManager for the MiniClusterConfiguration, but I don’t see
that in Flink 1.15.x
So is there a workaround to allow me to run a test from inside of
my IDE, using the MiniCluster, that reads from S3?
Thanks,
— Ken
--------------------------
Ken Krugler
http://www.scaleunlimited.com
Custom big data solutions
Flink, Pinot, Solr, Elasticsearch
