Hi all,
With Flink 1.15.x, is there a way to use the S3 Presto plugin when running code
on the MiniCluster?
I can’t just add that jar as a dependency when testing, as I get:
java.lang.NoClassDefFoundError: Could not initialize class
com.facebook.presto.hive.s3.PrestoS3FileSystem
at
org.apache.flink.fs.s3presto.S3FileSystemFactory.createHadoopFileSystem(S3FileSystemFactory.java:88)
~[flink-s3-fs-presto-1.15.1.jar:1.15.1]
at
org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory.create(AbstractS3FileSystemFactory.java:126)
~[flink-s3-fs-presto-1.15.1.jar:1.15.1]
I assume that’s because of this warning in the Flink docs:
> The s3 file systems (flink-s3-fs-presto and flink-s3-fs-hadoop) can only be
> used as plugins as we already removed the relocations. Placing them in libs/
> will result in system failures.
In the latest Flink JavaDocs, there’s a way to specify the PluginManager for
the MiniClusterConfiguration, but I don’t see that in Flink 1.15.x
So is there a workaround to allow me to run a test from inside of my IDE, using
the MiniCluster, that reads from S3?
Thanks,
— Ken
--------------------------
Ken Krugler
http://www.scaleunlimited.com
Custom big data solutions
Flink, Pinot, Solr, Elasticsearch
