Hi, John, AFAIK, Flink will automatically help you to ship the "plugins/" directory of your Flink distribution to Yarn[1]. So, you just need to make a directory in "plugins/" and put your custom jar into it. Do you meet any problem with this approach?
[1] https://github.com/apache/flink/blob/216f65fff10fb0957e324570662d075be66bacdf/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java#L770 Best, Yangze Guo On Wed, Jun 10, 2020 at 11:29 PM John Mathews <jmathews3...@gmail.com> wrote: > > Hello, > > I have a custom filesystem that I am trying to migrate to the plugins model > described here: > https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems/#adding-a-new-pluggable-file-system-implementation, > but it is unclear to me how to dynamically get the plugins directory to be > available when launching using a Yarn Cluster Descriptor. One thought was to > add the plugins to the shipFilesList, but I don't think that would result in > the plugins being in the correct directory location for Flink to discover it. > > Is there another way to get the plugins onto the host when launching the > cluster? Or is there a different recommended way of doing this? Happy to > answer any questions if something is unclear. > > Thanks so much for your help! > > John
