Thanks for your valuable inputs. To make deploying Flink on K8s easy as a normal Java application is certainly the mission of Flink Kubernetes Operator. Obviously, we are still a little far from this mission.
Back to the user jars download, I think it makes sense to introduce the artifact fetcher in the Flink Kubernetes Operator or directly in the Flink. Then things become quite easier. Users just need to configure the S3 access key and secret. Best, Yang Matt Casters <matt.cast...@neotechnology.com> 于2022年6月22日周三 15:52写道： > Hi Yang, > > Thanks for the suggestion! I looked into this volume sharing on EKS > yesterday but I couldn't figure it out right away. > The way that people come into the Apache Hop project is often with very > little technical knowledge since that's sort of the goal of the project: > make things easy. Following page after page of complicated instructions > just to get a few files into a pod container... I feel it's just a bit > much. > But again, this is my frustration with k8s, not with Flink ;-) > > Cheers, > Matt > > On Wed, Jun 22, 2022 at 5:32 AM Yang Wang <danrtsey...@gmail.com> wrote: > >> Matyas and Gyula have shared many great informations about how to make >> the Flink Kubernetes Operator work on the EKS. >> >> One more input about how to prepare the user jars. If you are more >> familiar with K8s, you could use persistent volume to provide the user jars >> and them mount the volume to JobManager and TaskManager. >> I think the EKS could support EBS, NFS and more other PVs. >> >> Best, >> Yang >> >> Őrhidi Mátyás <matyas.orh...@gmail.com> 于2022年6月21日周二 23:00写道： >> >>> Hi Matt, >>> >>> I believe an artifact fetcher (e.g >>> https://hub.docker.com/r/agiledigital/s3-artifact-fetcher ) + the pod >>> template ( >>> https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/pod-template/#pod-template) >>> is an elegant way to solve your problem. >>> >>> The operator uses K8s native integration under the hood: >>> https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/#application-mode >>> In >>> application mode, the main() method of the application is executed on the >>> JobManager, hence we need the jar locally. >>> >>> You can launch a session cluster (without job spec) on the operator that >>> allows submitting jars if you would like to avoid dealing with >>> authentication, but the recommended and safe approach is to use >>> sessionjobs for this purpose. >>> >>> >>> Cheers, >>> Matyas >>> >>> On Tue, Jun 21, 2022 at 4:03 PM Matt Casters < >>> matt.cast...@neotechnology.com> wrote: >>> >>>> Thank you very much for the help Matyas and Gyula! >>>> >>>> I just saw a video today where you were presenting the FKO. Really >>>> nice stuff! >>>> >>>> So I'm guessing we're executing "flink run" at some point on the master >>>> and that this is when we need the jar file to be local? >>>> Am I right in assuming that this happens after the flink cluster in >>>> question was started, as part of the job execution? >>>> >>>> On the one hand I agree with the underlying idea that authentication >>>> and security should not be a responsibility of the operator. On the other >>>> hand I could add a flink-s3 driver but then I'd also have to configure it >>>> and so on and it's just hard to get that configuration to be really clean. >>>> >>>> Do we have some service running on the flink cluster which would allow >>>> us to post/copy files from the client (running kubectl) to the master? If >>>> so, could we add an option to the job specification to that effect? Just >>>> brainstorming ;-) (and forking apache/flink-kubernetes-operator) >>>> >>>> All the best, >>>> Matt >>>> >>>> On Tue, Jun 21, 2022 at 2:52 PM Őrhidi Mátyás <matyas.orh...@gmail.com> >>>> wrote: >>>> >>>>> Hi Matt, >>>>> >>>>> - In FlinkDeployments you can utilize an init container to download >>>>> your artifact onto a shared volume, then you can refer to it as local:/.. >>>>> from the main container. FlinkDeployments comes with pod template support >>>>> https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/pod-template/#pod-template >>>>> >>>>> - FlinkSessionJobs comes with an artifact fetcher, but it may need >>>>> some tweaking to make it work on your environment: >>>>> >>>>> https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/overview/#flinksessionjob-spec-overview >>>>> >>>>> I hope it helps, let us know if you have further questions. >>>>> >>>>> Cheers, >>>>> Matyas >>>>> >>>>> >>>>> >>>>> On Tue, Jun 21, 2022 at 2:35 PM Matt Casters < >>>>> matt.cast...@neotechnology.com> wrote: >>>>> >>>>>> Hi Flink team! >>>>>> >>>>>> I'm interested in getting the new Flink Kubernetes Operator to work >>>>>> on AWS EKS. Following the documentation I got pretty far. However, when >>>>>> trying to run a job I got the following error: >>>>>> >>>>>> Only "local" is supported as schema for application mode. This >>>>>>> assumes t >>>>>>> hat the jar is located in the image, not the Flink client. An >>>>>>> example of such path is: >>>>>>> local:///opt/flink/examples/streaming/WindowJoin.jar >>>>>> >>>>>> >>>>>> I have an Apache Hop/Beam fat jar capable of running the Flink >>>>>> pipeline in my yml file: >>>>>> >>>>>> jarURI: s3://hop-eks/hop/hop-2.1.0-fat.jar >>>>>> >>>>>> So how could I go about getting the fat jar in a desired location for >>>>>> the operator? >>>>>> >>>>>> Getting this to work would be really cool for both short and >>>>>> long-lived pipelines in the service of all sorts of data integration >>>>>> work. >>>>>> It would do away with the complexity of setting up and maintaining your >>>>>> own >>>>>> Flink cluster. >>>>>> >>>>>> Thanks in advance! >>>>>> >>>>>> All the best, >>>>>> >>>>>> Matt (mcasters, Apache Hop PMC) >>>>>> >>>>>>
