Hello Deniz, Glad to hear that it worked for you! as this is a feature that might benefit others in the community I've just merged this to our main branch, which means that feature releases will have that feature :-) Currently there are no plans to backport this to 3.1 however.
Cheers, Igal. On Mon, Dec 13, 2021 at 3:33 PM Deniz Koçak <lend...@gmail.com> wrote: > Hello Igal, > > First of all, thanks for your effort and feedback on that issue. > We followed the steps you specified and it seems to be working, in > order to briefly summarize what we have done (nothing different > actually you specified on your e-mail) > > 1) Compiled `statefun-flink-distribution` artifact by using your > branch and added that custom build as a dependency to our project. > 2) Used the module definition format here : > > https://nightlies.apache.org/flink/flink-statefun-docs-master/docs/modules/overview/ > 3) We put our env. specific module definition to s3 and simply add the > `additionalDependencies` on Ververica UI as below: > > spec: > artifact: > additionalDependencies: > - > 's3://rttk8s-nxt-v2/vvp/artifacts/namespaces/default/module-drk.yaml' > entryClass: > org.apache.flink.statefun.flink.core.StatefulFunctionsJob > > We also interested in if that feature will be a part of next release > and is there any plan/possibility to backport it to a previous version > like 3.1? At the moment, we are planning to keep this custom build in > our own repo, but it would be very handy to see that change in an > official release. > > Thanks, > Deniz > > On Thu, Dec 9, 2021 at 2:39 PM Igal Shilman <i...@apache.org> wrote: > > > > Hello Deniz, > > > > Looking at /flink/usrlib and the way it is expected to be used, Flink > will only pick up .jar files and include > > them into the classpath, so unfortunately it is being excluded. > > If you want it to just make it work and get on with your day, you can > simply place module.yaml in a separate JAR , otherwise keep on reading :-) > > > > I've created a branch[1], that supports providing a custom name for the > module.yaml, if you are comfortable with building this branch and trying it > out, I can go forward with adding this to statefun, as I believe others > might need a similar functionality. > > > > To make it work you need: > > > > 1. Use @Fabian Paul's advice and upload your custom module.yaml as you > did before, you can also rename it now to whatever name you want. For > example prod.yaml. > > 2. This file will appear at /flink/usrlib/prod.yaml > > 3. You would also need to specify this file by adding the following to > your flink-conf.yaml: > > > > statefun.remote.module-name: /flink/usrlib/prod.yaml > > > > 4. At the bottom of this page [2] you can see a full example, and how to > add additional flink configurations (flinkConfiguration) property. > > > > I hope this helps, > > Igal. > > > > [1] https://github.com/igalshilman/flink-statefun/tree/custom_module > > [2] > https://docs.ververica.com/user_guide/application_operations/deployments/index.html#deployment-defaults > > > > > > On Thu, Dec 9, 2021 at 12:22 PM Deniz Koçak <lend...@gmail.com> wrote: > >> > >> Hi Fabian, > >> > >> Thanks for that solution.. I've removed the module.yaml file from the > >> jar file assuming that it should be fetched from s3 and used by the > >> job. I've tried this on our job, but still its seems to be failing. > >> > >> From the logs module.yaml file seems to be fetched from s3 bucket. > >> ---- > >> com.ververica.platform.artifactfetcher.ArtifactFetcherEntryPoint - > >> Finished fetching > >> s3://rttk8s-nxt-v2/vvp/artifacts/namespaces/default/module.yaml into > >> /flink/usrlib/module.yaml > >> ---- > >> > >> However we got that exception below: > >> > >> ---- > >> Caused by: java.lang.IllegalStateException: There are no ingress > >> defined. at > org.apache.flink.statefun.flink.core.StatefulFunctionsUniverseValidator.validate(StatefulFunctionsUniverseValidator.java:25) > >> ~[?:?] > >> ---- > >> > >> Please let me know if you need further information. Thanks again for > your help. > >> > >> Deniz > >> > >> On Wed, Dec 8, 2021 at 1:20 PM Fabian Paul <fp...@apache.org> wrote: > >> > > >> > Hi Deniz, > >> > > >> > Great to hear from someone using Ververica Platform with StateFun. > >> > When deploying your job you can specify `additionalConfigurations`[1] > >> > that are also pulled and put into the classpath. > >> > > >> > Hopefully, that is suitable for your scenario. > >> > > >> > Best, > >> > Fabian > >> > > >> > [1] > https://docs.ververica.com/user_guide/application_operations/deployments/artifacts.html?highlight=additionaldependencies > >> > > >> > On Fri, Dec 3, 2021 at 4:51 PM Deniz Koçak <lend...@gmail.com> wrote: > >> > > > >> > > Hi Igal, > >> > > > >> > > We are using official images from Ververica as the Flink > installation. > >> > > Actually, I was hoping to specify the name of file names to use > during > >> > > the runtime via `mainArgs` in the deployment configuration (or any > >> > > other way may be). By this way we can specify the target yaml files, > >> > > but I think this is not possible? > >> > > > >> > > ======================= > >> > > kind: JAR > >> > > mainArgs: '--active-profile nxt' > >> > > ======================= > >> > > > >> > > Therefore, it's easier to use single jar in our pipelines instead of > >> > > creating a different jar file for each env. (at least for > development > >> > > and production). > >> > > > >> > > For solution 2, you refer flink distro. , like /flink/lib folder in > >> > > the official Docker image? > >> > > > >> > > Thanks, > >> > > Deniz > >> > > > >> > > On Fri, Dec 3, 2021 at 3:06 PM Igal Shilman <i...@apache.org> > wrote: > >> > > > > >> > > > Hi Deniz, > >> > > > > >> > > > StateFun would be looking for module.yaml(s) in the classpath. > >> > > > If you are submitting the job to an existing Flink cluster this > really means that it needs to be either: > >> > > > 1. packaged with the jar (like you are already doing) > >> > > > 2. be present at the classpath, this means that you can place > your module.yaml at the /lib directory of your Flink installation, I > suppose that you have different installations in different environments. > >> > > > > >> > > > I'm not aware of a way to submit any additional files with the > jar via the flink cli, but perhaps someone else can chime in :-) > >> > > > > >> > > > Cheers, > >> > > > Igal. > >> > > > > >> > > > > >> > > > On Thu, Dec 2, 2021 at 3:29 PM Deniz Koçak <lend...@gmail.com> > wrote: > >> > > >> > >> > > >> Hi, > >> > > >> > >> > > >> We have a simple stateful-function job, consuming from Kafka, > calling > >> > > >> an HTTP endpoint (on AWS via an Elastic Load Balancer) and > publishing > >> > > >> the result back via Kafka again. > >> > > >> > >> > > >> * We created a jar file to be deployed on a standalone cluster > (it's > >> > > >> not a docker Image), therefore we add > `statefun-flink-distribution` > >> > > >> version 3.0.0 as a dependency in that jar file. > >> > > >> * Entry class in our job configuration is > >> > > >> `org.apache.flink.statefun.flink.core.StatefulFunctionsJob` and > we > >> > > >> simply keep a single module.yaml file in resources folder for the > >> > > >> module configuration. > >> > > >> > >> > > >> My question here is, we would like to deploy that jar to > different > >> > > >> environments (dev. and prod.) and not sure how we can pass > different > >> > > >> module configurations (module.yaml or > module_nxt.yaml/module_prd.yaml) > >> > > >> to the job during startup without creating separate jar files for > >> > > >> different environments? > >> > > >> > >> > > >> Thanks, > >> > > >> Deniz >
