Hi Sigalit, You can point to the main class with the --class org.apache.hop.beam.run.MainBeam option of flink run.
Then you need to pass along 3 arguments: https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-spark-pipeline-engine.html#_running_with_spark_submit But like I said, there's some library mismatch in the 1.1.0 snapshot build I'm fixing. After that I'll update the docs so we continue to Flink as before. All the best, Matt Op ma 3 jan. 2022 15:30 schreef Sigalit Eliazov <e.siga...@gmail.com>: > Thanks a lot for the quick response > I have one more question please: > I created a simple pipeline that generates fictitious data and filter > values. > i was able to create a far jar from HOP and i deployed it manually to my > flink cluster (via flink run command) > the problem i am getting now is > "org.apache.flink.client.program.ProgramInvocationException: No manifest > found in jar file '/opt/flink/bin/../lib/my-far.jar'. The manifest is need > to point to the program's main class." > > Is there any configuration/ definition I am missing before generating the > jar? > I know that flink requires a main class in order to deploy the job. In my > java beam pipeline implementation i created those as part of my project > How do I make sure it exists in the HOP jar? > > Thanks > Sigalit > > On Mon, Jan 3, 2022 at 2:54 PM Matt Casters <matt.cast...@neo4j.com> > wrote: > >> Hi Sigalit, >> >> Thanks for testing the Flink runner in Hop and thanks for bringing to our >> attention that the documentation is not up to spec. >> >> https://issues.apache.org/jira/browse/HOP-3599 >> >> > Our plan is to modify the pipelines and implement them in HOP. I >> understand I need to generate a 'far' jar and then deploy it to the flink >> master. >> >> That's right! The process is the same as for Apache Spark documented >> here: [1] >> The caveat being that the fat jar can be anywhere that the Beam Flink >> runner can find it, including in a hdfs:// location. >> I think that's where the real problem is with the flink run command as >> some library is forcing a really old hadoop-common library to be included >> with which there is an issue. >> >> https://issues.apache.org/jira/browse/HOP-3600 >> >> Finally, it *is* weird that in the Hop GUI it's always running with the >> embedded engine. I think something might have changed in the Flink runner >> API. I'll have a look there as well. >> >> https://issues.apache.org/jira/browse/HOP-3601 >> >> These things shouldn't be hard to fix but I'll keep you in the loop. >> >> Cheers, >> Matt >> >> [1] >> https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-spark-pipeline-engine.html#_running_with_spark_submit >> >> -- >> Neo4j Chief Solutions Architect >> *✉ *matt.cast...@neo4j.com >> >> >> >>
