Hello Matt Do you have any estimate for https://issues.apache.org/jira/browse/HOP-3601? Thanks Sigalit
On Tue, Jan 4, 2022 at 12:09 AM Matt Casters <[email protected]> wrote: > Hi Sigalit, > > You can point to the main class with the --class > org.apache.hop.beam.run.MainBeam option of flink run. > > Then you need to pass along 3 arguments: > https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-spark-pipeline-engine.html#_running_with_spark_submit > > But like I said, there's some library mismatch in the 1.1.0 snapshot build > I'm fixing. After that I'll update the docs so we continue to Flink as > before. > > All the best, > > Matt > > Op ma 3 jan. 2022 15:30 schreef Sigalit Eliazov <[email protected]>: > >> Thanks a lot for the quick response >> I have one more question please: >> I created a simple pipeline that generates fictitious data and filter >> values. >> i was able to create a far jar from HOP and i deployed it manually to my >> flink cluster (via flink run command) >> the problem i am getting now is >> "org.apache.flink.client.program.ProgramInvocationException: No manifest >> found in jar file '/opt/flink/bin/../lib/my-far.jar'. The manifest is need >> to point to the program's main class." >> >> Is there any configuration/ definition I am missing before generating the >> jar? >> I know that flink requires a main class in order to deploy the job. In >> my java beam pipeline implementation i created those as part of my project >> How do I make sure it exists in the HOP jar? >> >> Thanks >> Sigalit >> >> On Mon, Jan 3, 2022 at 2:54 PM Matt Casters <[email protected]> >> wrote: >> >>> Hi Sigalit, >>> >>> Thanks for testing the Flink runner in Hop and thanks for bringing to >>> our attention that the documentation is not up to spec. >>> >>> https://issues.apache.org/jira/browse/HOP-3599 >>> >>> > Our plan is to modify the pipelines and implement them in HOP. I >>> understand I need to generate a 'far' jar and then deploy it to the flink >>> master. >>> >>> That's right! The process is the same as for Apache Spark documented >>> here: [1] >>> The caveat being that the fat jar can be anywhere that the Beam Flink >>> runner can find it, including in a hdfs:// location. >>> I think that's where the real problem is with the flink run command as >>> some library is forcing a really old hadoop-common library to be included >>> with which there is an issue. >>> >>> https://issues.apache.org/jira/browse/HOP-3600 >>> >>> Finally, it *is* weird that in the Hop GUI it's always running with the >>> embedded engine. I think something might have changed in the Flink runner >>> API. I'll have a look there as well. >>> >>> https://issues.apache.org/jira/browse/HOP-3601 >>> >>> These things shouldn't be hard to fix but I'll keep you in the loop. >>> >>> Cheers, >>> Matt >>> >>> [1] >>> https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-spark-pipeline-engine.html#_running_with_spark_submit >>> >>> -- >>> Neo4j Chief Solutions Architect >>> *✉ *[email protected] >>> >>> >>> >>>
