Well I wanted to use HOP to avoid programming in Java or Scala😀. Since I am not a skilled Java coder.
On Thu, Jan 27, 2022, 15:02 Maximilian Michels <[email protected]> wrote: > Oh yes! I now remember that Hop uses Beam to run on Flink. Anyhow, the > basic checkpoint options are there. Very true, that we might have to add > any new ones. Maybe a chance for Hans-Peter to contribute? :) > > Cheers, > Max > > On 27.01.22 11:55, Matt Casters wrote: > > You are correct. Hop uses the Flink runner of Apache Beam though and we > > did upgrade to the Flink version 1.13 runner in our latest Hop version > > so there's always the chance that new options became available in Flink > > which are not yet reflected in the runner or in Hop. > > There's really no good communications channel that I know of to signal > > downstream projects of available changes so we just keep an eye out for > > them. > > > > Cheers, > > Matt > > > > Op do 27 jan. 2022 11:49 schreef Maximilian Michels <[email protected] > > <mailto:[email protected]>>: > > > > It looks to me, Hans-Peter's question was not about Beam but Flink. > The > > provided link should answer his question regarding setting the > > checkpoint interval. > > > > -Max > > > > On 27.01.22 11:21, Matt Casters wrote: > > > It can happen that we lag a bit behind when Apache Beam adds > > options for > > > specific runners. > > > So always let us know if there's an option that's not available. > > > In fact, if the option is not available in Beam either we can > > create a > > > change against Apache Beam itself. > > > Cheers, > > > Matt > > > > > > On Thu, Jan 27, 2022 at 10:50 AM Maximilian Michels > > <[email protected] <mailto:[email protected]> > > > <mailto:[email protected] <mailto:[email protected]>>> wrote: > > > > > > Hi Hans-Peter, > > > > > > Checkpoints are internal to Flink. Hop does not have any > > business with > > > it, other than configuring the checkpoint options. Those can > > also be > > > configured via the Flink config file. > > > > > > I see that Hop has checkpointing options listed here: > > > > > > https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options > > < > https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options > > > > > > > < > https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options > < > https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options > >> > > > > > > Cheers, > > > -Max > > > > > > On 23.01.22 19:03, HG wrote: > > > > Hi, > > > > > > > > Perhaps a dumb question. > > > > How is checkpointing implemented when the final framework > > is Flink? > > > > With Flink checkpoint barriers are inserted. > > > > Can such an interval also be configured with HOP? > > > > > > > > Regards Hans-Peter > > > > > > > > > > > > > > > > > >
