I see! AFAIK the watermark strategy is not configurable via Hop at the
moment. You would have to use the default WatermarkStrategy of the
source you are using.
Cheers,
Max
On 27.01.22 17:13, HG wrote:
Well I wanted to use HOP to avoid programming in Java or Scala😀.
Since I am not a skilled Java coder.
On Thu, Jan 27, 2022, 15:02 Maximilian Michels <[email protected]
<mailto:[email protected]>> wrote:
Oh yes! I now remember that Hop uses Beam to run on Flink. Anyhow, the
basic checkpoint options are there. Very true, that we might have to
add
any new ones. Maybe a chance for Hans-Peter to contribute? :)
Cheers,
Max
On 27.01.22 11:55, Matt Casters wrote:
> You are correct. Hop uses the Flink runner of Apache Beam though
and we
> did upgrade to the Flink version 1.13 runner in our latest Hop
version
> so there's always the chance that new options became available in
Flink
> which are not yet reflected in the runner or in Hop.
> There's really no good communications channel that I know of to
signal
> downstream projects of available changes so we just keep an eye
out for
> them.
>
> Cheers,
> Matt
>
> Op do 27 jan. 2022 11:49 schreef Maximilian Michels
<[email protected] <mailto:[email protected]>
> <mailto:[email protected] <mailto:[email protected]>>>:
>
> It looks to me, Hans-Peter's question was not about Beam but
Flink. The
> provided link should answer his question regarding setting the
> checkpoint interval.
>
> -Max
>
> On 27.01.22 11:21, Matt Casters wrote:
> > It can happen that we lag a bit behind when Apache Beam adds
> options for
> > specific runners.
> > So always let us know if there's an option that's not
available.
> > In fact, if the option is not available in Beam either we can
> create a
> > change against Apache Beam itself.
> > Cheers,
> > Matt
> >
> > On Thu, Jan 27, 2022 at 10:50 AM Maximilian Michels
> <[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>> wrote:
> >
> > Hi Hans-Peter,
> >
> > Checkpoints are internal to Flink. Hop does not have any
> business with
> > it, other than configuring the checkpoint options.
Those can
> also be
> > configured via the Flink config file.
> >
> > I see that Hop has checkpointing options listed here:
> >
>
https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options
<https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options>
>
<https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options <https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options>>
> >
>
<https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options <https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options> <https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options <https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options>>>
> >
> > Cheers,
> > -Max
> >
> > On 23.01.22 19:03, HG wrote:
> > > Hi,
> > >
> > > Perhaps a dumb question.
> > > How is checkpointing implemented when the final
framework
> is Flink?
> > > With Flink checkpoint barriers are inserted.
> > > Can such an interval also be configured with HOP?
> > >
> > > Regards Hans-Peter
> >
> >
> >
> >
> >
>
