Hi Monajit, This is the auto scaling nature of Flink fighting against the requirement of having a single threaded pipeline for Kafka messages (as we need to know when messages are finished. When running on Flink the best solution would be to use the Beam Kafka Consumer.
Another solution (but not yet tested here so not sure it will work) is to force it to a single thread by setting SINGLE_BEAM in the "number of copies". More information about this can be found on our documentation pages [1] Kind regards, Hans [1] https://hop.apache.org/manual/latest/pipeline/beam/getting-started-with-beam.html On Fri, 22 Apr 2022 at 06:50, monajit choudhury <[email protected]> wrote: > Hi, > > I am trying to test a simple kafka consumer using Apache Hop v1.2. When I > run the pipeline using the local runner, it works fine. But if I run it > using the flink runner I get the following error > > You can only have one copy of the injector transform 'output' to accept > the Kafka messages > > I have tried debugging the Hop code and looks like the root cause is the > initSubPipeline() method being invoked multiple times while using the Flink > runner. That's not the case when I use the local runner. Am I missing > something here? > > > Thanks > > Monajit Choudhury > > Linkedin <https://www.linkedin.com/in/monajit-choudhury-b1409a2/> >
