Hi Hans, Going through the log files I realized it had something to do with multithreaded executions. I tried using the Beam Kafka Consumer but the issue is it only supports AVRO. I need to consume json messages
Thanks Mono On Fri, Apr 22, 2022 at 12:21 AM Hans Van Akelyen < [email protected]> wrote: > Hi Monajit, > > This is the auto scaling nature of Flink fighting against the requirement > of having a single threaded pipeline for Kafka messages (as we need to know > when messages are finished. When running on Flink the best solution would > be to use the Beam Kafka Consumer. > > Another solution (but not yet tested here so not sure it will work) is to > force it to a single thread by setting SINGLE_BEAM in the "number of > copies". > More information about this can be found on our documentation pages [1] > > Kind regards, > Hans > > [1] > https://hop.apache.org/manual/latest/pipeline/beam/getting-started-with-beam.html > > On Fri, 22 Apr 2022 at 06:50, monajit choudhury <[email protected]> > wrote: > >> Hi, >> >> I am trying to test a simple kafka consumer using Apache Hop v1.2. When I >> run the pipeline using the local runner, it works fine. But if I run it >> using the flink runner I get the following error >> >> You can only have one copy of the injector transform 'output' to accept >> the Kafka messages >> >> I have tried debugging the Hop code and looks like the root cause is the >> initSubPipeline() method being invoked multiple times while using the Flink >> runner. That's not the case when I use the local runner. Am I missing >> something here? >> >> >> Thanks >> >> Monajit Choudhury >> >> Linkedin <https://www.linkedin.com/in/monajit-choudhury-b1409a2/> >> >
