GitHub user eleonora011 created a discussion: Using Apache Hop integrated with Kafka
I’d like to briefly explain the workflow I developed in Apache Hop and the issues I’m encountering. The flow works as follows: It starts with a Kafka Consumer that reads messages from a specific topic. This task triggers a specific pipeline, which begins with an Injector. Then, to properly format each row, I use a JSON Input step and several Select Values steps. My goal is to enrich the message coming from Kafka with data from the database. So, after a Table Input, I perform a Merge (between the data coming from Kafka and the data coming from the database), and finally, I output the result to a text file. The problem I’ve encountered is that the flow stops at the Merge step. The behavior is quite strange: if I pass a finite number of rows to the pipeline, it completes successfully; however, if I use a Generate Rows step (with rows generated at regular time intervals) or a Kafka Consumer (both of which can be considered “infinite” sources ) the flow freezes. I’d like to know whether this issue has already been experienced by others, and if so, how it can be resolved or worked around. Thank you GitHub link: https://github.com/apache/hop/discussions/5961 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
