Hi team, Thanks for your reply.
In my pipeline, multiple queries need to be executed for each row. However, opening and closing a database connection for each query is causing delays. Since my source is Kafka, a large number of messages(rows) are continuously arriving at the same time, leading to processing delays on the Apache Hop side. We want to reuse the same database connection for processing messages from the source instead of opening and closing a new connection for each query within a single row. I am using the Flink engine to run the pipeline. Is there a way to achieve connection pooling in Apache Hop, possibly through scripting or a custom transform? Looking forward to your insights. Regards, Arjun S On Wed, 19 Mar 2025 at 12:49, Bart Maertens <[email protected]> wrote: > Hi Arjun, > > Can you explain your use case in a bit more detail? > Each transform in a pipeline opens a connection and keeps it open for the > duration of the pipeline, or at least for the duration of that transform. > Connection pooling wouldn't really help in this situation. Also, > connection pooling usually helps in transactional systems, where the cost > of opening/closing a connection (only a couple of milliseconds) helps for > many short transactions. In the case of pipelines that run for at least a > couple of seconds, the cost of opening/closing a connection usually doesn't > make a huge difference. > > Bart > > On Wed, Mar 19, 2025 at 8:11 AM arjun s <[email protected]> wrote: > >> Hi team, >> >> I am currently using Apache Hop version 2.12 and wanted to check if >> connection pooling is supported in this version. If not, is there any plan >> to introduce connection pooling in future releases? >> >> Looking forward to your insights. >> >> Regards, >> Arjun S >> >> On Tue, 18 Mar 2025 at 12:12, arjun s <[email protected]> wrote: >> >>> Hi team, >>> >>> Is connection pooling available in Apache Hop? If so, how does >>> connection pooling work in Apache Hop? >>> >>> Thanks and regards, >>> Arjun S >>> >>
