We are not opening/closing connections per query. Each transform (instance) will open a connection.
We currently have no active developers on the Apache Beam implementation but I think that even with pooling it would have limited effect in Flink unless everything is running in the same TaskManager. Kr, Hans On 19 Mar 2025 at 12:13 +0100, arjun s <[email protected]>, wrote: > Hi team, > > Thanks for your reply. > > In my pipeline, multiple queries need to be executed for each row. However, > opening and closing a database connection for each query is causing delays. > Since my source is Kafka, a large number of messages(rows) are continuously > arriving at the same time, leading to processing delays on the Apache Hop > side. We want to reuse the same database connection for processing messages > from the source instead of opening and closing a new connection for each > query within a single row. > I am using the Flink engine to run the pipeline. Is there a way to achieve > connection pooling in Apache Hop, possibly through scripting or a custom > transform? > > Looking forward to your insights. > > Regards, > Arjun S > > On Wed, 19 Mar 2025 at 12:49, Bart Maertens <[email protected]> wrote: > > Hi Arjun, > > > > Can you explain your use case in a bit more detail? > > Each transform in a pipeline opens a connection and keeps it open for the > > duration of the pipeline, or at least for the duration of that transform. > > Connection pooling wouldn't really help in this situation. Also, connection > > pooling usually helps in transactional systems, where the cost of > > opening/closing a connection (only a couple of milliseconds) helps for many > > short transactions. In the case of pipelines that run for at least a couple > > of seconds, the cost of opening/closing a connection usually doesn't make a > > huge difference. > > > > Bart > > > > > On Wed, Mar 19, 2025 at 8:11 AM arjun s <[email protected]> wrote: > > > > Hi team, > > > > > > > > I am currently using Apache Hop version 2.12 and wanted to check if > > > > connection pooling is supported in this version. If not, is there any > > > > plan to introduce connection pooling in future releases? > > > > > > > > Looking forward to your insights. > > > > > > > > Regards, > > > > Arjun S > > > > > > > > > On Tue, 18 Mar 2025 at 12:12, arjun s <[email protected]> wrote: > > > > > > Hi team, > > > > > > > > > > > > Is connection pooling available in Apache Hop? If so, how does > > > > > > connection pooling work in Apache Hop? > > > > > > > > > > > > Thanks and regards, > > > > > > Arjun S
