Regards and Thanks, Vardhan Thigle, +919535346204 A small correction, I intended to link to JdbcIO.html <https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/jdbc/JdbcIO.html>
On Sat, May 4, 2024 at 5:48 PM Vardhan Thigle <vardhanvthi...@google.com> wrote: > Hi Beam Experts, > > I had a small query about `JdbcIO.PoolableDataSourceProvider` > > As per main the documentation of JdbcIO > <https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/jdbc/JdbcIO.html>, > (IIUC) `JdbcIO.PoolableDataSourceProvider` creates one DataSource per > execution thread by default which can overwhelm the source db. > > Where As > > As per the Java doc of > <https://beam.apache.org/releases/javadoc/2.29.0/org/apache/beam/sdk/io/jdbc/JdbcIO.PoolableDataSourceProvider.html> > JdbcIO.PoolableDataSourceProvider, > <https://beam.apache.org/releases/javadoc/2.29.0/org/apache/beam/sdk/io/jdbc/JdbcIO.PoolableDataSourceProvider.html> > > > At most a single DataSource instance will be constructed during pipeline > execution for each unique JdbcIO.DataSourceConfiguration > <https://beam.apache.org/releases/javadoc/2.29.0/org/apache/beam/sdk/io/jdbc/JdbcIO.DataSourceConfiguration.html> > within > the pipeline. > > If I want a singleton poolable connection for a given source database and > my pipeline is dealing with multiple source databases, do I need to wrap > the `JdbcIO.PoolableDataSourceProvider` in another concurrent hash map > (from the implementation it looks lit that's what it does already and it's > not needed)?I am a bit confused due to the variation in the 2 docs above > (it's quite possible that I am interpreting them wrong) > Would it be more recommended to rollout a custom class as suggested in the > main documentation of JdbcIO > <https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/jdbc/JdbcIO.ReadWithPartitions.html#withDataSourceConfiguration-org.apache.beam.sdk.io.jdbc.JdbcIO.DataSourceConfiguration->, > in cases like:1. configure the poolconfig 2. Use an alternative source > like say Hikari which If I understand correctly is not possible with > JdbcIO.PoolableDataSourceProvider > <https://beam.apache.org/releases/javadoc/2.29.0/org/apache/beam/sdk/io/jdbc/JdbcIO.PoolableDataSourceProvider.html> > . > > > > > Regards and Thanks, > Vardhan Thigle, > +919535346204 <+91%2095353%2046204> >
