Ah got it, thanks! On Thu, Feb 18, 2021 at 10:53 PM Chesnay Schepler <ches...@apache.org> wrote:
> Every works uses exactly 1 connection, so in order to increase the > number of connections you must indeed increase the worker parallelism. > > On 2/19/2021 6:51 AM, Li Peng wrote: > > Hey folks, > > > > I'm trying to use flink to write high throughput incoming data to a > > SQL db using the JDBC Connector as described here: > > > > > https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/jdbc.html > > < > https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/jdbc.html> > > > > > > > However, after enabling this, my data consumption rate slowed down to > > a crawl. After doing some digging, it seemed like the number of > > connections to the db was very low, almost 1 connection per worker > > (there are 9 workers). Which seemed like the culprit if Flink was > > blocking and waiting for a single db connection to do all this work? > > > > Is there a way to tell the JDBC connector to use more db connections? > > Or do I need to specifically increase the parallelism of the connector > > to beyond the default? > > > > And if I need to increase the parallelism, currently my setup is > > having 9 workers, 1 task slot for each (this cluster is dedicated to > > one job). That means I have to increase the number of task slots > > before increasing parallelism, right? > > > > My flink version is 1.10.1 and my jdbc connection > > is flink-connector-jdbc_2.11:1.11.0. > > > > Thanks! > > Li > > >
