[ https://issues.apache.org/jira/browse/BEAM-12044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17309566#comment-17309566 ]
Sylvain Veyrié commented on BEAM-12044: --------------------------------------- Updated PR: https://github.com/apache/beam/pull/14349 > JdbcIO should explicitly setAutoCommit to false > ----------------------------------------------- > > Key: BEAM-12044 > URL: https://issues.apache.org/jira/browse/BEAM-12044 > Project: Beam > Issue Type: Bug > Components: sdk-java-core > Affects Versions: 2.28.0 > Reporter: Sylvain Veyrié > Priority: P2 > Time Spent: 2h 50m > Remaining Estimate: 0h > > Hello, > Per [PostgreSQL JDBC > documentation|https://jdbc.postgresql.org/documentation/head/query.html#query-with-cursor], > autocommit must be explicitly disabled on the connection to allow cursor > streaming. > [~jkff] mentionned it [on the mailing > list|https://www.mail-archive.com/dev@beam.apache.org/msg16808.html], however > even if there is: > {code:java} > poolableConnectionFactory.setDefaultAutoCommit(false); > {code} > in JdbcIO:1555, currently, at least with JDBC driver 42.2.16, any read with > JdbcIO will memoize the whole dataset (which leads to OOM), since > {code:java} > connection.getAutoCommit() > {code} > returns true in JdbcIO#ReadFn#processElement. > I can provide a PR — the patch is pretty simple (and solves the problem for > us in 2.28.0): > {code:java} > if (connection == null) { > connection = dataSource.getConnection(); > } > connection.setAutoCommit(false); // line added > {code} > Thanks! -- This message was sent by Atlassian Jira (v8.3.4#803005)