[
https://issues.apache.org/jira/browse/BEAM-3506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333404#comment-16333404
]
Jean-Baptiste Onofré commented on BEAM-3506:
--------------------------------------------
That sounds interesting. By the way, you can tweak the batch size of the
{{WriteFn}} (there's an option for that). In the case of
{{PCollection<Iterable<T>>}} , I guess you are assuming the batch size will be
1 in the {{WriteFn}}, right ? So, you might want to have different batch size
"per collection" ?
> JdbcIO: Support writing iterables (i.e. collections) of rows instead of only
> single rows
> ----------------------------------------------------------------------------------------
>
> Key: BEAM-3506
> URL: https://issues.apache.org/jira/browse/BEAM-3506
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-extensions
> Affects Versions: 2.3.0
> Reporter: Knut Olav Loite
> Assignee: Jean-Baptiste Onofré
> Priority: Minor
> Labels: JdbcIO, jdbc
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> The current JdbcIO write interface expects a PCollection<T> where T is the
> row to be written. Each instance of T is then added to a batch and written to
> the database. The user has little control over how many rows will be added to
> one batch. If JdbcIO would also support writing a PCollection<Iterable<T>>
> the user would have more control over the number of rows in one batch.
> Especially when writing to cloud databases, such as Google Cloud Spanner, the
> batching of multiple rows together is important for performance.
> I already have a solution locally and I will submit a pull request.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
