dejii commented on code in PR #38149:
URL: https://github.com/apache/beam/pull/38149#discussion_r3075080795
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/RecordWriterManager.java:
##########
@@ -434,20 +431,6 @@ public void close() throws IOException {
state.dataFiles.clear();
}
} finally {
- // Close unique FileIO instances now that all writers are done.
- // table.io() may return a shared FileIO; we deduplicate by identity
- // so we close each underlying connection pool exactly once.
- Set<FileIO> closedIOs = new HashSet<>();
- for (DestinationState state : destinations.values()) {
- FileIO io = state.table.io();
- if (io != null && closedIOs.add(io)) {
- try {
- io.close();
Review Comment:
> call of teardown is best effort, do not use for side effects. are there
side effects of closing on those io?
There are none that impact correctness. Closing a connection pool is not in
the same category as data writes or transaction commits. If teardown is
skipped, the pool leaks until the worker is recycled - no data loss, no
correctness impact.
This is also the standard pattern for resource cleanup across Beam IOs:
- SpannerIO closes its connection pool in `@Teardown` -
[SpannerIO.java#L2557](https://github.com/apache/beam/blob/41968ad30e4cc9b80b1051d736281559199c6c1a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java#L2557)
- KinesisIO closes its writer in `@Teardown` -
[KinesisIO.java#L855](https://github.com/apache/beam/blob/41968ad30e4cc9b80b1051d736281559199c6c1a/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/KinesisIO.java#L855)
- SqsIO closes its handler in `@Teardown` -
[SqsIO.java#L457](https://github.com/apache/beam/blob/41968ad30e4cc9b80b1051d736281559199c6c1a/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sqs/SqsIO.java#L457)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
