Kaileshwar16 commented on code in PR #866:
URL: https://github.com/apache/wayang/pull/866#discussion_r4538354838
##########
wayang-platforms/wayang-jdbc-template/src/main/java/org/apache/wayang/jdbc/operators/SqlToStreamOperator.java:
##########
@@ -97,30 +99,50 @@ public Tuple<Collection<ExecutionLineageNode>,
Collection<ChannelInstance>> eval
final StreamChannel.Instance output = (StreamChannel.Instance)
outputs[0];
JdbcPlatformTemplate producerPlatform = (JdbcPlatformTemplate)
input.getChannel().getProducer().getPlatform();
+ final String sqlQuery = input.getSqlQuery();
final Connection connection = producerPlatform
.createDatabaseDescriptor(executor.getConfiguration())
.createJdbcConnection();
- Iterator<Record> resultSetIterator = new ResultSetIterator(connection,
input.getSqlQuery());
- Spliterator<Record> resultSetSpliterator =
Spliterators.spliteratorUnknownSize(resultSetIterator, 0);
- Stream<Record> resultSetStream =
StreamSupport.stream(resultSetSpliterator, false);
+ ResultSetIterator resultSetIterator = new
ResultSetIterator(connection, sqlQuery);
+ boolean success = false;
+ try {
+ Spliterator<Record> resultSetSpliterator =
Spliterators.spliteratorUnknownSize(resultSetIterator, 0);
+ Stream<Record> resultSetStream =
StreamSupport.stream(resultSetSpliterator, false)
+ .onClose(resultSetIterator::close);
- output.accept(resultSetStream);
+ // Terminal stream operations do not close streams. Keep a
job-level fallback for
+ // short-circuiting consumers, without changing StreamChannel's
ownership semantics.
+ CrossPlatformExecutor crossPlatformExecutor =
executor.getCrossPlatformExecutor();
Review Comment:
I added this so JDBC resources get cleaned up when the job ends, even if
downstream processing stops early. Stream operations don’t automatically close
the stream, and disposing the channel only clears its reference.
I also tried registering cleanup with JavaExecutor alone, but live channels
can prevent it from being disposed after a downstream failure. Both new
regression cases failed with that approach. I will add tests and have the
comment more precise. is it justifed to add it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]