Kaileshwar16 commented on code in PR #866:
URL: https://github.com/apache/wayang/pull/866#discussion_r4538354838
##########
wayang-platforms/wayang-jdbc-template/src/main/java/org/apache/wayang/jdbc/operators/SqlToStreamOperator.java:
##########
@@ -97,30 +99,50 @@ public Tuple<Collection<ExecutionLineageNode>,
Collection<ChannelInstance>> eval
final StreamChannel.Instance output = (StreamChannel.Instance)
outputs[0];
JdbcPlatformTemplate producerPlatform = (JdbcPlatformTemplate)
input.getChannel().getProducer().getPlatform();
+ final String sqlQuery = input.getSqlQuery();
final Connection connection = producerPlatform
.createDatabaseDescriptor(executor.getConfiguration())
.createJdbcConnection();
- Iterator<Record> resultSetIterator = new ResultSetIterator(connection,
input.getSqlQuery());
- Spliterator<Record> resultSetSpliterator =
Spliterators.spliteratorUnknownSize(resultSetIterator, 0);
- Stream<Record> resultSetStream =
StreamSupport.stream(resultSetSpliterator, false);
+ ResultSetIterator resultSetIterator = new
ResultSetIterator(connection, sqlQuery);
+ boolean success = false;
+ try {
+ Spliterator<Record> resultSetSpliterator =
Spliterators.spliteratorUnknownSize(resultSetIterator, 0);
+ Stream<Record> resultSetStream =
StreamSupport.stream(resultSetSpliterator, false)
+ .onClose(resultSetIterator::close);
- output.accept(resultSetStream);
+ // Terminal stream operations do not close streams. Keep a
job-level fallback for
+ // short-circuiting consumers, without changing StreamChannel's
ownership semantics.
+ CrossPlatformExecutor crossPlatformExecutor =
executor.getCrossPlatformExecutor();
Review Comment:
I understand the concern about coupling the operator directly to
CrossPlatformExecutor, I added this so JDBC resources get cleaned up when the
job ends, even if downstream processing stops early. Stream operations don’t
automatically close the stream, and disposing the channel only clears its
reference.
I also tried registering cleanup with JavaExecutor alone, but live channels
can prevent it from being disposed after a downstream failure. Both new
regression cases failed with that approach. I will add tests and have the
comment more precise. is it justifed to add it? do prefer anyother way to solve
it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]