Kaileshwar16 commented on code in PR #866:
URL: https://github.com/apache/wayang/pull/866#discussion_r4538354838


##########
wayang-platforms/wayang-jdbc-template/src/main/java/org/apache/wayang/jdbc/operators/SqlToStreamOperator.java:
##########
@@ -97,30 +99,50 @@ public Tuple<Collection<ExecutionLineageNode>, 
Collection<ChannelInstance>> eval
         final StreamChannel.Instance output = (StreamChannel.Instance) 
outputs[0];
 
         JdbcPlatformTemplate producerPlatform = (JdbcPlatformTemplate) 
input.getChannel().getProducer().getPlatform();
+        final String sqlQuery = input.getSqlQuery();
         final Connection connection = producerPlatform
                 .createDatabaseDescriptor(executor.getConfiguration())
                 .createJdbcConnection();
 
-        Iterator<Record> resultSetIterator = new ResultSetIterator(connection, 
input.getSqlQuery());
-        Spliterator<Record> resultSetSpliterator = 
Spliterators.spliteratorUnknownSize(resultSetIterator, 0);
-        Stream<Record> resultSetStream = 
StreamSupport.stream(resultSetSpliterator, false);
+        ResultSetIterator resultSetIterator = new 
ResultSetIterator(connection, sqlQuery);
+        boolean success = false;
+        try {
+            Spliterator<Record> resultSetSpliterator = 
Spliterators.spliteratorUnknownSize(resultSetIterator, 0);
+            Stream<Record> resultSetStream = 
StreamSupport.stream(resultSetSpliterator, false)
+                    .onClose(resultSetIterator::close);
 
-        output.accept(resultSetStream);
+            // Terminal stream operations do not close streams. Keep a 
job-level fallback for
+            // short-circuiting consumers, without changing StreamChannel's 
ownership semantics.
+            CrossPlatformExecutor crossPlatformExecutor = 
executor.getCrossPlatformExecutor();

Review Comment:
   I understand the concern about coupling the operator directly to 
CrossPlatformExecutor, I added this so JDBC resources get cleaned up when the 
job ends, even if downstream processing stops early. Stream operations don’t 
automatically close the stream, and disposing the channel only clears its 
reference.
     
   I also tried registering cleanup with JavaExecutor alone, but live channels 
can prevent it from being disposed after a downstream failure. Both new 
regression cases failed with that approach. I will add tests and have the 
comment more precise. is it justifed to add it? do prefer anyother way to solve 
it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to