tkaymak commented on code in PR #39253:
URL: https://github.com/apache/beam/pull/39253#discussion_r3579861146
##########
sdks/java/io/jms/src/main/java/org/apache/beam/sdk/io/jms/JmsIO.java:
##########
@@ -783,21 +896,43 @@ public void close() {
private void doClose() {
try {
closeAutoscaler();
- closeConsumer();
- ScheduledExecutorService executorService =
- options.as(ExecutorOptions.class).getScheduledExecutorService();
- executorService.schedule(
- () -> {
- LOG.debug("Closing connection after delay {}",
source.spec.getCloseTimeout());
- // Discard the checkpoints and set the reader as inactive
- checkpointMarkPreparer.discard();
- closeSession();
- closeConnection();
- },
- source.spec.getCloseTimeout().getMillis(),
- TimeUnit.MILLISECONDS);
+ // Discard the checkpoints and set the reader as inactive
+ checkpointMarkPreparer.discard();
+ if (source.spec.getAcknowledgeMode() ==
AcknowledgeMode.CLIENT_ACKNOWLEDGE) {
+ // checkpointMark holds session in CLIENT_ACKNOWLEDGE mode. Therefore
+ // we can close consumer and session immediately.
+ closeConsumer();
+ closeSession();
+ }
+ if (activeCheckpoints.get() == 0) {
+ closeConsumer();
+ closeSession();
+ closeConnection();
+ } else {
+ ScheduledExecutorService executorService =
+ options.as(ExecutorOptions.class).getScheduledExecutorService();
+ executorService.submit(
+ () -> {
+ long startTime = System.currentTimeMillis();
+ long timeoutMillis = source.spec.getCloseTimeout().getMillis();
+ while (activeCheckpoints.get() > 0
+ && System.currentTimeMillis() - startTime < timeoutMillis)
{
+ try {
+ Thread.sleep(1_000); // poll in 1 sec interval
+ } catch (InterruptedException ignored) {
+ break;
+ }
+ }
+ LOG.debug(
+ "Closing connection after checkpoints finalized or
timeout: {}",
+ source.spec.getCloseTimeout());
+ closeConsumer();
+ closeSession();
+ closeConnection();
+ });
Review Comment:
The close path `submit()`s a while (...) `Thread.sleep(1000)` loop onto the
shared `ExecutorOptions` scheduled pool. On any runner where checkpoint
finalization is best effort or delayed, the count may never reach 0, so the
loop runs up to the full `closeTimeout` (default 60s).
When many readers close at once, for example on scale down, that parks a lot
of pool threads and can starve other scheduled work. I am not sure how often
Dataflow drops finalization in practice, you would know better, but even
setting that aside the busy wait on a shared pool seems worth avoiding.
Suggest rescheduling via `schedule()` with a short period, or having the
last `finalizeCheckpoint` decrement trigger the close when a closed flag is set
(last one out), with a single `schedule(closeTimeout)` fallback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
