jkolash commented on issue #13297:
URL: https://github.com/apache/iceberg/issues/13297#issuecomment-2967949805
> Or is the issue here that we are saving this task context for the UI so we
don't actually ever drop the iterator reference and close is never called?
The close() will not likely be called until the job completes. the call back
being held onto is what will do the close() operation and that wont likely
happen until the job completes.
```
context.addTaskCompletionListener[Unit] { _ =>
// In case of early stopping before consuming the entire
iterator,
// we need to do one more metric update at the end of the task.
CustomMetrics
.updateMetrics(reader.currentMetricsValues.toImmutableArraySeq, customMetrics)
iter.forceUpdateMetrics()
reader.close()
}
```
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceRDD.scala#L90
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
