RussellSpitzer commented on issue #13297: URL: https://github.com/apache/iceberg/issues/13297#issuecomment-2968134018
Ok - for my own recap here 1. Spark is holding onto task contexts in order to invoke callbacks to get metric information till the end of the job 2. Task Context through this callback holds onto the iterator the DSV2 source makes 3. Our iterator ends up being slightly heavy, it holds onto a reader for every column in the table which means for a very wide table we end up holding onto a parquet reader for every column in the table. This ends up adding up to a significant amount of memory. Now I think there are a few avenues to make this better 1. As @jkolash suggests we can preemptively null out the reference to the readers when the iterator has been exhausted. One issue here is that a partially consumed iterator will still hold onto the model until the job has completed. 2. We can work on the Spark side to make sure that the iterator is actually released when exhausted or the task has completed. Thoughts @szehon-ho + @aokolnychyi ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
