stevenzwu commented on issue #13738: URL: https://github.com/apache/iceberg/issues/13738#issuecomment-3156903363
The analysis makes sense. This is problematic for long-running applications (like streaming jobs). I guess it hasn't been a big problem so far probably because the per-entry overhead is relatively low. From the heap dump, it seems like ~130 bytes per entry. 2M entries (which is a lot) retains <300 MBs, which probably wasn't causing issues for most production setup with plenty of memory. On the other hand, having `deleteOnExit` is a nice safety blanket in case the expected cleanup doesn't complete. This is more useful for local unit test or run. Modern cloud env with ephemeral containers/nodes are less of a concern. Weighing the tradeoffs, I am +1 for removing the `deleteOnExit` call. If we are going to fix this, we probably should also fix the `OSSOutputStream` in aliyun module. Other usages seem to be only in test code (including `RESTCatalogServer` from textFixtures). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
