kinolaev commented on PR #15727: URL: https://github.com/apache/iceberg/pull/15727#issuecomment-5554548545
Thanks @szehon-ho for the review! I moved the scan to the executors, parallelized delete manifest reading, and implemented the anti-join using `JavaRDD.subtractByKey()`, so the driver now only collects the dangling deletes (d1558460e3a52ac06d75f0e4cd20b1de6c54e519). My estimate is ~400 bytes per `DeleteFile` without stats, so a million delete files would need around 400MB of memory. The day I first ran this in production, it removed ~250k of ~400k delete files, with peak driver memory consumption at 4.5GB. Over the last week, it removed ~200k of ~300k delete files every night, consuming 3-4GB on average. I'm still running the old version, where delete files are stored with stats. My driver has 16GB of memory, so I suspect the JVM just doesn't bother collecting garbage aggressively, which is probably why the actual numbers are higher than the estimates. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
