jkolash commented on issue #13297: URL: https://github.com/apache/iceberg/issues/13297#issuecomment-2992083449
I filed https://issues.apache.org/jira/projects/SPARK/issues/SPARK-52516 and have a WIP branch that I need to test against the latest spark master https://github.com/apache/spark/compare/master...jkolash:spark:fix-memory-leak-completionListener I also created a gist here https://gist.github.com/jkolash/c13d48f97657787068bedc464e0c43c4 to generate synthetic data to reproduce the issue. at least with raw parquet if you were to execute the snapshot procedure https://iceberg.apache.org/docs/nightly/spark-procedures/#snapshot you would be able to reproduce it with iceberg. Some tuning needs to be done to generate synthetic data that doesn't hit other issues but hits this issue and the amount of ram you run the test with. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
