jayceslesar commented on issue #2152: URL: https://github.com/apache/iceberg-python/issues/2152#issuecomment-3067178339
I think this is really hard to do without fully materializing the table and would take a very long time to complete if you are writing a large stream of record batches. We would have to process the entire stream of batches here https://github.com/apache/iceberg-python/blob/dc439402e33d642c9a0c3261f10c078017d6566e/pyiceberg/io/pyarrow.py#L2711 at least, and then re-sink everything to files per partition, then process all of those temporary files that now need to be cleaned up as a stream. You are correct that it does not work with overwrite in its current state. I would really suggest just using a bigger compute runtime if you have to use pyiceberg for an operation like this otherwise use spark -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
