luoyuxia commented on issue #6104: URL: https://github.com/apache/iceberg/issues/6104#issuecomment-1322899747
> This means that in the CDC data that is streaming to Iceberg, don't have a viable data compression scheme for data streams that contain delete operations at this stage? Yes, I'm afraid so. > Here's a question,is it possible to pause the writer for data compression once, and when the data compression is completed, resume the data writing from the checkpoint again, and handle the above commit exception by cyclically suspending, compressing, and writing again? I think it's possible. From the code, IIUC, the exception happens when you start a compression, but it find delete files before finish compression. And once it won't produce any deletes files between start and finish a compression, there shouldn't throw exception. But I'm not sure, you can have a try and to see whether it works. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
