Re: [I] How to move Iceberg table from one location to another [iceberg]
anuragmantri commented on issue #3142: URL: https://github.com/apache/iceberg/issues/3142#issuecomment-2226580307 There is now a PR in review which rewrites metadata with new location and also does some other checks. Please take a look at the PR for some ideas https://github.com/apache/iceberg/pull/10024 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [I] How to move Iceberg table from one location to another [iceberg]
namangoel31 commented on issue #3142: URL: https://github.com/apache/iceberg/issues/3142#issuecomment-2226154660 @cccs-jc, how do you determine the schema for writing to avro? I'm not able to get anything useful. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [I] How to move Iceberg table from one location to another [iceberg]
cccs-jc commented on issue #3142: URL: https://github.com/apache/iceberg/issues/3142#issuecomment-2077081125 @ksmatharoo, thank you for providing your code to replace the paths within the iceberg metadata. I've implemented a similar solution using Python's fastavro to handle Avro files. In my approach, I read the Avro files, replace the paths, and then save the modified files. So far, everything seems to be working well. However, I noticed that your code is more sophisticated, particularly in how you manage lists of known paths. I'm curious if there's a specific reason for this. While my solution is straightforward, I'm wondering if there might be subtle issues that I haven't considered. Any insights you could provide would be appreciated! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [I] How to move Iceberg table from one location to another [iceberg]
ksmatharoo commented on issue #3142: URL: https://github.com/apache/iceberg/issues/3142#issuecomment-2036179649 @atifiu @meicanhong We encountered a similar issue when we switched S3 vendors, necessitating the copying of all data to another bucket, resulting in unreadable iceberg tables. Currently, we have two options: either attempt to rewrite metadata with custom code or inject custom file IO. We have experimented with both approaches, and I can provide you with our code for reference. https://github.com/ksmatharoo/IcebergCustomFileIO https://github.com/ksmatharoo/IcebergMetadataRewrite -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [I] How to move Iceberg table from one location to another [iceberg]
meicanhong commented on issue #3142: URL: https://github.com/apache/iceberg/issues/3142#issuecomment-1882206973 @atifiu I didn't solve this problem. For a discussion on this issue, check out this issue https://github.com/apache/iceberg/issues/1617 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [I] How to move Iceberg table from one location to another [iceberg]
atifiu commented on issue #3142: URL: https://github.com/apache/iceberg/issues/3142#issuecomment-1881540781 @anuragmantri How to rewrite the metadata after copying the files, I have tried using add_files but it does not support transformation on partition column @meicanhong Are you able to solve this ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [I] How to move Iceberg table from one location to another [iceberg]
atifiu commented on issue #3142: URL: https://github.com/apache/iceberg/issues/3142#issuecomment-1881537803 @anuragmantri Are you able to solve this ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
