DanielZhu58 commented on PR #6577: URL: https://github.com/apache/hive/pull/6577#issuecomment-4923064402
I’m generally supportive of this PR. The approach of fixing this on the add-partition/import path is better than only optimizing the read path later. By trying to reuse an existing column descriptor before creating a new CD_ID, this can prevent metadata bloat. I also support that the change is guarded by a new config. A couple of things I think we might need to pay attention later: We only checks a limited set of recent CD_IDs, so we should document that clearly. We should also test it across all supported metastore DBs, (does it work with oracle?) and consider a batch-level cache later to reduce repeated DirectSQL lookups during large imports. We can do the a possible follow-up dedup tool later. Overall, I think this is a good and working fix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
