kazdy commented on issue #7282: URL: https://github.com/apache/hudi/issues/7282#issuecomment-1327328186
I think the goal here would be to have good support for users who want to have something that resembles a table in DWH/RDBMS but use Hudi. I've seen some new users thinking it's a bug that preCombine is required and it was confusing at first. So happy path would be to make sure Spark SQL supports a model with PK and no precombine by default. This would hopefully increase adoption and make the learning curve shallow. For more advanced users, it's still great to have the option to use precombine as it's useful for incremental/streaming/cdc pipelines and event deduplication. Besides what we're missing is a modeling guide for Hudi datasets, it forces users to make certain assumptions about how hudi data pipelines should work (which is expected for a framework), as precombine and pk constraints are given. So a part of this work would be to provide a guide for: 1. pk + no_precombine model 2. pk + precombine model With clearly described pros and cons of both, when to use what, and how. Describe how to handle surrogate/factless keys for datasets where PK can not be clearly defined and so on. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
