[GitHub] [hudi] nsivabalan commented on issue #2284: [SUPPORT] : Is there a option to achieve SCD 2 in Hudi?
nsivabalan commented on issue #2284: URL: https://github.com/apache/hudi/issues/2284#issuecomment-812713158 CC @n3nash -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #2284: [SUPPORT] : Is there a option to achieve SCD 2 in Hudi?
nsivabalan commented on issue #2284: URL: https://github.com/apache/hudi/issues/2284#issuecomment-786611593 Unfortunately, I don't think hudi has support for updating already written records. All we can do is to compare old and new incoming records based on a field (PreCombine) and construct the new payload(akka row). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #2284: [SUPPORT] : Is there a option to achieve SCD 2 in Hudi?
nsivabalan commented on issue #2284: URL: https://github.com/apache/hudi/issues/2284#issuecomment-774317208 I don't think in any code flow we update existing records inplace in hudi as of today. everything is like an append and a newer version of the record. @vinothchandar @bvaradar @n3nash : your thoughts. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #2284: [SUPPORT] : Is there a option to achieve SCD 2 in Hudi?
nsivabalan commented on issue #2284: URL: https://github.com/apache/hudi/issues/2284#issuecomment-773374732 hey folks, let me try to understand your use-case better. I am not aware of SCD2 and found [this](https://adatis.co.uk/introduction-to-slowly-changing-dimensions-scd-types/) through my friend (google ;) ). I will illustrate w/ an example and let me know if I my understanding is right. At t1 (C1 commit) // incoming record recId | name | all cols ... | effective from | effective to - |---||---| rec1 | bob | ..| t1| null.| this record will be stored as is in hudi w/ some additional hudi meta fields recId | name | all cols ... | effective from | effective to| hudi_commit_time | ... other meta fields - |---||---| | --| --- rec1 | bob | ..| t1| null. | t1|. At t5(C2 commit) // incoming record recId | name | all cols ... | effective from | effective to - |---||---| rec1 | bob | ..| t5| null // when we merge this w/ hudi, you want to have the following rows in hudi recId | name | all cols ... | effective from | effective to| hudi_commit_time | ... other meta fields - |---||---| | --| --- rec1 | bob | ..| t1| t5 |t1 |. | rec1 | bob | ..| t5| null| t5 |. | Let me know if this is what you are looking for. We can discuss further. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #2284: [SUPPORT] : Is there a option to achieve SCD 2 in Hudi?
nsivabalan commented on issue #2284: URL: https://github.com/apache/hudi/issues/2284#issuecomment-773374732 hey folks, let me try to understand your use-case better. I am not aware of SCD2 and found [this](https://adatis.co.uk/introduction-to-slowly-changing-dimensions-scd-types/) through my friend (google ;) ). I will illustrate w/ an example and let me know if I my understanding is right. At t1 (C1 commit) // incoming record recId | name | all cols ... | effective from | effective to - |---||---| rec1 | bob | ..| t1| null.| this record will be stored as is in hudi w/ some additional hudi meta fields recId | name | all cols ... | effective from | effective to| hudi_commit_time | ... other meta fields - |---||---| | --| --- rec1 | bob | ..| t1| null. | t1|. At t5(C2 commit) // incoming record recId | name | all cols ... | effective from | effective to - |---||---| rec1 | bob | ..| t5| null // when we merge this w/ hudi, you want to have the following rows in hudi recId | name | all cols ... | effective from | effective to| hudi_commit_time | ... other meta fields - |---||---| | --| --- rec1 | bob | ..| t1| t5 |t1 |. | rec1 | bob | ..| t5| null| t5 |. | Let me know if this is what you are looking for. We can discuss further. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #2284: [SUPPORT] : Is there a option to achieve SCD 2 in Hudi?
nsivabalan commented on issue #2284: URL: https://github.com/apache/hudi/issues/2284#issuecomment-766436747 @sanket-khedikar : can you please respond if the suggested approaches work for you. or you still need more enhancements from Hudi? If it's solved, would appreciate if you can close this ticket. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #2284: [SUPPORT] : Is there a option to achieve SCD 2 in Hudi?
nsivabalan commented on issue #2284: URL: https://github.com/apache/hudi/issues/2284#issuecomment-766436747 @sanket-khedikar : can you please respond if the suggested approaches work for you. or you still need more enhancements from Hudi? If it's solved, would appreciate if you can close this ticket. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
