[GitHub] [hudi] trushev commented on pull request #5830: [HUDI-3981][RFC-33] Flink engine support for comprehensive schema evolution

2022-11-28 Thread GitBox
trushev commented on PR #5830: URL: https://github.com/apache/hudi/pull/5830#issuecomment-1330175541 It looks like azure doesn't run on this PR anymore. Verifying PR is opened https://github.com/apache/hudi/pull/7321 -- This is an automated message from the Apache Git Service. To resp

[GitHub] [hudi] trushev commented on pull request #5830: [HUDI-3981][RFC-33] Flink engine support for comprehensive schema evolution

2022-11-28 Thread GitBox
trushev commented on PR #5830: URL: https://github.com/apache/hudi/pull/5830#issuecomment-1330080879 CI build failure due to broken master branch. I've pushed the fix https://github.com/apache/hudi/pull/7319 -- This is an automated message from the Apache Git Service. To respond to the me

[GitHub] [hudi] trushev commented on pull request #5830: [HUDI-3981][RFC-33] Flink engine support for comprehensive schema evolution

2022-11-28 Thread GitBox
trushev commented on PR #5830: URL: https://github.com/apache/hudi/pull/5830#issuecomment-1330057955 > Can you scrash and force push here. I didn't see the Azure CI history, let's re-trigger it. Done, waiting for azure -- This is an automated message from the Apache Git Service. To

[GitHub] [hudi] trushev commented on pull request #5830: [HUDI-3981][RFC-33] Flink engine support for comprehensive schema evolution

2022-11-24 Thread GitBox
trushev commented on PR #5830: URL: https://github.com/apache/hudi/pull/5830#issuecomment-1326207485 > Would review it tomorrow, i see there is a conflict, can we resolve it first. https://github.com/apache/hudi/pull/6358 have just been merged. All my tests don't work. Schema evoluti

[GitHub] [hudi] trushev commented on pull request #5830: [HUDI-3981][RFC-33] Flink engine support for comprehensive schema evolution

2022-11-17 Thread GitBox
trushev commented on PR #5830: URL: https://github.com/apache/hudi/pull/5830#issuecomment-1318489360 @danny0405 It is hard to maintain this PR. Despite the fact that this feature is related only to flink, changes are needed in common part. For example, https://github.com/apache/hudi/pul

[GitHub] [hudi] trushev commented on pull request #5830: [HUDI-3981][RFC-33] Flink engine support for comprehensive schema evolution

2022-11-16 Thread GitBox
trushev commented on PR #5830: URL: https://github.com/apache/hudi/pull/5830#issuecomment-1318024566 > > is it possible to fetch the original schema when the file was committed if SE disabled? > > There is no need to fetch the original schema if SE is disabled. > > > Prepare in

[GitHub] [hudi] trushev commented on pull request #5830: [HUDI-3981][RFC-33] Flink engine support for comprehensive schema evolution

2022-11-16 Thread GitBox
trushev commented on PR #5830: URL: https://github.com/apache/hudi/pull/5830#issuecomment-1318006019 > 1. fetch the original schema when the file was committed, read the record as is is it possible to fetch the original schema when the file was committed if SE disabled? > I wo

[GitHub] [hudi] trushev commented on pull request #5830: [HUDI-3981][RFC-33] Flink engine support for comprehensive schema evolution

2022-11-15 Thread GitBox
trushev commented on PR #5830: URL: https://github.com/apache/hudi/pull/5830#issuecomment-1314934901 > What do you mean for the rest cases ? Isn't either schema evolution enabled or disabled ? Even though schema evolution enabled there is might not an `internalSchema`. So we use read

[GitHub] [hudi] trushev commented on pull request #5830: [HUDI-3981][RFC-33] Flink engine support for comprehensive schema evolution

2022-11-14 Thread GitBox
trushev commented on PR #5830: URL: https://github.com/apache/hudi/pull/5830#issuecomment-1314911375 > Either we have some tool for fetching the right avro schema in evolution use cases `avroSchema` does not support custom ids. We use them to build mergedInternalSchema to map old typ

[GitHub] [hudi] trushev commented on pull request #5830: [HUDI-3981][RFC-33] Flink engine support for comprehensive schema evolution

2022-11-14 Thread GitBox
trushev commented on PR #5830: URL: https://github.com/apache/hudi/pull/5830#issuecomment-1314898583 > Actually i'm confused totally by these schema use cases, can we list a summary here, in which case we use writer /reader schema, for schema evolution enabled/disabled ? Schema evolu

[GitHub] [hudi] trushev commented on pull request #5830: [HUDI-3981][RFC-33] Flink engine support for comprehensive schema evolution

2022-11-14 Thread GitBox
trushev commented on PR #5830: URL: https://github.com/apache/hudi/pull/5830#issuecomment-1314709284 > > withInternalSchema( > > Agree So as I understand, you don't mind if I revert all changes not related to flink, which means adding `withInternalSchema` to `HoodieUnMergedLog

[GitHub] [hudi] trushev commented on pull request #5830: [HUDI-3981][RFC-33] Flink engine support for comprehensive schema evolution

2022-11-14 Thread GitBox
trushev commented on PR #5830: URL: https://github.com/apache/hudi/pull/5830#issuecomment-1314705169 @danny0405 In fact, I'd prefer not to replace two schema approach with one scheme approach, as it does not look like part of flink schema evolution feature Moreover, two schema approach s

[GitHub] [hudi] trushev commented on pull request #5830: [HUDI-3981][RFC-33] Flink engine support for comprehensive schema evolution

2022-11-13 Thread GitBox
trushev commented on PR #5830: URL: https://github.com/apache/hudi/pull/5830#issuecomment-1313166445 > [3981.patch.zip](https://github.com/apache/hudi/files/9977268/3981.patch.zip) Thanks for the contribution, have reviewed some of the part, and left a local patch here and some comments ~

[GitHub] [hudi] trushev commented on pull request #5830: [HUDI-3981][RFC-33] Flink engine support for comprehensive schema evolution

2022-11-09 Thread GitBox
trushev commented on PR #5830: URL: https://github.com/apache/hudi/pull/5830#issuecomment-1309655985 @danny0405 rebased -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

[GitHub] [hudi] trushev commented on pull request #5830: [HUDI-3981][RFC-33] Flink engine support for comprehensive schema evolution

2022-11-09 Thread GitBox
trushev commented on PR #5830: URL: https://github.com/apache/hudi/pull/5830#issuecomment-1309651755 @danny0405 it's rebased -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [hudi] trushev commented on pull request #5830: [HUDI-3981][RFC-33] Flink engine support for comprehensive schema evolution

2022-10-23 Thread GitBox
trushev commented on PR #5830: URL: https://github.com/apache/hudi/pull/5830#issuecomment-1288409306 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To