Re: [PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data [hudi]

2024-02-24 Thread via GitHub
hudi-bot commented on PR #10736: URL: https://github.com/apache/hudi/pull/10736#issuecomment-1962322165 ## CI report: * 707fc464da051e02301c730b5b5402bbe3bf3a05 Azure:

Re: [PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data [hudi]

2024-02-23 Thread via GitHub
hudi-bot commented on PR #10736: URL: https://github.com/apache/hudi/pull/10736#issuecomment-1962287364 ## CI report: * dbe9cea4f203fe6f056b1f1e1f639e7ad775736c Azure:

Re: [PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data [hudi]

2024-02-23 Thread via GitHub
hudi-bot commented on PR #10736: URL: https://github.com/apache/hudi/pull/10736#issuecomment-1962285984 ## CI report: * dbe9cea4f203fe6f056b1f1e1f639e7ad775736c Azure:

Re: [PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data [hudi]

2024-02-23 Thread via GitHub
hudi-bot commented on PR #10736: URL: https://github.com/apache/hudi/pull/10736#issuecomment-1962009037 ## CI report: * dbe9cea4f203fe6f056b1f1e1f639e7ad775736c Azure:

Re: [PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data [hudi]

2024-02-23 Thread via GitHub
hudi-bot commented on PR #10736: URL: https://github.com/apache/hudi/pull/10736#issuecomment-1961941627 ## CI report: * 2b66c852e373113c8bd1bd66bd0376a8f537044e Azure:

Re: [PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data [hudi]

2024-02-23 Thread via GitHub
hudi-bot commented on PR #10736: URL: https://github.com/apache/hudi/pull/10736#issuecomment-196125 ## CI report: * 2b66c852e373113c8bd1bd66bd0376a8f537044e Azure:

Re: [PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data [hudi]

2024-02-23 Thread via GitHub
hudi-bot commented on PR #10736: URL: https://github.com/apache/hudi/pull/10736#issuecomment-1961715953 ## CI report: * 2b66c852e373113c8bd1bd66bd0376a8f537044e Azure:

Re: [PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data [hudi]

2024-02-23 Thread via GitHub
hudi-bot commented on PR #10736: URL: https://github.com/apache/hudi/pull/10736#issuecomment-1961554948 ## CI report: * 6c41fe5e29de60a4d33701ebfd4aefc898cc605f Azure:

Re: [PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data [hudi]

2024-02-23 Thread via GitHub
hudi-bot commented on PR #10736: URL: https://github.com/apache/hudi/pull/10736#issuecomment-1961542149 ## CI report: * 6c41fe5e29de60a4d33701ebfd4aefc898cc605f Azure:

Re: [PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data [hudi]

2024-02-23 Thread via GitHub
hudi-bot commented on PR #10736: URL: https://github.com/apache/hudi/pull/10736#issuecomment-1960909193 ## CI report: * 6c41fe5e29de60a4d33701ebfd4aefc898cc605f Azure:

Re: [PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data [hudi]

2024-02-22 Thread via GitHub
hudi-bot commented on PR #10736: URL: https://github.com/apache/hudi/pull/10736#issuecomment-1960756910 ## CI report: * 6c41fe5e29de60a4d33701ebfd4aefc898cc605f Azure:

Re: [PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data [hudi]

2024-02-22 Thread via GitHub
hudi-bot commented on PR #10736: URL: https://github.com/apache/hudi/pull/10736#issuecomment-1960752243 ## CI report: * 6c41fe5e29de60a4d33701ebfd4aefc898cc605f UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run

[PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data [hudi]

2024-02-22 Thread via GitHub
vinishjail97 opened a new pull request, #10736: URL: https://github.com/apache/hudi/pull/10736 ### Change Logs There were test failures in the original PR and had to be reverted, bringing back the change and fixed the tests now. https://github.com/apache/hudi/pull/10687

Re: [PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data [hudi]

2024-02-22 Thread via GitHub
linliu-code commented on PR #10687: URL: https://github.com/apache/hudi/pull/10687#issuecomment-1960447292 the failure is caused by a flaky test: https://issues.apache.org/jira/browse/HUDI-7437 -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data [hudi]

2024-02-22 Thread via GitHub
codope commented on PR #10687: URL: https://github.com/apache/hudi/pull/10687#issuecomment-1959747524 The failure in Azure CI is due to test being abandoned which is being investigated separately. It is not related to any of the code change in this PR. -- This is an automated message

Re: [PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data [hudi]

2024-02-22 Thread via GitHub
codope merged PR #10687: URL: https://github.com/apache/hudi/pull/10687 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data [hudi]

2024-02-22 Thread via GitHub
hudi-bot commented on PR #10687: URL: https://github.com/apache/hudi/pull/10687#issuecomment-1959127768 ## CI report: * 78c5f6e801b1c7428326456360dbc4282a9f9703 UNKNOWN * 001eb733b8122b7f4b0bc284d8677ccc73c114eb Azure:

Re: [PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data [hudi]

2024-02-22 Thread via GitHub
hudi-bot commented on PR #10687: URL: https://github.com/apache/hudi/pull/10687#issuecomment-1959045232 ## CI report: * 78c5f6e801b1c7428326456360dbc4282a9f9703 UNKNOWN * dada15ebf1b0b3a9f9c12be092b07aad47f1fcb2 Azure:

Re: [PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data [hudi]

2024-02-22 Thread via GitHub
hudi-bot commented on PR #10687: URL: https://github.com/apache/hudi/pull/10687#issuecomment-1958947957 ## CI report: * 78c5f6e801b1c7428326456360dbc4282a9f9703 UNKNOWN * dada15ebf1b0b3a9f9c12be092b07aad47f1fcb2 Azure:

Re: [PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data [hudi]

2024-02-22 Thread via GitHub
vinishjail97 commented on code in PR #10687: URL: https://github.com/apache/hudi/pull/10687#discussion_r1498842285 ## hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/SourceProfileSupplier.java: ## @@ -0,0 +1,28 @@ +/* + * Licensed to the Apache Software

Re: [PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data [hudi]

2024-02-22 Thread via GitHub
yihua commented on code in PR #10687: URL: https://github.com/apache/hudi/pull/10687#discussion_r1498835956 ## hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/SourceProfileSupplier.java: ## @@ -0,0 +1,28 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data [hudi]

2024-02-21 Thread via GitHub
hudi-bot commented on PR #10687: URL: https://github.com/apache/hudi/pull/10687#issuecomment-1956102876 ## CI report: * 78c5f6e801b1c7428326456360dbc4282a9f9703 UNKNOWN * dada15ebf1b0b3a9f9c12be092b07aad47f1fcb2 Azure:

Re: [PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data [hudi]

2024-02-20 Thread via GitHub
hudi-bot commented on PR #10687: URL: https://github.com/apache/hudi/pull/10687#issuecomment-1955925406 ## CI report: * 78c5f6e801b1c7428326456360dbc4282a9f9703 UNKNOWN * 2d33c841bb67b947626bd4df724df54ea4522c1b Azure:

Re: [PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data [hudi]

2024-02-20 Thread via GitHub
hudi-bot commented on PR #10687: URL: https://github.com/apache/hudi/pull/10687#issuecomment-1955918898 ## CI report: * 78c5f6e801b1c7428326456360dbc4282a9f9703 UNKNOWN * 2d33c841bb67b947626bd4df724df54ea4522c1b Azure:

Re: [PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data [hudi]

2024-02-20 Thread via GitHub
vinishjail97 commented on code in PR #10687: URL: https://github.com/apache/hudi/pull/10687#discussion_r1496874320 ## hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/StreamProfileSupplier.java: ## @@ -0,0 +1,28 @@ +/* + * Licensed to the Apache Software

Re: [PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data [hudi]

2024-02-20 Thread via GitHub
rmahindra123 commented on code in PR #10687: URL: https://github.com/apache/hudi/pull/10687#discussion_r1496727416 ## hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/StreamContext.java: ## @@ -0,0 +1,44 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

Re: [PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data [hudi]

2024-02-20 Thread via GitHub
yihua commented on code in PR #10687: URL: https://github.com/apache/hudi/pull/10687#discussion_r1496284240 ## hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/StreamSync.java: ## @@ -285,7 +285,7 @@ public StreamSync(HoodieStreamer.Config cfg, SparkSession

Re: [PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data [hudi]

2024-02-20 Thread via GitHub
yihua commented on code in PR #10687: URL: https://github.com/apache/hudi/pull/10687#discussion_r1496269281 ## hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/StreamProfileSupplier.java: ## @@ -0,0 +1,28 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data [hudi]

2024-02-20 Thread via GitHub
hudi-bot commented on PR #10687: URL: https://github.com/apache/hudi/pull/10687#issuecomment-1954427258 ## CI report: * 78c5f6e801b1c7428326456360dbc4282a9f9703 UNKNOWN * 2d33c841bb67b947626bd4df724df54ea4522c1b Azure:

Re: [PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data [hudi]

2024-02-20 Thread via GitHub
hudi-bot commented on PR #10687: URL: https://github.com/apache/hudi/pull/10687#issuecomment-1954298971 ## CI report: * 80007841abe42d908d6b6626e46edda7ac2bdc47 Azure:

Re: [PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data [hudi]

2024-02-20 Thread via GitHub
hudi-bot commented on PR #10687: URL: https://github.com/apache/hudi/pull/10687#issuecomment-1954217525 ## CI report: * 80007841abe42d908d6b6626e46edda7ac2bdc47 Azure:

Re: [PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data [hudi]

2024-02-20 Thread via GitHub
hudi-bot commented on PR #10687: URL: https://github.com/apache/hudi/pull/10687#issuecomment-1954201269 ## CI report: * 80007841abe42d908d6b6626e46edda7ac2bdc47 Azure:

Re: [PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data [hudi]

2024-02-20 Thread via GitHub
vinishjail97 commented on code in PR #10687: URL: https://github.com/apache/hudi/pull/10687#discussion_r1495798878 ## hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/Source.java: ## @@ -55,11 +57,17 @@ protected Source(TypedProperties props, JavaSparkContext

Re: [PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data [hudi]

2024-02-19 Thread via GitHub
yihua commented on code in PR #10687: URL: https://github.com/apache/hudi/pull/10687#discussion_r1495114452 ## hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/StreamProfile.java: ## @@ -0,0 +1,46 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data. [hudi]

2024-02-16 Thread via GitHub
hudi-bot commented on PR #10687: URL: https://github.com/apache/hudi/pull/10687#issuecomment-1948880151 ## CI report: * 80007841abe42d908d6b6626e46edda7ac2bdc47 Azure:

Re: [PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data. [hudi]

2024-02-16 Thread via GitHub
hudi-bot commented on PR #10687: URL: https://github.com/apache/hudi/pull/10687#issuecomment-1948458344 ## CI report: * 80007841abe42d908d6b6626e46edda7ac2bdc47 Azure:

Re: [PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data. [hudi]

2024-02-16 Thread via GitHub
hudi-bot commented on PR #10687: URL: https://github.com/apache/hudi/pull/10687#issuecomment-1948446603 ## CI report: * 80007841abe42d908d6b6626e46edda7ac2bdc47 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run

[PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data. [hudi]

2024-02-16 Thread via GitHub
vinishjail97 opened a new pull request, #10687: URL: https://github.com/apache/hudi/pull/10687 ### Change Logs Introducing a new class known as `StreamProfile` which contains details about how the next sync round in StreamSync should be consumed and written. For eg: