Re: [PR] adding blog entry on E-commerce Funnel Analysis with StarRocks: 87 Million Records, Apache Hudi, Apache Iceberg, Delta Lake (MinIO, Apache HMS, Apache xTable) [incubator-xtable]

2024-03-01 Thread via GitHub
alberttwong commented on PR #360: URL: https://github.com/apache/incubator-xtable/pull/360#issuecomment-1974239221 closing this PR and opening a clean PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[PR] Add blog post on E-commerce Funnel Analysis with StarRocks: 87 Millio… [incubator-xtable]

2024-03-01 Thread via GitHub
alberttwong opened a new pull request, #361: URL: https://github.com/apache/incubator-xtable/pull/361 …n Records, Apache Hudi, Apache Iceberg, Delta Lake (MinIO, Apache HMS, Apache xTable) ## *Important Read* add blog post ## Brief change log - *Manually verif

Re: [PR] adding blog entry on E-commerce Funnel Analysis with StarRocks: 87 Million Records, Apache Hudi, Apache Iceberg, Delta Lake (MinIO, Apache HMS, Apache xTable) [incubator-xtable]

2024-03-01 Thread via GitHub
alberttwong closed pull request #360: adding blog entry on E-commerce Funnel Analysis with StarRocks: 87 Million Records, Apache Hudi, Apache Iceberg, Delta Lake (MinIO, Apache HMS, Apache xTable) URL: https://github.com/apache/incubator-xtable/pull/360 -- This is an automated message from t

Re: [PR] Remove duplicate dependencies [incubator-xtable]

2024-03-01 Thread via GitHub
wuchunfu commented on code in PR #357: URL: https://github.com/apache/incubator-xtable/pull/357#discussion_r1509834826 ## core/pom.xml: ## @@ -115,11 +115,6 @@ org.apache.spark spark-sql_${scala.version.prefix} - -org.apach

Re: [PR] Remove duplicate dependencies [incubator-xtable]

2024-03-01 Thread via GitHub
ashvina commented on code in PR #357: URL: https://github.com/apache/incubator-xtable/pull/357#discussion_r1509778301 ## core/pom.xml: ## @@ -115,11 +115,6 @@ org.apache.spark spark-sql_${scala.version.prefix} - -org.apache

Re: [PR] Add license header [incubator-xtable]

2024-03-01 Thread via GitHub
wuchunfu commented on PR #359: URL: https://github.com/apache/incubator-xtable/pull/359#issuecomment-1974149136 @ashvina @vinothchandar PTAL, thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Add license header [incubator-xtable]

2024-03-01 Thread via GitHub
wuchunfu commented on PR #359: URL: https://github.com/apache/incubator-xtable/pull/359#issuecomment-1974145086 > Hi @wuchunfu. I checked out your branch and ran spotless on it (mvn spotless:apply). It removed the lines that were added by the plugin you used. This would cause the build to f

Re: [PR] Add license header [incubator-xtable]

2024-03-01 Thread via GitHub
wuchunfu commented on PR #359: URL: https://github.com/apache/incubator-xtable/pull/359#issuecomment-1974143451 > Overall, it looks good to me 👍 I presume you used some tool to add or fix the header. As Vinoth mentioned, was it the RAT plugin? Previously, we used the spotless plugin to form

Re: [PR] Add license header [incubator-xtable]

2024-03-01 Thread via GitHub
ashvina commented on PR #359: URL: https://github.com/apache/incubator-xtable/pull/359#issuecomment-1974140649 Hi @wuchunfu. I checked out your branch and ran spotless on it (mvn spotless:apply). It removed the lines that were added by the plugin you used. This would cause the build to fail

Re: [I] Enable Direct Writing to OneTable Format from Hudi Delta Streamer [incubator-xtable]

2024-03-01 Thread via GitHub
soumilshah1995 commented on issue #351: URL: https://github.com/apache/incubator-xtable/issues/351#issuecomment-1974134461 ``` spark-submit \ --class org.apache.hudi.utilities.streamer.HoodieStreamer \ --packages 'org.apache.hudi:hudi-spark3.4-bundle_2.12:0.14.0' \

Re: [I] Enable Direct Writing to OneTable Format from Hudi Delta Streamer [incubator-xtable]

2024-03-01 Thread via GitHub
soumilshah1995 commented on issue #351: URL: https://github.com/apache/incubator-xtable/issues/351#issuecomment-1974129509 well after lot of headache I got the java working and got the man and build running ``` RNING] Usually this is not harmful and you can skip these warnings, [W

Re: [PR] Add license header [incubator-xtable]

2024-03-01 Thread via GitHub
wuchunfu commented on PR #359: URL: https://github.com/apache/incubator-xtable/pull/359#issuecomment-1974120257 > This is great! Will review deeply. QQ: can we also introduce RAT plugin to complain about lack of license ? @vinothchandar With this in mind, I would like to add a CI to c

Re: [PR] Docs and Website XTable Rename [incubator-xtable]

2024-03-01 Thread via GitHub
vinothchandar commented on code in PR #352: URL: https://github.com/apache/incubator-xtable/pull/352#discussion_r1509707287 ## website/blog/OneTable-is-now-Apache-XTable.md: ## @@ -0,0 +1,67 @@ +--- +title: "OneTable is now “Apache XTable (Incubating)”" +excerpt: "XTable is now

Re: [PR] Docs and Website XTable Rename [incubator-xtable]

2024-03-01 Thread via GitHub
vinothchandar commented on code in PR #352: URL: https://github.com/apache/incubator-xtable/pull/352#discussion_r1509707287 ## website/blog/OneTable-is-now-Apache-XTable.md: ## @@ -0,0 +1,67 @@ +--- +title: "OneTable is now “Apache XTable (Incubating)”" +excerpt: "XTable is now

Re: [PR] Docs and Website XTable Rename [incubator-xtable]

2024-03-01 Thread via GitHub
kywe665 commented on PR #352: URL: https://github.com/apache/incubator-xtable/pull/352#issuecomment-1974082764 Thank you @zabetak and @vinothchandar for the good reviews, I believe I have addressed all of your feedback above -- This is an automated message from the Apache Git Service. To

Re: [PR] Docs and Website XTable Rename [incubator-xtable]

2024-03-01 Thread via GitHub
kywe665 commented on code in PR #352: URL: https://github.com/apache/incubator-xtable/pull/352#discussion_r1509658119 ## website/docusaurus.config.js: ## @@ -45,12 +45,12 @@ const config = { /** @type {import('@docusaurus/preset-classic').ThemeConfig} */ ({ navb

Re: [PR] Docs and Website XTable Rename [incubator-xtable]

2024-03-01 Thread via GitHub
kywe665 commented on code in PR #352: URL: https://github.com/apache/incubator-xtable/pull/352#discussion_r1509657859 ## website/docs/athena.md: ## @@ -4,16 +4,16 @@ title: "Amazon Athena" --- # Querying from Amazon Athena -To read a OneTable synced target table (regardless

Re: [PR] Docs and Website XTable Rename [incubator-xtable]

2024-03-01 Thread via GitHub
kywe665 commented on code in PR #352: URL: https://github.com/apache/incubator-xtable/pull/352#discussion_r1509657276 ## website/blog/OneTable-is-now-Apache-XTable.md: ## @@ -0,0 +1,67 @@ +--- +title: "OneTable is now “Apache XTable”" +excerpt: "XTable is now Incubating in the A

Re: [PR] Docs and Website XTable Rename [incubator-xtable]

2024-03-01 Thread via GitHub
kywe665 commented on code in PR #352: URL: https://github.com/apache/incubator-xtable/pull/352#discussion_r1509642987 ## website/blog/OneTable-is-now-Apache-XTable.md: ## @@ -0,0 +1,67 @@ +--- +title: "OneTable is now “Apache XTable”" +excerpt: "XTable is now Incubating in the A

Re: [PR] Docs and Website XTable Rename [incubator-xtable]

2024-03-01 Thread via GitHub
kywe665 commented on code in PR #352: URL: https://github.com/apache/incubator-xtable/pull/352#discussion_r1509642829 ## website/blog/OneTable-is-now-Apache-XTable.md: ## @@ -0,0 +1,67 @@ +--- +title: "OneTable is now “Apache XTable”" Review Comment: thanks for catching this

Re: [PR] Docs and Website XTable Rename [incubator-xtable]

2024-03-01 Thread via GitHub
kywe665 commented on code in PR #352: URL: https://github.com/apache/incubator-xtable/pull/352#discussion_r1509640582 ## website/README.md: ## @@ -63,4 +63,4 @@ npm run serve ## Maintainers -OneTable Community +XTable Community Review Comment: Good idea, added the link

Re: [PR] Docs and Website XTable Rename [incubator-xtable]

2024-03-01 Thread via GitHub
kywe665 commented on code in PR #352: URL: https://github.com/apache/incubator-xtable/pull/352#discussion_r1509640380 ## README.md: ## @@ -1,15 +1,15 @@ -# OneTable +# XTable Review Comment: Thanks! Updated -- This is an automated message from the Apache Git Service. To

Re: [PR] Docs and Website XTable Rename [incubator-xtable]

2024-03-01 Thread via GitHub
kywe665 commented on code in PR #352: URL: https://github.com/apache/incubator-xtable/pull/352#discussion_r1509640088 ## website/static/images/xtable-svg.svg: ## @@ -0,0 +1,70 @@ +http://www.w3.org/2000/svg";> Review Comment: Hand-crafted from scratch with Figma and exported

Re: [PR] Docs and Website XTable Rename [incubator-xtable]

2024-03-01 Thread via GitHub
kywe665 commented on code in PR #352: URL: https://github.com/apache/incubator-xtable/pull/352#discussion_r1509639970 ## website/README.md: ## @@ -1,6 +1,6 @@ -# OneTable Website Source Code +# XTable Website Source Code -This repo hosts the source code of [OneTable](https://o

Re: [PR] Docs and Website XTable Rename [incubator-xtable]

2024-03-01 Thread via GitHub
kywe665 commented on code in PR #352: URL: https://github.com/apache/incubator-xtable/pull/352#discussion_r1509639834 ## README.md: ## @@ -1,15 +1,15 @@ -# OneTable +# XTable Review Comment: Great catch, thank you! I updated this instance and the first instance in the docs

Re: [PR] Docs and Website XTable Rename [incubator-xtable]

2024-03-01 Thread via GitHub
vinothchandar commented on code in PR #352: URL: https://github.com/apache/incubator-xtable/pull/352#discussion_r1509637956 ## website/docusaurus.config.js: ## @@ -45,12 +45,12 @@ const config = { /** @type {import('@docusaurus/preset-classic').ThemeConfig} */ ({

Re: [PR] adding blog entry on E-commerce Funnel Analysis with StarRocks: 87 Million Records, Apache Hudi, Apache Iceberg, Delta Lake (MinIO, Apache HMS, Apache xTable) [incubator-xtable]

2024-03-01 Thread via GitHub
alberttwong commented on PR #360: URL: https://github.com/apache/incubator-xtable/pull/360#issuecomment-1973993111 I was on an old version and I think with I did the git pull it just auto merged. I'm hoping that it'll do the same and just add the delta. If you want, I can make a clean PR

Re: [I] AWS Glue with Iceberg Source and Delta Target [incubator-xtable]

2024-03-01 Thread via GitHub
MrDerecho commented on issue #354: URL: https://github.com/apache/incubator-xtable/issues/354#issuecomment-1973799886 @the-other-tim-brown - I am also stuck on this particular issue, do you have an example of what this config would look like? Thanks. -- This is an automated message from

Re: [I] AWS Glue with Iceberg Source and Delta Target [incubator-xtable]

2024-03-01 Thread via GitHub
ForeverAngry commented on issue #354: URL: https://github.com/apache/incubator-xtable/issues/354#issuecomment-1973797515 > Yes you will need a catalog yaml to specify which catalog implementation you are using for the Iceberg source. If none is specified, it assumes you're not using an ext

[PR] adding blog entry on E-commerce Funnel Analysis with StarRocks: 87 Million Records, Apache Hudi, Apache Iceberg, Delta Lake (MinIO, Apache HMS, Apache xTable) [incubator-xtable]

2024-03-01 Thread via GitHub
alberttwong opened a new pull request, #360: URL: https://github.com/apache/incubator-xtable/pull/360 ## What is the purpose of the pull request adding a new blog entry ## Verify this pull request *Manually verified the change by running the website locally.* -

Re: [PR] add the ability to support minio and write open table formats in s3 buckets [incubator-xtable]

2024-03-01 Thread via GitHub
alberttwong commented on PR #337: URL: https://github.com/apache/incubator-xtable/pull/337#issuecomment-1973748868 You can try on your end. I'm only sharing what worked for me. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] add the ability to support minio and write open table formats in s3 buckets [incubator-xtable]

2024-03-01 Thread via GitHub
alberttwong closed pull request #337: add the ability to support minio and write open table formats in s3 buckets URL: https://github.com/apache/incubator-xtable/pull/337 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

Re: [PR] add the ability to support minio and write open table formats in s3 buckets [incubator-xtable]

2024-03-01 Thread via GitHub
alberttwong commented on code in PR #337: URL: https://github.com/apache/incubator-xtable/pull/337#discussion_r1509413445 ## utilities/src/main/resources/onetable-hadoop-defaults.xml: ## @@ -72,4 +72,34 @@ false - \ No newline at end of file + + +fs.s3

Re: [PR] add the ability to support minio and write open table formats in s3 buckets [incubator-xtable]

2024-03-01 Thread via GitHub
alberttwong commented on code in PR #337: URL: https://github.com/apache/incubator-xtable/pull/337#discussion_r1509413158 ## utilities/src/main/resources/onetable-hadoop-defaults.xml: ## @@ -72,4 +72,34 @@ false - \ No newline at end of file + Review Comment:

Re: [PR] add the ability to support minio and write open table formats in s3 buckets [incubator-xtable]

2024-03-01 Thread via GitHub
the-other-tim-brown commented on code in PR #337: URL: https://github.com/apache/incubator-xtable/pull/337#discussion_r1509381058 ## utilities/src/main/resources/onetable-hadoop-defaults.xml: ## @@ -72,4 +72,34 @@ false - \ No newline at end of file + + +

Re: [I] AWS Glue with Iceberg Source and Delta Target [incubator-xtable]

2024-03-01 Thread via GitHub
the-other-tim-brown commented on issue #354: URL: https://github.com/apache/incubator-xtable/issues/354#issuecomment-1973688718 Yes you will need a catalog yaml to specify which catalog implementation you are using for the Iceberg source. If none is specified, it assumes you're not using a

Re: [PR] Docs and Website XTable Rename [incubator-xtable]

2024-03-01 Thread via GitHub
vinothchandar commented on code in PR #352: URL: https://github.com/apache/incubator-xtable/pull/352#discussion_r1509332841 ## website/blog/OneTable-is-now-Apache-XTable.md: ## @@ -0,0 +1,67 @@ +--- +title: "OneTable is now “Apache XTable”" Review Comment: ```suggestion t

Re: [PR] Fix google-java-format version mismatch [incubator-xtable]

2024-03-01 Thread via GitHub
wuchunfu closed pull request #358: Fix google-java-format version mismatch URL: https://github.com/apache/incubator-xtable/pull/358 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

Re: [PR] Fix google-java-format version mismatch [incubator-xtable]

2024-03-01 Thread via GitHub
wuchunfu commented on PR #358: URL: https://github.com/apache/incubator-xtable/pull/358#issuecomment-1973576168 > @wuchunfu you are running on java 8? [Readme](https://github.com/apache/incubator-xtable?tab=readme-ov-file#building-the-project-and-running-tests) states to use java 11.

[PR] Add license header [incubator-xtable]

2024-03-01 Thread via GitHub
wuchunfu opened a new pull request, #359: URL: https://github.com/apache/incubator-xtable/pull/359 ## Add license header -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Fix google-java-format version mismatch [incubator-xtable]

2024-03-01 Thread via GitHub
the-other-tim-brown commented on PR #358: URL: https://github.com/apache/incubator-xtable/pull/358#issuecomment-1973523367 @wuchunfu you are running on java 8? [Readme](https://github.com/apache/incubator-xtable?tab=readme-ov-file#building-the-project-and-running-tests) states to use java 1

Re: [I] Enable Direct Writing to OneTable Format from Hudi Delta Streamer [incubator-xtable]

2024-03-01 Thread via GitHub
soumilshah1995 commented on issue #351: URL: https://github.com/apache/incubator-xtable/issues/351#issuecomment-1973494877 Here is Error after adding --enable-sync \ ``` 24/03/01 11:28:00 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20240301112751750__deltaco

[PR] Fix google-java-format version mismatch [incubator-xtable]

2024-03-01 Thread via GitHub
wuchunfu opened a new pull request, #358: URL: https://github.com/apache/incubator-xtable/pull/358 ## Fix the issue of Google Java format version mismatch causing compilation failure https://github.com/apache/incubator-xtable/assets/29206593/a2e36c22-fb4e-44f2-ba6c-97d1a4b200de";>

[PR] Remove duplicate dependencies [incubator-xtable]

2024-03-01 Thread via GitHub
wuchunfu opened a new pull request, #357: URL: https://github.com/apache/incubator-xtable/pull/357 ## Remove duplicate dependencies -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [I] Enable Direct Writing to OneTable Format from Hudi Delta Streamer [incubator-xtable]

2024-03-01 Thread via GitHub
the-other-tim-brown commented on issue #351: URL: https://github.com/apache/incubator-xtable/issues/351#issuecomment-1973456286 The utilities bundle should not be required. You can just use the extensions. Are there any logs from the job that worked? I think you may need to include `--enab

Re: [I] Enable Direct Writing to OneTable Format from Hudi Delta Streamer [incubator-xtable]

2024-03-01 Thread via GitHub
soumilshah1995 commented on issue #351: URL: https://github.com/apache/incubator-xtable/issues/351#issuecomment-1973198967 @the-other-tim-brown if I add both the jar which are provided on GH * hudi-extensions-0.1.0-beta1.jar * utilities-0.1.0-beta1-bundled.jar * ```

Re: [I] Enable Direct Writing to OneTable Format from Hudi Delta Streamer [incubator-xtable]

2024-03-01 Thread via GitHub
soumilshah1995 commented on issue #351: URL: https://github.com/apache/incubator-xtable/issues/351#issuecomment-1973129493 @the-other-tim-brown followed the steps mentioned above Here is Spark submit ``` spark-submit \ --class org.apache.hudi.utilities.streamer.Hoo

Re: [PR] Docs and Website XTable Rename [incubator-xtable]

2024-03-01 Thread via GitHub
zabetak commented on code in PR #352: URL: https://github.com/apache/incubator-xtable/pull/352#discussion_r1508647679 ## website/README.md: ## @@ -1,6 +1,6 @@ -# OneTable Website Source Code +# XTable Website Source Code -This repo hosts the source code of [OneTable](https://o